Repository navigation
numba / dask defensive hardening #1804
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
cmdupuis3
wants to merge
21
commits into
UXARRAY:main
Choose a base branch
from
cmdupuis3:cmd/h1h2
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
71ef47b
CI clears numba cache
cmdupuis3 92cc4fc
Guard numba flags the compensated EFT kernels depend on
cmdupuis3 4815763
Merge branch 'main' into cmd/h1h2
cmdupuis3 250018f
Merge branch 'main' into cmd/h1h2
cmdupuis3 0948ce8
Merge branch 'main' into cmd/h1h2
cmdupuis3 589fffb
Merge branch 'main' into cmd/h1h2
cmdupuis3 aa10a3c
Merge branch 'main' into cmd/h1h2
cmdupuis3 355a8cb
h1h2: some comments
cmdupuis3 4a8aa28
h1h2: fix false positive on
cmdupuis3 d234d35
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 730f55a
Merge branch 'main' into cmd/h1h2
cmdupuis3 5e7c664
h1h2: more comments
cmdupuis3 4a472d7
Merge branch 'main' into cmd/h1h2
cmdupuis3 dae7554
Merge branch 'main' into cmd/h1h2
cmdupuis3 c6cf41a
Address review comments on numba flag tests
cmdupuis3 9020f89
Expand documentation of numba flag tests
cmdupuis3 0e40fc2
Expand the fastmath assertion message in numba flag tests
cmdupuis3 cd00287
Merge branch 'main' into cmd/h1h2
cmdupuis3 010e232
Merge branch 'main' into cmd/h1h2
cmdupuis3 1607aec
Clear stale numba caches at pytest startup instead of in pre-commit
cmdupuis3 b6ba499
Test that SHARED_MODULES covers every cross-module kernel call
cmdupuis3 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| """Delete numba's on-disk cache for ``uxarray``. | ||
|
|
||
| Numba stamps each ``cache=True`` index (``*.nbi``) with the ``(mtime, size)`` of the | ||
| source file that *defines* the function, and nothing else. A kernel that calls a jitted | ||
| function from another module has that callee's code linked into its own cached object, | ||
| so editing the callee leaves the caller's cache entry "valid" with the old body baked in. | ||
|
|
||
| The test suite calls ``clear_if_shared_modules_changed`` at session start, which clears | ||
| the cache only when the jitted modules that others call into (``SHARED_MODULES``) differ | ||
| from the last run. Outside pytest (notebooks, scripts), run this by hand after editing | ||
| one of them:: | ||
|
|
||
| python ci/clean_numba_cache.py [--dry-run] | ||
| """ | ||
|
|
||
| import argparse | ||
| import hashlib | ||
| import os | ||
| from pathlib import Path | ||
|
|
||
| PACKAGE_DIR = Path(__file__).resolve().parent.parent / "uxarray" | ||
| CACHE_SUFFIXES = {".nbi", ".nbc"} | ||
| # Jitted modules whose kernels are compiled into kernels of *other* modules. Add a module | ||
| # here when another module's jitted code starts calling into it; test_numba_flags.py's | ||
| # test_shared_modules_complete fails if one is missing. | ||
| SHARED_MODULES = [ | ||
| "utils/computing.py", | ||
| "utils/numba_math.py", | ||
| "grid/angles.py", | ||
| "grid/arcs.py", | ||
| "grid/area.py", | ||
| "grid/coordinates.py", | ||
| "grid/geometry.py", | ||
| "grid/intersections.py", | ||
| "grid/point_in_face.py", | ||
| "grid/utils.py", | ||
| ] | ||
| # Digest of SHARED_MODULES as of the last clear; __pycache__ keeps it out of git. | ||
| STAMP = PACKAGE_DIR / "__pycache__" / "numba_shared_modules.sha256" | ||
| # Numba files caches in NUMBA_CACHE_DIR under a hash of each source directory, so they | ||
| # can't be told apart from other projects' entries; leave them alone rather than guess. | ||
| EXTERNAL_CACHE_WARNING = ( | ||
| "NUMBA_CACHE_DIR is set, so uxarray's numba cache lives there and was not cleared " | ||
| "(to avoid touching other projects' entries); clear it by hand." | ||
| ) | ||
|
|
||
|
|
||
| def _cache_files(): | ||
| """Yield numba cache files stored next to the package's sources.""" | ||
| for path in PACKAGE_DIR.rglob("__pycache__/*"): | ||
| if path.suffix in CACHE_SUFFIXES: | ||
| yield path | ||
|
|
||
|
|
||
| def _shared_modules_digest(): | ||
| digest = hashlib.sha256() | ||
| for name in SHARED_MODULES: | ||
| path = PACKAGE_DIR / name | ||
| digest.update(name.encode()) | ||
| digest.update(path.read_bytes() if path.exists() else b"<missing>") | ||
| return digest.hexdigest() | ||
|
|
||
|
|
||
| def _write_stamp(digest): | ||
| STAMP.parent.mkdir(exist_ok=True) | ||
| STAMP.write_text(digest) | ||
|
|
||
|
|
||
| def clear(dry_run=False): | ||
| """Delete every cache file; return how many there were.""" | ||
| files = sorted(set(_cache_files())) | ||
| for path in files: | ||
| if dry_run: | ||
| print(path) | ||
| else: | ||
| path.unlink(missing_ok=True) | ||
|
|
||
| if not dry_run: | ||
| _write_stamp(_shared_modules_digest()) | ||
| return len(files) | ||
|
|
||
|
|
||
| def clear_if_shared_modules_changed(): | ||
| """Clear the cache if SHARED_MODULES changed since the last clear (or there is no | ||
| record of one); return the number of files removed, or None if nothing changed. | ||
| """ | ||
| digest = _shared_modules_digest() | ||
| try: | ||
| if STAMP.read_text() == digest: | ||
| return None | ||
| except OSError: | ||
| pass | ||
| return clear() | ||
|
|
||
|
|
||
| def main(argv=None): | ||
| parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) | ||
| parser.add_argument( | ||
| "-n", | ||
| "--dry-run", | ||
| action="store_true", | ||
| help="list what would be removed without deleting anything", | ||
| ) | ||
| args = parser.parse_args(argv) | ||
|
|
||
| removed = clear(dry_run=args.dry_run) | ||
| if os.environ.get("NUMBA_CACHE_DIR"): | ||
| print(f"WARNING: {EXTERNAL_CACHE_WARNING}") | ||
| verb = "would remove" if args.dry_run else "removed" | ||
| print(f"{verb} {removed} numba cache file(s)") | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| """Guard the numba flags that uxarray's compensated (EFT) kernels depend on. | ||
|
|
||
| A kernel declared ``inline="always"`` is compiled with its *caller's* flags, not its own, | ||
| so every kernel it is inlined into, directly or transitively, needs the same flags. | ||
| Taking ``acc_sqrt_re`` (``uxarray.utils.computing``) as the example: | ||
|
|
||
| - ``error_model="numpy"``: division follows IEEE 754, so ``acc_sqrt_re(0.0)`` returns | ||
| ``(0.0, nan)`` (the correction is 0/0) and the status layer's isfinite mask rejects it. | ||
| Under numba's default ``error_model="python"`` the same call raises ZeroDivisionError. | ||
| (A negative input gives ``(nan, nan)`` under either model; only division raises.) | ||
| - no ``fastmath``: compensated arithmetic relies on IEEE 754 forbidding reassociation. | ||
| ``fastmath`` lets LLVM simplify two_sum's ``e = (a - (s - bp)) + (b - bp)``, with | ||
| ``s = a + b``, to 0, discarding the rounding error the term exists to recover: | ||
| ``acc_sqrt_re(10.0)`` returns correction -1.9e-16 normally, 0.0 under fastmath. | ||
|
|
||
| ``test_numpy_error_model`` checks every uxarray kernel that inlines one of | ||
| ``NUMPY_ERROR_MODEL_KERNELS``; ``test_no_fastmath`` checks every kernel that inlines | ||
| anything from ``uxarray.utils.computing``. ``test_shared_modules_complete`` checks that | ||
| the cache clearing in test/conftest.py watches every module whose kernels are compiled | ||
| into another module's. | ||
| """ | ||
|
|
||
| import ast | ||
| import importlib | ||
| import importlib.util | ||
| import inspect | ||
| import pkgutil | ||
| import sys | ||
| import textwrap | ||
| from pathlib import Path | ||
|
|
||
| import pytest | ||
| from numba.core.registry import CPUDispatcher | ||
|
|
||
| import uxarray | ||
|
|
||
| EFT_MODULE = "uxarray.utils.computing" | ||
| # The kernels whose own arithmetic relies on 1/0 -> inf and 0/0 -> nan. This is the | ||
| # list of *sources*: the tests scan every kernel in uxarray and flag any one these are | ||
| # inlined into that lacks error_model="numpy". It has to be maintained by hand, because | ||
| # a kernel's own flags can't tell us this: e.g. ``_try_gca_gca_intersection`` sets | ||
| # error_model="numpy" only because it inlines ``_accux_gca``, not because it needs it. | ||
| NUMPY_ERROR_MODEL_KERNELS = { | ||
| "uxarray.utils.computing.acc_sqrt_re", | ||
| "uxarray.grid.intersections._accux_gca", | ||
| "uxarray.grid.intersections._accux_constlat", | ||
| } | ||
|
|
||
|
|
||
| def _qualname(kernel): | ||
| return f"{kernel.py_func.__module__}.{kernel.py_func.__name__}" | ||
|
|
||
|
|
||
| def _callees(kernel): | ||
| """Yield the jitted functions called by name in ``kernel``'s body.""" | ||
| namespace = kernel.py_func.__globals__ | ||
| for node in ast.walk(ast.parse(textwrap.dedent(inspect.getsource(kernel.py_func)))): | ||
| if not isinstance(node, ast.Call): | ||
| continue | ||
|
|
||
| func = node.func | ||
| # A direct function call (e.g., print()) | ||
| if isinstance(func, ast.Name): | ||
|
jayeshkrishna marked this conversation as resolved.
Resolved
|
||
| target = namespace.get(func.id) | ||
| # An attribute method call (e.g. val.print()) | ||
| elif isinstance(func, ast.Attribute) and isinstance(func.value, ast.Name): | ||
|
jayeshkrishna marked this conversation as resolved.
Resolved
|
||
| target = getattr(namespace.get(func.value.id), func.attr, None) | ||
| else: | ||
| continue | ||
|
|
||
| # Only consider Numba JIT func calls | ||
| if isinstance(target, CPUDispatcher): | ||
|
jayeshkrishna marked this conversation as resolved.
Resolved
|
||
| yield target | ||
|
|
||
|
|
||
| def _get_inlined_funcs_in_call_tree(kernel): | ||
| """Names of ``kernel`` and every kernel inlined into it, all built with its flags.""" | ||
| names, stack = {_qualname(kernel)}, [kernel] | ||
| while stack: | ||
| for callee in _callees(stack.pop()): | ||
| name = _qualname(callee) | ||
| if callee.targetoptions.get("inline") == "always" and name not in names: | ||
| names.add(name) | ||
| stack.append(callee) | ||
| return names | ||
|
|
||
|
|
||
| @pytest.fixture(scope="module") | ||
| def kernels(): | ||
| """Map each module-level kernel in uxarray to ``(kernel, names compiled into it)``.""" | ||
| found = {} | ||
| for info in pkgutil.walk_packages(uxarray.__path__, "uxarray."): | ||
| try: | ||
| module = importlib.import_module(info.name) | ||
| except ImportError: # missing optional dependency: its kernels can't run either | ||
| continue | ||
| for obj in vars(module).values(): | ||
| if isinstance(obj, CPUDispatcher) and obj.py_func.__module__ == info.name: | ||
| found[_qualname(obj)] = (obj, _get_inlined_funcs_in_call_tree(obj)) | ||
| return found | ||
|
|
||
|
|
||
| def test_numpy_error_model(kernels): | ||
|
jayeshkrishna marked this conversation as resolved.
Resolved
|
||
| """Every kernel that inlines a NUMPY_ERROR_MODEL_KERNELS entry sets error_model="numpy".""" | ||
| # If a source kernel is renamed, moved or fails to import, nothing would match it | ||
| # below and this test would pass without checking anything. | ||
| missing = NUMPY_ERROR_MODEL_KERNELS - kernels.keys() | ||
| assert not missing, f"NUMPY_ERROR_MODEL_KERNELS names unknown kernels: {missing}" | ||
|
|
||
| offenders = sorted( | ||
| name | ||
|
jayeshkrishna marked this conversation as resolved.
Resolved
|
||
| for name, (kernel, is_compiled) in kernels.items() | ||
| if is_compiled & NUMPY_ERROR_MODEL_KERNELS # inlines a kernel that needs it... | ||
| and kernel.targetoptions.get("error_model") != "numpy" # ...but doesn't set it | ||
| ) | ||
| assert not offenders, ( | ||
| f"Missing error_model='numpy' on {offenders}: they inline kernels that rely on " | ||
| f"1/0 -> inf and 0/0 -> nan, which raise ZeroDivisionError under the default model" | ||
| ) | ||
|
|
||
|
|
||
| def test_no_fastmath(kernels): | ||
|
jayeshkrishna marked this conversation as resolved.
Resolved
|
||
| """No kernel that inlines EFT code from uxarray.utils.computing enables fastmath.""" | ||
| eft = { | ||
| name: kernel | ||
| for name, (kernel, compiled) in kernels.items() | ||
| if any(n.startswith(EFT_MODULE + ".") for n in compiled) | ||
| } | ||
| assert eft, f"No kernels found in {EFT_MODULE}; was it renamed?" # else vacuous | ||
| offenders = [ | ||
| name for name, kernel in eft.items() if kernel.targetoptions.get("fastmath") | ||
| ] | ||
| # Also scan the source, for kernels the import walk can't see: nested ones and the | ||
| # ``two_prod`` variant not selected on this machine. Only a literal ``False`` is known | ||
| # to be safe; anything else (``True``, a flag set, a variable) is flagged. | ||
| for module in {kernel.py_func.__module__ for kernel in eft.values()}: | ||
| tree = ast.parse(Path(sys.modules[module].__file__).read_text()) | ||
| offenders += [ | ||
| f"{module}:{node.lineno}" | ||
| for node in ast.walk(tree) | ||
| if isinstance(node, ast.keyword) | ||
| and node.arg == "fastmath" | ||
| and not (isinstance(node.value, ast.Constant) and node.value.value is False) | ||
| ] | ||
| assert not offenders, ( | ||
| f"fastmath enabled on {sorted(offenders)} (kernels, or module:line of a fastmath= " | ||
| f"that isn't a literal False): they inline EFT code from {EFT_MODULE}, whose " | ||
| f"error terms fastmath cancels to 0. Remove fastmath or set it to False." | ||
| ) | ||
|
|
||
|
|
||
| def test_shared_modules_complete(kernels): | ||
| """SHARED_MODULES lists every module whose kernels another module's kernels call.""" | ||
| script = Path(__file__).parents[3] / "ci" / "clean_numba_cache.py" | ||
| if not script.exists(): | ||
| pytest.skip("ci/clean_numba_cache.py is not available outside the repository") | ||
| spec = importlib.util.spec_from_file_location("clean_numba_cache", script) | ||
| cleaner = importlib.util.module_from_spec(spec) | ||
| spec.loader.exec_module(cleaner) | ||
| shared = { | ||
| "uxarray." + name.removesuffix(".py").replace("/", ".") | ||
| for name in cleaner.SHARED_MODULES | ||
| } | ||
|
|
||
| # Any cross-module call counts, not only inline="always": numba links a cached | ||
| # callee's code into the caller's cached object either way. | ||
| called_from = {} | ||
| for kernel, _ in kernels.values(): | ||
| caller = kernel.py_func.__module__ | ||
| for callee in _callees(kernel): | ||
| if callee.py_func.__module__ != caller: | ||
| called_from.setdefault(callee.py_func.__module__, set()).add(caller) | ||
| assert called_from, "found no cross-module kernel calls; is _callees broken?" | ||
|
|
||
| missing = {m: sorted(c) for m, c in sorted(called_from.items()) if m not in shared} | ||
| assert not missing, ( | ||
| f"Add these to SHARED_MODULES in ci/clean_numba_cache.py: {missing} " | ||
| f"(module -> its callers). Kernels in the callers compile in code from these " | ||
| f"modules, and numba won't notice when they change, so pytest must clear the " | ||
| f"cache whenever they do." | ||
| ) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.