test: raise coverage 48% -> 78% (+653 tests), fix 13 bugs found on the way

.github/workflows/tests.yml

@@ -51,3 +51,53 @@ jobs:
       uses: codecov/codecov-action@v4
       with:
         token: ${{ secrets.CODECOV_TOKEN }}
+        flags: light
+
+  build-full-coverage:
+    # The matrix above installs only the dev group, so every module behind torch /
+    # nnunetv2 / hf-deepali is a hard zero there however well it is tested. This job
+    # installs those on CPU so that coverage actually reaches Codecov. One job, one
+    # python version - the matrix stays the fast compatibility gate.
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.11
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.11"
+        cache: "pip"
+    - name: Configure python
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install poetry
+    - name: Install system deps for CPU-only mesh3D tests
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y xvfb libgl1
+    - name: Install dependencies
+      run: |
+        python -m poetry install --with dev
+    - name: Install heavy optional deps (CPU only)
+      # The default torch wheel pulls ~2.5 GB of CUDA libraries that this job cannot
+      # use. `hf-deepali` is the correct distribution name - a bare `deepali` on PyPI
+      # is an unrelated project. `spineps` is deliberately absent: it depends on
+      # TPTBox itself, so installing it would pin a *released* TPTBox over this
+      # checkout. The tests that touch it fake the module instead.
+      run: |
+        python -m poetry run pip install torch --index-url https://download.pytorch.org/whl/cpu
+        python -m poetry run pip install antspyx nnunetv2 acvl-utils batchgenerators hf-deepali
+    - name: Test with pytest and create coverage report
+      run: |
+        python -m poetry run coverage run --source=TPTBox -m pytest unit_tests/ TPTBox/tests/
+        python -m poetry run coverage xml
+        echo "--- with the curated omit list (the reported number) ---"
+        python -m poetry run coverage report
+        echo "--- without it, for comparison ---"
+        python -m poetry run coverage report --omit="TPTBox/__no_such_path__/*"
+    - name: Upload coverage results to Codecov (Only on merge to main)
+      if: github.ref == 'refs/heads/main' && github.event_name == 'push'
+      uses: codecov/codecov-action@v4
+      with:
+        token: ${{ secrets.CODECOV_TOKEN }}
+        flags: full

.github/workflows/tests_mr.yml

@@ -43,3 +43,30 @@ jobs:
     - name: Test with pytest
       run: |
         python -m poetry run pytest
+
+  heavy-deps:
+    # Same heavy-dependency set as the coverage job on main, so a PR that breaks a
+    # mocked torch / deepali test fails here instead of after the merge. No coverage
+    # upload - this is only a gate.
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
+      with:
+        python-version: "3.11"
+        cache: "pip"
+    - name: Install system deps for CPU-only mesh3D tests
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y xvfb libgl1
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install poetry
+        python -m poetry install --with dev
+        python -m poetry run pip install torch --index-url https://download.pytorch.org/whl/cpu
+        python -m poetry run pip install antspyx nnunetv2 acvl-utils batchgenerators hf-deepali
+    - name: Test with pytest
+      run: |
+        python -m poetry run pytest unit_tests/

CLAUDE.md

@@ -83,7 +83,13 @@ Public API is re-exported from `TPTBox/__init__.py`. All major classes and utili
 
 ### Test Layout
 
-Tests live in `unit_tests/` (not `TPTBox/tests/`). `TPTBox/tests/` contains test utilities and sample data (CT/MRI NIfTIs) used by the unit tests. Some generated test files are very large (>20K LOC) — they are autogenerated and should not be edited by hand.
+Tests live in `unit_tests/` (not `TPTBox/tests/`). `TPTBox/tests/` contains test utilities and sample data (CT/MRI NIfTIs) used by the unit tests.
+
+Three files carry a `# Generated by CodiumAI` marker on line 1 — `test_poi_autogen.py`, `test_nii_wrapper_auto.py` and `test_poi_global.py`. There is no generator script in the repo; they were committed once and have been hand-edited since, so treat them as ordinary (if oddly styled) test files.
+
+House style for new tests, as followed by every file in `unit_tests/`: `unittest.TestCase` classes (no pytest fixtures, no `conftest.py`), `with self.subTest(...)` for table-driven cases, `tempfile.TemporaryDirectory()` for scratch files, and optional heavy dependencies guarded with `importlib.util.find_spec` plus `@unittest.skipUnless` (see `test_mesh3D.py`). New tests must stay Python 3.9-compatible — CI runs a 3.9 leg, so no `itertools.pairwise` (use `TPTBox.core.compat.pairwise`), no `zip(..., strict=...)` (use `compat.zip_strict`) and no `match` statements.
+
+`[tool.coverage.report].omit` in `pyproject.toml` is curated: it lists only code a unit test cannot legitimately reach (vendored nnU-Net internals, nnU-Net training drivers, the dead vendored deepali engine, the NAKO dataset pipelines, `nipy`-dependent scripts). Code that is merely awkward to test does not belong there.
 
 ## Sub-package notes
 

TPTBox/core/bids_constants.py

@@ -51,16 +51,13 @@
     "Chimap",
     "S0map",
     "M0ma",
-    "T1w",
-    "T2w",
     "PDw",
     "T2starw",
     "FLAIR",
     "inplaneT1",
     "inplaneT2",
     "PDT2",
     "angio",
-    "T2star",
     "FLASH",
     "VF",
     "defacemas",
@@ -102,13 +99,11 @@
     "electrode",
     "marker",
     "me",
-    "sti",
     "PE",
     "bloo",
     "pe",
     "event",
     "physio",
-    "sti",
     "TEM",
     "SEM",
     "uCT",

TPTBox/core/bids_files.py

@@ -308,7 +308,7 @@ def _scan_tree(path, lvl=1, filter_folder=None, verbose=False):
         if entry.is_dir(follow_symlinks=False):
             if filter_folder is not None and not filter_folder(Path(entry.path), lvl):
                 continue
-            yield from _scan_tree(entry.path, lvl=lvl + 1, verbose=verbose)
+            yield from _scan_tree(entry.path, lvl=lvl + 1, filter_folder=filter_folder, verbose=verbose)
         elif entry.name[0] != ".":
             if verbose:
                 print(f"[{_cont:8}]", end="\r")
@@ -336,9 +336,10 @@ def __init__(
             datasets (typing.List[str]): List of dataset paths
             parents (typing.List[str]): List of parents (like ["rawdata","sourcedata","derivatives"])
             additional_key (list, optional): Additional keys that are not in the default BIDS but should not raise a warning. Defaults to ["sequ", "seg", "ovl"].
-            filter_folder: Filter function, input is the path of the folder and the level of the folder structure. Return True if we should continue searching
-                        Example:
-                        filter_folder = lambda p, lvl: True if (lvl != 2 or p.name in ["sub-123","sub-456"]) else False
+            filter_folder: Filter function, input is the path of the folder and the level of the folder structure.
+                Return True if we should continue searching. Level 1 is the first folder inside
+                ``<dataset>/<parent>``, i.e. the subject folder; level 2 is the session folder.
+                Example: ``filter_folder = lambda p, lvl: lvl != 1 or p.name in ["sub-123", "sub-456"]``
         """
         self.count_file = 0
         self._p_counter = 0
@@ -1872,8 +1873,11 @@ def filter_dixon_water(self, _keys: list[str] | None = None) -> None:
         assert self._flatten
 
         def json_filter(x: dict) -> bool:
-            """Return True if JSON ImageType contains all required channel keys."""
-            return "ImageType" not in x or all(k in x["ImageType"] for k in _keys)
+            """Return True if JSON ImageType contains any of the channel keys."""
+            # `_keys` lists alternative spellings of one channel ("W"/"WATER"), so this is
+            # `any`, matching `lam_filter` below. With `all` the filter dropped every file
+            # that had a sidecar, because no ImageType carries both spellings.
+            return "ImageType" not in x or any(k in x["ImageType"] for k in _keys)
 
         def lam_filter(x: str) -> bool:
             """Return True if the entity value matches one of the target channel keys."""

TPTBox/core/compat.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from collections.abc import Iterator
+
 
 def zip_strict(*iterables) -> zip:
     """A strict version of zip that raises a ValueError if the input iterables have different lengths.
@@ -20,3 +22,36 @@ def zip_strict(*iterables) -> zip:
     if len(set(lengths)) != 1:
         raise ValueError(f"Length mismatch: {lengths}")
     return zip(*lists)
+
+
+def pairwise(iterable) -> Iterator:
+    """Yield successive overlapping pairs from ``iterable``.
+
+    Backport of :func:`itertools.pairwise`, which is only available from Python
+    3.10 onwards while this package still supports 3.9. On 3.10+ the stdlib
+    implementation is used directly.
+
+    Args:
+        iterable: Any iterable. It is consumed lazily, so generators are fine.
+
+    Returns:
+        An iterator over ``(s[0], s[1]), (s[1], s[2]), ...``. Empty for inputs
+        with fewer than two elements.
+    """
+    try:
+        from itertools import pairwise as _pairwise  # noqa: PLC0415
+    except ImportError:
+        return _pairwise_py39(iterable)
+    return _pairwise(iterable)
+
+
+def _pairwise_py39(iterable):
+    """Pure-python ``pairwise`` used on Python 3.9."""
+    it = iter(iterable)
+    try:
+        prev = next(it)
+    except StopIteration:
+        return
+    for cur in it:
+        yield prev, cur
+        prev = cur

TPTBox/core/dicom/dicom_renamer.py

@@ -487,6 +487,7 @@ def _leftover_move_plan(
     scan_root: Path,
     subject_map: dict[str, str],
     parent: str | None,
+    already_planned: set[Path] | None = None,
 ) -> list[tuple[Path, Path]]:
     """Compute (src, dst) moves for files left behind after the JSON pass.
 
@@ -498,6 +499,11 @@ def _leftover_move_plan(
     (matched by ``sequ-<N>``) so BIDS stem-linkage stays intact. Files that
     can't be twinned drop at the session level of the new subject folder as
     a safe fallback.
+
+    ``already_planned`` holds the sources Pass 1 has lined up. In a real run those
+    files are gone from disk by the time this runs, but under ``dry_run`` they are
+    all still there, and without this filter every one of them would be reported a
+    second time with the fallback destination.
     """
     del parent  # kept for signature symmetry; scan_root already resolves it
     moves: list[tuple[Path, Path]] = []
@@ -571,6 +577,8 @@ def _leftover_move_plan(
                 dst = new_dir / rel.parent / new_name
             if src == dst or dst.exists():
                 continue
+            if already_planned and src in already_planned:
+                continue
             moves.append((src, dst))
     return moves
 
@@ -688,7 +696,7 @@ def rerun_bids_naming(
     # Pass 2 — sweep orphan companions (.bval / .bvec / _ADC.nii.gz / other
     # non-sidecar files) that Pass 1 didn't see because their JSON already
     # moved on a previous run or they never had one.
-    leftover = _leftover_move_plan(scan_root, subject_map, parent_norm)
+    leftover = _leftover_move_plan(scan_root, subject_map, parent_norm, already_planned={src for src, _ in all_moves})
     for src, dst in leftover:
         if verbose:
             logger.on_neutral(f"{'[dry]' if dry_run else '[lft]'} {src.relative_to(dataset_root)}  ->  {dst.relative_to(dataset_root)}")

TPTBox/core/dicom/xa_pairing.py

@@ -208,7 +208,10 @@ def pair_biplane_xa(
                 if b_sequ == a_sequ:
                     continue
                 # Build the target path with A's sequ; keep all other entities.
-                new_bids = bf.get_changed_bids(info={"sequ": a_sequ}, non_strict_mode=True)
+                # `parent` must be the file's own parent: get_changed_bids defaults to
+                # "derivatives", which silently moved the B-side out of rawdata (where a
+                # rawdata-scoped scan - including the flip_z pass below - then lost it).
+                new_bids = bf.get_changed_bids(info={"sequ": a_sequ}, parent=bf.parent, non_strict_mode=True)
                 src_paths = {ext: Path(p) for ext, p in bf.file.items()}
                 new_nii = Path(new_bids.file.get("nii.gz", ""))
                 new_stem = str(new_nii).removesuffix(".nii.gz")

TPTBox/core/internal/elastic_deform.py

@@ -4,6 +4,7 @@
 from numpy.typing import NDArray
 
 from TPTBox import NII
+from TPTBox.core.compat import zip_strict
 
 
 def _elasticdeform_install_hint() -> str:
@@ -109,7 +110,7 @@ def deformed_nii(
         order=[0 if v.seg else 3 for v in values],  # type: ignore
     )
     out2: dict[str, NII] = {}
-    for (k, nii), arr in zip(nii_dic.items(), out, strict=True):
+    for (k, nii), arr in zip_strict(nii_dic.items(), out):
         out2[k] = nii.set_array(arr[p:-p, p:-p, p:-p])
     print("Deformation took", round(time.time() - t, 1), "Seconds")
     if joint_normalize:

TPTBox/core/internal/train_nnUnet/fastProcessor.py

@@ -65,7 +65,7 @@ def _preprocess_dataset(
     print(f"Preprocessing dataset {dataset_name}")
     plans_file = join(nnUNet_preprocessed, dataset_name, plans_identifier + ".json")  # type: ignore
     plans_manager = PlansManager(plans_file)
-    for n, c in zip(num_processes, configurations, strict=False):
+    for n, c in zip(num_processes, configurations):
         print(f"Configuration: {c}...")
         if c not in plans_manager.available_configurations:
             print(f"INFO: Configuration {c} not found in plans file {plans_identifier + '.json'} of dataset {dataset_name}. Skipping.")

TPTBox/core/nii_wrapper.py

@@ -2705,7 +2705,10 @@ def to_stl(
                 if out_path.is_dir():
                     out_path = out_path / f"mask_{label}.stl"
                 elif number_path:
-                    out_path.with_name(f"{out_path.stem}_{label}.stl")
+                    # with_name returns a new path; the result used to be discarded, so
+                    # to_stls() wrote every label to the same file and only the last
+                    # label's mesh survived.
+                    out_path = out_path.with_name(f"{out_path.stem}_{label}.stl")
                 log.on_save(f"Saving STL to {out_path}")
                 out_path.parent.mkdir(exist_ok=True,parents=True)
                 cube.save(str(out_path))

TPTBox/core/np_utils.py

@@ -2218,9 +2218,11 @@ def np_calc_boundary_mask(
     """
     if adjust_intensity_for_ct:
         img = img + 1000
-    boundary = img.copy()
-    boundary[boundary > threshold] = 2
-    boundary[boundary <= threshold] = 0
+    # Compute the foreground mask *before* writing the sentinel: the two assignments
+    # used to run in place one after the other, so the 2 written by the first line was
+    # then compared against the threshold by the second. For any threshold >= 2 that
+    # wiped the foreground again and the function returned an empty mask.
+    boundary = np.where(img > threshold, 2, 0)
     infect_list = []
 
     def infect(x, y, z):
@@ -2242,7 +2244,6 @@ def infect(x, y, z):
                 (-1, 0, 0),
                 (0, 1, 0),
                 (0, -1, 0),
-                (0, 1, 0),
                 (0, 0, 1),
                 (0, 0, -1),
             ]:

TPTBox/registration/_deformable/_grid_search_vert.py

@@ -132,7 +132,7 @@ def get_additonal_point(poi: POI, cc: NII, seg_id):
     # Crop to speed up and decreas gpu memory consumption.
     c0 = femur_0_.compute_crop(dist=10)
     c1 = femur_1_reg.compute_crop(dist=10)
-    ex_slice = [slice(min(a.start, b.start), max(a.stop, b.stop)) for a, b in zip(c1, c0, strict=False)]
+    ex_slice = [slice(min(a.start, b.start), max(a.stop, b.stop)) for a, b in zip(c1, c0)]
     femur_0 = femur_0_.apply_crop(ex_slice)
     femur_1_reg.apply_crop_(ex_slice)
 

TPTBox/segmentation/nnUnet_utils/predictor.py

@@ -587,7 +587,7 @@ def _internal_maybe_mirror_and_predict(self, x: torch.Tensor, network) -> torch.
             mirror_axes = [m + 2 for m in mirror_axes]
             axes_combinations = [c for i in range(len(mirror_axes)) for c in itertools.combinations(mirror_axes, i + 1)]
             for axes in axes_combinations:
-                prediction += torch.flip(self.network(torch.flip(x, axes)), axes)
+                prediction += torch.flip(network(torch.flip(x, axes)), axes)
             prediction /= len(axes_combinations) + 1
 
         return prediction

TPTBox/segmentation/rib/_rib_assign.py

@@ -405,7 +405,11 @@ def assign_ribs_to_vert_segmentation(
     logger.on_debug(f"{rib_inst.unique()=}")
     # Merge rib assignments back into the original vert / sem segmentations
     # without disturbing existing (non-rib) labels. Skip unmatched CCs (sentinel error_value=255).
-    matched = (rib_inst != 0) & (rib_inst != error_value)
+    # Combine on the arrays, not on NII: `rib_inst != 0` yields a bool-backed NII and
+    # NII.__and__ rejects non-integer arrays, so the NII-level `&` raised TypeError and
+    # this function could never run to completion.
+    rib_inst_arr = rib_inst.get_seg_array()
+    matched = (rib_inst_arr != 0) & (rib_inst_arr != error_value)
     vert_seg[matched] = rib_inst[matched]
 
     sem_seg[rib_seg != 0] = rib_seg.map_labels({left_id: Location.Rib_Left.value, right_id: Location.Rib_Right.value})[rib_seg != 0]  # type: ignore
@@ -418,7 +422,9 @@ def assign_ribs_to_vert_segmentation(
             vert_seg2 = vert_seg.remove_labels(error_value).infect(vert_seg.extract_label(error_value), verbose=False)
             vert_seg[vert_seg != vert_seg2] = vert_seg2[vert_seg != vert_seg2]
             vert_seg[np.logical_and(rib_inst == error_value, vert_seg == 0)] = error_value
-        vert_seg[np.logical_and(vert_seg == 0, sem_seg.extract_label([Location.Rib_Left.value, Location.Rib_Right.value] == 1))] = (
-            error_value
-        )
+        # The closing bracket used to sit one token too far left, so this evaluated
+        # `[Rib_Left, Rib_Right] == 1` -> False -> extract_label(0), which trips the
+        # "zero label is the background" assertion and took the whole function down.
+        rib_sem = sem_seg.extract_label([Location.Rib_Left.value, Location.Rib_Right.value]).get_seg_array() == 1
+        vert_seg[np.logical_and(vert_seg.get_seg_array() == 0, rib_sem)] = error_value
     return vert_seg.reorient_(ori), sem_seg.reorient_(ori)

TPTBox/spine/spinestats/README.md

@@ -58,7 +58,7 @@ not on the Python API. A standalone copy of this reference lives at
 | Key | Source function | What it covers |
 |---|---|---|
 | `ivd_geometry` | `measure_ivd_and_vertebra_geometry(..., structure_label=100)` | intervertebral discs (now also carries wedge angles/indices per label) |
-| `vert_geometry` | `measure_ivd_and_vertebra_geometry(..., structure_label=50)` | vertebral bodies (now also carries wedge angles/indices per label) |
+| `vert_geometry` | `measure_ivd_and_vertebra_geometry(..., structure_label=0)` | vertebral bodies (now also carries wedge angles/indices per label) |
 | `VBQ_score` | `VBQ_score` | vertebral bone quality (T2 signal ratio) |
 | `body_composition_score` | `body_composition_score` | axial CSA per tissue at chosen vertebral levels |
 | `muscle_fat_infiltration` | `muscle_fat_infiltration` | Dixon fat-fraction based muscle-quality metrics |

TPTBox/spine/spinestats/all_output_reference.md

@@ -13,7 +13,7 @@ Python API.
 | Key | Source function | What it covers |
 |---|---|---|
 | `ivd_geometry` | `measure_ivd_and_vertebra_geometry(..., structure_label=100)` | intervertebral discs (per-label entries carry wedge angles/indices) |
-| `vert_geometry` | `measure_ivd_and_vertebra_geometry(..., structure_label=50)` | vertebral bodies (per-label entries carry wedge angles/indices) |
+| `vert_geometry` | `measure_ivd_and_vertebra_geometry(..., structure_label=0)` | vertebral bodies (per-label entries carry wedge angles/indices) |
 | `VBQ_score` | `VBQ_score` | vertebral bone quality (T2 signal ratio) |
 | `body_composition_score` | `body_composition_score` | axial CSA per tissue at chosen vertebral levels |
 | `muscle_fat_infiltration` | `muscle_fat_infiltration` | Dixon fat-fraction based muscle-quality metrics |