Tracking: Efficient Lazy/Chronic Computations in SortingAnalyzer

[alejoe91]

Overview

This issue tracks a series of improvements aimed at making SortingAnalyzer computations more efficient, particularly for long-term / chronic recordings and large-scale pipelines. The overarching goal is to reduce redundant I/O, enable incremental computation, and allow results to be written directly to efficient storage backends (e.g. Zarr) without unnecessary in-memory buffering.

Motivation

Current SortingAnalyzer workflows load and compute extensions eagerly, which becomes a bottleneck when:

• Working with chronic/long-term recordings that accumulate large amounts of waveform data
• Re-running partial computations (e.g. only a subset of units or a new time range changed)
• Saving results to Zarr-based pipelines where intermediate in-memory copies are wasteful

Efforts

1. Lazy Loading

PR: #4623

Support for lazily loading SortingAnalyzer extensions — extension data is not loaded into memory until explicitly accessed. This reduces startup time and memory footprint when only a subset of extensions are needed.

2. Lazy Compute

PR: #4713

Distinct from lazy loading: allows extension computations to be deferred and triggered on demand rather than eagerly computed when compute() is called. This enables incremental / on-demand pipelines without full recomputation.

3. Gather to Zarr

PR: #4703

Enables writing extension results directly to Zarr arrays during computation, bypassing in-memory accumulation. Critical for chronic recording workflows where the full result set does not fit in RAM.

4. Extract Waveforms to Zarr

PR: #4708

Extends the gather-to-Zarr backend to waveform extraction specifically, allowing ComputeWaveforms to stream extracted snippets directly into Zarr storage rather than accumulating them in memory first.

5. Optimize QC Computation

PR: #4715

6. Reduce UnitRefine function RAM (run from metrics)

PR: #4716

Quality metric computations are optimized to take advantage of the lazy/streaming infrastructure above, reducing redundant passes over waveform data and improving throughput for large unit counts.

Checklist

• Lazy-mode for loading Sortinganalyzer #4623 — Lazy loading
• SortingAnalyzer: add read_only mode (different than lazy) #4713 — Lazy compute
• Add GatherToZarr and save pipeline node extension data to disk #4703 — Gather to Zarr
• Extract waveforms to zarr dataset #4708 — Extract waveforms to Zarr
• [DRAFT] Optimize quality metrics performance #4715 — Optimize QC computation
• Model_based_curation + unitrefine prediction: add unit_ids input #4716 — Reduce UnitRefine function RAM (run from metrics)

Notes

• These PRs are interdependent: lazy compute and gather-to-Zarr both build on the lazy loading infrastructure.
• Testing should cover chronic recording use cases (many segments, large unit counts) to validate memory and performance gains end-to-end.