Multi pass decoding by oscarhiggott · Pull Request #256 · quantumlib/tesseract-decoder

oscarhiggott · 2026-05-22T22:13:26Z

Summary

This adds one- and two-pass decoding with exactly two detector components. Each component has its own Tesseract decoder. In two-pass mode, predictions from the first pass reweight errors in the other component before the final pass. The component decoders are constructed once and reused across shots.

Causal scheduling is the default: only component decodes needed for the final logical prediction are scheduled. Experimental static scheduling runs both components in each pass. More than two components or two passes is out of scope.

Decoder interface

MultiPassTesseractDecoder takes a MultiPassTesseractConfig, separately from TesseractDecoder(TesseractConfig). Both, and SimplexDecoder, implement the common Decoder interface and return a DecodeResult. The result carries predictions, cost, low confidence, and an explicit flag indicating whether predicted error indices are populated.

Multipass owns the DEM splitting, component decoding, scheduling, and cross-component reweighting. Generated detector orders resolve separately against each component DEM, using the DetectorOrder infrastructure from #277. The CLI chooses the decoder during construction; the worker loop uses the common interface and initialization remains parallel. Execution-plan statistics are computed on demand by get_execution_plan().

Python and Sinter

The ordinary workflow is:

from multi_pass_sinter_decoders import MultiPassSinterDecoder, get_sinter_decoders

decoder = MultiPassSinterDecoder()
custom_decoders = get_sinter_decoders()

Standard Tesseract configuration keywords can be passed directly. Unknown keywords raise. The existing long-beam registry names remain available. Custom X/Z classification is one optional keyword, detector_basis_classifier; detector_classifier remains a compatibility alias for integer component labels.

GARI, the Python DEM decomposer, and multipass share the detector-basis interface. The named automatic classifier checks, in order:

Top-level measure_basis.
md.measure_basis.
Top-level basis.
md.basis.
Chromobius fourth coordinate: exactly 0/1/2 means X and 3/4/5 means Z.

An invalid reached metadata field blocks fallback. Nonintegral fourth coordinates are rejected. Stim surface-code parity is an explicit named adapter, not an automatic fallback; the decomposer's generic last-coordinate adapter remains available without claiming its labels are necessarily X/Z.

Python resolves classification once and passes a component vector to native code. Users do not need to normalize their DEM manually for Python/Sinter.

CLI

--multipass selects multipass decoding, with --num-passes 1|2 (default 2) and --multipass-strategy causal|static. --print-multipass-plan prints model statistics and the schedule on demand. --multipass --dem-out is rejected.

The standalone CLI accepts only canonical top-level JSON measure_basis tags:

detector[{"measure_basis":"X"}](0, 0, 0) D0
detector[{"measure_basis":"Z"}](1, 0, 0) D1

Canonically tagged circuit DETECTOR instructions survive circuit-to-DEM conversion, so tagged .stim input works too. Otherwise, demutil.annotate_detector_bases(...) produces a canonical DEM in Python. It preserves coordinates, instruction order, repeats, shifts, errors and their tags, and unrelated JSON metadata. Invalid or conflicting existing top-level measure_basis values and non-JSON tags that would be overwritten are errors. Lower-priority metadata is retained without requiring agreement. Top-level measure_basis is authoritative in both Python and native decoding; basis-only DEMs remain supported automatically in Python/Sinter and can be normalized with this helper for the CLI.

Reweighting and correctness

The reweighting rule is a correlated-matching-style heuristic, not an exact conditional probability. It divides the XOR-combined probability of mechanisms containing both symptoms by the XOR-combined probability of the source symptom. One-sided mechanisms contribute to the denominator; independent coincidences are not included in the numerator. Reweighted probabilities are capped at 0.499 to keep costs positive.

Two-pass decoding requires merge_errors=True: the heuristic acts on aggregate symptoms, so applying its probability separately to duplicate unmerged mechanisms would be incorrect. One-pass decoding still supports merge_errors=False.

Existing ^ decomposition groups retain their boundaries and tags; each group must belong to one component. For undecomposed mixed-component errors, observable assignment must have exactly one solution. Impossible or ambiguous assignments, logical-only groups, and other detectorless groups are rejected with context rather than silently reinterpreted.

Temporary cost updates validate indices, use deterministic tie ordering, and restore costs even when decoding throws. Repeated and sparse shots must not inherit reweighting state. Reported cost comes only from the final pass; low confidence is aggregated across passes and propagated through Sinter's discard byte. Packed NumPy inputs support arbitrary strides.

Tests

Coverage includes observable-assignment uniqueness, existing decomposition groups, the reweighting formula and probability cap, merged/unmerged policies, cost restoration and sparse decoding, basis precedence and normalization, tagged circuit input, component detector orders, Sinter discards and strided arrays, and zero-configuration Python/Sinter use. The full bazel test --jobs=1 src/... suite passed during implementation, as did the benchmark-workflow and tutorial-sync tests.

Reorganise the Python package layout: - Rename pybind module from tesseract_decoder to _core - Move _tesseract_py_util to tesseract_decoder.utils with relative imports - Add tesseract_decoder/__init__.py with top-level re-exports - Add sinter_decoders.py with MultiPassSinterDecoder wrapper - Add setup.py for pip-installable builds via Bazel - Update stub_test.py for new API surface - Update CMakeLists.txt and BUILD for new module name

Prepare TesseractDecoder for multi-pass decoding support: - Add update_internal_costs() for incremental resynchronisation of internal cost structures (error_costs, d2e sort order) after external modification of error likelihoods - Add early return in decode_to_errors for empty syndromes - Add TesseractDebugger friend class for test access to internals - Reserve error_costs capacity before initial fill - Fix int/size_t mismatch in flip_detectors_and_block_errors - Update and simplify tesseract tests

Add foundational libraries for multi-pass decoding: - bern_utils: Bernoulli probability utilities (log-likelihood conversion, probability clamping) - tanner_graph: Union-Find-based connected component analysis of the detector-error Tanner graph - error_correlations: Correlation extraction pipeline computing marginal, joint, and conditional error probabilities from first-pass decoding results - dem_decomposition: DEM decomposition by detector class, error splitting across components, observable assignment, and DEM merging for multi-component decoding

Add the multi-pass Tesseract decoder, which decomposes a detector error model into independent components by detector class and decodes each component separately across multiple passes. Between passes, first-pass decoding correlations are used to reweight error probabilities in subsequent components, improving accuracy. Key components: - MultiPassTesseractDecoder: core decoder with static and causal scheduling across detector classes - FastTwoPassTesseractDecoder: optimised two-pass specialisation - multi_pass_sinter_compat.pybind.h: pybind11 bindings exposing MultiPassSinterDecoder and MultiPassSinterCompiledDecoder - Python integration tests for multi-pass bindings - Theory and architecture documentation Performance: 10-100x wall-clock speedup over single-pass Tesseract by decomposing the DEM into smaller independent components.

…g, and Sinter integration (#255) This Pull Request integrates the complete C++ and Python Multi-Pass Prior Propagation Decoding Engine into the repository, surgically resolving several upstream alignment bugs, reindexing logic mismatches, degeneracy mappings, and packaging issues present in the baseline branch. All changes are fully validated under Bazel and replicate the optimal private baseline logical error rate (LER) results down to the single shot! --- We surgically resolved several critical reindexing and prior propagation bugs inside the C++ core library to align it with high-performance multi-pass decoding: * **Global Detector Reindexing**: Maintained absolute global detector indices in all local Component DEMs. This keeps `global_to_local_det` as a clean identity map, preventing out-of-bounds array lookup crashes. * **Sweep Seed Alignment**: Synchronized all component decoders to use a single consistent deterministic `seed` (instead of `seed + i`) during BFS traversal orderings, preventing search-tree sweep divergence. * **Degenerate Symptom Vector Mapping**: Refactored `symptom_to_error_index` to map degenerate symptoms to `std::vector<size_t>` and updated C++ rule propagation to broadcast LLR reweights across **all** degenerate causal and target error states. * **Max-Prob Prior Updates**: Replaced basic priority overwriting inside `decode()` with the mathematically correct **Max-Prob prior combination rule** (`std::max(current_p, conditional_prob)`), safely capped at `0.5` to prevent negative edge weights. * **Persistent Intermediate Predictions**: Introduced a persistent `component_predictions` map to store predictions across passes, ensuring clean Logical Observable Extraction before the final Surgical Reset restores modified costs. * **Clean Validation Encapsulation**: Added a public static validator `MultiPassTesseractDecoder::validate_annotations` to enforce component partition validations at the CLI layer (`src/tesseract_main.cc`), preserving C++ core library constructor flexibility for programmatic and single-detector subproblem tests. * **Wall-Clock Time Accuracy**: Replaced thread-accumulated execution times with real elapsed wall-clock time measurements (`global_elapsed`), reporting accurate multi-threaded throughput in console stats outputs. --- We resolved several outstanding Bazel build and Python dependency reference errors: * **Wheel Packaging Targets**: Updated the root `BUILD` file `py_wheel` dependencies to correctly map to Oscar's renamed pybind extension target `//src:_core` and python target `//src/py:tesseract_decoder`. * **Pip Sandbox Dependency**: Declared the missing `@pypi//sinter` dependency on the `:tesseract_decoder` python target in `src/py/BUILD` to cleanly pass Sinter-compat python unit tests. * **Pristine stream redirection**: Added the `scoped_ostream_redirect` pybind call guard to `decode_shots_bit_packed` to pipe C++ stdout natively back to Python standard streams. --- We ran full-scale $1,000$-shot Multi-Pass decoding benchmarks on the newly compiled public binary. The logical error counts **match our optimal private baseline exactly down to the single shot**: * **Replicated Error Count**: **`145` / 1,000** (Expected: `145`). * **Wall-Clock Execution Time**: **`7.93 seconds`** (instead of `182` seconds of thread-accumulated time!). * **Command**: ```bash ./bazel-bin/src/tesseract \ --circuit testdata/annotated_surface_codes/style=surface_code,d=7,basis=X,num_rounds=10,max_qubits_per_module=91,total_qubits=118,k=1,noise=SI1000,p=0.00500.stim \ --sample-num-shots 1000 \ --multipass \ --num-passes 2 \ --multipass-strategy causal \ --pqlimit 1000000 \ --beam 20 \ --beam-climbing \ --no-revisit-dets \ --num-det-orders 21 \ --det-order-seed 2384753 \ --sample-seed 2384753 \ --print-stats ``` * **Replicated Error Count**: **`72` / 1,000** (Expected: `72`). * **Wall-Clock Execution Time**: **`0.13 seconds`**! * **Command**: ```bash ./bazel-bin/src/tesseract \ --circuit testdata/colorcodes/r=5,d=5,p=0.003,noise=si1000,c=midout_color_code_X,q=23,gates=cz.stim \ --sample-num-shots 1000 \ --multipass \ --num-passes 2 \ --multipass-strategy static \ --pqlimit 1000000 \ --beam 20 \ --beam-climbing \ --no-revisit-dets \ --num-det-orders 21 \ --det-order-seed 2384753 \ --sample-seed 2384753 \ --print-stats ``` ---

…_internal_costs

In GitHub Actions CI and PyPI binary releases, building with -march=native causes the compiler to emit host-specific vector instructions (e.g. AVX-512) that are masked or unsupported by VM hypervisors. When running compiled C++ tests or importing _core.so during stub generation, the virtual CPU throws an Illegal instruction (SIGILL) signal. This commit updates the default build architecture to -march=x86-64-v3: - Enables full AVX, AVX2, FMA3, BMI1, BMI2, SSE4.2, and POPCNT vector SIMD acceleration for maximum math performance. - Ensures 100% execution safety on CI runner VMs and PyPI manylinux wheels. Opt-in native host CPU tuning remains fully supported: - Bazel: Pass --config=native or --copt=-march=native. - CMake: Pass -DTESSERACT_NATIVE_ARCH=ON.

- Add MultiPassDecodeResult struct containing predictions, low_confidence, and total_cost, resolving hardcoded low_confidence=false and cost=0 in CLI. - Disallow combining --multipass and --dem-out flags at CLI option parsing time. - Validate --multipass-strategy against expected values ('static', 'causal') and report a CLI error for invalid values. - Return -1 for unclassified detectors in default classifier and throw a descriptive exception in validate_annotations identifying unclassified detectors.

… field

# Conflicts: # src/utils.h

…pass-decoding

… orders to decoder CLI (#277) ## Summary This connects the existing Python GARI transform to the Tesseract CLI. It also introduces `DetectorOrder`, so we can specify how to construct an order before the decoding DEM is known. This is useful here and for the component decoders in #256. ## Detector orders One `DetectorOrder` represents one permutation. It either contains a literal permutation (`Method::Literal`) or a method (`BFS`, `Index`, or `Coordinate`) and seed. `resolve(dem)` fills in the order in place; for literal orders it validates the supplied permutation. Callers do not need separate handling for the two cases. `TesseractConfig` holds a vector of these objects. Generated orders are resolved against the actual decoding DEM, with graph/coordinate preparation shared across the batch. Order k uses seed + k, rather than depending on a shared random-number stream. Every order must be a complete permutation of the DEM's detector IDs. The Tesseract CLI can combine generated methods and JSON order files, in command-line order: ```bash ./tesseract --circuit source.stim --dem transformed.dem \ --num-det-orders 3 --det-order-bfs \ --detector-orders gari-orders.json --det-order-coordinate ``` Each generated source contributes `--num-det-orders` orders, using `--det-order-seed` as its base seed. `--detector-orders FILE` can be repeated and contributes every permutation in that file. The format is a JSON list of lists, just like Python's `det_orders`. With no source flags, the default is Index. A file-only invocation stays file-only; add `--det-order-index` to combine it with Index orders. Count/seed flags with files but no generated method are rejected. The bookkeeping for these CLI sources stays local to the CLI, not in the library API. ## GARI workflow GARI transformation and GARI-aware order construction stay in Python. `demutil.gari.circuit_to_gari(...)` returns the transformed DEM directly, with source detector IDs first and virtual detectors appended. No layout sidecar is needed. `demutil.gari.build_detector_orders(...)` constructs orders from the source DEM, appends virtual detector IDs, and checks that the decoding DEM has the expected GARI check and logical matrices. Probability-only reweighting is allowed. When given both a circuit and a larger decoding DEM, the Tesseract and Simplex CLIs read/sample the circuit-width syndrome and leave the extra DEM detectors zero. The observable counts must agree. The order-file option is specific to the Tesseract CLI. Python keeps the list-of-lists interface and existing detector-order aliases. Ordinary `TesseractSinterDecoder` construction and registry names are unchanged; generated orders resolve when the compilation DEM is available. This does not add a GARI-specific Sinter frontend. GARI requires undecomposed source errors: `^` groups and logical-only errors are rejected. Its basis convention here is fourth coordinate 0/1/2 for X and 3/4/5 for Z. GARI-aware orders require the default source-aligned layout; `row_order="block"` remains available for matrix work. The transformed DEM is for decoding, not sampling. ## Tests Coverage includes generated/literal resolution, per-order seeds, permutation and file validation, Python compatibility, deferred Sinter orders, GARI layouts and matrix checks, and source-width shots decoded against a larger DEM. The full `bazel test --jobs=1 src/...` suite passed during implementation. --------- Co-authored-by: Noah Shutty <noahshutty@gmail.com> Co-authored-by: Aria Shahingohar <ariash@google.com> Co-authored-by: Noureldin <noureldinyosri@google.com> Co-authored-by: Noah Shutty <noajshu@users.noreply.github.com>

Resolve squash-merge overlap while preserving the existing multipass tree and subsequent cleanup.

noajshu

LGTM
@oscarhiggott will this break your workflow?

oscarhiggott requested a review from a team as a code owner May 22, 2026 22:13

oscarhiggott requested review from noajshu and removed request for a team May 22, 2026 22:13

mhucka mentioned this pull request Jul 27, 2026

feat(multipass): C++ multi-pass engine fixes, degenerate error mapping, and Sinter integration #255

Merged

oscarhiggott added 4 commits July 28, 2026 17:36

aria-googler force-pushed the multi-pass-decoding branch from c72bca0 to 425762d Compare July 28, 2026 17:56

aria-googler force-pushed the multi-pass-decoding branch from 425762d to c699816 Compare July 28, 2026 18:00