Skip to content

Add OBN Continuous Receiver Gathers (CRG) template - #850

Closed
andgineer wants to merge 5 commits into
TGSAI:mainfrom
andgineer:crg-continuous-receiver-gathers
Closed

Add OBN Continuous Receiver Gathers (CRG) template#850
andgineer wants to merge 5 commits into
TGSAI:mainfrom
andgineer:crg-continuous-receiver-gathers

Conversation

@andgineer

@andgineer andgineer commented Jul 27, 2026

Copy link
Copy Markdown

Draft — opening early to start the discussion on the two API questions at the bottom. Code and tests are complete and green locally; happy to reshape the public surface before this is reviewed properly.

What this adds

A template for OBN continuous receiver gathers (CRG). Unlike shot-organised OBN data, CRG receivers record continuously, so there is no shot index to grid on. The natural key is (component, receiver_line, receiver), and each receiver holds a long run of time segments sharing that key.

Rather than introduce a new grid-override mechanism, this extends the existing HasDuplicates path, which already solves exactly this problem — it appends a per-tuple counter as a synthetic trace dimension. Two knobs were missing for the CRG scale, so those come first and the template builds on them.

Resulting layout: (component, receiver_line, receiver, trace, time).

Chunk size and dtype for the inserted trace dimension

The inserted trace dimension was hardcoded to chunk size 1 and an int16 counter. That caps a gather at 32,767 duplicates — past which the counter raises OverflowError — and forces one chunk per trace. A full CRG campaign runs to roughly 2.6e5 segments per receiver, and chunk-per-trace gives unusably small objects.

  • GridOverrides.chunksize now applies to has_duplicates as well as non_binned.
  • New optional GridOverrides.trace_dtype sets the dtype of the inserted dimension — both the counter and the coordinate it is stored as. It is typed ScalarType and restricted to integer types, so a dtype that cannot count traces is rejected at construction instead of failing deep inside numpy. It reaches the non_binned path too, since that inserts the same axis and has the same ceiling.

Omitting both reproduces today's behaviour exactly: chunk 1, an int16 counter stored as an int32 coordinate. That pair of legacy defaults is named (LEGACY_COUNTER_DTYPE, LEGACY_TRACE_DTYPE) and pinned by explicit backwards-compatibility tests.

This part also generalizes _resolve_synthesize_dims to read a template's synthesize_missing_dims attribute instead of isinstance-checking Seismic3DObnReceiverGathersTemplate. The OBN template already sets that attribute, so its behaviour is unchanged; this just lets other templates opt in. ingestion/segy/validation.py keys its optional-dimension exemption off the same attribute so the two cannot drift, and the attribute is now declared on AbstractDatasetTemplate itself rather than assigned in __init__, so consumers can read the hook off any template without a getattr fallback.

The template

ObnContinuousReceiverGathers3D, registered as a built-in. Grids on the three spatial dimensions and leaves the segment axis to HasDuplicates. component is declared in synthesize_missing_dims, so single-component files carrying no component header ingest against the same template with the dimension synthesized — one template covers both layouts.

Tests

tests/unit tests/integration -m "not cloud" → 569 passed.

  • Template contract, coordinate-spec sync, registry inventory.
  • Grid-override plumbing: the inserted trace honours chunksize and trace_dtype in both the header counter and the stored coordinate, plus BC tests asserting the default path still yields chunk 1, an int16 counter and an int32 coordinate.
  • A non-integer trace_dtype is rejected at construction, and a gather that outgrows the counter raises OverflowError rather than wrapping to a wrong index.
  • Two integration tests ingesting mock CRG SEG-Y: one multi-component, asserting the uint32 segment axis and the realized chunk shape, and one exercising the synthesized-component path.

Questions for maintainers

  1. Is the API shape right for the new knobs? I put trace_dtype on GridOverrides alongside the existing chunksize, since that is where non_binned already carries its chunk size and it keeps the two paths symmetric. It governs the counter and the stored coordinate together, on both paths that insert a trace axis. The alternative is to carry it on the template instead, which would keep GridOverrides narrower but split the configuration for one dimension across two objects. Happy to move it.

  2. Is "extend HasDuplicates" the direction you want? The alternative is a dedicated override (something like CalculateTimeSegmentIndex) that does not touch the existing path at all. Extending seemed better because the semantics are genuinely identical — a counter disambiguating repeated index tuples — and a separate override would duplicate the machinery. But it does widen a shipping public surface, so say the word if you would rather have this isolated.

Andrei.Sorokin and others added 5 commits July 27, 2026 12:50
The inserted `trace` dimension was hardcoded to chunk size 1 and an int16
counter, which caps a gather at ~32k duplicates and forces one chunk per
trace. Continuous receiver gathers need both knobs: ~2.6e5 segments per
receiver, and a chunk size large enough to reach a sane object size.

`GridOverrides.chunksize` now applies to the `has_duplicates` path as well
as `non_binned`, and a new optional `GridOverrides.trace_dtype` selects the
counter dtype. Both default to the previous behaviour (chunk 1, int16), so
existing payloads that omit them are unaffected.

Also generalizes `_resolve_synthesize_dims` to read the template's
`synthesize_missing_dims` attribute instead of isinstance-checking the OBN
template, so any template can declare dimensions it synthesizes when they
are absent from the SEG-Y spec. The ingestion validator keys off the same
attribute.

Co-authored-by: Cursor <cursoragent@cursor.com>
Continuous receiver gathers record every receiver continuously rather than
per shot, so there is no shot index to grid on: the natural key is
(component, receiver_line, receiver), and each receiver holds a long run of
time segments that share that key.

`ObnContinuousReceiverGathers3D` grids on those three spatial dimensions and
leaves the segment axis to the `HasDuplicates` override, which inserts
`trace` ahead of the vertical axis. The resulting layout is
(component, receiver_line, receiver, trace, time). `component` is declared in
`synthesize_missing_dims`, so single-component files that carry no component
header ingest against the same template with the dimension synthesized.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The knob only reached the in-flight counter: the stored `trace` coordinate kept
the DimensionSpec default, so a caller asking for uint32 still got an int32
axis. Both trace-inserting schema effects now take the dtype and pass it to
DimensionSpec explicitly, and the strategy forwards its own.

The dtype is also forwarded on the `non_binned` path, which inserts the same
`trace` axis and had the same int16 ceiling while silently ignoring the knob.

Typing the field as ScalarType and restricting it to integers rejects dtypes
that cannot count traces (float32, bool, object, bytes, datetime64) at
construction instead of failing deep inside numpy.

`None` still means the legacy pair - an int16 counter stored as an int32
coordinate - now named as LEGACY_COUNTER_DTYPE / LEGACY_TRACE_DTYPE and pinned
by tests, including one asserting the counter raises OverflowError rather than
wrapping when a gather outgrows it.

`synthesize_missing_dims` moves to a class attribute on
AbstractDatasetTemplate so consumers can read the hook off any template without
a getattr fallback; it was previously set in __init__, which hid it from
`MagicMock(spec=...)`.

Co-authored-by: Cursor <cursoragent@cursor.com>
Fail early when CRG imports omit the override that supplies their trace dimension, and trim redundant tests and documentation.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.21%. Comparing base (c3ba558) to head (5ac13c0).
⚠️ Report is 247 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #850      +/-   ##
==========================================
+ Coverage   85.30%   93.21%   +7.90%     
==========================================
  Files          46      148     +102     
  Lines        2219     8588    +6369     
  Branches      306      474     +168     
==========================================
+ Hits         1893     8005    +6112     
- Misses        281      509     +228     
- Partials       45       74      +29     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@andgineer

Copy link
Copy Markdown
Author

abandon in favor of better Brian's PR #851

@andgineer andgineer closed this Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant