Add OBN Continuous Receiver Gathers (CRG) template - #850
Closed
andgineer wants to merge 5 commits into
Closed
Conversation
The inserted `trace` dimension was hardcoded to chunk size 1 and an int16 counter, which caps a gather at ~32k duplicates and forces one chunk per trace. Continuous receiver gathers need both knobs: ~2.6e5 segments per receiver, and a chunk size large enough to reach a sane object size. `GridOverrides.chunksize` now applies to the `has_duplicates` path as well as `non_binned`, and a new optional `GridOverrides.trace_dtype` selects the counter dtype. Both default to the previous behaviour (chunk 1, int16), so existing payloads that omit them are unaffected. Also generalizes `_resolve_synthesize_dims` to read the template's `synthesize_missing_dims` attribute instead of isinstance-checking the OBN template, so any template can declare dimensions it synthesizes when they are absent from the SEG-Y spec. The ingestion validator keys off the same attribute. Co-authored-by: Cursor <cursoragent@cursor.com>
Continuous receiver gathers record every receiver continuously rather than per shot, so there is no shot index to grid on: the natural key is (component, receiver_line, receiver), and each receiver holds a long run of time segments that share that key. `ObnContinuousReceiverGathers3D` grids on those three spatial dimensions and leaves the segment axis to the `HasDuplicates` override, which inserts `trace` ahead of the vertical axis. The resulting layout is (component, receiver_line, receiver, trace, time). `component` is declared in `synthesize_missing_dims`, so single-component files that carry no component header ingest against the same template with the dimension synthesized. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The knob only reached the in-flight counter: the stored `trace` coordinate kept the DimensionSpec default, so a caller asking for uint32 still got an int32 axis. Both trace-inserting schema effects now take the dtype and pass it to DimensionSpec explicitly, and the strategy forwards its own. The dtype is also forwarded on the `non_binned` path, which inserts the same `trace` axis and had the same int16 ceiling while silently ignoring the knob. Typing the field as ScalarType and restricting it to integers rejects dtypes that cannot count traces (float32, bool, object, bytes, datetime64) at construction instead of failing deep inside numpy. `None` still means the legacy pair - an int16 counter stored as an int32 coordinate - now named as LEGACY_COUNTER_DTYPE / LEGACY_TRACE_DTYPE and pinned by tests, including one asserting the counter raises OverflowError rather than wrapping when a gather outgrows it. `synthesize_missing_dims` moves to a class attribute on AbstractDatasetTemplate so consumers can read the hook off any template without a getattr fallback; it was previously set in __init__, which hid it from `MagicMock(spec=...)`. Co-authored-by: Cursor <cursoragent@cursor.com>
Fail early when CRG imports omit the override that supplies their trace dimension, and trim redundant tests and documentation. Co-authored-by: Cursor <cursoragent@cursor.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #850 +/- ##
==========================================
+ Coverage 85.30% 93.21% +7.90%
==========================================
Files 46 148 +102
Lines 2219 8588 +6369
Branches 306 474 +168
==========================================
+ Hits 1893 8005 +6112
- Misses 281 509 +228
- Partials 45 74 +29 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Author
|
abandon in favor of better Brian's PR #851 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — opening early to start the discussion on the two API questions at the bottom. Code and tests are complete and green locally; happy to reshape the public surface before this is reviewed properly.
What this adds
A template for OBN continuous receiver gathers (CRG). Unlike shot-organised OBN data, CRG receivers record continuously, so there is no shot index to grid on. The natural key is
(component, receiver_line, receiver), and each receiver holds a long run of time segments sharing that key.Rather than introduce a new grid-override mechanism, this extends the existing
HasDuplicatespath, which already solves exactly this problem — it appends a per-tuple counter as a synthetictracedimension. Two knobs were missing for the CRG scale, so those come first and the template builds on them.Resulting layout:
(component, receiver_line, receiver, trace, time).Chunk size and dtype for the inserted
tracedimensionThe inserted
tracedimension was hardcoded to chunk size 1 and anint16counter. That caps a gather at 32,767 duplicates — past which the counter raisesOverflowError— and forces one chunk per trace. A full CRG campaign runs to roughly 2.6e5 segments per receiver, and chunk-per-trace gives unusably small objects.GridOverrides.chunksizenow applies tohas_duplicatesas well asnon_binned.GridOverrides.trace_dtypesets the dtype of the inserted dimension — both the counter and the coordinate it is stored as. It is typedScalarTypeand restricted to integer types, so a dtype that cannot count traces is rejected at construction instead of failing deep inside numpy. It reaches thenon_binnedpath too, since that inserts the same axis and has the same ceiling.Omitting both reproduces today's behaviour exactly: chunk 1, an
int16counter stored as anint32coordinate. That pair of legacy defaults is named (LEGACY_COUNTER_DTYPE,LEGACY_TRACE_DTYPE) and pinned by explicit backwards-compatibility tests.This part also generalizes
_resolve_synthesize_dimsto read a template'ssynthesize_missing_dimsattribute instead of isinstance-checkingSeismic3DObnReceiverGathersTemplate. The OBN template already sets that attribute, so its behaviour is unchanged; this just lets other templates opt in.ingestion/segy/validation.pykeys its optional-dimension exemption off the same attribute so the two cannot drift, and the attribute is now declared onAbstractDatasetTemplateitself rather than assigned in__init__, so consumers can read the hook off any template without agetattrfallback.The template
ObnContinuousReceiverGathers3D, registered as a built-in. Grids on the three spatial dimensions and leaves the segment axis toHasDuplicates.componentis declared insynthesize_missing_dims, so single-component files carrying no component header ingest against the same template with the dimension synthesized — one template covers both layouts.Tests
tests/unit tests/integration -m "not cloud"→ 569 passed.tracehonourschunksizeandtrace_dtypein both the header counter and the stored coordinate, plus BC tests asserting the default path still yields chunk 1, anint16counter and anint32coordinate.trace_dtypeis rejected at construction, and a gather that outgrows the counter raisesOverflowErrorrather than wrapping to a wrong index.uint32segment axis and the realized chunk shape, and one exercising the synthesized-componentpath.Questions for maintainers
Is the API shape right for the new knobs? I put
trace_dtypeonGridOverridesalongside the existingchunksize, since that is wherenon_binnedalready carries its chunk size and it keeps the two paths symmetric. It governs the counter and the stored coordinate together, on both paths that insert atraceaxis. The alternative is to carry it on the template instead, which would keepGridOverridesnarrower but split the configuration for one dimension across two objects. Happy to move it.Is "extend
HasDuplicates" the direction you want? The alternative is a dedicated override (something likeCalculateTimeSegmentIndex) that does not touch the existing path at all. Extending seemed better because the semantics are genuinely identical — a counter disambiguating repeated index tuples — and a separate override would duplicate the machinery. But it does widen a shipping public surface, so say the word if you would rather have this isolated.