perf(ogc): faster page parsing and chunk combination#345
Open
thodson-usgs wants to merge 3 commits into
Open
Conversation
Add parallel_chunks(n) so large Water Data and NGWMN queries can split into more independently paginated sub-requests while preserving the conservative byte-limit-only default. Cap optional fan-out across all chunk axes, validate positive integer inputs, export the helper publicly, and document usage, tradeoffs, and benchmark results. Add planner, context, and integration coverage.
- Replace pd.json_normalize with pd.DataFrame in _get_resp_data. OGC API properties are flat dicts; json_normalize's recursive flattening is unnecessary overhead (~2.4× faster per page: 3.2 ms → 1.35 ms on 2500-feature pages). - Skip drop_duplicates in _combine_chunk_frames when the plan has no filter axis. List-axis chunks produce non-overlapping partitions, so dedup is pure overhead in the common parallel_chunks case (~2.7× faster combine: 3.4 ms → 1.25 ms on 52k-row results). - Add ChunkPlan.has_filter_axis property to distinguish plans that need dedup (filter-axis overlap possible) from those that don't (list-axis only).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two client-side performance improvements to the OGC page-parsing and chunk-combination paths, identified through profiling the
parallel_chunksfan-out code.Changes
Replace
pd.json_normalizewithpd.DataFramein_get_resp_data(shaping.py)OGC API feature properties are flat dictionaries —
json_normalize's recursive nesting support is unnecessary overhead. Directpd.DataFrameconstruction is 2.4× faster per page (3.2 ms → 1.35 ms on 2500-feature pages).Skip
drop_duplicateswhen the plan has no filter axis (planning.py, chunking.py)List-axis chunks (the common
parallel_chunkscase — splitting onmonitoring_location_id,parameter_code, etc.) produce non-overlapping partitions by construction. Only filter-axis chunks (splitting on CQLORclauses) can overlap._combine_chunk_framesnow accepts adedupflag andChunkedCall._combine_rawpassesdedup=self.plan.has_filter_axis— 2.7× faster combine (3.4 ms → 1.25 ms on 52k-row results) for the common case.Impact
For a typical 32-chunk query fetching 32 pages (~52k rows):
These are additive to the network-latency savings
parallel_chunksalready provides.Testing
All 574 tests pass. No behavioral change — list-axis chunks were never producing duplicates (the dedup was a no-op); filter-axis chunks still dedup as before.