Repository navigation
[core] Prune merged Data Evolution reads with shared row-ID ranges - #1469
Open
SteNicholas wants to merge 1 commit into
Open
SteNicholas wants to merge 1 commit into
SteNicholas wants to merge 1 commit into
Conversation
A Data Evolution read could decode an entire merge group for a filter matching one row. The merge selects columns by field ID and file version, so older files can hold stale values. Parquet pages can also start at different rows in each file. Independent file filters could discard matches or combine values from different rows. - Derive candidate row ranges from row-group and page statistics only when one file supplies every field in a conjunct, addressing apache#1445. - Intersect candidates with caller ranges and file-index hits, exclude deleted rows, and read the same positions from every provider. Attach row IDs per batch and apply the full predicate to reconstructed rows before LIMIT. - Reuse footers and indexes for decoding. Discard invalid offset indexes and keep the original ranges if file and manifest row counts differ. - Honor Java's data-evolution.merged-read.stats-pushdown.enabled option (true by default). Measure rows returned to the merge, bytes read and scan time for selective and full-range filters in both modes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #1445.
A selective filter on a Data Evolution table can make the reader decode an entire group of column files. Older files can hold overwritten values, and the current providers can have different Parquet page boundaries. This change identifies candidate row IDs from the files that supply the filtered columns and reads those positions from every contributing file.
Brief change log
LIMIT.bucket=-1. Dedicated BLOB and vector groups, nested-field evolution and ROW sidecar reads use their existing read paths.Tests
--all-targets --workspace --features vortex -- -D warnings.API and Format
CoreOptions::data_evolution_merged_read_stats_pushdown_enabled()reads the Java-compatible table optiondata-evolution.merged-read.stats-pushdown.enabled. It defaults totrue. Set it tofalseto disable merged-read statistics pruning. The reader uses metadata already stored in existing Parquet files.Documentation
docs/src/getting-started.mddescribes when pruning applies, how it falls back and how to disable it.docs/src/benchmark.mdincludes the reproduction command, byte counts and seven-run median scan times for both filters.🤖 Assisted with Codex.