Skip to content

[core] Prune merged Data Evolution reads with shared row-ID ranges - #1469

Open
SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1445
Open

SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1445

Conversation

@SteNicholas

Copy link
Copy Markdown
Member

Purpose

Closes #1445.

A selective filter on a Data Evolution table can make the reader decode an entire group of column files. Older files can hold overwritten values, and the current providers can have different Parquet page boundaries. This change identifies candidate row IDs from the files that supply the filtered columns and reads those positions from every contributing file.

Brief change log

  • Resolve column ownership by stable field ID, historical schema and file version. Use a file's row-group statistics and page indexes for a conjunct only when it supplies every referenced field. Evaluate predicates spanning providers on the reconstructed rows.
  • Intersect candidate row IDs with requested ranges and file-index hits, exclude deleted rows, and apply the same selection to every source. Attach row IDs per batch and evaluate the full predicate before LIMIT.
  • Reuse the Parquet footers and indexes loaded during planning. Fall back to row-group statistics when page positions or byte ranges are invalid, and restore the original selection when file and manifest row counts disagree.
  • Enable pruning for ordinary Parquet files that share a complete row-ID range on row-tracked append tables with bucket=-1. Dedicated BLOB and vector groups, nested-field evolution and ROW sidecar reads use their existing read paths.

Tests

  • 4,996 core unit tests passed with 8 ignored by default. 45 integration tests passed across bitmap merges, ROW sidecars, vector compaction, system columns, partition restoration and Parquet page indexes. Coverage includes renamed and recreated columns, nulls, mismatched page boundaries and corrupt offsets. Counting-I/O tests verify one footer fetch per provider and reuse of a separately fetched page index.
  • Ran the selective and full-range benchmark over 262,144 logical rows on an Apple M4 Pro. The selective filter returned the same row in both modes. Pushdown reduced rows delivered to the merge from 524,288 to 128, column-data bytes from 2,342,012 to 756, and median scan time from 61.8 ms to 13.0 ms. The full-range filter returned all rows in both modes, with a 4.4% increase in median scan time.
  • Formatting and full-workspace Clippy passed on Rust 1.98.0 with --all-targets --workspace --features vortex -- -D warnings.

API and Format

CoreOptions::data_evolution_merged_read_stats_pushdown_enabled() reads the Java-compatible table option data-evolution.merged-read.stats-pushdown.enabled. It defaults to true. Set it to false to disable merged-read statistics pruning. The reader uses metadata already stored in existing Parquet files.

Documentation

docs/src/getting-started.md describes when pruning applies, how it falls back and how to disable it. docs/src/benchmark.md includes the reproduction command, byte counts and seven-run median scan times for both filters.

🤖 Assisted with Codex.

A Data Evolution read could decode an entire merge group for a filter
matching one row. The merge selects columns by field ID and file
version, so older files can hold stale values. Parquet pages can also
start at different rows in each file. Independent file filters could
discard matches or combine values from different rows.

- Derive candidate row ranges from row-group and page statistics only
  when one file supplies every field in a conjunct, addressing apache#1445.
- Intersect candidates with caller ranges and file-index hits, exclude
  deleted rows, and read the same positions from every provider. Attach
  row IDs per batch and apply the full predicate to reconstructed rows
  before LIMIT.
- Reuse footers and indexes for decoding. Discard invalid offset indexes
  and keep the original ranges if file and manifest row counts differ.
- Honor Java's data-evolution.merged-read.stats-pushdown.enabled option
  (true by default). Measure rows returned to the merge, bytes read and
  scan time for selective and full-range filters in both modes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Prune Data Evolution merged reads using shared candidate row ranges

1 participant