perf(parquet): skip RowFilter on statically fully-matched row groups - #23696
Open
zhuqi-lucas wants to merge 20 commits into
Open
perf(parquet): skip RowFilter on statically fully-matched row groups#23696zhuqi-lucas wants to merge 20 commits into
zhuqi-lucas wants to merge 20 commits into
Conversation
github-actions
Bot
added
core
zhuqi-lucas
marked this pull request as draft
July 19, 2026 03:28
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Parquet scan performance by avoiding per-row RowFilter evaluation for row groups proven (via statistics/static pruning) to be fully matched, and by enabling cheap per-row-group rebuilds of the row filter using prebuilt candidates. This builds on arrow-rs’s peek_next_row_group() to keep DataFusion’s row-group plan aligned with the decoder’s actual frontier.
Changes:
- Track per-row-group
fully_matchedstate in the Parquet access plan and preserve it through reorder/reverse operations. - Toggle the decoder’s
RowFilterat row-group boundaries (empty filter for fully-matched RGs; real filter otherwise) and add a new metric for observability. - Refactor row-filter construction to prebuild filter candidates once per file and rebuild cheaply per row group; update integration + SQL logic tests accordingly.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| datafusion/datasource-parquet/src/access_plan.rs | Carries per-RG fully_matched flags into the prepared plan and strips empty-selection RGs to keep plan/decoder alignment. |
| datafusion/datasource-parquet/src/metrics.rs | Adds row_filter_skipped_fully_matched metric to Parquet file metrics. |
| datafusion/datasource-parquet/src/push_decoder.rs | Adds per-RG plan entries and implements boundary-time RowFilter toggling using peek_next_row_group(). |
| datafusion/datasource-parquet/src/row_filter.rs | Splits row-filter building into “prebuild once per file” + “instantiate per RG” and expands nested/struct field planning logic. |
| datafusion/datasource-parquet/src/opener/mod.rs | Wires the prebuilt row-filter context into decoder construction and propagates new metric/state into the stream. |
| datafusion/datasource-parquet/src/sort.rs | Updates prepared-plan reverse test to reflect stripping of all-empty selections. |
| datafusion/core/tests/parquet/dynamic_row_group_pruning.rs | Adds an end-to-end integration test asserting fully-matched RGs trigger row-filter skipping behavior. |
| datafusion/sqllogictest/test_files/push_down_filter_parquet.slt | Updates EXPLAIN ANALYZE expectations to include the new metric. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+64
to
+68
| /// Number of row groups for which the per-row | ||
| /// [`RowFilter`](parquet::arrow::arrow_reader::RowFilter) was skipped | ||
| /// because the static stats proved every row of the RG satisfies the | ||
| /// predicate. The decoder is rebuilt at the boundary with an empty | ||
| /// row filter so the upcoming RG decodes without per-row evaluation. |
Contributor
Author
There was a problem hiding this comment.
Fixed in fea0264 — the doc now states explicitly that this counts suppression events, not row groups: a run of consecutive fully-matched RGs shares a single toggle.
| && (!DataType::is_nested(return_type) | ||
| || self.is_nested_type_supported(return_type)) | ||
| { | ||
| // try to resolve all field name arguments to strinrg literals |
Comment on lines
+192
to
+196
| /// The result of resolving which Parquet leaf columns and Arrow schema fields | ||
| /// are needed to evaluate an expression against a Parquet file | ||
| /// | ||
| /// This is the shared output of the column resolution pipeline used by both | ||
| /// the row filter to build `ArrowPredicate`s and the opener to build `ProjectionMask`s |
Contributor
Author
There was a problem hiding this comment.
Fixed in fea0264 — row_filter.rs now reuses the shared ParquetReadPlan / PushdownChecker machinery from projection_read_plan.rs instead of carrying its own copy (~700 lines removed).
| .map(|p| replace_columns_with_literals(p, &literal_columns)) | ||
| .transpose()?; | ||
| } | ||
|
|
Contributor
Author
There was a problem hiding this comment.
Good catch — this was real: the rebase had silently dropped the input_file_name() rewrite. Restored in fea0264 along with a regression test (test_input_file_name_projection).
Contributor
Author
|
run benchmarks env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark sort_tpch env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: sort_tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagesort_tpch — base (merge-base)
sort_tpch — branch
File an issue against this benchmark runner |
zhuqi-lucas
force-pushed
the
perf/parquet-skip-filter-for-full-match
branch
from
July 31, 2026 14:13
a3047ed to
fea0264
Compare
Builds on the `peek_next_row_group` API landed in apache/arrow-rs#10158. When Parquet stats prove that every row of a row group already satisfies the pushdown predicate (`fully_matched`), running the per-row `RowFilter` inside that row group is pure overhead — every row passes anyway. This change installs an *empty* `RowFilter` on fully-matched runs and only pays the row-level machinery on RGs that still need filtering. * `access_plan.rs` — `ParquetAccessPlan` now tracks per-RG `fully_matched` state, produced during static pruning. * `metrics.rs` — new `row_filter_skipped_fully_matched` counter to observe how often the toggle fires. * `push_decoder.rs` — `RgPlanEntry` carries the per-RG toggle; `RowFilterContext` swaps between the real filter (from the `prebuild_row_filter_candidates` cache) and an empty filter as the decoder crosses row-group boundaries. * `row_filter.rs` — split `build_row_filter` into `prebuild_*` and `row_filter_from_prebuilt`: the expensive tree walk + candidate construction runs once per file, and the cheap per-RG bind runs at each toggle. Preserves the existing public `build_row_filter` API for non-toggling callers. * `opener/mod.rs` — wires the prebuilt cache into the decoder builder; skips page-index loading for fully-matched RGs (their per-row filter is a no-op, so page pruning saves nothing). * `sort.rs` — sort-order-aware RG reorder preserves the per-RG toggle through reordering. * `fully_matched_rgs_skip_row_filter` (`dynamic_row_group_pruning.rs`) — 4 RGs of 3 rows each, predicate `v >= 3` makes RGs 1..=3 fully matched. Asserts: (a) results are correct, (b) `row_filter_skipped_fully_matched` fires at least once at the RG 0 → RG 1 boundary. * All 6 pre-existing `dynamic_row_group_pruning` tests still pass. * Full `datasource-parquet` lib suite (158 tests) passes.
…t_file_name handling lost in rebase Three fixes on top of the rebase onto latest main: 1. row_filter.rs carried a stale pre-apache#23638 copy of the column-resolution machinery (ParquetReadPlan, PushdownChecker, StructFieldAccess, build_filter_schema, etc.) that main has since moved into projection_read_plan.rs. Rebuilt the file on main's shared-module version, keeping only this PR's real additions (PrebuiltRowFilterCandidate, prebuild_row_filter_candidates, row_filter_from_prebuilt, and the RowFilterGenerator removal): 2194 → 1483 lines, -798 net. The stray "strinrg" typo lived in the deleted copy. 2. The rebase silently dropped main's input_file_name() handling in ParquetMorselizer (import, the per-file literal rewrite in open(), and the test_input_file_name_projection test) because this PR's restructure of open() overlapped the region apache#23638 later touched. Restored all three; the test passes again. 3. metrics: the row_filter_skipped_fully_matched doc claimed to count row groups, but the counter increments once per suppression event (a run of consecutive fully-matched RGs shares a single toggle). Doc now states the actual semantics. Verified: 158 datasource-parquet tests pass (with PARQUET_TEST_DATA), cargo doc clean, clippy clean on touched files.
…redAccessPlan::new, regenerate slt for new metric, fix redundant doc link
zhuqi-lucas
force-pushed
the
perf/parquet-skip-filter-for-full-match
branch
from
August 6, 2026 13:30
fea0264 to
2445739
Compare
Contributor
Author
|
Update on the split:
On PR 2 (prebuild once per file) — I don't think it stands alone, and I'd suggest folding the prebuild into PR 4 (#23696) instead. The perf premise doesn't hold on current The only per-RG rebuild is #23696's own Does folding PR 2 into PR 4 sound right to you? If so I'll rebase #23696 onto |
Contributor
…ilter-for-full-match # Conflicts: # datafusion/datasource-parquet/src/access_plan.rs # datafusion/datasource-parquet/src/opener/mod.rs # datafusion/datasource-parquet/src/push_decoder.rs # datafusion/datasource-parquet/src/sort.rs # datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt
… slt churn) Follow the add_page_index_pages_skipped_by_fully_matched pattern from apache#21637: the counter is registered only when a suppression actually fires, via a RowFilterSkippedFullyMatchedMetric holder that keeps a live Count handle after first registration (the stream records suppressions as they happen, so a fire-once helper is not enough). - ParquetFileMetrics loses the eager pub field (no public API change, no semver flag). - 41 zero-valued row_filter_skipped_fully_matched=0 entries disappear from EXPLAIN ANALYZE baselines across push_down_filter_parquet.slt, explain_analyze.slt and dynamic_filter_pushdown_config.slt; the =1 assertions in dynamic_row_group_pruning.slt remain. Suggested by @adriangb in the apache#23696 decomposition review.
The test was byte-identical but had moved ~60 lines during an earlier rebase, inflating the diff. Restore it to its position on main.
…ilter-for-full-match # Conflicts: # datafusion/datasource-parquet/src/opener/mod.rs
zhuqi-lucas
added a commit
to zhuqi-lucas/arrow-datafusion
that referenced
this pull request
Aug 22, 2026
…r-setup block (apache#24572) ## Which issue does this PR close? - Closes apache#24286. ## Rationale for this change Split out from apache#23696 (per @adriangb's [decomposition proposal](apache#23696 (comment)), PR 3). The decoder-setup block in the opener returns a bare `(decoder, rg_plan, has_row_selection)` tuple. Giving it a named `InitialDecoderState` struct keeps the values carried out of the block self-documenting, and means the follow-up feature PR (apache#23696) adds fields to a named struct instead of reshaping a positional tuple. ## What changes are included in this PR? - `push_decoder.rs`: new `pub(crate) struct InitialDecoderState { decoder, rg_plan, has_row_selection }`. - `opener/mod.rs`: the decoder-setup block returns `InitialDecoderState { .. }` and is destructured back into the same locals. Pure refactor, no behavior change. ## Are these changes tested? Covered by the existing `dynamic_row_group_pruning` integration tests (they drive the opener path); all pass. `clippy --all-targets --all-features -D warnings` is clean. ## Are there any user-facing changes? No.
… + row_filter_from_prebuilt One path now splits conjuncts, orders candidates (sort_unstable_by_key on required_bytes under reorder_predicates), and wires metrics (every predicate shares pushdown_rows_pruned, only the last counts pushdown_rows_matched) — the once-per-file open path and the per-row-group RowFilterContext rebuild path can no longer drift. DatafusionArrowPredicate::try_new becomes a #[cfg(test)] convenience; the production paths construct predicates from prebuilt candidates. Folds the PR 2 piece of @adriangb's decomposition into this PR: on main the prebuilt list would have no consumer (into_builder preserves the RowFilter across runtime-prune rebuilds), so it lands together with the per-RG toggle that consumes it.
Contributor
Author
|
@adriangb done — this PR is now reshaped per your proposal:
Net: 11 files / +844 −207 → 8 files / +636 −170, with zero churn in unrelated slt baselines. PR description updated. Ready for another look. |
adriangb
reviewed
Aug 23, 2026
Comment on lines
+678
to
+684
| None => { | ||
| // Filter could not be rebuilt; install an empty filter | ||
| // so the decoder runs unfiltered for this run rather | ||
| // than failing. | ||
| builder = builder.with_row_filter(RowFilter::new(vec![])); | ||
| self.filter_installed = false; | ||
| } |
Contributor
There was a problem hiding this comment.
Could we make build_row_filter return a RowFilter instead of an Option when self.prebuilt.is_empty() and make this arm unreacheable?
Contributor
Author
There was a problem hiding this comment.
Thanks, done in dbdfcee — build_row_filter now returns RowFilter directly (try_new guarantees a non-empty prebuilt list), the unreachable arm and PrebuiltRowFilterCandidateList::is_empty are gone.
Comment on lines
+1555
to
+1558
| let initial_filter = precomputed_context | ||
| .as_ref() | ||
| .and_then(|ctx| ctx.build_row_filter()); | ||
| let row_filter_context = precomputed_context; |
Contributor
There was a problem hiding this comment.
This is discarded if the first row group is fully matched - can we construct it lazily?
Contributor
Author
There was a problem hiding this comment.
Thanks, done in dbdfcee — the initial filter is now built lazily in the non-fully-matched branch only, nothing is constructed just to be discarded.
| /// Extracting this from [`build_row_filter`] lets callers pay the tree-walk + | ||
| /// column-resolution + `reassign_expr_columns` cost **once per file** instead | ||
| /// of once per row group, which is the hot path for | ||
| /// [`RowFilterContext::build`](crate::push_decoder::RowFilterContext) rebuilds |
Contributor
There was a problem hiding this comment.
I think the method is build_row_filter not build
Contributor
|
run benchmark parquet_row_filter_skip |
Contributor
|
run benchmark parquet_row_filter_skip env:
RG_SIZE: "100000" |
Contributor
|
run benchmark clickbench_pushdown clickbench_sorted env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5a173c9) to fc15e25 (merge-base) diff Run configurationrun benchmark parquet_row_filter_skipResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5a173c9) to fc15e25 (merge-base) diff Run configurationrun benchmark parquet_row_filter_skip
env:
RG_SIZE: "100000"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5a173c9) to fc15e25 (merge-base) diff Run configurationrun benchmark clickbench_sorted
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5a173c9) to fc15e25 (merge-base) diff Run configurationrun benchmark clickbench_pushdown
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (5a173c9) to fc15e25 (merge-base) diff Run configurationrun benchmark clickbench_pushdown
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)Details
Resource Usageclickbench_pushdown — base (merge-base)
clickbench_pushdown — branch
File an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (5a173c9) to fc15e25 (merge-base) diff Run configurationrun benchmark clickbench_sorted
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)Details
Resource Usageclickbench_sorted — base (merge-base)
clickbench_sorted — branch
File an issue against this benchmark runner |
…nitial filter, doc name - RowFilterContext::try_new only constructs a context when the prebuilt list is non-empty, so build_row_filter now returns RowFilter directly and the unreachable empty arm (and PrebuiltRowFilterCandidateList::is_empty) is gone. - The opener no longer builds the initial RowFilter eagerly just to discard it when the first row group is fully matched; it is built lazily in the non-fully-matched branch only. - Fix doc reference: RowFilterContext::build -> build_row_filter.
Contributor
Author
|
The benchmark result looks good: │ QQuery 0 │ 85.42 / 165.14 ±152.92 / 470.95 ms │ 84.91 / 107.81 ±17.92 / 139.75 ms │ +1.53x faster │ |
Contributor
|
run benchmark parquet_row_filter_skip env:
CARGO_BUILD_JOBS: "3" |
Contributor
|
run benchmark parquet_row_filter_skip env:
CARGO_BUILD_JOBS: "3"
RG_SIZE: "100000" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (dbdfcee) to fc15e25 (merge-base) diff Run configurationrun benchmark parquet_row_filter_skip
env:
CARGO_BUILD_JOBS: "3"
RG_SIZE: "100000"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (dbdfcee) to fc15e25 (merge-base) diff Run configurationrun benchmark parquet_row_filter_skip
env:
CARGO_BUILD_JOBS: "3"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (dbdfcee) to fc15e25 (merge-base) diff Run configurationrun benchmark parquet_row_filter_skip
env:
CARGO_BUILD_JOBS: "3"
RG_SIZE: "100000"CPU Details (lscpu)Details
Resource Usageparquet_row_filter_skip — base (merge-base)
parquet_row_filter_skip — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (dbdfcee) to fc15e25 (merge-base) diff Run configurationrun benchmark parquet_row_filter_skip
env:
CARGO_BUILD_JOBS: "3"CPU Details (lscpu)Details
Resource Usageparquet_row_filter_skip — base (merge-base)
parquet_row_filter_skip — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
fully_matchedRowFilter skip · needs arrow-rs#10158 (peek_next_row_group) #23067.Rationale for this change
When Parquet statistics prove that every row of a row group already satisfies the pushdown predicate (
fully_matched, tracked byParquetAccessPlansince #21637), running the per-rowRowFilterinside that row group is pure overhead — every row passes anyway. This PR teaches the push-decoder stream to toggle theRowFilteroff across fully-matched runs and back on at the next straddling row group, so fully-matched runs decode without per-row evaluation (and two-phase reads skip fetching filter-only columns for them). Common shape: time-partitioned tables (WHERE ts >= X AND ts < Ywith whole row groups inside[X, Y)) and the middle stretch of a TopK scan.This PR was decomposed per @adriangb's proposal; the standalone pieces already merged:
strip_empty_row_groupsinPreparedAccessPlan::prepare([Follow-up #23696] Extract generic empty-row-group stripping from strip_empty_row_groups #24287)InitialDecoderStateextraction ([Follow-up #23696] Extract the initial push-decoder setup into a named struct #24286)What remains here is the feature itself plus the prebuild split it consumes (the proposal's PR 2 is folded in rather than standalone: on
main,into_builder()preserves theRowFilteracross runtime-prune rebuilds, so a prebuilt candidate list would have no consumer and no perf story on its own — the per-RG toggle here is its only consumer).What changes are included in this PR?
access_plan.rs—PreparedAccessPlancarries per-RGfully_matchedflags, mapped from the access plan after the (generic) empty-row-group strip;reorder_by_statistics/reversepermute them alongside the indexes.row_filter.rs— split intoprebuild_row_filter_candidates(once per file: conjunct split, candidate construction,reassign_expr_columns) androw_filter_from_prebuilt(per boundary: metric wiring + optionalrequired_bytesordering, no tree walks). The publicbuild_row_filteris reimplemented on top of the two, so the open-time path and the per-RG rebuild path share one conjunct-split/order/metric implementation and cannot drift.DatafusionArrowPredicate::try_newbecomes a#[cfg(test)]convenience.push_decoder.rs—RgPlanEntry.fully_matched;RowFilterContext(the prebuilt list + settings) letsrebuild_decoder_at_boundaryswap between the real filter and an empty one as the stream crosses row-group boundaries; at most oneinto_builderrebuild per boundary, shared with the runtime pruner.opener/mod.rs— prebuilds once per file, installs the initial filter based on the first row group'sfully_matchedstate (two new fields onInitialDecoderState).metrics.rs— newrow_filter_skipped_fully_matchedcounter, registered lazily on first suppression (RowFilterSkippedFullyMatchedMetric), following thepage_index_pages_skipped_by_fully_matchedpattern from Skip RowFilter and page pruning for fully matched row groups #21637: no public field onParquetFileMetrics(no API/semver change) and no zero-valued entries inEXPLAIN ANALYZEbaselines — the previously-regenerated 41 baselines across 3 slt files are untouched by this PR. It counts suppression events, not row groups: a run of consecutive fully-matched RGs shares one toggle.Considered but deferred: carrying one
Vec<{index, fully_matched}>instead of the two positionally-aligned vectors. Alignment is now maintained in exactly two adjacent permutations (reorder_by_statistics,reverse) plus adebug_assert, and the type change would churn ~40 assertion lines inmain-side tests — the opposite of shrinking this PR. Happy to do it as a tiny follow-up.Are these changes tested?
fully_matched_rgs_skip_row_filter— 4 RGs, predicatev >= 3 AND v <= 10makes RG 0 and RG 3 straddlers with a fully-matched run in between, covering the full toggle lifecycle (ON → OFF → back ON); asserts correctness, that 11/12 are filtered out by the reinstalled filter, androw_filter_skipped_fully_matched >= 1.dynamic_row_group_pruning.slt— anEXPLAIN ANALYZEcase surfacingrow_filter_skipped_fully_matched=1, so the optimization is observable from SQL and a regression shows up as the metric disappearing.dynamic_row_group_pruningintegration tests (10), thedatafusion-datasource-parquetlib suite (240), and the affected slt files pass;clippy --all-targets --all-features -D warningsclean.Are there any user-facing changes?
row_filter_skipped_fully_matchedappears inEXPLAIN ANALYZEon parquet scans only when the toggle actually fired.build_row_filterkeeps its signature and behavior.