Skip to content

[core] Prune manifests using individual partition values - #11277

Open
Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:fix/prune-full-compaction-by-delete-partitions
Open

Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:fix/prune-full-compaction-by-delete-partitions

Conversation

@Zouxxyy

@Zouxxyy Zouxxyy commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Addresses #5683 in the shared multi-partition statistics predicate.

Suppose the requested partitions are (day=1, event=A) and (day=19, event=B). A manifest spanning days 9 through 11 passes the combined bounds even though neither requested partition occurs in that range. This can cause scans and overwrite commits to open unnecessary manifest bodies.

After the combined bounds pass, compare the field values of each requested partition with the per-field manifest statistics. Stop at the first possible match. Skip the manifest only when at least one field rules out each requested partition. Unknown statistics and null counts are handled conservatively. A single requested partition is already covered by the existing bounds check.

The existing fromMultiple callers share this behavior, including ordinary manifest scans, overwrites, full compaction, and manifest sort. Per-partition values are decoded once during predicate construction. Manifest bounds are decoded once for the inner loop.

Tests

  • Exercised multi-partition manifest scanning, block and legacy full compaction, and full compaction with manifest sort enabled. The tests verify candidate manifests and surviving file names. Predicate tests cover correlated values, range gaps, inclusive bounds, NULLs, missing statistics and serialization.
  • With Spark 3.5.9 on a local filesystem, created 38 real base manifests, each covering two hours. Tracing file opens showed that both an ordinary two-partition file scan and a five-column dynamic overwrite opened 38 base manifest bodies on master and 2 with the change. The scan planned the two expected data files, and the final overwrite result matched all 76 expected rows in both runs.
  • Ran six JDK 8 suites: PartitionPredicateTest, ManifestFileMetaTest, ManifestFileTest, NoPartitionManifestFileMetaTest, KeyValueFileStoreScanTest and FileStoreCommitTest.

The worst case takes work proportional to candidate manifests × distinct requested partitions × partition fields. In a single-thread local JDK 8 microbenchmark with 13,151 synthetic candidate manifests and five string fields, only the last partition matched every candidate. Checking 288 partitions took about 0.4 s, 682 took about 0.9 s, and 10,000 took 19–24 s. The previous combined-bounds check took under 0.02 s for these cases. These timings measure calls to the statistics predicate alone, with zero files pruned.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I just add it directly, rather than passing a bool? Also, the name "tuple" seems a bit odd—aren't these just statistics at the field level?

@Zouxxyy Zouxxyy changed the title [core] Prune full compaction manifests by individual delete partitions [core] Prune manifests using individual partition values Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants