Skip to content

[core][flink][spark] Parallelize data evolution DV rewrites - #11272

Open
zhoulii wants to merge 1 commit into
apache:masterfrom
zhoulii:improve/de-compaction-parallel-dv-rewrite
Open

zhoulii wants to merge 1 commit into
apache:masterfrom
zhoulii:improve/de-compaction-parallel-dv-rewrite

Conversation

@zhoulii

@zhoulii zhoulii commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Purpose

Closes #11271.

Run Data Evolution DV index rewrites on Flink workers and Spark executors, with multiple tasks per partition. Metadata planning and final commits remain centralized.

Keep retained DV entries separate for each source index and from rewritten anchors, preserving the conflict boundaries of subsequent DELETEs. Track worker-attempt outputs for failure cleanup and handle Spark's checked exceptions when aborting known retry artifacts.

Document orphan-file recovery for outputs from already completed tasks or lost workers, and verify the recovery path with a failed Spark job.

Tests

Validated locally on JDK 11:

  • Core: 68 tests, including disjoint DELETEs prepared from the same snapshot and DV write failure cleanup.
  • Flink 1.20: 23 tests covering worker failure/restart, cancellation, normal completion, SQL DELETE/materialization, and range sort.
  • Spark 3.5.9: 69 tests across CompactProcedureTest and SortDataEvolutionProcedureTest, including a real DV rewrite job failure and orphan-file recovery.
  • Spotless, Checkstyle, and git diff --check.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parallel DV rewriting has value for large Data Evolution compactions. I ran the submitted head locally on JDK 8. All 65 Core, 13 Flink, and 60 Spark tests passed. The current master merge conflicts in DataEvolutionRowIdRewriteSink.java after the range-sort change, so the combined rewrite paths still need an integration run after rebasing.

[P2] Avoid coupling independently updated retained anchors. I used a Parquet table with two old DV indexes. Each index contained a compacted anchor and a distinct untouched sibling. After compaction the old path kept the two retained sibling vectors in separate index files. The new writer packed both siblings into the same file. I prepared two DELETE commits from that post-compaction snapshot, each touching one of the distinct siblings, then committed them in order. Both commits succeeded on master. The second commit on this PR failed with Trying to delete file ... which is not previously added. Please preserve the old index ownership boundaries for retained entries, or handle shared index replacement without failing these disjoint deletes.

[P2] Clean up earlier Flink task outputs when a worker attempt fails. I ran bounded Flink compaction on a partition with two DV rewrite tasks and made the second old index unavailable. The snapshot stayed at 6. With one configured restart, the current PR left two uncommitted DV files, one from each attempt. The serial control left zero. DataEvolutionDeletionVectorRewriteOperator emits each successful rewrite before running the next, and the commit-preparation stage never receives a complete attempt on failure. Please track outputs per worker attempt and remove the earlier uncommitted DV files when a later task in that attempt fails.

@JingsongLi

Copy link
Copy Markdown
Contributor

I also verified the failed rewrite in Spark 3.5.9. Two rewrite tasks ran in one local job and the second task could not open its source DV. The serial control left zero new index files. The distributed path left one uncommitted DV file and the snapshot stayed unchanged.

The driver received a SparkException, which extends Exception on this version. It bypasses the RuntimeException catch around rewriteDeletionVectors, including the cleanup for any already-known rebased artifacts. Please cover the actual exception type in that cleanup path. The successful task's output also needs an attempt-level cleanup mechanism or a documented recovery path.

@zhoulii
zhoulii force-pushed the improve/de-compaction-parallel-dv-rewrite branch from ce99c68 to bf334d6 Compare November 20, 2026 07:35
@zhoulii

zhoulii commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Thanks for the reproductions. Addressed in bf334d6:

  • Preserve retained DV boundaries per source index, with a regression for two disjoint DELETEs prepared from the same snapshot.
  • Clean tracked Flink worker outputs on failure or cancellation, including a restart regression.
  • Handle checked SparkExceptions and clean known retry artifacts. Document and test orphan-file recovery for outputs from completed tasks.

Rebased onto master and verified the range-sort paths. Local validation on JDK 11 passed: 68 core, 23 Flink, and 69 Spark tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Parallelize DV index rewriting for Data Evolution compaction

2 participants