Repository navigation
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Parallel DV rewriting has value for large Data Evolution compactions. I ran the submitted head locally on JDK 8. All 65 Core, 13 Flink, and 60 Spark tests passed. The current master merge conflicts in DataEvolutionRowIdRewriteSink.java after the range-sort change, so the combined rewrite paths still need an integration run after rebasing.
[P2] Avoid coupling independently updated retained anchors. I used a Parquet table with two old DV indexes. Each index contained a compacted anchor and a distinct untouched sibling. After compaction the old path kept the two retained sibling vectors in separate index files. The new writer packed both siblings into the same file. I prepared two DELETE commits from that post-compaction snapshot, each touching one of the distinct siblings, then committed them in order. Both commits succeeded on master. The second commit on this PR failed with Trying to delete file ... which is not previously added. Please preserve the old index ownership boundaries for retained entries, or handle shared index replacement without failing these disjoint deletes.
[P2] Clean up earlier Flink task outputs when a worker attempt fails. I ran bounded Flink compaction on a partition with two DV rewrite tasks and made the second old index unavailable. The snapshot stayed at 6. With one configured restart, the current PR left two uncommitted DV files, one from each attempt. The serial control left zero. DataEvolutionDeletionVectorRewriteOperator emits each successful rewrite before running the next, and the commit-preparation stage never receives a complete attempt on failure. Please track outputs per worker attempt and remove the earlier uncommitted DV files when a later task in that attempt fails.
|
I also verified the failed rewrite in Spark 3.5.9. Two rewrite tasks ran in one local job and the second task could not open its source DV. The serial control left zero new index files. The distributed path left one uncommitted DV file and the snapshot stayed unchanged. The driver received a |
ce99c68 to
bf334d6
Compare
|
Thanks for the reproductions. Addressed in bf334d6:
Rebased onto master and verified the range-sort paths. Local validation on JDK 11 passed: 68 core, 23 Flink, and 69 Spark tests. |
Purpose
Closes #11271.
Run Data Evolution DV index rewrites on Flink workers and Spark executors, with multiple tasks per partition. Metadata planning and final commits remain centralized.
Keep retained DV entries separate for each source index and from rewritten anchors, preserving the conflict boundaries of subsequent DELETEs. Track worker-attempt outputs for failure cleanup and handle Spark's checked exceptions when aborting known retry artifacts.
Document orphan-file recovery for outputs from already completed tasks or lost workers, and verify the recovery path with a failed Spark job.
Tests
Validated locally on JDK 11:
CompactProcedureTestandSortDataEvolutionProcedureTest, including a real DV rewrite job failure and orphan-file recovery.git diff --check.