[Flink] Fix dedicated streaming compaction after snapshot expiration - #9548
Open
lilei1128 wants to merge 1 commit into
Open
[Flink] Fix dedicated streaming compaction after snapshot expiration#9548lilei1128 wants to merge 1 commit into
lilei1128 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Make Flink Dedicated Streaming Compact jobs self-recoverable when snapshot
expiration races with compaction source reads or checkpoint recovery.
A previous compaction may expire the snapshot or manifest files currently
being read by the source, causing OutOfRangeException or
FileNotFoundException. The same expired snapshot or pending split may also be
restored from a checkpoint after failover, causing the job to repeatedly retry
the invalid snapshot and remain stuck.
The fix rebuilds the compaction baseline from the latest valid snapshot,
clears expired source state, synchronizes rebase information with all
compaction writers, and resumes incremental compaction without manual
intervention.
The core idea is to treat snapshot expiration as a coordinated source and
sink recovery:
dedicated compaction source.
valid snapshot.
compaction writers.
splits already covered by the rebase.
can continue from a valid baseline.
Tests