Retry moveMasterPodsOffNode on failure instead of aborting - #3179
Open
pdixit26 wants to merge 2 commits into
Open
Retry moveMasterPodsOffNode on failure instead of aborting#3179pdixit26 wants to merge 2 commits into
pdixit26 wants to merge 2 commits into
Conversation
attemptToMoveMasterPodsOffNode's transient errors (e.g. no synced standby available yet) were treated as terminal, so the retry loop gave up after a single ~4-5s attempt instead of retrying over master_pod_move_timeout. This left masters stuck on draining nodes indefinitely once a PodDisruptionBudget started rejecting evictions. Log the failure and return (false, nil) so the retry loop keeps polling every minute until the timeout expires. Fixes zalando#3176
pdixit26
requested review from
FxKu,
Jan-M,
idanovinda,
jopadi and
mikkeloscar
as code owners
September 1, 2026 18:06
Author
|
This is a bugfix for #3176. I can't add the label myself. |
Author
|
@FxKu Could I get review on this PR? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
moveMasterPodsOffNodegave up after a single failed attempt instead ofretrying over the configured
master_pod_move_timeout. Any error fromattemptToMoveMasterPodsOffNode(e.g. no synced standby available yet)was treated as terminal by
retryutil.Retry, so the operator got one~4-5s attempt and then permanently gave up, leaving the master stuck on
a draining node.
Fix
Log the failure and return
(false, nil)on error so the retry loopkeeps polling every minute until
master_pod_move_timeoutexpires,matching the documented behavior.
Testing
Added
TestMoveMasterPodsOffNodeRetriesOnError, which forcesattemptToMoveMasterPodsOffNodeto fail and asserts the retry loopruns to timeout rather than aborting on the first error.
Fixes #3176