Repository navigation
[FLINK-41586][runtime] Obtain delegation tokens for the lifetime of the JobManager process - #30043
Open
gaborgsomogyi wants to merge 3 commits into
Open
gaborgsomogyi wants to merge 3 commits into
gaborgsomogyi wants to merge 3 commits into
Conversation
…he JobManager process
…s with the obtain vocabulary
… from the DefaultDelegationTokenManager
Collaborator
Contributor
Author
|
I've intentionally split the changes into 3 parts to see that there are 3 intentions. At the end this will be a single one... |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
A JobManager obtains its delegation tokens when the process starts. The periodic obtain of new tokens was started by the
ResourceManagerof a leader session and stopped with it. A standby JobManager, and a JobManager which lost the leadership, therefore never obtains new tokens for the process-local receivers it holds, for example the S3 or HDFS credentials the high availability storage is read with.When such a JobManager becomes the leader after its tokens expired, the Dispatcher recovery reads the HA storage right away, while the
ResourceManager, which would obtain new tokens, is still starting. The read fails withCould not retrieve JobResults of globally-terminated jobs from JobResultStore, which is a fatal error, and the new leader exits.This PR makes obtaining the tokens a property of the process and the distribution of the tokens to the TaskManagers a property of the leader session.
Brief change log
DelegationTokenManager(@Internal):start()andstop()belong to the process,attachListener()anddetachListener()to a leader session.start(Listener)and the no-argumentobtainDelegationTokens()are removed, the overload with aDelegationTokenContainerstays (YARN client).DefaultDelegationTokenManager: one chain of obtains per process (CREATED -> RUNNING -> STOPPED).start()obtains inline and fails fast, as the one-time obtain did.attachListener()hands the newest tokens to the listener without contacting the token services. The callbacks to the listener are serialized, so it never gets older tokens after newer ones.detachListener()waits for the callback in progress, so none runs after it returns (bounded by the RPC timeout of the TaskManager update). TheClockcan be injected.ResourceManagerattaches and detaches itself instead of starting and stopping the periodic obtain.ClusterEntrypointandMiniClusterstart the manager where they obtained the tokens once, and stop it when they shut down.ClusterEntrypoint#createDelegationTokenManageris protected so that the test can see the calls. A failingstop()of the manager is collected like the failures of the other services, so the remaining services are stopped and the termination future of the entrypoint completes.security-delegation-token.md: section "Delegation Tokens and High Availability".DefaultDelegationTokenManagersay obtain and expiry (validUntil) instead of renewal and next renewal. The keys of the config options keep their "renewal" wording.DelegationTokenProviderRepository(new,@Internal, the counterpart ofDelegationTokenReceiverRepository): the provider handling moved out ofDefaultDelegationTokenManagerunchanged (loading, the enabled flag, init, duplicate and prefix checks, obtaining the tokens and their expiry). The manager keeps the lifecycle, the scheduling, the listener, the check that every provider has a receiver, andobtainDelegationTokensAndGetExpiryas the seam for tests. The log lines of the provider handling come from the new logger (third commit, no behavior change).Behavior changes:
validUntilobtain tokens only when the process starts.Errorof a provider during an obtain now also ends a standby JobManager, as it ended the leader before.Verifying this change
Added new automated tests + triggered HA failover on real cluster.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-bytrailer of the commit)Generated-by: Claude Code with Claude Sonnet 5.5