Skip to content

[FLINK-41586][runtime] Obtain delegation tokens for the lifetime of the JobManager process - #30043

Open
gaborgsomogyi wants to merge 3 commits into
apache:masterfrom
gaborgsomogyi:FLINK-41586
Open

gaborgsomogyi wants to merge 3 commits into
apache:masterfrom
gaborgsomogyi:FLINK-41586

Conversation

@gaborgsomogyi

Copy link
Copy Markdown
Contributor

What is the purpose of the change

A JobManager obtains its delegation tokens when the process starts. The periodic obtain of new tokens was started by the ResourceManager of a leader session and stopped with it. A standby JobManager, and a JobManager which lost the leadership, therefore never obtains new tokens for the process-local receivers it holds, for example the S3 or HDFS credentials the high availability storage is read with.

When such a JobManager becomes the leader after its tokens expired, the Dispatcher recovery reads the HA storage right away, while the ResourceManager, which would obtain new tokens, is still starting. The read fails with Could not retrieve JobResults of globally-terminated jobs from JobResultStore, which is a fatal error, and the new leader exits.

This PR makes obtaining the tokens a property of the process and the distribution of the tokens to the TaskManagers a property of the leader session.

Brief change log

  • DelegationTokenManager (@Internal): start() and stop() belong to the process, attachListener() and detachListener() to a leader session. start(Listener) and the no-argument obtainDelegationTokens() are removed, the overload with a DelegationTokenContainer stays (YARN client).
  • DefaultDelegationTokenManager: one chain of obtains per process (CREATED -> RUNNING -> STOPPED). start() obtains inline and fails fast, as the one-time obtain did. attachListener() hands the newest tokens to the listener without contacting the token services. The callbacks to the listener are serialized, so it never gets older tokens after newer ones. detachListener() waits for the callback in progress, so none runs after it returns (bounded by the RPC timeout of the TaskManager update). The Clock can be injected.
  • ResourceManager attaches and detaches itself instead of starting and stopping the periodic obtain.
  • ClusterEntrypoint and MiniCluster start the manager where they obtained the tokens once, and stop it when they shut down. ClusterEntrypoint#createDelegationTokenManager is protected so that the test can see the calls. A failing stop() of the manager is collected like the failures of the other services, so the remaining services are stopped and the termination future of the entrypoint completes.
  • security-delegation-token.md: section "Delegation Tokens and High Availability".
  • Naming (second commit, no behavior change): the manager never renews a token, it obtains new ones before the old ones expire. The names, Javadoc and log texts of DefaultDelegationTokenManager say obtain and expiry (validUntil) instead of renewal and next renewal. The keys of the config options keep their "renewal" wording.
  • DelegationTokenProviderRepository (new, @Internal, the counterpart of DelegationTokenReceiverRepository): the provider handling moved out of DefaultDelegationTokenManager unchanged (loading, the enabled flag, init, duplicate and prefix checks, obtaining the tokens and their expiry). The manager keeps the lifecycle, the scheduling, the listener, the check that every provider has a receiver, and obtainDelegationTokensAndGetExpiry as the seam for tests. The log lines of the provider handling come from the new logger (third commit, no behavior change).

Behavior changes:

  • Standby JobManagers keep obtaining new delegation tokens, so they contact the configured token services as well. N JobManagers issue N times as many tokens.
  • The start of a leader session does not obtain tokens any more, it uses the tokens its process holds. Providers which report no validUntil obtain tokens only when the process starts.
  • An Error of a provider during an obtain now also ends a standby JobManager, as it ended the leader before.

Verifying this change

Added new automated tests + triggered HA failover on real cluster.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code, see the Generated-by trailer of the commit)

Generated-by: Claude Code with Claude Sonnet 5.5

@flinkbot

flinkbot commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@gaborgsomogyi

Copy link
Copy Markdown
Contributor Author

I've intentionally split the changes into 3 parts to see that there are 3 intentions. At the end this will be a single one...

@gaborgsomogyi

Copy link
Copy Markdown
Contributor Author

cc @gyfora @Savonitar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants