feat: Enable Bound Token for Agentic Identities - #13873
macastelaz merged 41 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces Agent Identity token binding support for Cloud Run. It adds AgentIdentityUtils to resolve, load, and verify certificates and private keys, and updates ComputeEngineCredentials to request bound tokens via POST requests when a valid certificate chain is present. The review feedback suggests a cohesive improvement to implement a single-read pattern for certificate files. By reading the certificate chain once, caching it in CertInfo, and passing it to parseCertificate and getBoundTokenPayload, the implementation can avoid redundant disk I/O and prevent potential race conditions during certificate rotation.
macastelaz
changed the title
| // Environment variables | ||
| static final String GOOGLE_API_CERTIFICATE_CONFIG = "GOOGLE_API_CERTIFICATE_CONFIG"; | ||
| static final String GOOGLE_API_PREVENT_TOKEN_SHARING_FOR_GCP_SERVICES = | ||
| "GOOGLE_API_PREVENT_TOKEN_SHARING_FOR_GCP_SERVICES"; |
There was a problem hiding this comment.
Note that based on googleapis/google-cloud-python#17698 (comment) this is not yet finalized
f5e81cc to
db1c39c
Compare
db1c39c to
3ada55f
Compare
1. POST request to MDS with cert-chain 2. Cert-key matching 3. Included logic to consider the user's choice by looking at GOOGLE_API_USE_CLIENT_CERTIFICATE env variable 4. Bound ID tokens. # Conflicts: # google-auth-library-java/oauth2_http/javatests/com/google/auth/oauth2/ComputeEngineCredentialsTest.java # google-auth-library-java/oauth2_http/javatests/com/google/auth/oauth2/MockMetadataServerTransport.java
…etry logic. Nit fixes. # Conflicts: # google-auth-library-java/oauth2_http/javatests/com/google/auth/oauth2/ComputeEngineCredentialsTest.java
# Conflicts: # google-auth-library-java/oauth2_http/java/com/google/auth/oauth2/ComputeEngineCredentials.java
| if (san.size() >= 2 | ||
| && san.get(0) instanceof Integer | ||
| && (Integer) san.get(0) == SAN_URI_TYPE) { | ||
| Object value = san.get(1); | ||
| if (value instanceof String) { |
There was a problem hiding this comment.
nit: Is it possible to invert the if checks to have guards here to reduce the nesting for this method?
if san.size < 1 || !san.get(0) instanceof Integer || san.get(0) != SAN_URI_TYPE, same with value !isntanceof String?
There was a problem hiding this comment.
Done — inverted the conditions into continue guard clauses in shouldRequestBoundToken to flatten the loop nesting.
| static boolean isCachedInfoValid( | ||
| final CachedAgentIdentityInfo cached, | ||
| final String certConfigPath, | ||
| final String wellKnownDir) { |
There was a problem hiding this comment.
nit: the wellKnownDir path shouldn't change from different invocations? Can we just refernece the wellknowndir constant instead of using it as a param?
There was a problem hiding this comment.
Done — removed the wellKnownDir parameter from isCachedInfoValid and referenced AgentIdentityUtils.getWellKnownDir() directly inside the method.
| * @throws IOException If an I/O error occurs while reading the files, or if the key-pair | ||
| * verification fails after retries. | ||
| */ | ||
| static CertInfo getAgentIdentityCertInfo() throws IOException { |
There was a problem hiding this comment.
This logic is a bit hard for me to follow and I think we can try to simplify it.
- getCachedAgentIdentityInfo() looks like it can return null and we should guard against that
- The Fast-Path checks sense, I think we should clarify why we can hard-code certPresent = true for this case.
- From what I see, if ResolvedCertAndKeyPaths == null, then
configExistsalways is false. If that's the case, thenshouldEnableMtlsshould always return false. I think we guard against that, then shouldEnableMtls doesn't need the configExists param as we can check it here in getAgentIdentityCertInfo()
There was a problem hiding this comment.
Updated getAgentIdentityCertInfo() to address these points:
- Added an explicit
initialCached != nullguard before callingisCachedInfoValid. - Added an inline comment explaining that
isCachedInfoValidverifies the cached certificate file still exists on disk with matching metadata, which guaranteescertsPresent = truein the fast-path. - Removed the redundant
paths != nullchecks (sinceresolveCertAndKeyPathsnever returnsnull). Note thatshouldEnableMtls(certsPresent, configExists)still needsconfigExistsbecauseconfigExists == falsedoes not always evaluate tofalse: whencertsPresent == true && configExists == false(certificates discovered in the well-known directory without a config file),shouldEnableMtls(true, false)returnstruewhenGOOGLE_API_USE_CLIENT_CERTIFICATE="true"(Case 1), whereas it returnsfalsewhenGOOGLE_API_USE_CLIENT_CERTIFICATEis unset (Case 3). Added Javadoc onshouldEnableMtlsto clarify this.
… and well-known path resolution
| final CachedAgentIdentityInfo initialCached) throws IOException { | ||
| String useClientCert = getUseClientCertificateEnv(); | ||
| boolean explicitMtls = isMtlsExplicitlyEnabled(); | ||
| if (!explicitMtls && !Files.exists(Paths.get(wellKnownDir))) { |
There was a problem hiding this comment.
getWellKnownCertificatePathWithRetry only runs when GOOGLE_API_CERTIFICATE_CONFIG is unset, where configExists is always false. Since shouldEnableMtls now returns false whenever GOOGLE_API_USE_CLIENT_CERTIFICATE is not "true" and configExists is false, getWellKnownCertificatePathWithRetry can return new ResolvedCertAndKeyPaths(null, null, false) immediately when !explicitMtls instead of probing wellKnownDir and its certificate files on every token refresh.
There was a problem hiding this comment.
Done — updated getWellKnownCertificatePathWithRetry to return new ResolvedCertAndKeyPaths(null, null, false) immediately when !isMtlsExplicitlyEnabled().
| } | ||
| } | ||
| } | ||
| initialStartupCompleted = true; |
There was a problem hiding this comment.
Setting initialStartupCompleted = true here before the debug log at line 734 means initialStartupCompleted always prints true in that log message, even on the initial startup call. Also, when isMtlsExplicitlyDisabled() is true, we should skip logging the missing well-known certificate fallback message just like getPathsFromConfigWithRetry does.
There was a problem hiding this comment.
With the early return when !isMtlsExplicitlyEnabled() at the top of getWellKnownCertificatePathWithRetry (from the comment above), the remainder of this method only runs when explicit mTLS is enabled, so the !explicitMtls fallback log at the end was unreachable and has been removed.
| for (int cycle = 0; cycle < maxCycles; cycle++) { | ||
| try { | ||
| if (AgentIdentityCacheUtils.checkExistsOrAccessDenied(Paths.get(certConfigPath))) { | ||
| ResolvedCertAndKeyPaths paths = extractPathsFromConfig(certConfigPath); |
There was a problem hiding this comment.
small nit (possible I may be missing some edge case), so if we can't or shouldn't do this then please ignore.
I think all paths that lead to a valid ResolvedCertAndKeyPaths should just have the certpath and keypaths already validated so we don't need to validate this in the calling method.
L567 and L570 don't need explicit checkExistsOrAccessDenied calls here as it'll know that it's either valid or null since it got back a ResolvedCertAndKeyPaths object.
There are also a lot of paths.getCertPath() calls from a ResolvedCertAndKeyPaths so perhaps certPath and keyPath fields should just be a Path object.
There was a problem hiding this comment.
- On validating file existence inside
extractPathsFromConfig: The edge case here is whencertificate_config.jsonitself resides outsidewellKnownDir(shouldPoll == falseinitially) while itscert_path/key_pathpoint insidewellKnownDirand haven't been delivered yet on cycle 0.getPathsFromConfigWithRetryneeds the parsedcertPathandkeyPathfromextractPathsFromConfigbefore checking file existence so it can inspectisPathInWellKnownDir(...)and enable startup polling (shouldPoll = true; maxCycles = TOTAL_POLL_CYCLES). - On using
Pathvs.StringinResolvedCertAndKeyPaths: Both the inputs (JSON config strings andfallbackCached.certMetadata.getPath()) and the downstream consumers (loadAndVerifyCredentials,FileMetadata,isPostResolutionCacheHit,readCertificateChain, andreadPrivateKey) take and storeString, so keepingStringavoidsString -> Path -> Stringround-trip conversions and null-guarded.toString()calls.
| && (isPathInWellKnownDir(paths.getCertPath()) | ||
| || isPathInWellKnownDir(paths.getKeyPath()))) { |
There was a problem hiding this comment.
maybe worth adding a comment in the code, I'm not sure why this needs to check if it exists in the well known dir here?
There was a problem hiding this comment.
Done — added an inline comment explaining that the config file itself may reside outside wellKnownDir while referencing cert_path or key_path inside wellKnownDir that are still being delivered at startup.
|
I think the code looks good to me. I will do a final pass through the tests tomorrow to see if there is any cases that I may have missed. |
| // Incomplete workload config (missing cert_path or key_path) will never become ready; | ||
| // fail fast without polling. | ||
| lastParseException = e; | ||
| break; |
There was a problem hiding this comment.
qq: for this case, it mentions fail-fast but this ends up going to the fallback value in the cache. Is this intended or should we bubble this up to the user/ calling method?
There was a problem hiding this comment.
Good catch on the comment wording — "fail fast without polling" here meant skipping the 30-second startup polling loop (via break;), not bypassing the steady-state cache fallback:
- On initial startup (
fallbackCached == null), breaking out of the loop immediately throws theIOException(withIncompleteWorkloadConfigExceptionas the cause) without waiting 30s. - In steady state (
fallbackCached != null, meaning a valid config and verified certificate were already cached earlier), breaking out of the loop falls back to the previously validated cached paths if a later config rewrite is incomplete, matching the behavior when the config file is deleted or unreadable after caching.
Updated the inline comment in 9afe807bca3 to make this distinction explicit.
| * @throws IOException If an I/O error occurs while reading the files, or if the key-pair | ||
| * verification fails after retries. | ||
| */ | ||
| static CertInfo getAgentIdentityCertInfo() throws IOException { |
There was a problem hiding this comment.
additional thoughts:
Tracing this from ComputeEngineCredentials, I see this is called from refreshAcessToken() -> getBoundToken() which already handles the request coalescing to mitigate the thundering herd possibility. Perhaps it maybe worth a small callout in the javadocs the mention this for future maintainers, so they know why we don't have any need for coalescing here or concerns about syncing across multiple requests
There was a problem hiding this comment.
Good call — added a note to getAgentIdentityCertInfo()'s Javadoc in 9afe807bca3 calling out that callers like ComputeEngineCredentials.refreshAccessToken() invoke this via getBoundTokenPayload() under OAuth2Credentials's token refresh coalescing.
| * Utility class for in-memory caching and filesystem metadata validation of Agent Identity | ||
| * certificates and configuration files. | ||
| */ | ||
| final class AgentIdentityCacheUtils { |
There was a problem hiding this comment.
For the util classes, can you add some quick @NullMarked and @nullable annotations. Feel free to add them in follow up PRs (I don't think annotations are blocking for this)
There was a problem hiding this comment.
Done in 9afe807bca3 — added @NullMarked and @Nullable annotations across AgentIdentityCacheUtils, AgentIdentityCertificateValidationUtils, and AgentIdentityUtils.
lqiu96
left a comment
lqiu96
left a comment
There was a problem hiding this comment.
LGTM, thanks for all the quick iterations! I took one final pass throughout the PR and I think the logic makes sense. I don't have any additional concerns with the code and I think we can make follow fixes after additional testing/ feedback.
acc7c9d
into
googleapis:agentic-identities-bound-token
5 participants
This PR introduces a feature which enables the auth library to acquire bound access-tokens and bound id-tokens in Agentic Environments.
We detect certs in default paths and check if they match the SPIFFE format for agents.
If 1. is a yes then we call the MDS endpoint in a POST request with the certificate in the body.
Note this PR was based on #13169
Manual Testing & End-to-End Verification
We verified this feature end-to-end across both a Live Cloud Run Agent Identity environment (testing against the live Google Metadata Server, Security Token Service, and Vertex AI with the Java Agent Development Kit (
com.google.adk:google-adk:1.9.0)) and a 10-Scenario Local Mock MDS Simulation Harness (testing exact HTTP request payloads, true cryptographic certificate/key rotation, combined bundle private-key stripping, well-known directory discovery, non-agent SPIFFE fallback, environment variable precedence, non-atomic rotation retries, and asynchronous container startup polling).1. Live Cloud Run Agent Identity Verification (
<PROJECT_ID>,us-central1)We deployed a containerized Java test application built against this branch (
google-auth-library-oauth2-http:1.50.0-SNAPSHOT@27776f6d62a) + Java ADK (com.google.adk:google-adk:1.9.0) to Cloud Run with Agent Identity enabled (--functional-type=agent --identity-type=agent-identity).Execution A: Default Bound Token Acquisition (
GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKENunset / defaulttrue)agent-bound-token-java-test-cjrgp(latest run with Step 4D) &agent-bound-token-java-test-dd9gm/var/run/secrets/workload-spiffe-credentials/credentials.jsonspiffe://agents.global.org-<ORG_NUMBER>.system.id.goog/resources/run/projects/<PROJECT_NUMBER>/locations/us-central1/jobs/agent-bound-token-java-testx5t#S256):5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8ComputeEngineCredentials):POSTto Metadata Server (ya29.d.c0AZ4bNp...).IdTokenCredentials) & Cryptographic Binding Verification (RFC 8705 § 3.1):https://example-target-service.run.app.textPayload) and verified live Google STS embedded thecnf(Confirmation) claim containing the SHA-256 thumbprint (x5t#S256) of the workload's leaf X.509 certificate:[PASS] JWT x5t#S256 thumbprint EXACTLY matches local leaf certificate SHA-256!).com.google.adk:google-adk:1.9.0) +google-genai& mTLS Proof-of-Possession Verification (Steps 4A, 4B, 4C, 4D):HttpClientFactoryNon-mTLS Transport): Inspected ADK's sharedOkHttpClient(sun.security.ssl.SSLSocketFactoryImpl, no client cert). Calling Google APIs over this non-mTLS channel with the bound token is rejected at the auth layer withHTTP 401 UNAUTHENTICATED.LlmAgent+GeminiTurn on Vertex AI): InvokingInMemoryRunner.runAsync(...)withGemini(gemini-2.5-flash) fails on turn 1 withcom.google.genai.errors.ClientException: 401 . Request had invalid authentication credentials, confirming the known limitation wheregoogle-genaisends bound tokens over a non-mTLS channel.OkHttpClient(configured with/var/run/secrets/workload-spiffe-credentials/certificates.pem+private_key.pem,x5t#S256 = 5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8) tohttps://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>passes authentication (HTTP 403 PERMISSION_DENIEDIAM check instead of401 UNAUTHENTICATED), proving Google API Frontend verified the token binding against the TLS client certificate handshake.https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>over an mTLSOkHttpClientconfigured with a different X.509 client certificate (x5t#S256 = -57ZjYVm89oczfdO02Hf3Sz-FaYQIVH1DD3c9zaBIKA) is rejected at the auth layer withHTTP 401 UNAUTHENTICATED, confirming that Google API Frontend enforces cryptographic thumbprint matching (cnf.x5t#S256 == SHA256(TLS client cert)).Execution B: Opt-Out Unbound Token Acquisition (
GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false)agent-bound-token-java-test-t7b98--set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true".ComputeEngineCredentialsandIdTokenCredentialsfell back to standardHTTP GETrequests against MDS and issued standard unbound tokens (decoded JWT payload confirmed absence of thecnfclaim).Step 4B): WithGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false, the unbound token works overgoogle-genai's non-mTLS channel and the live ADKLlmAgent+Geminiturn SUCCEEDS:[ADK Event] author=bound-token-verify-agent, content=Hello! Yes, as an agent designed to test bound token behavior with the Java ADK, I am expected to receive and process bound tokens.2. Local End-to-End Simulation & Wire Verification (10 Scenarios)
To verify internal wire-level, discovery, rotation, environment-variable, and error-handling behavior between the client and MDS, we executed our local simulation suite (
LocalSimulationRunner.java) spinning up a local Mock MDS (HttpServer) across 10 end-to-end scenarios:ComputeEngineCredentials+GOOGLE_API_CERTIFICATE_CONFIG): VerifiedHTTP POSTto/computeMetadata/v1/instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/cloud-platform, verified JSON body{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}(serialized as a single PEM string), and verified no extra fields are included in the JSON payload.IdTokenCredentials+GOOGLE_API_CERTIFICATE_CONFIG): VerifiedHTTP POSTto/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://target.run.app(withaudiencepassed as a URL query parameter) and JSON body{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}.Gen-1Gen-2): Generated a new X.509 SPIFFE certificate and matching 2048-bit RSA key pair on disk, updated filemtime, and verifiedAgentIdentityUtils.getAgentIdentityCertInfo()invalidated its cache, verified the new key pair, and transmitted the rotatedGen-2certificate chain (!req3.body.equals(req1.body)).GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false): Verified settingGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=falseswitches requests back toHTTP GETwith an empty request body.credentialbundle.pem) + Private Key Stripping: UnsetGOOGLE_API_CERTIFICATE_CONFIG, wrote a combinedcredentialbundle.pemcontaining both-----BEGIN CERTIFICATE-----and-----BEGIN PRIVATE KEY-----in the well-known directory, and verified thatAgentIdentityUtilsresolvedcertPath == keyPath, verified the key pair, and stripped thePRIVATE KEYblock from the transmittedPOSTpayload.certificates.pem+private_key.pem): UnsetGOOGLE_API_CERTIFICATE_CONFIG, removedcredentialbundle.pem, placed separatecertificates.pemandprivate_key.pemin the well-known directory, and verified boundHTTP POSTacquisition.shouldRequestBoundToken == false): Configured a valid X.509 certificate with a standard GKE Workload Identity SAN (spiffe://my-standard-gke-project.svc.id.goog/ns/default/sa/my-ksa); verifiedAgentIdentityUtilsdid not throw, cachedshouldRequestBoundToken = false, and fell back to standardHTTP GET.GOOGLE_API_USE_CLIENT_CERTIFICATEMatrix:GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false(withGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKENunset)HTTP GET.GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=trueoverrides legacyGOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=falseHTTP POST.GOOGLE_API_USE_CLIENT_CERTIFICATE=falsewith valid agent certs on diskHTTP GET) without startup polling.GOOGLE_API_USE_CLIENT_CERTIFICATE=truewith missing cert filesIOExceptionafter retries instead of silently downgrading to an unbound token.CERT_KEY_MATCH_RETRIES): Wrote a newGen-3non-prod SPIFFE certificate (spiffe://agents-nonprod.global.org-54321.system.id.goog/...) first while delaying the matchingprivate_key.pemupdate on a background thread; verifiedloadAndVerifyCredentials()retried cleanly viaCERT_KEY_MATCH_RETRIESand transmittedGen-3.TOTAL_POLL_CYCLES): Started with an empty well-known directory on initial startup (GOOGLE_API_USE_CLIENT_CERTIFICATE=true), deliveredcertificates.pem+private_key.pemasynchronously from a background thread after ~180ms, and verified the initialrefreshAccessToken()polled until the files arrived and succeeded with a boundHTTP POST.3. Reproducible Test Artifacts & Execution Logs (
gpaste- Internal Corp Access Only)cjrgp& Opt-Outt7b98) + Java ADK + Local 10-Scenario SimulationCloudRunAgentVerifyApp.javacnf.x5t#S256match, and ADK Steps 4A/4B/4C/4D)LocalSimulationRunner.javaHttpServer) testing all 10 scenariosDockerfilegoogle-genaiand overlaying our local PR JARdeploy_cloud_run_job.sh--identity-type=agent-identity, and executebuild_and_run_local.shcjrgp)gcloud logging readfor default Bound Token execution (with inline decoded JWT & ADK Steps 4A/4B/4C/4D)t7b98)gcloud logging readforGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=falseexecution (showing live ADK Gemini response)Quick Reproduction Steps
To replicate the live Cloud Run test in any GCP project with Agent Identity enabled: