Repository navigation
Description
🔴 Required Information
Describe the Bug:
AdkWebServer.get_fast_api_app() (src/google/adk/cli/fast_api.py) installs a @app.middleware("http") hook (context_propagation) that calls get_propagated_context(request) (src/google/adk/telemetry/_agent_engine.py:59-84) and attaches the returned OpenTelemetry context for the duration of the request.
#4767 and #4972 were closed on the assumption that this middleware extracts inbound W3C traceparent headers so adk api_server / adk web continues the caller's distributed trace. However, get_propagated_context() only runs TraceContextTextMapPropagator().extract(...) on the proprietary Google-Agent-Engine-Traceparent header:
# src/google/adk/telemetry/_agent_engine.py:59-84
def get_propagated_context(request: fastapi.Request) -> context.Context:
"""Propagates context from the request headers."""
ctx = context.get_current()
if _GOOGLE_TRACEPARENT_HEADER in request.headers:
original_traceparent = request.headers[_GOOGLE_TRACEPARENT_HEADER]
ctx = baggage.set_baggage(
_GOOGLE_TRACEPARENT_BAGGAGE_KEY,
original_traceparent,
context=ctx,
)
if _GOOGLE_AE_TRACEPARENT_HEADER in request.headers:
ae_traceparent = request.headers[_GOOGLE_AE_TRACEPARENT_HEADER]
extracted_ctx = tracecontext.TraceContextTextMapPropagator().extract(
carrier={"traceparent": ae_traceparent}, context=ctx
)
if extracted_ctx is not ctx:
ctx = baggage.set_baggage(
_TRACEPARENT_BAGGAGE_KEY, ae_traceparent, context=extracted_ctx
)
return ctxAs a result, when any non-Agent-Engine caller (an API gateway, load balancer, or upstream microservice) sends a standard W3C traceparent (and/or baggage) header to get_fast_api_app():
traceparentis only stashed inbaggageunder"google_traceparent"(whichTopSpanProcessorreads to stampsupportID), never extracted as the parentSpanContext. Every inbound/runor/run_sserequest starts a brand-new root trace ID instead of parenting under the caller's span.- Inbound W3C
baggageheaders (e.g., tenant, experiment, or evaluation metadata) are not extracted at all.
Steps to Reproduce:
- Create a FastAPI app via
get_fast_api_app(...)with an activeTracerProviderandInMemorySpanExporter. - Send a POST request to
/runwith a valid W3C header:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
andbaggage: tenant_id=acme. - Inspect the exported
invocation/invoke_agentspans.
Expected Behavior:
- When
Google-Agent-Engine-Traceparentis present, it continues to take precedence as the extracted parent span context (preserving Vertex Agent EnginesupportIDbehavior). - When
Google-Agent-Engine-Traceparentis absent and a standard W3Ctraceparentheader is present,get_propagated_context()extractstraceparent(andtracestate) as the parentSpanContextso ADK spans join the caller's trace (trace_id == 0x4bf92f3577b34da6a3ce929d0e0e4736). - Inbound W3C
baggageheaders are extracted viaW3CBaggagePropagatorwithout clobbering the internalgoogle_traceparent/traceparentbaggage keys.
Observed Behavior:
The exported ADK spans have a newly generated trace_id unrelated to 4bf92f3577b34da6a3ce929d0e0e4736, and inbound W3C baggage entries are dropped.
Environment Details:
- ADK Library Version:
main(2.13.0+) - Desktop OS: macOS / Linux
- Python Version: 3.11+
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A (telemetry / FastAPI middleware layer)
🟡 Optional Information
Regression:
#4767 reported this gap and #4972 was closed assuming commit ffa057c1 resolved it via get_propagated_context, but get_propagated_context only extracts Google-Agent-Engine-Traceparent.
Minimal Reproduction Code:
from fastapi import FastAPI, Request
from fastapi.testclient import TestClient
from opentelemetry import baggage, trace
from google.adk.telemetry._agent_engine import get_propagated_context
app = FastAPI()
@app.get("/test")
def endpoint(request: Request):
ctx = get_propagated_context(request)
span_ctx = trace.get_current_span(ctx).get_span_context()
return {
"trace_id": f"{span_ctx.trace_id:032x}",
"tenant": baggage.get_baggage("tenant_id", context=ctx),
}
client = TestClient(app)
resp = client.get(
"/test",
headers={
"traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
"baggage": "tenant_id=acme",
},
)
# Returns trace_id="00000000000000000000000000000000" and tenant=None
print(resp.json())How often has this issue occurred?:
- Always (100%)
Activity
fix(tracing): extract standard W3C traceparent and baggage headers on…