Skip to content

fix(cli): stop unbounded debug trace retention on the api server - #6916

Open
a2105z wants to merge 1 commit into
google:mainfrom
a2105z:fix/api-server-unbounded-trace-retention
Open

fix(cli): stop unbounded debug trace retention on the api server#6916
a2105z wants to merge 1 commit into
google:mainfrom
a2105z:fix/api-server-unbounded-trace-retention

Conversation

@a2105z

@a2105z a2105z commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
get_fast_api_app(web=False) unconditionally installed ApiServerSpanExporter and InMemoryExporter on the process tracer. Those buffers store full llm_request / llm_response attributes and every span, with no eviction. The debug endpoints that read them (/dev/apps/.../debug/trace) are only mounted with web=True, so a production Cloud Run / GKE process pays unbounded memory for buffers nobody reads. Measured ~200 KB/LLM turn and OOM about every 6 days.

Solution:

  • Do not register the internal exporters on ApiServer (web=False). DevServer (web=True / adk web) still registers them.
  • FIFO-cap the event dict (512), span deque (4096), and session index (512) so a long-lived adk web session cannot retain every span either.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.
uv run python -m pytest tests/unittests/cli/test_fast_api.py::test_api_server_span_exporter_records_only_llm_and_tool_spans tests/unittests/cli/test_fast_api.py::test_api_server_span_exporter_skips_span_without_event_id tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_returns_only_spans_of_requested_session tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_falls_back_to_conversation_id tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_clear_drops_spans_but_keeps_session_index tests/unittests/cli/test_fast_api.py::test_api_server_span_exporter_evicts_oldest_events tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_evicts_oldest_spans tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_evicts_oldest_sessions tests/unittests/cli/test_fast_api.py::test_debug_trace_exporters_only_registered_for_dev_server tests/unittests/cli/test_fast_api.py::test_dev_only_endpoints_absent_when_web_disabled tests/unittests/cli/test_fast_api.py::test_debug_trace -q
11 passed

Manual End-to-End (E2E) Tests:

from google.adk.cli.fast_api import get_fast_api_app
from opentelemetry import trace

app = get_fast_api_app(agents_dir="agents", web=False)
tp = trace.get_tracer_provider()
procs = tp._active_span_processor._span_processors
print([type(p.span_exporter).__name__ for p in procs])
# no ApiServerSpanExporter / InMemoryExporter

adk web still serves /dev/apps/{app}/debug/trace/{event_id} (covered by test_debug_trace).

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

get_fast_api_app(web=False) still installed ApiServerSpanExporter and
InMemoryExporter even though the debug endpoints that read them are only
mounted with web=True. Those buffers retain every span (including full
llm_request / llm_response) for the life of the process and OOM
long-lived Cloud Run / GKE instances.

Register the exporters only on DevServer, and FIFO-cap the event dict,
span deque, and session index so adk web cannot grow without bound either.

Closes google#6915
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant