Repository navigation
LiteLlm with SSE streaming records the answer twice in gen_ai.output.messages聽#7630
Description
馃敶 Required Information
Describe the Bug:
with StreamingMode.SSE and a LiteLlm model the generate_content span ends up with every streamed chunk in gen_ai.output.messages and then the full answer again at the end. so the answer shows up twice and all the chunks have an empty finish_reason. without streaming its one message like expected. only visible with message content capture on (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT)
looks like the text chunks get yielded as partial responses with no finish reason (lite_llm.py#L4475-L4485) and only the final aggregated response gets one (#L4556-L4558). record_llm_response keeps recording until it sees a finish reason (_instrumentation.py#L568-L595) so every chunk plus the aggregate goes in. since its in the LiteLlm wrapper i'd expect real providers to do the same but i only tested with litellm's mock_response
Steps to Reproduce:
pip install google-adk litellm- run this (no api key needed, it uses litellm's
mock_response)
import asyncio
import json
import os
os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "SPAN_ONLY"
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
spans = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(spans))
trace.set_tracer_provider(provider)
from google.adk.agents import Agent
from google.adk.agents.run_config import RunConfig, StreamingMode
from google.adk.models.lite_llm import LiteLlm
from google.adk.runners import InMemoryRunner
from google.genai import types
async def run(mode):
spans.clear()
model = LiteLlm(model="openai/gpt-4o", mock_response="Oslo is 4C and cloudy today.", api_key="unused")
runner = InMemoryRunner(agent=Agent(name="a", model=model), app_name="p")
session = await runner.session_service.create_session(app_name="p", user_id="u")
msg = types.Content(role="user", parts=[types.Part(text="weather?")])
async for _ in runner.run_async(user_id="u", session_id=session.id, new_message=msg,
run_config=RunConfig(streaming_mode=mode)):
pass
await runner.close()
span = next(s for s in spans.get_finished_spans() if s.name.startswith("generate_content"))
messages = json.loads(span.attributes["gen_ai.output.messages"])
print(f"{mode.name}: {len(messages)} output message(s)")
for m in messages:
print(" ", repr(m.get("finish_reason")), "".join(p.get("content", "") for p in m["parts"]))
asyncio.run(run(StreamingMode.NONE))
asyncio.run(run(StreamingMode.SSE))Expected Behavior:
one output message with the full answer, same as without streaming
Observed Behavior:
NONE: 1 output message(s)
'stop' Oslo is 4C and cloudy today.
SSE: 11 output message(s)
'' Osl
'' o i
'' s 4
'' C a
'' nd
'' clo
'' udy
'' to
'' day
'' .
'stop' Oslo is 4C and cloudy today.
Environment Details:
- ADK Library Version (pip show google-adk): 2.12.0 (main at 77a74f0)
- Desktop OS: macOS
- Python Version (python -V): 3.12.15
Model Information:
- Are you using LiteLLM: Yes
- Which model is being used: openai/gpt-4o via litellm
mock_response
馃煛 Optional Information
Regression:
havent checked older versions
maybe skip the partial chunks when recording output messages and keep just the aggregate? happy to send a PR if that sounds right
Activity
GBX-Max1220 commentedon Oct 20, 2026
Thanks for the clear reproduction. I looked into the telemetry path and noticed that
set_operation_details_attributes_from_response()deliberately appends each response togen_ai.output.messages, while LiteLLM streaming also emits a final aggregated response.Before proposing a fix, could we clarify the intended contract here?
Should a streaming model call record one canonical completed output message, or preserve individual streamed chunks without duplicating the final aggregate?
I think the distinction matters for incomplete streams, finish reasons, and usage metadata.
I'm happy to help with focused offline regression tests if useful. Since the reporter has already offered to prepare a fix, I wouldn't open a competing PR.
ishan-1010 commentedon Oct 21, 2026
i think one completed message per call. thats what you already get with streaming off and the partial chunks have no finish reason anyway. if the stream gets cut off before the final response i'd record whatever text came in so far with no finish reason. would like a maintainer to confirm before i send the PR
GBX-Max1220 commentedon Oct 21, 2026
Thanks, that makes sense as a proposed contract. I agree we should wait for maintainer confirmation, especially on interrupted streams.
Once confirmed, I鈥檓 happy to help with offline regression tests covering completed streams without duplicate text, interrupted streams, and finish-reason and usage metadata. I鈥檒l leave the implementation PR with you.
surajksharma07 commentedon Oct 22, 2026
Reproduced this on 2.12.0 and main (facd7dd) @ishan-1010. 11 messages with SSE vs 1 without. LiteLlm鈥檚 chunks carry no finish_reason and only the final aggregate has one so record_llm_response keeps recording until the aggregate and adds the whole answer again. Gemini doesn鈥檛 hit this because its last chunk already has the finish_reason so the aggregate is never recorded.
Something that worked as a workaround: in InferenceScope.record_llm_response when a non-partial response with a finish_reason follows only partial chunks that have none pop gen_ai.output.messages from span.operation_details_attributes before recording it. LiteLlm SSE then gives one 'stop' message and Gemini-style streams keep their per-chunk messages which the existing telemetry tests expect.
Could you check if that holds up for you with a real provider too plus a tool-call stream and a stream cut off mid-way (those should keep their chunks)? Please test it thoroughly before raising the PR. @GBX-Max1220鈥檚 regression tests would fit well alongside it.
Will also ask a telemetry maintainer to confirm the contract since this gives LiteLlm one completed message while Gemini stays per-chunk.
ishan-1010 commentedon Oct 22, 2026
@surajksharma07 the pop holds up.
gen_ai.output.messageson main (facd7dd) vs with the change:openai/gpt-oss-20b, text answeropenai/gpt-oss-20b, tool calltool_callwitharguments: nulltool_call({"city": "Oslo"})arguments: null)same in the completion log with
EVENT_ONLY. a stream that raises before its aggregate keeps its chunks, and the existing telemetry tests incl. the functional goldens don't change (1023 passed).two things:
finish_reason="stop", so telemetry can't tell it was cut off. that's in lite_llm.py, not this change.separately, on groq the call after the tool fails because the history sends
reasoning_contentback (property 'reasoning_content' is unsupported). i'll open an issue for that one.GBX-Max1220 commentedon Oct 22, 2026
Thanks for reproducing this and spelling out the workaround. I鈥檇 be happy to coordinate on the regression cases, particularly completed responses, tool-call streams, and interrupted streams.
It would help to get the maintainer鈥檚 view on the intended contract first, since the proposed behavior differs between LiteLlm and Gemini. I haven鈥檛 validated the workaround against a real provider.
GBX-Max1220 commentedon Oct 22, 2026
Thanks, Ishan鈥攜our provider results are helpful. I put together a small offline regression-preparation pack against main
6c67240.Seven isolated checks pass using extracted source functions with lightweight test doubles. These aren鈥檛 native ADK tests or a reproduction of the network cutoff.
The pack includes cases for duplicate aggregates, partial tool arguments, and missing terminal responses. Your cutoff finding is especially useful: removing duplicate messages won鈥檛 recover completion information that was already lost upstream.
Happy to coordinate these cases with your fix once the telemetry contract is confirmed.
ADK_7630_Offline_Regression_Preparation_20261023.zip
surajksharma07 commentedon Oct 23, 2026
That鈥檚 a solid set of runs @ishan-1010. On the tool call: keeping only the finished call is fine. The partial chunks only carry partial_args which is why they show up as arguments: null so dropping them loses nothing. The cut-off one I can confirm too: when the stream ends without a finish chunk lite_llm.py falls back to last_finish_reason or "stop" (still the case on main, 2de5876) so telemetry has no way to tell it was cut off. Please open that one separately same as the groq reasoning_content one and keep this PR to the telemetry change.
Feel free to open the PR as a draft now so the telemetry contract can be reviewed on the actual diff. @GBX-Max1220 it would help to add your cases as native tests under tests/unittests/telemetry in that PR (completed stream, tool-call stream, a stream that raises before the aggregate, Gemini unchanged) instead of keeping them in the zip.
GBX-Max1220 commentedon Oct 23, 2026
Thanks, Suraj鈥攖hat gives me a clear scope. I鈥檓 happy to adapt the offline cases into native tests under
tests/unittests/telemetry, covering completed streams, tool-call streams, an exception before the aggregate, and unchanged Gemini behavior.@ishan-1010 please link the draft PR when it鈥檚 open so I can coordinate against your branch. I鈥檒l keep the silent-cutoff and Groq history issues separate.
ishan-1010 commentedon Oct 23, 2026
draft PR is #7694. the four cases you listed are already native tests in it (completed stream, tool-call stream, a stream that raises before the aggregate, Gemini unchanged). @GBX-Max1220 if you have cases beyond those, open a PR against my branch or paste them here. cut-off issue: #7693
GBX-Max1220 commentedon Oct 23, 2026
Thanks, Ishan! I see the four native cases in #7694. I鈥檒l review those against the offline cases and check for any additional gaps, rather than duplicate what you鈥檝e already covered. If I find something useful, I鈥檒l share a focused reproducer here or a test patch against your branch.