fix(google): preserve non-blocking realtime generation - #6741
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| while not gen._done: | ||
| elapsed = time.monotonic() - started_at | ||
| if elapsed >= NON_BLOCKING_TOOL_DRAIN_TIMEOUT_SECONDS: | ||
| break | ||
|
|
||
| await asyncio.sleep( | ||
| min( | ||
| NON_BLOCKING_TOOL_DRAIN_QUIESCENCE_SECONDS, | ||
| NON_BLOCKING_TOOL_DRAIN_TIMEOUT_SECONDS - elapsed, | ||
| ) | ||
| ) | ||
|
|
||
| current_activity_at = gen._last_output_activity_at | ||
| if current_activity_at == last_activity_at: | ||
| break | ||
| last_activity_at = current_activity_at | ||
|
|
||
| self._mark_generation_done(gen) |
There was a problem hiding this comment.
🔴 Agent speech is cut off five seconds after a non-blocking tool call
The spoken reply is force-ended (_mark_generation_done(gen) at livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py:1526) five seconds after a non-blocking tool call even while the model is still actively talking, so the reply is chopped off mid-sentence.
Impact: Users hear the agent's answer truncated and then abruptly restarted whenever it speaks for more than five seconds following a non-blocking tool call.
Hard drain deadline ignores ongoing output activity
_finalize_non_blocking_generation_after_output_drain computes elapsed from started_at (the moment the tool call arrived) and breaks out of the loop as soon as elapsed >= NON_BLOCKING_TOOL_DRAIN_TIMEOUT_SECONDS (realtime_api.py:1510-1512), regardless of whether gen._last_output_activity_at is still being refreshed by incoming audio/transcription (realtime_api.py:1285-1288). The constant is documented as a fallback "for models that wait for the tool response instead of completing the turn" (realtime_api.py:50-54), i.e. an idle timeout, but it is implemented as an absolute cap on the whole generation.
When the model keeps streaming audio past 5s, _mark_generation_done closes audio_ch/text_ch via _close_output_streams and closes message_ch/function_ch (realtime_api.py:1400-1408), ending the audio segment mid-speech. Subsequent audio then hits the _current_generation._done branch in the receive loop (realtime_api.py:1106), which starts a brand-new generation; _start_new_generation emits input_speech_started (realtime_api.py:1259), interrupting whatever playout was left.
Making the timeout relative to the last observed output activity (or only arming it while the stream is quiet) would preserve the intended fallback without truncating active speech.
Prompt for agents
In _finalize_non_blocking_generation_after_output_drain (livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py), the NON_BLOCKING_TOOL_DRAIN_TIMEOUT_SECONDS budget is measured from the moment the tool call was received (started_at) and is checked unconditionally, so the generation is finalized 5 seconds after the tool call even when the model is still streaming audio/transcription (which refreshes gen._last_output_activity_at in _handle_server_content). This truncates the agent's speech and causes a new generation (and an input_speech_started interrupt) to be started for the remaining audio. The constant is intended as a safety net for models that never emit generation_complete/turn_complete while waiting for a tool response, i.e. an idle bound. Rework the loop so the timeout is measured against the last observed output activity (or only applies while no output has been seen), letting an actively streaming turn continue until generation_complete/turn_complete finalizes it.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
NON_BLOCKINGtool callsgeneration_complete/turn_completeevents close the output streams and generationThis replaces #6526 with a clean change based on the current LiveKit Agents 1.6.8 codebase.
Tests
uv run pytest tests/test_plugin_google_realtime.py -q(5 passed)uv run ruff check livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py tests/test_plugin_google_realtime.pyuv run ruff format --check livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py tests/test_plugin_google_realtime.py