feat(voice): auto-disable realtime server-side turn detection#6495
Open
longcw wants to merge 6 commits into
Open
feat(voice): auto-disable realtime server-side turn detection#6495longcw wants to merge 6 commits into
longcw wants to merge 6 commits into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 21, 2026 09:27
9c2cc63 to
9caf4ba
Compare
longcw
force-pushed
the
longc/realtime-barge-in
branch
from
July 22, 2026 01:49
a557e26 to
2926544
Compare
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 22, 2026 01:49
f9c7088 to
b525a7a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 22, 2026 02:57
48359d0 to
9c988f3
Compare
longcw
force-pushed
the
longc/realtime-barge-in
branch
from
July 23, 2026 06:01
82a241c to
2b70881
Compare
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 23, 2026 06:01
9c988f3 to
eca4e01
Compare
| # model to auto-generate a tool reply (auto_tool_reply_generation=True). | ||
| self._pending_auto_tool_reply_fut: asyncio.Future[None] | None = None | ||
|
|
||
| def _resolve_rt_turn_detection_enabled(self) -> bool: |
Member
There was a problem hiding this comment.
should we handle update_options with turn_detection for realtime too?
Contributor
Author
There was a problem hiding this comment.
toggle turn detection at runtime will need a restart of rt_session if the server-side turn detection is disabled. I added a warning for now.
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 24, 2026 03:03
eca4e01 to
9e9feca
Compare
Disable a realtime model's server-side turn detection for the session when the user configured client-side turn-taking (a TurnDetector, "vad", "manual", or an explicit interruption mode) and the model supports it, so client-side EoT/barge-in works without also passing turn_detection=None to the RealtimeModel. - add RealtimeCapabilities.can_disable_turn_detection; OpenAI sets it from the ctor turn_detection given-ness - add session(turn_detection_disabled=...), applied on the initial session.update via the per-session opts copy so the model is never mutated (stays reusable) - AgentActivity resolves the effective state once (_rt_turn_detection_enabled) and routes every server-side turn-detection check through it Other realtime plugins accept-and-ignore the flag (can_disable_turn_detection stays False): xAI (client turn-taking not stable in testing) and Gemini/others (no commit_audio/clear_audio; turns are activity-signal based).
RealtimeModel.update_options force-synced turn_detection=self._opts.turn_detection to every session unconditionally, so changing an unrelated field (e.g. voice) re-enabled server VAD on a session that opted out of server-side turn detection. Only propagate turn_detection when the caller set it; an explicit value still propagates so callers can re-enable it.
…tection On an agent handoff the realtime session is reused as-is. If the new agent resolves server-side turn detection differently (it can override turn_detection, interruption, or vad), the reused session kept the previous agent's mode, causing duplicate or missing replies. Refuse reuse on a mismatch so a fresh session is created with the correct setting.
_resolve_interruption_detection and the on_end_of_turn backchannel-drop guard still read the model's static capabilities.turn_detection instead of the session-resolved _rt_turn_detection_enabled. When a realtime model with built-in turn detection has it auto-disabled for the session (client-side turn-taking configured), can_gatekeep stayed False so the adaptive detector was never created, and the backchannel-drop path was skipped — silently disabling the client-side barge-in this stack is meant to enable.
with_azure applied the Azure default turn detection before constructing the model, so can_disable_turn_detection was always False for Azure even when the caller passed nothing. Capture the caller's intent before defaulting and set the capability accordingly, so the framework can still auto-disable server-side turn detection.
A realtime model's server-side turn detection is resolved once at session start and isn't re-synced at runtime yet. Warn when an explicit update_options(turn_detection=...) would flip that resolved state, so the mismatch isn't silent. Reverting to None (automatic re-selection) is left unwarned. Also move the auto-disable info log to the resolver's call site so the resolver stays a pure query, and mark the session turn detection explicit when set via update_options.
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 24, 2026 03:23
9e9feca to
24154ca
Compare
| self._rt_session.update_options(tool_choice=self._tool_choice) | ||
|
|
||
| if utils.is_given(turn_detection): | ||
| # a realtime model's server-side turn detection is resolved once at session start; |
Member
There was a problem hiding this comment.
we should have a similar warning for RealtimeModel.update_options(turn_detection=...) too
Member
|
Another thing I noticed in testing: if the interruption is detected by the barge-in model, it leads to the false interruption path, and for realtime model, transcript isn't always available immediately for that interruption to take effect. Should we consider disable false interruption when no STT is configured with realtime model? |
chenghao-mou
approved these changes
Jul 27, 2026
chenghao-mou
left a comment
chenghao-mou
left a comment
Member
There was a problem hiding this comment.
LGTM overall. Tested locally with OAI Realtime and VAD or turn detector. Two nits and two issues from previous comments.
| """Whether the model can produce audio output directly""" | ||
| manual_function_calls: bool | ||
| """Whether function call items already in the chat context can be resumed""" | ||
| can_disable_turn_detection: bool = False |
Member
There was a problem hiding this comment.
nit: maybe optional_turn_detection so it matches other adj_noun patterns
|
|
||
| @abstractmethod | ||
| def session(self) -> RealtimeSession: ... | ||
| def session(self, *, turn_detection_disabled: bool = False) -> RealtimeSession: |
Member
There was a problem hiding this comment.
nit: should we rename it enable_turn_detection: bool = False instead?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-disables a realtime model's server-side turn detection for the session when the user configured client-side turn-taking (a TurnDetector,
vad,manual, or an explicit interruption mode) and the model supports it, so client-side EoT/barge-in works without also passingturn_detection=Noneto the RealtimeModel.Adds
RealtimeCapabilities.can_disable_turn_detection(OpenAI sets it from the ctorturn_detectiongiven-ness) and asession(turn_detection_disabled=...)seam applied on the initialsession.updatevia the per-session opts copy, so the model is never mutated and stays reusable.AgentActivityresolves the effective state once (_rt_turn_detection_enabled) and routes every server-side turn-detection check through it.Other realtime plugins accept-and-ignore the flag (
can_disable_turn_detectionstays False): xAI (client turn-taking not stable in testing) and Gemini/others (nocommit_audio/clear_audio; turns are activity-signal based).Stacked on
longc/realtime-barge-in; this PR targets that branch, not main.