Skip to content

fix(voice): unblock SpeechHandle.wait_for_playout() immediately on interruption (#5359) - #7249

Closed
rrfunde wants to merge 9 commits into
livekit:mainfrom
rrfunde:fix/speech-handle-wait-for-playout-interruption
Closed

rrfunde wants to merge 9 commits into
livekit:mainfrom
rrfunde:fix/speech-handle-wait-for-playout-interruption

Conversation

@rrfunde

@rrfunde rrfunde commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • In SpeechHandle.wait_for_playout(), wait for either _done_fut or _interrupt_fut using asyncio.FIRST_COMPLETED.
  • Return immediately when self.interrupted or self._interrupt_fut.done() is true instead of remaining deadlocked until the 5.0s INTERRUPTION_TIMEOUT expires.
  • Added regression tests in tests/test_speech_handle_interruption.py.

Fixes #5359

@rrfunde
rrfunde requested a review from a team as a code owner September 12, 2026 11:19
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-14T01:50:49.724200Z 9e78b9b New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

devin-ai-integration[bot]

This comment was marked as resolved.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e818c727c9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +258 to +260
await asyncio.wait(
{self._done_fut, self._interrupt_fut},
return_when=asyncio.FIRST_COMPLETED,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the documented await-and-inspect contract

When an interruption wins this wait, await handle now returns while _done_fut is still pending, as the new test explicitly asserts. However, AgentSession.generate_reply() documents that awaiting the handle waits for the reply to finish and directs callers to inspect handle.exception() afterward; that method raises InvalidStateError until _done_fut completes. Consequently, callers following the public contract can fail whenever speech is interrupted, so either provide a separate completion wait for exception inspection or update the API contract and make post-await inspection safe.

Useful? React with 👍 / 👎.

@rrfunde

rrfunde commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback:

  • SpeechHandle.done() now returns True when interrupted (self._done_fut.done() or self.interrupted).
  • SpeechHandle.exception() checks self.done(), making post-await inspection safe when interrupted without raising InvalidStateError.
  • Registered done_callbacks are triggered upon interruption as well.
  • _wait_for_generation() also unblocks immediately on interruption.

devin-ai-integration[bot]

This comment was marked as resolved.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8843ff6bfc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +263 to +265
await asyncio.wait(
{self._done_fut, self._interrupt_fut},
return_when=asyncio.FIRST_COMPLETED,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Wait for chat finalization before starting a barge-in reply

When a user turn interrupts current pipeline/TTS speech, _user_turn_completed_impl relies on await current_speech.interrupt() (agent_activity.py:2776) before copying the agent chat context. This FIRST_COMPLETED wait now returns as soon as _interrupt_fut is set, but _tts_task_impl only derives and inserts the synchronized partial assistant transcript afterward (agent_activity.py:3270-3337). The next LLM is therefore launched from a snapshot missing text the user actually heard, while that interrupted item can be inserted after the newer turn; retain a true finalization wait for this internal transition path.

Useful? React with 👍 / 👎.

logger.warning(f"error in done_callback: {cb}", exc_info=e)

self._done_fut.add_done_callback(_on_done)
self._interrupt_fut.add_done_callback(_on_done)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep RunResult callbacks tied to actual completion

When a SpeechHandle watched by RunResult is interrupted, this invokes _mark_done_if_needed before _done_fut or the linked speech tasks finish. RunResult._mark_done() then freezes the result using incomplete _error and _maybe_run_final_output state, and _item_added() rejects synchronized partial assistant messages produced during subsequent interruption cleanup. Consequently, an interrupted session.run() or remote run can resolve with missing events, trigger an erroneous structured-output retry, or report success before a late generation failure is recorded; completion consumers need a callback tied to actual _done_fut finalization rather than the interruption notification.

Useful? React with 👍 / 👎.

@rrfunde

rrfunde commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Fixed:

  1. Kept SpeechHandle.done() and _done_callbacks strictly tied to actual _done_fut finalization rather than the interruption event, avoiding premature state freeze in RunResult or skipping speech during activity delay.
  2. Made SpeechHandle.exception() safe post-interruption without raising InvalidStateError.
  3. In wait_for_playout(), drained running linked tasks upon interruption before returning so tasks perform their interruption cleanup.

devin-ai-integration[bot]

This comment was marked as resolved.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5565cde686

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +263 to +265
running_tasks = [t for t in self._tasks if not t.done() and t is not curr]
if running_tasks:
await asyncio.wait(running_tasks)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Stop draining linked tasks in the public playout wait

For session-generated handles, _create_speech_task appends the active generation task to _tasks (agent_activity.py:970-978), so an interrupted handle normally enters this branch rather than returning on _interrupt_fut. If that task is stuck in a provider or user async source—the scenario INTERRUPTION_TIMEOUT is intended to recover—this still blocks for five seconds; if the task delays or suppresses cancellation, it can remain blocked even after the timeout calls _mark_done(), because this asyncio.wait no longer observes _done_fut. Thus await handle and wait_for_playout() do not reliably gain the immediate interruption behavior asserted by the new tests, which only use handles with an empty _tasks list; keep task draining on a separate internal finalization path.

Useful? React with 👍 / 👎.

@rrfunde

rrfunde commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Fixed:

  1. Gated SpeechHandle.exception() strictly on self.done() so unfinalized state is never exposed during interruption.
  2. Removed task draining from wait_for_playout(), preserving prompt unblocking upon interruption.
  3. Added synchronous interrupt callbacks on SpeechHandle so AgentActivity immediately cancels in-flight realtime generation upon speech interruption without waiting on event loop task scheduling.

devin-ai-integration[bot]

This comment was marked as resolved.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1e56187cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# the response already landed, cancelling the future no longer reaches it
self._rt_session.interrupt()
return
speech_handle._add_interrupt_callback(self._rt_session.interrupt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Hook interruptions while awaiting realtime say

When VoiceAgent.say() uses a realtime model with supports_say, this callback is never registered because the text is not None branch returns above after directly awaiting _rt_session.say(text). If callers interrupt that SpeechHandle while the provider's say future is pending, the newly immediate wait_for_playout() returns but no provider interrupt is sent; the provider can continue generating or begin speaking until the internal task is cancelled by the five-second interruption timeout. Apply the same interrupt-aware cancellation around the realtime say() future.

Useful? React with 👍 / 👎.

@rrfunde

rrfunde commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review feedback: omitted exc_info when logging callback failures in _cancel() to prevent leaking provider internals, exception tracebacks, or sensitive data.

devin-ai-integration[bot]

This comment was marked as resolved.

@rrfunde

rrfunde commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review feedback:

  • Hooked speech_handle interruption callbacks and wait_if_not_interrupted around _rt_session.say(text) when awaiting realtime say(), properly cancelling/interrupting realtime providers.
  • Omitted callback identity (cb) from the interrupt failure log message to avoid exposing bound arguments or closures.
  • Added unit test covering interruption while awaiting realtime say().

devin-ai-integration[bot]

This comment was marked as resolved.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1f87a58bb2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +4055 to +4056
if not say_fut.done():
say_fut.cancel()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Cancel Phonic's pending say task on interruption

When Phonic is used with a delayed say()—for example, while waiting for readiness or consuming an async iterable—cancelling say_fut only cancels its returned _pending_generate_reply_fut; Phonic's say() separately starts _generate_reply_task, and _send_say() continues to send_say() without checking that future, while interrupt() is a no-op before a generation exists. The handle can therefore finish as interrupted but stale text can still be sent and spoken during a later turn. Fresh evidence since the earlier comment is the current Phonic implementation in livekit-plugins/livekit-plugins-phonic/livekit/plugins/phonic/realtime/realtime_model.py, where lines 1021-1044 decouple the send task from the future and lines 1046-1069 send without a cancellation check.

Useful? React with 👍 / 👎.

@rrfunde

rrfunde commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review feedback:

  1. In Phonic's RealtimeSession, coupled the pending _send_say task to the future returned by say(), so cancelling the future cancels _generate_reply_task immediately.
  2. In RealtimeSession.interrupt(), cancel any active _generate_reply_task and pending say future.
  3. In _send_say, check for future cancellation both before/after waiting for readiness and while streaming chunks, preventing stale text from being submitted to the socket after an interruption.
  4. Added unit tests in test_say.py verifying that future cancellation and session interruption prevent blocked say tasks from submitting text once readiness opens.

devin-ai-integration[bot]

This comment was marked as resolved.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7cdce04bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1048 to +1049
if f.cancelled() and self._generate_reply_task and not self._generate_reply_task.done():
self._generate_reply_task.cancel()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Capture the task belonging to each say future

When say() replaces an earlier pending call, cancelling the old future schedules this callback for the next event-loop iteration, but by then self._generate_reply_task points to the new call's task. The old callback therefore cancels the replacement task, so back-to-back say() calls drop the latest text and leave its future pending until the 10-second timeout; capture the task created for this specific future instead of looking up the mutable session field.

Useful? React with 👍 / 👎.

@rrfunde

rrfunde commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review feedback:

  • Captured the specific say() task in the future's done callback rather than referencing the mutable session attribute, and cleared _pending_generate_reply_fut before cancelling the old future so back-to-back say() calls do not inadvertently cancel replacement generation tasks.
  • Used asyncio.get_running_loop() for the timeout handle.
  • Added unit test verifying back-to-back say() calls successfully execute the replacement speech.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8dea7b90fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pytest.mark.asyncio
async def test_say_cancellation_cancels_pending_send_task() -> None:
model = RealtimeModel(api_key="fake")
sess = RealtimeSession(model)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Close Phonic sessions in the unit tests

Each RealtimeSession(model) immediately starts the phonic-realtime-session task (realtime_model.py:519), but none of these tests calls await sess.aclose(). Because every test yields with asyncio.sleep, that task begins opening a real provider connection and remains pending; the autouse fail_on_leaked_tasks fixture in tests/conftest.py:341-372 will therefore fail teardown, in addition to violating the hermetic unit category. Close each session in a finally block or fixture.

AGENTS.md reference: AGENTS.md:L40-L42

Useful? React with 👍 / 👎.

@rrfunde

rrfunde commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review feedback:

  • Wrapped all RealtimeSession unit tests in test_say.py with try ... finally: await sess.aclose() to ensure background session tasks are cleanly torn down.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e78b9b7ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pytest.mark.asyncio
async def test_say_cancellation_cancels_pending_send_task() -> None:
model = RealtimeModel(api_key="fake")
sess = RealtimeSession(model)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Mock the Phonic connection in unit tests

Each RealtimeSession(model) immediately schedules _main_task(), and the first asyncio.sleep() lets that task call the real Phonic SDK's connect().__aenter__(); assigning an AsyncMock to _socket does not mock that connection. These unit tests can therefore make external network requests with the fake key and become environment-dependent. Fresh evidence after the earlier cleanup comment is that the added aclose() runs only in finally, after the sleeps have already allowed the connection attempt; mock sess._client.conversations.connect or prevent _main_task from starting.

AGENTS.md reference: AGENTS.md:L40-L44

Useful? React with 👍 / 👎.

@longcw

longcw commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

closing since SpeechHandle.wait_for_playout() should wait for the playout actually stopped.

@longcw longcw closed this Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SpeechHandle.wait_for_playout() ignores interruption, causing 5s deadlock during tool preamble

2 participants