fix(duplex): keep late transcripts on the audio they describe - #2491
rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 3400f7b The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (burst.anchorMs === undefined) { | ||
| burst.anchorMs = fragment.startMs - burst.audioStartMs; | ||
| this.lastBurst = burst; | ||
| } |
There was a problem hiding this comment.
🟡 Fully delayed transcripts join the next burst
When a burst closes before its first timestamped fragment, lastBurst remains unset. Its delayed transcript waits, then attachFragments assigns it to the next speech burst.
Learn more
lastBurst is recorded only when attachFragments establishes an anchor from a fragment. A burst that ends before any transcript arrives therefore leaves no completed burst for onTranscriptDelta to examine. If another speech burst starts within the three-second unclaimed-transcript window, its first audio frame calls attachFragments and consumes the old transcript. The resulting generation and chat message pair the old words with the new audio.
Example: A model emits burst A from 2000–2300 ms, and the gate closes it before any transcript arrives. At 2400 ms, Hello arrives with startMs: 2000; no text-only generation is created because lastBurst is undefined. If burst B starts at 3000 ms, B consumes Hello, although the text describes burst A.
Recommended fix: Preserve enough timing state for every completed speech burst, even before it receives a transcript. For stamped audio, retain the burst timeline directly; for unstamped audio, establish or reconcile the anchor when the first delayed fragment arrives. Add a regression test where an entire timestamped transcript arrives after its burst closes and before the next burst starts.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#7264 to keep delayed duplex transcript fragments associated with the audio they describe.
Late fragments are attached when an audio burst closes. Fragments whose timestamps place them on an already-finished burst are emitted immediately as a text-only generation, while only a subsequent speech burst can claim a pending
generateReplyrequest.Source: livekit/agents#7264
Source diff coverage
Source diff coverage
livekit-agents/livekit/agents/llm/duplex_adapter.py: adapted toagents/src/llm/duplex_adapter.ts. Ported all burst bookkeeping, close-time fragment attachment, immediate text-only generations for transcript tied to already-played audio, reply-kind handling, context insertion, and reconnect reset. Adaptations are limited to TypeScript naming, Web Streams/queue primitives, and the target's millisecond-native conventions.tests/test_duplex_adapter.py: adapted toagents/src/llm/duplex_adapter.test.ts. Ported all four added regression tests into the existing Vitest fake-duplex harness, preserving each source scenario and assertion intent.No source files are omitted and no target infrastructure gap remains.
Validation
pnpm build: passed, 40/40 workspace buildspnpm test agents: passed, 157 files and 2,635 tests; 5 skippedpnpm --filter @livekit/agents lint: passed with existing warningspnpm --filter @livekit/agents typecheck: passedpnpm format:check: passedpnpm lint: blocked by the pre-existing unrelated@typescript-eslint/no-misused-promiseserror inplugins/openai/src/ws/llm.ts:127cue-cli: attempted against a built, explicitly dispatched GPT-Live agent; the configured LiveKit environment rejected both worker and driver with401 Unauthorized - invalid API key, so no runtime event assertion completedPorted from livekit/agents#7264
Original PR description
Problem:
DuplexRealtimeAdapterattaches queued transcript fragments only inside the frame handler, while the gate is open, so a fragment that arrives after the last accepted frame stays queued. The next utterance takes it as its first fragment and sets its time anchor from it, or the 3 s unclaimed timeout emits it under a new id with an error log.Fix:
_close_burstattaches due fragments before it closes the burst, so words that arrive between the last frame and the close still join it. A fragment whose span, by the last anchored burst's anchor, sits on audio that burst already played is emitted at once as a text-only generation under a fresh id, the shape the unclaimed path already produces, so it can no longer anchor the next burst.Fixes #7227. Supersedes #7244.
Context for reviewing and coding agents