openai: keep GPT-Live's input clock running when no audio is pushed - #7239
Open
u9g wants to merge 3 commits into
Open
openai: keep GPT-Live's input clock running when no audio is pushed#7239u9g wants to merge 3 commits into
u9g wants to merge 3 commits into
Conversation
GPT-Live speaks only while its input clock runs, and that clock is the audio the client appends. A session with no microphone (a text simulation, a text-mode console, muted input) never appended anything, so every ask timed out with "the model did not start speaking when asked". Verified against the live service: the same ask that times out with no input opens a reply in under two seconds once silence flows. GPTLiveSession now appends 100 ms of silence whenever nothing has been pushed for 200 ms, once the session has started. A live microphone is never padded.
The guard from #7236 treated the missing audio as fatal. The GPT-Live plugin now supplies the input clock itself, so a text simulation runs a duplex model like any other.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GPT-Live speaks only while its input clock runs, and that clock is the audio the client appends. A session with no microphone never appends anything, so every ask times out after 10 seconds with:
That is what happens under a text simulation, in
lk agent console --text, and whenever input audio is disabled. #7236 papered over the first case by refusing to start; the other two still hang.Measured against the live service with the adapter's
generate_reply: with nothing pushed the ask times out; with 100 ms silence frames pushed at 10 Hz the reply opens in 1.6 s with transcript and audio.session.input_audio.mutealone does not do it.Change
GPTLiveSessionappends 100 ms of silence whenever nothing has been pushed for 200 ms, once the session has started. A live microphone is never padded: the feeder only fills gaps.push_audiosplit into the timestamp and a shared_append_audio.Tests
tests/test_gpt_live_model.py: idle session appends silence; a live microphone is not padded. The fake sockets ignore silent appends by default so the existing sequence assertions stay exact; the two tests that assert on silent input opt back in. The two #7236 tests are removed with the guard. 99 tests pass across the GPT-Live and duplex adapter files.End to end: a GPT-Live agent under
lk agent simulate(text mode) now holds full conversations on this branch. Same run on main fails every scenario on turn one.What text mode still cannot do well
GPT-Live has no typed user-input channel; the plugin delivers typed turns as
session.commentary.appendwith an instruction to reply. In the end-to-end run the voice model delegated to the backend (and so reached the tools) in one of three scenarios, versus three of three in audio mode. Text mode is now usable for exercising conversation flow, not for asserting tool behavior. That limit is the protocol's, not this change's.agents-js counterpart: livekit/agents-js#2484.