Repository navigation
realtime: announce speakers in transcription-only sessions - #12871
Merged
Merged
Conversation
With voice_recognition.identity.announce on, a realtime session sends a conversation.item.speaker event for each user turn. Only generateResponse sent this event. A transcription-only session never calls generateResponse, so it resolved the speaker of each turn and then dropped the result. In a transcription-only session with announce on, send the event before the completed transcription event of the turn, with the same item_id. The rules are the same as on the response path: no event for an unknown speaker unless announce_unknown is set, and no event for a speaker that an enforcing gate rejects. Sessions with announce off and response sessions do not change. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mudler
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With
voice_recognition.identity.announce: true, a realtime session sends aconversation.item.speakerevent for each user turn. OnlygenerateResponsesends this event. A transcription-only session never callsgenerateResponse. It resolves the speaker of each turn (IdentityEnabled()starts the lookup), but it then drops the result, and the client gets no event.This PR adds the event to transcription-only sessions:
When the session is transcription-only and
identity.announceis on, the commit path waits for the speaker lookup that is already running and sendsconversation.item.speakerbefore the turn'sconversation.item.input_audio_transcription.completedevent. Both events have the sameitem_id. A client therefore knows who spoke when the final transcript arrives.The event shape does not change:
{"type": "conversation.item.speaker", "item_id": "item_abc", "speaker": {"name": "...", "id": "...", "confidence": 92.0, "distance": 0.1, "matched": true}}The rules are the same as on the response path, through one shared helper (
announceSpeaker): no event for an unknown speaker unlessannounce_unknownis set, and no event for a speaker that an enforcing gate rejects.The lookup result is joined once and cached, so the gate check later in the commit path uses the same result.
emitTranscriptionandemitPrecomputedTranscriptionget an optionalbeforeCompletedhook. All call sites other than the transcription-only path passnil.The feature is off unless the pipeline sets
identity.announce. Sessions with announce off and response sessions do not change. In a response session, the event still comes after the user conversation item.The docs for the
conversation.item.speakerevent now say how it is ordered in a transcription-only session.Tests
New specs in
realtime_voicegate_integration_test.godrive the realcommitUtterancepath with the existing fake transport, model and registry:item_id, and no responsepersonalizeonly): no speaker eventThe first spec fails when the new path is disabled.
go vet ./core/http/endpoints/openai/andgo test -race ./core/http/endpoints/openai/pass (aftermake protogen-go).Notes for Reviewers
In a transcription-only session with announce on, the completed event now waits for the speaker lookup. The lookup runs at the same time as transcription, so the extra wait is the time by which the lookup is slower than transcription, if any.
🤖 Generated with Claude Code