Skip to content

realtime: announce speakers in transcription-only sessions - #12871

Merged
mudler merged 1 commit into
masterfrom
feat/realtime-transcription-speaker
Oct 23, 2026
Merged

mudler merged 1 commit into
masterfrom
feat/realtime-transcription-speaker

Conversation

@mudler-agent

Copy link
Copy Markdown
Collaborator

Description

With voice_recognition.identity.announce: true, a realtime session sends a conversation.item.speaker event for each user turn. Only generateResponse sends this event. A transcription-only session never calls generateResponse. It resolves the speaker of each turn (IdentityEnabled() starts the lookup), but it then drops the result, and the client gets no event.

This PR adds the event to transcription-only sessions:

  • When the session is transcription-only and identity.announce is on, the commit path waits for the speaker lookup that is already running and sends conversation.item.speaker before the turn's conversation.item.input_audio_transcription.completed event. Both events have the same item_id. A client therefore knows who spoke when the final transcript arrives.

  • The event shape does not change:

    {"type": "conversation.item.speaker", "item_id": "item_abc",
     "speaker": {"name": "...", "id": "...", "confidence": 92.0, "distance": 0.1, "matched": true}}
  • The rules are the same as on the response path, through one shared helper (announceSpeaker): no event for an unknown speaker unless announce_unknown is set, and no event for a speaker that an enforcing gate rejects.

  • The lookup result is joined once and cached, so the gate check later in the commit path uses the same result.

  • emitTranscription and emitPrecomputedTranscription get an optional beforeCompleted hook. All call sites other than the transcription-only path pass nil.

The feature is off unless the pipeline sets identity.announce. Sessions with announce off and response sessions do not change. In a response session, the event still comes after the user conversation item.

The docs for the conversation.item.speaker event now say how it is ordered in a transcription-only session.

Tests

New specs in realtime_voicegate_integration_test.go drive the real commitUtterance path with the existing fake transport, model and registry:

  • transcription-only with announce on: one speaker event, before the completed event, same item_id, and no response
  • transcription-only with announce off (no identity block, or personalize only): no speaker event
  • transcription-only with an enforcing gate that rejects the speaker: no speaker event
  • response session with announce on: unchanged, and the speaker event comes after the completed event

The first spec fails when the new path is disabled. go vet ./core/http/endpoints/openai/ and go test -race ./core/http/endpoints/openai/ pass (after make protogen-go).

Notes for Reviewers

In a transcription-only session with announce on, the completed event now waits for the speaker lookup. The lookup runs at the same time as transcription, so the extra wait is the time by which the lookup is slower than transcription, if any.

  • Documentation updated (docs/content/) for user-facing changes

🤖 Generated with Claude Code

With voice_recognition.identity.announce on, a realtime session sends
a conversation.item.speaker event for each user turn. Only
generateResponse sent this event. A transcription-only session never
calls generateResponse, so it resolved the speaker of each turn and
then dropped the result.

In a transcription-only session with announce on, send the event
before the completed transcription event of the turn, with the same
item_id. The rules are the same as on the response path: no event for
an unknown speaker unless announce_unknown is set, and no event for a
speaker that an enforcing gate rejects.

Sessions with announce off and response sessions do not change.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mudler
mudler merged commit 3d68e72 into master Oct 23, 2026
18 of 23 checks passed
@mudler
mudler deleted the feat/realtime-transcription-speaker branch October 23, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants