Skip to content

Feature/gemini live - #61

Open
juliantoledo wants to merge 2 commits into
android:mainfrom
juliantoledo:feature/gemini-live
Open

juliantoledo wants to merge 2 commits into
android:mainfrom
juliantoledo:feature/gemini-live

Conversation

@juliantoledo

Copy link
Copy Markdown

Title

feat(agent): add Gemini Live (gemini-3.8-live) voice session support for AppFunctions

Description

Summary

Adds real-time bidirectional voice interaction to the Agent Demo screen using the Gemini Live API (models/gemini-3.8-live), enabling users to invoke Android AppFunctions and internal agent tools by voice while viewing live transcriptions, tool execution status, generated image attachments, and PendingIntent action confirmation cards directly in the active chat thread.

Key Changes

  • Gemini Live WebSocket Session (GeminiLiveSessionManager & LiveSessionState):

    • Connects to the Gemini v1beta BidiGenerateContent WebSocket endpoint using Ktor WebSockets (DataModule.provideHttpClient) and the configured Gemini API key from SettingsRepository.
    • Configures "models/gemini-3.8-live" with AUDIO response modality, input/output audio transcriptions, system instructions, and all enabled AppFunction and internal tool declarations.
    • Streams user speech and model audio responses while flushing input and output transcriptions into the active chat thread as USER and ASSISTANT messages.
  • Low-Latency PCM Audio I/O (GeminiLiveAudioHelper):

    • Captures 16 kHz mono 16-bit PCM microphone audio (VOICE_COMMUNICATION with AcousticEchoCanceler enabled when available) and plays back 24 kHz mono 16-bit PCM model audio via AudioTrack.
    • Supports immediate playback queue clearing on user interruption and recording pause/resume while executing tool calls.
  • AppFunction & Internal Tool Bridge (AgentOrchestrator & GeminiToolConverter):

    • Added AgentOrchestrator.getEnabledTools and AgentOrchestrator.executeLiveToolCalls to execute AppFunctions and internal tools (getCurrentLocation, geocodeAddress, generateImage) during a Live session, updating AgentStatus.InvokingTool, posting generated image attachments and PendingIntent confirmation cards to the thread, and returning functionResponses over the WebSocket.
    • Added GeminiToolConverter.convertForLiveApi to normalize tool schemas (enum field naming and OpenAPI format compatibility) for BidiGenerateContent.
  • Agent Demo UI & Permissions (AgentDemoScreen, AgentDemoViewModel, AgentUiState, AndroidManifest.xml):

    • Added a microphone / stop toggle button in the chat input bar with runtime RECORD_AUDIO permission handling and @App scoping support.
    • Added a LiveSessionIndicator status banner when connecting or listening, and updated ModelDropdown in the top bar to display gemini-3.8-live while a Live session is active.
    • Declared INTERNET and RECORD_AUDIO permissions in AndroidManifest.xml.
  • Unit Tests:

    • Added GeminiLiveSessionManagerTest, GeminiToolConverterTest.convertForLiveApi_normalizesEnumsAndStripsFileReferenceFormat, and AgentDemoViewModelTest coverage for OnToggleLiveSession.

@juliantoledo
juliantoledo requested a review from a team as a code owner October 23, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant