Built to deliver real-time AI voice conversations without expensive third-party voice API lock-in.
AI voice assistants today are either cloud-dependent (OpenAI Realtime API at $5–$20/hour) or lack real-time responsiveness. Open-source speech models exist but require significant integration work to achieve sub-200ms voice latency across mobile and desktop.
ConvoGPT is a cross-platform AI conversation app that runs custom fine-tuned Piper TTS models locally, processes speech-to-text via sherpa-onnx, and orchestrates a 3-layer "mind" architecture for natural turn-taking — all without recurring per-minute API fees.
📹 Demo: [Insert 30-second screen recording showing voice conversation across mobile and desktop]
🔗 Deployment: React Native (iOS/Android) + Electron (Desktop)
- Voice latency: Sub-200ms STT → AI → TTS pipeline via local sherpa-onnx inference
- Custom voice model: Fine-tuned Piper TTS (61MB ONNX) trained for the application — no cloud TTS costs
- Cross-platform: Single TypeScript core deploys to iOS, Android, macOS, and Windows
- Real-time canvas: AI-generated artifacts stream via EventSource/WebSocket with <500ms perceived latency
- 3-layer mind architecture: Subconscious (intent detection) → Conscious (response selection) → Unconscious (memory consolidation) enables context-aware conversations without stateless API churn
- Session broker: Express middleware handles 50+ concurrent device pairings via QR-code workspace auth
| Layer | Technology |
|---|---|
| Frontend (Mobile) | React Native (Expo), TypeScript, Zustand |
| Frontend (Desktop) | Electron, Web Audio API |
| Backend | Express, SSE streaming, REST |
| Speech (STT) | sherpa-onnx (local inference) |
| Speech (TTS) | Piper TTS (custom fine-tuned model) |
| AI | DeepSeek API, OpenAI-compatible provider abstraction |
| State | Zustand, async-storage, conversation storage |
| Auth | QR-code workspace pairing, broker sessions |
# 1. Install dependencies
npm install
# 2. Download the custom TTS voice model
npm run tts:package-piper
# 3. Start the development server
npx expo startSee .env.example for required environment variables (API keys, broker URL, workspace config).
src/
├── api/ # Backend client libraries (broker, canvas, voice, workspace)
├── audio/ # Audio session coordination and playback
├── canvas/ # AI artifact generation and streaming
├── diagnostics/ # Telemetry and event tracking
├── memory/ # Conversation storage and retrieval
├── theme/ # Design system (palette, spacing, radii)
├── ui/ # Shared components (tab bar, voice orb, error boundary)
└── voice/ # STT/TTS provider abstraction and orchestration
- voice-clone-server — Piper TTS training pipeline used to create the custom voice model
- projector — P2P storage layer used by the desktop app for offline conversation sync
MIT — see LICENSE