Add Vui TTS plugin (local streaming TTS on CUDA / Apple Silicon) - #7243
Add Vui TTS plugin (local streaming TTS on CUDA / Apple Silicon)#7243mogwai wants to merge 5 commits into
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Vui Nano is a small, context-aware text-to-speech model trained on real conversations (219M active / 305M total parameters, Apache 2.0). The plugin runs the model in-process via the `vui-tts` package — on an NVIDIA GPU, or on MLX on Apple Silicon — with no API key; weights and voice prompts download from Hugging Face on first use (prewarm() does it ahead of time). - synthesize(): one-shot render, rewound to the voice prompt afterwards. - stream(): sentences are rendered as the LLM produces them, in one conversation row, so prosody carries across the reply. - All engine work (load, prefill, decode, rewind) runs on one worker thread — CUDA graphs and MLX streams are bound to the thread that created them. - voice= is one of the shipped voices, a prompt .safetensors baked with Vui's scripts/build_prompts.py, or a .wav to clone. Measured on an M4 (MLX, int8): one-shot 5.5s of audio in 3.1s; streamed input 5.5s in 2.3s with first audio at 407ms.
|
Addressed the three Devin Review flags (they're hidden on the PR by the org settings, so summarising here):
Rebased on current |
…vs, require vui-tts 1.1.3 - A SynthesizeStream holds the engine lock from its first sentence to its end, so two streams can no longer interleave sentences on the single row or rewind each other's context; ChunkedStream takes the lock per call. - A .wav voice without a transcript now raises a clear error (sibling .txt, a baked prompt .safetensors, or `vui-tts[server]` for transcription) instead of failing on the missing openai-whisper import. - vui-tts >= 1.1.3: CUDA Row.rewind() now re-seeds the codec context from the voice prompt, so a turn no longer decodes against the previous turn's tail. Lock regenerated.
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 1 new potential issue.
5 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| async def aclose(self) -> None: | ||
| self._executor.shutdown(wait=False) |
There was a problem hiding this comment.
🔴 Synthesis can outlive shutdown
When synthesize() or stream() runs after shutdown begins, it escapes aclose()’s stream snapshot. It can recreate _engine after cleanup clears it, leaving GPU allocations alive after executor shutdown.
Learn more
Shutdown marks the TTS instance closed and snapshots the current weak set before releasing the row and executor. The synthesis entry points do not check that state. A new stream created after the snapshot starts its base-class task immediately, but shutdown never cancels it. Once cleanup clears _row and _engine, the missed task can enter _render_blocking, initialize another engine, and leave that engine outside the row being closed.
Example: aclose() snapshots one active stream and awaits its cancellation. Another task calls synthesize("hello") during that await. Cleanup then clears the engine, while the new stream later initializes a replacement that cleanup never closes.
Recommended fix: Reject synthesize(), stream(), and prewarm() once closing starts. Also synchronize stream registration with the transition to closing so no stream can appear between the state check and the shutdown snapshot.
Was this helpful? React with 👍 or 👎 to provide feedback.
…errides aclose() now cancels live streams (tracked in a WeakSet, as the other streaming plugins do), waits for the worker thread off the event loop, closes the row and drops the engine reference, so a session that closes mid-sentence no longer leaves a render running or the model reachable. Idempotent. mypy: vui-tts and torchcodec are analysed as opaque (ignore_missing_imports + follow_imports = "skip"), matching the existing overrides for other third-party SDKs — torchcodec re-exports its decoders without declaring them, and vui-tts only ships py.typed from 1.1.4.
|
Two more updates:
Re-verified after the change: MLX one-shot and streamed synthesis 0.0% WER with |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
`_closed` was set before cleanup ran, so an aclose() cancelled part-way left the row, worker thread and model allocated with every later call returning immediately. Now every step is idempotent and the flag is set only once all of them complete, under a lock, so a retry finishes the job. Verified: cancel aclose() mid-render, retry completes with the engine released and the executor shut down.
|
Fixed the cancelled-shutdown case: |
2 participants
What
A
livekit-plugins-vuiTTS plugin for Vui Nano — a small, context-aware text-to-speech model trained on real conversations (219M active / 305M total parameters, Apache 2.0). It runs in-process through thevui-ttspackage: on an NVIDIA GPU, or on MLX on Apple Silicon. No API key; weights and the shipped voice prompts download from Hugging Face on first use (prewarm()does it ahead of the first request).Design
synthesize()renders one text and rewinds the model to the voice prompt.stream()renders each sentence as the LLM produces it (viatokenize.basic.SentenceTokenizer) in one conversation row, so the model conditions each sentence on what it just said and prosody carries across the reply. The row is rewound when input ends.Verified
On an M4 (MLX, int8): one-shot 5.5 s of audio in 3.1 s (1.8×); streamed input 5.5 s in 2.3 s (2.4×), first audio at 407 ms. Output transcribed with moonshine ASR.
ruff check/ruff formatclean;uv lockregenerated.I'm the author of the model (Fluxions AI) and will maintain the plugin.
Update — CUDA verified too (RTX 5090, torch 2.11, plain pip install without flash-attn, sharing the GPU with another job): one-shot 6.6 s of audio in 2.7 s (2.4×); streamed 8.2 s in 3.3 s (2.5×) with first audio at 48 ms; both 0.0% WER by moonshine. Devin Review flags addressed — see the comment below.
Scenario verification (exact PR head, MLX on M4 and CUDA on a 5090): one-shot; a three-sentence streamed reply in one row; a stream interrupted after 6 frames (
aclose()) followed by a one-shot on the same row; two concurrentsynthesize()calls (serialised, both complete); a voice switch viaupdate_options; andaclose()cancelled mid-render then retried (engine released, executor shut down, no live streams). Every rendered wav transcribed by moonshine: 0.0% WER on both backends apart from one ASR-normalisation "OK"/"okay" and one single-word slip in the 40-word streamed reply on CUDA. First audio 48 ms on CUDA, ~400 ms on M4.