feat(breeze_tts): sub-chunk streaming via resumable AR stepper

[Th-Underscore]

Summary

BreezeTTS streaming currently emits at most one audio event per text chunk, so first audio only becomes available once the chunk's full autoregressive run plus decode has finished. This adds a sub-chunk streaming path: a resumable autoregressive stepper advances the backbone by a configurable number of frames per event (reusing the existing streaming primitives) and emits the decoded prefix audio every event, withholding a configurable lookahead margin.

This directly addresses the streaming time-to-first-audio concern raised in issue #467, where chunk-granular streaming buffers an entire text chunk before emitting any audio.

• New session options: breeze_tts.stream_subchunk (bool, default false — existing chunk-granular streaming is unchanged), breeze_tts.stream_frames_per_event (default 32), breeze_tts.stream_lookahead_margin (default 12; must stay below frames-per-event or the emit watermark stalls into the 256-iteration guard).
• Offline generate() delegates to the same stepper as a single event; output is bit-identical to the pre-stepper path (verified).
• The VRAM fix (free the decoder graph before rebuilding its replacement) applies to all decode paths, capping the peak at one graph rather than old+new.

Known limitation: per-event prefix re-decode is O(N²)

Each event decodes the entire accumulated code prefix (the Mimi decoder has no incremental state), so per-event cost is c_AR·E + c_decode·prefix and total decode work grows quadratically in frames within a chunk. Measured on a 988-codepoint single-run corpus (below), per-event cost climbs monotonically through the run. Two facts bound the practical impact:

• The prefix resets to the reference at every chunk boundary, so total cost is linear in total text length with a slope proportional to chunk size — small text_chunk_size already bounds both cost and drift.
• The fix is incremental per-frame decoder state (Mimi's upstream StreamingConv1d.previous / StreamingConvTranspose1d.partial ring buffers) across ~30 decoder layers, with a mandatory byte-identical parity gate. That is a separate, large change and is out of scope here; it is the planned follow-up.

Spec note

For a released GGUF the server prefers the GGUF-embedded contract spec, so the new options are unknown to already-published artifacts and are rejected (unknown BreezeTTS request option: stream_lookahead_margin) until the server is launched with --model-spec-override pointing at a spec directory containing the updated model_specs/breeze_tts.json. This is a stale-artifact issue, not a code bug.

Validation

Tested on 2×V100-16GB, server on GPU 2, released breeze-tts-2-q8_0 GGUF + --model-spec-override, 24/24 unit-gate green on the current base (upstream moved the extended tests out of the unit gate after this PR's base PR #423 merged; 24/24 is the full gate).

• Bit-identity: offline vs streaming (single event) output is identical.
• Live web client plays PCM deltas over WebAudio as they arrive; intermediate deltas are inaudible (global decoder), final audio clean.
• Short clip, 16/8: 13 deltas, 15.29s wall, 13.27s spread, mean gap 1.11s (median 1.31, max 1.41).
• Long 988-codepoint single-run corpus (tags expand to real vocal events, so decoded audio exceeds word count), identical 87.120s audio across arms:
  • 32/12: 35 deltas, 112.60s spread, drift 2.54s → 4.18s (+64.6%), 115.86s wall (audio 75.2% of wall), RTF 1.33.
  • 16/8: 69 deltas, 143.06s spread, drift 1.28s → 2.94s (+40.9%), 145.28s wall (audio 60.0%), RTF 1.67.

The 16/8 arm costs +25.3% wall on the long corpus (vs ~+5% on short clips) — the O(N²) penalty growing as a fraction of wall with length, exactly as the model above predicts. 16/8 is a clean win for short clips (lower TTFT, ~2× event cadence); for long-form within one chunk 32/12 is the cheaper trade until the incremental-decode follow-up lands.

Test artifacts (gists)

• Web client (interactive testing): https://gist.github.com/Th-Underscore/79cc55024c13a3724aa4ca32731a931f — standalone, stdlib-only. python3 stream_client.py --port 5025 --server http://127.0.0.1:5023 --voice-dir <dir-with-clone-wavs>
• Measurement scripts: https://gist.github.com/Th-Underscore/b6ab3cc761fafb162306da6c7ffdc94e — fpe_measure.py <frames> <margin> (short-clip A/B) and fpe_long.py <frames> <margin> (long-corpus A/B; forces one AR run via text_chunk_size=2048)
• Long test corpus (988 codepoints): https://gist.github.com/Th-Underscore/55013c7468b53e70c5aeee658316f361 — copy to /tmp/long_test.txt before running fpe_long.py

Voice reference for streaming tests

Smoke tests used a VCTK speaker p233 clone: 15.75s at 48 kHz, a deterministic same-text concat of utterances 016+025+030 with 0.3s silence gaps, reproducibly rebuildable from the source dataset (https://huggingface.co/datasets/badayvedat/VCTK — the source copy is declared apache-2.0; note VCTK is Cambridge corpus data). Transcripts sidecar: https://gist.github.com/Th-Underscore/bbfd2a7d65953e44025a739cc9cdfd26. The WAV itself cannot be published (gists do not accept binary files); a maintainer can rebuild it from the utterance list above if desired.

Limitations

• Sub-chunk streaming is BreezeTTS-only: its Mimi decoder can decode arbitrary prefixes, so intermediate prefix decodes are coherent (differ slightly, decay to exact at completion). Other families have no equivalent.
• Events are still chunk-boundary-aligned in time; no family streams within a single event.
• The O(N²) prefix re-decode above is inherited as a known limitation until the incremental-decode follow-up lands.
• The existing audio.cpp web UI does not support realtime streaming — its streaming mode is chunk-granular only — so sub-chunk streaming requires a custom client (the gist above) or direct API use.

Complete and validated as described; the incremental per-frame decoder state (Mimi previous/partial) is the planned follow-up.

[0xShug0]

@Th-Underscore For the streaming design, I'd suggest following the PocketTTS pattern more closely: keep the codec decoder stateful and decode only the new streaming step/window, instead of repeatedly decoding the accumulated prefix and slicing the tail. Also, please provide an A/B performance regression test for the offline path if it is changed.

[Th-Underscore]

@0xShug0 As mentioned, a full stateful decoder upgrade ("incremental per-frame decoder state") would be a much larger PR (~1500 lines on top of the current changes). I can work on it in the background, but I'd suggest having the subchunk streaming foundation merged first. Of course, it's ultimately your choice, so say the word and I'll probably get it done before the end of this week.

Anyway, the offline path is almost completely untouched (only the StreamState memory management optimization and use_cfg threading changes). But just to be sure, here are the A/B results via ab_harness.py (274-codepoint text, seed=7):

	A: f6277c1 (old-mono)	B: 1000c2a (stepper)
run0 (cold)	26.00s	25.97s
run1 (warm)	14.01s	13.95s
run2 (warm)	14.15s	13.87s
best warm RTF	1.00	0.99
audio dur	14.0s	14.0s
peak VRAM	6062 MiB	6115 MiB
output sha256	cbc2db2bec0e9797	cbc2db2bec0e9797

-> byte-identical output, perf within noise, +53 MiB (StreamState).

[0xShug0]

@Th-Underscore

a full stateful decoder upgrade ("incremental per-frame decoder state") would be a much larger PR (~1500 lines on top of the current changes).

Let me check. I’m happy to take this on, and you can follow up with improvements afterward.