feat(audio8_tts): Falcon-H1 0.1B (Mamba2 + attention) support and codec/AR performance - #444
Open
gqf2008 wants to merge 55 commits into
Open
feat(audio8_tts): Falcon-H1 0.1B (Mamba2 + attention) support and codec/AR performance#444gqf2008 wants to merge 55 commits into
gqf2008 wants to merge 55 commits into
Conversation
…d port Implement the previously-stubbed Falcon-H1 slow-AR path: - stateful per-token forward (falcon_forward_step): RMSNorm -> (Mamba2 || GQA attention) -> residual -> gated FFN, with conv/SSM states and KV cache - Mamba2: in_proj [z,xBC,dt] split -> ssm_conv (kernel flipped to match HF causal_conv1d) -> ssm_scan -> D -> silu(z) gate -> out_proj - GQA attention: q/k/v proj -> RoPE NEOX (base 1e11) -> flash_attn with KV cache - fix: lm_head_multiplier does NOT apply to ArkttsModel compact semantic head - fix: embedding_multiplier applies to (text_emb + codebook_sum) together Verified against transformers reference: logits scale/argmax match; CPU runs 627 tokens stable. (The ggml-metal.metal change from the original af7bcd4 was split into the two following commits: a mechanical LF renormalize, then the ssm_scan fix.)
Mechanical line-ending normalization per .gitattributes (text=auto eol=lf); no semantic change (diff -w against the parent is empty). Split out of af7bcd4 so the ssm_scan fix that follows is visible as its own hunk.
- ggml_set_output(scan): keep the SSM state tail alive for host read-back (gallocr was reusing the scan result buffer, corrupting the read state) - conv1d flip: GGUF layout is [d_conv,1,conv_dim], not HF [conv_dim,1,d_conv] - embedding_multiplier applies to (text_emb + codebook_sum) together - lm_head_multiplier does NOT apply to ArkttsModel compact semantic head - ggml-metal ssm_scan: fix reduction garbage for d_state>32, n_t<sgptg Result: 0.1b logits match transformers reference scale/argmax; CPU runs 185 tokens before a dt-clamping-related zero-out. Still TODO: clamp dt_softplus to [time_step_min, time_step_max] to prevent state death on long sequences.
Document the two remaining issues (logits argmax mismatch vs reference, recurrent SSM state blow-up) plus the debugging approach and cleanup notes, for hand-off to a follow-up agent.
The fast (audio-codebook) AR runs one graph submission per generated codebook token, making it submit+sync latency bound on GPU backends: ~2.9ms/step on Metal versus ~0.6ms/step for the same graph on CPU. When the main backend is a GPU, give the fast AR a dedicated CPU backend: byte-copy the fast-layer projections and fast_output onto it (same ggml type, lossless for q8_0/f16/f32), build the fast graph, state buffers, and constants cache there, and route run() at it. The slow path (Qwen or Falcon-H1) and codec stay on the main backend; the only cross-backend traffic is small host vectors. Measured on M4/Metal, q8_0, 62-char prompt (278 frames): - 0.1B: fast AR 7.28s -> 1.68s, session 16.1s -> 10.2s (RTF ~1.3 -> ~0.8) - 0.6B: fast AR 8.6s -> 3.39s, session 17.4s -> 12.7s (RTF 1.26 -> ~1.0) - --backend cpu unchanged; ASR round-trip verbatim on both backends
Split audio8_tts.codec_decode_ms into graph_build_ms (graph construction, weight upload, gallocr reserve) and graph_compute_ms (submit + GPU execution), and log the node count. On M4/Metal q8_0 the 1105-node decode graph builds in ~200ms and computes in ~4.8s, so per-phase attribution now points at GPU execution directly.
The audio codecs store activations as [frames, channels] (time-fast), the transpose of the LLM layout ggml kernels are tuned for. On that layout ggml_conv_1d's im2col is a strided gather (kernel taps sit C*4 bytes apart, ~16x read amplification), costing ~200ms per conv at [569k, 96] on M4 -- ~4.5s of the audio8_tts codec decode. Add a Metal-only fast path in Conv1dModule for padding=0, stride=1, batch=1, contiguous F32 input: transpose the input to channel-fast once, run one contiguous GEMM per kernel tap over shifted views, and transpose the accumulator back. Weights are regrouped to per-tap rows with a single cont(permute). audio8_tts long-text A/B/A (normalized by fast_graph_ms load indicator, quiet machine): codec graph compute 4970-4981ms -> 2248-2279ms (~2.2x), session wall 9.9s -> 6.6-6.9s. Same-seed output byte-identical across runs; in-graph parity vs ggml_conv_1d (sum_abs_diff ~ 0); ASR round-trips unchanged.
The falcon_forward_step graph is ~600 tiny nodes per token; on Metal it is dispatch-latency bound (~5.9 ms/step) while the same graph computes in ~2.0 ms on CPU (measured both ways via new falcon_step_* profile timers). Mirror the fast-AR treatment: retarget the Falcon-H1 layer weights (incl. ssm_A/ssm_D, so the per-step A/D reads become plain CPU memcpys instead of GPU->host syncs) and the semantic head onto the existing dedicated CPU backend, and call falcon_forward_step with it. --backend cpu behavior is unchanged (no retarget, same backend). Long-text session: ar_generate 6284 -> 3040 ms, session wall ~6.8s -> 5.3s (RTF ~0.48 -> ~0.38). Audio duration byte-identical, ASR round-trips verbatim on both backends.
…slots in-graph The zero-copy path no longer uploads the KV cache per step nor reads the fresh k/v back for a host-side append: per-layer caches live in geometrically grown padded buffers ([head_dim, cap, n_kv]) bound as external leaves; the new token's k/v are copied into slot seq by in-graph ggml_cpy (expanded before the attention nodes so the write precedes the read on sequential backends), and flash_attn_ext reads a strided prefix view over slots [0, seq+1) (CPU flash only requires contiguous rows). This removes the per-step concat of the full cache prefix as well as the host re-striding append. Measured (M4, quiet, interleaved A/B): falcon download 92->0.8 ms, compute -20 ms, ar_generate -100 ms; same-seed output wavs bit-identical. The exact-size cache + concat path is kept for non-host backends.
The per-token step graph no longer depends on the sequence length: flash attention reads the full padded KV cache under an -inf mask (masked slots contribute exactly zero to the softmax, so the reduction is bitwise identical to the exact-prefix one), and the fresh k/v land in their slot via ggml_set_rows with the slot index read from host memory. The graph, its context, and its allocator are baked once per 128-slot bucket (FalconStepPlan) and reused for every step in the bucket; the per-step feed is a 2 KB embedding memcpy plus three scalar writes (position, slot, mask). Bucket sizes stay below the CPU flash kernel's split-KV threshold (512) as long as possible so the masked padded reduction stays in the same code path as the exact-prefix one. Measured (M4, interleaved A/B vs the per-step-build version): falcon build 90->0.7 ms, gallocr 60->0.6 ms, init 0.7->0.005 ms per generation (~3 bucket rebuilds total); same-seed output wavs bit-identical on the 380-step benchmark, metal/cpu gates and ASR round-trip pass. The per-call upload/download fallback for non-host backends is unchanged.
…ul_mat_acc The channel-fast per-tap conv ran one ggml_mul_mat per kernel tap and folded the partials in with ggml_add, so every non-first tap paid a temporary write plus the add's three-way traffic (read acc + read partial + write acc). Add a new ggml op, GGML_OP_MUL_MAT_ACC, computing acc += a * b with the result a view of acc (the ggml_cpy in-place idiom), and switch taps 1..K-1 to it. The new Metal kernel_mul_mm_acc mirrors kernel_mul_mm through the MMA (both the simdgroup and tensor variants), so the partial products are bit-identical; only the epilogue differs, folding the result tile into the destination with one F32 add per element (matching ggml_add). The simdgroup variant always stages the tile through threadgroup memory so partial tiles clip identically; the tensor variant loads the destination tile into a second cooperative tensor and adds element-wise. Add-only per open-closed: new op, kernels, encoder, pipeline getter, and supports-op cases; no existing function modified. The CPU backend gets a naive single-threaded reference forward (the op is exercised on Metal only; the per-tap path is Metal-gated). Measured (M4, interleaved A/B vs the mul_mat+add chain under residual background load): codec graph_compute_ms -~400 ms median (-~570 ms min), consistent with the ~365 ms predicted by the skip-taps probe; output wavs bit-identical across 13/13 same-seed runs, CPU backend path untouched.
…odec Add a dedicated ggml elementwise op computing x + sin^2(alpha*x)/alpha in a single pass (Metal kernel + scalar CPU reference) and route the codec's snake1d through it, replacing a 5-kernel / 11-pass elementwise chain over the largest decoder tensors. Audio8-TTS 0.6B on M4 (342-char zh, seed 1234, interleaved A/B n=8): codec.graph_compute_ms 2624 -> 2056 min-to-min (-21.7%, 8/8 pairwise), wall -669 ms; falcon fast_graph unchanged. Numerics: scalar-reference max_abs 1.9e-6 (metal sin ulp); full-utterance wav vs the chain max int16 delta 12. AUDIO8_TTS_CODEC_SNAKE_FUSED=0 falls back to the chain.
…ckend Bind loop-invariant weights (norms, conv kernel/bias, A=-exp(A_log), expanded D) and all recurrent state (conv, ssm, KV cache) to the per-token step graph as external views of their host vectors; gallocr skips tensors whose data is set externally, so the per-step upload collapses from ~300 ms/generation to ~0.15 ms. The conv/ssm next-state write-backs now run in-graph as ggml_cpy into the host state vectors (each cpy depends on the nodes that consumed the old state, so reads strictly precede writes), eliminating the per-step state read-back. A/D are resolved once per generation instead of via four tensor_get calls per layer per step (the build-time pair was dead code and is removed). Embedding/ids/pos are bound directly as well. Measured (M4, interleaved A/B under background load, fast_graph as the load indicator): falcon upload 300->0.15 ms, download 182->107 ms, compute +~170 ms (the in-graph write-backs), ar_generate -150..-350 ms; same-seed output wavs are bit-identical, metal/cpu gates and ASR round-trip pass. Non-CPU backends keep the previous explicit upload/download path.
The per-tap conv fast path still paid two transposes per conv plus a materialized bias repeat: 52 transposes + 26 repeats per decode. The ops between convs (snake, residual add) are all elementwise and hence layout-agnostic, so the whole block region -- snake, convT upsample, three residual units -- can run channel-fast [channels, frames] with transposes only at region edges (6 instead of 52). - conv_modules: extract the per-tap GEMM core; expose raw channel-fast helpers conv1d_pertap_channel_fast (bias via broadcast add, no repeat) and conv_transpose1d_col2im_channel_fast (skips the col2im path's internal transpose). Guard the module fast path on F32 weights (the per-tap weight views assume 4-byte rows). - codec: chain each decoder block through the raw helpers behind an env kill-switch (AUDIO8_TTS_CODEC_CHANNEL_FAST=0), causal pad via scaled-to-zero prefix columns + concat; falls back to the module path elsewhere. Same-seed output is byte-identical to the previous commit across repeated runs (pure data-movement change). Interleaved A/B under background load, fast_graph_ms-normalized: codec graph compute ratio 1.73 -> 1.45 (~550ms saved).
Each token's output is the sum over all simdgroups of that token's partial sums (shared_sums[t*NW + g] for g in 0..sgptg-1). The previous simd_sum(shared_sums[sgitg*NW + tiisg]) read garbage columns whenever sgptg < NW (e.g. d_state=64 -> sgptg=2) with few tokens, corrupting the SSM state. Compute the token sum redundantly on every thread instead. Split out of af7bcd4 so the fix is reviewable on its own.
Three root causes made the Falcon-H1 slow-AR path diverge from the transformers reference (first-frame argmax 3620 vs 2732, ASR round-trip said the wrong word) and blew the SSM state up on long sequences: 1. conv1d kernel was flipped at load time, but ggml ssm_conv and the HF FalconH1 decode (nn.Conv1d prefill and the cached torch.sum path alike) use the same cross-correlation orientation with window[0] the oldest frame. Feed the GGUF kernel unflipped. 2. sx (conv window) and k_r/v (fresh K/V) were read back on the host without ggml_set_output, so gallocr reused their buffers and the conv/SSM/KV states were fed garbage every step. Pin exactly those three tensors per layer. 3. The host KV cache used the current sequence length as the per-head stride while appending only the new token, so from the second token on the append overwrote the previous head blocks and every head past the first read corrupted context. The resulting residual-stream garbage - not the recurrent scan itself - drove the SSM state to 1e15..1e18 around token 180. append_falcon_kv_token now re-lays the cached tokens into the new stride before appending. Verified against an f32 transformers 4.57.6 reference forced onto the recurrent path: per-layer conv/SSM/KV states match within bf16 rounding over the whole prompt, first-frame argmax = 2732, synthesized speech round-trips through ASR verbatim, and a 605-position generation stays bounded (max state ~1.1e3, no NaN) on both CPU and Metal backends. Adds a regression test for the KV cache re-stride that fails against the previous append logic at the second token.
gqf2008
force-pushed
the
feat/audio8-tts-falcon-h1-01b
branch
from
September 4, 2026 06:26
9c852cb to
fcba3a4
Compare
…g0#393) * breeze: pack qkv and gate/up projection weights, document weight_type * ggml, qwen_decoder: fuse bf16 activation rounding into a single kernel nsys on the 2080 Ti shows the f32 -> bf16 -> f32 cast pairs behind every activation rounding point cost ~19% of GPU time on the bf16 path and ~29% on the q4_k path (144k tiny cpy kernels per 20-token run), because ggml has no fused round-to-bf16 op and the CUDA backend runs each ggml_cast as a separate kernel. Add GGML_UNARY_OP_ROUND_BF16 (CPU + CUDA implementations; HIP shares the ggml-cuda sources) that rounds f32 values to bf16 precision in one pass, bit-identical to the cast round trip (same __float2bfloat16 / __bfloat162float sequence as cpy). The qwen decoder activation cast policy gains a fused_round flag, enabled for CUDA/HIP only; Vulkan keeps the round trip. Non-contiguous views also keep the round trip, as the unary op requires contiguous input. Verified on RTX 2080 Ti with Breeze-TTS 2: generated codes are bit-identical to the round trip build in all four test cases (bf16/q4_k, fixed 100-token case and both Chinese regression prompts). RTF on the fixed 100-token case: bf16 0.760 -> 0.695, q4_k 0.484 -> 0.419; Chinese regression q4_k 0.861 -> 0.736 (short) and 0.556 -> 0.464 (long). * ggml, breeze: support row-strided inputs in fused bf16 rounding Rounding points fed by non-contiguous views (rope/cache paths) still used the cast round trip: a strided f32 -> bf16 cpy plus a contiguous bf16 -> f32 cpy, ~10% of GPU time on the q4_k path. Add a row-strided variant of the round_bf16 kernel (dst is contiguous by construction) and relax the backend/framework gates from ggml_is_contiguous to ggml_is_contiguous_rows, so those points fuse too. Codes remain bit-identical in all four test cases. RTF on RTX 2080 Ti, q4_k: 100-token 0.419 -> 0.399, Chinese long 0.464 -> 0.441; bf16 100-token 0.695 -> 0.677. * ggml-cuda: allow CUDA graphs on pre-Ampere GPUs via GGML_CUDA_GRAPHS_PRE_AMPERE Upstream disables CUDA graphs below sm_80. Keep that default, but add an env-var escape hatch so pre-Ampere behavior can be tested without recompiling. On the RTX 2080 Ti (sm_75) Breeze-TTS 2 decode the graphs do capture and replay correctly (bit-identical codes), but RTF is neutral to slightly worse (0.399 without vs 0.408 with on the q4_k 100-token case), so the upstream default stands for this workload. * ggml, breeze: generalize fused bf16 rounding to f16/bf16 inputs ggml_round_bf16 now always produces a contiguous f32 result regardless of input type (f32/f16/bf16), matching the cast round trip bit for bit: bf16 input is already rounded so the op degenerates to an exact widening, f16 input rounds through bf16 and widens, both landing on the same real values as cast -> bf16 -> cast -> f32. This fixes a HIP crash where rounding points fed by the bf16 KV cache hit an f32/f16-only assert in the unary kernel, and recovers the fusion for f16 inputs (CUDA f16 KV cache paths) that the previous f32-only gate skipped. The activation cast no longer needs per-type special cases. Verified bit-identical codes in all 8 cases (CUDA + HIP x q4_k/bf16 x 100-token + 2 Chinese regression prompts). RTF, q4_k 100-token: CUDA 0.417 -> 0.405, HIP 0.73 (unchanged); HIP q4_k vs pre-fusion baseline: 0.84 -> 0.73, long 0.92 -> 0.80, bf16 1.50 -> 1.37. * conv_transpose1d: enable col2im fast path on Vulkan The col2im path (mul_mat + ggml_col2im_1d) only ran on CUDA/HIP/Metal; Vulkan fell back to ggml_conv_transpose_1d, whose Vulkan shader is a naive per-element kernel. All ops the col2im path needs are already supported by the Vulkan backend, including col2im_1d (f32/f16 pipelines). Breeze-TTS 2 speech decoder on Radeon 8060S: 190 ms -> 98 ms; greedy output codes identical to the generic path (wav correlation 0.99998). * breeze: skip the unconditional branch when guidance_scale == 1 CFG combines logits as uncond + scale * (cond - uncond), which is exactly cond at the default guidance_scale of 1. Running the unconditional backbone there is pure waste: skipping it removes half the backbone prefill and decode work. The depth projector's logits_cfg also gets a scale == 1 shortcut that copies the conditional half directly, avoiding an inexact uncond + 1 * (cond - uncond) round trip. guidance_scale = 0 (pure unconditional) is now accepted as well. On an RTX 2080 Ti, Breeze-TTS 2 fixed 100-token case, native weights: RTF 0.705 -> 0.605; greedy output is bit-identical with and without the skip. guidance_scale = 1.5 still runs the full CFG path unchanged. * ggml-vulkan: add bf16<->f32/f16 cpy pipelines * breeze: round activations to bf16 on GPU backends to match reference The official Breeze-TTS 2 inference runs the backbone and depth decoder with bf16 activations and a bf16 KV cache. A pure fp32 AR loop drifts into degenerate trajectories on some prompts (mispronounced tokens, repetition collapse, missing EOS), so round activations to bf16 at every op boundary via the qwen decoder activation_cast policy, mirroring the reference torch bf16 semantics. CUDA/HIP use the fused round-to-bf16 op; Vulkan uses the cast round trip. KV cache stays F16 on CUDA and Vulkan: bf16 flash attention is only accelerated with native bf16 MMA (sm_80+) and is ~3x slower on older GPUs. HIP uses a bf16 KV cache like the reference. (Ported onto the perf branch; fused_round requires the ROUND_BF16 op from the preceding commits.)
* breeze: chunk the speech-encoder conv stack to bound clone VRAM The encoder graph was built at the exact reference-audio length, so conv activations grew linearly (~45 MiB/s of reference) and every new length triggered a full graph rebuild; a 60 s reference cost ~2.5 GB extra over a 6 s one. Split the encoder into two graphs. The conv stack now runs on fixed 5 s chunks (120000 samples) preceded by a 9600-sample left overlap that covers the stack's exact 5240-sample receptive field; chunk lengths are multiples of the 960x transformer stride, so no per-stage right padding occurs and the discarded overlap frames absorb the zero left pads that represent audio start in the first chunk. Stitched outputs are bit-identical to a single-pass encode of the same input (verified over 68 frames x 16 codebooks). The transformer, downsample, and projections run once over the full frame sequence at frame scale, where even minute-long references cost only tens of MiB. Measured on a 2080 Ti (Vulkan, native q8_0 GGUF, peak minus idle baseline): the VRAM slope over reference length drops from ~45 MiB/s to ~11 MiB/s (remaining slope is the frame-scale transformer graph and the longer AR prefill from reference codes), and a 60 s reference peaks ~1.4 GB lower. Encode time for 60 s improves from 3561 ms to 2197 ms. * breeze: bucket speech-encoder transformer graph capacity The transformer graph was rebuilt at the exact frame count for every distinct reference length. Round the capacity up to 125-frame (5 s) buckets so lengths within a bucket share one graph. Unused bucket frames are replicate-padded to match the downsample conv's Replicate right pad; causal attention keeps padding frames invisible to real frames. Verified bit-identical reference codes vs exact-length graphs at 6 s and 15 s; odd lengths show sub-1% last-frame diffs from flash-attention tiling, the same accepted noise class as the pre-existing length sensitivity. Single-run peak VRAM is unchanged. * ggml-vulkan, breeze: fused round-to-bf16 unary op on Vulkan Vulkan previously paid a cast round trip (f32->bf16->f32, two kernels, a bf16 intermediate tensor) at every activation-rounding point of the breeze decoder. Add a round_bf16 compute shader (f32/f16/bf16 in, always f32 out, round-to-nearest-even via the same fp32_to_bf16 bit trick the cpy shaders use), register pipelines indexed by source type, handle the widened f32 dst in the unary pipeline selection and op-support checks, and enable fused_round for Vulkan in the breeze activation-cast policy. Verified bit-identical breeze reference codes vs the cast round trip at 6 s and 15 s references. Peak VRAM on a 2080 Ti drops ~250 MiB at a 60 s reference (5491 -> 5239 MiB); no measurable change at 6 s. * ggml-vulkan: handle row-strided inputs in fused round-to-bf16 The breeze activation-rounding policy admits row-strided views into ggml_round_bf16 (ggml_is_contiguous_rows gate in qwen_decoder). The Vulkan port dispatched every input to the flat shader, which indexes the source as a contiguous array, so row-strided views read garbage and clone output degenerated into noise. Route non-contiguous inputs to a new round_bf16_strided shader built on generic_unary_head (same pattern as sigmoid_strided), keeping the flat fast path for contiguous inputs.
Merge Breeze and CosyVoice3 from dev into main
…eASR (1/2) (0xShug0#447) * ggml: add GGML_TYPE_I8_S / I2_S and five fused CPU INT8 ops * ggml-cpu: SIMD kernels for the I8_S fused ops * ggml-cpu: carry the I8_S in-band scale through dup/cont * ggml-cpu: ternary I2_S matmul kernel
…elds (0xShug0#401) * server: add /v1/audio/transcriptions/details for transcript detail fields ASR models that align words, segment speech or separate speakers report that work through TaskResult::word_timestamps, speech_segments and speaker_turns. /v1/audio/transcriptions serialises text and timing only, so for those models the alignment is computed and then discarded on the way out of the server. Rather than widen the existing response, which callers already build against, this adds an opt-in route with the same request shape. /v1/audio/transcriptions and /v1/tasks/run are byte-identical to before. The detail response is a superset of the plain one: text first, timing last, with language, segments, speaker_turns and words in between where the model produced them. Spans are sample offsets because that is what the models report, so sample_rate travels with them and is emitted only when at least one of the arrays is present. stream=true is rejected with a 400 on the detail route. The SSE response carries transcript deltas only and has nowhere to put the arrays, so accepting the request would return none of what the route exists to return. The serialisation the generic task route already performed is factored into write_transcript_detail_fields and shared, rather than duplicated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ATa5YkLUPMDPRL7w1gCo9p * docs: fix transcription details timing field --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: 0xShug0 <231717474+0xShug0@users.noreply.github.com>
* feat(package_manager): support ModelScope snapshot downloads in native manager Add modelscope_snapshot as a download kind alongside huggingface_snapshot: - schema: accept modelscope_snapshot (repo required, revision optional) - manager: ms_url() + AUDIOCPP_MS_BASE_URL override (default https://www.modelscope.cn); kind-aware revision default (master for ModelScope, main for Hugging Face) - remote info via the ModelScope file-list API (per-file Size + Sha256), cached per repo+revision; sha256 doubles as the manifest etag so size checks and inventory version comparison work unchanged; falls back to HEAD + X-Linked-Etag when the listing is unavailable - downloads reuse the existing HTTP layer (302 CDN redirects already followed); auth failure message no longer hardcodes Hugging Face - tests: ModelScope endpoints in the native manager fixture, new package_manager_modelscope_test, schema kind coverage - docs: download sources section in model_manager.md and the new kind in maintainers/model_specs.md * feat(model_manager_v2): support ModelScope snapshot downloads in Python manager Mirror the native package manager's ModelScope support in tools/model_manager_v2.py: - ms_endpoint() honors AUDIOCPP_MS_BASE_URL (default https://www.modelscope.cn); ms_url() builds /models/{repo}/resolve/{revision}/{path} - kind-aware revision default via package_revision(): master for modelscope_snapshot, main for huggingface_snapshot - remote info comes from the ModelScope file-list API (per-file Size + Sha256), cached per endpoint+repo+revision; sha256 doubles as the manifest etag so size checks and version_state comparison work unchanged; falls back to HEAD + X-Linked-Etag with unknown size when the listing is unavailable - downloads reuse the existing urlopen flow (302 CDN redirects already followed); 401/403 message is provider-appropriate; gated/HF-token handling stays Hugging Face-only - docs: note that both the native manager and model_manager_v2 support both download sources * feat(model_manager_v2): add --source/--source-repo download source override Allow redirecting any spec package to ModelScope at install/sizes time without editing model_specs: --source modelscope rewrites the download kind, --source-repo names the ModelScope repo (defaults to the spec's repo), and an unset or 'main' revision becomes 'master' while explicit revisions pass through. Errors under an override hint at --source-repo. Document the switch and the cross-source etag caveat. * fix(model_manager): scope request auth per provider Request headers were built by one shared helper that attached the Hugging Face token (HF_TOKEN / HUGGING_FACE_HUB_TOKEN / cached HF token) to every request, so ModelScope requests could leak the HF credential to modelscope.cn or an AUDIOCPP_MS_BASE_URL mirror. Both the native manager and tools/model_manager_v2.py now pick auth by provider: Hugging Face requests may carry the HF token, ModelScope requests carry only the new AUDIOCPP_MS_TOKEN (optional). The fixture server gained --hf-token/--ms-token guards that reject HF credentials on ModelScope paths, and the ModelScope package test now runs with both tokens set to prove the isolation.
…7M params, en/vi/id, GGUF FP32) (0xShug0#449) * model_specs: add the sanoTTS community family Two GGUF packages from ampixa/sanoTTS on Hugging Face: heart-nano (294,279 parameters, int8, 357 KB) as the default, and heart (2,272,145, f32, 9.1 MB). Both are converted losslessly from the blobs the project already ships. Rebuilding those blobs from the GGUF reproduces them byte for byte, and the golden gate on the rebuilt weights gives the same correlation against the float PyTorch references as the originals -- 0.989703 and 1.000000 against a 0.98 threshold -- including when the GGUF is fetched from Hugging Face rather than built locally. Session options mirror inflect_v2's, since sanoTTS needs the same external eSpeak-ng phonemizer and must not embed it. * sanotts: model spec, GGUF assets and the eSpeak-ng front end Groundwork for a ggml-native sanoTTS community model. model_specs/sanotts.json heart-nano (294,279 params, 24 kHz) from ampixa/sanoTTS on Hugging Face. Session options mirror inflect_v2's, since sanoTTS needs the same external eSpeak-ng phonemizer. assets.{h,cpp} Reads config.json and the GGUF tensors. config.json carries (tensor, offset) regions emitted by the packaging tool, so nothing here translates region names and the two cannot drift. Every shape constant is compared against the build's own, because a lineage mismatch would read weights at the wrong offsets and synthesize noise rather than fail. frontend.{h,cpp} Text -> the 62-symbol phoneme ids the model was trained on: eSpeak-ng IPA, then misaki's E2M rewrite, then the character-level vocabulary. Ported from the project's own JavaScript and Python front ends so all three agree symbol for symbol. eSpeak-ng is opened at runtime and never linked -- it is GPL-3.0 and must not be embedded here, the same treatment inflect_v2 gives it. The packaging is verified upstream: rebuilding both weight blobs from the GGUF reproduces the originals byte for byte, and the golden gate on the rebuilt weights matches the float PyTorch reference at 0.989703 against a 0.98 threshold -- including when the GGUF is fetched from Hugging Face. The inference graph is next, built on the framework's module library rather than a vendored runtime, so sanoTTS gets the shared backends like every other model here. * sanotts: ggml-native runtime, session and docs Three cached graphs (duration, token stage, frame stage + ConvNeXt decoder) with the reference implementations' exact semantics: ATen-compatible MT19937 noise, torch.linspace/expand_features float behavior, LayerNorm eps 1e-6, erf GELU, torch.istft trim, and the 0.9973-pole DC blocker. The frontend gains phonemizer-compatible punctuation preservation and the correct eSpeak-ng tie mode so token streams are byte-identical to the Python front end. Verified against the project's numpy reference (same text, seed, and eSpeak-ng build): correlation 0.999999985, identical sample count; the reference is itself gated 0.987-1.000 against float PyTorch. 38.7 s of audio renders in 0.22 s wall on CPU (peak RSS 76 MB). Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we * sanotts: bisect chunks that phonemize past the duration token limit The codepoint chunker cannot see phoneme counts, so a dense 280-codepoint chunk can exceed the duration model's 207-token training limit. encode() now throws a typed SanoTtsTooLongError and the session splits the chunk at the whitespace nearest its middle and recurses, so the shared long-form case (6 kB of text, 6.2 minutes of audio) renders instead of failing. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we * sanotts: add the heart 2.27M voice as a second package Same graph, wider and deeper; the runtime now derives the expected tensor count from the config instead of hardcoding heart-nano's 103, and the weight arena covers the 9.1 MB FP32 payload. Verified like heart-nano: correlation 0.999999985 against the numpy reference at identical sample count, installed end to end from the published Hugging Face package. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we * sanotts: piperlite lineage -- amy, hfc, kristin, vi and id voices Second graph in the family: duration and acoustic students into a 192-channel latent, then a 3-stage ConvTranspose1d decoder with dilated residual banks (kristin adds a learned post filter). Deterministic, 22.05 kHz. The shared front-end structure moves into graph_common.h; the session dispatches on the config's graph field. The piperlite front end reproduces Piper's convention exactly: untied eSpeak-ng phonemes through the phonemizer punctuation pipeline, NFD decomposition to codepoints, the per-voice phoneme_id_map with [BOS, PAD, (id, PAD)..., EOS] framing, the schwa fallback for ids outside a component's trained vocab, and regional-variant-first voice selection (phonemizer rejects bare language codes on espeak-ng >= 1.49, so 'en' must resolve to en-us in both stacks). All five voices verified against the project's numpy reference with the same eSpeak-ng build: correlation >= 0.99999996 at identical sample counts, installed end to end from the published Hugging Face packages. Vietnamese and Indonesian exercise their own espeak voices and language validation. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we * sanotts: deduplicate the two runtimes through graph_common The duration and token stages run under identical tensor names in both lineages, so their four graph builders collapse into one shared build_front_graph over a small stage spec; the frame-level acoustic stage both decoders open with becomes acoustic_frame_stage. BackendState now carries the backend/weights/CUDA-duration-mirror boilerplate once, a cached_graph template replaces the six cache accessors, and the host-side feature builders and duration rounding are shared (rounding unified to double, which is what the numpy references both runtimes are gated against actually compute). All seven voices re-verified after the refactor: correlations unchanged (>= 0.99999996) at identical sample counts; unit test and the shared long-form case pass. Net -363 lines. Claude-Session: https://claude.ai/code/session_01P1iL37FdfJkGxdGrpjH1we
… of the stream (0xShug0#454) The Higgs codec decoder is a non-causal conv stack, so the last frames of a stream are decoded against zero padding on their right instead of real context. That decodes as a rising hiss over the final ~300 ms, clearly audible on short utterances ("Hello, how are you doing?" with a cloned voice ends in a swell that is cut off). This is the same mechanism as the periodic click at chunk seams (0xShug0#429) but at the end of the stream, where the chunked path's context frames do not apply. Repeat the last frame kCodecTailContextFrames (8) times before decoding and trim the extra samples again. 16 and 32 frames give sample-identical output, so 8 is enough. decode_codes() keeps its contract (same sample count as before); the previous body becomes decode_codes_impl(). Measured on Metal (M4), Q8_0, voice clone, same seed before/after, RMS of the last three 40 ms frames and peak of the last 100 ms: short, seed 1: -39 -44 -34 dBFS, peak 3990 -> -41 -48 -46, peak 864 short, seed 2: -39 -46 -32 dBFS, peak 3951 -> -40 -47 -41, peak 1121 long (11.4 s): -48 -53 -46 dBFS -> -48 -53 -53, peak 334 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* community_models: VibeASR I8_S VAE encoder * docs: state the vibeasr/vibevoice_asr split as a decision * vibeasr: add ternary I2_S LM decoder graph, loader, and session * vibeasr: point the weights package at microsoft/VibeVoice-ASR-BitNet
(english) and [中文] tags are supported
* gguf: let a finished GGUF be re-converted Re-quantising a GGUF that audiocpp_gguf produced fails today, and the error points at the wrong thing (0xShug0#457): $ audiocpp_gguf --input ACE-Step1.5-XL-Turbo-bf16.gguf --type q8_0 --output test.gguf error: no GGUF sidecars were found ... Three things stand in the way. The conversion's namespaces come from the `namespace=` labels on the command line, so a GGUF whose tensors are already named `dit_xl_turbo_weights/...` presents as one unnamed namespace and matches nothing in its own family. With no `--family`, the catalog search then settles on whichever unrelated spec accepts a single unnamed namespace -- `sense_asr` -- and the run dies later, in sidecar embedding. When it does not die it is worse: re-converting the Kroko GGUF with any small text file beside it succeeds and writes `model_spec_family=sense_asr` into the output. And the sidecars the input already carries go unused, because collection only walks `--root`. So: read the namespaces back out of a GGUF input's tensor names, register its embedded model spec as the top-priority candidate and let it set the default family, and fall back to its embedded sidecars when neither `--root` nor `--sidecar` was given. A re-conversion also stops requiring every namespace the spec declares -- a package built with `--exclude-prefix` (an ACE-Step XL GGUF carries no turbo or base DiT) ships fewer than the spec lists, and the runtime loads it happily. docs/models/ace_step.md said building an XL GGUF "needs the other variants' safetensors on hand". It does not: the namespace check only reads the labels and `--exclude-prefix` drops those tensors before any data is touched, so a 76-byte placeholder works and the build needs 24 GB of downloads rather than 33 GB. The section now shows that, notes that only the two config.json files are genuinely required, and documents quantising the DiT alone -- q8_0's "planner sampling can fail" grade is about the planner LM, and keeping it at bf16 holds a 0.989 waveform correlation against the bf16 build at a fixed seed where a fully quantised build scores 0.09. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYQxP9KPBeGmGRRhptbQWp * ace_step: add package rows for the mixed-precision XL GGUFs `ace_step_xl_turbo_q8dit` and `ace_step_xl_sft_q8dit` install the q8_0-DiT builds next to the bf16 ones, from the same repo the bf16 rows already point at. 9.97 GiB against 14.2 GiB, and at a fixed seed the output holds a 0.989 waveform correlation with the bf16 build (0.997 on a sung take, 0.999 for XL SFT) where a fully quantised build scores 0.09. `precision` is the validated enum, so these rows carry `q8_0` — the type the conversion was run at. What the planner LM, text encoder and VAE keep is in the id and display name instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYQxP9KPBeGmGRRhptbQWp --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat: add native MiraTTS community model * feat: add MiraTTS segment streaming * test: add MiraTTS parity and performance benchmarks * perf: accelerate MiraTTS Q8 inference * perf: optimize MiraTTS CPU synthesis and stabilize Vulkan generation * fix(mira-tts): enable stable Vulkan GPU generation * fix(flashsr): support builds without OpenMP * refactor(mira-tts): keep sparse output head model-local
0xShug0#353) * feat(model): add sopro tts * feat(model): add sopro tts streaming * fix(sopro_tts): address review findings - session: seed the first segment's AR carry from the tail of the reference instead of its head, matching every later segment and the order the acoustic head concatenates in - text_tokenizer: keep the EOS marker when truncating, and take the budget from config.model.max_text_len instead of the 512 constructor default, so long segments are no longer silently cut by roughly a quarter - session: reject negative min_seconds and min_seconds > max_seconds, which left min_steps > max_steps and suppressed EOS for every segment - session: clamp style_tokens from below, symmetric with prompt_tokens - semantic_lm: bounds-check bos/eos ids before indexing the logit row - semantic_encoder: derive the resample source rate and pinned length from config instead of hardcoding 24 kHz; bit-identical for the shipped checkpoint - acoustic/vocoder/semantic_encoder/speaker_encoder: release the old graph arena before allocating its replacement, as dramabox and confucius4_tts already do - webui: add the missing min_seconds control and rebuild the bundle Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(sopro_tts): use a local kPi constant instead of M_PI MSVC does not define M_PI without _USE_MATH_DEFINES, breaking the Windows CPU build. Matches f5_tts, which documents the same trap for the same sway-time grid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(sopro_tts): port upstream band limit and boost-only reference gain Ports two upstream sopro 2.1 changes flagged in review: - Band-limit vocoder synthesis (samuel-vitorino/sopro 3d25c6f): new vocoder.band_limit_hz (default 10900), zeroing every ISTFT bin at or above the cut. Removes the high-frequency vocoder hiss, and with it the non-real Nyquist bin the head used to synthesise. - Boost-only, peak-guarded reference normalization (253a7f4): a reference already at or above the -19.8 dB prompt level is no longer attenuated, and a boost is capped at the headroom below 0.95 peak. normalize_reference now reports the level it lands on, carried on SoproReference and threaded into the output-gain fallback. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(sopro_tts): cover the reference level chain and the ISTFT band limit The reference level chain and the band limit are pure functions over plain buffers, so they can be checked without the checkpoint - which matters here because no GGUF is published and the safetensors set is a multi-file download. Eight checks over the public audio_ops surface: speech_level_db on a flat buffer and its short-input fallback, the boost-only rule (a hot reference passes through untouched), the 0.95 peak guard, the 30 dB gain limit, output_gain against the reference level, both match_gain paths, and the band_limit_bin arithmetic. Lifts band_limit_bin out of the anonymous namespace in vocoder.cpp and declares it in vocoder.h so the bin arithmetic is reachable from a test; istft_from_head now calls it. No behaviour change. Both behaviours were mutation-tested: restoring the old two-sided clamp fails the hot-reference check, and bypassing the band limit lifts the probe round trip from -63.9 to -28.8 dB across 11-12 kHz. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Rebuild WebUI bundle after rebase --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: 0xShug0 <231717474+0xShug0@users.noreply.github.com>
…xShug0#423) * Add BreezeTTS and CosyVoice3 model wiring * Normalize CosyVoice3 and Breeze GGUF package names * Add CosyVoice3 and Breeze UI support * Move Breeze voice design to design route * perf(breeze_tts): pre-allocate depth projection & generation staging buffers (0xShug0#348) * Add BreezeTTS streaming mode * Breeze-TTS 2 performance: weight packing + fused bf16 rounding (0xShug0#393) * breeze: pack qkv and gate/up projection weights, document weight_type * ggml, qwen_decoder: fuse bf16 activation rounding into a single kernel nsys on the 2080 Ti shows the f32 -> bf16 -> f32 cast pairs behind every activation rounding point cost ~19% of GPU time on the bf16 path and ~29% on the q4_k path (144k tiny cpy kernels per 20-token run), because ggml has no fused round-to-bf16 op and the CUDA backend runs each ggml_cast as a separate kernel. Add GGML_UNARY_OP_ROUND_BF16 (CPU + CUDA implementations; HIP shares the ggml-cuda sources) that rounds f32 values to bf16 precision in one pass, bit-identical to the cast round trip (same __float2bfloat16 / __bfloat162float sequence as cpy). The qwen decoder activation cast policy gains a fused_round flag, enabled for CUDA/HIP only; Vulkan keeps the round trip. Non-contiguous views also keep the round trip, as the unary op requires contiguous input. Verified on RTX 2080 Ti with Breeze-TTS 2: generated codes are bit-identical to the round trip build in all four test cases (bf16/q4_k, fixed 100-token case and both Chinese regression prompts). RTF on the fixed 100-token case: bf16 0.760 -> 0.695, q4_k 0.484 -> 0.419; Chinese regression q4_k 0.861 -> 0.736 (short) and 0.556 -> 0.464 (long). * ggml, breeze: support row-strided inputs in fused bf16 rounding Rounding points fed by non-contiguous views (rope/cache paths) still used the cast round trip: a strided f32 -> bf16 cpy plus a contiguous bf16 -> f32 cpy, ~10% of GPU time on the q4_k path. Add a row-strided variant of the round_bf16 kernel (dst is contiguous by construction) and relax the backend/framework gates from ggml_is_contiguous to ggml_is_contiguous_rows, so those points fuse too. Codes remain bit-identical in all four test cases. RTF on RTX 2080 Ti, q4_k: 100-token 0.419 -> 0.399, Chinese long 0.464 -> 0.441; bf16 100-token 0.695 -> 0.677. * ggml-cuda: allow CUDA graphs on pre-Ampere GPUs via GGML_CUDA_GRAPHS_PRE_AMPERE Upstream disables CUDA graphs below sm_80. Keep that default, but add an env-var escape hatch so pre-Ampere behavior can be tested without recompiling. On the RTX 2080 Ti (sm_75) Breeze-TTS 2 decode the graphs do capture and replay correctly (bit-identical codes), but RTF is neutral to slightly worse (0.399 without vs 0.408 with on the q4_k 100-token case), so the upstream default stands for this workload. * ggml, breeze: generalize fused bf16 rounding to f16/bf16 inputs ggml_round_bf16 now always produces a contiguous f32 result regardless of input type (f32/f16/bf16), matching the cast round trip bit for bit: bf16 input is already rounded so the op degenerates to an exact widening, f16 input rounds through bf16 and widens, both landing on the same real values as cast -> bf16 -> cast -> f32. This fixes a HIP crash where rounding points fed by the bf16 KV cache hit an f32/f16-only assert in the unary kernel, and recovers the fusion for f16 inputs (CUDA f16 KV cache paths) that the previous f32-only gate skipped. The activation cast no longer needs per-type special cases. Verified bit-identical codes in all 8 cases (CUDA + HIP x q4_k/bf16 x 100-token + 2 Chinese regression prompts). RTF, q4_k 100-token: CUDA 0.417 -> 0.405, HIP 0.73 (unchanged); HIP q4_k vs pre-fusion baseline: 0.84 -> 0.73, long 0.92 -> 0.80, bf16 1.50 -> 1.37. * conv_transpose1d: enable col2im fast path on Vulkan The col2im path (mul_mat + ggml_col2im_1d) only ran on CUDA/HIP/Metal; Vulkan fell back to ggml_conv_transpose_1d, whose Vulkan shader is a naive per-element kernel. All ops the col2im path needs are already supported by the Vulkan backend, including col2im_1d (f32/f16 pipelines). Breeze-TTS 2 speech decoder on Radeon 8060S: 190 ms -> 98 ms; greedy output codes identical to the generic path (wav correlation 0.99998). * breeze: skip the unconditional branch when guidance_scale == 1 CFG combines logits as uncond + scale * (cond - uncond), which is exactly cond at the default guidance_scale of 1. Running the unconditional backbone there is pure waste: skipping it removes half the backbone prefill and decode work. The depth projector's logits_cfg also gets a scale == 1 shortcut that copies the conditional half directly, avoiding an inexact uncond + 1 * (cond - uncond) round trip. guidance_scale = 0 (pure unconditional) is now accepted as well. On an RTX 2080 Ti, Breeze-TTS 2 fixed 100-token case, native weights: RTF 0.705 -> 0.605; greedy output is bit-identical with and without the skip. guidance_scale = 1.5 still runs the full CFG path unchanged. * ggml-vulkan: add bf16<->f32/f16 cpy pipelines * breeze: round activations to bf16 on GPU backends to match reference The official Breeze-TTS 2 inference runs the backbone and depth decoder with bf16 activations and a bf16 KV cache. A pure fp32 AR loop drifts into degenerate trajectories on some prompts (mispronounced tokens, repetition collapse, missing EOS), so round activations to bf16 at every op boundary via the qwen decoder activation_cast policy, mirroring the reference torch bf16 semantics. CUDA/HIP use the fused round-to-bf16 op; Vulkan uses the cast round trip. KV cache stays F16 on CUDA and Vulkan: bf16 flash attention is only accelerated with native bf16 MMA (sm_80+) and is ~3x slower on older GPUs. HIP uses a bf16 KV cache like the reference. (Ported onto the perf branch; fused_round requires the ROUND_BF16 op from the preceding commits.) * Breeze encoder chunked vram (0xShug0#431) * breeze: chunk the speech-encoder conv stack to bound clone VRAM The encoder graph was built at the exact reference-audio length, so conv activations grew linearly (~45 MiB/s of reference) and every new length triggered a full graph rebuild; a 60 s reference cost ~2.5 GB extra over a 6 s one. Split the encoder into two graphs. The conv stack now runs on fixed 5 s chunks (120000 samples) preceded by a 9600-sample left overlap that covers the stack's exact 5240-sample receptive field; chunk lengths are multiples of the 960x transformer stride, so no per-stage right padding occurs and the discarded overlap frames absorb the zero left pads that represent audio start in the first chunk. Stitched outputs are bit-identical to a single-pass encode of the same input (verified over 68 frames x 16 codebooks). The transformer, downsample, and projections run once over the full frame sequence at frame scale, where even minute-long references cost only tens of MiB. Measured on a 2080 Ti (Vulkan, native q8_0 GGUF, peak minus idle baseline): the VRAM slope over reference length drops from ~45 MiB/s to ~11 MiB/s (remaining slope is the frame-scale transformer graph and the longer AR prefill from reference codes), and a 60 s reference peaks ~1.4 GB lower. Encode time for 60 s improves from 3561 ms to 2197 ms. * breeze: bucket speech-encoder transformer graph capacity The transformer graph was rebuilt at the exact frame count for every distinct reference length. Round the capacity up to 125-frame (5 s) buckets so lengths within a bucket share one graph. Unused bucket frames are replicate-padded to match the downsample conv's Replicate right pad; causal attention keeps padding frames invisible to real frames. Verified bit-identical reference codes vs exact-length graphs at 6 s and 15 s; odd lengths show sub-1% last-frame diffs from flash-attention tiling, the same accepted noise class as the pre-existing length sensitivity. Single-run peak VRAM is unchanged. * ggml-vulkan, breeze: fused round-to-bf16 unary op on Vulkan Vulkan previously paid a cast round trip (f32->bf16->f32, two kernels, a bf16 intermediate tensor) at every activation-rounding point of the breeze decoder. Add a round_bf16 compute shader (f32/f16/bf16 in, always f32 out, round-to-nearest-even via the same fp32_to_bf16 bit trick the cpy shaders use), register pipelines indexed by source type, handle the widened f32 dst in the unary pipeline selection and op-support checks, and enable fused_round for Vulkan in the breeze activation-cast policy. Verified bit-identical breeze reference codes vs the cast round trip at 6 s and 15 s references. Peak VRAM on a 2080 Ti drops ~250 MiB at a 60 s reference (5491 -> 5239 MiB); no measurable change at 6 s. * ggml-vulkan: handle row-strided inputs in fused round-to-bf16 The breeze activation-rounding policy admits row-strided views into ggml_round_bf16 (ggml_is_contiguous_rows gate in qwen_decoder). The Vulkan port dispatched every input to the flat shader, which indexes the source as a contiguous array, so row-strided views read garbage and clone output degenerated into noise. Route non-contiguous inputs to a new round_bf16_strided shader built on generic_unary_head (same pattern as sigmoid_strided), keeping the flat fast path for contiguous inputs. * Attention fallback for GPUs without flash MMA kernels (sm70) Auto-resolve flash vs eager attention from CUDA compute capability: Volta/Turing (700 <= cc < 800) fall back to eager, since large prefill shapes select the MMA kernel which has no usable device code there ('flash_attn_ext_f16 has no device code compatible with CUDA arch 700'). - New engine::core::attention_fallback unit: preference parsing (per-model '<family>.attention' session option + AUDIOCPP_ATTENTION), CC gating via the CUDA driver (supports_op cannot detect this: it returns true on sm70 for shapes that later crash at launch). - Wire auto fallback + session options into higgs_audio_tts and breeze_tts (backbone, depth, encoder, decoder); process-wide AUDIOCPP_ATTENTION=eager backstop in the shared SDPA/GQA/QwenDecoder modules; trace logging of the resolved path. - Fix latent QwenDecoder prefix-concat dtype assert exposed by the eager path (cast cached prefix KV on every path, not just flash). - Unit test, breeze_tts model-spec entry, docs. * Allow Breeze attention option with older GGUF specs --------- Co-authored-by: 0xShug0 <231717474+0xShug0@users.noreply.github.com> Co-authored-by: Fraser <42195943+FraserHum@users.noreply.github.com> Co-authored-by: Laika <2432896620@qq.com>
* Buffer chunked HTTP uploads * Fix server memory guard CI test gate
ggml-hip publicly defines GGML_USE_CUDA for its consumers (hipified CUDA sources), so attention_fallback.cpp compiled the CUDA driver API probe (cuDeviceGet/cuDeviceGetAttribute) in HIP builds too, where no libcuda exists to link against. Every HIP link of the runtime then fails with undefined symbols. Define ENGINE_GGML_HIP_BACKEND on engine_core when ENGINE_ENABLE_HIP is on and use it to compile the probe out. HIP devices are named "ROCmN" and were already skipped by the runtime name check, so behavior on HIP is unchanged: the probe stays fail-open and flash attention remains enabled.
# Conflicts: # CMakeLists.txt # external/ggml/include/ggml.h # external/ggml/src/ggml.c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Companion to #333 (Audio8-TTS 0.6B): native support for the Audio8-TTS-Preview-0.1B checkpoint, whose AR stage is a Falcon-H1 hybrid — stateful Mamba2 + grouped-query attention per token — instead of the 0.6B's pure transformer.
Port
falcon_forward_step: RMSNorm → (Mamba2 ‖ GQA attention) → residual → gated FFN, stateful per token: conv/SSM states + KV cache. Mamba2 path:in_proj[z,xBC,dt]split →ssm_conv(kernel flipped to match HF causal_conv1d) →ssm_scan→ D → silu(z) gate →out_proj. Attention: q/k/v → RoPE NEOX (base 1e11) → flash attention with KV cache.lm_head_multiplierdoes not apply to the compact semantic head;embedding_multiplierapplies to (text_emb + codebook_sum) jointly.falcon_kv_cache.h+ regression test.Performance (M4, Metal, q8_0)
GGML_OP_MUL_MAT_ACC(accumulate-in-place per-tap GEMMs), and a fusedGGML_OP_SNAKE_1Dop replacing the 5-kernel/11-pass activation chain (codec compute −21.7%). 0.6B codec decode 4.8s → ~2.25s at the first step of this series; all gated work paths kept byte-identical output.ssm_scanreduction read garbage whensgptg < NW(Metal);ggml-metal.metalCRLF renormalized first for a clean diff.Verification
trust_remote_codereference (first-frame argmatch; bf16-GGUF noise stays below top-2 gap except known ties).audio8_tts_falcon_kv_cache_testcovers the KV stride bug.docs/community_models/audio8_tts_falcon_h1_status.md(root causes for the three mid-port misdiagnoses included).