Conversation
Port upstream PR handy-computer#162 which reduces memory usage across all model families by bounding intermediate tensors and trimming padded rows in-graph. Conformer infrastructure: - Add inplace_pre_encode and pre_encode_dw_time_chunk to ConvPolicy - Inplace conv bias + relu, chunked depthwise convs for bounded activation memory - Apply to canary_qwen and cohere encoders GigaAM: - Query-tiled attention for long sequences (T > 2048), tiling only the query axis to bound the O(T^2) score matrix Granite-NAR: - Replace host-side BPE-CTC pool+decode with graph-based bounded projection (512 windows per chunk), avoiding a full-utterance [vocab, T_enc] tensor Moonshine Streaming: - F16 sliding-window masks (no F32->F16 cast at graph build) - Per-layer mask deduplication (tiny: 6 layers share 2 unique masks) - Hard 81.92s input limit from 4096-row adapter position table - Input limit check in run()/run_batch/stream_feed - KV cache respects n_ctx cap Qwen3 ASR: - Removed encoder key-pad mask (build_cu_seqlens_mask, mask_in) - Bounded chunked subsample (32-chunk batches) replaces inline conv - KV cache init uses qwen3_context_ceiling instead of hardcoded 2048 Transcribe.cpp: - Update max_kv_bytes comment for sessions that default to F32 KV Tests: - Rename moonshine batch_truncation -> input_limit (tests adapter position table gate, not output truncation) - GigaAM smoke: 45s -> 90s (crosses query-tiling threshold) - Update qwen3_asr truncation test comment Docs: - Document moonshine_streaming 81.92s adapter-position limit
|
Couple notes from another session: Step graphs are built once at max_n_kv, and attention reads the full window every step even when barely filled (causal_lm.cpp:420). The fix already exists: run_batched_encdec_step_loop::ensure_window() (transcribe-batch-util.cpp:254) doubles the window and rebuilds the step graph mid-decode — but it's only wired into the enc-dec batch path (canary/cohere/moonshine). Port it to causal_lm::run_batched_step_loop and the single-utterance GPU paths. Measured, qwen3-0.6b / 4750U / 8 threads: per-step ≈94 ms at max_n_kv=8192, of which the KV read (896 MiB) is ≈32 ms. Halving the effective window ≈ 17%/step. Pure perf, no correctness impact. Buffer is sized once per run; overrun = truncation. Realloc+copy on overrun would allow sizing budgets below the safe max(floor, n_audio) bound. Only worth doing if measurement shows that bound is too generous. Complication: kv_init is layer-major with position in the middle, so growing n_ctx is n_layer × n_batch strided copies, not an append (~400 MiB / tens of ms for a 5-min qwen3 clip, once). |
This branch has not been deployed
1 participant
NOTE THIS MAY BREAK MODELS, NOT SUITABLE FOR MERGING YET
Some notes from my 4750U machine
Granite Speech 4.1 2B NAR — Unrestricted run caused system OOM; isolated run exhausted 8 GiB RAM + 2 GiB swap and was killed. Clear usability failure.
Moonshine Streaming Tiny — ~6.3 GiB RSS for a 48 MiB model before aborting; explicit streaming still aborted at ~677 MiB. Extreme amplification plus a likely indexing/streaming bug.
Qwen3-ASR 0.6B — ~7.6 GiB RSS, then hit its generation/context cap. Very high for an ~811 MiB model; likely duration-dependent growth.
GigaAM v3 RNNT — ~4.0 GiB RSS for a ~261 MiB model. Roughly 15× model size and strong growth with duration.
Multitalker Parakeet — One-shot: 4.45 GiB; explicit streaming: 1.26 GiB. Large mode-dependent difference; perhaps avoid or bound the one-shot path.
Canary-Qwen 2.5B — CPU remained at ≥8 GiB; Vulkan ~8.17 GiB after this PR. Still unsuitable for many 6–8 GiB GPUs.
Cohere Transcribe — ~7.5 GiB CPU, 4.34 GiB Vulkan. High absolute requirement, although its 2.25 GiB weights explain part of it.
Granite 4.0 1B Speech — ~5.6 GiB RSS, with an isolated cgroup peak around 8 GiB. High total working set relative to its 2.4 GiB weights.
SenseVoice Small — ~1.2 GiB for a ~139 MiB model.
MOSS Transcribe-Diarize — ~2.7 GiB and extremely slow on the five-minute recording.
Granite TurboCTC — ~1.2–1.4 GiB depending on quantization.
Moonshine Tiny — ~697 MiB before hitting its generation cap.
CJ Notes
Introduced slop limit on moonshine streaming