Repository navigation
Conversation
…(STRATA_PF_MMVQ, gfx906)
…re latched (no mid-prompt free/sync)
KevinX8
force-pushed
the
gfx906-pf-mmvq
branch
from
October 28, 2026 09:53
5507f38 to
a834032
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result: see #4157. Agent turns of 100-256 new tokens on gfx906 spend most of the batched pass dequantizing the dense projections' GGUF blocks and running small rocBLAS GEMMs. With
STRATA_PF_MMVQ=1those chunks use decode's native MMVQ on the blocks instead: a 165-token turn on an MI50 goes from 1.83 to 1.57 s (two passes each); larger chunks are unchanged.Related discussion: #4157 (scope question open; draft until it is answered).
Approach:
Gemm::native_mmvq(src/prefill/gemm.cu, declared ininclude/strata/prefill/gemm.hpp), called first inGemm::native, reusingnative_mmvq_supported()/native_quantize_q8_1()/native_mmvq()frominclude/strata/kernels/native_mmvq.hpp. The quantizer and its byte count take 1..8 columns; Q8_1 blocks are per 32 values with column j atj * K / 32, so a slice is quantized as one column ofn * Kvalues (byte-identical; the quantizer itself runs one flat kernel overn_in * ncols). One 16 MiB workspace perGemm, allocated at first use, never grown (no mid-prompt free/sync); a failed allocation turns the path off for thatGemm. Off unlessSTRATA_PF_MMVQ=1; with it unset the cost is one static bool test pernative()call. Based on main v0.2.7.1; the gfx906 build needs #3986 to compile at all (dflash2.cpp), so this branch was built with #3986 applied under it (not part of this PR).Validation:
tests/hip/pf_mmvq_parity.cpp(registered in theSTRATA_HIP_GFX906test block), this branch on v0.2.7.1 + [Engine] gfx906: fix the build of src/core/dflash2.cpp #3986, MI50, ROCm 10.1: Q4_K, Q6_K, IQ4_XS, Q8_0 at T = 1 / 7 / 8 / 129 / 512 against a host double product on the dequantized weights, one row x1000 — rel RMS 5.3e-3 to 6.3e-3, max 4.7e-3 to 9.0e-3 per token row; ldy = N + 16, beta = 1, T above the threshold, K = 640 and BF16 are declined and leave Y untouched; device memory does not change between a 1-token and a 512-token call.cteston the gfx906 build, 150/157. The 7 failures are the same with the flag unset and do not touch this path:mmvq_il_parity,verify_parity(CUDA-only kernels),pf_wmma_parity(RDNA WMMA),ple_parity(Q2_0 fixture not present),split_lend_complement_test,platform_memory_test,isa_guard_check.serve/server.pywith the switch on: a 30-minute soak of one growing coding conversation (276 turns, context to 76k), two turn benchmarks and a 118k-token fresh read — no verify timeouts, 910 MiB VRAM free after load. Decode unchanged (44.1 -> 45.5 tok/s short, 31.1 -> 30.7 at 118k depth).Limits: measured on one gfx906 card; the 256 default rests on the 165- and 480-token points of one sweep (break-even ~290 by interpolation). Not covered by the test yet: Inf/NaN rows, a K large enough to hit the 16 MiB slice cap, an
ldx = K + 64decline. Below the threshold the result depends on the chunk size (Q8_1 activations).