Skip to content

perf(glm): write MLA projections directly in token-major layout - #1710

Merged
shihaobai merged 2 commits into
mainfrom
bsh/glm-prefill-mla-token-major-bmm
Nov 12, 2026
Merged

shihaobai merged 2 commits into
mainfrom
bsh/glm-prefill-mla-token-major-bmm

Conversation

@shihaobai

@shihaobai shihaobai commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

GLM sparse MLA BMMs produce head-major outputs, requiring layout copies before prefill attention and when flattening multi-token V outputs. Write Q prefill and V prefill/decode results directly into token-major buffers through the existing bmm(out=...) interface, with explicit dimensions at each call site.

Validation on H200:

  • 86 related GPU tests pass, covering BF16/FP16, 16/64 heads, padded noncontiguous inputs, guard rows, CUDA graph replay, and prefill/decode output projections.
  • With real GLM-5.3-Flash layer-3 BF16 weights, decode V projection plus head flattening takes 16.61 → 12.90 µs at batch 32 and 22.75 → 15.68 µs at batch 128 (64 heads, five alternating AB/BA rounds). Batch 1 remains approximately 12 µs. These measurements cover projection and layout; full-model throughput was not measured.
  • Repository-pinned Black, Flake8, and whitespace checks pass.

@shihaobai shihaobai changed the title perf(glm): write prefill MLA projections in token-major layout perf(glm): write MLA projections directly in token-major layout Nov 12, 2026
@shihaobai
shihaobai merged commit d60f6fc into main Nov 12, 2026
1 check passed
@shihaobai
shihaobai deleted the bsh/glm-prefill-mla-token-major-bmm branch November 12, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant