Skip to content

perf(glm): skip unused NextN prefill computation - #1708

Merged
shihaobai merged 1 commit into
mainfrom
bsh/glm-prefill-nextn-cache-only
Nov 12, 2026
Merged

shihaobai merged 1 commit into
mainfrom
bsh/glm-prefill-nextn-cache-only

Conversation

@shihaobai

@shihaobai shihaobai commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

GLM NextN draft Prefill can discard the draft output after its cache state is committed. This change keeps the shared BaseModel flow unchanged and moves the optimization into the GLM MTP model:

  • The draft-specific Prefill layer writes MLA KV, indexer K-pools, and the raw tail directly, then returns a zero-row placeholder.
  • The draft post layer converts that placeholder into an empty output, skipping the unused Q-B projection, sparse attention, FFN, final hidden collection, and logits.
  • EAGLE uses the cache-only path for all draft steps. Vanilla With-Att uses it for a single chained draft; multi-level Vanilla still runs the intermediate outputs needed by the next draft.
  • The same writer handles BF16/FP8 layouts, TP/TPSP, DP prefill balance, mixed prefill, and overlap execution.

Cache-only writes are byte-exact with the full writer across HOLD rows, short and 8K prompts, long-prefix/pool-boundary requests, and mixed request layouts.

@shihaobai
shihaobai force-pushed the bsh/glm-prefill-nextn-cache-only branch from 7e8788d to 4a5ebb7 Compare November 12, 2026 15:31
@shihaobai
shihaobai merged commit d69f6d5 into main Nov 12, 2026
1 check passed
@shihaobai
shihaobai deleted the bsh/glm-prefill-nextn-cache-only branch November 12, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant