[Agentx][sglang] config update#2145
Conversation
Rework the B300 agentic sglang recipe into three regimes: - DP-attention (megamoe): MegaMoE DeepGEMM MoE (--moe-a2a-backend megamoe + mega_moe env + fused shared experts + autotune), mem-fraction 0.835, swa 0.075, prefill-delayer. Rank-adjusted sizing: DEP8 chunk 65536 / cuda-graph-max-bs-decode 544; DEP4 chunk 32768 / decode 128 (effective chunk 8192 for both). Measured DEP8 conc128: 24,466 -> 33,220 tok/s/gpu (vLLM v0.23.0 reference 28,962). - TP-only low-latency (TP8 or TP4, non-DP, conc <= 16): SGLang cookbook low-latency single-node recipe with speculative decoding removed: flashinfer_mxfp4 + --enable-deepseek-v4-fp4-indexer + fused shared experts, chunked-prefill 8192, mem-fraction 0.90. - TP8 mid concurrency (32-52): unchanged flashinfer_mxfp4 baseline. Builds on #2112 (image bump to nightly-20260707). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ecipe
B200 sglang agentic (dsv4_fp4_b200_sglang.sh):
- low-latency TP-only path (DP_ATTENTION=false): mem-fraction 0.90 (was 0.88),
matching the SGLang cookbook low-latency recipe (flashinfer_mxfp4, chunked
8192, swa 0.1, no spec, GPU-only).
- DEP path now splits by concurrency via DEP_HIGH_CONC:
conc < 54 -> conservative recipe (chunked 32768, mem 0.88, swa 0.1,
NUM_MAX_TOKENS_PER_RANK 4096, --cuda-graph-max-bs)
conc >= 54 -> cookbook high-throughput recipe (chunked 65536, mem 0.835,
swa 0.075, NUM_MAX_TOKENS_PER_RANK 8192,
--cuda-graph-max-bs-decode 544, --enable-prefill-delayer).
The 8192 tokens/rank cap keeps chunked 65536 on the DeepGEMM MoE path
instead of the fp4-incompatible Triton fallback.
configs/nvidia-master.yaml:
- dsv4-fp4-b200-sglang-agentic-hicache: refresh TP/DEP conc-lists.
- dsv4-fp4-b300-sglang-agentic-hicache: refresh conc-lists.
中文：B200 sglang 智能体脚本按并发拆分 DEP 配方（conc>=54 使用 cookbook 高吞吐配方，
含 chunked 65536 / prefill delayer / mem 0.835 / swa 0.075，并将 tokens/rank 上限提到
8192，避免 fp4 MoE 回退到 Triton 导致崩溃）；低延迟 TP-only 路径 mem 提到 0.90；
更新 master.yaml 中 B200/B300 sglang 智能体的并发列表。
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Oasis-Git
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # contexts. Leave the same HBM headroom used by the B300 recipe so a nearly | ||
| # full GPU KV cache does not OOM while HiCache is spilling to host memory. | ||
| MEM_FRACTION_STATIC=0.88 | ||
| # The low-latency TP-only path (conc <= 16) runs GPU-only with no HiCache | ||
| # spill, so it can take a larger static fraction for more KV headroom. | ||
| if [ "$DP_ATTENTION" = "true" ]; then | ||
| if [ "$DEP_HIGH_CONC" = "true" ]; then | ||
| MEM_FRACTION_STATIC=0.835 | ||
| else | ||
| MEM_FRACTION_STATIC=0.88 | ||
| fi | ||
| else | ||
| MEM_FRACTION_STATIC=0.90 | ||
| fi | ||
|
|
||
| # AgentX concurrency counts live session trees, not individual requests. | ||
| # Allow subagent fan-out to exceed CONC without clipping request bursts. | ||
| MAX_RUNNING_REQUESTS=$((2 * CONC)) | ||
| CUDA_GRAPH_MAX_BS=$CONC | ||
| [ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64 | ||
| # The cookbook DEP tail captures a large decode graph (batch 544); other paths | ||
| # scale the unified cuda-graph batch with CONC (capped at 64). |
There was a problem hiding this comment.
🟡 B200 non-DP branch now sets MEM_FRACTION_STATIC=0.90 unconditionally, but the accompanying comment justifies this as "the low-latency TP-only path (conc <= 16) runs GPU-only with no HiCache spill" — the gate is only DP_ATTENTION=false, with no CONC or KV_OFFLOADING check. The paired configs/nvidia-master.yaml still contains the non-DP HiCache row { tp: 8, kv-offloading: dram, kv-offload-backend: hicache, conc-list: [8, 16, 32] }, so that whole row (including conc=32, above the stated <=16 window) will now run at 0.90 while HiCache spills, contradicting the retained "leave the same HBM headroom … so a nearly full GPU KV cache does not OOM while HiCache is spilling" comment three lines up. Either mirror the B300 gate (elif { [ "$TP" = "8" ] || [ "$TP" = "4" ]; } && [ "${CONC:-999}" -le 16 ]; then … MEM_FRACTION_STATIC=0.90 at dsv4_fp4_b300_sglang.sh:156), gate on KV_OFFLOADING=none, or drop the non-DP HiCache row from the config.
Extended reasoning...
What the bug is
In benchmarks/single_node/agentic/dsv4_fp4_b200_sglang.sh (post-PR lines ~149–161), the MEM_FRACTION_STATIC selector now reads:
# DeepGEMM's DSv4 indexer needs a multi-GiB temporary allocation at long
# contexts. Leave the same HBM headroom used by the B300 recipe so a nearly
# full GPU KV cache does not OOM while HiCache is spilling to host memory.
# The low-latency TP-only path (conc <= 16) runs GPU-only with no HiCache
# spill, so it can take a larger static fraction for more KV headroom.
if [ "$DP_ATTENTION" = "true" ]; then
if [ "$DEP_HIGH_CONC" = "true" ]; then
MEM_FRACTION_STATIC=0.835
else
MEM_FRACTION_STATIC=0.88
fi
else
MEM_FRACTION_STATIC=0.90
fiThe new comment scopes the 0.90 raise to "the low-latency TP-only path (conc <= 16) runs GPU-only with no HiCache spill". But the actual gate is only DP_ATTENTION=false — it does not check CONC and does not check KV_OFFLOADING. Every non-DP invocation gets 0.90, including HiCache-enabled and conc>16 cases.
Why the retained comment matters
The comment immediately above the new one is the previous author's explicit invariant:
DeepGEMM's DSv4 indexer needs a multi-GiB temporary allocation at long contexts. Leave the same HBM headroom used by the B300 recipe so a nearly full GPU KV cache does not OOM while HiCache is spilling to host memory.
That is exactly why 0.88 was chosen. Raising to 0.90 for the non-DP + HiCache case removes ~2% of 192 GB ≈ ~3.8 GB of HBM headroom the previous author intentionally reserved for the DSv4 indexer while HiCache spills.
Step-by-step proof from the paired config
The PR also modifies configs/nvidia-master.yaml (line ~14088) but keeps a non-DP HiCache row:
- { tp: 8, kv-offloading: dram, kv-offload-backend: hicache, conc-list: [8, 16, 32] }Trace the runner for conc=32 from that row:
- Runner sets
DP_ATTENTION=false,KV_OFFLOADING=dram,KV_OFFLOAD_BACKEND=hicache,CONC=32,TP=8. DP_ATTENTION=truebranch → false; falls into the else (line ~160):MEM_FRACTION_STATIC=0.90.require_agentic_kv_offload_backend hicacheis true →--enable-hierarchical-cacheis passed, HiCache is active and will spill.- Result: non-DP + HiCache spilling +
MEM_FRACTION_STATIC=0.90, which is exactly the case the retained comment says must run at 0.88. Additionally conc=32 exceeds the "conc <= 16" bound the new comment claims. Same failure mode for conc=8 and conc=16 in that row (still HiCache-enabled).
Pre-PR that same run used 0.88; post-PR it uses 0.90.
Cross-check against the sibling script
The companion benchmarks/single_node/agentic/dsv4_fp4_b300_sglang.sh (line ~156) does this correctly in the same PR:
elif { [ "$TP" = "8" ] || [ "$TP" = "4" ]; } && [ "${CONC:-999}" -le 16 ]; then
…
MEM_FRACTION_STATIC=0.90So the author knew the correct pattern — gate 0.90 on the low-latency window and keep 0.88 for the general non-DP path — it's just missing on B200.
Fix
Any one of these resolves the code/comment/config three-way inconsistency:
- Mirror the B300 gate on the B200 non-DP branch: only raise to 0.90 when
{ TP=8 || TP=4 } && CONC <= 16(or equivalently gate onKV_OFFLOADING=none); keep 0.88 for non-DP + HiCache. - Drop the non-DP HiCache row
{ tp: 8, kv-offloading: dram, kv-offload-backend: hicache, conc-list: [8, 16, 32] }fromconfigs/nvidia-master.yaml, so the 0.90 non-DP path never coincides with HiCache spill. - If 0.90 has been empirically validated safe for non-DP + HiCache on B200 (with the DSv4 indexer's multi-GiB indexer alloc), delete the retained "leave the same HBM headroom … while HiCache is spilling" comment and reword the new one so it no longer claims "conc <= 16 … no HiCache spill" as the justification.
Severity note
Marking nit because the ~3.8 GB delta may or may not trigger an actual OOM at long-context HiCache spill in practice — full-sweep-fail-fast validation will catch it if it does. But the code violates its own documented invariant, and the B300 mirror in the same PR shows the intended shape; the fix is a two-line gate change.
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
cquil11
added
the
agentx
ChatGPT Codex Connector
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
2 participants
No description provided.