[AMD][MI355X] DSv4 SGLang agentic: v0.5.14 image, router, HiCache#2146
[AMD][MI355X] DSv4 SGLang agentic: v0.5.14 image, router, HiCache#2146seungrokj wants to merge 14 commits into
Conversation
…th router and HiCache support Align the agentic launcher with the fixed-seq-len sibling: dsv4 attention backend, fp8_e4m3 KV cache, disable-radix-cache, DP-attention env exports (SGLANG_SHARED_EXPERT_TP1, DP_USE_GATHERV, etc.), and two-batch-overlap. Add SGLang router for DP-attention configs and HiCache KV offloading. Sweep conc=48 across TP8 +/- DPA +/- HiCache. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
seungrokj
requested review from
billishyahao,
chunfangamd and
yctseng0211
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
seungrokj
added
AMD
full-sweep-fail-fast
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| *) | ||
| echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: native, mooncake, lmcache)" >&2 | ||
| exit 1 |
There was a problem hiding this comment.
🟡 The default arm of the KV_OFFLOAD_BACKEND case at line 85 prints (expected: native, mooncake, lmcache), but the case above only accepts hicache — the listed values look copy-pasted from the sibling vLLM launchers (which do accept native/mooncake/lmcache) and would themselves fall into the same default arm here. Sweep configs only use hicache so CI never trips this branch, but anyone debugging by hand will be misdirected. Fix: change the list to (expected: hicache).
Extended reasoning...
What the bug is
The case "${KV_OFFLOAD_BACKEND:-}" block at lines 56-88 of benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh has exactly one accepting arm: hicache). Anything else falls into the default *) arm at lines 84-87, which emits:
Error: unsupported KV_OFFLOAD_BACKEND '<value>' (expected: native, mooncake, lmcache)
None of native, mooncake, or lmcache are recognized by this case statement — they are all backends supported by other launchers in the repo (native for kimik2.5-fp4-mi355x-vllm-agentic, mooncake for minimaxm3-fp8-mi300x-vllm-agentic, lmcache for the sibling dsv4-fp4-mi355x-vllm-agentic). The message is a copy-paste from those vLLM-family launchers.
Why the message is self-contradictory
A user who sees unsupported KV_OFFLOAD_BACKEND 'foo' and reads the parenthetical hint will pick one of native, mooncake, or lmcache — and hit the exact same default arm and the exact same message. In the case where they picked native, the output literally reads unsupported KV_OFFLOAD_BACKEND 'native' (expected: native, mooncake, lmcache), which is self-contradictory.
Step-by-step proof
- Operator invokes the launcher manually with
KV_OFFLOADING=dram KV_OFFLOAD_BACKEND=native(a natural attempt after seeing another dsv4 launcher uselmcache). agentic_kv_offload_enabledreturns true becauseKV_OFFLOADING=dram.- Bash enters the
caseat line 56 and evaluates the arms in order. The only arm ishicache). nativedoes not matchhicache), so control falls to*)at line 84.- Line 85 prints:
Error: unsupported KV_OFFLOAD_BACKEND 'native' (expected: native, mooncake, lmcache). - Line 86 exits 1.
- Operator, taking the message at face value, retries with
mooncakeorlmcache. Same path — steps 3-6 repeat.
Why existing code doesn't prevent it
The header comment at line 7 states "KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=hicache", so the canonical documented value is hicache. The error text just wasn't updated to match when this launcher was adapted from the vLLM sibling.
Impact
Cosmetic misdirection in a diagnostic-only path. The four sweep entries in configs/amd-master.yaml for this recipe all pass kv-offload-backend: hicache, so CI never hits the default arm. The cost is a minute or two of wasted debug time for someone experimenting with a non-hicache backend by hand.
Fix
Change line 85's parenthetical to (expected: hicache):
echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: hicache)" >&2On the refutations
Both refutations are duplicate-management: bug_002 and bug_004 target the same defect at the same lines with the same fix, and the refuters explicitly note the finding itself is real — they only refuted one of the two to avoid posting the same comment twice. The synthesis agent merged them into a single report, which resolves that concern.
…config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29076727890 |
…figs for initial validation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29077198891 |
…unning-requests, re-enable all sweep configs - Fix attention-backend typo (compresseed -> compressed) - Remove duplicate CUDA_GRAPH_MAX_BS block - Set MAX_RUNNING_REQUESTS to 2*CONC for subagent fan-out headroom - Use MEM_FRACTION_STATIC=0.90 and swa-full-tokens-ratio=0.10 - Add METRICS_ARGS array, guard capture_cache_metrics on metrics enabled - Remove duplicate SGLANG_ENABLE_UNIFIED_RADIX_TREE export from hicache block - Uncomment all 4 sweep configs (TP8 +/- DPA +/- HiCache) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081203249 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081361033 |
…ust sweep concurrency hicache-ratio=8 tried to pin ~495 GB via cudaHostRegister, exceeding ROCm's per-process pinnable memory limit. Lower to ratio=4 and sweep graduated concurrency levels across configs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29094196523 |
…concurrency to 96 Device KV utilization peaked at only 27% even at conc=48, and non-DPA configs underperformed DPA in both throughput and completed requests. Drop non-DPA sweep entries and extend DPA concurrency range to [16..96] (none) and [48..96] (hicache) to find the saturation point. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29108708524 |
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
please update sglang cookbook with ur current optimized recipes, sglang cookbook has support for HiCache
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29108764431 |
… conc 1-8) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29108764431 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136063462 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136075686 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136311108 |
…c hicache recipe Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136685794 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136685794 |
2 participants
Summary
dsv4-fp4-mi355x-sglang-agenticimage tolmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710dsv4attention backend,fp8_e4m3KV cache,--disable-radix-cache,--cuda-graph-max-bs, DP-attention exports,--enable-two-batch-overlap)SGLANG_CMDarray pattern with command logging and env-var dumpcapture_cache_metricsfor pre/post-benchmark cache statsconc=48across TP8 +/- DPA +/- HiCacheTest plan
dsv4-fp4-mi355x-sglang-agenticfull sweep on MI355X cluster🤖 Generated with Claude Code