Conversation
将 DeepSeek-V4-Pro MI355X ATOM AgentX 的 c48 及以上测试切换到本地验证过的原生 RCCL DEP 配置，并保持低并发 TP 测试不变。 同步固定的 post-merge ATOM 镜像、EP8 元数据、关闭 TBO/EPLB、真实 MTP 接受率以及本地验证过的路由和 AIPerf 参数。
将 DeepSeek-V4-Pro RCCL DEP 性能变更记录中的占位链接替换为实际的 InferenceX PR 链接。
精简 DeepSeek-V4-Pro AgentX RCCL DEP 配置，移除与 CLI 或公共默认值重复的环境变量，并补齐本地验证使用的 3600 秒 warmup grace。
Restore the request-equivalent weight, prefill delayer, and decode interval requested for the AgentX run. Remove the newly introduced terminal MTP overrides while keeping the rest of the cleanup unchanged.
Keep the 3600-second agentic warmup allowance, but rely on the server keep-alive setting and AIPerf default benchmark grace period.
Remove the DEP-only 32K override so both TP and DEP retain the original 8192-token state checkpoint interval.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5c91ae8. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding, I also checked the dropped AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID export in the DEP branch — the surrounding comment now only describes the dynamo-specific session-id header for DPA routing, so removing the generic fallback appears to be an intentional simplification rather than a regression.
Extended reasoning...
One confirmed finding (perf-changelog.yaml claiming zero request equivalence and a 32768-token checkpoint interval that contradict the script's ATOM_DP_LB_REQ_EQUIV=512 and hardcoded STATE_CHECKPOINT_INTERVAL_TOKENS=8192) is already queued for posting as an inline comment, so it is not restated here. I independently reviewed the script diff, confirming this discrepancy is real (STATE_CHECKPOINT_INTERVAL_TOKENS=8192 is set unconditionally at line ~57 with no path to 32768, and ATOM_DP_LB_REQ_EQUIV=512 is set explicitly for the DP_ATTENTION branch). Given a verified finding exists, approval is not appropriate. I additionally examined the other ruled-out candidate — the removal of AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID alongside AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID — and concluded it looks intentional: the updated comment describes only the dynamo-specific session header path for the new DEP transport, with no remaining reference to a generic fallback, consistent with dropping the redundant export. This is worth surfacing briefly as it was a real candidate concern that a reviewer might otherwise re-raise, but does not need extensive discussion. No new top-level architectural or security concerns were found beyond what's already captured in the inline finding.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
4 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34388489723 |
为 DSV4-Pro 的 DPA/DEP 路径补齐小 batch CUDA graph，避免非二次幂 decode batch 向上填充造成的 attention、MoE 和 RCCL 无效计算。保留 TP 路径默认行为，并为高并发配置保留大 batch graph。
Dense DEP decode graphs / DEP decode 稠密图优化Added commit On the current PR configuration (8K state checkpoint, default scheduler controls), a controlled C64 900-second A/B changed only the graph ladder and kept synthetic acceptance at 2.49:
The official C64 server log shows that 2,140 of 3,270 scheduler samples (65.44%) have non-power-of-two running-request counts, most commonly 5/6/7, so the gain matches the observed padding mechanism. Cost: about +13 seconds startup and +0.12 GB graph memory per GPU. Correctness was also checked with real MTP on all 1,319 GSM8K samples: flexible/strict exact match = 95.15% / 95.22%, with 65.95% MTP acceptance; no accuracy regression versus 94.69% / 94.84% before the graph change. 提交 在当前 PR 配置（8K state checkpoint、默认调度参数）上做了 900 秒 C64 隔离 A/B，唯一变量是 graph ladder，synthetic acceptance 同为 2.49：
官方 C64 日志的 3,270 个调度采样中，有 2,140 个（65.44%）是非二次幂 running-request 数，最常见为 5/6/7，和减少 padding 的机制一致。代价约为启动增加 13 秒、每张 GPU graph 显存增加 0.12 GB。 精度也使用真实 MTP 跑完 1,319 道 GSM8K：flexible/strict exact match 为 95.15% / 95.22%，MTP acceptance 65.95%；相对改动前 94.69% / 94.84% 没有精度回退。 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34965660977 |
| mkdir -p "$RESULT_DIR" | ||
| python3 "$(dirname "$0")/check_dsv4_dspark_checkpoint.py" \ | ||
| --model-path "$MODEL_PATH" --revision "$DSV4_MODEL_REVISION" \ | ||
| --output "$RESULT_DIR/checkpoint_preflight.json" |
There was a problem hiding this comment.
what is the purpose of this ?
There was a problem hiding this comment.
This is a fail-fast checkpoint integrity and provenance check. MODEL_PATH may point to a pre-populated shared cache, while the download logic only verifies that the directory is non-empty. Before allocating GPUs, this verifies the pinned DeepSeek-V4-Pro-0813 config/index, required DSpark heads, all 66 shard boundaries, and tokenizer readability. The resulting JSON is also included in runtime_manifest.json, so the exact checkpoint used by the benchmark is auditable.
中文
这是一个 checkpoint 完整性快速失败及运行溯源检查。MODEL_PATH 可能指向预先填充的共享缓存，而下载逻辑只能确认目录非空。它会在分配 GPU 前检查固定的 DeepSeek-V4-Pro-0813 config/index、必要的 DSpark heads、全部 66 个权重分片边界以及 tokenizer 可读性。生成的 JSON 也会写入 runtime_manifest.json，用于记录 benchmark 实际使用的 checkpoint。
Restore run-sweep.yml to the PR base and replace the temporary rocm/atom-dev:pr2233-4f3a808 image with official nightly_202609161445, which includes the merged ROCm/ATOM#2233 inference-mode fix. 将 run-sweep.yml 还原到 PR base，并把临时镜像 pr2233-4f3a808 换成 包含已合入 ATOM#2233 修复的官方 nightly_202609161445。
There was a problem hiding this comment.
I reviewed this PR and didn't find any new bugs. My earlier note (the changelog's "32768-token"/"zero request equivalence" claims not matching the actual 8192-token interval and ATOM_DP_LB_REQ_EQUIV=512) has since been corrected via append-only entries in perf-changelog.yaml, and the CHANGES_REQUESTED objection about pinning to the temporary rocm/atom-dev:pr2233-4f3a808 image has been addressed by the latest commit switching to the official rocm/atom-dev:nightly_202609161445. Given the size of this change and that it touches speculative-decoding rollout logic, a hash-guarded runtime patch to AITER internals, and new checkpoint/shard validation, a human look is still worthwhile as an additional step.
What was reviewed:
- perf-changelog.yaml correction entries (the DSpark image-bump and checkpoint-interval entries) now correctly state the 8192-token interval and request-equivalence 512, resolving my prior finding.
- configs/amd-master.yaml confirms the pin to the official nightly image rather than the temporary PR-built one, addressing the functionstackx CHANGES_REQUESTED thread.
- This run's other candidates (DEP session-affinity env var, DPA8 sharding flag, benchmark_lib.sh tokenizer-arg quoting) were examined and not confirmed as bugs.
Extended reasoning...
Overview
This PR migrates the MI355X ATOM AgentX DeepSeek-V4 recipe from MTP to DSpark K6 speculative decoding, bumps the model checkpoint to DeepSeek-V4-Pro-0813, updates the ATOM image multiple times (ending on an official nightly), adds a new checkpoint/shard preflight validation script, changes the KV-cache precision split across concurrency bands, and appends a long series of corrective/updating entries to perf-changelog.yaml. It also at one point carried a runtime source patch to AITER internals guarded by exact-source hashing, later removed once the fix landed upstream in the pinned image.
Security risks
No injection, auth, or credential-exposure concerns were found in this diff. The AITER runtime patch (now removed in the final state per the changelog) mutated third-party source inside the container guarded by hash checks — a reasonable safety mechanism, and it has since been dropped in favor of the fix landing in the base image.
Level of scrutiny
This is a large, non-mechanical change (205 lines in the main benchmark script, a new 119-line Python checkpoint-validation script, spec-decoding-method changes, and KV-precision policy changes across concurrency bands) touching a benchmark/perf pipeline rather than production auth/crypto code, but its correctness materially affects reported throughput/accuracy numbers that downstream engineers rely on. That warrants defer rather than approve, consistent with the guidance against approving large/complex changes.
Other factors
The previously outstanding CHANGES_REQUESTED review (functionstackx, image pin) was addressed by a subsequent commit switching from a temporary PR-built image to the official nightly build, and my own earlier changelog-mismatch finding was corrected via append-only entries — both checked directly against the current diff. No new bugs were reported by this run's bug hunt, and the four candidate issues considered were not independently confirmed as bugs by me beyond what's already in the ruled-out list.
Keep this PR's DSpark DEP recipe, official nightly image, and 0813 HF-cache mount. Take main's AgentX tokenizer path (--tokenizer $MODEL). Restore perf-changelog.yaml from main and re-append this PR's entries at the tail. 将 main 合入本分支：保留 DSpark DEP 配方与官方 nightly；tokenizer 跟随 main； changelog 以 main 为底，再把本 PR 条目接到末尾。
The main merge left the first PR #2912 changelog entry without its - config-keys: header, so process_changelog could not parse the added YAML. 修复合并后第一条 2912 changelog 条目缺少 - config-keys: 导致的 YAML 解析失败。
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
@seungrokj @functionstackx @billishyahao could you plz help reveiw & merge? I have cleaned up review comments |
将两份 CI 流程文档恢复为 main 版本，使其不再出现在本 PR 的变更中。
将中英文 CI 流程文档恢复为 PR 当前基础版本，确保它们不再出现在本 PR 的变更中。
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
- recipe at https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-V4-Agentic-InferenceX.md
- is not mutating model config.json nor atom configuration.
InferenceX/benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.sh
Lines 120 to 209 in 5ac385d
Signed: seungrokj
❌❌❌ REJECTED ❌❌❌@seungrokj two blockers at the signed-off commit ❌ Check 2 (Evals pass on the in-PR run): FAIL — the eval used a different inference-engine image than this PR's config. The only executed green sweep/eval on an in-PR commit, run 34965660977 (commit ❌ Check 4 (Reuse command posted): FAIL — No authorized reuse command has been posted on this PR. The current issue comments contain no Passed and not applicable checks✅ Check 0 (Sign-off author is a CODEOWNER): PASS — ✅ Check 1 (Passing sweep on an in-PR commit): PASS — commit ➖ Check 3 (Recipe linked, merged, complete): N/A — the entry is ✅ Check 5 (Latest checklist template): PASS — all 15 items of the current template are present and checked. ✅ Check 6 (Upstream images, engine-first ordering): PASS — the entry is ✅ Check 7 (No deprecated models/scenarios): PASS — ✅ Check 8 (No architecture-changing hacks): PASS — no ✅ Check 9 (Spec-decode uses chat templates): PASS — the agentic replay goes through ✅ Check 10 (No engine patches): PASS — at ✅ Check 11 (Agentic spec-decode uses golden AL): PASS — throughput points pin ➖ Check 12 (Append-only changes): N/A — the new Assessed commit: |
4 participants
Summary
Enable DSpark K6 at every MI355X ATOM AgentX concurrency using
deepseek-ai/DeepSeek-V4-Pro-0813revision72e1d3230f6c080a530b0a1d46f8eb4602340597. All throughput points use golden AL 3.77; full C256 GSM8K uses real draft verification. These DSpark parameters match the SGLang reference. Preserve this PR's TP/DEP bands and GPU KV policy.rocm/atom-dev:nightly_202609121454, digestsha256:972e37f6e0f7eb12de83188b07709ed0578a2eb4aebc20fbdcfcec562ca8d60c.c61fbfb0e5fabd478978ba04801fb80028240176contains ATOM#2190; AITER4676838b5607542585a9a04d19f61b6055d341facontains AITER#5467 and AITER#4159. FP8 KV does not exercise #4159's BF16 FlyDSL paged SWA branch.--method dspark --num-speculative-tokens 6means six draft tokens and target verify length 7. Confidence schedules and ragged verification remain disabled.ATOM_DP_LB_REQ_EQUIV=512.1..16,32,48,64,128, plus256,512for C96+. Verify fresh target q7 and DSpark draft capture in runtime logs.draft_modelmetadata to the existing recipe and mount shared HF cache for 0813. Pin server and AgentX tokenizer to the same snapshot. Before GPU startup, check exact config/index identity, DSpark heads, tokenizer and all 66 shard headers/payload boundaries. This does not hash entire weight files.torch.inference_mode()before updating preallocated inference buffers. This fixes q7 padding through compiled custom ops while retaining communication fusion and graphs; DEP does not use this backend. Record original/patched hashes inaiter_runtime_fix.jsonand the runtime manifest. TP AITER source is intentionally dirty only for this recorded fix.checkpoint_preflight.json,runtime_manifest.jsonandserver_command.txt. Retain the CI priority-classifier correction andnodes:1scheduling contract.Sweep
Current repaired sweep 34828203043 runs commit
0bb90e0be1da007ab433d52c26ec63d86397238eand requests the complete 10+1 matrix. The preceding sweep was cancelled after TP C2/C4/C8 and local C16 reproduced the same startup failure; its completed C256 eval is retained as evidence for the unchanged DEP path. All 11 GPU jobs havenodes:1and priority3.500; they are currently queued.Validation
main.MTP baseline 34768147457, attempt 4 passed ten throughput points and all 1319 GSM8K samples (strict 96.51%, flexible 96.44%). C2/C4 power was invalid; C256 had a post-profile drain timeout. This is baseline evidence only. Both checkpoint and acceptance policy change, so comparison measures the complete migration rather than an isolated DSpark speedup.
中文
MI355X ATOM AgentX 的全部并发切换为固定 revision 的 DeepSeek-V4-Pro-0813 和 DSpark K6/q7。十个性能点统一使用 golden AL 3.77；C256 全量 GSM8K 不传强制接受率。DSpark 参数对齐所链接的 SGLang 任务，保留本 PR 的 TP/DEP 分档与 GPU KV 策略。
保留已核实的固定镜像及 ATOM#2190、AITER#5467/#4159、原生 RCCL DEP、dense FULL graph 和现有 KV/调度配置；FP8 KV 不启用 #4159 的 BF16 FlyDSL 分支。修复
draft_model路由和 0813 共享缓存选择，统一服务与 AgentX tokenizer snapshot。GPU 启动前检查 config/index 身份、DSpark head、tokenizer 和全部分片；不计算完整权重文件哈希。前一轮远程 TP C2/C4/C8 和本地 C16 均复现 AITER inference tensor 写入错误，该轮已取消。现已提交带源码哈希保护的 TP Stage2 callback InferenceMode 修复，保留通信融合及 graph，并启动完整的新一轮；DEP 不使用这一 TP backend。三项固定镜像回归测试通过；修复后的 TP C16 和 DEP C256 本地 smoke 均已通过，分别完成 32/32 和 512/512 个两轮请求，以及各 16/16 道 GSM8K，零请求错误、无截断。所有 q7 target graph 和 DSpark draft warmup 均完成，服务正常退出，8 张本地 GPU 已恢复空闲。当前提交、任务范围和调度标签见上表及链接。Bash/Python、YAML、changelog、矩阵、固定镜像 checkpoint 预检、四组真实参数解析、路由/tokenizer 和现有 eval 测试均已通过。远程 C256 全量 GSM8K 已通过，成绩见上文；已逐项核实全部 1,319 道题、固定 snapshot、K6、真实接受率验证、22 档 dense FULL capture 及同阶梯的 DSpark draft warmup，固定镜像默认启用 draft graph。新一轮 11 个正式任务已全部生成并带有
nodes:1、priority3.500，当前排队中。旧 MTP sweep 仅作基线，不作为 DSpark 的通过证据。该基线的功耗无效项和采样后收尾超时见上文；本次同时改变 checkpoint 和接受率口径，比较结果应解释为整体迁移收益。