Repository navigation
[NVIDIA][AgentX] Update GB300 DSV4.1-Flash disagg topology, caching and routing / [NVIDIA][AgentX] 更新 GB300 DSV4.1-Flash 分离式拓扑、缓存与路由 - #4303
Conversation
中文：将 changelog 条目的 pr-link 设置为 #4303。
…20 nightly The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains sgl-project/sglang#44580 and #45062, so the setup script now applies only the external linker pools patch (#44623). Add the #4303 changelog entry on top of the merged #4180 entry.
9b7d62b to
974edc5
Compare
weireweire
changed the title
… nightly with the Mooncake linker Move dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg to lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089 (multiarch digest pinned) and the ai-dynamo 1.6.0.dev20261018 it ships, with the ZMQ event plane and no NATS service. - c1228 keeps 4x DEP2 prefill + 1x DEP16 decode; c2884 moves from 6 to 5 DEP2 prefill workers (3 prefill nodes, 26 GPUs) and caps prefill batches at 32768 tokens. - Both roles use the flashmla_mega attention backend, local DP control broadcast and the Mooncake external cache linker with a Mooncake master service (high watermark 0.90, client TTL 60 s, 64 RPC threads). The linker replaces prefill HiCache, so the master rows declare Mooncake 0.3.13.post1 as the KV offload backend. - Decode adds late admission, stream interval 32, polling interval 8, 2048 SWA prefix tails and decoder SWA bounded replay; the NCCL DP-batch sync env is dropped because the image exchanges MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622). - A setup script applies four SGLang patches the image lacks, with git apply under a lock, idempotently, failing on any hunk that does not apply: sgl-project/sglang#44623 (open), #44580 (merged as 834bfe4ef0), #45062 (merged as 06342acad2) and decode late admission (weireweire/sglang@407e912a1e). - Revise this PR's changelog entry to the final state. 中文：将 dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg 迁移到 lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089（固定 multiarch digest）及镜像自带的 ai-dynamo 1.6.0.dev20261018，使用 ZMQ event plane， 不再启动 NATS 服务。 - c1228 保持 4 个 DEP2 prefill + 1 个 DEP16 decode；c2884 的 DEP2 prefill 由 6 个减为 5 个（3 个 prefill 节点，共 26 个 GPU），并将 prefill 批次 上限设为 32768 token。 - 两个角色均使用 flashmla_mega attention 后端、本地 DP control broadcast 和 Mooncake external cache linker，并启动 Mooncake master 服务（高水位 0.90，client TTL 60 秒，64 个 RPC 线程）。linker 取代了 prefill HiCache，因此 master 配置将 KV offload 后端声明为 Mooncake 0.3.13.post1。 - Decode 新增 late admission、stream interval 32、polling interval 8、 2048 SWA prefix tails 与 decoder SWA bounded replay；镜像已通过 gloo gather + broadcast 交换 MLP 同步数据（sgl-project/sglang#44622），因此 删除 NCCL DP batch 同步环境变量。 - 新增 setup 脚本，在锁保护下用 git apply 幂等地应用镜像缺少的四个 SGLang 补丁，任一 hunk 无法应用即失败：sgl-project/sglang#44623（未合并）、 #44580（已合并为 834bfe4ef0）、#45062（已合并为 06342acad2）以及 decode late admission（weireweire/sglang@407e912a1e）。 - 将本 PR 的 changelog 条目更新为最终状态。
…isagg points A same-time A/B on the 20261019 nightly showed no effect from decode late admission: P90 interactivity 230/232 without it and 230/228 with it at c1228. The image already exchanges the scheduler's MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622) and avoids blocking Engram history copies during PD admission (#44787). Remove the patch, its setup script entry and SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION. 中文：在 20261019 nightly 上的同时段 A/B 显示 decode late admission 没有 效果：c1228 下关闭时 P90 interactivity 为 230/232，开启时为 230/228。镜像 已通过 gloo gather + broadcast 交换调度器的 MLP 同步数据 （sgl-project/sglang#44622），并避免了 PD admission 期间阻塞的 Engram 历史拷贝（#44787）。因此删除该补丁、setup 脚本中的对应条目以及 SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION。
…20 nightly The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains sgl-project/sglang#44580 and #45062, so the setup script now applies only the external linker pools patch (#44623). Add the #4303 changelog entry on top of the merged #4180 entry.
…96 at 7P1D With the Mooncake linker, three DEP2 prefill workers keep up at c1228 (364.5k/362.1k vs 336.7k tok/s/GPU at 4P1D, same interactivity). Add c4096 with seven DEP2 prefill workers and 64 requests per decode rank (630.1k tok/s/GPU, P90 137 tok/s/user). Both decode roles use a 140gb Mooncake segment.
ffe8082 to
26be6a3
Compare
Root cause: the previous image needed an external-linker patch and the submitted recipe omitted supported routing and worker settings used locally. Fix: pin the October 21 public image, remove the linker patch and setup script, enable per-role routing, total-token balancing, worker fastokens and BF16 decode index logits, and use eight DEP2 prefill workers at c4096 within the same four prefill nodes. Update only the disaggregated matrix and changelog. Validation: the repository matrix generator resolves all three points with 3600-second measurements and evaluation enabled. The exact new image/config combination has not yet completed a performance sweep or native evaluation. 原因：旧镜像需要外部缓存 linker 补丁，提交配置也未包含本地使用的部分原生配置项。 修复：固定 10 月 21 日公开镜像，删除补丁，补齐路由、负载均衡、tokenizer 和 decode index logits 配置；c4096 在原有四个 prefill 节点使用八个 DEP2 worker。 验证：三个配置点均通过矩阵生成，保留 3600 秒测量和 eval；新版完整组合尚未实跑。
Keep the complete upstream changelog and append the disaggregated configuration entry after it. 保留上游全部历史条目，将本 PR 的分离式配置条目追加到末尾。
weireweire
changed the title
|
cookbook pr: sgl-project/sglang#45495 |
weireweire
added
the
full-sweep-enabled
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @weireweire's task in 49s —— View job LGTM - no blocking issues found |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39695060576 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39695060576 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Root cause: prefill-only Prometheus counters were treated as deployment totals, preventing record-based token fallback and omitting decode KV metrics. Fix: enable SGLang metrics on decode for all three disaggregated variants. Validation: Ruff check and format checks passed; perf-changelog matrix validation passed. Full GPU sweep remains pending. 原因：仅 prefill 导出指标会使汇总缺少 decode 数据。修复：为三个 disagg 配置恢复 decode metrics。校验：Ruff 和 changelog 矩阵检查通过，GPU sweep 尚待验证。
Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup. Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass. 修复 KV-event 端口与节点监控监听端口冲突，回移 srt-slurm #617；保持镜像及模型参数不变。
Retain only the port-base change and concise public descriptions. 简化端口补丁及说明。
Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission. Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布，保留预填充事件和双方指标，并移除端口补丁。
weireweire
changed the title
将现有 TP4 c1/c4/c16 和 TP2 c16 纳入本 PR 的 sweep 与 eval 选点，覆盖低延迟区间；保留原有 agg 配置及镜像。矩阵校验生成 7 个性能点和 5 个 eval，Ruff 检查通过。
保留 main 的全部历史 changelog，并将本 PR 的 agg/disagg 选点条目置于末尾。确认 agg 配置未变化，矩阵仍生成 7 个性能点和 5 个 eval。
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
/use 39695060576 |
|
@nvpohanh staged run 39695060576. This run remains available across future |
|
[by Claude Code] sgl-project/sglang#45749 has been merged. This PR's AgentX config ( This PR's |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a listed owner of ✅ Check 1 (Sweep on in-PR commit): PASS — head ✅ Check 2 (Evals): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#45749 is MERGED (2026-10-22T06:15:20Z). Its ✅ Check 4 (Reuse command): PASS — @nvpohanh (COLLABORATOR) posted ✅ Check 5 (Latest template): PASS — all 19 items of the current ✅ Check 6 (Upstream image / ordering): PASS — both entries use upstream ✅ Check 7 (Deprecated models): PASS — the MODELS.md row for ✅ Check 8 (No architecture hacks): PASS — no ✅ Check 9 (Spec-decode chat template): PASS — the AgentX client drives the chat endpoint, with ✅ Check 10 (No engine patches): PASS — the PR adds no patches, setup scripts, sitecustomize or wheel installs; the only srt-slurm patch, ✅ Check 11 (Agentic golden AL): PASS — ➖ Check 12 (Append-only): N/A — no new changelog entry sets ✅ Check 13 (Draft precision): PASS — the draft is the checkpoint's embedded DSpark head, served with no draft path, quantization, dtype or KV-dtype override. Pinned SGLang ✅ Check 14 (Pareto coverage): PASS — curve dsv41flash/agentic-coding/gb300_dynamo-sglang/fp4 (agg and disagg points combined) from ✅ Check 15 (PR description): PASS — the claimed topology, MRR/graph BS/extra slots, Mooncake segment sizes, routing, KV events, scheduling flags, image digest and unchanged agg points all match the pinned recipe. sglang#44623 (merge ✅ Check 16 (NVIDIA overview): PASS — DeepSeek-V4.1-Flash has a populated SGLang cell in the Framework Recipes table of the NVIDIA overview. ➖ Check 17 (Metrics publication): N/A — the affected configs are Dynamo + SGLang, not Dynamo TRT-LLM, so these Dynamo metrics settings do not apply ( Assessed commit: |
Update the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang disaggregated AgentX configuration: replace prefill HiCache with Mooncake external cache linking, revise the prefill topology, and separate prefill KV routing from decode round-robin routing.
Use digest-pinned
lmsysorg/sglang:nightly-dev-cu13-20261021-c1c287a4with bundled Dynamo1.6.0.dev20261019and SGLang0.0.0.dev1+gc1c287a42. The image includes sgl-project/sglang#44623; no SGLang engine patch is added.MOONCAKE_GLOBAL_SEGMENT_SIZE=80gbon prefill and140gbon decode. Prefill retains KV routing with a 14400-second session-affinity TTL; decode uses round-robin routing and SGLangtotal_tokensload balancing. Keep prefill KV events, disable unused decode KV events, and retain metrics on both roles. Dynamo uses the ZMQ event plane.flashmla_mega, fastokens and local DP control broadcast on both roles; enable BF16 index logits on decode. Remove the NCCL DP-batch-sync environment override. BF16 index logits changes indexer precision and is covered by the eval sweep.Include the existing aggregated TP4 c1/c4/c16 and TP2 c16 points in the same sweep and eval selection to cover the low-latency portion of the curve. Their configurations and pinned October 15 image are unchanged.
中文
更新 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang 的分离式 AgentX 配置，涵盖拓扑、缓存、路由和执行参数，而不只是升级镜像。
nightly-dev-cu13-20261021-c1c287a4，以及镜像自带的 Dynamo1.6.0.dev20261019、SGLang0.0.0.dev1+gc1c287a42，不添加 SGLang 引擎补丁。80gb/140gb。Prefill 使用 KV 路由和 14400 秒会话亲和；decode 使用 round-robin 和total_tokensDP 负载均衡。保留 prefill KV events，关闭 decode KV events，两侧 metrics 保留；Dynamo event plane 使用 ZMQ。flashmla_mega、fastokens 和本地 DP control broadcast；decode 启用 BF16 index logits，移除 NCCL DP-batch-sync 环境覆盖。使用 2048 SWA prefix tails、decoder SWA bounded replay、decode polling interval 8 和 stream interval 32；c2884/c4096 的 prefill batch token 上限为 32768。BF16 index logits 的精度变化由 eval sweep 验证。将现有 aggregated TP4 c1/c4/c16 和 TP2 c16 纳入同一轮 sweep 与 eval 选点，覆盖曲线的低延迟区间；其配置和固定的 10 月 15 日镜像保持不变。
AI model disclosure
claude-opus-5-5), as recorded in the previous PR description.