From 1f91d15de104b6d53d699337773649cbd2f6490e Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Mon, 19 Oct 2026 11:52:25 +0800 Subject: [PATCH 01/11] feat(agentx): GB300 DSV4.1-Flash Dynamo+SGLang disagg on the 20261019 nightly with the Mooncake linker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg to lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089 (multiarch digest pinned) and the ai-dynamo 1.6.0.dev20261018 it ships, with the ZMQ event plane and no NATS service. - c1228 keeps 4x DEP2 prefill + 1x DEP16 decode; c2884 moves from 6 to 5 DEP2 prefill workers (3 prefill nodes, 26 GPUs) and caps prefill batches at 32768 tokens. - Both roles use the flashmla_mega attention backend, local DP control broadcast and the Mooncake external cache linker with a Mooncake master service (high watermark 0.90, client TTL 60 s, 64 RPC threads). The linker replaces prefill HiCache, so the master rows declare Mooncake 0.3.13.post1 as the KV offload backend. - Decode adds late admission, stream interval 32, polling interval 8, 2048 SWA prefix tails and decoder SWA bounded replay; the NCCL DP-batch sync env is dropped because the image exchanges MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622). - A setup script applies four SGLang patches the image lacks, with git apply under a lock, idempotently, failing on any hunk that does not apply: sgl-project/sglang#44623 (open), #44580 (merged as 834bfe4ef0), #45062 (merged as 06342acad2) and decode late admission (weireweire/sglang@407e912a1e). - Revise this PR's changelog entry to the final state. 中文：将 dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg 迁移到 lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089（固定 multiarch digest）及镜像自带的 ai-dynamo 1.6.0.dev20261018，使用 ZMQ event plane， 不再启动 NATS 服务。 - c1228 保持 4 个 DEP2 prefill + 1 个 DEP16 decode；c2884 的 DEP2 prefill 由 6 个减为 5 个（3 个 prefill 节点，共 26 个 GPU），并将 prefill 批次 上限设为 32768 token。 - 两个角色均使用 flashmla_mega attention 后端、本地 DP control broadcast 和 Mooncake external cache linker，并启动 Mooncake master 服务（高水位 0.90，client TTL 60 秒，64 个 RPC 线程）。linker 取代了 prefill HiCache，因此 master 配置将 KV offload 后端声明为 Mooncake 0.3.13.post1。 - Decode 新增 late admission、stream interval 32、polling interval 8、 2048 SWA prefix tails 与 decoder SWA bounded replay；镜像已通过 gloo gather + broadcast 交换 MLP 同步数据（sgl-project/sglang#44622），因此 删除 NCCL DP batch 同步环境变量。 - 新增 setup 脚本，在锁保护下用 git apply 幂等地应用镜像缺少的四个 SGLang 补丁，任一 hunk 无法应用即失败：sgl-project/sglang#44623（未合并）、 #44580（已合并为 834bfe4ef0）、#45062（已合并为 06342acad2）以及 decode late admission（weireweire/sglang@407e912a1e）。 - 将本 PR 的 changelog 条目更新为最终状态。 --- .../configs/dsv41flash-sglang-patches.sh | 32 ++ .../sglang-decode-late-admission.patch | 48 +++ .../sglang-pr44580.patch | 22 ++ .../sglang-pr44623.patch | 290 ++++++++++++++++++ .../sglang-pr45062.patch | 28 ++ .../gb300-fp4/agentx/disagg-variants.yaml | 80 +++-- inferencex-e2e/configs/nvidia-master.yaml | 12 +- 7 files changed, 485 insertions(+), 27 deletions(-) create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch create mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh new file mode 100644 index 0000000000..6d3403ff9c --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -eo pipefail + +sglang_root=/sgl-workspace/sglang +patch_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/dsv41flash-sglang-patches" +# SGLang changes missing from nightly-dev-cu13-20261019-3c560089; drop each once the image has it. +patches=( + sglang-pr44623.patch # sgl-project/sglang#44623 (open) + sglang-pr44580.patch # sgl-project/sglang#44580, merged as 834bfe4ef0 + sglang-pr45062.patch # sgl-project/sglang#45062, merged as 06342acad2 + sglang-decode-late-admission.patch # weireweire/sglang@407e912a1e, not yet upstream +) + +origin=$(python3 -c 'import importlib.util; print(importlib.util.find_spec("sglang").origin)') +if [[ "$origin" != "$sglang_root/python/sglang/__init__.py" ]]; then + echo "sglang imports from $origin, not the editable checkout in $sglang_root" >&2 + exit 1 +fi + +# Workers sharing a node and container must not patch the tree concurrently. +exec 9<"$sglang_root" +flock 9 +cd / +for name in "${patches[@]}"; do + patch="$patch_dir/$name" + if git apply --directory="${sglang_root#/}" --reverse --check "$patch" 2>/dev/null; then + echo "SGLang patch $name: already applied" + else + git apply --directory="${sglang_root#/}" --verbose "$patch" + echo "SGLang patch $name: applied" + fi +done diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch new file mode 100644 index 0000000000..031acde5d1 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch @@ -0,0 +1,48 @@ +diff --git a/python/sglang/srt/disaggregation/decode.py b/python/sglang/srt/disaggregation/decode.py +index 8fbeddb2df..1046c7516c 100644 +--- a/python/sglang/srt/disaggregation/decode.py ++++ b/python/sglang/srt/disaggregation/decode.py +@@ -3084,6 +3084,8 @@ class SchedulerDisaggregationDecodeMixin: + tmp_batch, tmp_result = self.result_queue.popleft() + self.process_batch_result(tmp_batch, tmp_result) + ++ late_admission = envs.SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION.get() ++ + while True: + if self.gracefully_exit: + break +@@ -3097,7 +3099,8 @@ class SchedulerDisaggregationDecodeMixin: + if self._engine_paused: + self._record_scheduler_state_for_paused_engine() + continue +- self.process_decode_queue() ++ if not late_admission: ++ self.process_decode_queue() + + # Get the next batch to run + plan = self.get_next_disagg_decode_batch_to_run( +@@ -3126,6 +3129,11 @@ class SchedulerDisaggregationDecodeMixin: + batch_result = None + self._sched_idled = True + ++ # Admit while the launched forward runs, so one rank's admission work ++ # does not hold every DP rank at the next MLP sync. ++ if late_admission: ++ self.process_decode_queue() ++ + # Process the last batch + if self.last_batch: + if not disable_overlap_for_batch: +diff --git a/python/sglang/srt/environ.py b/python/sglang/srt/environ.py +index 8b73503ecf..8ab999624c 100644 +--- a/python/sglang/srt/environ.py ++++ b/python/sglang/srt/environ.py +@@ -677,6 +677,8 @@ class Envs: + SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH = EnvBool(False) + # Opt-in: keep receiving prefill requests while forward results are pending. + SGLANG_ENABLE_DISAGG_PREFILL_CONTINUOUS_INPUT_POLLING = EnvBool(False) ++ # Opt-in: admit transferred decode requests after launching the step's forward. ++ SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION = EnvBool(False) + # Override NVML Confidential Computing autodetection. None = autodetect. + SGLANG_FORCE_CONFIDENTIAL_COMPUTE = EnvBool(None) + diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch new file mode 100644 index 0000000000..2f4cb713a4 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch @@ -0,0 +1,22 @@ +diff --git a/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh b/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh +index dffffd1979..b41821b2d0 100644 +--- a/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh ++++ b/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh +@@ -1,6 +1,7 @@ + #include + #include + ++#include + #include + + #include +@@ -270,6 +271,9 @@ void flashmla_sched_meta( + + const std::size_t smem = sizeof(int) * static_cast(b * 5 + 1 + num_sm_parts * kMetaInts); + RuntimeCheck(smem <= 48 * 1024, "schedule does not fit in shared memory: ", smem, " bytes"); ++ // The kernel also has static shared memory. Opt in so schedules near ++ // 48 KiB do not exceed the default combined static + dynamic limit. ++ runtime::set_max_dynamic_smem_per_device(device_.unwrap(), 48 * 1024); + LaunchKernel(1, kBlockSize, device_.unwrap(), smem)(flashmla_sched_meta_kernel, p); + } + diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch new file mode 100644 index 0000000000..a1567c7179 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch @@ -0,0 +1,290 @@ +diff --git a/python/sglang/srt/arg_groups/deepseek_v4_hook.py b/python/sglang/srt/arg_groups/deepseek_v4_hook.py +index e9e04ec..21c6054 100644 +--- a/python/sglang/srt/arg_groups/deepseek_v4_hook.py ++++ b/python/sglang/srt/arg_groups/deepseek_v4_hook.py +@@ -295,7 +295,6 @@ def validate_deepseek_v41_features(server_args: ServerArgs) -> None: + # ROCm replays hits in a forward of their own, which only the hitting + # attention-DP ranks would enter; CUDA folds them into the extend. + ("DP attention on ROCm", get_platform().is_hip and attn_dp_enabled_of(cfg)), +- ("external cache linker", cfg.enable_unified_cache_external_linker), + ("unified memory", cfg.enable_unified_memory), + # Decode receives the last SWA window from prefill and never replays. + ("PD decode", cfg.disaggregation_mode == "decode"), +diff --git a/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py b/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py +index d552662..01219f7 100644 +--- a/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py ++++ b/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py +@@ -794,14 +794,10 @@ def _dsv4_indexer_regions(kvcache: Any, page_size: int) -> list[_IndexerRegion]: + ] + + +-def _dsv4_low_ratio_entries( +- kvcache: Any, page_size: int, num_host_pages: int, transfer_layer_id_max: int +-): +- """Mirror each shared source once, in FULL-page units. Prefixes end on an even +- page boundary, so ratio-2's request-scoped ring is rebuilt, not cached.""" ++def _dsv4_low_ratio_regions(kvcache: Any, page_size: int): ++ """Expose shared source KV and indexer buffers in FULL-page units.""" + import torch + +- entries = [] + for ratio, names in ( + ( + 1, +@@ -852,24 +848,35 @@ def _dsv4_low_ratio_entries( + ) + regions.append((name, index_pool, rows)) + for name, device_pool, buffers in regions: +- entries.append( +- build_pool_entry( +- name=name, +- host_pool=DeepSeekV4PagedHostPool( +- pool_name=str(name), +- device_buffers=buffers, +- item_bytes=buffers[0].shape[1] * buffers[0].element_size(), +- num_host_pages=num_host_pages, +- slot_page_size=page_size, +- layout=get_memory().hicache_mem_layout, +- allocator_type=_get_allocator_type(), +- page_aligned_only=True, +- ), +- device_pool=device_pool, +- layer_mapping=layer_mapping, +- transfer_layer_id_max=transfer_layer_id_max, +- ) ++ yield name, device_pool, buffers, layer_mapping ++ ++ ++def _dsv4_low_ratio_entries( ++ kvcache: Any, page_size: int, num_host_pages: int, transfer_layer_id_max: int ++): ++ """Mirror shared KV; ratio-2 request state is rebuilt at page boundaries.""" ++ entries = [] ++ for name, device_pool, buffers, layer_mapping in _dsv4_low_ratio_regions( ++ kvcache, page_size ++ ): ++ entries.append( ++ build_pool_entry( ++ name=name, ++ host_pool=DeepSeekV4PagedHostPool( ++ pool_name=str(name), ++ device_buffers=buffers, ++ item_bytes=buffers[0].shape[1] * buffers[0].element_size(), ++ num_host_pages=num_host_pages, ++ slot_page_size=page_size, ++ layout=get_memory().hicache_mem_layout, ++ allocator_type=_get_allocator_type(), ++ page_aligned_only=True, ++ ), ++ device_pool=device_pool, ++ layer_mapping=layer_mapping, ++ transfer_layer_id_max=transfer_layer_id_max, + ) ++ ) + return entries + + +diff --git a/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py b/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py +index 9d66379..334f0df 100644 +--- a/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py ++++ b/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py +@@ -264,6 +264,7 @@ def _build_deepseek_v4_device_pool_group( + from sglang.srt.mem_cache.hybrid_cache.hybrid_pool_assembler import ( + _dsv4_compressed_region_buffers, + _dsv4_indexer_regions, ++ _dsv4_low_ratio_regions, + _require_single_row_dsv4_swa_pages, + _resolve_deepseek_v4_layer_mappings, + ) +@@ -273,8 +274,11 @@ def _build_deepseek_v4_device_pool_group( + + mappings = _resolve_deepseek_v4_layer_mappings(kvcache) + is_unified_kv = getattr(kvcache, "_unified_kv", False) ++ # Encoder SWA replay keeps the target window per request; the draft SWA is ++ # dropped with it, a known draft accept-length cost. ++ has_paged_swa = not is_unified_kv and kvcache.swa_kv_pool is not None + entries = [] +- if not is_unified_kv: ++ if has_paged_swa: + _require_single_row_dsv4_swa_pages( + logical_page_size=kvcache.swa_page_size, + physical_page_size=kvcache.swa_kv_pool.page_size, +@@ -321,31 +325,36 @@ def _build_deepseek_v4_device_pool_group( + ) + ) + +- c4_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 4) +- c128_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 128) +- +- add( +- PoolName.DEEPSEEK_V4_C4, +- PoolName.KV, +- kvcache.c4_kv_pool, +- c4_buffers, +- mappings.c4, +- ) +- for region in _dsv4_indexer_regions(kvcache, page_size): ++ if mappings.c4: ++ c4_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 4) + add( +- region.name, ++ PoolName.DEEPSEEK_V4_C4, + PoolName.KV, +- kvcache.c4_indexer_kv_pool, +- region.device_buffers, ++ kvcache.c4_kv_pool, ++ c4_buffers, + mappings.c4, + ) +- add( +- PoolName.DEEPSEEK_V4_C128, +- PoolName.KV, +- kvcache.c128_kv_pool, +- c128_buffers, +- mappings.c128, +- ) ++ for region in _dsv4_indexer_regions(kvcache, page_size): ++ add( ++ region.name, ++ PoolName.KV, ++ kvcache.c4_indexer_kv_pool, ++ region.device_buffers, ++ mappings.c4, ++ ) ++ if mappings.c128: ++ c128_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 128) ++ add( ++ PoolName.DEEPSEEK_V4_C128, ++ PoolName.KV, ++ kvcache.c128_kv_pool, ++ c128_buffers, ++ mappings.c128, ++ ) ++ for name, device_pool, buffers, layer_mapping in _dsv4_low_ratio_regions( ++ kvcache, page_size ++ ): ++ add(name, PoolName.KV, device_pool, buffers, layer_mapping) + + # fp8 two-pool unified_kv: one row index addresses both kv_buffer (fp8 nope) and + # kv_buffer_rope (bf16 rope), but unified_region_buffers() above returns the nope +@@ -384,6 +393,13 @@ def _build_deepseek_v4_device_pool_group( + ), + mappings.c4_state, + ) ++ if not has_paged_swa and any( ++ entry.indices_from_pool == PoolName.SWA for entry in entries ++ ): ++ raise ValueError( ++ "DeepSeek-V4 compress state pools take SWA page indices, which " ++ "encoder SWA replay does not keep." ++ ) + return DevicePoolGroup( + entries, + mappings.transfer_layer_id_max, +diff --git a/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py b/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py +index f95b00a..3aca7d5 100644 +--- a/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py ++++ b/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py +@@ -416,6 +416,9 @@ class MooncakeDirectLinker(UnifiedCacheLinker): + def pop_completed_offload(self) -> bool: + return self.offload_results.get_nowait() + ++ def restores_swa(self) -> bool: ++ return PoolName.SWA in self.pools ++ + def reset(self) -> None: + self.pending_loads.clear() + self.load_queue.join() +diff --git a/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py b/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py +index 204d533..5a74d32 100644 +--- a/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py ++++ b/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py +@@ -65,20 +65,20 @@ def _ordered_layers(entry) -> list[int]: + f"UMBP pool {entry.name} components have different layer counts." + ) + pool_layer_count = component_lengths.pop() +- if pool_layer_count != len(entry.layer_mapping): +- raise ValueError( +- f"UMBP pool {entry.name} has {pool_layer_count} buffers per component " +- f"but {len(entry.layer_mapping)} mapped layers." +- ) +- by_buffer = { +- buffer_index: logical_layer +- for logical_layer, buffer_index in entry.layer_mapping.items() +- } ++ by_buffer = {} ++ for logical_layer, mapped in entry.layer_mapping.items(): ++ indices = [mapped] if isinstance(mapped, int) else list(mapped) ++ for index in indices: ++ if index in by_buffer: ++ raise ValueError( ++ f"UMBP pool {entry.name} maps buffer {index} more than once." ++ ) ++ by_buffer[index] = logical_layer + if sorted(by_buffer) != list(range(pool_layer_count)): + raise ValueError( + f"UMBP pool {entry.name} layer mapping is not a contiguous bijection." + ) +- return [by_buffer[index] for index in range(pool_layer_count)] ++ return list(dict.fromkeys(by_buffer[index] for index in range(pool_layer_count))) + + + @dataclass +@@ -712,15 +712,17 @@ class UMBPDirectLinker(UnifiedCacheLinker): + entry = self.pools[plan.name] + items: list[list[tuple[int, int, int, int]]] = [] + for logical_layer in layers: +- buffer_index = entry.layer_mapping.get(logical_layer) +- if buffer_index is None: ++ mapped = entry.layer_mapping.get(logical_layer) ++ if mapped is None: + continue ++ indices = [mapped] if isinstance(mapped, int) else list(mapped) + items.append( + [ +- (*component[buffer_index], offsets[buffer_index]) ++ (*component[index], offsets[index]) + for component, offsets in zip( + entry.buffer_meta, entry._component_offsets + ) ++ for index in indices + ] + ) + return items +@@ -989,6 +991,9 @@ class UMBPDirectLinker(UnifiedCacheLinker): + def pop_completed_offload(self) -> bool: + return self._offload_results.get_nowait() + ++ def restores_swa(self) -> bool: ++ return PoolName.SWA in self.pools ++ + def reset(self) -> None: + self._pending.clear() + self._load_queue.join() +diff --git a/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py b/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py +index 3d31507..6661b8c 100644 +--- a/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py ++++ b/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py +@@ -189,6 +189,10 @@ class UnifiedCacheLinker(ABC): + def close(self) -> None: + """Quiesce all transfers and release backend resources.""" + ++ def restores_swa(self) -> bool: ++ """Whether the store carries SWA pages; False tombstones restored tails.""" ++ return True ++ + + class ExternalCacheHitMarker(NamedTuple): + """What ``match`` found in the external store, consumed by ``load_back``. +@@ -228,8 +232,9 @@ class UnifiedCacheLinkerWrapper: + self.cache = cache + self.cache_linker = cache_linker + swa = cache.components.get(ComponentType.SWA) +- self._skip_swa = swa is not None and is_swa_req_ring( +- cache.token_to_kv_pool_allocator ++ self._skip_swa = swa is not None and ( ++ is_swa_req_ring(cache.token_to_kv_pool_allocator) ++ or not cache_linker.restores_swa() + ) + self._components = tuple( + component diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch new file mode 100644 index 0000000000..8b1cc435c7 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch @@ -0,0 +1,28 @@ +diff --git a/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py b/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py +index 2feb9fa117..260dc617bb 100644 +--- a/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py ++++ b/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py +@@ -7,6 +7,7 @@ import torch + from sglang.srt.distributed.utils import all_gather_single + from sglang.srt.layers.dp_attention import DpPaddingMode, set_dp_buffer_len_from_batch + from sglang.srt.runtime_context import get_parallel ++from sglang.srt.utils.common import async_h2d + + if TYPE_CHECKING: + from sglang.srt.model_executor.forward_batch_info import ForwardBatch +@@ -70,11 +71,10 @@ def enter_late_layer_dp(forward_batch: ForwardBatch, counts: List[int]) -> tuple + # Device ops only: this runs inside prefill graph capture, where a host copy + # would be recorded against pageable memory. + gpu = fb.global_num_tokens_gpu +- fb.global_num_tokens_gpu = ( +- torch.full_like(gpu, counts[0]) +- if len(set(counts)) == 1 +- else gpu.new_tensor(counts) +- ) ++ if len(set(counts)) == 1: ++ fb.global_num_tokens_gpu = torch.full_like(gpu, counts[0]) ++ else: ++ fb.global_num_tokens_gpu = async_h2d(counts, dtype=gpu.dtype, device=gpu.device) + fb.global_num_tokens_padded_cpu = counts + # The MoE gatherv reads its sizes from the logprob counts when present. + fb.global_num_tokens_for_logprob_cpu = counts diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index 0a8178acc8..7f08e114a0 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -5,11 +5,13 @@ base: identity: model: {} frameworks: - dynamo: "1.6.0.dev20261014" - sglang: "0.0.0.dev1+ga6a8c5601" + dynamo: "1.6.0.dev20261018" + sglang: "0.0.0.dev1+g3c560089e" dynamo: install: false request_plane: tcp + event_plane: zmq + setup_script: dsv41flash-sglang-patches.sh # gb300-nv serves this model from the Hub; read it from the shared mounted cache. environment: HF_HUB_CACHE: /hf_hub_cache @@ -31,10 +33,12 @@ base: type: etcd placement: node: infra - - name: nats - type: nats - placement: - node: infra + - name: mooncake-master + type: mooncake-master + args: + - --eviction_high_watermark_ratio=0.90 + - --client_ttl=60 + - --rpc_thread_num=64 # Disables Dynamo's per-request lifecycle span export; InferenceX does not read it. - name: otel-collector type: otel-collector @@ -42,7 +46,8 @@ base: frontend: type: dynamo enable_multiple_frontends: false - # One CPU node hosts the frontend, etcd, NATS and the AgentX client. + numa_bind: true + # One CPU node hosts the frontend, etcd, the Mooncake master and the AgentX client. placement: node: dedicated env: @@ -77,6 +82,17 @@ base: NCCL_MNNVL_ENABLE: "1" NCCL_CUMEM_ENABLE: "1" MC_FORCE_MNNVL: "1" + MOONCAKE_PROTOCOL: rdma + # The store registers host buffers only over RDMA; without a device list it + # picks NVLink for same-domain peers. + MOONCAKE_DEVICE: "@fabric.rdma-devices" + MOONCAKE_GLOBAL_SEGMENT_SIZE: 80gb + MOONCAKE_STANDALONE_STORAGE: "0" + MC_STORE_CLIENT_METRIC_INTERVAL: "5" + MC_ENABLE_DEST_DEVICE_AFFINITY: "1" + # Some GB300 nodes list a BMC interface with the same address on every node first. + MC_TCP_BIND_ADDRESS: "{node}" + WITH_NVIDIA_PEERMEM: "0" args: served-model-name: deepseek-ai/DeepSeek-V4.1-Flash trust-remote-code: true @@ -85,20 +101,24 @@ base: expert-parallel-size: 2 enable-dp-attention: true enable-dp-lm-head: true + enable-dp-attention-local-control-broadcast: true moe-a2a-backend: megamoe - dsv4-attn-backend: flashmla + dsv4-attn-backend: flashmla_mega disaggregation-transfer-backend: mooncake chunked-prefill-size: 65536 max-running-requests: 128 swa-prefix-tails: 2048 enable-decoder-swa-bounded-replay: true - enable-hierarchical-cache: true + enable-unified-cache-external-linker: true + unified-cache-external-linker-backend: mooncake + hicache-storage-backend-extra-config: '{"enable_group_semantics":true}' hicache-write-policy: write_back - hicache-io-backend: direct + hicache-ratio: 1.02 hicache-mem-layout: page_first_direct mem-fraction-static: 0.85 speculative-algorithm: DSPARK speculative-dspark-block-size: 5 + speculative-num-steps: 5 reasoning-parser: auto tool-call-parser: auto watchdog-timeout: 3600 @@ -106,7 +126,6 @@ base: weight-loader-prefetch-checkpoints: true weight-loader-drop-cache-after-load: false model-loader-extra-config: '{"enable_multithread_load":true}' - # Only prefill publishes KV events to the Dynamo KV router. kv_events: true decode: nodes: 4 @@ -120,13 +139,20 @@ base: DYN_SGL_GATEWAY_WORKERS: "8" SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600" SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600" - # Sync the per-step DP batch over NCCL instead of gloo. - SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH: "1" + # Admit transferred requests after launching the step, off the DP sync path. + SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION: "1" SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True" SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "16384" NCCL_MNNVL_ENABLE: "1" NCCL_CUMEM_ENABLE: "1" MC_FORCE_MNNVL: "1" + MOONCAKE_PROTOCOL: rdma + MOONCAKE_DEVICE: "@fabric.rdma-devices" + MOONCAKE_STANDALONE_STORAGE: "0" + MC_STORE_CLIENT_METRIC_INTERVAL: "5" + MC_ENABLE_DEST_DEVICE_AFFINITY: "1" + MC_TCP_BIND_ADDRESS: "{node}" + WITH_NVIDIA_PEERMEM: "0" args: served-model-name: deepseek-ai/DeepSeek-V4.1-Flash trust-remote-code: true @@ -135,19 +161,28 @@ base: expert-parallel-size: 16 enable-dp-attention: true enable-dp-lm-head: true + enable-dp-attention-local-control-broadcast: true moe-a2a-backend: megamoe - dsv4-attn-backend: flashmla + dsv4-attn-backend: flashmla_mega disaggregation-transfer-backend: mooncake + disaggregation-decode-polling-interval: 8 + stream-interval: 32 + swa-prefix-tails: 2048 + enable-decoder-swa-bounded-replay: true + enable-unified-cache-external-linker: true + unified-cache-external-linker-backend: mooncake mem-fraction-static: 0.85 speculative-algorithm: DSPARK speculative-dspark-block-size: 5 + speculative-num-steps: 5 reasoning-parser: auto tool-call-parser: auto watchdog-timeout: 3600 - enable-metrics: true + enable-metrics: false weight-loader-prefetch-checkpoints: true weight-loader-drop-cache-after-load: false model-loader-extra-config: '{"enable_multithread_load":true}' + kv_events: true sbatch_directives: mem: "0" exclusive: "" @@ -173,23 +208,26 @@ override_4p1d_dep2_dep16_c1228: prefill: nodes: 2 workers: 4 - args: - hicache-ratio: 1.02 decode: + env: + MOONCAKE_GLOBAL_SEGMENT_SIZE: 100gb args: max-running-requests: 384 cuda-graph-max-bs-decode: 24 disaggregation-decode-extra-slots: 36 -override_6p1d_dep2_dep16_c2884: - name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-6p1d-dep2-dep16-c2884 +override_5p1d_dep2_dep16_c2884: + name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-5p1d-dep2-dep16-c2884 roles: prefill: + # Five DEP2 workers leave two GPUs of the third node idle. nodes: 3 - workers: 6 + workers: 5 args: - hicache-ratio: 1.1 + max-prefill-tokens: 32768 decode: + env: + MOONCAKE_GLOBAL_SEGMENT_SIZE: 140gb args: max-running-requests: 640 cuda-graph-max-bs-decode: 40 diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index d3bef73dd6..09a62429a7 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -9109,13 +9109,13 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg: dp-attn: false dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: - image: lmsysorg/sglang:nightly-dev-cu13-20261015-a6a8c560@sha256:0a1ba33aa4e032b8c5bcebdc69c45664074c032074add4f7db01629ec1fcc381 + image: lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089@sha256:c1f4d9fc6fcf50c2269adbc757ce7b1778883af04155edb888fa9eea3b6648c0 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:gb300-nv precision: fp4 framework: dynamo-sglang - router: { name: dynamo-router, version: "1.6.0.dev20261014" } + router: { name: dynamo-router, version: "1.6.0.dev20261018" } kv-p2p-transfer: mooncake multinode: true disagg: true @@ -9128,7 +9128,7 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: conc-list: [1228] srt-recipe: disagg-variants.yaml:override_4p1d_dep2_dep16_c1228 kv-offloading: dram - kv-offload-backend: { name: hicache } + kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } prefill: num-worker: 4 tp: 2 @@ -9141,11 +9141,11 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: dp-attn: true - spec-decoding: mtp conc-list: [2884] - srt-recipe: disagg-variants.yaml:override_6p1d_dep2_dep16_c2884 + srt-recipe: disagg-variants.yaml:override_5p1d_dep2_dep16_c2884 kv-offloading: dram - kv-offload-backend: { name: hicache } + kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } prefill: - num-worker: 6 + num-worker: 5 tp: 2 ep: 2 dp-attn: true From acb21592ed8d749169b443fc987f55c5a4acd57e Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Mon, 19 Oct 2026 12:51:11 +0800 Subject: [PATCH 02/11] fix(agentx): drop decode late admission from the GB300 DSV4.1-Flash disagg points MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A same-time A/B on the 20261019 nightly showed no effect from decode late admission: P90 interactivity 230/232 without it and 230/228 with it at c1228. The image already exchanges the scheduler's MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622) and avoids blocking Engram history copies during PD admission (#44787). Remove the patch, its setup script entry and SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION. 中文：在 20261019 nightly 上的同时段 A/B 显示 decode late admission 没有 效果：c1228 下关闭时 P90 interactivity 为 230/232，开启时为 230/228。镜像 已通过 gloo gather + broadcast 交换调度器的 MLP 同步数据 （sgl-project/sglang#44622），并避免了 PD admission 期间阻塞的 Engram 历史拷贝（#44787）。因此删除该补丁、setup 脚本中的对应条目以及 SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION。 --- .../configs/dsv41flash-sglang-patches.sh | 7 ++- .../sglang-decode-late-admission.patch | 48 ------------------- .../gb300-fp4/agentx/disagg-variants.yaml | 2 - 3 files changed, 3 insertions(+), 54 deletions(-) delete mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh index 6d3403ff9c..5c047a38f9 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh @@ -5,10 +5,9 @@ sglang_root=/sgl-workspace/sglang patch_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/dsv41flash-sglang-patches" # SGLang changes missing from nightly-dev-cu13-20261019-3c560089; drop each once the image has it. patches=( - sglang-pr44623.patch # sgl-project/sglang#44623 (open) - sglang-pr44580.patch # sgl-project/sglang#44580, merged as 834bfe4ef0 - sglang-pr45062.patch # sgl-project/sglang#45062, merged as 06342acad2 - sglang-decode-late-admission.patch # weireweire/sglang@407e912a1e, not yet upstream + sglang-pr44623.patch # sgl-project/sglang#44623 (open) + sglang-pr44580.patch # sgl-project/sglang#44580, merged as 834bfe4ef0 + sglang-pr45062.patch # sgl-project/sglang#45062, merged as 06342acad2 ) origin=$(python3 -c 'import importlib.util; print(importlib.util.find_spec("sglang").origin)') diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch deleted file mode 100644 index 031acde5d1..0000000000 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-decode-late-admission.patch +++ /dev/null @@ -1,48 +0,0 @@ -diff --git a/python/sglang/srt/disaggregation/decode.py b/python/sglang/srt/disaggregation/decode.py -index 8fbeddb2df..1046c7516c 100644 ---- a/python/sglang/srt/disaggregation/decode.py -+++ b/python/sglang/srt/disaggregation/decode.py -@@ -3084,6 +3084,8 @@ class SchedulerDisaggregationDecodeMixin: - tmp_batch, tmp_result = self.result_queue.popleft() - self.process_batch_result(tmp_batch, tmp_result) - -+ late_admission = envs.SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION.get() -+ - while True: - if self.gracefully_exit: - break -@@ -3097,7 +3099,8 @@ class SchedulerDisaggregationDecodeMixin: - if self._engine_paused: - self._record_scheduler_state_for_paused_engine() - continue -- self.process_decode_queue() -+ if not late_admission: -+ self.process_decode_queue() - - # Get the next batch to run - plan = self.get_next_disagg_decode_batch_to_run( -@@ -3126,6 +3129,11 @@ class SchedulerDisaggregationDecodeMixin: - batch_result = None - self._sched_idled = True - -+ # Admit while the launched forward runs, so one rank's admission work -+ # does not hold every DP rank at the next MLP sync. -+ if late_admission: -+ self.process_decode_queue() -+ - # Process the last batch - if self.last_batch: - if not disable_overlap_for_batch: -diff --git a/python/sglang/srt/environ.py b/python/sglang/srt/environ.py -index 8b73503ecf..8ab999624c 100644 ---- a/python/sglang/srt/environ.py -+++ b/python/sglang/srt/environ.py -@@ -677,6 +677,8 @@ class Envs: - SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH = EnvBool(False) - # Opt-in: keep receiving prefill requests while forward results are pending. - SGLANG_ENABLE_DISAGG_PREFILL_CONTINUOUS_INPUT_POLLING = EnvBool(False) -+ # Opt-in: admit transferred decode requests after launching the step's forward. -+ SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION = EnvBool(False) - # Override NVML Confidential Computing autodetection. None = autodetect. - SGLANG_FORCE_CONFIDENTIAL_COMPUTE = EnvBool(None) - diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index 7f08e114a0..c49dca7c5c 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -139,8 +139,6 @@ base: DYN_SGL_GATEWAY_WORKERS: "8" SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600" SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600" - # Admit transferred requests after launching the step, off the DP sync path. - SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION: "1" SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True" SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "16384" NCCL_MNNVL_ENABLE: "1" From e7efe4bf4f96b789da1f27ef945eedf795abd7ad Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Tue, 20 Oct 2026 14:23:30 +0800 Subject: [PATCH 03/11] feat(agentx): move the GB300 DSV4.1-Flash disagg points to the 20261020 nightly The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains sgl-project/sglang#44580 and #45062, so the setup script now applies only the external linker pools patch (#44623). Add the #4303 changelog entry on top of the merged #4180 entry. --- .../configs/dsv41flash-sglang-patches.sh | 4 +-- .../sglang-pr44580.patch | 22 --------------- .../sglang-pr45062.patch | 28 ------------------- .../gb300-fp4/agentx/disagg-variants.yaml | 4 +-- inferencex-e2e/configs/nvidia-master.yaml | 4 +-- inferencex-e2e/perf-changelog.yaml | 11 ++++++++ 6 files changed, 16 insertions(+), 57 deletions(-) delete mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch delete mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh index 5c047a38f9..74b4ae7b1a 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh @@ -3,11 +3,9 @@ set -eo pipefail sglang_root=/sgl-workspace/sglang patch_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/dsv41flash-sglang-patches" -# SGLang changes missing from nightly-dev-cu13-20261019-3c560089; drop each once the image has it. +# SGLang changes missing from nightly-dev-cu13-20261020-ab580b2e; drop each once the image has it. patches=( sglang-pr44623.patch # sgl-project/sglang#44623 (open) - sglang-pr44580.patch # sgl-project/sglang#44580, merged as 834bfe4ef0 - sglang-pr45062.patch # sgl-project/sglang#45062, merged as 06342acad2 ) origin=$(python3 -c 'import importlib.util; print(importlib.util.find_spec("sglang").origin)') diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch deleted file mode 100644 index 2f4cb713a4..0000000000 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44580.patch +++ /dev/null @@ -1,22 +0,0 @@ -diff --git a/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh b/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh -index dffffd1979..b41821b2d0 100644 ---- a/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh -+++ b/python/sglang/kernels/jit/csrc/deepseek_v4/flashmla_sched_meta.cuh -@@ -1,6 +1,7 @@ - #include - #include - -+#include - #include - - #include -@@ -270,6 +271,9 @@ void flashmla_sched_meta( - - const std::size_t smem = sizeof(int) * static_cast(b * 5 + 1 + num_sm_parts * kMetaInts); - RuntimeCheck(smem <= 48 * 1024, "schedule does not fit in shared memory: ", smem, " bytes"); -+ // The kernel also has static shared memory. Opt in so schedules near -+ // 48 KiB do not exceed the default combined static + dynamic limit. -+ runtime::set_max_dynamic_smem_per_device(device_.unwrap(), 48 * 1024); - LaunchKernel(1, kBlockSize, device_.unwrap(), smem)(flashmla_sched_meta_kernel, p); - } - diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch deleted file mode 100644 index 8b1cc435c7..0000000000 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr45062.patch +++ /dev/null @@ -1,28 +0,0 @@ -diff --git a/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py b/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py -index 2feb9fa117..260dc617bb 100644 ---- a/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py -+++ b/python/sglang/srt/models/deepseek_common/decoder_swa_bounded_replay.py -@@ -7,6 +7,7 @@ import torch - from sglang.srt.distributed.utils import all_gather_single - from sglang.srt.layers.dp_attention import DpPaddingMode, set_dp_buffer_len_from_batch - from sglang.srt.runtime_context import get_parallel -+from sglang.srt.utils.common import async_h2d - - if TYPE_CHECKING: - from sglang.srt.model_executor.forward_batch_info import ForwardBatch -@@ -70,11 +71,10 @@ def enter_late_layer_dp(forward_batch: ForwardBatch, counts: List[int]) -> tuple - # Device ops only: this runs inside prefill graph capture, where a host copy - # would be recorded against pageable memory. - gpu = fb.global_num_tokens_gpu -- fb.global_num_tokens_gpu = ( -- torch.full_like(gpu, counts[0]) -- if len(set(counts)) == 1 -- else gpu.new_tensor(counts) -- ) -+ if len(set(counts)) == 1: -+ fb.global_num_tokens_gpu = torch.full_like(gpu, counts[0]) -+ else: -+ fb.global_num_tokens_gpu = async_h2d(counts, dtype=gpu.dtype, device=gpu.device) - fb.global_num_tokens_padded_cpu = counts - # The MoE gatherv reads its sizes from the logprob counts when present. - fb.global_num_tokens_for_logprob_cpu = counts diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index c49dca7c5c..a0a7130d47 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -5,8 +5,8 @@ base: identity: model: {} frameworks: - dynamo: "1.6.0.dev20261018" - sglang: "0.0.0.dev1+g3c560089e" + dynamo: "1.6.0.dev20261019" + sglang: "0.0.0.dev1+gab580b2e0" dynamo: install: false request_plane: tcp diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 09a62429a7..73b6191930 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -9109,13 +9109,13 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg: dp-attn: false dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: - image: lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089@sha256:c1f4d9fc6fcf50c2269adbc757ce7b1778883af04155edb888fa9eea3b6648c0 + image: lmsysorg/sglang:nightly-dev-cu13-20261020-ab580b2e@sha256:0af411bb320a95bb5e1c58b4c20c9852920a19aa9f12f148f69731b0dca3b576 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:gb300-nv precision: fp4 framework: dynamo-sglang - router: { name: dynamo-router, version: "1.6.0.dev20261018" } + router: { name: dynamo-router, version: "1.6.0.dev20261019" } kv-p2p-transfer: mooncake multinode: true disagg: true diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 14df99b78f..6b321f2d67 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10256,3 +10256,14 @@ - "Raise DP chunked-prefill-size from 65536 to 131072 (16384 tokens per rank)." - "MegaMoE: raise FlyDSL MTPR to 16384, add decode MTPR / fixed-slot caps, EPLB P2P batch chunk size 7." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4315 + +- config-keys: + - dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg + scenario-type: + - agentic-coding + description: + - "Move the disaggregated points to lmsysorg/sglang:nightly-dev-cu13-20261020-ab580b2e (digest sha256:0af411bb320a95bb5e1c58b4c20c9852920a19aa9f12f148f69731b0dca3b576) with the ai-dynamo 1.6.0.dev20261019 it ships and the ZMQ event plane. One CPU node hosts etcd, the Mooncake master, the frontend and the AgentX client." + - "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 100gb at c1228 and 140gb at c2884 on decode). The higher prefix hit rate lets c2884 run 5x DEP2 prefill instead of 6x; c1228 keeps 4x DEP2 prefill." + - "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32, polling interval 8 and KV events, with the NCCL DP-batch sync env removed; c2884 caps prefill batches at 32768 tokens." + - "A setup script applies the DeepSeek-V4.1 external linker pools (sgl-project/sglang#44623), which the image does not have yet." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4303 From 26be6a3664fe749aea8099eb01ee0db286a1e758 Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Tue, 20 Oct 2026 14:57:39 +0800 Subject: [PATCH 04/11] feat(agentx): run GB300 DSV4.1-Flash disagg c1228 at 3P1D and add c4096 at 7P1D With the Mooncake linker, three DEP2 prefill workers keep up at c1228 (364.5k/362.1k vs 336.7k tok/s/GPU at 4P1D, same interactivity). Add c4096 with seven DEP2 prefill workers and 64 requests per decode rank (630.1k tok/s/GPU, P90 137 tok/s/user). Both decode roles use a 140gb Mooncake segment. --- .../gb300-fp4/agentx/disagg-variants.yaml | 26 ++++++++++++++++--- inferencex-e2e/configs/nvidia-master.yaml | 19 ++++++++++++-- inferencex-e2e/perf-changelog.yaml | 5 ++-- 3 files changed, 42 insertions(+), 8 deletions(-) diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index a0a7130d47..492cae991e 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -200,15 +200,16 @@ base: AIPERF_EXTRA_INPUTS: '{"chat_template_kwargs":{"reasoning_effort":"high","thinking":true},"temperature":1}' AIPERF_DATASET_MMAP_CACHE_ENABLED: "true" -override_4p1d_dep2_dep16_c1228: - name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-4p1d-dep2-dep16-c1228 +override_3p1d_dep2_dep16_c1228: + name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-3p1d-dep2-dep16-c1228 roles: prefill: + # Three DEP2 workers leave two GPUs of the second node idle. nodes: 2 - workers: 4 + workers: 3 decode: env: - MOONCAKE_GLOBAL_SEGMENT_SIZE: 100gb + MOONCAKE_GLOBAL_SEGMENT_SIZE: 140gb args: max-running-requests: 384 cuda-graph-max-bs-decode: 24 @@ -230,3 +231,20 @@ override_5p1d_dep2_dep16_c2884: max-running-requests: 640 cuda-graph-max-bs-decode: 40 disaggregation-decode-extra-slots: 80 + +override_7p1d_dep2_dep16_c4096: + name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-7p1d-dep2-dep16-c4096 + roles: + prefill: + # Seven DEP2 workers leave two GPUs of the fourth node idle. + nodes: 4 + workers: 7 + args: + max-prefill-tokens: 32768 + decode: + env: + MOONCAKE_GLOBAL_SEGMENT_SIZE: 140gb + args: + max-running-requests: 1024 + cuda-graph-max-bs-decode: 64 + disaggregation-decode-extra-slots: 128 diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index 73b6191930..b5ac2f85a6 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -9126,11 +9126,11 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: search-space: - spec-decoding: mtp conc-list: [1228] - srt-recipe: disagg-variants.yaml:override_4p1d_dep2_dep16_c1228 + srt-recipe: disagg-variants.yaml:override_3p1d_dep2_dep16_c1228 kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } prefill: - num-worker: 4 + num-worker: 3 tp: 2 ep: 2 dp-attn: true @@ -9154,6 +9154,21 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: tp: 16 ep: 16 dp-attn: true + - spec-decoding: mtp + conc-list: [4096] + srt-recipe: disagg-variants.yaml:override_7p1d_dep2_dep16_c4096 + kv-offloading: dram + kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } + prefill: + num-worker: 7 + tp: 2 + ep: 2 + dp-attn: true + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg: image: vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4 diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 6b321f2d67..c0ac1936c5 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10263,7 +10263,8 @@ - agentic-coding description: - "Move the disaggregated points to lmsysorg/sglang:nightly-dev-cu13-20261020-ab580b2e (digest sha256:0af411bb320a95bb5e1c58b4c20c9852920a19aa9f12f148f69731b0dca3b576) with the ai-dynamo 1.6.0.dev20261019 it ships and the ZMQ event plane. One CPU node hosts etcd, the Mooncake master, the frontend and the AgentX client." - - "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 100gb at c1228 and 140gb at c2884 on decode). The higher prefix hit rate lets c2884 run 5x DEP2 prefill instead of 6x; c1228 keeps 4x DEP2 prefill." - - "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32, polling interval 8 and KV events, with the NCCL DP-batch sync env removed; c2884 caps prefill batches at 32768 tokens." + - "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 140gb on decode). The higher prefix hit rate lets c1228 run 3x DEP2 prefill instead of 4x and c2884 5x instead of 6x." + - "Add c4096 with 7x DEP2 prefill (4 nodes) and one DEP16 decode: 64 requests per decode rank, 128 extra decode slots." + - "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32, polling interval 8 and KV events, with the NCCL DP-batch sync env removed; c2884 and c4096 cap prefill batches at 32768 tokens." - "A setup script applies the DeepSeek-V4.1 external linker pools (sgl-project/sglang#44623), which the image does not have yet." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4303 From 3b4998c8ad408e31b27b3227556531de5b5ac2b4 Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Wed, 21 Oct 2026 10:44:24 +0800 Subject: [PATCH 05/11] perf(agentx): update DSV4.1 disagg to patch-free October 21 nightly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: the previous image needed an external-linker patch and the submitted recipe omitted supported routing and worker settings used locally. Fix: pin the October 21 public image, remove the linker patch and setup script, enable per-role routing, total-token balancing, worker fastokens and BF16 decode index logits, and use eight DEP2 prefill workers at c4096 within the same four prefill nodes. Update only the disaggregated matrix and changelog. Validation: the repository matrix generator resolves all three points with 3600-second measurements and evaluation enabled. The exact new image/config combination has not yet completed a performance sweep or native evaluation. 原因：旧镜像需要外部缓存 linker 补丁，提交配置也未包含本地使用的部分原生配置项。 修复：固定 10 月 21 日公开镜像，删除补丁，补齐路由、负载均衡、tokenizer 和 decode index logits 配置；c4096 在原有四个 prefill 节点使用八个 DEP2 worker。 验证：三个配置点均通过矩阵生成，保留 3600 秒测量和 eval；新版完整组合尚未实跑。 --- .../configs/dsv41flash-sglang-patches.sh | 29 -- .../sglang-pr44623.patch | 290 ------------------ .../gb300-fp4/agentx/disagg-variants.yaml | 17 +- inferencex-e2e/configs/nvidia-master.yaml | 6 +- inferencex-e2e/perf-changelog.yaml | 8 +- 5 files changed, 19 insertions(+), 331 deletions(-) delete mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh delete mode 100644 inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh deleted file mode 100644 index 74b4ae7b1a..0000000000 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env bash -set -eo pipefail - -sglang_root=/sgl-workspace/sglang -patch_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/dsv41flash-sglang-patches" -# SGLang changes missing from nightly-dev-cu13-20261020-ab580b2e; drop each once the image has it. -patches=( - sglang-pr44623.patch # sgl-project/sglang#44623 (open) -) - -origin=$(python3 -c 'import importlib.util; print(importlib.util.find_spec("sglang").origin)') -if [[ "$origin" != "$sglang_root/python/sglang/__init__.py" ]]; then - echo "sglang imports from $origin, not the editable checkout in $sglang_root" >&2 - exit 1 -fi - -# Workers sharing a node and container must not patch the tree concurrently. -exec 9<"$sglang_root" -flock 9 -cd / -for name in "${patches[@]}"; do - patch="$patch_dir/$name" - if git apply --directory="${sglang_root#/}" --reverse --check "$patch" 2>/dev/null; then - echo "SGLang patch $name: already applied" - else - git apply --directory="${sglang_root#/}" --verbose "$patch" - echo "SGLang patch $name: applied" - fi -done diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch deleted file mode 100644 index a1567c7179..0000000000 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/dsv41flash-sglang-patches/sglang-pr44623.patch +++ /dev/null @@ -1,290 +0,0 @@ -diff --git a/python/sglang/srt/arg_groups/deepseek_v4_hook.py b/python/sglang/srt/arg_groups/deepseek_v4_hook.py -index e9e04ec..21c6054 100644 ---- a/python/sglang/srt/arg_groups/deepseek_v4_hook.py -+++ b/python/sglang/srt/arg_groups/deepseek_v4_hook.py -@@ -295,7 +295,6 @@ def validate_deepseek_v41_features(server_args: ServerArgs) -> None: - # ROCm replays hits in a forward of their own, which only the hitting - # attention-DP ranks would enter; CUDA folds them into the extend. - ("DP attention on ROCm", get_platform().is_hip and attn_dp_enabled_of(cfg)), -- ("external cache linker", cfg.enable_unified_cache_external_linker), - ("unified memory", cfg.enable_unified_memory), - # Decode receives the last SWA window from prefill and never replays. - ("PD decode", cfg.disaggregation_mode == "decode"), -diff --git a/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py b/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py -index d552662..01219f7 100644 ---- a/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py -+++ b/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py -@@ -794,14 +794,10 @@ def _dsv4_indexer_regions(kvcache: Any, page_size: int) -> list[_IndexerRegion]: - ] - - --def _dsv4_low_ratio_entries( -- kvcache: Any, page_size: int, num_host_pages: int, transfer_layer_id_max: int --): -- """Mirror each shared source once, in FULL-page units. Prefixes end on an even -- page boundary, so ratio-2's request-scoped ring is rebuilt, not cached.""" -+def _dsv4_low_ratio_regions(kvcache: Any, page_size: int): -+ """Expose shared source KV and indexer buffers in FULL-page units.""" - import torch - -- entries = [] - for ratio, names in ( - ( - 1, -@@ -852,24 +848,35 @@ def _dsv4_low_ratio_entries( - ) - regions.append((name, index_pool, rows)) - for name, device_pool, buffers in regions: -- entries.append( -- build_pool_entry( -- name=name, -- host_pool=DeepSeekV4PagedHostPool( -- pool_name=str(name), -- device_buffers=buffers, -- item_bytes=buffers[0].shape[1] * buffers[0].element_size(), -- num_host_pages=num_host_pages, -- slot_page_size=page_size, -- layout=get_memory().hicache_mem_layout, -- allocator_type=_get_allocator_type(), -- page_aligned_only=True, -- ), -- device_pool=device_pool, -- layer_mapping=layer_mapping, -- transfer_layer_id_max=transfer_layer_id_max, -- ) -+ yield name, device_pool, buffers, layer_mapping -+ -+ -+def _dsv4_low_ratio_entries( -+ kvcache: Any, page_size: int, num_host_pages: int, transfer_layer_id_max: int -+): -+ """Mirror shared KV; ratio-2 request state is rebuilt at page boundaries.""" -+ entries = [] -+ for name, device_pool, buffers, layer_mapping in _dsv4_low_ratio_regions( -+ kvcache, page_size -+ ): -+ entries.append( -+ build_pool_entry( -+ name=name, -+ host_pool=DeepSeekV4PagedHostPool( -+ pool_name=str(name), -+ device_buffers=buffers, -+ item_bytes=buffers[0].shape[1] * buffers[0].element_size(), -+ num_host_pages=num_host_pages, -+ slot_page_size=page_size, -+ layout=get_memory().hicache_mem_layout, -+ allocator_type=_get_allocator_type(), -+ page_aligned_only=True, -+ ), -+ device_pool=device_pool, -+ layer_mapping=layer_mapping, -+ transfer_layer_id_max=transfer_layer_id_max, - ) -+ ) - return entries - - -diff --git a/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py b/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py -index 9d66379..334f0df 100644 ---- a/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py -+++ b/python/sglang/srt/mem_cache/hybrid_cache/linker_pool_assembler.py -@@ -264,6 +264,7 @@ def _build_deepseek_v4_device_pool_group( - from sglang.srt.mem_cache.hybrid_cache.hybrid_pool_assembler import ( - _dsv4_compressed_region_buffers, - _dsv4_indexer_regions, -+ _dsv4_low_ratio_regions, - _require_single_row_dsv4_swa_pages, - _resolve_deepseek_v4_layer_mappings, - ) -@@ -273,8 +274,11 @@ def _build_deepseek_v4_device_pool_group( - - mappings = _resolve_deepseek_v4_layer_mappings(kvcache) - is_unified_kv = getattr(kvcache, "_unified_kv", False) -+ # Encoder SWA replay keeps the target window per request; the draft SWA is -+ # dropped with it, a known draft accept-length cost. -+ has_paged_swa = not is_unified_kv and kvcache.swa_kv_pool is not None - entries = [] -- if not is_unified_kv: -+ if has_paged_swa: - _require_single_row_dsv4_swa_pages( - logical_page_size=kvcache.swa_page_size, - physical_page_size=kvcache.swa_kv_pool.page_size, -@@ -321,31 +325,36 @@ def _build_deepseek_v4_device_pool_group( - ) - ) - -- c4_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 4) -- c128_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 128) -- -- add( -- PoolName.DEEPSEEK_V4_C4, -- PoolName.KV, -- kvcache.c4_kv_pool, -- c4_buffers, -- mappings.c4, -- ) -- for region in _dsv4_indexer_regions(kvcache, page_size): -+ if mappings.c4: -+ c4_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 4) - add( -- region.name, -+ PoolName.DEEPSEEK_V4_C4, - PoolName.KV, -- kvcache.c4_indexer_kv_pool, -- region.device_buffers, -+ kvcache.c4_kv_pool, -+ c4_buffers, - mappings.c4, - ) -- add( -- PoolName.DEEPSEEK_V4_C128, -- PoolName.KV, -- kvcache.c128_kv_pool, -- c128_buffers, -- mappings.c128, -- ) -+ for region in _dsv4_indexer_regions(kvcache, page_size): -+ add( -+ region.name, -+ PoolName.KV, -+ kvcache.c4_indexer_kv_pool, -+ region.device_buffers, -+ mappings.c4, -+ ) -+ if mappings.c128: -+ c128_buffers, _ = _dsv4_compressed_region_buffers(kvcache, 128) -+ add( -+ PoolName.DEEPSEEK_V4_C128, -+ PoolName.KV, -+ kvcache.c128_kv_pool, -+ c128_buffers, -+ mappings.c128, -+ ) -+ for name, device_pool, buffers, layer_mapping in _dsv4_low_ratio_regions( -+ kvcache, page_size -+ ): -+ add(name, PoolName.KV, device_pool, buffers, layer_mapping) - - # fp8 two-pool unified_kv: one row index addresses both kv_buffer (fp8 nope) and - # kv_buffer_rope (bf16 rope), but unified_region_buffers() above returns the nope -@@ -384,6 +393,13 @@ def _build_deepseek_v4_device_pool_group( - ), - mappings.c4_state, - ) -+ if not has_paged_swa and any( -+ entry.indices_from_pool == PoolName.SWA for entry in entries -+ ): -+ raise ValueError( -+ "DeepSeek-V4 compress state pools take SWA page indices, which " -+ "encoder SWA replay does not keep." -+ ) - return DevicePoolGroup( - entries, - mappings.transfer_layer_id_max, -diff --git a/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py b/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py -index f95b00a..3aca7d5 100644 ---- a/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py -+++ b/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_direct_linker.py -@@ -416,6 +416,9 @@ class MooncakeDirectLinker(UnifiedCacheLinker): - def pop_completed_offload(self) -> bool: - return self.offload_results.get_nowait() - -+ def restores_swa(self) -> bool: -+ return PoolName.SWA in self.pools -+ - def reset(self) -> None: - self.pending_loads.clear() - self.load_queue.join() -diff --git a/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py b/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py -index 204d533..5a74d32 100644 ---- a/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py -+++ b/python/sglang/srt/mem_cache/storage/umbp/umbp_direct_linker.py -@@ -65,20 +65,20 @@ def _ordered_layers(entry) -> list[int]: - f"UMBP pool {entry.name} components have different layer counts." - ) - pool_layer_count = component_lengths.pop() -- if pool_layer_count != len(entry.layer_mapping): -- raise ValueError( -- f"UMBP pool {entry.name} has {pool_layer_count} buffers per component " -- f"but {len(entry.layer_mapping)} mapped layers." -- ) -- by_buffer = { -- buffer_index: logical_layer -- for logical_layer, buffer_index in entry.layer_mapping.items() -- } -+ by_buffer = {} -+ for logical_layer, mapped in entry.layer_mapping.items(): -+ indices = [mapped] if isinstance(mapped, int) else list(mapped) -+ for index in indices: -+ if index in by_buffer: -+ raise ValueError( -+ f"UMBP pool {entry.name} maps buffer {index} more than once." -+ ) -+ by_buffer[index] = logical_layer - if sorted(by_buffer) != list(range(pool_layer_count)): - raise ValueError( - f"UMBP pool {entry.name} layer mapping is not a contiguous bijection." - ) -- return [by_buffer[index] for index in range(pool_layer_count)] -+ return list(dict.fromkeys(by_buffer[index] for index in range(pool_layer_count))) - - - @dataclass -@@ -712,15 +712,17 @@ class UMBPDirectLinker(UnifiedCacheLinker): - entry = self.pools[plan.name] - items: list[list[tuple[int, int, int, int]]] = [] - for logical_layer in layers: -- buffer_index = entry.layer_mapping.get(logical_layer) -- if buffer_index is None: -+ mapped = entry.layer_mapping.get(logical_layer) -+ if mapped is None: - continue -+ indices = [mapped] if isinstance(mapped, int) else list(mapped) - items.append( - [ -- (*component[buffer_index], offsets[buffer_index]) -+ (*component[index], offsets[index]) - for component, offsets in zip( - entry.buffer_meta, entry._component_offsets - ) -+ for index in indices - ] - ) - return items -@@ -989,6 +991,9 @@ class UMBPDirectLinker(UnifiedCacheLinker): - def pop_completed_offload(self) -> bool: - return self._offload_results.get_nowait() - -+ def restores_swa(self) -> bool: -+ return PoolName.SWA in self.pools -+ - def reset(self) -> None: - self._pending.clear() - self._load_queue.join() -diff --git a/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py b/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py -index 3d31507..6661b8c 100644 ---- a/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py -+++ b/python/sglang/srt/mem_cache/unified_cache/unified_cache_linker.py -@@ -189,6 +189,10 @@ class UnifiedCacheLinker(ABC): - def close(self) -> None: - """Quiesce all transfers and release backend resources.""" - -+ def restores_swa(self) -> bool: -+ """Whether the store carries SWA pages; False tombstones restored tails.""" -+ return True -+ - - class ExternalCacheHitMarker(NamedTuple): - """What ``match`` found in the external store, consumed by ``load_back``. -@@ -228,8 +232,9 @@ class UnifiedCacheLinkerWrapper: - self.cache = cache - self.cache_linker = cache_linker - swa = cache.components.get(ComponentType.SWA) -- self._skip_swa = swa is not None and is_swa_req_ring( -- cache.token_to_kv_pool_allocator -+ self._skip_swa = swa is not None and ( -+ is_swa_req_ring(cache.token_to_kv_pool_allocator) -+ or not cache_linker.restores_swa() - ) - self._components = tuple( - component diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index 492cae991e..4d5e5d3639 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -6,12 +6,11 @@ base: model: {} frameworks: dynamo: "1.6.0.dev20261019" - sglang: "0.0.0.dev1+gab580b2e0" + sglang: "0.0.0.dev1+gc1c287a42" dynamo: install: false request_plane: tcp event_plane: zmq - setup_script: dsv41flash-sglang-patches.sh # gb300-nv serves this model from the Hub; read it from the shared mounted cache. environment: HF_HUB_CACHE: /hf_hub_cache @@ -97,6 +96,9 @@ base: served-model-name: deepseek-ai/DeepSeek-V4.1-Flash trust-remote-code: true tensor-parallel-size: 2 + tokenizer-backend: fastokens + router-mode: kv + router-session-affinity-ttl-secs: "14400" data-parallel-size: 2 expert-parallel-size: 2 enable-dp-attention: true @@ -141,6 +143,7 @@ base: SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600" SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True" SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "16384" + SGLANG_OPT_DSV41_BF16_INDEX_LOGITS: "1" NCCL_MNNVL_ENABLE: "1" NCCL_CUMEM_ENABLE: "1" MC_FORCE_MNNVL: "1" @@ -155,6 +158,9 @@ base: served-model-name: deepseek-ai/DeepSeek-V4.1-Flash trust-remote-code: true tensor-parallel-size: 16 + tokenizer-backend: fastokens + router-mode: round-robin + load-balance-method: total_tokens data-parallel-size: 16 expert-parallel-size: 16 enable-dp-attention: true @@ -232,13 +238,12 @@ override_5p1d_dep2_dep16_c2884: cuda-graph-max-bs-decode: 40 disaggregation-decode-extra-slots: 80 -override_7p1d_dep2_dep16_c4096: - name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-7p1d-dep2-dep16-c4096 +override_8p1d_dep2_dep16_c4096: + name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-8p1d-dep2-dep16-c4096 roles: prefill: - # Seven DEP2 workers leave two GPUs of the fourth node idle. nodes: 4 - workers: 7 + workers: 8 args: max-prefill-tokens: 32768 decode: diff --git a/inferencex-e2e/configs/nvidia-master.yaml b/inferencex-e2e/configs/nvidia-master.yaml index b5ac2f85a6..ef2afa2f23 100644 --- a/inferencex-e2e/configs/nvidia-master.yaml +++ b/inferencex-e2e/configs/nvidia-master.yaml @@ -9109,7 +9109,7 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg: dp-attn: false dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: - image: lmsysorg/sglang:nightly-dev-cu13-20261020-ab580b2e@sha256:0af411bb320a95bb5e1c58b4c20c9852920a19aa9f12f148f69731b0dca3b576 + image: lmsysorg/sglang:nightly-dev-cu13-20261021-c1c287a4@sha256:b0473834763b2753e069cb96c1acaf235ba7fd632ccd9bbeb44da809c437fdeb model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:gb300-nv @@ -9156,11 +9156,11 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg: dp-attn: true - spec-decoding: mtp conc-list: [4096] - srt-recipe: disagg-variants.yaml:override_7p1d_dep2_dep16_c4096 + srt-recipe: disagg-variants.yaml:override_8p1d_dep2_dep16_c4096 kv-offloading: dram kv-offload-backend: { name: mooncake, version: "0.3.13.post1" } prefill: - num-worker: 7 + num-worker: 8 tp: 2 ep: 2 dp-attn: true diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index c0ac1936c5..49fd3d49ce 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10262,9 +10262,11 @@ scenario-type: - agentic-coding description: - - "Move the disaggregated points to lmsysorg/sglang:nightly-dev-cu13-20261020-ab580b2e (digest sha256:0af411bb320a95bb5e1c58b4c20c9852920a19aa9f12f148f69731b0dca3b576) with the ai-dynamo 1.6.0.dev20261019 it ships and the ZMQ event plane. One CPU node hosts etcd, the Mooncake master, the frontend and the AgentX client." + - "Move the disaggregated points to lmsysorg/sglang:nightly-dev-cu13-20261021-c1c287a4 (digest sha256:b0473834763b2753e069cb96c1acaf235ba7fd632ccd9bbeb44da809c437fdeb) with the ai-dynamo 1.6.0.dev20261019 it ships and the ZMQ event plane. One CPU node hosts etcd, the Mooncake master, the frontend and the AgentX client." - "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 140gb on decode). The higher prefix hit rate lets c1228 run 3x DEP2 prefill instead of 4x and c2884 5x instead of 6x." - - "Add c4096 with 7x DEP2 prefill (4 nodes) and one DEP16 decode: 64 requests per decode rank, 128 extra decode slots." + - "Add c4096 with 8x DEP2 prefill (4 nodes) and one DEP16 decode: 64 requests per decode rank, 128 extra decode slots." - "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32, polling interval 8 and KV events, with the NCCL DP-batch sync env removed; c2884 and c4096 cap prefill batches at 32768 tokens." - - "A setup script applies the DeepSeek-V4.1 external linker pools (sgl-project/sglang#44623), which the image does not have yet." + - "Use the image's upstream DeepSeek-V4.1 external linker pool support (sgl-project/sglang#44623) without runtime source patches." + - "Keep KV routing and session affinity on prefill; use round-robin routing on decode so SGLang places requests with total_tokens load balancing." + - "Use fastokens in both worker roles and BF16 decode index logits." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4303 From 12d8b3c73f1b88fe9ebb0c884f8099d2c9ee6722 Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Wed, 21 Oct 2026 11:39:52 +0800 Subject: [PATCH 06/11] config: remove unused HiCache write policy from DSV4.1 linker recipe --- .../dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index 4d5e5d3639..32f2948132 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -114,7 +114,6 @@ base: enable-unified-cache-external-linker: true unified-cache-external-linker-backend: mooncake hicache-storage-backend-extra-config: '{"enable_group_semantics":true}' - hicache-write-policy: write_back hicache-ratio: 1.02 hicache-mem-layout: page_first_direct mem-fraction-static: 0.85 From 4ff6426fa42a34285e2d05739852cf5e64677275 Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Wed, 21 Oct 2026 12:54:56 +0800 Subject: [PATCH 07/11] fix: restore decode metrics in DSV4.1 disagg recipes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: prefill-only Prometheus counters were treated as deployment totals, preventing record-based token fallback and omitting decode KV metrics. Fix: enable SGLang metrics on decode for all three disaggregated variants. Validation: Ruff check and format checks passed; perf-changelog matrix validation passed. Full GPU sweep remains pending. 原因：仅 prefill 导出指标会使汇总缺少 decode 数据。修复：为三个 disagg 配置恢复 decode metrics。校验：Ruff 和 changelog 矩阵检查通过，GPU sweep 尚待验证。 --- .../dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index 32f2948132..824e25a114 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -181,7 +181,7 @@ base: reasoning-parser: auto tool-call-parser: auto watchdog-timeout: 3600 - enable-metrics: false + enable-metrics: true weight-loader-prefetch-checkpoints: true weight-loader-drop-cache-after-load: false model-loader-extra-config: '{"enable_multithread_load":true}' From 2fd15cb901b611a49254bd11f6e9e0943ba077ef Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Wed, 21 Oct 2026 17:45:57 +0800 Subject: [PATCH 08/11] fix: backport KV-event port range correction for disagg runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup. Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass. 修复 KV-event 端口与节点监控监听端口冲突，回移 srt-slurm #617；保持镜像及模型参数不变。 --- inferencex-e2e/perf-changelog.yaml | 3 ++- .../patches/617-kv-events-port-range.patch | 15 +++++++++++++++ .../runners/srt-slurm/patches/README.md | 1 + 3 files changed, 18 insertions(+), 1 deletion(-) create mode 100644 inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 58c729c8f4..b3a0e1a445 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10501,7 +10501,8 @@ - "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 140gb on decode). The higher prefix hit rate lets c1228 run 3x DEP2 prefill instead of 4x and c2884 5x instead of 6x." - "Add c4096 with 8x DEP2 prefill (4 nodes) and one DEP16 decode: 64 requests per decode rank, 128 extra decode slots." - "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32, polling interval 8 and KV events, with the NCCL DP-batch sync env removed; c2884 and c4096 cap prefill batches at 32768 tokens." - - "Use the image's upstream DeepSeek-V4.1 external linker pool support (sgl-project/sglang#44623) without runtime source patches." + - "Use the image's upstream DeepSeek-V4.1 external linker pool support (sgl-project/sglang#44623) without SGLang source patches." - "Keep KV routing and session affinity on prefill; use round-robin routing on decode so SGLang places requests with total_tokens load balancing." - "Use fastokens in both worker roles and BF16 decode index logits." + - "Backport NVIDIA/srt-slurm#617 to move KV-event allocation below 9000 and avoid monitoring listeners on port 5252." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4303 diff --git a/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch b/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch new file mode 100644 index 0000000000..0788c8a786 --- /dev/null +++ b/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch @@ -0,0 +1,15 @@ +diff --git a/src/srtctl/ports.py b/src/srtctl/ports.py +index 431bfb67..e35901bb 100644 +--- a/src/srtctl/ports.py ++++ b/src/srtctl/ports.py +@@ -22,7 +22,9 @@ FRONTEND_INTERNAL_PORT = 8180 + # Shared worker endpoint ports. + # SGLang uses this for --kv-events-config; vLLM uses it for + # DYN_VLLM_KV_EVENT_PORT. +-KV_EVENTS_PORT_BASE = 5200 ++# Keep publisher blocks below the 9000-based ephemeral range used on some clusters ++# and clear of node monitoring listeners in the 5200 range. ++KV_EVENTS_PORT_BASE = 3000 + + # SGLang backend ports. + SGLANG_HTTP_PORT_BASE = 6100 diff --git a/inferencex-e2e/runners/srt-slurm/patches/README.md b/inferencex-e2e/runners/srt-slurm/patches/README.md index d9052d1aa8..194a612fd5 100644 --- a/inferencex-e2e/runners/srt-slurm/patches/README.md +++ b/inferencex-e2e/runners/srt-slurm/patches/README.md @@ -9,3 +9,4 @@ Each patch backports an upstream fix that the shared submodule pin does not incl | Patch | Upstream PR | Fix | |-------|-------------|-----| | `573-participating-gpus.patch` | [NVIDIA/srt-slurm#573](https://github.com/NVIDIA/srt-slurm/pull/573) | Record only the GPUs workers occupy on a worker node; an exclusive node's exporter (DCGM or AMD) reports every GPU, which the collector otherwise flags as `unexpected_device`. The pin includes #573, but upstream merged a different design: it keeps every row and excuses idle worker-node GPUs only after a DCGM worker GPU UUID probe. [`infx/results/power/multinode.py`](../../../infx/results/power/multinode.py) does not apply that excusal yet, so keep this filter until it does. | +| `617-kv-events-port-range.patch` | [NVIDIA/srt-slurm#617](https://github.com/NVIDIA/srt-slurm/pull/617) | Start KV-event publisher allocation at 3000 to avoid monitoring listeners on 5252 and the 9000–65000 ephemeral range, while retaining per-DP-rank port blocks. | From 648be6382fa8adf3659b77a52ec6abc4e1443805 Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Wed, 21 Oct 2026 17:49:34 +0800 Subject: [PATCH 09/11] docs: keep the launcher port backport minimal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retain only the port-base change and concise public descriptions. 简化端口补丁及说明。 --- inferencex-e2e/perf-changelog.yaml | 2 +- .../srt-slurm/patches/617-kv-events-port-range.patch | 6 ++---- inferencex-e2e/runners/srt-slurm/patches/README.md | 2 +- 3 files changed, 4 insertions(+), 6 deletions(-) diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index b3a0e1a445..f6a0a11c10 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10504,5 +10504,5 @@ - "Use the image's upstream DeepSeek-V4.1 external linker pool support (sgl-project/sglang#44623) without SGLang source patches." - "Keep KV routing and session affinity on prefill; use round-robin routing on decode so SGLang places requests with total_tokens load balancing." - "Use fastokens in both worker roles and BF16 decode index logits." - - "Backport NVIDIA/srt-slurm#617 to move KV-event allocation below 9000 and avoid monitoring listeners on port 5252." + - "Backport NVIDIA/srt-slurm#617 to avoid monitoring listeners on port 5252 when allocating KV-event ports." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4303 diff --git a/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch b/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch index 0788c8a786..958cf91404 100644 --- a/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch +++ b/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch @@ -1,14 +1,12 @@ diff --git a/src/srtctl/ports.py b/src/srtctl/ports.py -index 431bfb67..e35901bb 100644 +index 431bfb67..e1093027 100644 --- a/src/srtctl/ports.py +++ b/src/srtctl/ports.py -@@ -22,7 +22,9 @@ FRONTEND_INTERNAL_PORT = 8180 +@@ -22,7 +22,7 @@ FRONTEND_INTERNAL_PORT = 8180 # Shared worker endpoint ports. # SGLang uses this for --kv-events-config; vLLM uses it for # DYN_VLLM_KV_EVENT_PORT. -KV_EVENTS_PORT_BASE = 5200 -+# Keep publisher blocks below the 9000-based ephemeral range used on some clusters -+# and clear of node monitoring listeners in the 5200 range. +KV_EVENTS_PORT_BASE = 3000 # SGLang backend ports. diff --git a/inferencex-e2e/runners/srt-slurm/patches/README.md b/inferencex-e2e/runners/srt-slurm/patches/README.md index 194a612fd5..d480a54e54 100644 --- a/inferencex-e2e/runners/srt-slurm/patches/README.md +++ b/inferencex-e2e/runners/srt-slurm/patches/README.md @@ -9,4 +9,4 @@ Each patch backports an upstream fix that the shared submodule pin does not incl | Patch | Upstream PR | Fix | |-------|-------------|-----| | `573-participating-gpus.patch` | [NVIDIA/srt-slurm#573](https://github.com/NVIDIA/srt-slurm/pull/573) | Record only the GPUs workers occupy on a worker node; an exclusive node's exporter (DCGM or AMD) reports every GPU, which the collector otherwise flags as `unexpected_device`. The pin includes #573, but upstream merged a different design: it keeps every row and excuses idle worker-node GPUs only after a DCGM worker GPU UUID probe. [`infx/results/power/multinode.py`](../../../infx/results/power/multinode.py) does not apply that excusal yet, so keep this filter until it does. | -| `617-kv-events-port-range.patch` | [NVIDIA/srt-slurm#617](https://github.com/NVIDIA/srt-slurm/pull/617) | Start KV-event publisher allocation at 3000 to avoid monitoring listeners on 5252 and the 9000–65000 ephemeral range, while retaining per-DP-rank port blocks. | +| `617-kv-events-port-range.patch` | [NVIDIA/srt-slurm#617](https://github.com/NVIDIA/srt-slurm/pull/617) | Move KV-event publisher allocation away from monitoring listeners on port 5252, retaining per-DP-rank port blocks. | From 70d63156d511f52e8139db3c53bc7b5281d2cf73 Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Wed, 21 Oct 2026 18:06:59 +0800 Subject: [PATCH 10/11] config: disable unused decode KV events and drop port backport MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission. Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布，保留预填充事件和双方指标，并移除端口补丁。 --- .../sglang/gb300-fp4/agentx/disagg-variants.yaml | 2 +- inferencex-e2e/perf-changelog.yaml | 4 ++-- .../patches/617-kv-events-port-range.patch | 13 ------------- inferencex-e2e/runners/srt-slurm/patches/README.md | 1 - 4 files changed, 3 insertions(+), 17 deletions(-) delete mode 100644 inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml index 824e25a114..d05ff9634b 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml @@ -185,7 +185,7 @@ base: weight-loader-prefetch-checkpoints: true weight-loader-drop-cache-after-load: false model-loader-extra-config: '{"enable_multithread_load":true}' - kv_events: true + kv_events: false sbatch_directives: mem: "0" exclusive: "" diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index f6a0a11c10..7084a0bda5 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10500,9 +10500,9 @@ - "Move the disaggregated points to lmsysorg/sglang:nightly-dev-cu13-20261021-c1c287a4 (digest sha256:b0473834763b2753e069cb96c1acaf235ba7fd632ccd9bbeb44da809c437fdeb) with the ai-dynamo 1.6.0.dev20261019 it ships and the ZMQ event plane. One CPU node hosts etcd, the Mooncake master, the frontend and the AgentX client." - "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 140gb on decode). The higher prefix hit rate lets c1228 run 3x DEP2 prefill instead of 4x and c2884 5x instead of 6x." - "Add c4096 with 8x DEP2 prefill (4 nodes) and one DEP16 decode: 64 requests per decode rank, 128 extra decode slots." - - "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32, polling interval 8 and KV events, with the NCCL DP-batch sync env removed; c2884 and c4096 cap prefill batches at 32768 tokens." + - "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32 and polling interval 8, with the NCCL DP-batch sync env removed; c2884 and c4096 cap prefill batches at 32768 tokens." - "Use the image's upstream DeepSeek-V4.1 external linker pool support (sgl-project/sglang#44623) without SGLang source patches." - "Keep KV routing and session affinity on prefill; use round-robin routing on decode so SGLang places requests with total_tokens load balancing." - "Use fastokens in both worker roles and BF16 decode index logits." - - "Backport NVIDIA/srt-slurm#617 to avoid monitoring listeners on port 5252 when allocating KV-event ports." + - "Disable decode KV-event publishing for round-robin routing; retain prefill KV events and metrics on both roles." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4303 diff --git a/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch b/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch deleted file mode 100644 index 958cf91404..0000000000 --- a/inferencex-e2e/runners/srt-slurm/patches/617-kv-events-port-range.patch +++ /dev/null @@ -1,13 +0,0 @@ -diff --git a/src/srtctl/ports.py b/src/srtctl/ports.py -index 431bfb67..e1093027 100644 ---- a/src/srtctl/ports.py -+++ b/src/srtctl/ports.py -@@ -22,7 +22,7 @@ FRONTEND_INTERNAL_PORT = 8180 - # Shared worker endpoint ports. - # SGLang uses this for --kv-events-config; vLLM uses it for - # DYN_VLLM_KV_EVENT_PORT. --KV_EVENTS_PORT_BASE = 5200 -+KV_EVENTS_PORT_BASE = 3000 - - # SGLang backend ports. - SGLANG_HTTP_PORT_BASE = 6100 diff --git a/inferencex-e2e/runners/srt-slurm/patches/README.md b/inferencex-e2e/runners/srt-slurm/patches/README.md index d480a54e54..d9052d1aa8 100644 --- a/inferencex-e2e/runners/srt-slurm/patches/README.md +++ b/inferencex-e2e/runners/srt-slurm/patches/README.md @@ -9,4 +9,3 @@ Each patch backports an upstream fix that the shared submodule pin does not incl | Patch | Upstream PR | Fix | |-------|-------------|-----| | `573-participating-gpus.patch` | [NVIDIA/srt-slurm#573](https://github.com/NVIDIA/srt-slurm/pull/573) | Record only the GPUs workers occupy on a worker node; an exclusive node's exporter (DCGM or AMD) reports every GPU, which the collector otherwise flags as `unexpected_device`. The pin includes #573, but upstream merged a different design: it keeps every row and excuses idle worker-node GPUs only after a DCGM worker GPU UUID probe. [`infx/results/power/multinode.py`](../../../infx/results/power/multinode.py) does not apply that excusal yet, so keep this filter until it does. | -| `617-kv-events-port-range.patch` | [NVIDIA/srt-slurm#617](https://github.com/NVIDIA/srt-slurm/pull/617) | Move KV-event publisher allocation away from monitoring listeners on port 5252, retaining per-DP-rank port blocks. | From 42c6e9b1e1ae4651d087ac81e32d6d5a2039911f Mon Sep 17 00:00:00 2001 From: weireweire <20922698+weireweire@users.noreply.github.com> Date: Thu, 22 Oct 2026 07:40:48 +0800 Subject: [PATCH 11/11] ci: Include unchanged GB300 DSV4.1 aggregated points in sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将现有 TP4 c1/c4/c16 和 TP2 c16 纳入本 PR 的 sweep 与 eval 选点，覆盖低延迟区间；保留原有 agg 配置及镜像。矩阵校验生成 7 个性能点和 5 个 eval，Ruff 检查通过。 --- inferencex-e2e/perf-changelog.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 7084a0bda5..ffd90e03c8 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -10494,9 +10494,11 @@ - config-keys: - dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg + - dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg scenario-type: - agentic-coding description: + - "Include the unchanged aggregated TP4 c1/c4/c16 and TP2 c16 points in the sweep and evaluation coverage, retaining their pinned October 15 image." - "Move the disaggregated points to lmsysorg/sglang:nightly-dev-cu13-20261021-c1c287a4 (digest sha256:b0473834763b2753e069cb96c1acaf235ba7fd632ccd9bbeb44da809c437fdeb) with the ai-dynamo 1.6.0.dev20261019 it ships and the ZMQ event plane. One CPU node hosts etcd, the Mooncake master, the frontend and the AgentX client." - "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 140gb on decode). The higher prefix hit rate lets c1228 run 3x DEP2 prefill instead of 4x and c2884 5x instead of 6x." - "Add c4096 with 8x DEP2 prefill (4 nodes) and one DEP16 decode: 64 requests per decode rank, 128 extra decode slots."