[WIP] Kimi K2.6 agentic #2158
Conversation
xinli-sw
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
xinli-sw
added
full-sweep-fail-fast
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| image: vllm/vllm-openai:nightly-09663abde0f50944a8d5ea30120666024b503faa | ||
| model: Kimi-K2.6-NVFP4 | ||
| model-prefix: kimik2.5 | ||
| runner: cluster:b300 |
There was a problem hiding this comment.
🔴 The new kimik2.5-fp4-b300-vllm-agentic-mtp config sets runner: cluster:b300 at configs/nvidia-master.yaml:2985, but that label doesn't exist in configs/runners.yaml — only cluster:b300-nv and cluster:b300-cw are defined. Because one search-space entry uses kv-offloading: dram, matrix generation calls agentic_dram_offload_gb → runner_available_cpu_dram_mib('cluster:b300', ...) which raises ValueError (no hardware entry), aborting sweep-config generation before any benchmark runs. Change to cluster:b300-nv to match the sibling non-mtp config at line 2970.
Extended reasoning...
What the bug is
The new kimik2.5-fp4-b300-vllm-agentic-mtp recipe added in this PR (configs/nvidia-master.yaml:2981-2996) declares runner: cluster:b300 at line 2985. This label does not exist anywhere in configs/runners.yaml. Grepping the runners file shows only two B300 cluster labels are defined: cluster:b300-cw (line 216) and cluster:b300-nv (line 222), and only cluster:b300-nv has a hardware: entry (line 306). The sibling non-mtp recipe kimik2.5-fp4-b300-vllm-agentic at line 2970 correctly uses cluster:b300-nv on what is clearly the same physical fleet.
How it manifests
The recipe contains a search-space entry with kv-offloading: dram, kv-offload-backend: native (line 2995). In utils/matrix_logic/generate_sweep_configs.py, generate_full_sweep calls agentic_dram_offload_gb(agentic_config, bmk, runner, runner_data) for that entry. That function calls runner_available_cpu_dram_mib('cluster:b300', runner_data) → runner_hardware_int(...) which looks up hardware['cluster:b300']. Since there is no such key (only cluster:b300-nv is registered under hardware:), runner_hardware_int raises:
ValueError: Runner 'cluster:b300' requires 'available-cpu-dram-mib' in runner hardware metadata.
Available hardware keys: cluster:b300-nv, ...
That exception propagates out of generate_full_sweep, so the entire sweep-config generation step (the get-jobs/setup job in the workflow) fails with a non-zero exit code. No benchmark job in the matrix ever dispatches — not even the non-dram entries at lines 2993/2994/2996.
Why existing code doesn't catch it
AgenticCodingConfig.validate_dram_offload_capacity in validation.py only checks that dram-utilization is set when kv-offloading == 'dram'; it does not cross-check that the recipe's runner: label actually appears in runners.yaml. Master-config validation is purely structural, so an arbitrary string is accepted for runner. The mistake only surfaces at matrix-generation time.
Impact
This is the whole point of the PR: the sweep for the new MTP recipe cannot generate configs, so no runs will occur. Even for the non-dram entries where the missing-hardware code path is not exercised, GitHub Actions would try to schedule jobs with runs-on: cluster:b300, which no self-hosted runner group carries — so they would sit in "waiting for a runner" forever and eventually time out.
Fix
One-character change: replace runner: cluster:b300 with runner: cluster:b300-nv at configs/nvidia-master.yaml:2985, matching the sibling recipe at line 2970. If the CW fleet was intended instead, use cluster:b300-cw — but note that cluster:b300-cw has no hardware: entry either, so the dram search-space entry would still fail; only cluster:b300-nv currently supports DRAM offload matrix generation.
Step-by-step proof
configs/runners.yamlline 306:hardware:only containscluster:b300-nv(nocluster:b300).run-sweep.yml/e2e-tests.yml'setup' job runspython3 utils/matrix_logic/generate_sweep_configs.py ....generate_full_sweepiterates configs; reacheskimik2.5-fp4-b300-vllm-agentic-mtp,is_multinode = False,runner = "cluster:b300".- Enters the agentic loop; for the dram entry
{tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: native, conc-list: [2, 4, 16]}. total_cpu_dram_gb = agentic_dram_offload_gb(agentic_config, bmk, "cluster:b300", runner_data).agentic_dram_offload_gb→runner_available_cpu_dram_mib("cluster:b300", runner_data)→runner_hardware_int("cluster:b300", runner_data, "available-cpu-dram-mib").runner_hardware(runner_data).get("cluster:b300", {})returns{};value = None→ raisesValueError.- Python exits non-zero;
CONFIG_JSON=$(python3 ...)fails; theget-jobs/setupstep fails; no downstream benchmark job dispatches.
| "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use\":$CPU_OFFLOAD_BYTES,\"lazy_offload\":false}}" | ||
| ) | ||
| ;; | ||
| *) echo "Error: unsupported KV_OFFLOAD_BACKEND value '$KV_OFFLOAD_BACKEND' with EAGLE3 (expected: native)" >&2; exit 1 ;; | ||
| esac | ||
| fi | ||
|
|
||
| DCP_SIZE="${DCP_SIZE:-1}" | ||
| DCP_ARGS=() | ||
| if [[ "$DCP_SIZE" -gt 1 ]]; then | ||
| DCP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE") | ||
| NUM_SPEC_TOKENS=3 | ||
| SYNTHETIC_ACCEPT_LEN=2.88 | ||
| else | ||
| NUM_SPEC_TOKENS=4 | ||
| SYNTHETIC_ACCEPT_LEN=3.24 | ||
| fi |
There was a problem hiding this comment.
thanks for the contribution @xinli-sw
can u please update recipes.vllm.ai with ur new configs?
vllm recipes doesn't currently support offloading tho the inferact guys said they are open to contributions to support mooncake store, lmcache, vllm native offloading support in vllm recipes
as in sglang cookbook recipes , they also have support for cpu offloading recipes
similar applies to DCP, doubt the inferact folks won't mind contributions to add DCP support too
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29121027372 |
xinli-sw
force-pushed
the
kimik2.6-fp4-b300-mtp
branch
from
9f46169 to
3b42122
Compare
xinli-sw
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29137748102 |
…vllm-agentic-mtp)
xinli-sw
force-pushed
the
kimik2.6-fp4-b300-mtp
branch
from
3b42122 to
e5923ca
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29138755696 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29139024185 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29140751554 |
2 participants
No description provided.