[PowerX] collect native TileRT 8k1k power / 采集 TileRT 8k1k 原生功耗 - #3057
[PowerX] collect native TileRT 8k1k power / 采集 TileRT 8k1k 原生功耗#3057edwingao28 wants to merge 17 commits into
Conversation
为固定序列场景增加默认关闭的功耗要求，并贯通矩阵及工作流；保持 AgentX 和评估隔离。
拆分原生多节点功耗采集契约、UTC 上下文和生命周期屏障，不启用新路径。
整合按场景启用功耗的矩阵契约。
为 TileRT 8k1k 收集原生功耗，保留本任务分配、角色退出状态及采集证据。
同步原生采集器前置交付。 # Conflicts: # perf-changelog.yaml # runners/test_native_collector_barriers.py
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
未启用功耗时跳过 TileRT 采集目录、设备设置和挂载。
同步功耗上下文产物保留规则。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e44fe07. Configure here.
| step_rc=0 | ||
| wait_owned_step "$pid" || step_rc=$? | ||
| [[ "$rc" != 0 ]] || rc=$step_rc | ||
| done |
There was a problem hiding this comment.
Successful TileRT jobs exit 127
High Severity
The EXIT handler re-waits PREFILL_SRUN_PID after the main script already reaped it. Bash then returns 127, and because that overwrites a zero status, a successful prefill becomes a failed job and decode's real exit code is dropped. This hits every TileRT launch, not only native-power 8k1k.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit e44fe07. Configure here.
整合原生采集器退出时限和单角色功耗修复。
将新增功耗回归测试与相关路径接入现有 CPU 工作流。
保留共享原生功耗与运行时测试命令的并集。
矩阵生成同时识别校验允许的字段名与别名，避免静默丢失功耗门槛。
同步矩阵字段别名的回归修复。
复用共享原生分支的信号就绪屏障，消除测试进程 exec 竞态。
保留普通固定序列与 AgentX 路由隔离、失败结果保留和 CPU 测试并集。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| for pid in "${PREFILL_SRUN_PID:-}" "${DECODE_SRUN_PID:-}"; do | ||
| [[ -n "$pid" ]] || continue | ||
| step_rc=0 | ||
| wait_owned_step "$pid" || step_rc=$? | ||
| [[ "$rc" != 0 ]] || rc=$step_rc | ||
| done |
There was a problem hiding this comment.
🔴 finish_tilert_submit's EXIT trap re-waits on PREFILL_SRUN_PID (line 152 already did wait "$PREFILL_SRUN_PID" || PREFILL_RC=$?), and a second wait on an already-reaped PID fails with bash's "not a child of this shell" (exit 127), which then overwrites rc=0 on a fully successful run because of [[ "$rc" != 0 ]] || rc=$step_rc. Every successful sweep run would exit 127 instead of 0, so CI/sweep automation treats passing benchmarks as failures. Fix: only wait_owned_step on DECODE_SRUN_PID in the trap (or track which PIDs were already reaped) so a prior explicit wait is never repeated.
Extended reasoning...
Main flow: run_role prefill ... & sets PREFILL_SRUN_PID (line 150), then wait "$PREFILL_SRUN_PID" || PREFILL_RC=$? (line 152) reaps it. exit "$PREFILL_RC" fires the EXIT trap finish_tilert_submit with rc=$PREFILL_RC (0 on success). The for-loop at lines 80-85 iterates PREFILL_SRUN_PID first and calls wait_owned_step, whose kill -0 fails immediately (process gone) then does wait "$pid" again (line 64) — bash returns 127 for a pid no longer tracked as a child. step_rc=127, and since rc was 0, rc=$step_rc sets rc=127. The later DECODE_SRUN_PID iteration can't fix it because the guard now sees rc!=0. Final exit "$rc" returns 127 even though both roles succeeded.
Verification: normal. New code (all added by this PR). On a fully successful run the main flow reaps prefill at line 152 (wait "$PREFILL_SRUN_PID" || PREFILL_RC=$?), then exit "$PREFILL_RC" (line 153, =0) fires the EXIT trap finish_tilert_submit with rc=0. The loop at lines 80-85 iterates PREFILL_SRUN_PID first and calls wait_owned_step, whose line 64 wait "$pid" || rc=$? re-waits the… | normal.…
同步结果前置的 sidecar 排除及旧版处理器失败传播。
同步共享原生契约对角色身份与边界行审计的修复。
同步原生采集器就绪及结束身份失败收据修复。
1 participant
Description
Require native measured power only for TileRT 8192/1024. Keep execution inside its owned allocation, drain collectors, and preserve both role statuses and per-node audits.
Allocation lifecycle also changes 1k1k; standard AgentX retains SRT routing.
Testing: CPU CI passed 455 cases at
608c56d06. Prior 426-pass/1-fail exposed a shared SIGINT fixture race, now fixed. Hardware sweep/evals and CODEOWNER approval remain pending.中文
仅 TileRT 8192/1024 要求原生实测功耗；始终使用本任务分配，等待采集器排空，保留两个角色状态与各节点审计。分配生命周期也影响 1k1k；标准 AgentX 保留 SRT 路由。
测试：
608c56d06的 CPU CI 全部 455 项通过。此前 426 项通过、一项失败，暴露共享 SIGINT 测试就绪竞态；已修复。硬件 sweep/eval 与 CODEOWNER 审批仍待完成。拆自 #3027，依赖 #3051、#3050。
Related Issue
Split from #3027; depends on #3051 and #3050.
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.Note
Medium Risk
Changes Slurm allocation/teardown and benchmark power collection for TileRT multinode runs; scope is gated to 8k1k + require-power but failures could affect job exit codes and sweep reliability.
Overview
Adds an opt-in
require-powerflag on fixed-sequence matrix rows (validated for 8192/1024 only) and wires it through sweep/E2E workflows so throughput jobs can demand validated GPU power while eval-only rows never inherit it.glm5.1-fp8-b200-tilert8k1k is the first enabled scenario in master config.TileRT Slurm launcher (
submit.sh/run_node.sh) is reworked to run inside the owningsallocallocation, run decode and prefill as tracked child steps, and when power is required for 8k1k: mount native collector paths, start/stop collectors vianative_power_lifecycle.sh, drain before completion, stage per-node audits, and propagate the prefill exit code through the decode done sentinel. Non-agentic TileRT on B200 nscale multinode now routes through the compat launcher that uses this path.Matrix generation, validation schema, docs (EN/ZH), perf-changelog, and CPU lifecycle tests cover the new behavior.
Reviewed by Cursor Bugbot for commit 87e074c. Bugbot is set up for automated code reviews on this repo. Configure here.