[Power] feat: extend dcgm energy lanes to gb dsv4/qwen3.5 fp4 / 扩展 dcgm 能耗采集到 gb dsv4 与 qwen3.5 fp4

[edwingao28]

Extends the dcgm-power measurement lanes (built in #2323/#2437/#2456) to the fp4 recipes the scale campaign needs: five DSV4 8k1k Pareto points (gb200 c1/c512/c256, gb300 c1/c1024) and the qwen3.5-fp4 gb300 low-latency ladder. Recipes opt in through the same telemetry block the fp8 lanes use; gb300 recipes take port 19401 for the known cluster-exporter collision.

Launcher changes, kept surgical:

• gb200: the existing dynamo-sglang && dsv4 branch clones the pinned producer when the resolved recipe carries an enabled dcgm-power block, and keeps its current NVIDIA/srt-slurm:main clone otherwise. The recipe overlay is unchanged.
• gb300: a new branch routes only power runs of dsv4 dynamo-sglang to the pinned producer plus the in-repo recipe overlay. Non-power dsv4 keeps the generic fallthrough exactly as before.
• The fp8-only fail-fast becomes a framework fail-fast: power lanes are validated for dynamo-sglang only; any other framework still exits early instead of silently cloning a diverging ref (aflowers branch / sa-submission).

Safety checks done before opening:

• All six recipes parse clean with the pinned producer's (6fc1bed) config loader — none of them use sa-submission-only schema fields (warmup_req_rate, spread placement, request-plane options).
• sa_bench_tokenizers.sglang_deepseek_v4 (used by the gb300 DSV4 recipes) is present at the pin.
• The gb200 DSV4 lane already runs on the main lineage the pin descends from; the qwen3.5 fp4 recipe pins dynamo by version (pip path), which the pin supports.
• Contract tests updated: the power-recipe allowlist grows to 8 entries and every entry's telemetry block is asserted per platform port. 16/16 pass locally; both launchers pass bash -n.

Scope: the config keys already exist; dispatches for validation use exact-key test-config. One master-yaml line changes: the dsv4-fp4-gb300-dynamo-sglang image pin, whose nightly (20260520) has been garbage-collected from Docker Hub — bumped to the live 20260721 nightly the B300 DSV4 family already runs (recipe container fields bumped to match). Heads-up for maintainers: several other multinode pins (gb200 DSV4 20260528, gb300 qwen3.5-fp4 20260624) are also gone from the Hub and currently survive only on per-cluster squash caches.

Canary validation (both from this branch, require-power=true):

• DSV4 gb200 1P1D TP8/TP8 c1 (4 nodes, 16 GPUs): 31081927656 — power_valid=1, manifest producer == the pin, 16/16 devices observed, publication_valid=true, prefill/decode energy split present. First multi-node DSV4 power capture.
• Qwen3.5-fp4 gb300 low-latency ladder c1–c256 (1P1D TP4, 8 GPUs, eval job included): 31081941962 — 7/7 windows power_valid=1, producer == pin, publication_valid=true. Against the merged fp8 N=3 ladder on the same topology, fp4 saves 4% J/out-token at c1 growing to 23% at c64.
• DSV4 gb300 1P1D TP4/TP4 c1 (2 nodes, 8 GPUs): 31092050733 — power_valid=1, producer == pin, publication_valid=true, on the bumped 20260721 image (fresh arm64 pull, no cache).

The first canary attempt failed at srtctl preflight ("model alias resolved to a path unavailable") and produced the second commit: these fp4 models live on compute-node-local NVMe (gb200 /mnt/numa1, gb300 /scratch), which the login-side runner cannot stat — the same situation the existing agentic/glm5.1/qwen3.5-fp8 --no-preflight exemptions cover. The power lanes join that list, scoped to USES_DCGM_POWER=1; the gb200 power lane also points MODEL_PATH at the /mnt/numa1 checkpoint the agentic path already loads from (the shared lustre alias stays untouched for the external-cluster staging).

中文:把能耗采集扩展到 scale campaign 需要的 fp4 配方(DSV4 五个点位 + qwen3.5-fp4 gb300 低延迟梯子)。launcher 改动保持外科式:gb200 dsv4 分支仅在 power 时改克隆 pinned producer;gb300 新增仅 power 生效的 dsv4 路由,非 power 路径逐字节不变;fp8 精度门换成 framework 门。六个配方已在 pin 的 loader 下解析通过,契约测试 16/16 绿。canary 链接补充后转正式 review。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Claude finished @edwingao28's task in 4m 58s —— View job

---

Review of PR #2507

• Gather context (diff, both launchers, master config, contract tests)
• Review recipe telemetry blocks (ports, exporter config)
• Review launcher routing changes (gb200/gb300)
• Review contract test updates
• Post review feedback

Found 1 blocking issue (inline comment on runners/launch_gb300-nv.sh): the two new gb300 power lanes (dsv4-fp4 and qwen3.5-fp4) now clone the pinned producer — which follows the main lineage carrying srtctl's login-node model-path preflight — but the --no-preflight condition at runners/launch_gb300-nv.sh:436 still only covers IS_AGENTIC/glm5.1/qwen3.5+fp8. Both new lanes resolve model.path to compute-node-only /scratch/models/... that the GHA runner pod can't stat, so srtctl apply should fail before sbatch. Until now these fp4 lanes ran on sa-submission-q2-2026, which (per the launcher's own comments) has no preflight, which is why this never bit. Suggested fix: add || "$USES_DCGM_POWER" == "1" to the condition. A gb300 canary run would confirm or refute this before merge.

Everything else checked out clean:

• The routing is correctly ordered — the new gb300 dsv4 power branch sits after the agentic branches and can't be shadowed; non-power dsv4 sglang still reaches the generic fallthrough.
• The awk telemetry detection matches the exact 2-space indentation the six recipe blocks use, and gb200/gb300 ports (9401 vs 19401) are consistent between recipes and the contract test's POWER_RECIPES table.
• gb200 side is fine as-is: its dsv4 and qwen3.5 model paths are on login-visible Lustre, so no preflight concern there.
• Contract test changes are statically sound (imports, helpers, and the v1.0.25 not in launcher assertion all hold against the new launcher text).

Note: this sandbox's permissions blocked running pytest / bash -n and cloning the pin (6fc1bed) to confirm the preflight exists there, so those verifications are static-only; the author reports 16/16 tests passing locally, and nothing in the test file contradicts that.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ad142d752

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[edwingao28]

Refreshed on current main and finished the loose ends:

• Merged main in (routing tables and recipes moved a lot since 2026-08-06).
• dsv4 GB300 image pins were dead — both nightlies referenced by the 7 power recipes and nvidia-master.yaml are gone from the registry. All synced to the digest-pinned lmsysorg/sglang:v0.5.14-cu130.
• Removed a dead elif in launch_gb300-nv.sh — the late dynamo-sglang && dsv4 && USES_DCGM_POWER branch was unreachable (shadowed by the earlier generic dynamo-sglang && dsv4 match); power/non-power now fork inside the single dsv4 branch. Also || exit 1 on every clone/checkout/cp in that region.
• +4 contract tests on the GB300 side: the two lane tests execute the launcher's real routing region with stubbed git/mkdir/cp and assert the power lane clones FORK_URL@PRODUCER_PIN + stamps power-producer-sha.txt, while the non-power lane clones upstream v1.0.25 with no stamp. Plus master-image↔recipes consistency and the AGENT_OPERATIONS.md wording.
• perf-changelog entry for the three config keys.

Contract suites: 20/20 passing.