Emit per-role average power on multinode disaggregated runs / 多节点分离式运行输出每角色平均功率

[edwingao28]

The dashboard has had two chart axes wired for prefill_avg_power_w and decode_avg_power_w for a while, but this repo never emitted those keys — we emit per-role energy instead. Vendor-submitted rows already populate those axes, so our own disaggregated runs are the ones missing from the comparison. Emit the two keys.

Purely additive: two new keys on multinode disaggregated runs. No renames, no change to the meaning of any existing key, single-node untouched.

Definition: role energy divided by the same full formal serving window and by the role's declared GPU count — the mean board draw of that role's GPUs across the whole window, not a kernel-level phase power. Same window for both roles, because ParsedWindow is keyed by (benchmark_type, concurrency) and carries no role dimension. Dividing by the declared prefill_gpus / decode_gpus rather than a recount means a consumer holding prefill_gpu_energy_j, the window and num_prefill_gpu can re-derive the published value exactly.

Registering the keys in ROLE_METRIC_KEYS is load-bearing, not bookkeeping: that tuple feeds the _patch_agg scrub loop, so an emitted-but-unregistered key would survive as a stale value on a later power_valid: 0 re-run of the same aggregate. The hand-copied mirror in process_result.py is updated for the same reason on the internal-error path.

Validated against 32 real disaggregated aggregates (GB200 and GB300, concurrency 1 to 128, 4P+4D): the GPU-count-weighted mean of the two proposed values reproduces the published avg_power_w in 32/32, and prefill_gpu_energy_j + decode_gpu_energy_j == total_gpu_energy_j in 32/32. Example (GB200, concurrency 8): prefill 368.419 W, decode 576.040 W, weighted 472.229 W against a published avg_power_w of 472.229.

Worth noting for whoever reads these axes first: decode draws more per GPU than prefill in all 32 runs, converging only at the top of the concurrency ladder. That gap is the whole reason these two axes exist.

pytest utils/test_aggregate_power_multinode.py utils/test_process_result.py utils/test_aggregate_power.py utils/test_gb200_power_official_contract.py utils/test_gb300_power_official_contract.py — 141 passed.

中文说明

仪表板早已为 prefill_avg_power_w 和 decode_avg_power_w 接好了两条图表坐标轴，但本仓库从未输出这两个键——我们输出的是每角色的能量。厂商提交的数据行已经填充了这两条轴，因此缺席对比的反而是我们自己的分离式运行。本次改动输出这两个键。

纯增量改动：仅在多节点分离式运行上新增两个键。不涉及重命名，不改变任何现有键的含义，单节点路径不受影响。

定义：角色能量除以同一个完整的正式服务窗口，再除以该角色声明的 GPU 数量——即该角色 GPU 在整个窗口内的平均板级功耗，而非内核级的阶段功率。两个角色使用同一个窗口，因为 ParsedWindow 以 (benchmark_type, concurrency) 为键，不带角色维度。除以声明的 prefill_gpus / decode_gpus 而不是重新统计，意味着消费方只要拿到 prefill_gpu_energy_j、窗口和 num_prefill_gpu，就能精确复现发布出去的数值。

把新键注册进 ROLE_METRIC_KEYS 是必需的，不是形式工作：该元组驱动 _patch_agg 的清理循环，输出但未注册的键会在同一份 aggregate 后续以 power_valid: 0 重跑时，把上一次的旧数值残留下来。process_result.py 中那份手抄镜像出于同样原因一并更新，覆盖内部错误路径。

已针对 32 份真实分离式 aggregate 验证（GB200 与 GB300，并发 1 至 128，4P+4D）：两个新值按 GPU 数量加权平均后，32/32 精确复现已发布的 avg_power_w；prefill_gpu_energy_j + decode_gpu_energy_j == total_gpu_energy_j 同样 32/32 成立。示例（GB200，并发 8）：prefill 368.419 W，decode 576.040 W，加权 472.229 W，与已发布的 avg_power_w 472.229 一致。

一个值得先说明的现象：全部 32 次运行中，decode 的单卡功耗都高于 prefill，只有在并发梯度的顶端才收敛。这个差距正是这两条坐标轴存在的意义。

pytest utils/test_aggregate_power_multinode.py utils/test_process_result.py utils/test_aggregate_power.py utils/test_gb200_power_official_contract.py utils/test_gb300_power_official_contract.py —— 141 项通过。

[claude]

LGTM — straightforward additive metric emission following the existing pattern for role energy.

Extended reasoning...

Overview

This PR adds two new metrics, prefill_avg_power_w and decode_avg_power_w, to multinode disaggregated aggregate output. The computation is a one-line division of the already-computed role energy (role_energy["prefill"] / role_energy["decode"]) by the existing duration_s and the role's declared GPU count, guarded by the same if prefill_gpus > 0 / if decode_gpus > 0 checks used for the existing energy and joules-per-token metrics. The new keys are registered in ROLE_METRIC_KEYS in utils/aggregate_power_multinode.py and mirrored in the hand-copied _MULTINODE_ROLE_METRIC_KEYS tuple in utils/process_result.py, which is required for both the scrub loop on invalid re-runs and the internal-error fallback path. Docs in .github/AGENT_OPERATIONS.md were updated to describe the new keys. No existing keys are renamed or reinterpreted, and the single-node path is untouched.

Security risks

None. This is a pure arithmetic addition to an internal metrics-aggregation script; no new inputs, no auth/crypto/permission logic touched, no new file or network I/O paths introduced.

Level of scrutiny

Low-to-moderate is appropriate: the change is small, mechanical, and follows an existing well-established pattern in the same function (the energy and joules-per-token metrics right next to it). The surrounding file contains a lot of complex validation logic, but none of that logic was touched — the diff only adds two divisions and two tuple entries, each already covered by the existing null/zero GPU-count guards.

Other factors

Test coverage is strong: new assertions were added for the exact expected values in the existing fixture, plus a new test (test_role_watts_close_over_the_whole_deployment) that verifies the GPU-count-weighted mean of the two new values reproduces avg_total_gpu_power_w/avg_power_w under a non-uniform power ramp, which is a good regression guard against the type of bug (e.g., dividing by total device count instead of role GPU count) this metric is prone to. The PR description also cites validation against 32 real disaggregated runs. The bug hunting system found no bugs; the one candidate issue raised (duplicate role-metric logic) was a style nit, not correctness, and was already ruled out.

What was reviewed: the two new metric computations and their divide-by-zero guards, the ROLE_METRIC_KEYS/_MULTINODE_ROLE_METRIC_KEYS registration used by the stale-value scrub loop and internal-error fallback, and the new weighted-mean regression test. Confirmed the rounding path in _patch_agg (3 decimals for _w/_j suffixed keys) applies correctly to the new _w keys.