Skip to content

fix(rdc): report the socket power of MI300 series and later GPUs (#13637) - #15114

Closed
adam360x wants to merge 1 commit into
release/therock-10.2from
users/adam360x/ROCM-33968-therock-10.2
Closed

adam360x wants to merge 1 commit into
release/therock-10.2from
users/adam360x/ROCM-33968-therock-10.2

Conversation

@adam360x

@adam360x adam360x commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Cherry-pick of d0357d2 (#13637) to release/therock-10.2.

Motivation

RDC_FI_POWER_USAGE reads 4294967295000000 µW, with a success status, on AMD Instinct MI300 series and newer GPUs, while amd-smi metric --power shows the real socket power. It was reported on MI300A with ROCm 10.1.0. The ROCm 10.2.0 pre-release behaves the same: release/therock-10.2 was cut on 2026-10-14, two days before the fix merged to develop.

RDC treats a socket power of 65535 as "not available", but AMD SMI has reported an unreadable power as UINT32_MAX since ROCm 7.11. These GPUs report no average socket power, so RDC scaled UINT32_MAX to microwatts.

Technical Details

  • Library and test changes are identical to fix(rdc): report the socket power of MI300 series and later GPUs #13637 (RdcMetricFetcherImpl.cc, new tests/unit/test_power_usage.cc):
    • Per-field fetch: UINT32_MAX and UINT16_MAX both mean "not available" for the average and current socket power. GPUs without an average socket power report the current one; partitions without any socket power report N/A, and RDC logs an error on each fetch, as on develop.
    • Bulk fetch (RDC_BULK_FETCH_ENABLED, off by default): a GPU with neither reading falls through to the per-field fetch instead of reporting 65535 W.
  • Conflicts resolved by keeping only this change's lines:
  • Not included: fix(rdc): report the partition count from the partition profile #13638 (RDC_FI_GPU_PARTITION_COUNT reports 65535 for the same reason).
  • To reach ROCm 10.2.0, TheRock release/therock-10.2 also needs a rocm-systems submodule bump after this merges.

Issue Tracking

JIRA ID: ROCM-33968

Test Plan

  • GPU-less rdc_power_usage_tests from this branch, and again with RdcMetricFetcherImpl.cc reverted to release/therock-10.2.
  • pytest tests/ci/test_gtest_runner.py -k "power or client or help" against a private install of this branch.
  • 4x MI300A (gfx942, SPX), non-root, the embedded-RDC Python script from the ticket, amd-smi as reference:
  • This branch with GPU 0 under a compute load.
  • rdcd -u and rdci dmon -u -e 300 in a private network namespace: the pre-release and this branch.
  • RDC CI (rdc-ci.yml) does not run for pull requests to this branch; dispatched on the branch: https://github.com/ROCm/rocm-systems/actions/runs/42056116910.

Test Result

  • Unit tests: 9 of 9 pass; with RdcMetricFetcherImpl.cc as on release/therock-10.2, 3 of 9 fail.
  • pytest: the 3 selected tests pass.
  • RDC CI: build and GPU-less tests pass on the third attempt (61 passed, test_rdc_power_usage among them; 20 skipped, no GPU). The first two attempts stopped before checkout on a Docker Hub pull rate limit (infrastructure).
  • MI300A: ROCm 10.1.0, the 10.2.0 pre-release and the local release/therock-10.2 build report 4294967295000000 on all 4 GPUs. This branch and the nightly report 53-66 W at idle, the same as amd-smi (current socket power; average socket power N/A).
  • Under load, GPU 0: 100-135 W from RDC and 91-136 W from amd-smi over the same 8 s.
  • rdci dmon: the pre-release reports 4294967295000000, this branch 53000000-66000000 µW.
  • Not re-run on hardware: TPX/CPX partition modes. The unit tests cover partitions without a socket power, and fix(rdc): report the socket power of MI300 series and later GPUs #13637 was run on MI300A in SPX, TPX and CPX.

Submission Checklist

)

`rdci dmon -e 300` (`RDC_FI_POWER_USAGE`) reports 4294967295000000 µW on
AMD Instinct MI300 and newer GPUs, while `amd-smi metric --power` shows
the real socket power (140 W on the MI325X in the ticket).

RDC treats a socket power of 65535 as "not available", but AMD SMI has
reported an unreadable power as UINT32_MAX since ROCm 7.11 (ROCm 7.2 and
earlier: 65535). These GPUs have no average socket power, so RDC scaled
UINT32_MAX to microwatts, and power policies triggered on every check.

- Per-field fetch: UINT32_MAX (current AMD SMI) and UINT16_MAX (older
AMD SMI) both mean "not available" for the average and current socket
power. GPUs without an average socket power report the current one;
partitions without any socket power report N/A (was the bogus value).
- Bulk fetch (`RDC_BULK_FETCH_ENABLED`, off by default): a GPU whose
16-bit GPU metrics report neither an average nor a current socket power
now falls through to the per-field fetch instead of reporting 65535 W.
- Tests: new GPU-less `rdc_power_usage_tests` (9 tests) builds the
metric fetcher against fake AMD SMI replies;
`tests/ci/test_gtest_runner.py` runs it in the no-GPU CI job.
- The tests share one metric fetcher and never destroy it (it keeps no
state for the field they fetch): `~RdcMetricFetcherImpl()` can miss
waking its worker thread and then wait for it forever. That race
predates this PR and can also hang rdcd when it stops; it is not fixed
here.
- CHANGELOG entry under Resolved Issues.

**Not changed here**
- Logging is not improved here: partitions without socket power now log,
on every fetch, the errors any unavailable field logs, plus "AMDSMI:
cannot get POWER_USAGE". #13570 reworks that logging and edits the
adjacent line.
- Sibling fix for `RDC_FI_GPU_PARTITION_COUNT`: #13638.

JIRA ID: ROCM-33008

- `rdc_power_usage_tests` against develop's fetcher and the fix; CI
pytest wrapper against a private install.
- `rdcd -u` + `rdci dmon -u -e 300` in a private network/mount
namespace: 1x MI455X (ROCm 10.2.0 nightly; stock, develop, fix;
non-root; bulk fetch on), 4x MI300A (gfx942) in SPX/TPX/CPX, 2x Radeon
(RX 6800 XT, RX 7900 XT).
- MI455X: 2000 W power policy, print-only action, `rdci policy --reg`.
- Trial merges with #13386, #13570, #13638 and current develop.

- Tests: 3 of 9 fail when built against develop's fetcher; all 9 pass
with the fix.
- Hang check: with 16 copies of the test binary running 5000 repeats
each at once, all 16 hung with a fetcher per test and none with the
shared fetcher.
- MI455X: 4294967295000000 -> 1125000000-1129000000 µW (amd-smi:
1126-1127 W), as root and non-root. With the bulk fetch on, develop
already reported the current power.
- MI300A: 4294967295000000 -> 59-133 W per GPU; TPX/CPX partitions
without socket power -> N/A.
- Radeon (has an average socket power): unchanged.
- Power policy: stock RDC reported "A Power Limit exceeds the threshold
2000 with the value 4294967295" every 5 s at ~1126 W; the fix reported
nothing in 12 s.
- Trial merges clean in every order and with current develop; the
combined tree builds and all GPU-less tests pass.

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

---------

Conflicts resolved by keeping only this change's lines; the neighbouring
develop changes are not on this branch:
- projects/rdc/CHANGELOG.md: the entry goes first under Resolved Issues.
- projects/rdc/tests/unit/CMakeLists.txt,
  projects/rdc/tests/ci/test_gtest_runner.py: without the
  rdc_link_status_tests target and test from #13384.

Signed-off-by: Adam360x <Adam.pryor@amd.com>
(cherry picked from commit d0357d2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant