Repository navigation
Conversation
) `rdci dmon -e 300` (`RDC_FI_POWER_USAGE`) reports 4294967295000000 µW on AMD Instinct MI300 and newer GPUs, while `amd-smi metric --power` shows the real socket power (140 W on the MI325X in the ticket). RDC treats a socket power of 65535 as "not available", but AMD SMI has reported an unreadable power as UINT32_MAX since ROCm 7.11 (ROCm 7.2 and earlier: 65535). These GPUs have no average socket power, so RDC scaled UINT32_MAX to microwatts, and power policies triggered on every check. - Per-field fetch: UINT32_MAX (current AMD SMI) and UINT16_MAX (older AMD SMI) both mean "not available" for the average and current socket power. GPUs without an average socket power report the current one; partitions without any socket power report N/A (was the bogus value). - Bulk fetch (`RDC_BULK_FETCH_ENABLED`, off by default): a GPU whose 16-bit GPU metrics report neither an average nor a current socket power now falls through to the per-field fetch instead of reporting 65535 W. - Tests: new GPU-less `rdc_power_usage_tests` (9 tests) builds the metric fetcher against fake AMD SMI replies; `tests/ci/test_gtest_runner.py` runs it in the no-GPU CI job. - The tests share one metric fetcher and never destroy it (it keeps no state for the field they fetch): `~RdcMetricFetcherImpl()` can miss waking its worker thread and then wait for it forever. That race predates this PR and can also hang rdcd when it stops; it is not fixed here. - CHANGELOG entry under Resolved Issues. **Not changed here** - Logging is not improved here: partitions without socket power now log, on every fetch, the errors any unavailable field logs, plus "AMDSMI: cannot get POWER_USAGE". #13570 reworks that logging and edits the adjacent line. - Sibling fix for `RDC_FI_GPU_PARTITION_COUNT`: #13638. JIRA ID: ROCM-33008 - `rdc_power_usage_tests` against develop's fetcher and the fix; CI pytest wrapper against a private install. - `rdcd -u` + `rdci dmon -u -e 300` in a private network/mount namespace: 1x MI455X (ROCm 10.2.0 nightly; stock, develop, fix; non-root; bulk fetch on), 4x MI300A (gfx942) in SPX/TPX/CPX, 2x Radeon (RX 6800 XT, RX 7900 XT). - MI455X: 2000 W power policy, print-only action, `rdci policy --reg`. - Trial merges with #13386, #13570, #13638 and current develop. - Tests: 3 of 9 fail when built against develop's fetcher; all 9 pass with the fix. - Hang check: with 16 copies of the test binary running 5000 repeats each at once, all 16 hung with a fetcher per test and none with the shared fetcher. - MI455X: 4294967295000000 -> 1125000000-1129000000 µW (amd-smi: 1126-1127 W), as root and non-root. With the bulk fetch on, develop already reported the current power. - MI300A: 4294967295000000 -> 59-133 W per GPU; TPX/CPX partitions without socket power -> N/A. - Radeon (has an average socket power): unchanged. - Power policy: stock RDC reported "A Power Limit exceeds the threshold 2000 with the value 4294967295" every 5 s at ~1126 W; the fix reported nothing in 12 s. - Trial merges clean in every order and with current develop; the combined tree builds and all GPU-less tests pass. - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Conflicts resolved by keeping only this change's lines; the neighbouring develop changes are not on this branch: - projects/rdc/CHANGELOG.md: the entry goes first under Resolved Issues. - projects/rdc/tests/unit/CMakeLists.txt, projects/rdc/tests/ci/test_gtest_runner.py: without the rdc_link_status_tests target and test from #13384. Signed-off-by: Adam360x <Adam.pryor@amd.com> (cherry picked from commit d0357d2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of d0357d2 (#13637) to
release/therock-10.2.Motivation
RDC_FI_POWER_USAGEreads 4294967295000000 µW, with a success status, on AMD Instinct MI300 series and newer GPUs, whileamd-smi metric --powershows the real socket power. It was reported on MI300A with ROCm 10.1.0. The ROCm 10.2.0 pre-release behaves the same:release/therock-10.2was cut on 2026-10-14, two days before the fix merged todevelop.RDC treats a socket power of 65535 as "not available", but AMD SMI has reported an unreadable power as UINT32_MAX since ROCm 7.11. These GPUs report no average socket power, so RDC scaled UINT32_MAX to microwatts.
Technical Details
RdcMetricFetcherImpl.cc, newtests/unit/test_power_usage.cc):develop.RDC_BULK_FETCH_ENABLED, off by default): a GPU with neither reading falls through to the per-field fetch instead of reporting 65535 W.CHANGELOG.md: the entry goes first under Resolved Issues of the branch's top section, whose heading still names ROCm 10.1.0. docs(rdc): split the RDC 1.3.1 and 1.4.0 changelog sections #13874 splits that section into 10.2.0 and 10.1.0 entries ondevelopand needs its own cherry-pick.tests/unit/CMakeLists.txt,tests/ci/test_gtest_runner.py: without therdc_link_status_testslines of fix(rdc): don't report XGMI links amdsmi did not return #13384, which is not on this branch.RDC_FI_GPU_PARTITION_COUNTreports 65535 for the same reason).release/therock-10.2also needs arocm-systemssubmodule bump after this merges.Issue Tracking
JIRA ID: ROCM-33968
Test Plan
rdc_power_usage_testsfrom this branch, and again withRdcMetricFetcherImpl.ccreverted torelease/therock-10.2.pytest tests/ci/test_gtest_runner.py -k "power or client or help"against a private install of this branch.amd-smias reference:release/therock-10.2, built locally against the pre-release's AMD SMI;rdcd -uandrdci dmon -u -e 300in a private network namespace: the pre-release and this branch.rdc-ci.yml) does not run for pull requests to this branch; dispatched on the branch: https://github.com/ROCm/rocm-systems/actions/runs/42056116910.Test Result
RdcMetricFetcherImpl.ccas onrelease/therock-10.2, 3 of 9 fail.pytest: the 3 selected tests pass.test_rdc_power_usageamong them; 20 skipped, no GPU). The first two attempts stopped before checkout on a Docker Hub pull rate limit (infrastructure).release/therock-10.2build report 4294967295000000 on all 4 GPUs. This branch and the nightly report 53-66 W at idle, the same asamd-smi(current socket power; average socket power N/A).amd-smiover the same 8 s.rdci dmon: the pre-release reports 4294967295000000, this branch 53000000-66000000 µW.Submission Checklist