test(rccl): add host microtests for the ROCm runtime and SMI wrappers by aduforest · Pull Request #13335 · ROCm/rocm-systems

aduforest · 2026-10-13T07:54:09Z

JIRA ID: AICOMRCCL-2822

What

Host-only microtests for src/misc/roctx.cc, src/misc/nvmlwrap_stub.cc, src/misc/rocmwrap.cc and src/misc/amdsmi_wrap.cc.
Complementary to test/AmdSmiWrapTests.cpp.

How

rccl-UnitTestsMicroRoctxNvml (plus a standalone-only -roctx variant that builds the ROCTX_ENABLE arm against a stand-in roctracer/roctx.h): roctx.cc and nvmlwrap_stub.cc. A malloc seam scoped to roctx.cc reaches roctxAlloc's failure paths.
rccl-UnitTestsMicroRocmwrap: the kernel-version probe, the cuMem probe and capability gates, and rocmLibraryInit's ROCr and DMA-BUF setup. kernel_config is added to rccl_mark_oracle_tus_generated so an in-tree configure works before hipify runs.
rccl-UnitTestsMicroAmdsmiWrap: init, GPU count, PCI bus-id lookups, link info, firmware version, and fabric discovery and telemetry, on the WSL2, amd_smi_lib and ARSMI backends, through a scripted FakeSmi in the file's own pfn_* table.
Pins 6 production bugs as KnownBug_ tests:
- roctx decodes an unmapped nvtx payload type as INT and adds it to its map.
- rocmLibraryInit leaves the DMA-BUF export gate armed when the kernel-config check fails.
- rocmLibraryInit still enables DMA-BUF when /proc/kallsyms can't be read.
- amd_smi_getNumDevice counts CPU processors as GPUs.
- amd_smi_getDeviceIndexByPciBusId keeps only the device number's low hex digit (AICOMRCCL-2935).
- amd_smi_getLinkInfo logs an unknown index, then queries a null handle and returns success.
Not pinned: stringify() overflows its buffer past MAX_MESSAGE_LENGTH (AICOMRCCL-2914). Not covered: the in-window arms of ncclCuMemEnable/ncclCuMemHostEnable, which only compile on ROCm 7.12+.

Coverage

Every line of roctx.cc, nvmlwrap_stub.cc and amdsmi_wrap.cc; rocmwrap.cc 209/212 lines (ROCm 7.1).

Mutation Testing

76 mutations, 75 killed; the survivor is equivalent (dropping amdsmi_wrap's per-thread fast path only skips a lock).

Test Plan

Standalone build, each × 20 shuffled: rccl-UnitTestsMicroRoctxNvml and -roctx 31/31 each, rccl-UnitTestsMicroRocmwrap 35/35, rccl-UnitTestsMicroAmdsmiWrap 69/69.
CI runs 68 AmdsmiWrap tests: ProcessorHandle_MissingSymbolFailsQuietly only compiles without the amd_smi headers.

aduforest · 2026-10-13T07:54:21Z

@argus review

therock-pr-bot · 2026-10-13T07:55:09Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

amd-argus

ARGUS

Bottom line: Looks approval ready. Everything open is minor; the three sanitizer-only checks at roctx-nvml-stub-test.cc:166 are the one I would read first.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

aduforest · 2026-10-13T16:41:11Z

Precheckin's "Failed in Compile stage" was a real link error my standalone build couldn't reproduce. In-tree builds copy the rccl target's COMPILE_DEFINITIONS (PRIVATE ones included) into test targets, so ROCTX_ENABLE reaches this binary whenever roctracer is present. roctx.cc then calls roctxRangePushA/roctxRangePop, and nothing on the micro-test link line provides them. dbbfd41 defines both in the test TU to record calls, and the range tests now cover both arms, including the exact message pushed. Verified locally with -DROCTX_ENABLE and a stand-in roctracer header: 30/30 under ASan/LSan, 100% coverage, 4 more mutations killed.

@argus review

amd-argus

ARGUS

Bottom line: Worth settling before approval: rccl-UnitTestsMicroRoctxNvml never gets ROCTX_ENABLE in the standalone arm run_host_tests.sh builds, so the push/pop assertions added this round compile out in CI.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

aduforest · 2026-10-13T19:40:16Z

@argus review

amd-argus

ARGUS

Bottom line: LGTM once MICROTEST_README.md:222 is corrected: in-tree the base binary inherits ROCTX_ENABLE, so it covers the roctracer arm, not the WARN fallback.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

aduforest · 2026-10-13T22:21:52Z

@argus review

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Covers the kernel-version probe, the cuMem VMM functional probe and its cleanup, the cuMem capability gates, the out-of-window NCCL_CUMEM_ENABLE / cuMem-host arms, and rocmLibraryInit's ROCr version and DMA-BUF setup, in a new rccl-UnitTestsMicroRocmwrap binary. Co-Authored-By: Claude <noreply@anthropic.com>

…dress ARGUS findings on rocmwrap microtests Co-Authored-By: Claude <noreply@anthropic.com>

…tub.cc New rccl-UnitTestsMicroRoctxNvml binary compiling both files for real (comm_fakes.cc stubs roctx_scoped_range_in for every other target). The test TU supplies its own ncclDebugLog and g_loadParam, so no shared fakes are linked; param_redirect.h makes RCCL_LOG_ROCTX controllable per test, and a malloc seam scoped to roctx.cc reaches both roctxAlloc failure paths. stringify() overflows its 1024-byte buffer once a message is truncated (confirmed under ASan); filed as AICOMRCCL-2914 and not pinned here. Co-Authored-By: Claude <noreply@anthropic.com>

In-tree builds copy the rccl target's COMPILE_DEFINITIONS (PRIVATE ones included) into every test target, so ROCTX_ENABLE reaches this binary whenever roctracer is found. roctx.cc then calls roctxRangePushA/Pop, which nothing on the micro-test link line provides; that was the precheckin "Failed in Compile stage". The standalone build never defines it, so this did not reproduce locally. Define both entry points in the test TU to record calls, and make the scoped-range tests hold in either build: push/pop events and the exact message pushed with ROCTX_ENABLE, the WARN fallback without it. Co-Authored-By: Claude <noreply@anthropic.com>

…pop checks, address ARGUS round 2 Co-Authored-By: Claude <noreply@anthropic.com>

…l README entry Co-Authored-By: Claude <noreply@anthropic.com>

aduforest · 2026-10-14T18:27:31Z

@argus review

…rt gate stays unarmed when ROCr lacks DMA-BUF Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Covers init/shutdown caching, GPU count, PCI bus-id lookups both ways, link info, firmware version, fabric discovery (including the amd_smi fabric ABI probe and its sysfs fallbacks) and telemetry, across the WSL2, amd_smi_lib and ARSMI backends, in a new rccl-UnitTestsMicroAmdsmiWrap binary. Co-Authored-By: Claude <noreply@anthropic.com>

Seed a nonzero firmware version in the WSL2 test so only the early return can produce 0, drive install/clear/dlsym from one symbol table and check every pfn slot after dlopen, count calls to prove getProcessorHandle's early returns, name the typed-fabric-layout condition once, and drop a dead store in the too-many-devices test. Co-Authored-By: Claude <noreply@anthropic.com>

Give the lib PCIe case a nonzero bandwidth pair and add an ARSMI PCIe link with a direct-link weight, so both XGMI type checks are needed. Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Add a hip-id-1 CPU after GPU1 so the link lookup needs its GPU type check, and populate the out-of-range fabric slot so the bounds checks are needed. Co-Authored-By: Claude <noreply@anthropic.com>

…vml-host-microtests # Conflicts: # projects/rccl/test/host/CMakeLists.txt # projects/rccl/test/host/MICROTEST_README.md # projects/rccl/tools/scripts/test_runner/configs/ci-precheckin.json # projects/rccl/tools/scripts/test_runner/lib/test_executor.py

…vml-host-microtests # Conflicts: # projects/rccl/test/host/CMakeLists.txt # projects/rccl/test/host/MICROTEST_README.md # projects/rccl/test/host/run_host_tests.sh # projects/rccl/tools/scripts/test_runner/configs/ci-precheckin.json # projects/rccl/tools/scripts/test_runner/lib/test_executor.py

aduforest · 2026-10-15T19:31:50Z

@argus review

amd-argus

ARGUS

This PR changes 3,075 lines across 12 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

…comrccl-2822-roctx-nvml-host-microtests

aduforest · 2026-10-16T17:11:16Z

@argus review

amd-argus

ARGUS

This PR changes 3,074 lines across 12 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Also reset g_legacyTelemName in Reset() and drop the Jira key from the roctx-nvml-stub-test.cc header. Co-Authored-By: Claude <noreply@anthropic.com>

aduforest · 2026-10-16T21:27:29Z

@argus review

amd-argus

ARGUS

This PR changes 3,074 lines across 12 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

aduforest · 2026-10-16T22:04:44Z

@argus re-review

amd-argus · 2026-10-16T22:07:16Z

@aduforest the current head (48a7554e) already has a review from me above; nothing has changed since then for me to look at again. Push a change and I'll pick it up, or ask again once you have.

aduforest · 2026-10-16T22:29:57Z

@argus review

amd-argus · 2026-10-16T22:35:13Z

@aduforest the current head (48a7554e) already has a review from me above; nothing has changed since then for me to look at again. Push a change and I'll pick it up, or ask again once you have.

# Conflicts: # projects/rccl/test/host/MICROTEST_README.md

aduforest · 2026-10-19T17:26:37Z

@argus review

amd-argus

ARGUS

This PR changes 3,102 lines across 12 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

# Conflicts: # projects/rccl/test/host/MICROTEST_README.md # projects/rccl/tools/scripts/test_runner/lib/test_executor.py

aduforest · 2026-10-19T17:53:46Z

@argus review

amd-argus

ARGUS

This PR changes 3,092 lines across 12 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

systems-assistant · 2026-10-21T13:32:29Z

TheRock Submodule Bump Activity

Newest first

2026-10-21 — Included in TheRock via ROCm/TheRock#9398.

github-actions Bot added the project: rccl label Oct 13, 2026

systems-assistant Bot added the organization: ROCm label Oct 13, 2026

amd-argus Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread projects/rccl/test/host/CMakeLists.txt Resolved

Comment thread projects/rccl/test/host/roctx-nvml-stub-test.cc Resolved

Comment thread projects/rccl/test/host/roctx-nvml-stub-test.cc Resolved

aduforest force-pushed the users/aduforest/aicomrccl-2822-roctx-nvml-host-microtests branch from dbbfd41 to 8440af2 Compare October 13, 2026 19:38

amd-argus Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread projects/rccl/test/host/MICROTEST_README.md Resolved

aduforest mentioned this pull request Oct 13, 2026

test(rccl): add host microtests for misc/rocmwrap.cc #13418

Closed

amd-argus Bot reviewed Oct 13, 2026

View reviewed changes

aduforest marked this pull request as ready for review October 14, 2026 02:55

aduforest requested review from a team October 14, 2026 02:55

aduforest and others added 6 commits October 14, 2026 10:51

test(rccl): mark kernel_config.cc generated for in-tree configure, ad…

17147e8

…dress ARGUS findings on rocmwrap microtests Co-Authored-By: Claude <noreply@anthropic.com>

test(rccl): build roctx.cc's ROCTX_ENABLE arm standalone, split push/…

c9c918a

…pop checks, address ARGUS round 2 Co-Authored-By: Claude <noreply@anthropic.com>

docs(rccl): note the in-tree ROCTX_ENABLE inheritance in the RoctxNvm…

090feb0

…l README entry Co-Authored-By: Claude <noreply@anthropic.com>

aduforest force-pushed the users/aduforest/aicomrccl-2822-roctx-nvml-host-microtests branch from 68e6d90 to 090feb0 Compare October 14, 2026 18:27

test(rccl): pin cuMem off in the DMABUF_ENABLE=0 test, check the expo…

5083ca2

…rt gate stays unarmed when ROCr lacks DMA-BUF Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus Bot reviewed Oct 14, 2026

View reviewed changes

aduforest mentioned this pull request Oct 14, 2026

test(rccl): add host microtests for misc/amdsmi_wrap.cc #13501

Closed

aduforest and others added 2 commits October 14, 2026 14:04

test(rccl): pin the XGMI-only gates in amdsmi_wrap link info

05c89f6

Give the lib PCIe case a nonzero bandwidth pair and add an ARSMI PCIe link with a direct-link weight, so both XGMI type checks are needed. Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus Bot reviewed Oct 15, 2026

View reviewed changes

aduforest and others added 3 commits October 15, 2026 09:36

test(rccl): pin amdsmi_wrap's link-info type check and fabric bounds

74e100d

Add a hip-id-1 CPU after GPU1 so the link lookup needs its GPU type check, and populate the out-of-range fabric slot so the bounds checks are needed. Co-Authored-By: Claude <noreply@anthropic.com>

aduforest changed the title ~~test(rccl): add host microtests for misc/roctx.cc and misc/nvmlwrap_stub.cc~~ test(rccl): add host microtests for the ROCm runtime and SMI wrappers Oct 15, 2026

amd-argus Bot reviewed Oct 15, 2026

View reviewed changes

pvallem approved these changes Oct 15, 2026

View reviewed changes

Merge remote-tracking branch 'origin/develop' into users/aduforest/ai…

086d1c2

…comrccl-2822-roctx-nvml-host-microtests

amd-argus Bot reviewed Oct 16, 2026

View reviewed changes

test(rccl): name the roctx/rocmwrap/amdsmi pins KnownBug_

48a7554

Also reset g_legacyTelemName in Reset() and drop the Jira key from the roctx-nvml-stub-test.cc header. Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus Bot reviewed Oct 16, 2026

View reviewed changes

Merge remote-tracking branch 'origin/develop' into agent/2822-pr

a87f794

# Conflicts: # projects/rccl/test/host/MICROTEST_README.md

amd-argus Bot reviewed Oct 19, 2026

View reviewed changes

Merge remote-tracking branch 'origin/develop' into agent/2822-pr

e5ec934

# Conflicts: # projects/rccl/test/host/MICROTEST_README.md # projects/rccl/tools/scripts/test_runner/lib/test_executor.py

amd-argus Bot reviewed Oct 19, 2026

View reviewed changes

aduforest merged commit 60f0ac2 into develop Oct 20, 2026
72 of 76 checks passed

aduforest deleted the users/aduforest/aicomrccl-2822-roctx-nvml-host-microtests branch October 20, 2026 00:59

This was referenced Oct 21, 2026

[rocjitsu] Correct legacy floating min/max and median selection #13898

Merged

[rocjitsu] Expose backed CDNA3 and CDNA4 hardware identity fields #13899

Merged

[rocjitsu] Implement scratch LDS transfers and scalar cache discard #13900

Merged

Conversation

aduforest commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Coverage

Mutation Testing

Test Plan

Uh oh!

aduforest commented Oct 13, 2026

Uh oh!

therock-pr-bot Bot commented Oct 13, 2026

✅ All Policy Checks Passed

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aduforest commented Oct 13, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aduforest commented Oct 13, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aduforest commented Oct 13, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 14, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 15, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 16, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 16, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 16, 2026

Uh oh!

amd-argus Bot commented Oct 16, 2026

Uh oh!

aduforest commented Oct 16, 2026

Uh oh!

amd-argus Bot commented Oct 16, 2026

Uh oh!

aduforest commented Oct 19, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 13, 2026 •

edited

Loading