test(rccl): Add symmetric memory tests in testbed framework by PJAvinash · Pull Request #11953 · ROCm/rocm-systems

PJAvinash · 2026-09-21T17:39:08Z

Motivation

This PR adds changes to testing framework to test symmetric memory for all collectives.

Technical Details

Add Single-Process Memory-Registration Coverage

Branches

Head: users/apotnuru/single-proc-mem-reg-tests
Base: users/apotnuru/testbed-process-infrastructure
Commit: 0f8a5531f49144ec315b27f6eee29f0f4ca93d47

Overall summary

This PR adds focused TestBed coverage for NCCL_SINGLE_PROC_MEM_REG_ENABLE across eight collectives.

The diff contains only unit tests and shared test helpers; it does not modify production RCCL behavior.

The enabled-path tests use symmetric-window allocations and exercise multiple data types, deliberately unaligned element counts, in-place and out-of-place layouts where supported, and HIP graph and non-graph execution. Compact flag-disabled smoke tests run the same allocation path with single-process memory registration disabled, guarding the fallback behavior.

A shared helper owns the environment setup, runtime-version prerequisite, TestBed sweep configuration, and cleanup. Environment changes are scoped and restored after each test so they do not leak into unrelated unit tests.

This PR is intentionally stacked on the TestBed infrastructure PR because it requires MEM_ALLOC_SYMMETRIC_WIN, grouped window registration, and HIP runtime-version detection added there.

Common test behavior

Forces UT_COMM_POOL=0 so each environment configuration receives fresh workers.
Forces NCCL_CUMEM_ENABLE=1.
Sets NCCL_SINGLE_PROC_MEM_REG_ENABLE to 1 for enabled-path coverage and 0 for disabled smoke coverage.
Skips when the detected HIP runtime version is below 71260540.
Uses unmanaged memory with MEM_ALLOC_SYMMETRIC_WIN.
Uses ncclSum; non-reduction collectives ignore the reduction operation.
Runs flag-disabled smoke cases with 4,314 elements, graph and non-graph modes, and both placements when the collective supports in-place execution.

File-wise summary

`projects/rccl/test/AllGatherTests.cpp`

Adds enabled single-process registration coverage for Uint64, Uint32, Bfloat16, and Uint8.
Covers 1, 3, 7, 4,314, 5,003, 1,048,575, and 1,048,576 elements.
Exercises in-place/out-of-place and graph/non-graph combinations.
Adds a flag-disabled smoke test.

`projects/rccl/test/AllReduceTests.cpp`

Adds enabled coverage for Float64, Float32, Float16, Bfloat16, FP8 E4M3, and FP8 E5M2.
Covers 1, 3, 7, 4,314, 5,003, and 10,000,001 elements.
Includes deliberately non-16-byte-aligned sizes to stress registration boundaries.
Exercises in-place/out-of-place and graph/non-graph combinations.
Adds a flag-disabled smoke test.

`projects/rccl/test/AllToAllTests.cpp`

Adds enabled non-graph coverage for Uint8, Bfloat16, Uint32, and Uint64 at 1, 4,096, and 4,314 elements.
Adds separate enabled graph coverage at 1, 4,096, 5,003, and 5,012 elements.
Keeps AllToAll out-of-place, matching its TestBed support.
Adds a flag-disabled smoke test.

`projects/rccl/test/BroadcastTests.cpp`

Adds enabled coverage for Uint8, Bfloat16, Uint32, and Uint64.
Covers 1 and 4,314 elements.
Exercises in-place/out-of-place and graph/non-graph combinations.
Adds a flag-disabled smoke test.

`projects/rccl/test/GatherTests.cpp`

Adds enabled coverage for Uint8, Bfloat16, Uint32, and Uint64 with root rank 1.
Covers 1 and 4,314 elements.
Exercises in-place/out-of-place and graph/non-graph combinations.
Adds a flag-disabled smoke test.

`projects/rccl/test/ReduceScatterTests.cpp`

Adds enabled coverage for Float64, Float32, Float16, Bfloat16, FP8 E4M3, and FP8 E5M2.
Covers 1, 3, 7, 4,314, 5,003, 1,048,575, and 1,048,576 elements.
Exercises in-place/out-of-place and graph/non-graph combinations.
Adds a flag-disabled smoke test.

`projects/rccl/test/ReduceTests.cpp`

Adds enabled coverage for Float64, Float32, Float16, Bfloat16, FP8 E4M3, and FP8 E5M2 with root rank 1.
Covers 1 and 4,314 elements.
Exercises in-place/out-of-place and graph/non-graph combinations.
Adds a flag-disabled smoke test.

`projects/rccl/test/ScatterTests.cpp`

Adds enabled non-graph and graph tests for Uint8, Bfloat16, Uint32, and Uint64.
Covers 1 and 4,096 elements.
Exercises in-place and out-of-place buffers.
Adds a flag-disabled smoke test.

`projects/rccl/test/common/ScopedEnvVar.hpp`

Adds a non-copyable RAII environment-variable helper.
Saves and restores an existing value or removes the variable if it was originally unset.
Prevents test-specific RCCL and TestBed settings from leaking into later tests.

`projects/rccl/test/common/SingleProcMemRegTestUtils.hpp`

Defines a shared enabled/disabled test configuration.
Applies scoped TestBed and RCCL environment settings.
Checks the HIP runtime prerequisite using the TestBed's isolated runtime probe.
Runs the requested collective matrix through RunSimpleSweep with symmetric-window allocation.
Provides compact, consistent flag-disabled smoke coverage.

Dependency

The PR base must remain users/apotnuru/testbed-process-infrastructure until that PR merges. Afterward, this branch can be rebased onto develop and the PR target can be changed to develop.

Issue Tracking

JIRA ID : AICOMRCCL-1169
JIRA ID : AICOMRCCL-1533

Test Plan

Test Result

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

therock-pr-bot · 2026-09-21T17:39:50Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-21T17:39:52Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

amd-argus

ARGUS

Bottom line: Worth settling before approval: neither the enabled nor the flag-off sweeps assert anything that depends on NCCL_SINGLE_PROC_MEM_REG_ENABLE, so they cannot show the feature is doing anything.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

mch

Would be good to address at least some of the argus comments. A few suggestions from me.

PJAvinash · 2026-09-25T08:57:05Z

@argus re-review

amd-argus

ARGUS

Bottom line: Worth settling before approval: the - cleanup at test_categories.yaml:157 and :163 newly excludes AllReduce.ManagedMemUserBufferRegistration, AllReduce.ROCTX and AllGather.ManagedMemUserBufferRegistration from all six full_pow2_* categories.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

PJAvinash · 2026-10-13T09:02:59Z

Passes all tests [ INFO ] [ TIMING ] TEST SUITE [ TIMING ] AllGather [ TIMING ] AllGather [ TIMING ] AllGather [ TIMING ] AllReduce [ TIMING ] AllReduce [ TIMING ] AllReduce [ TIMING ] AlltoAll [ TIMING ] AlltoAll [ TIMING ] AlltoAll [ TIMING ] AlltoAll [ TIMING ] Broadcast [ TIMING ] Broadcast [ TIMING ] Broadcast [ TIMING ] Gather [ TIMING ] Gather [ TIMING ] Gather [ TIMING ] ReduceScatter [ TIMING ] ReduceScatter [ TIMING ] ReduceScatter [ TIMING ] Reduce [ TIMING ] Reduce [ TIMING ] Reduce [ TIMING ] Scatter [ TIMING ] Scatter [ TIMING ] Scatter [ TIMING ] Scatter [ TIMING ] Total time: on gfx942 and gfx950
Total executed cases: 4428
: TEST NAME : TIME ms (STATUS)
: SingleProcMemReg : 108.98 sec (PASS)
: SingleProcMemRegDisabled: 92.72 sec (PASS)
: TOTAL : 201.69 sec (PASS)
: SingleProcMemReg : 118.32 sec (PASS)
: SingleProcMemRegDisabled: 90.31 sec (PASS)
: TOTAL : 208.63 sec (PASS)
: SingleProcMemReg : 94.05 sec (PASS)
: SingleProcMemRegGraph: 96.48 sec (PASS)
: SingleProcMemRegDisabled: 94.30 sec (PASS)
: TOTAL : 284.84 sec (PASS)
: SingleProcMemReg : 105.45 sec (PASS)
: SingleProcMemRegDisabled: 94.08 sec (PASS)
: TOTAL : 199.53 sec (PASS)
: SingleProcMemReg : 107.45 sec (PASS)
: SingleProcMemRegDisabled: 93.36 sec (PASS)
: TOTAL : 200.81 sec (PASS)
: SingleProcMemReg : 107.60 sec (PASS)
: SingleProcMemRegDisabled: 90.15 sec (PASS)
: TOTAL : 197.75 sec (PASS)
: SingleProcMemReg : 104.34 sec (PASS)
: SingleProcMemRegDisabled: 87.71 sec (PASS)
: TOTAL : 192.05 sec (PASS)
: SingleProcMemReg : 91.75 sec (PASS)
: SingleProcMemRegGraph: 93.28 sec (PASS)
: SingleProcMemRegDisabled: 90.87 sec (PASS)
: TOTAL : 275.92 sec (PASS)
29.35 minutes

CI failure is probably due to missing

ulimit -n 65536
ulimit -l unlimited in the environment for these tests.

github-actions · 2026-10-14T15:40:07Z

No undocumented RCCL or NCCL environment-variable changes.

amd-argus · 2026-10-14T16:09:22Z

@cppccppccppc @sunguobao PTAL

amd-argus

ARGUS

Bottom line: Nothing new worth raising; still open from earlier: SingleProcMemRegTestUtils.hpp:46. Worth clearing before approval.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

This PR changes 1,129 lines across 26 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Nothing new worth raising; still open from earlier: SingleProcMemRegTestUtils.hpp:59. Worth clearing before approval.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

Bottom line: Nothing new worth raising; still open from earlier: SingleProcMemRegTestUtils.hpp:46. Worth clearing before approval.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Co-authored-by: Cursor <cursoragent@cursor.com>

This reverts commit 65c3b11.

Co-authored-by: Cursor <cursoragent@cursor.com>

Temporary diagnostic to surface the failing HIP VMM call behind the AlltoAll.SingleProcMemReg teardown failure on gfx942 CI. Revert before merge. Co-authored-by: Cursor <cursoragent@cursor.com>

…workers Revert before merge. Preloads libc_malloc_debug with MALLOC_CHECK_=3 and MALLOC_PERTURB_=165 in the exec'd TestBed workers so the gfx942 CI run aborts at the first detected heap corruption instead of at an unrelated later malloc. Co-authored-by: Cursor <cursoragent@cursor.com>

Revert before merge. TestBed workers print the fatal signal and a backtrace before dying, so a silent worker crash is no longer reported only as 'Child N pipe closed unexpectedly'. Single-proc mem reg tests also log NCCL INFO for ALLOC,REG to show the last VMM operation before the crash. Co-authored-by: Cursor <cursoragent@cursor.com>

…G INFO logs" This reverts commit 64e14dc.

…mem reg workers" This reverts commit 60ef679.

… reg tests" This reverts commit e0d396c.

This reverts commit 7c36a3f.

…er dies A worker that crashed (for example SIGSEGV inside the HIP runtime) closed its pipe, and PIPE_READ called exit(1). That ended rccl-UnitTests in the middle of a test: no failure was recorded for it, the remaining tests never ran, and the parent never said how the worker died. In multi-process configurations the parent could instead hang reading a surviving worker that was blocked in a collective on the dead rank. Now a closed pipe, short read or failed write fails the current test with the worker's exit status, kills and reaps every worker of that TestBed, and leaves it with no active children, so the sweep winds down and the next test starts fresh workers. Reads also watch every active worker's pipe, so a peer's death is noticed while waiting on a different worker. RunSimpleSweep now also stops after a failed DeallocateMem; it used to carry on into the next configuration, which with no active ranks divides by zero. Co-authored-by: Cursor <cursoragent@cursor.com>

systems-assistant · 2026-10-26T14:41:43Z

TheRock Submodule Bump Activity

Newest first

2026-10-26 — Included in TheRock via ROCm/TheRock#9486.

PJAvinash requested review from a team September 21, 2026 17:39

PJAvinash self-assigned this Sep 21, 2026

therock-pr-bot Bot added the Not ready to Review label Sep 21, 2026

github-actions Bot added the project: rccl label Sep 21, 2026

PJAvinash changed the title ~~Users/apotnuru/single proc mem reg tests~~ test(rccl) Add support for symmetric memory tests in testbed framework Sep 21, 2026

PJAvinash changed the base branch from develop to users/apotnuru/testbed-process-infrastructure September 21, 2026 17:55

PJAvinash added this pull request to stack #11957 September 21, 2026 17:56

PJAvinash changed the title ~~test(rccl) Add support for symmetric memory tests in testbed framework~~ test(rccl) Add symmetric memory tests in testbed framework Sep 21, 2026

PJAvinash changed the title ~~test(rccl) Add symmetric memory tests in testbed framework~~ test(rccl): Add symmetric memory tests in testbed framework Sep 21, 2026

systems-assistant Bot added the organization: ROCm label Sep 21, 2026

PJAvinash requested review from alex-breslow-amd, atulkulk, corey-derochie-amd and mch September 21, 2026 19:02

amd-argus Bot reviewed Sep 21, 2026

View reviewed changes

charest requested review from pvallem and removed request for alex-breslow-amd and atulkulk September 23, 2026 15:51

mch approved these changes Sep 23, 2026

View reviewed changes

Comment thread projects/rccl/test/common/ScopedEnvVar.hpp Resolved

Comment thread projects/rccl/test/common/ScopedEnvVar.hpp Resolved

Comment thread projects/rccl/test/common/SingleProcMemRegTestUtils.hpp Resolved

mch reviewed Sep 23, 2026

View reviewed changes

Comment thread projects/rccl/test/common/SingleProcMemRegTestUtils.hpp Resolved

PJAvinash force-pushed the users/apotnuru/single-proc-mem-reg-tests branch from 0f8a553 to b518a40 Compare September 23, 2026 17:22

pvallem reviewed Sep 24, 2026

View reviewed changes

Comment thread projects/rccl/test/common/SingleProcMemRegTestUtils.hpp Resolved

pvallem approved these changes Sep 24, 2026

View reviewed changes

PJAvinash force-pushed the users/apotnuru/single-proc-mem-reg-tests branch 2 times, most recently from 07a28f2 to b2ca727 Compare September 25, 2026 08:56

amd-argus Bot reviewed Sep 25, 2026

View reviewed changes

Comment thread projects/rccl/test/test_categories.yaml Resolved

Comment thread projects/rccl/test/common/SingleProcMemRegTestUtils.hpp Resolved

Comment thread projects/rccl/test/common/EnvVars.cpp Resolved

Comment thread projects/rccl/test/BroadcastTests.cpp Resolved

PJAvinash force-pushed the users/apotnuru/single-proc-mem-reg-tests branch from 44832a3 to 0d00589 Compare October 13, 2026 16:05

PJAvinash force-pushed the users/apotnuru/single-proc-mem-reg-tests branch from 9048319 to df173ed Compare October 14, 2026 19:06

amd-argus Bot reviewed Oct 14, 2026

View reviewed changes

amd-argus Bot reviewed Oct 15, 2026

View reviewed changes

PJAvinash force-pushed the users/apotnuru/single-proc-mem-reg-tests branch from cd4a3e5 to 6e33154 Compare October 21, 2026 21:25

PJAvinash and others added 17 commits October 22, 2026 09:59

test(rccl): add single-process memory registration coverage

e628d3c

Co-authored-by: Cursor <cursoragent@cursor.com>

test(rccl): address review comments on single-proc mem reg tests

a3d5cb7

test(rccl): keep pow2 coverage and widen symmetric reg counts

ba70822

Address comments

50e87d9

Rebase and address comments

f246362

fix(rccl-tests): deregister buffers before releasing allocations

4021b70

fix(rccl): synchronize DDA IPC teardown across ranks

dc0aad6

Revert "fix(rccl): synchronize DDA IPC teardown across ranks"

f8d9e69

This reverts commit 65c3b11.

rccl: harden symmetric memory VMM teardown

3890fe8

Co-authored-by: Cursor <cursoragent@cursor.com>

DEBUG(rccl-tests): enable NCCL_DEBUG=WARN for single-proc mem reg tests

bc9d46d

Temporary diagnostic to surface the failing HIP VMM call behind the AlltoAll.SingleProcMemReg teardown failure on gfx942 CI. Revert before merge. Co-authored-by: Cursor <cursoragent@cursor.com>

Revert "DEBUG(rccl-tests): print worker crash backtraces and ALLOC/RE…

1dff1f6

…G INFO logs" This reverts commit 64e14dc.

Revert "DEBUG(rccl-tests): enable glibc heap checking in single-proc …

b45727f

…mem reg workers" This reverts commit 60ef679.

Revert "DEBUG(rccl-tests): enable NCCL_DEBUG=WARN for single-proc mem…

41b3353

… reg tests" This reverts commit e0d396c.

Revert "rccl: harden symmetric memory VMM teardown"

22fecfd

This reverts commit 7c36a3f.

PJAvinash force-pushed the users/apotnuru/single-proc-mem-reg-tests branch from 6e33154 to c1a4367 Compare October 22, 2026 14:59

PJAvinash merged commit deaa630 into develop Oct 25, 2026
240 of 256 checks passed

PJAvinash deleted the users/apotnuru/single-proc-mem-reg-tests branch October 25, 2026 22:02

Conversation

PJAvinash commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Add Single-Process Memory-Registration Coverage

Branches

Overall summary

Common test behavior

File-wise summary

projects/rccl/test/AllGatherTests.cpp

projects/rccl/test/AllReduceTests.cpp

projects/rccl/test/AllToAllTests.cpp

projects/rccl/test/BroadcastTests.cpp

projects/rccl/test/GatherTests.cpp

projects/rccl/test/ReduceScatterTests.cpp

projects/rccl/test/ReduceTests.cpp

projects/rccl/test/ScatterTests.cpp

projects/rccl/test/common/ScopedEnvVar.hpp

projects/rccl/test/common/SingleProcMemRegTestUtils.hpp

Dependency

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Sep 21, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PJAvinash commented Sep 25, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PJAvinash commented Oct 13, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amd-argus Bot commented Oct 14, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 26, 2026

PJAvinash commented Sep 21, 2026 •

edited

Loading

`projects/rccl/test/AllGatherTests.cpp`

`projects/rccl/test/AllReduceTests.cpp`

`projects/rccl/test/AllToAllTests.cpp`

`projects/rccl/test/BroadcastTests.cpp`

`projects/rccl/test/GatherTests.cpp`

`projects/rccl/test/ReduceScatterTests.cpp`

`projects/rccl/test/ReduceTests.cpp`

`projects/rccl/test/ScatterTests.cpp`

`projects/rccl/test/common/ScopedEnvVar.hpp`

`projects/rccl/test/common/SingleProcMemRegTestUtils.hpp`

therock-pr-bot Bot commented Sep 21, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading