fix(rccl/gin): reject Anvil GIN peers outside the LSA team by dlamd1dai · Pull Request #13429 · ROCm/rocm-systems

dlamd1dai · 2026-10-14T01:29:42Z

This PR is rebased onto develop (e7cb4b477a, head 126528dfdf; only the CHANGELOG context changed since the tested head 7237ae7b96) after #12751, and the check lives in that PR's ginAnvilRegisterSignalSpan. Merge #14216 first: with NCCL_GIN_TYPE unset on a multi-node communicator, Anvil can still be the selected backend until #14216 drops it there, and this check would then fail ncclDevCommCreate for Device API users instead of the communicator falling back to the GIN proxy. #12841 and #12742 from the CPX stack have merged; this PR does not depend on #13235 or #13237 and can merge in any order with them. The defect was found while fixing AICOMRCCL-2387 and is follow-up 4 on #12742. The CPX hardware runs below used the CPX stack so that Anvil could initialize on partitions.

Motivation

On MI300X CPX with 8 ranks and NCCL_LSA_TEAM_SIZE=4, the Anvil SDMA GIN backend (NCCL_GIN_TYPE=7) publishes signal addresses that are not mapped on the device. At init it only warns:

signal nRanks=8 != devr->lsaSize=4
ctx->rank=4 != devr->lsaSelf=0

It then skips the connectivity check and keeps going.

The signal binding path (ginAnvilBuildSignalPeerAddrs, called from ginAnvilRegisterSignalSpan and ginAnvilBindContextSignals) builds the peer signal table as lsaSelf + (pe - ginRank) * bigSize for every GIN peer. The device indexes this table by GIN peer (remoteSignalAddr, resolveRemotePeerVa), but each entry is a slot in this rank's LSA window. That is only correct for peers in the same LSA team, where GIN position and LSA slot agree. On rank 0, entries for GIN peers 4–7 point past the end of the window. On rank 4, entries for peers 0–3 point before its start.

The ticket suggests rebasing the table on the LSA origin and extent. That would move the in-team peers' entries off their real slots, so this PR does not do that. Writing 0 for out-of-team peers is not safe either: signalPeer returns silently on a null address, so the wait would hang, and ginAnvilResolvePeerVa turns a null base into 0 + offset.

The same problem affects any NCCL_GIN_CONNECTION_RAIL GIN team with more than one rank. Its peers are on other hosts and never have a slot in the local window.

This PR makes these configurations fail at registration with a clear error. The symmetric kernels, which request GIN internally and whose GIN peers are always rail peers outside the LSA team, now skip GIN on Anvil, so applications that never ask for GIN are not affected.

Technical Details

src/gin/gin_plugin_anvil_sdma.cc:
- New helper ginAnvilFirstSignalPeerOutsideLsa(comm, nRanks, ginRank). For each GIN peer, it computes the LSA slot that the table entry points at. It accepts the peer only when that slot is inside [0, lsaSize) and lsaRankList[slot] is the peer's world rank. The GIN-to-world mapping uses the same stride as teamRankToGinRank: contiguousRanksPerHost under RAIL, otherwise 1.
- ginAnvilRegisterSignalSpan calls the helper before it builds the peer table or registers the span in the IPC table. It runs once per resource-window bind, so it covers every logical context of every pending GIN context. If a peer fails, it logs a WARN naming the peer, GIN rank and size, lsaSelf and lsaSize. The warning tells the user to set NCCL_LSA_TEAM_SIZE to cover the GIN team or select another NCCL_GIN_TYPE. Registration then returns ncclInvalidUsage, which ncclGinAnvilBindResourceWindowSignals propagates to the caller that created the device communicator (see the next list).
- The two old warnings are removed. Communicators whose GIN team matches their LSA team take the same path as before.
test/gin/GinAnvilPlugin_test.cpp:
- New helper expectSignalsUnbound. It checks that a failed registration left signals and signal_remote_addrs null in every logical context of the handle, added nothing to the IPC table, and ran no connectivity check.
- ConnCheck_SkippedWhenLsaSizeMismatch is now BindSignals_GinPeerPastLsaWindowFailsRegistration.
- New test BindSignals_GinPeerBeforeLsaWindowFailsRegistration: the CPX case seen from the second LSA team (rank 2 of 4, lsaSize 2).
- ConnCheck_SkippedOnRailConnection is now BindSignals_RailGinTeamFailsRegistration. In its layout, the rail peer's slot is inside the window but holds the wrong world rank, so the test only passes if the RAIL stride is applied.
- ConnCheck_UsesLsaTeamForBootstrap used a reversed lsaRanks list that no real communicator produces. It now uses rank 1 with lsaSelf 1 and checks that the bootstrap uses the intra-node rank and peer list.
- connectColl now takes the GIN rank.
- BindSignals_SpanBaseProbeSizedForWorldRanks (added by fix(gin): de-alias Anvil signals across logical contexts #12751) used a two-rank RAIL GIN team, which now fails registration. It now uses one host of 8 ranks, where the rail team is this rank alone, so the world (8 ranks) is still wider than the GIN team (1 rank) and the probe buffer sizing is still tested.
src/sym_kernels.cc, src/include/sym_kernels.h:
- New ncclSymkState::ginBackendUnsupported. ncclSymkInitOnce sets it when the symmetric kernels need GIN (LSA team smaller than the communicator) and ncclGetRailedGinType reports NCCL_GIN_TYPE_ANVIL_SDMA. In that case it logs an INFO line and does not add the GIN requirement to its internal device communicator, so Anvil signals are never bound during init.
- ncclSymkMask treats GIN as unavailable when the flag is set. Because the mask keeps only GIN kernels when GIN is needed, this disables the symmetric kernels for that communicator, the same as NCCL_SYM_GIN_KERNELS_ENABLE=0; collectives use the regular kernels.
- ncclGinSetDefaultBackend runs before ncclSymkInitOnce in init.cc, so the backend is known when the flag is computed.
test/host/sym-kernels-test.cc: replaces ncclGetRailedGinType with a hook. SymKernelMicrotest.GinBackendUnsupported_OnlyForAnvilSdma checks that only Anvil sets the flag (Anvil, proxy, none). SymKernelMaskTest.NeedGinTrue_GinBackendUnsupported_EverythingCleared checks that the mask is empty when GIN is needed and the flag is set.
CHANGELOG.md: added a "Resolved issues" entry. It notes that ncclDevCommCreate calls requesting GIN in these configurations used to succeed and now fail, and that the symmetric kernels no longer use GIN with Anvil, so communicator init and collectives are unaffected.

Behavior changes reviewers should know about:

Narrowed LSA team, including CPX across OAMs: with NCCL_GIN_TYPE=7 and NCCL_LSA_TEAM_SIZE smaller than the communicator, ncclDevCommCreate calls that request GIN used to succeed and now fail with ncclInvalidUsage and the new warning. ncclCommInitRank and collectives are unaffected: the symmetric kernels skip GIN on Anvil (see sym_kernels.cc above), so for these communicators they are disabled instead of failing init. On hardware, a plain all_gather_perf in this setup passes on this PR (see Test Result).
RAIL with more than one GIN rank: Device API GIN requests always fail on Anvil now. The symmetric kernels never use Anvil for it.
Multi-backend runs: with NCCL_GIN_TYPE unset, Anvil can be one of several active backends and ranks ahead of the GIN proxy. ncclGinDevCommSetup falls back between backends, but signal binding happens after that choice and has no fallback. fix(rccl/gin): drop Anvil SDMA backend on multi-node communicators #14216 drops Anvil on multi-node communicators so the proxy is chosen there; merge it first. On a single node with a narrowed LSA team, a Device API communicator that selected Anvil now fails instead of signaling through unmapped addresses. This has not been run multi-node.
Connectivity check: a communicator that passes registration reaches the skip branches for "GIN team size differs from LSA team size" and for RAIL only when its GIN team is smaller than its LSA team, or is a single-rank rail team. They are left in place.

fix(rccl/gin): drop Anvil SDMA backend on multi-node communicators #14216 (merge first) drops Anvil when comm->topo->nHosts > 1.
fix(gin): de-alias Anvil signals across logical contexts #12751 (merged) rewrote this function as ginAnvilBuildSignalPeerAddrs / ginAnvilRegisterSignalSpan. The rebase carries the check into ginAnvilRegisterSignalSpan and removes the old warnings there; the second commit adapts the tests to per-context signal spans. The branch is now three commits; an attempt to refuse out-of-team peers in createContext, and its revert, were dropped.
The buffer path's IPC-table fallback in regMrSym (RegisterVmm with lsaSelf / lsaSize) has the same index-space mismatch. The stride path in resolveRemotePeerVa usually bypasses it, and this PR does not change it.

Issue Tracking

JIRA ID: AICOMRCCL-2706

Test Plan

rccl-UnitTestsGinAnvilPlugin --gtest_filter='*BindSignals*:*ConnCheck*', then the full rccl-UnitTestsGinAnvilPlugin suite. Requires ENABLE_ROCSHMEM_GIN and ROCm 6.4 or later.
rccl-UnitTestsMicroSymKernels (host microtests).
rccl-tests alltoall_perf and all_gather_perf with NCCL_GIN_TYPE=7, on CPX ranks split across two OAMs (MI355X and MI300X) and on SPX (MI300X), each on this PR's library and on a baseline library with the plugin file from develop:
- LSA team narrowed below the communicator: expect init to fail with the new warning instead of the old "nRanks != lsaSize" warnings.
- LSA team unset: expect init and the connectivity check to pass, and -D 3 runs to complete with the data check passing.
- Plain collectives (no -D) with a narrowed LSA team: expect them to pass, with the symmetric-kernel skip logged on every rank.

Test Result

On MI355X with Linux 6.8, Anvil passes alltoall_perf -D 3 and all_gather_perf -D 3 data checks when the GIN and LSA teams match, in SPX with 8 ranks and in CPX with 4. When the LSA team is narrowed, this PR fails ncclDevCommCreate on every rank with the new warning, and plain collectives pass. On the same layout, the baseline skips the connectivity check and then hangs or fails. The MI300X host repeats the failure-path results.

With the symmetric-kernel skip (`develop` `daf7ef33e4`, head `7237ae7b96`)

Same MI355X host and build as the next section, SPX, 8 ranks, this PR's library only. Logs are under ~/pr13429/logs-port2.

Full rccl-UnitTestsGinAnvilPlugin: 48 of 48 passed.
rccl-UnitTestsMicroSymKernels: 69 of 69 passed, including the two new tests.

Run	GIN team vs. LSA team	Result	Symmetric GIN skip logged
`alltoall_perf -D 3`, `NCCL_LSA_TEAM_SIZE=8`	8 vs. 8	Signals bound, conn-check OK on all 8 ranks, 0 wrong values	0 ranks
`all_gather_perf -D 3`, `NCCL_LSA_TEAM_SIZE=8`	8 vs. 8	Signals bound, conn-check OK on all 8 ranks, 0 wrong values	0 ranks
`alltoall_perf -D 3`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Init passes; the `ncclDevCommCreate` group fails (`common.cu:2705`, `invalid usage`) with the new warning on all 8 ranks	8 ranks
`all_gather_perf -D 3`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	rccl-tests refuses the kernel before creating a device communicator ("GinHybridAllGatherKernel requires CUDA P2P connectivity across all ranks"), rc 6; the baseline library prints the same refusal	8 ranks
Plain `all_gather_perf`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Passes, 0 wrong values (failed `ncclCommInitRank` at the previous head)	8 ranks
Plain `all_gather_perf -R 2`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Passes, 0 wrong values	8 ranks
Plain `alltoall_perf -R 2`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Passes, 0 wrong values	8 ranks

Example lines:

NCCL INFO Symmetric GIN kernels disabled: the Anvil SDMA GIN backend cannot reach rail peers outside the LSA team (lsaSize=4, nRanks=8)
NCCL WARN GIN anvil-sdma: GIN peer 4 is outside the LSA team of rank 2 (ginRank=2, ginRanks=8, lsaSelf=2, lsaSize=4). Anvil SDMA can only signal LSA team peers; set NCCL_LSA_TEAM_SIZE to cover the GIN team or select another NCCL_GIN_TYPE.

The sections below are from earlier heads, before the symmetric-kernel skip. Where they say "Fails in ncclCommInitRank", this head passes instead.

After the rebase onto `develop` (`1ba7048298`, head `2cb35ac234`)

Built on the MI355X host below (gfx950, ROCm 10.1.0, Linux 6.8.0-101) with install.sh --rocshmem-gin, then -DBUILD_TESTS=ON -DGIN_ANVIL_UNIT_TESTS=ON. The baseline library uses gin_plugin_anvil_sdma.cc from develop (1ba7048298), which includes #12751. Logs are under ~/pr13429/logs-port on that host.

rccl-UnitTestsGinAnvilPlugin --gtest_filter='*BindSignals*:*ConnCheck*': 30 of 30 passed.
Full rccl-UnitTestsGinAnvilPlugin: 48 of 48 passed.

SPX, 8 ranks, one per GPU:

Run	GIN team vs. LSA team	This PR	Baseline
`alltoall_perf`, `NCCL_LSA_TEAM_SIZE=8`	8 vs. 8	Signals bound (4 logical contexts per rank), conn-check OK on all 8 ranks, 0 wrong values	Same as this PR
`all_gather_perf`, `NCCL_LSA_TEAM_SIZE=8`	8 vs. 8	Signals bound, conn-check OK on all 8 ranks, 0 wrong values	Same as this PR
`alltoall_perf`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Init fails with `invalid usage` on all 8 ranks, new warning	Old warnings and conn-check skipped on all ranks, then hangs (killed at 300 s)
`all_gather_perf`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Init fails, new warning on all 8 ranks	Old warnings and conn-check skipped, then the test fails (rc 6)
Plain `all_gather_perf`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Fails in `ncclCommInitRank`, new warning	Passes, 0 wrong values, old warnings printed
Plain `all_gather_perf`, `NCCL_LSA_TEAM_SIZE=4`, `NCCL_SYM_GIN_KERNELS_ENABLE=0`	8 vs. 4	Passes, 0 wrong values	Passes
`alltoall_perf`, `NCCL_LSA_TEAM_SIZE=4`, `NCCL_SYM_GIN_KERNELS_ENABLE=0`	8 vs. 4	Fails in the `ncclDevCommCreate` group (`common.cu:2705`), new warning	(not run)

The CPX cases were not re-run after the rebase, because the host was in SPX and changing its partition mode would disturb other users. The tables below are from the pre-rebase head.

Before the rebase

Unit tests pass (MI300X host below, ROCm 7.2.0, built at 9e444f3ab5 with install.sh --rocshmem-gin, then -DBUILD_TESTS=ON -DGIN_ANVIL_UNIT_TESTS=ON):

rccl-UnitTestsGinAnvilPlugin --gtest_filter='*BindSignals*:*ConnCheck*': 23 of 23 passed, including the three new BindSignals_*FailsRegistration tests.
Full rccl-UnitTestsGinAnvilPlugin: 37 of 37 passed.

Each hardware case ran twice: on this PR's library, and on a baseline library built from the same tree with gin_plugin_anvil_sdma.cc from develop (2a4afbe5c4). Common settings were NCCL_GIN_TYPE=7, NCCL_CUMEM_ENABLE=1 and -R 2 -D 3, unless a row says "plain" (no -D). Runs were killed at 300 s.

MI355X: `smci355-ccs-aus-m03-17`

8× MI355X (gfx950), NPS1, Linux 6.8.0-101, amdgpu 7.1-dev, ROCm 10.1.0, built at 8a97123fff. No kernel-gate bypass was needed. Logs are under ~/pr13429/logs on that host.

SPX, 8 ranks, one per GPU (this PR's tree only):

Run	GIN team vs. LSA team	This PR	Baseline
`alltoall_perf`, `NCCL_LSA_TEAM_SIZE=8`	8 vs. 8	Signals bound (`lsaSize=8`), conn-check OK on all 8 ranks, 0 wrong values	Same as this PR
`all_gather_perf`, `NCCL_LSA_TEAM_SIZE=8`	8 vs. 8	Signals bound, conn-check OK on all 8 ranks, 0 wrong values	Same as this PR
`alltoall_perf`, `NCCL_LSA_TEAM_SIZE=4`	8 vs. 4	Init fails with `invalid usage` on all 8 ranks, new warning	(not run)

CPX, 4 ranks (this PR plus the CPX stack: #12841, #13235, #13237 and #12742 at their 2026-10-14 heads, when the even-engine change was still in #13237). Ranks 0–1 are on two partitions of one OAM and ranks 2–3 on two partitions of another (HIP_VISIBLE_DEVICES=0,1,8,9, NCCL_MULTI_RANK_GPU_ENABLE=1):

Run	GIN team vs. LSA team	This PR	Baseline
`alltoall_perf`, LSA team unset	4 vs. 4	Signals bound, conn-check OK on all 4 ranks, 0 wrong values	Same as this PR
`all_gather_perf`, LSA team unset	4 vs. 4	Signals bound, conn-check OK on all 4 ranks, 0 wrong values	Same as this PR
`alltoall_perf`, `NCCL_LSA_TEAM_SIZE=2`	4 vs. 2	Init fails with `invalid usage` on all 4 ranks, new warning	Old warnings and conn-check skipped on all ranks, then hangs (killed at 300 s)
`all_gather_perf`, `NCCL_LSA_TEAM_SIZE=2`	4 vs. 2	Init fails, new warning on all 4 ranks	Old warnings and conn-check skipped, then the test fails (rc 6)
Plain `all_gather_perf`, `NCCL_LSA_TEAM_SIZE=2`	4 vs. 2	Fails in `ncclCommInitRank`, new warning	Passes, 0 wrong values, old warnings printed
Plain `all_gather_perf`, `NCCL_LSA_TEAM_SIZE=2`, `NCCL_SYM_GIN_KERNELS_ENABLE=0`	4 vs. 2	Passes, 0 wrong values	Passes
`alltoall_perf`, `NCCL_LSA_TEAM_SIZE=2`, `NCCL_SYM_GIN_KERNELS_ENABLE=0`	4 vs. 2	Fails in the `ncclDevCommCreate` group (`common.cu:2705`), new warning	(not run)

Example warning (rank 2, on the second OAM):

GIN anvil-sdma: GIN peer 0 is outside the LSA team of rank 2 (ginRank=2, ginRanks=4, lsaSelf=0, lsaSize=2). Anvil SDMA can only signal LSA team peers; set NCCL_LSA_TEAM_SIZE to cover the GIN team or select another NCCL_GIN_TYPE.

The baseline prints the warnings from the ticket on the same run, for example signal nRanks=4 != devr->lsaSize=2 (rank 0) and ctx->rank=2 != devr->lsaSelf=0 (signal peer indexing may be wrong).

MI300X: `banff-cyxtera-cx56-5`

8× MI300X (gfx942), NPS1, Linux 5.15, amdgpu 6.16.13, ROCm 10.1. Logs are under ~/pr13429/logs-r10 on that host. On this host, RCCL turns cuMem off below Linux 6.8, and without cuMem rccl-tests skips every -D 3 run. These runs therefore add an experiment-only patch that skips that kernel gate (RCCL_EXP_SKIP_KERNEL_GATE=1). On this kernel, cuMem peer mappings are broken, so the matched-team runs fail the connectivity check on both libraries. The failure path still shows, because the new check runs before any peer is signaled. NCCL_RMA_DISABLE=1 was used whenever the LSA team was narrowed, because the RMA proxy's memory registration fails on this host before GIN is reached.

Run	GIN team vs. LSA team	This PR	Baseline
SPX, 8 ranks, `NCCL_LSA_TEAM_SIZE=4`, `alltoall_perf`	8 vs. 4	Init fails with `invalid usage` on all 8 ranks with the new warning	Old warnings and conn-check skipped on all ranks, then hangs (killed at 300 s)
SPX, same, `all_gather_perf`	8 vs. 4	Init fails, new warning on all 8 ranks	Old warnings and conn-check skipped, then the test fails (rc 6)
CPX, 6 ranks, 3 on each of two OAMs, `NCCL_LSA_TEAM_SIZE=3`, `alltoall_perf`	6 vs. 3	Init fails, new warning on all 6 ranks	Old warnings and conn-check skipped, then hangs (killed at 300 s)
CPX, same 6 ranks, LSA team unset	6 vs. 6	Signals bound, conn-check runs and fails on this kernel's broken cuMem mapping	Same as this PR
SPX, 8 ranks, LSA team unset	8 vs. 8	Signals bound, conn-check runs and fails on the broken mapping	Same as this PR
SPX, 8 ranks, `NCCL_LSA_TEAM_SIZE=4`, plain `all_gather_perf`	8 vs. 4	Fails in `ncclCommInitRank`	Passes, 0 wrong values, old warnings printed
Same, `NCCL_SYM_GIN_KERNELS_ENABLE=0`	8 vs. 4	Passes	Passes
SPX, 8 ranks, `NCCL_LSA_TEAM_SIZE=4`, `alltoall_perf`, `NCCL_SYM_GIN_KERNELS_ENABLE=0`	8 vs. 4	Fails in the `ncclDevCommCreate` group	(not run)
SPX, 8 ranks, `NCCL_LSA_TEAM_SIZE=4`, plain `all_gather_perf`, `NCCL_GIN_TYPE` unset	8 vs. 4	Init fails in the GIN proxy backend (`gin_host_proxy.cc:477`), before Anvil binds	Same failure

Notes

The ticket's exact layout (8 CPX ranks, 4 on each OAM) could not be run on either host. On the CPX stack, every rank runs out of SDMA queues while connecting to 7 peers (SDMA queue budget exhausted: 7 queue(s) taken by this process + 1 requested, limit 8; on MI355X the driver returns NO_MEMORY for the 8th queue). That happens in connect, before signal binding, on this PR and on the baseline alike. On MI355X it happens with NCCL_LSA_TEAM_SIZE=8 too, whether the 8 partitions are on one OAM, split 4+4, or spread one per OAM. The 4-rank (MI355X) and 6-rank (MI300X) layouts stand in for it. The limit of 8 is the even-engine CPX budget, which fix(rocshmem): make Anvil SDMA usable on MI300X CPX and QPX, DPX up to two OAMs #12841 now ships; fix(rocshmem): skip GPU-unreachable SDMA doorbell slots in Anvil (ROCM-32598 workaround) #13237's slot-aware budget raises it to 12, and the 8-rank layout has not been retried with that budget.
The CPX stack needed one local build fix at the time: RCCL's copy list lacked anvil_engine_map.hpp. fix(rocshmem): make Anvil SDMA usable on MI300X CPX and QPX, DPX up to two OAMs #12841 now includes it.
Not run: multi-node, and the RAIL connection type on hardware.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

therock-pr-bot · 2026-10-14T01:30:35Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

amd-argus

ARGUS

Findings

5 more finding(s), not posted inline

projects/rccl/src/gin/gin_plugin_anvil_sdma.cc:752 (Low) Anvil LSA refusal runs at bind, after ncclGinDevCommSetup chose the backend, so backend fallback cannot run
projects/rccl/CHANGELOG.md:59 (Low) CHANGELOG entry ties the new init failure to NCCL_GIN_TYPE=7 only, not the unset-type case where Anvil is selected
projects/rccl/test/gin/GinAnvilPlugin_test.cpp:551 (Low) No passing bind test has comm->rank, GIN rank and lsaSelf differ, so swapped indices in the helper go undetected
projects/rccl/test/gin/GinAnvilPlugin_test.cpp:1008 (Low) ConnCheck_UsesLsaTeamForBootstrap now uses an identity lsaRanks list, so it cannot tell LSA-team from world bootstrap
projects/rccl/test/gin/GinAnvilPlugin_test.cpp:1002 (Low) ConnCheck_UsesLsaTeamForBootstrap inlines startTwoRankGin's body only to pass ginRank=1

Bottom line: Looks approval ready.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

) This PR is the **RCCL half** of the CPX GIN fix; the rocSHMEM half is #12841, now merged to `develop` as `c34c1b1f56`. `CODEOWNERS` assigns `projects/rocshmem/` to `@ROCm/rocshmem-reviewers` and `projects/rccl/` to `@ROCm/rccl-reviewers`/`@ROCm/rccl-leads`, so the change is split by owner. This half is 996 insertions across 11 files, all under `projects/rccl/`. **Rebased onto `develop` after #12841 merged.** `gin_anvil_sdma_factory.cc` calls `disconnectDevice()` and its test calls `oamMapEngineNeedsFold()` and `pciFunctionBus()`, all added by #12841, which also adds `anvil_engine_map.hpp` to RCCL's `--rocshmem-gin` copy list. That dependency is now satisfied by `develop` itself, so the branch carries only the 11 RCCL commits and no rocSHMEM files. **Replaces #12603.** Review history lives there: four rounds of Argus review ending in LGTM, with all 37 threads resolved. #12603 was opened from a fork where `RCCL PR gate` can never pass on its own. ## Summary GIN Anvil SDMA (`NCCL_GIN_TYPE=7`) failed to initialize on MI300X in CPX mode. Four defects stacked on top of each other, each masking the next. Two are in RCCL and are fixed here; the two Anvil defects were fixed in #12841. **Note on the JIRA title:** the ticket says GIN Type 6, but Anvil SDMA is `NCCL_GIN_TYPE=7` on this branch (`6` is rocSHMEM GDA). All verification used `NCCL_GIN_TYPE=7`. 1. **The RCCL GIN gate rejected every CPX partition** (`init.cc`). `fillInfo` stamps `mloPart = fn` on every CPX partition, but the `hasMloPart` carve-out only exempted partition 0, so ranks on partitions 1-7 set `comm->hasMloPart` and the gate turned GIN off. That dropped `symmetricSupport`, which routed `ncclCommWindowRegister` into `windowRegisterNonSym` and produced `hipIpcGetMemHandle failed: invalid argument`. Fix: `NCCL_GIN_MLOPART`, mirroring `NCCL_NET_GDR_MLOPART` at `graph/paths.cc:522`. The parameter is tri-state. By default a partitioned communicator keeps GIN on a single node and loses it across several: only the single-node shape has been run, and past one node admitting a partition additionally reaches the hierarchical Copy Engine path that `ncclHierCeAllowed` gates on `nNodes > 1` and `symmetricSupport` together. `1` admits partitions on any number of nodes, which is the opt-in a multi-node partitioned deployment needs; `0` restores the pre-parameter exclusion. A refused multi-node partitioned communicator names the override through `NCCL_DEBUG=INFO`, and the `GIN/RMA support` line from #11133 now reports `nNodes` beside `mloPart`. The condition is lifted into `ginGateAllows()` so it can be asserted directly. 2. **`windowRegisterNonSym` had no cuMem path** (`dev_runtime.cc`). Even with the gate fixed, the fallback only implemented legacy IPC, and GIN requires `NCCL_CUMEM_ENABLE=1`. Probe with `cuMemRetainAllocationHandle` and import through `ncclP2pImportShareableBuffer`; legacy IPC still covers genuine `cudaMalloc` buffers. Teardown branches per peer on `ipcPeerIsCuMem`. A window spanning several segments is refused outright, since the exchange publishes one handle and one size. ### Argus High findings fixed - **Resume could not re-match the import.** The import previously passed `ncclMemOffload` with the exporter's device VA as `ownerPtr`. Resume matches an imported entry against the owner-published list, which `mem_manager.cc:806` builds only from entries with `numExportedPeers > 0`, and that counter is incremented solely by `ncclP2pAllocateShareableBuffer`. `windowRegisterNonSym` exports a user buffer it never allocated, so no owner entry exists and the match at `:894` failed into `ncclSystemError`. Now imported as `ncclMemPersist` with a null owner, matching `transport/net.cc:699-701`. `ownerPtr` became dead and was removed from the record along with its two duplicated `static_assert`s. - **A failed GIN create destroyed another communicator's queues.** `gin_anvil_sdma_create`'s failure path called `anvil.disconnect()`, which clears the process-global channel map. Each communicator gets a fresh `ginState` and the connect loop deliberately reuses queues a previous create left behind, so a second create hitting the budget refusal destroyed the first's queues while their handles were already live on the GPU. Now it records the destinations it newly connected and releases only those via `disconnectDevice()`. ### Fold tests `AnvilOamMapEngine.*` asserts when the CPX fold runs, not which engine it picks. That choice is rocSHMEM's contract and differs between #12841 (even engines only) and #13237 (every engine), so it is pinned only by rocSHMEM's `anvil_engine_map_gtest`. ## Rebase onto `develop` The rebase from the old #12841 tip (`b47b4f7ccf`) onto `develop` (`c34c1b1f56`) had one conflict, in `init.cc`'s GIN/RMA log. #11133 replaced the conditional `symmetricSupport ... crossNicSupport` INFO line, which this PR had extended with `hasMloPart` and `nNodes`, with an always-on `GIN/RMA support` line that already reports `mloPart`. The resolution keeps #11133's line, adds `nNodes` to it, and keeps the `NCCL_GIN_MLOPART` multi-node hint after it. Every other file's change is identical to the pre-rebase branch. ## Test plan - [x] **Host tests on the rebased head (`fbf8a6d3c3`): 6,068 passed, 0 failed, 5 skipped** (the pre-existing `SymKernelMaskTest` skips), run through `test/host/run_host_tests.sh` in the CI image `rocm/dev-ubuntu-22.04:7.2`. Includes all `InitMicrotest.GinGate_*` and `WindowRegisterNonSym*` cases. Everything below was taken before the rebase (previous head `84a64a92dc`, before #12841 merged). Since then the only code changes on this branch are the fold-test trim above and the log-line resolution. - [x] Host tests on the split branch: 4,912 passed, 0 failed, 5 skipped across 10 binaries, with no compile errors. Run on `dell300x-ccs-aus-k13-09`. - [x] New coverage `WindowRegisterNonSymCuMemTest.VmmWindow_ExportsHandleAndImportsPeers` asserts the import is `ncclMemPersist` with null owners for both peers, plus `.RetainFails_FallsBackToLegacyIpc`, `.MultiSegmentWindow_FailsRegistrationWithoutGathering` and `.LaterSegmentAddressRangeFails_FailsRegistrationWithoutGathering`. - [x] `ginGateAllows()` asserted directly in `init-test.cc` across seven cases: an explicit `0` and `1` on both one and two nodes, the unpartitioned carve-out, and the three default cases (partitioned/one node admitted, partitioned/multi-node refused, unpartitioned/multi-node admitted). Mutating the node term to `return true` fails only `GinGate_MloPartDefaultRefusesPartitionAcrossNodes`, so the new branch is load-bearing. - [x] Gate confirmed on CPX hardware by A/B on `dell300x-ccs-aus-k13-09` (8 ranks, two OAMs, `NCCL_LSA_TEAM_SIZE=4`): `NCCL_GIN_MLOPART=1` reports `globalGinSupport 1` and `=0` reports `globalGinSupport 0`, with `hasMloPart 1` in both. `symmetricSupport` could not be exercised there: that host's driver fails the `HIP_VERSION >= 7.2.0` cuMem check, so `cuMemEnable` is 0 and `symmetricSupport` is 0 whatever the gate decides. - [x] 8-rank CPX reduce-scatter across two OAMs on `dell300x-ccs-aus-k13-09`: `Out of bounds values : 0 OK` at every size, no `hipIpcGetMemHandle` failures, no queue-create failures, no init abort. This run predates #12867 and needed `NCCL_MULTI_RANK_GPU_ENABLE=1` (see below). - [x] `install.sh -l --rocshmem-gin` builds clean (0 errors). - [x] `RCCL env-var changelog`: `NCCL_GIN_MLOPART` is described on the RCCL env-var page, so the check passes instead of asking for `[internal-env]`. - [x] The remaining `AnvilOamMapEngine.*` and `AnvilPciFunctionBus.*` cases (10 tests) compiled standalone against `anvil_engine_map.hpp` from both the #12841 tip (`b47b4f7ccf`) and #13237 (`a674a04501`): all pass on both. The removed fold-result expectations fail 5 of 8 on #12841 and pass on #13237. ## CPX partition identity (AICOMRCCL-2749): fixed by #12867 During CPX verification, `init.cc` refused a multi-rank CPX communicator with "Multiple Ranks are using the same GPU/Partition", because every CPX partition of one OAM reports the same `gpuUuid`. #12867 (merged) fixed the duplicate-GPU guard by also comparing `busId`, which carries the partition's PCI function. The 8-rank run above predates that fix. ## Follow-ups (not in this PR) 1. `NCCL_GIN_MLOPART` gate host assertion at the real call site. The init rung still cannot reach `init.cc`'s connect block: `ncclTopoComputeP2pChannels`, `rcclCommSetP2pShiftSize`, `ncclProfilerPluginInit`, `ncclTransportCheckP2pType` and `ncclProxyCreate` are fail-loud in between. 2. Multi-segment VMM window support in `windowRegisterNonSym`, which needs per-segment handles in `ExchangeEntry`. 3. RCCL-side graceful fallback when a GIN backend fails to connect, so an over-subscribed mesh disables GIN instead of failing `ncclCommInitRank`. #13235 is the rocSHMEM-side fallback and does not cover RCCL. 4. Anvil signal path indexes peers by GIN team rank instead of LSA team position (`signal nRanks=8 != devr->lsaSize=4`): #13429 (AICOMRCCL-2706). 5. `rccl-UnitTestsGinAnvilSdmaFactory` is appended to `RCCL_TEST_EXECUTABLES` but never registered anywhere that runs, unlike the sibling GPU targets at `test/CMakeLists.txt:522` and `:552`. An `add_test()` is not the fix: `test/CMakeLists.txt:42-46` records that nothing in RCCL CI runs ctest, so it would never gate. The gap predates this PR and lives outside its files. JIRA ID: AICOMRCCL-2387 --------- Co-authored-by: Cursor <cursoragent@cursor.com>

amd-argus

ARGUS

Bottom line: Worth settling before approval: Anvil refuses non-LSA GIN peers at signal bind, after the backend is chosen, so with NCCL_GIN_TYPE unset a multi-node init fails instead of falling back to proxy or GDA (gin_plugin_anvil_sdma.cc:815).

_{Comment @argus review to re-run, or @argus re-review after pushing.}

github-actions · 2026-10-26T23:37:08Z

pr-rocm-systems-13429-oci-rccl-pr-smoke

Builds

Stack	Component	Version	Commit	Status
pr-13429_rccl-debug_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl-pr-13429	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl_gtest-pr-13429-debug	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-pr-13429	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	`108a1e0`	cached
pr-13429_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-13429_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-pr-13429	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`108a1e0`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`108a1e0`	cached
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`108a1e0`	cached

Failures

No failing tests in this run.

15 tests skipped.

github-actions · 2026-10-26T23:37:09Z

pr-rocm-systems-13429-ruby64-rccl-pr-smoke

Builds

Stack	Component	Version	Commit	Status
pr-13429_rccl-debug_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl-pr-13429	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl_gtest-pr-13429-debug	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-pr-13429	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	`b798048`	cached
pr-13429_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-13429_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-pr-13429	`2cb35ac`	`2cb35ac`	cached
pr-13429_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	`b798048`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`b798048`	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	`b798048`	cached

Failures

No failing tests in this run.

15 tests skipped.

amd-argus

ARGUS

Bottom line: Looks approval ready. Signal-free Anvil contexts still put outside the LSA window (gin_plugin_anvil_sdma.cc:1032). That gap predates this PR and is worth its own ticket.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

Bottom line: Nothing new worth raising; still open from earlier: gin_plugin_anvil_sdma.cc:815 and 1 more. Worth clearing before approval.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

github-actions · 2026-10-27T04:11:09Z

pr-rocm-systems-13429-oci-rccl-pr-smoke

Builds

Stack	Component	Version	Commit	Status
pr-13429_rccl-debug_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl-pr-13429	`0891a02`	`0891a02`	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl_gtest-pr-13429-debug	`0891a02`	`0891a02`	cached
pr-13429_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-pr-13429	`0891a02`	`0891a02`	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	`a556b0b`	cached
pr-13429_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-13429_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-pr-13429	`0891a02`	`0891a02`	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`a556b0b`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`a556b0b`	cached
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`a556b0b`	cached

Failures

No failing tests in this run.

15 tests skipped.

github-actions · 2026-10-27T04:11:09Z

pr-rocm-systems-13429-ruby64-rccl-pr-smoke

Builds

Stack	Component	Version	Commit	Status
pr-13429_rccl-debug_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl-pr-13429	`0891a02`	`0891a02`	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl_gtest-pr-13429-debug	`0891a02`	`0891a02`	cached
pr-13429_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-pr-13429	`0891a02`	`0891a02`	cached
pr-13429_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	`05f8852`	cached
pr-13429_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-13429_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-pr-13429	`0891a02`	`0891a02`	cached
pr-13429_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	`05f8852`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`05f8852`	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	`05f8852`	cached

Failures

No failing tests in this run.

15 tests skipped.

dlamd1dai · 2026-10-27T04:57:01Z

RCCL-Reviewers: this still needs one more approval from that team. @abouteiller's approval already covers rccl-leads and one of the two RCCL-Reviewers approvals.

The bind-time LSA check stays in this PR. Unset NCCL_GIN_TYPE on multi-node, where Anvil would otherwise be chosen before proxy, is #14216. Both Argus threads are resolved on that basis. Required checks (HIP NVIDIA CI Summary, Multi-Arch CI Summary, RCCL PR gate) are green.

dlamd1dai · 2026-10-27T06:19:34Z

Thanks @amd-argus for the Oct 16 review (summary). Here's where each of the remaining low-severity findings stands. The bind-time refusal at gin_plugin_anvil_sdma.cc:752 is answered in its own thread.

CHANGELOG.md:59, entry only mentions NCCL_GIN_TYPE=7: Fixed in 5efdb81. The entry now covers "NCCL_GIN_TYPE=7, or NCCL_GIN_TYPE unset when Anvil is the selected backend". It also says where the failure appears: ncclCommInitRank, or ncclDevCommCreate with NCCL_SYM_GIN_KERNELS_ENABLE=0.
GinAnvilPlugin_test.cpp:551, no passing bind test where comm->rank, GIN rank and lsaSelf all differ: Good point, and still true on the tip. BindSignals_SpanBaseProbeSizedForWorldRanks separates the GIN rank from the world rank, but lsaSelf still equals comm->rank there. I'd like to add that case as a test-only follow-up rather than grow this PR.
GinAnvilPlugin_test.cpp:1008, ConnCheck_UsesLsaTeamForBootstrap uses an identity lsaRanks: Agreed. With two world ranks and lsaRanks = {0, 1}, the test can't tell the LSA-team bootstrap from the world bootstrap. The earlier reversed list wasn't a layout a real communicator produces, which is why it changed. A wider world with a non-identity LSA team would cover this, and it fits the same follow-up.
GinAnvilPlugin_test.cpp:1002, the test inlines startTwoRankGin only to pass ginRank=1: Fair. Giving startTwoRankGin a ginRank parameter would remove the duplication. I'll fold that into the same follow-up.

None of these change runtime behavior, so I'm keeping this PR scoped to the LSA check. Thanks again for the careful review.

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

github-actions · 2026-10-28T11:30:42Z

pr-rocm-systems-13429-oci-rccl-pr-smoke

Builds

Stack	Component	Version	Commit	Status
pr-13429_rccl-debug_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl-pr-13429	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl_gtest-pr-13429-debug	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-pr-13429	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	`eb17a6b`	cached
pr-13429_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-13429_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-13429_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-pr-13429	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`eb17a6b`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`eb17a6b`	cached
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`eb17a6b`	cached

Failures

No failing tests in this run.

15 tests skipped.

github-actions · 2026-10-28T11:30:43Z

pr-rocm-systems-13429-ruby64-rccl-pr-build

Builds

Stack	Component	Version	Commit	Status	Build time
pr-13429_rccl-backtrace_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-backtrace_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-backtrace_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-backtrace_rocm-10.1-ga	rccl-pr-13429-backtrace	`7237ae7`	`7237ae7`	built	366s
pr-13429_rccl-backtrace_rocm-10.1-ga	rccl_gtest-pr-13429	`7237ae7`	`7237ae7`	built
pr-13429_rccl-debug_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl-pr-13429	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl_gtest-pr-13429-debug	`7237ae7`	`7237ae7`	built	448s
pr-13429_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-pr-13429	`7237ae7`	`7237ae7`	built	285s
pr-13429_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	`88849ae`	built	68s
pr-13429_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	built	25s
pr-13429_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-pr-13429	`7237ae7`	`7237ae7`	built	320s
pr-13429_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	`88849ae`	built	67s
pr-13429_rccl_rocm-7.0.2.2	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl_rocm-7.0.2.2	rocm-7.0.2.2	7.0.2.2	7.0.2.2	cached
pr-13429_rccl_rocm-7.0.2.2	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl_rocm-7.0.2.2	rccl-pr-13429	`7237ae7`	`7237ae7`	built	792s
pr-13429_rccl_rocm-7.0.2.2	rccl_gtest-pr-13429	`7237ae7`	`7237ae7`	built
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	built	23s
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`88849ae`	built	655s
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	`88849ae`	built	73s

Failures

No failing tests in this run.

183 tests skipped.

pr-rocm-systems-13429-ruby64-rccl-pr-smoke

Builds

Stack	Component	Version	Commit	Status
pr-13429_rccl-debug_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-debug_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl-pr-13429	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-debug_rocm-10.1-ga	rccl_gtest-pr-13429-debug	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-13429_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.1-ga	rccl-pr-13429	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	`88849ae`	cached
pr-13429_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-13429_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-13429_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-13429_rccl-perf_rocm-10.2-latest	rccl-pr-13429	`7237ae7`	`7237ae7`	cached
pr-13429_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	`88849ae`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`88849ae`	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	`88849ae`	cached

Failures

Stack	Type	Tag	Test	Topology	Status	Message
pr-13429_rccl-perf_rocm-10.1-ga	rccl_test	regression-allgather-fp32	all_gather_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.1-ga	rccl_test	regression-alltoall-fp32	alltoall_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.1-ga	rccl_test	regression-bf16-sum	all_reduce_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.1-ga	rccl_test	regression-bf16-sum	reduce_scatter_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.1-ga	rccl_test	regression-reduce-scatter-fp32-sum	reduce_scatter_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.2-latest	rccl_test	regression-allgather-fp32	all_gather_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.2-latest	rccl_test	regression-alltoall-fp32	alltoall_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.2-latest	rccl_test	regression-bf16-sum	all_reduce_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.2-latest	rccl_test	regression-bf16-sum	reduce_scatter_perf	4n_8ppn_interactive	Failed	Exited with code 2
pr-13429_rccl-perf_rocm-10.2-latest	rccl_test	regression-reduce-scatter-fp32-sum	reduce_scatter_perf	4n_8ppn_interactive	Failed	Exited with code 2
rocm-10.1-ga_rccl-develop-perf	rccl_test	regression-allgather-fp32	all_gather_perf	4n_8ppn_interactive	Failed	Exited with code 2
rocm-10.1-ga_rccl-develop-perf	rccl_test	regression-alltoall-fp32	alltoall_perf	4n_8ppn_interactive	Failed	Exited with code 2
rocm-10.1-ga_rccl-develop-perf	rccl_test	regression-bf16-sum	all_reduce_perf	4n_8ppn_interactive	Failed	Exited with code 2
rocm-10.1-ga_rccl-develop-perf	rccl_test	regression-bf16-sum	reduce_scatter_perf	4n_8ppn_interactive	Failed	Exited with code 2
rocm-10.1-ga_rccl-develop-perf	rccl_test	regression-reduce-scatter-fp32-sum	reduce_scatter_perf	4n_8ppn_interactive	Failed	Exited with code 2

15 tests skipped.

dlamd1dai · 2026-10-28T20:19:07Z

@ROCm/rccl-ci requesting an override of the required RCCL PR gate check, for the same reason as #14216. It fails only because of Ruby / ruby64-rccl-pr-smoke (run): the same 5 4n_8ppn_interactive regression cases exit with code 2 on both PR stacks and on the rocm-10.1-ga_rccl-develop-perf baseline in the same run. The debug stack's 209 gtests, the Comm Libs build, the CPU-only host unit tests and the Multi-Arch and HIP NVIDIA summaries all pass.

This PR is approved and depends on #14216, which merges first.

Conversation

dlamd1dai commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

With the symmetric-kernel skip (develop daf7ef33e4, head 7237ae7b96)

After the rebase onto develop (1ba7048298, head 2cb35ac234)

Before the rebase

MI355X: smci355-ccs-aus-m03-17

MI300X: banff-cyxtera-cx56-5

Notes

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Oct 14, 2026

✅ All Policy Checks Passed

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Findings

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 26, 2026

pr-rocm-systems-13429-oci-rccl-pr-smoke

Builds

Failures

Uh oh!

github-actions Bot commented Oct 26, 2026

pr-rocm-systems-13429-ruby64-rccl-pr-smoke

Builds

Failures

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 27, 2026

pr-rocm-systems-13429-oci-rccl-pr-smoke

Builds

Failures

Uh oh!

github-actions Bot commented Oct 27, 2026

pr-rocm-systems-13429-ruby64-rccl-pr-smoke

Builds

Failures

Uh oh!

dlamd1dai commented Oct 27, 2026

Uh oh!

dlamd1dai commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 28, 2026

pr-rocm-systems-13429-oci-rccl-pr-smoke

Builds

Failures

Uh oh!

github-actions Bot commented Oct 28, 2026

pr-rocm-systems-13429-ruby64-rccl-pr-build

Builds

Failures

pr-rocm-systems-13429-ruby64-rccl-pr-smoke

Builds

Failures

Uh oh!

dlamd1dai commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

dlamd1dai commented Oct 14, 2026 •

edited

Loading

With the symmetric-kernel skip (`develop` `daf7ef33e4`, head `7237ae7b96`)

After the rebase onto `develop` (`1ba7048298`, head `2cb35ac234`)

MI355X: `smci355-ccs-aus-m03-17`

MI300X: `banff-cyxtera-cx56-5`

dlamd1dai commented Oct 27, 2026 •

edited

Loading