[dlamd1dai]

Motivation

Follow-up from #14216 (AICOMRCCL-3417), suggested by @abouteiller in review. #14216 makes ncclGinSetDefaultBackend drop Anvil SDMA when comm->topo->nHosts > 1. Its tests cover the multi-host side. No end-to-end test covers the other side: in a 2-node job, a communicator that spans one host must keep Anvil SDMA.

Stacked on #14506 (AICOMRCCL-3450), which adds the 2-node CI suite with NCCL_GIN_TYPE unset. Merge after #14216 and #14506; only the last three commits belong to this PR.

Technical Details

• New MPI test GinMPIDeviceTests.MixedBackends_TwoNodePerHostSplitKeepsAnvil (test/transport/GinDeviceMPITests.cpp). It needs 4 ranks, 2 on each of 2 hosts, and NCCL_GIN_TYPE unset. It reuses mixedBackendTwoNodeSkipReason(), so it skips unless rocSHMEM GIN is built and Anvil SDMA is probeable.
  • Precondition: the world comm has topo->nHosts == 2 and has already dropped Anvil SDMA (ginSupport[ANVIL_SDMA] clear). There is no GIN gate on the world comm: only the per-host comms use GIN, and without a GIN-capable NIC the world comm is left with no backend once Anvil SDMA is dropped.
  • ncclCommSplit with topo->hostIdx as color. splitShare comes from the parent, so the test asserts the child has its own sharedRes. A shared child would reuse the world comm's ginState, which no longer holds Anvil SDMA.
  • On each per-host comm: topo->nHosts == 1, props.ginType == NCCL_GIN_TYPE_ANVIL_SDMA, props.ginSupport[NCCL_GIN_TYPE_ANVIL_SDMA] set, and globalGinSupport == FULL. Then a default DevComm completes a put + signal between the two local ranks (runPutWaitRoundTrip).
  • This fails if the drop is keyed on the job or the world comm instead of the comm's own nHosts: the per-host comm would lose Anvil SDMA and fail the ginType check.
• CI: the case joins the gin_devmpi_mixed_2node suite (gin_rocshmem_backends_mellanox.json) with its own shape: num_ranks: 4, num_gpus: 2 (2 ranks per node), num_nodes: 2. test_gin_rocshmem_backends_config.py now locks the shape of both cases in that suite, since a wrong shape becomes a silent skip.

Issue Tracking

JIRA ID: AICOMRCCL-3501

Test Plan

• 2 nodes × 2 GPUs, ENABLE_ROCSHMEM_GIN=ON, ENABLE_MPI_TESTS=ON, NCCL_CUMEM_ENABLE=1, NCCL_GIN_TYPE unset:
  mpirun -np 4 --map-by ppr:2:node ... rccl-UnitTestsMPI --gtest_filter=GinMPIDeviceTests.MixedBackends_TwoNodePerHostSplitKeepsAnvil
  Expect [ PASSED ], not [ SKIPPED ].
• Negative check (scratch build, not committed): make the Anvil drop in ncclGinSetDefaultBackend job-wide (drop on every comm once any comm in the process has spanned more than one host), then confirm the test fails on props.ginType.
• Same run with a working IB fabric, so the world comm also has the proxy backend (CI suite gin_devmpi_mixed_2node).
• CPU-only: python3 -m unittest discover -s tests -t . in tools/scripts/test_runner.
• Host-side compile of GinDeviceMPITests.cpp with -Wall (ENABLE_ROCSHMEM_GIN, MPI_TESTS_ENABLED).

Test Result

• Test runner unit tests: 151 run, OK (37 skipped), including the updated MixedBackendsTwoNodeSuiteTest.
• Host-side compile: no diagnostics in GinDeviceMPITests.cpp.
  2 × MI300X (gfx942, SPX), ctr-cx65-mi300x-3 + ctr-cx66-mi300x-3, 2 ranks per node, Release build with ENABLE_ROCSHMEM_GIN=ON and ENABLE_MPI_TESTS=ON, NCCL_CUMEM_ENABLE=1, NCCL_GIN_TYPE unset. The IB ports on both nodes were in INIT (no subnet manager), so NCCL used NET/Socket over eth1 and the world comm had no proxy GIN.

Pass (rank 0 log; mpirun exited 0 on all 4 ranks):

NCCL INFO GIN backend ANVIL_SDMA is single-host only; removing it from active backends (nHosts=2)
NCCL INFO GIN/RMA support: ginConnection 0 hostRma 0 symmetric 0 (ginTypeMask 0x80 ... nNodes 2 ...)
NCCL INFO GIN/RMA support: ginConnection 1 hostRma 1 symmetric 1 (ginTypeMask 0x80 ... nNodes 1 ...)
[       OK ] GinMPIDeviceTests.MixedBackends_TwoNodePerHostSplitKeepsAnvil (2734 ms)
[  PASSED  ] 1 test.

Negative check, scratch job-wide gate (reverted afterwards; the pass above is from the clean source):

NCCL INFO GIN backend ANVIL_SDMA is single-host only; removing it from active backends (nHosts=1)
GinDeviceMPITests.cpp:8180: Failure
Rank 0: ASSERT_MPI_EQ(NCCL_GIN_TYPE_ANVIL_SDMA, props.ginType) failed
[  FAILED  ] GinMPIDeviceTests.MixedBackends_TwoNodePerHostSplitKeepsAnvil (2704 ms)

Before the world-comm GIN gate was dropped, the same run skipped with GIN full connectivity is required; communicator is NONE or RAIL.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[amd-argus]

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[amd-argus]

Low

Two new comments repeat the code below them. The header here restates the asserts, and test_gin_rocshmem_backends_config.py:8-10 restates _EXPECTED_SHAPES. Could we keep only the reasoning?

Suggested change

	// is keyed on the job or the world comm instead of the comm's own nHosts.
	// 2 ranks per host, NCCL_GIN_TYPE unset (AICOMRCCL-3501): a per-host split keeps Anvil SDMA,
	// since the drop keys on the comm's own nHosts, not the job's.

[amd-argus]

Resolved by dd30354b61.

[amd-argus]

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}