[nikitaxgusev]

Motivation

ROCM-27070: RCCL segfaults during ncclCommInitRank on AINIC (Pensando ionic RoCE) hosts. Two root causes:

1. The GIN plugin did not honor NCCL_GIN_ENABLE=0, so GIN device enumeration still ran (and crashed) even when GIN was explicitly disabled.
2. On AINIC the data path uses net_ib_cast (rcclUseAinic()), but GIN registered the generic IB proxy backend (ncclGinIbProxy, name "GIN_IB_PROXY") instead of the ionic-aware cast backend. This produced a second, independent RDMA device enumeration (separate ibv_get_device_list / netRefCount) that crashed inside libibverbs/ionic during commAlloc.

Technical Details

projects/rccl/src/plugin/gin.cc:

• Early enable gate: ncclGinInit now returns early (ncclSuccess) when ncclParamGinEnable() == 0, before initPluginLibsOnceFunc() and any device enumeration, so NCCL_GIN_ENABLE=0 fully skips GIN.
• Mutually-exclusive internal IB backend registration: the internal IB GIN backend is now registered exclusively. When NCCL_NET=IB-CAST/ROCM-IB, or on AINIC autodetect (NCCL_NET unset and rcclUseAinic()), only the ionic-aware IbCastGinIb entry point is registered. Otherwise the generic ncclGinIbProxy backend is registered. This mirrors the selection logic in plugin/net.cc and prevents dual RDMA device enumeration.

Note: IbCastGinIb is a morphing entry point — at init() it memcpys the concrete ionic sub-backend (IbCastGinIbGdaki or IbCastGinIbProxy) over itself. Consequently the ionic proxy sub-backend and the generic ncclGinIbProxy both report the name "GIN_IB_PROXY" at runtime; they are distinct backends from different transports (net_ib_cast vs net_ib) and are only distinguishable by their function tables.

Tests / runner:

• projects/rccl/test/transport/GinMPI/GinInitMPITests.cpp (new) — GinInitMPITest MPI gtests:
  • GinEnableZeroSkipsInit: with NCCL_GIN_ENABLE=0, ncclCommInitRank succeeds and comm->sharedRes->ginState.ncclGin is null (GIN skipped, no crash).
  • AinicSelectsCastBackend: with GIN enabled on AINIC, the assigned GIN backend is an ionic net_ib_cast backend and not the generic net_ib proxy. Because the cast entry point morphs at init() and both the ionic proxy sub-backend and the generic backend report the name "GIN_IB_PROXY", identity is verified through the backend's function table — gin->finalize == IbCastGinIbProxy.finalize (shared IbCastGinIbFinalize across both ionic sub-backends) and gin->finalize != ncclGinIbProxy.finalize — rather than by name or by a raw pointer to the struct (which is unreliable across the librccl.so boundary due to copy relocations).
• projects/rccl/test/CMakeLists.txt — builds GinInitMPITests.cpp into rccl-UnitTestsMPI.
• projects/rccl/tools/scripts/test_runner/configs/ainic.json — adds runner configs A14 (a_gin_enable_gate) and A15 (a_gin_ainic_backend) invoking the two new tests with appropriate env, and registers them in test_suites.

Issue Tracking

JIRA ID : ROCM-27070

Test Plan

• Build RCCL in Debug with MPI tests enabled: ./install.sh --enable-mpi-tests on gfx950 AINIC nodes.
• Run cross-node (2 nodes x 1 GPU) to avoid single-node IB loopback issues:
  • GinInitMPITest.GinEnableZeroSkipsInit (with NCCL_GIN_ENABLE=0)
  • GinInitMPITest.AinicSelectsCastBackend (GIN enabled, AINIC autodetect)
  • NetIbMPITest.InitializePlugin as a cross-node control.
• Equivalent test-runner entries: ainic.json suites A14 (a_gin_enable_gate) and A15 (a_gin_ainic_backend).

Test Result

Debug build (./install.sh --debug -t --enable-mpi-tests --amdgpu_targets gfx950), run cross-node on 2 nodes × 1 GPU (mia1-p01-g27 + g25, MI355 / gfx950, ROCm 7.0.2, RCCL_AINIC_ROCE=1):

• NetIbMPITest.InitializePlugin control — PASSED cross-node (ionic devices rdma0..7 enumerated, libionic.so loaded).
• P1 GinInitMPITest.GinEnableZeroSkipsInit — PASSED: with NCCL_GIN_ENABLE=0, ncclCommInitRank succeeds with ginState.ncclGin == nullptr and no crash.
• P2 GinInitMPITest.AinicSelectsCastBackend — PASSED: assigned GIN backend is the ionic net_ib_cast backend (gin->finalize == IbCastGinIbProxy.finalize, != ncclGinIbProxy.finalize); data path uses NET/IB-CAST — the generic IB proxy is not selected, so there is no dual enumeration.

[ RUN      ] GinInitMPITest.GinEnableZeroSkipsInit
[       OK ] GinInitMPITest.GinEnableZeroSkipsInit
[ RUN      ] GinInitMPITest.AinicSelectsCastBackend
[       OK ] GinInitMPITest.AinicSelectsCastBackend
[  PASSED  ] 2 tests.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

❌ PR Check — Action Required

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	❌ Fail	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*). Current: code file(s) changed: projects/rccl/src/plugin/gin.cc, projects/rccl/test/transport/GinMPI/GinMPITests.cpp; no test file found
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

• ❌ Unit Test

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

[therock-pr-bot]

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

• ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

[Copilot]

Pull request overview

This PR addresses ROCM-27070 by ensuring the RCCL GIN plugin (a) fully honors NCCL_GIN_ENABLE=0 to avoid any GIN initialization/device enumeration, and (b) selects the ionic-aware ib-cast GIN backend on AINIC hosts to prevent a second, crashing RDMA device enumeration path.

Changes:

• Add an early NCCL_GIN_ENABLE gate in ncclGinInit() to skip all GIN init when disabled.
• Make internal IB GIN backend registration mutually exclusive: select ib-cast backend on AINIC/IB-CAST, otherwise select the generic IB proxy backend.
• Add new MPI gtests and AINIC test-runner suite entries to cover the regression scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
projects/rccl/src/plugin/gin.cc	Adds early enable gating and changes internal IB backend registration logic to avoid dual enumeration on AINIC.
projects/rccl/test/transport/GinMPI/GinInitMPITests.cpp	Adds MPI gtests to validate NCCL_GIN_ENABLE=0 gating and AINIC backend selection by name.
projects/rccl/test/CMakeLists.txt	Wires the new GIN init MPI tests into the MPI unit test target.
projects/rccl/tools/scripts/test_runner/configs/ainic.json	Adds two new AINIC runner configs/suites to execute the new regression tests.

Comments suppressed due to low confidence (1)

projects/rccl/src/plugin/gin.cc:274

• ncclGinVersion is assigned twice for the same plugin entry in the ncclGinIbProxy branch; the second assignment is redundant.

    ginPluginLibs[pluginCounter].ncclGinVersion = ncclGinVersion[0];
    ginPluginLibs[pluginCounter].ncclRma = ginPluginLibs[pluginCounter].ncclGin;
    ginPluginLibs[pluginCounter].ncclRmaPluginState = ncclGinPluginStateInitReady;
    ginPluginLibs[pluginCounter].ncclGinVersion = ncclGinVersion[0];
    pluginCounter++;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

RCCL Perf-Regression Gate: ✅ PASS

Mode: detect (reference vs candidate)
Thresholds (per-tier, calibrated): small 17.2% · mid 12.3% · large 12.9%
Keys compared: 0 · Confirmed regressions: 0 · Inconclusive: 0

Per-collective breakdown

group	keys	regressions	inconclusive
all_gather_perf-d=bfloat16-default	0	0	0
all_gather_perf-d=float-default	0	0	0
all_reduce_perf-d=bfloat16-default	0	0	0
all_reduce_perf-d=float-default	0	0	0
broadcast_perf-d=bfloat16-default	0	0	0
broadcast_perf-d=float-default	0	0	0
reduce_scatter_perf-d=bfloat16-default	0	0	0
reduce_scatter_perf-d=float-default	0	0	0

[nikitaxgusev]

manual check:

A14. GIN enable gate (ROCM-27070)    A14_Gin_EnableZeroSkipsInit              PASSED   6.48s
A15. GIN AINIC backend (ROCM-27070)  A15_Gin_AinicSelectsCastBackend          PASSED   6.33s
------------------------------------------------------------------------------------------------------

[ RUN      ] GinInitMPITest.GinEnableZeroSkipsInit
NCCL INFO NCCL_GIN_ENABLE set by environment to 0.
NCCL INFO GIN disabled by NCCL_GIN_ENABLE=0; skipping GIN plugin init     gate worked
NCCL INFO [Rank 0] ncclCommInitRank_impl ... - Init START
NCCL INFO ncclCommInitRank_impl .......... - Init COMPLETE                         comm init  passed
[0] TEST INFO Test-specific communicator created successfully
[       OK ] GinInitMPITest.GinEnableZeroSkipsInit (5713 ms)
[  PASSED  ] 1 test.
[ RUN      ] GinInitMPITest.AinicSelectsCastBackend
[0] TEST INFO Assigned GIN backend: GIN_IB_PROXY (finalize=0x78b679388a00)
[       OK ] GinInitMPITest.AinicSelectsCastBackend (5677 ms)
[  PASSED  ] 1 test.

[ RUN      ] GinInitMPITest.AinicSelectsCastBackend
NCCL INFO Loaded dlvsym from libionic.so[.1]                ionic path
NCCL INFO Assigned GIN plugin GIN_IB_PROXY to comm          
NCCL INFO Assigned RMA plugin GIN_IB_PROXY to comm
NCCL INFO Using network IB-CAST                             datapath  IB-CAST
NCCL INFO ncclCommInitRank_impl ... - Init COMPLETE
[0] TEST INFO Assigned GIN backend: GIN_IB_PROXY (finalize=0x7bedf10aba00)
[       OK ] GinInitMPITest.AinicSelectsCastBackend (5450 ms)
[  PASSED  ] 1 test.

[nikitaxgusev]

CI is green except for a single flaky timeout on one node, unrelated to this change. All other test stages passed on the remaining GPU archs, and compile/package succeeded everywhere. The failing test (AllReduce.Channels on #8633 / Register.ProcessIsolatedRegisterTests on #9085) timed out but reported 0 failures, and passed on the other architectures. The remaining red checks (therock-pr-bot, build, Provision) are infra/provisioning issues. Safe to merge.