[Kapil-Shyam-Pawar]

Motivation

Implement GIN-SDMA alltoall inside RCCL, which is called from within ncclAllToAll if the eligibility criteria are satisfied.

With this, rccl-tests no longer needs to be linked with rocSHMEM GIN host symbols for the SDMA backend, and the -D 3 option when calling alltoall_perf can be dropped.

It also lets external apps use the GIN-SDMA alltoall through ncclAllToAll without linking the rocSHMEM SDMA symbols themselves.

Technical Details

Currently, when rccl is built with the --rocshmem-gin flag, the host symbols are left unresolved, which are resolved while building rccl-tests with ENABLE_ROCSHMEM_GIN=ON.

With this change, librccl is made self-contained with the SDMA host symbols, so the internal alltoall implementation can reach the SDMA backend without the application supplying them. They are extracted from librocshmem.a at build time and seeded by the gin_anvil_sdma_* entry points, so upstream file renames do not break it.

The extracted symbols are kept local to librccl with --exclude-libs, so rccl-tests built with ENABLE_ROCSHMEM_GIN=ON can still run rocSHMEM GDA or SDMA with -D 3/4

Issue Tracking

JIRA ID: AICOMRCCL-1682

Test Plan

• Add NCCL_GIN_TYPE=6 alltoall_perf -R 2 run to the GIN CI and monitor - Passing

• Steps to manually test:

  Build & Run:

  cd rccl
  bash install.sh -l --rocshmem-gin
  
  cd rccl-tests
  make clean
  GPU_TARGETS=gfx950 make -j \
    MPI=1 \
    ENABLE_DEVICE_API=1 \
    MPI_HOME="${MPI_HOME}" \
    NCCL_HOME="${ROCM_SYSTEMS}/rccl/build/release"
  
  mpirun \
    --allow-run-as-root \
    -np 8 \
    --map-by ppr:8:node \
    --mca pml ob1 \
    --mca btl ^vader,openib \
    -x PATH \
    -x LD_LIBRARY_PATH \
    -x NCCL_DEBUG=VERSION \
    -x NCCL_GIN_TYPE=6 \
    -x NCCL_CUMEM_ENABLE=1 \
    rccl-tests/build/alltoall_perf \
    -b 1 \
    -e 4G \
    -f 2 \
    -g 1 \
    -R 2
  

• Created stacked test(rccl): gin device tests for sdma #10388 for tests

Test Result

• GIN Device tests in GinDeviceMPITests.cpp‎ passes for SDMA
• New GinMPIDeviceTests.Alltoall_SdmaInternal passes

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	⚠️ Warning	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*). Current: code file(s) changed: projects/rccl/src/algorithms/gin/gin_alltoall.h, projects/rccl/src/collectives.cc, projects/rccl/src/include/comm.h, projects/rccl/src/init.cc; no test file found
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[Copilot]

Pull request overview

Implements an internal GIN-SDMA alltoall fast-path inside RCCL, invoked from ncclAlltoAll when eligibility criteria are met, and updates CI to exercise the new internal path so external apps and rccl-tests no longer need to supply/link rocSHMEM SDMA host symbols for this mode.

Changes:

• Adds a new GIN-SDMA alltoall implementation (gin_alltoall_sdma.cu) with eligibility checks and per-communicator lazy initialization/teardown.
• Routes ncclAlltoAll through the new internal GIN-SDMA path when eligible, and finalizes that state during communicator teardown.
• Updates build/device-link integration and CI test matrix to support and validate the self-contained SDMA host-symbol extraction and internal alltoall execution.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
projects/rccl/tools/ci/lib/gin-tests.json	Adds a CI rccl-tests run for the new internal GIN-SDMA alltoall path (no -D 3).
projects/rccl/src/init.cc	Ensures GIN alltoall state is finalized during communicator teardown.
projects/rccl/src/include/gin_alltoall.h	Introduces the public/internal host API and per-comm state struct for GIN-SDMA alltoall.
projects/rccl/src/include/comm.h	Adds ginA2AState to ncclComm and includes the new header.
projects/rccl/src/gin_alltoall_sdma.cu	Implements eligibility + kernel launch + init/finalize for internal GIN-SDMA alltoall.
projects/rccl/src/collectives.cc	Hooks ncclAlltoAll to take the internal GIN-SDMA path when eligible.
projects/rccl/src/CMakeLists.txt	Extracts/links required rocSHMEM SDMA host objects into librccl and hides them via --exclude-libs.
projects/rccl/cmake/DeviceLinker.cmake	Adjusts rocSHMEM device-bitcode injection logic and adds a dedicated device-linker compile for the new kernel TU.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

RCCL Perf-Regression Gate: ⚠️ NO VERDICT (not measured)

This run did not produce a usable answer. It is neither a PASS nor a regression — treat the perf gate as not run for this change.

Why:

• this report predates the trustworthiness flag, or a group could not be scored

Mode: detect (reference vs candidate)
Thresholds: small 17.2% · mid 12.3% · large 12.9%
Keys compared: 0 · Confirmed regressions: 0 · Inconclusive: 0
Provenance: n/a

Per-collective breakdown

group	keys	regressions	inconclusive
all_gather_perf-d=bfloat16-default	0	0	0
all_gather_perf-d=float-default	0	0	0
all_reduce_perf-d=bfloat16-default	0	0	0
all_reduce_perf-d=float-default	0	0	0
broadcast_perf-d=bfloat16-default	0	0	0
broadcast_perf-d=float-default	0	0	0
reduce_scatter_perf-d=bfloat16-default	0	0	0
reduce_scatter_perf-d=float-default	0	0	0

[nusislam]

@dlamd1dai @wenkaidu - Can you review this PR?

[dlamd1dai]

Review summary

Solid direction overall: self-contained SDMA host symbols, device-linker integration, and the ncclAlltoAll fast path without rccl-tests -D 3 are the right architecture. CMake and lifecycle work look good.

Please address the inline comments before merge:

1. Dispatch ordering — move GIN after DDA so small single-node messages stay on DDA when eligible.
2. SDMA kernel — use ncclGinBarrierSession and strided peer puts to match reference kernels.
3. Eligibility — add DDA deferral guard and sharedRes null check.
4. CI — add NCCL_MSCCL_ENABLE=0 to the new internal test.

Merge sequencing: #10386 can land once the above are addressed and CI is green. Merge stacked #10388 immediately after for device-kernel regression coverage. Eligibility unit tests for ncclAllToAllGinSdmaEligible are still a gap — please track as follow-up.

[dlamd1dai]

Re-review summary (post Aug 28 updates)

Thanks for the follow-up commits — ginA2ASdmaThreads, separate LSA/SDMA barrier paths, and the 16B alignment gate all look good. Build/device-linker integration remains solid.

Policy (confirmed on our side): GIN-first dispatch before DDA is acceptable given the measured parity/better perf on small sizes (AICOMRCCL-1682).

Before merge, please:

1. Reconcile the 8 MiB GIN_A2A_SDMA_MIN_BYTES default vs the 32 MiB note in the review thread (inline ask).
2. Prefer switching the SDMA peer loop to the strided reference pattern — not a hard blocker from us, but recommended.

Follow-ups (non-blocking): eligibility unit tests; align #10388 device tests with the hybrid ncclBarrierSession SDMA contract.

Overall: approve direction — close to merge pending the threshold clarification.

[systems-assistant]

TheRock Submodule Bump Activity

Newest first

• 2026-09-03 — Included in TheRock via ROCm/TheRock#7862.