fix(rccl): Make device API / GIN examples build on ROCm by Kapil-Shyam-Pawar · Pull Request #11579 · ROCm/rocm-systems

Kapil-Shyam-Pawar · 2026-09-14T15:31:38Z

Motivation

The 06_device_api examples were synced from NCCL but never wired up for ROCm, so they could not be built at all

Technical Details

Added the missing examples/06_device_api/Makefile
Pointed the three leaf c/Makefiles at rccl_examples.mk - the old include path resolved outside the repo and pulled in CUDA settings
Added a build-time hipify-perl step, so the upstream .cu sources stay byte-identical to NCCL and future syncs apply cleanly.
Search RCCL_HOME before ROCM_PATH, and accept a CMake build tree as RCCL_HOME.
Ignore the built binaries and hipify staging directory.

Issue Tracking

JIRA ID: AICOMRCCL-1529

Test Plan

Run from examples/06_device_api:

export RCCL_HOME=<path/to/rccl/build/release>
export MPI_HOME=<path/to/mpi/install>
export GPU_TARGETS=<gfx_arch>
export NCCL_CUMEM_ENABLE=1 NCCL_GIN_TYPE=2

# 1. LSA allreduce
cd 01_allreduce_lsa/c && make clean && make && \
  $MPI_HOME/bin/mpirun -np 8 ./allreduce_lsa

# 2. Pure GIN alltoall
cd 02_alltoall_gin/c && make clean && make MPI=1 && \
  $MPI_HOME/bin/mpirun -np 8 ./alltoall_gin

# 3. Hybrid GIN+LSA alltoall
cd 03_alltoall_hybrid/c && make clean && make MPI=1 && \
  $MPI_HOME/bin/mpirun -np 8 ./alltoall_hybrid

# 4. All three via the category Makefile
make clean && make test MPI=1

Test Result

All three examples build and pass

AllReduce completed. Result verification: PASSED
Pure GIN AlltoAll result: PASSED
Hybrid AlltoAll result: PASSED

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

therock-pr-bot · 2026-09-14T15:32:26Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

Copilot

🟢 Approval recommended

Remaining documentation nits are minor and non-blocking.

Pull request overview

Adds ROCm build support for the NCCL Device API and GIN examples using hipification and RCCL-aware linking.

Changes:

Adds category-level build, test, install, and clean targets.
Adds build-time hipify rules and RCCL/CMake build-tree discovery.
Updates example Makefiles and ignores generated artifacts.

File summaries

File	Summary
`projects/rccl/examples/makefiles/rccl_examples.mk`	Shared ROCm, RCCL, MPI, and hipify build rules.
`projects/rccl/examples/06_device_api/Makefile`	Category-level orchestration.
`projects/rccl/examples/06_device_api/03_alltoall_hybrid/c/Makefile`	Hybrid example ROCm build integration; README documentation nit noted.
`projects/rccl/examples/06_device_api/02_alltoall_gin/c/Makefile`	GIN example ROCm build integration; README documentation nit noted.
`projects/rccl/examples/06_device_api/01_allreduce_lsa/c/Makefile`	LSA example ROCm build integration; README documentation nit noted.
`projects/rccl/examples/.gitignore`	Ignores generated staging files and binaries.

Review details

Suppressed comments (2)

projects/rccl/examples/06_device_api/02_alltoall_gin/c/Makefile:9

The corresponding c/README.md still tells users to build with NCCL_HOME/CUDA_HOME and describes this as a CUDA/NCCL example, but this Makefile now consumes RCCL_HOME/ROCM_PATH and invokes hipify-perl/hipcc. Please update the README so the documented command actually configures this ROCm build.

include ../../../makefiles/rccl_examples.mk

projects/rccl/examples/06_device_api/03_alltoall_hybrid/c/Makefile:9

The corresponding c/README.md still tells users to build with NCCL_HOME/CUDA_HOME and describes this as a CUDA/NCCL example, but this Makefile now consumes RCCL_HOME/ROCM_PATH and invokes hipify-perl/hipcc. Please update the README so the documented command actually configures this ROCm build.

include ../../../makefiles/rccl_examples.mk

Files reviewed: 6/6 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

alex-breslow-amd

LGTM

amd-argus

ARGUS

Bottom line: Looks approval ready. The four notes are all Low and mechanical, so they do not need another pass from me.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

Findings

1 finding(s) that could not be posted as inline comments

projects/rccl/examples/06_device_api/02_alltoall_gin/c/main.cu:152 (Low) make test in the new category Makefile exits 0 on a host without GIN, having run no collective 06_device_api make test reports success on a host without GIN. 02_alltoall_gin/c/main.cu:149 prints ERROR: ... does not support GIN! and 02_alltoall_gin/c/main.cu:152 returns NULL without running the collective, then examples/common/src/utils.cc:261 passes NULL to pthread_join so that result is discarded and the process exits 0. Both files sit outside this diff, so this looks like follow-up material rather than a change here.

Bottom line: LGTM. The include repoint, hipify staging and RCCL_HOME search order all check out; the two notes are follow-ups, not changes for this PR.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Kapil-Shyam-Pawar · 2026-09-16T01:25:37Z

topo_expl build failure is unrelated and is being tracked by AICOMRCCL-2359

## Motivation The `06_device_api` examples were synced from NCCL but never wired up for ROCm, so they could not be built at all ## Technical Details - Added the missing `examples/06_device_api/Makefile` - Pointed the three leaf `c/Makefiles` at `rccl_examples.mk` - the old include path resolved outside the repo and pulled in CUDA settings - Added a build-time `hipify-perl` step, so the upstream `.cu` sources stay byte-identical to NCCL and future syncs apply cleanly. - Search `RCCL_HOME` before `ROCM_PATH`, and accept a CMake build tree as `RCCL_HOME`. - Ignore the built binaries and hipify staging directory. ## Issue Tracking JIRA ID: AICOMRCCL-1529 ## Test Plan Run from `examples/06_device_api`: ``` export RCCL_HOME=<path/to/rccl/build/release> export MPI_HOME=<path/to/mpi/install> export GPU_TARGETS=<gfx_arch> export NCCL_CUMEM_ENABLE=1 NCCL_GIN_TYPE=2 # 1. LSA allreduce cd 01_allreduce_lsa/c && make clean && make && \ $MPI_HOME/bin/mpirun -np 8 ./allreduce_lsa # 2. Pure GIN alltoall cd 02_alltoall_gin/c && make clean && make MPI=1 && \ $MPI_HOME/bin/mpirun -np 8 ./alltoall_gin # 3. Hybrid GIN+LSA alltoall cd 03_alltoall_hybrid/c && make clean && make MPI=1 && \ $MPI_HOME/bin/mpirun -np 8 ./alltoall_hybrid # 4. All three via the category Makefile make clean && make test MPI=1 ``` ## Test Result All three examples build and pass ``` AllReduce completed. Result verification: PASSED Pure GIN AlltoAll result: PASSED Hybrid AlltoAll result: PASSED ``` ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

systems-assistant · 2026-09-17T14:01:13Z

TheRock Submodule Bump Activity

Newest first

2026-09-17 — Included in TheRock via ROCm/TheRock#8258.

## Motivation The `06_device_api` examples were synced from NCCL but never wired up for ROCm, so they could not be built at all ## Technical Details - Added the missing `examples/06_device_api/Makefile` - Pointed the three leaf `c/Makefiles` at `rccl_examples.mk` - the old include path resolved outside the repo and pulled in CUDA settings - Added a build-time `hipify-perl` step, so the upstream `.cu` sources stay byte-identical to NCCL and future syncs apply cleanly. - Search `RCCL_HOME` before `ROCM_PATH`, and accept a CMake build tree as `RCCL_HOME`. - Ignore the built binaries and hipify staging directory. ## Issue Tracking JIRA ID: AICOMRCCL-1529 ## Test Plan Run from `examples/06_device_api`: ``` export RCCL_HOME=<path/to/rccl/build/release> export MPI_HOME=<path/to/mpi/install> export GPU_TARGETS=<gfx_arch> export NCCL_CUMEM_ENABLE=1 NCCL_GIN_TYPE=2 # 1. LSA allreduce cd 01_allreduce_lsa/c && make clean && make && \ $MPI_HOME/bin/mpirun -np 8 ./allreduce_lsa # 2. Pure GIN alltoall cd 02_alltoall_gin/c && make clean && make MPI=1 && \ $MPI_HOME/bin/mpirun -np 8 ./alltoall_gin # 3. Hybrid GIN+LSA alltoall cd 03_alltoall_hybrid/c && make clean && make MPI=1 && \ $MPI_HOME/bin/mpirun -np 8 ./alltoall_hybrid # 4. All three via the category Makefile make clean && make test MPI=1 ``` ## Test Result All three examples build and pass ``` AllReduce completed. Result verification: PASSED Pure GIN AlltoAll result: PASSED Hybrid AlltoAll result: PASSED ``` ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

Motivation NCCL 2.30.7 added the RMSNorm kernel fusion examples (examples/07_kernel_fusion: LSA, Multimem, GIN, Hybrid LSA/GIN). They came into RCCL with the sync but still included the upstream makefiles/{common,examples}.mk (outside the repo) and built with nvcc, so none of them could be built on ROCm. This PR wires them up the same way #11579 did for 06_device_api. Validating them also exposed a defect in the public device API headers that affects every application kernel using the GIN device API, not only these examples. This PR fixes it as well (see Device API fix below). Technical Details The four leaf c/Makefiles include examples/makefiles/rccl_examples.mk. The .cu sources are hipified at build time with hipify-perl into .hipify/ and stay byte-identical to NCCL, so future syncs apply cleanly. rccl_examples.mk: new $(HIPIFY_DIR)/%.cuh rule. The shared include/rmsnorm_utils.cuh includes <cuda_runtime.h>, so it is hipified too. The quoted #include in the hipified main.cu searches .hipify/ first and picks up the HIP copy. The leaf READMEs use the ROCm build variables (RCCL_HOME, ROCM_PATH, GPU_TARGETS), and the Makefile help targets drop the CUDA requirements, as in 06_device_api. examples/README.md lists 07_kernel_fusion, and its build line covers 01..07, matching examples/Makefile. The four binaries are added to examples/.gitignore.

Update device API / GIN examples to be functional on ROCm

9c2f71c

Kapil-Shyam-Pawar requested review from a team and a lite review from Copilot September 14, 2026 15:31

github-actions Bot added the project: rccl label Sep 14, 2026

Copilot started reviewing on behalf of Kapil-Shyam-Pawar September 14, 2026 15:32

View session

Copilot AI reviewed Sep 14, 2026

View reviewed changes

Comment thread projects/rccl/examples/06_device_api/01_allreduce_lsa/c/Makefile Resolved

systems-assistant Bot added the organization: ROCm label Sep 14, 2026

Update docs

a3e553a

alex-breslow-amd approved these changes Sep 14, 2026

View reviewed changes

atulkulk approved these changes Sep 14, 2026

View reviewed changes

amd-argus Bot reviewed Sep 14, 2026

View reviewed changes

Address review comments

f31703f

amd-argus Bot reviewed Sep 15, 2026

View reviewed changes

Kapil-Shyam-Pawar merged commit 5b1bec5 into develop Sep 16, 2026
45 of 53 checks passed

Kapil-Shyam-Pawar deleted the users/kpawar/fix_examples branch September 16, 2026 01:25

mkuznet1 mentioned this pull request Oct 20, 2026

fix(rccl): build the RMSNorm kernel fusion examples on ROCm #13815

Merged

Navigation Menu

Repository navigation

fix(rccl): Make device API / GIN examples build on ROCm - #11579

fix(rccl): Make device API / GIN examples build on ROCm#11579
Kapil-Shyam-Pawar merged 3 commits into
developfrom
users/kpawar/fix_examples

Kapil-Shyam-Pawar commented Sep 14, 2026

Uh oh!

therock-pr-bot Bot commented Sep 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

alex-breslow-amd left a comment

Uh oh!

amd-argus Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amd-argus Bot left a comment

Uh oh!

Kapil-Shyam-Pawar commented Sep 16, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Sep 17, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Kapil-Shyam-Pawar commented Sep 14, 2026

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Sep 14, 2026

✅ All Policy Checks Passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Suppressed comments (2)

Uh oh!

Uh oh!

alex-breslow-amd left a comment

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Findings

Uh oh!

Kapil-Shyam-Pawar commented Sep 16, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Sep 17, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants