Skip to content
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
78918c5
feat(rccl): Enable hierarchical CE scale-out collectives
thomas-huber Aug 7, 2026
69d542a
test(rccl): Cover hierarchical CE scale-out collectives
thomas-huber Aug 7, 2026
a555140
Merge branch 'develop' into users/thomas-huber/AICOMRCCL-630-ce-scaleout
thomas-huber Aug 27, 2026
73ebdc7
Merge remote-tracking branch 'origin/develop' into users/thomas-huber…
thomas-huber Sep 1, 2026
fb7299f
fix(rccl): Address CE scale-out review feedback
thomas-huber Sep 8, 2026
d8d27f2
fix(rccl): Address CE scale-out review round 2
thomas-huber Sep 9, 2026
eb1869e
Merge remote-tracking branch 'origin/develop' into users/thomas-huber…
thomas-huber Sep 9, 2026
d26a711
Merge remote-tracking branch 'origin/develop' into users/thomas-huber…
thomas-huber Sep 15, 2026
59b90eb
fix(rccl): Address CE scale-out review round 3
thomas-huber Sep 15, 2026
7bbec52
Merge remote-tracking branch 'origin/develop' into users/thomas-huber…
thomas-huber Sep 15, 2026
dacfc2c
fix(rccl): address CE scale-out review feedback
thomas-huber Sep 15, 2026
96cc65d
fix(rccl): report hierarchical CE AllGather as CE for -A 1
thomas-huber Sep 16, 2026
9df520c
test(rccl): pin CE eligibility mocks and FreeBatchOps runner filter
thomas-huber Sep 16, 2026
e964ac6
Merge remote-tracking branch 'origin/develop' into users/thomas-huber…
thomas-huber Sep 16, 2026
4c05e51
test(rccl): cover the CE LSA-coverage gate and sharpen CHANGELOG entries
thomas-huber Sep 16, 2026
0f400cf
Merge remote-tracking branch 'origin/develop' into users/thomas-huber…
thomas-huber Sep 17, 2026
1088dde
fix(rccl): restore the CE batch-param resets and cover the RMA-contex…
thomas-huber Sep 17, 2026
5a4b712
fix(rccl): repair two NCCL v2.31.2-1 sync regressions outside CE scal…
thomas-huber Sep 17, 2026
7ca726a
fix(rccl): guard the CE tuning model against an uninitialized sync fr…
thomas-huber Sep 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions projects/rccl/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Full documentation for RCCL is available at [https://rccl.readthedocs.io](https:
* Added `install.sh --all_unrolls` (`-DBUILD_ALL_UNROLLS=ON`) to generate every unroll factor (1, 2, 4, 8, 16, 32) for the targeted GPU architecture(s), for measuring unroll factors that the default per-arch matrix does not build. The flag also drops the per-architecture pin, so such a build accepts every `RCCL_UNROLL_FACTOR` value on any targeted architecture.
* Added the strict `--enable-full-coverage` install flag for unified host + device LLVM source-based code coverage. It requires `--debug`, the device linker, and ROCm 7.15 or newer. The test runner uses the CMake-level `ENABLE_FULL_COVERAGE=AUTO` mode, which falls back to host-only coverage when device instrumentation is unavailable.
* Enabled the Copy-Engine profiler path (`ncclProfiler_v6`): Copy-Engine events are emitted for CE collectives, and the example profiler plugin reports them.
* Enabled hierarchical Copy Engine `ncclAllGather` and `ncclAlltoAll` on multi-node communicators: the intra-node phase runs over SDMA/CE across the LSA team and the inter-node phase over one-sided RMA through the CPU (GIN) proxy. Selected when every node has the same number of ranks, `NCCL_CTA_POLICY=2` with symmetric send and receive windows, `NCCL_LOCAL_REGISTER=0`, `NCCL_CUMEM_ENABLE=1`, `NCCL_NUM_RMA_CTX` greater than zero, and a GIN backend available. Validated for correctness and path selection from 1 to 32 nodes.
Comment thread
alex-breslow-amd marked this conversation as resolved.
Resolved

### Changed
* Adapted the device-initiated GIN backends (Anvil SDMA and rocSHMEM GDA) to the NCCL 2.30.7 GIN API v14: added the new `getGinProperties` host op, dropped the data-path ops (`iput`/`iputSignal`/`iget`/`iflush`/`test`) that moved out of GIN under the GIN/RMA split, switched `createContext` to `ncclGinConfig_v14_t`, updated the device dispatch signatures, and matched the GIN type renumbering (`ROCSHMEM_GDA` and `ANVIL_SDMA` shifted after the new `GIN_GPI` type). The plugins now use the generic (unversioned) `ncclGin_t` / `ncclGinConfig_t` / `ncclGinProperties_t` typedefs so future ABI bumps do not require touching call sites.
Expand All @@ -29,6 +30,7 @@ Full documentation for RCCL is available at [https://rccl.readthedocs.io](https:
* `RCCL_UNROLL_FACTOR` is now rejected when the requested unroll factor's device functions were not compiled for the running architecture, instead of being accepted and then dispatching into an empty device function table. Unroll factor 32 is compiled for gfx1250 only, but a multi-arch build reported every unroll factor as available, so requesting it on another GPU crashed on the device. That value now fails communicator initialization with a warning naming the architecture, and the default selection falls back to the highest unroll factor actually compiled for the GPU rather than trusting the heuristic's choice. The WarpSpeed auto-tuner's preference for unroll factor 2 is subject to the same check, so it no longer replaces a validated unroll factor with one this build cannot dispatch. gfx1250 is unaffected and still defaults to 32.
* Fixed the per-unroll device function tables being misaligned in multi-arch builds. The LL128 `SendRecv` kernel was skipped for unrolls 8/16/32, so every function after `SendRecv` in those tables sat one index below the id the host had computed from the unroll-1 ordering, and the last entry of the host lookup table was dropped. `AlltoAllPivot`, `AlltoAllGda`, `AlltoAllvGda` and `AllGatherV` were affected on gfx1250.
* Fixed the AMD SMI fabric ABI guard rejecting the layout amd_smi 27.x introduced, which broke the RCCL build outright. The fabric payload union gained a second member, enlarging `amdsmi_fabric_info_t` without moving the v1 fields RCCL reads. RCCL now recognizes that layout, identifies the loaded runtime by how much of the probe buffer it writes, and falls back to the sysfs fabric backend on a 27.x runtime rather than reading a payload it does not model. Fabric topology on such a runtime therefore comes from sysfs, and RCCL warns once per process when it makes that switch.
* Copy Engine collectives are no longer selected when the LSA team does not cover the communicator. The LSA-local CE routines address peers by LSA rank, so on a single node with `NCCL_LSA_TEAM_SIZE` below the local rank count they gathered only `lsaSize` slices and placed them at LSA rather than communicator indices. Such communicators now fall back to the kernel path.
Comment thread
thomas-huber marked this conversation as resolved.
Resolved
* Fixed gfx1250 LL and LL128 comm-FIFO hangs on sibling DPX partitions. The FIFO store now uses system-scope b128 (`RCCL_LL_FIFO_SYS_SCOPE`) alongside the existing system-scope load, preventing hangs at slot reuse starting from the 9th collective operation.

### Known issues
Expand Down
Loading
Loading