[RCCL] Cherry-pick gfx1250 tuning and AlltoAllv fixes (#12984, #13336, #13416, #13466) [bkc 20261013] by akankjai · Pull Request #13736 · ROCm/rocm-systems

akankjai · 2026-10-19T12:48:30Z

Cherry-picks:

fix(rccl): make CE AlltoAllv intra-batch sync count rank-invariant (AICOMRCCL-2474) #12984 fix(rccl): make CE AlltoAllv intra-batch sync count rank-invariant (AICOMRCCL-2474)
[RCCL] gfx1250: RCCL Perf tuning #13336 [RCCL] gfx1250: RCCL Perf tuning
[RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416 [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE)
fix(rccl): compile rccl_arch_thresholds.cc into topo_expl #13466 fix(rccl): compile rccl_arch_thresholds.cc into topo_expl

Cherry-picks of RCCL changes already merged to develop into release/bkc/therock-10.2-20261013 for BKC MI455 26.13 (release 10/23).

Jira: ROCM-33119

Notes

Applied in the order they merged to develop, one commit per source PR, original authors preserved, with (cherry picked from commit ...) lines.
The code files merged without conflicts. Two documentation files had context conflicts with develop commits that are not on this branch, in [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416's commit: projects/rccl/CHANGELOG.md and projects/rccl/test/host/MICROTEST_README.md. Each is resolved to the same single-line addition that [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416 makes in the original.
[RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416's CHANGELOG context mentions RCCL_TUNER_EMBEDDED_CONFIG (feat(rccl): embed CSV tuner configs in librccl.so #13038, merged to develop on 10/13). feat(rccl): embed CSV tuner configs in librccl.so #13038 is not on this branch and is not included here. [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416's code has no dependency on it.
Not included: fix(rccl): compile rccl_arch_thresholds.cc into topo_expl once #13706, an open follow-up to fix(rccl): compile rccl_arch_thresholds.cc into topo_expl #13466 on develop.

This PR is in draft until the original authors have confirmed the cherry-picks.

🤖 Generated with Claude Code

…ICOMRCCL-2474) (ROCm#12984) CE AlltoAllv deadlocks at 16+ ranks with uneven send sizes because each rank chose its intra-batch barrier count from its own nonzero peers. Fix: decide from the full size matrix and barrier per round of destination slots, so every rank issues the same count. Test results: - MI450 (gfx1250), 2 nodes, DPX, 16 ranks, rccl-tests alltoallv_perf default sparse matrix, 512M, -R 2: shipped RCCL 2.31.2 hangs 3/3, develop hangs 5/5, this fix passes 5/5 (~207 GB/s busbw). Uniform sizes pass 2/2 on develop. A 1M to 1G sweep hangs at exactly 512M on develop and passes every size with the fix. -G 1 also hangs on the shipped RCCL. - MI355X (gfx950), 1 node, 4 and 8 ranks, with NCCL_CUMEM_ENABLE=1 and a test-only sync frequency of 2 or 4 to reach the condition: develop hangs 15/15, the fix passes 15/15, controls pass, and bandwidth is unchanged. - Host microtest rccl-UnitTestsMicroCeColl: 40/40 pass with the fix, 20/40 fail on develop's ce_coll.cc; the full host suite passes. Hardware arms ran this fix before the capture-only parity barrier was added, a path CE AlltoAllv does not take since capture is not eligible. JIRA ID: AICOMRCCL-2474 (cherry picked from commit 014a242)

## Motivation The AICOMRCCL-1756 sweep on gfx1250 (8–64 ranks, every collective × datatype, 1K–16G) found that the out-of-box algorithm, protocol and channel choices leave a lot of bandwidth unused. The crossovers scale with the rank count: the per-rank constants hold from 8 to 64 ranks. On gfx1250, `RCCL_FORCE_CE=1` also forced unregistered AllGather / AlltoAll / Gather / Scatter onto the Copy Engine, which is slower than the kernel paths it replaced. This PR moves all of that tuning into the per-architecture table (`rcclArchThresholds`). No environment-variable default changes. JIRA ID : AICOMRCCL-1756 Also covers AICOMRCCL-2874, AICOMRCCL-2875, AICOMRCCL-2876. ## Technical details ### CE-Scratch (AICOMRCCL-2876) - New `rcclArchThresholds::forceCeScratch`: true on gfx942 and gfx950 (unchanged), false on gfx1250. - `RCCL_FORCE_CE` keeps its default of 1: - an explicit `RCCL_FORCE_CE=1` still forces CE-Scratch, and `0` still turns it off; - a comm-level `NCCL_CTA_POLICY_ZERO` still forces it. - gfx1250 DDA LL128 AllGather ceiling: 64 → 128 MiB. ### AllReduce LL128 (AICOMRCCL-2874 / 2875), single-node gfx1250 - Tree keeps LL128 at every size. - Ring keeps LL128 up to nRanks × 32 MiB. The fp8 Ring precision guard is unchanged. - DDA LL128 tier: 32 → 256 MiB. - The DDA LL128 two-shot ceiling follows the tier cap. `RCCL_DDA_LL128_TWOSHOT_THRESHOLD` (default 64 MiB) still overrides it. ### Rank-scaled crossovers (AICOMRCCL-1756), gfx1250 single-node comms of ≥ 8 ranks A DDA tier cap becomes nRanks × the per-rank entry, clipped to 256 MiB. Comms below 8 ranks keep the totals. | Collective | Change | |---|---| | AllReduce | DDA LL two-shot ≤ nRanks × 1 MiB; from 32 ranks, DDA VMM ahead of LL in (2 MiB, nRanks × 512 KiB] for unregistered buffers; Tree over Ring ≤ nRanks × 1 MiB, Tree LL ≤ nRanks × 32 KiB | | AllGather | DDA LL ≤ nRanks × 16 KiB; DDA LL128 ≤ min(nRanks × 16 MiB, 256 MiB); Ring LL128 ≤ 64 MiB/rank | | ReduceScatter | DDA LL / LL128 / VMM ≤ nRanks × 256 KiB / 512 KiB / 2 MiB; Ring LL128 512 KiB–8 MiB/rank (32 MiB for 1-byte types); Ring Simple capped at 128 channels for (8, 32] MiB/rank | | DDA VMM AR / RS | 2048 / nRanks blocks, clamped to [32, 128], never above the init-time block count | ### Hooks and user overrides - Two new hooks in `topoGetAlgoInfo`: - `rcclUpdateCollectiveAlgorithm`, before the protocol hook; - `rcclUpdateCollectiveChannels`, after `rcclOverrideChannels`. - User settings still win: `NCCL_ALGO`, `NCCL_PROTO`, `NCCL_MIN/MAX_NCHANNELS` and `RCCL_DDA_*_THRESHOLD`. ## Test plan / results **Host microtests** (`test/host`): - New tests cover: - the CE-Scratch resolver; - the tier-cap and two-shot resolvers: rank scaling, the 256 MiB clip, env precedence, empty env; - the AllReduce VMM-first window: bounds, unregistered only, yields to the env thresholds; - the VMM block count; - the algorithm, protocol and channel hooks, with multi-node and env controls; - `topoGetAlgoInfo` calling each hook exactly once, in order. - The full suite passes, including the `ENABLE_LL128` build. - Mutation gate: each of the 41 mutants of the tuning branches fails at least one test. **GPU unit tests** (`rccl-UnitTestsFixturesDebug`): - per-arch `forceCeScratch`; - rank scaling applies only on gfx1250; - AllReduce LL / LL128 two-shot ceilings; - DDA VMM launch block count; - AllGather LL128 cap at 8 and 16 ranks. `DdaFabricLL128EligibilityTest.*` is added to the fixtures-debug categories. **Env-var CI check** (`rccl_check_default_behavior.py`): no undocumented env-variable changes. ### Targets: gains measured by the 1756 sweep (best-tuned vs out-of-box, out-of-place busbw) | Collective | Ranks | Sizes | Sweep gain | Path this PR picks | |---|---|---|---|---| | AllReduce float/bf16 | 8 / 16 | 4M–8M / 4M–16M | 1.37–1.91x | DDA LL two-shot | | AllReduce float/bf16 | 32 / 64 | 4M–32M | 1.36–3.66x | DDA VMM first, DDA LL two-shot | | AllReduce float/bf16 | 32 / 64 | 1G–2G | 1.23–1.34x | Ring LL128 (nRanks × 32 MiB) | | AllReduce fp8 | 8 | 1K–256K | 2.0–2.3x | Tree LL | | AllReduce fp8 | 8 | 512K–4M | 1.44–1.81x | Tree LL128 | | AllReduce fp8 | 8 | 128M–256M | 1.15–1.39x | Ring LL128 | | AllReduce fp8 | 16–64 | 256K–2M | 1.18–1.64x | Tree LL | | AllGather | 8–64 | 128M–4G | 1.32–3.56x | CE-Scratch off → DDA / Ring LL128 | | AllGather | 16 / 64 | 256M | 1.39x / 1.96x | DDA LL128 (256 MiB) | | AllGather | 16–64 | 256M–2G | up to 1.9x | Ring LL128 (64 MiB/rank) | | AllGather | 16 / 32 | 256K–512K | 1.18–1.44x | DDA LL | | ReduceScatter | 8–64 | 4M–128M | 1.16–2.87x | DDA LL128 / VMM | | ReduceScatter | 8–64 | 64M–512M | 1.36–1.48x | Ring LL128 | | ReduceScatter fp8 | 8–64 | 8M–2G | 1.15–1.45x | Ring LL128 (1-byte band) | | ReduceScatter | 8–64 | next 1–2 sizes up | 1.10–1.33x | Ring Simple, 128 channels | ### H13 validation (this build, SPX) **Method:** - Run the 1756 out-of-box recipe (`-b 1K -e 16G -f 2 -g 1 -n 50 -w 20 -c 1 -A 1 -M 1 -G 0 -R 0`) against this build. - Compare each size against the sweep's out-of-box (pre-tune) number. - Cover every collective × datatype at 8, 16 and 32 ranks. - Flag anything below 0.95x of pre-tune as a regression. | Ranks | Collective | Size | Develop path | This PR path | Develop busbw (GB/s) | This PR busbw (GB/s) | Develop latency per call (µs) | This PR latency per call (µs) | Gain | |---|---|---|---|---|---|---|---|---|---| | 8 | AllReduce float | 4M | DDA LL128 | DDA LL | 105.66 | 167.08 | 69.5 us | 43.9 us | 1.58x | | 8 | AllReduce float | 8M | DDA LL128 | DDA LL | 170.53 | 242.23 | 86.1 us | 60.6 us | 1.42x | | 8 | AllReduce float | 64M | RING SIMPLE | DDA LL128 | 323.42 | 474.94 | 363.1 us | 247.3 us | 1.47x | | 8 | AllReduce float | 128M | RING SIMPLE | DDA LL128 | 483.12 | 581.88 | 486.2 us | 403.7 us | 1.20x | | 8 | AllReduce bfloat16 | 4M | DDA LL128 | DDA LL | 113.50 | 164.29 | 64.7 us | 44.7 us | 1.45x | | 8 | AllReduce bfloat16 | 8M | DDA LL128 | DDA LL | 167.89 | 249.68 | 87.4 us | 58.8 us | 1.49x | | 8 | AllReduce bfloat16 | 64M | RING SIMPLE | DDA LL128 | 328.01 | 484.09 | 358.0 us | 242.6 us | 1.48x | | 8 | AllReduce bfloat16 | 128M | RING SIMPLE | DDA LL128 | 488.33 | 584.87 | 481.0 us | 401.6 us | 1.20x | | 8 | AllReduce fp8 e4m3 | 1K | RING LL | TREE LL | 0.02 | 0.04 | 116.8 us | 47.0 us | 2.00x | | 8 | AllReduce fp8 e4m3 | 2K | RING LL | TREE LL | 0.03 | 0.08 | 112.3 us | 44.3 us | 2.67x | | 8 | AllReduce fp8 e4m3 | 4K | RING LL | TREE LL | 0.06 | 0.15 | 112.2 us | 47.0 us | 2.58x | | 8 | AllReduce fp8 e4m3 | 8K | RING LL | TREE LL | 0.13 | 0.29 | 114.2 us | 49.6 us | 2.19x | | 8 | AllReduce fp8 e4m3 | 16K | RING LL | TREE LL | 0.24 | 0.57 | 119.8 us | 50.2 us | 2.38x | | 8 | AllReduce fp8 e4m3 | 32K | RING LL | TREE LL | 0.47 | 1.12 | 120.1 us | 51.1 us | 2.36x | | 8 | AllReduce fp8 e4m3 | 64K | RING LL | TREE LL | 0.95 | 2.20 | 120.4 us | 52.1 us | 2.30x | | 8 | AllReduce fp8 e4m3 | 128K | RING LL | TREE LL | 1.90 | 4.35 | 121.1 us | 52.8 us | 2.29x | | 8 | AllReduce fp8 e4m3 | 256K | RING LL128 | TREE LL | 2.94 | 8.29 | 155.9 us | 55.4 us | 2.82x | | 8 | AllReduce fp8 e4m3 | 32M | RING SIMPLE | RING LL128 | 185.46 | 288.56 | 316.6 us | 203.5 us | 1.56x | | 8 | AllReduce fp8 e4m3 | 64M | RING SIMPLE | RING LL128 | 277.36 | 367.00 | 423.4 us | 321.5 us | 1.32x | | 8 | AllReduce fp8 e5m2 | 1K | RING LL | TREE LL | 0.02 | 0.04 | 115.6 us | 46.2 us | 2.00x | | 8 | AllReduce fp8 e5m2 | 2K | RING LL | TREE LL | 0.03 | 0.08 | 111.2 us | 44.1 us | 2.67x | | 8 | AllReduce fp8 e5m2 | 4K | RING LL | TREE LL | 0.06 | 0.15 | 111.6 us | 46.8 us | 2.58x | | 8 | AllReduce fp8 e5m2 | 8K | RING LL | TREE LL | 0.12 | 0.29 | 114.6 us | 49.1 us | 2.32x | | 8 | AllReduce fp8 e5m2 | 16K | RING LL | TREE LL | 0.24 | 0.57 | 119.6 us | 50.3 us | 2.38x | | 8 | AllReduce fp8 e5m2 | 32K | RING LL | TREE LL | 0.48 | 1.12 | 119.5 us | 51.4 us | 2.32x | | 8 | AllReduce fp8 e5m2 | 64K | RING LL | TREE LL | 0.95 | 2.20 | 120.8 us | 52.2 us | 2.31x | | 8 | AllReduce fp8 e5m2 | 128K | RING LL | TREE LL | 1.87 | 4.33 | 122.6 us | 52.9 us | 2.32x | | 8 | AllReduce fp8 e5m2 | 256K | RING LL128 | TREE LL | 2.96 | 8.32 | 155.1 us | 55.1 us | 2.81x | | 8 | AllReduce fp8 e5m2 | 32M | RING SIMPLE | RING LL128 | 183.75 | 283.10 | 319.6 us | 207.4 us | 1.54x | | 8 | AllReduce fp8 e5m2 | 64M | RING SIMPLE | RING LL128 | 281.64 | 379.33 | 417.0 us | 315.4 us | 1.35x | | 8 | ReduceScatter float | 16M | RING LL128 | DDA SIMPLE | 140.19 | 222.18 | 104.7 us | 66.1 us | 1.58x | | 8 | ReduceScatter float | 64M | RING SIMPLE | RING LL128 | 287.11 | 353.02 | 204.5 us | 168.8 us | 1.23x | | 8 | ReduceScatter bfloat16 | 16M | RING LL128 | DDA SIMPLE | 144.00 | 209.76 | 102.0 us | 70.0 us | 1.46x | | 8 | ReduceScatter bfloat16 | 64M | RING SIMPLE | RING LL128 | 287.77 | 411.06 | 204.1 us | 142.9 us | 1.43x | | 8 | ReduceScatter fp8 e4m3 | 64M | RING SIMPLE | RING LL128 | 233.13 | 278.25 | 251.9 us | 211.1 us | 1.19x | | 8 | ReduceScatter fp8 e5m2 | 64M | RING SIMPLE | RING LL128 | 236.88 | 271.75 | 247.9 us | 216.2 us | 1.15x | | 8 | AllGather float | 128M | CE-Scratch | DDA LL128 | 420.67 | 503.71 | 279.2 us | 233.2 us | 1.20x | | 8 | AllGather float | 256M | CE-Scratch | RING LL128 | 474.95 | 568.12 | 494.6 us | 413.5 us | 1.20x | | 8 | AllGather float | 512M | CE-Scratch | RING LL128 | 508.27 | 597.82 | 924.2 us | 786.1 us | 1.18x | | 8 | AllToAll float | 8M | CE-Scratch | Direct SIMPLE | 126.46 | 153.56 | 58.0 us | 47.8 us | 1.21x | | 8 | AllToAll float | 16M | CE-Scratch | Direct SIMPLE | 173.81 | 252.06 | 84.5 us | 58.4 us | 1.45x | | 8 | AllToAll float | 32M | CE-Scratch | Direct SIMPLE | 256.82 | 331.63 | 114.3 us | 88.8 us | 1.29x | | 8 | AllToAll float | 64M | CE-Scratch | Direct SIMPLE | 345.90 | 404.91 | 169.8 us | 145.0 us | 1.17x | | 8 | AllToAll float | 128M | CE-Scratch | Direct SIMPLE | 421.47 | 471.89 | 278.7 us | 248.9 us | 1.12x | | 8 | AllToAll float | 256M | CE-Scratch | Direct SIMPLE | 473.12 | 538.44 | 496.5 us | 436.2 us | 1.14x | | 8 | AllToAll float | 512M | CE-Scratch | Direct SIMPLE | 507.23 | 569.11 | 926.1 us | 825.4 us | 1.12x | | 8 | Gather float | 1K | CE-Scratch* | P2P kernel* | 0.03 | 0.04 | 27.3 us | 21.2 us | 1.33x | | 8 | Gather float | 2K | CE-Scratch* | P2P kernel* | 0.07 | 0.12 | 27.9 us | 14.4 us | 1.92x | | 8 | Gather float | 4K | CE-Scratch* | P2P kernel* | 0.12 | 0.24 | 27.7 us | 14.4 us | 1.96x | | 8 | Gather float | 8K | CE-Scratch* | P2P kernel* | 0.26 | 0.53 | 27.4 us | 13.8 us | 2.02x | | 8 | Gather float | 16K | CE-Scratch* | P2P kernel* | 0.53 | 1.02 | 27.5 us | 14.1 us | 1.94x | | 8 | Gather float | 32K | CE-Scratch* | P2P kernel* | 1.05 | 1.98 | 27.4 us | 14.5 us | 1.89x | | 8 | Gather float | 64K | CE-Scratch* | P2P kernel* | 2.08 | 3.00 | 27.6 us | 19.1 us | 1.44x | | 8 | Gather float | 128K | CE-Scratch* | P2P kernel* | 4.11 | 5.83 | 27.9 us | 19.7 us | 1.42x | | 8 | Gather float | 256K | CE-Scratch* | P2P kernel* | 8.20 | 11.62 | 28.0 us | 19.7 us | 1.42x | | 8 | Gather float | 512K | CE-Scratch* | P2P kernel* | 16.13 | 22.67 | 28.4 us | 20.2 us | 1.41x | | 8 | Gather float | 1M | CE-Scratch* | P2P kernel* | 32.10 | 42.86 | 28.6 us | 21.4 us | 1.34x | | 8 | Gather float | 2M | CE-Scratch* | P2P kernel* | 62.34 | 87.41 | 29.4 us | 21.0 us | 1.40x | | 8 | Gather float | 4M | CE-Scratch* | P2P kernel* | 119.63 | 157.50 | 30.7 us | 23.3 us | 1.32x | | 8 | Gather float | 8M | CE-Scratch* | P2P kernel* | 204.02 | 291.04 | 36.0 us | 25.2 us | 1.43x | | 8 | Gather float | 16M | CE-Scratch* | P2P kernel* | 188.58 | 460.22 | 77.8 us | 31.9 us | 2.44x | | 8 | Gather float | 32M | CE-Scratch* | P2P kernel* | 245.15 | 570.25 | 119.8 us | 51.5 us | 2.33x | | 8 | Gather float | 64M | CE-Scratch* | P2P kernel* | 310.10 | 649.12 | 189.4 us | 90.5 us | 2.09x | | 8 | Gather float | 128M | CE-Scratch* | P2P kernel* | 439.15 | 679.25 | 267.4 us | 172.9 us | 1.55x | | 8 | Gather float | 256M | CE-Scratch* | P2P kernel* | 504.27 | 691.88 | 465.8 us | 339.5 us | 1.37x | | 8 | Gather float | 512M | CE-Scratch* | P2P kernel* | 540.88 | 713.38 | 868.5 us | 658.5 us | 1.32x | | 8 | Scatter float | 1K | CE-Scratch* | P2P kernel* | 0.03 | 0.04 | 27.7 us | 21.7 us | 1.33x | | 8 | Scatter float | 2K | CE-Scratch* | P2P kernel* | 0.07 | 0.12 | 27.3 us | 15.7 us | 1.77x | | 8 | Scatter float | 4K | CE-Scratch* | P2P kernel* | 0.13 | 0.26 | 27.6 us | 13.7 us | 2.00x | | 8 | Scatter float | 8K | CE-Scratch* | P2P kernel* | 0.26 | 0.50 | 27.5 us | 14.3 us | 1.92x | | 8 | Scatter float | 16K | CE-Scratch* | P2P kernel* | 0.52 | 1.02 | 27.6 us | 14.1 us | 1.95x | | 8 | Scatter float | 32K | CE-Scratch* | P2P kernel* | 1.04 | 1.88 | 27.6 us | 15.2 us | 1.81x | | 8 | Scatter float | 64K | CE-Scratch* | P2P kernel* | 2.07 | 3.02 | 27.7 us | 19.0 us | 1.46x | | 8 | Scatter float | 128K | CE-Scratch* | P2P kernel* | 4.14 | 5.55 | 27.7 us | 20.7 us | 1.34x | | 8 | Scatter float | 256K | CE-Scratch* | P2P kernel* | 8.23 | 10.75 | 27.8 us | 21.3 us | 1.31x | | 8 | Scatter float | 512K | CE-Scratch* | P2P kernel* | 16.27 | 21.34 | 28.2 us | 21.5 us | 1.31x | | 8 | Scatter float | 1M | CE-Scratch* | P2P kernel* | 31.91 | 42.84 | 28.8 us | 21.4 us | 1.34x | | 8 | Scatter float | 2M | CE-Scratch* | P2P kernel* | 55.33 | 81.12 | 33.2 us | 22.6 us | 1.47x | | 8 | Scatter float | 4M | CE-Scratch* | P2P kernel* | 98.10 | 141.26 | 37.4 us | 26.0 us | 1.44x | | 8 | Scatter float | 8M | CE-Scratch* | P2P kernel* | 150.78 | 242.16 | 48.7 us | 30.4 us | 1.61x | | 8 | Scatter float | 16M | CE-Scratch* | P2P kernel* | 204.38 | 359.59 | 71.8 us | 40.9 us | 1.76x | | 8 | Scatter float | 32M | CE-Scratch* | P2P kernel* | 316.86 | 439.60 | 92.7 us | 66.8 us | 1.39x | | 8 | Scatter float | 64M | CE-Scratch* | P2P kernel* | 440.25 | 528.46 | 133.4 us | 111.1 us | 1.20x | | 16 | AllReduce float | 4M | DDA LL128 | DDA LL | 62.12 | 96.31 | 126.6 us | 81.7 us | 1.55x | | 16 | AllReduce float | 8M | DDA LL128 | DDA LL | 89.16 | 172.88 | 176.4 us | 91.0 us | 1.94x | | 16 | AllReduce float | 16M | DDA LL128 | DDA LL | 171.61 | 253.22 | 183.3 us | 124.2 us | 1.48x | | 16 | AllReduce float | 64M | RING SIMPLE | DDA LL128 | 234.62 | 437.57 | 536.3 us | 287.6 us | 1.86x | | 16 | AllReduce float | 128M | RING SIMPLE | DDA LL128 | 345.06 | 473.54 | 729.3 us | 531.5 us | 1.37x | | 16 | AllReduce bfloat16 | 4M | DDA LL128 | DDA LL | 54.63 | 92.61 | 144.1 us | 84.9 us | 1.70x | | 16 | AllReduce bfloat16 | 8M | DDA LL128 | DDA LL | 89.00 | 171.86 | 176.7 us | 91.5 us | 1.93x | | 16 | AllReduce bfloat16 | 16M | DDA LL128 | DDA LL | 171.57 | 251.19 | 183.4 us | 125.3 us | 1.46x | | 16 | AllReduce bfloat16 | 64M | RING SIMPLE | DDA LL128 | 239.94 | 445.31 | 524.4 us | 282.6 us | 1.86x | | 16 | AllReduce bfloat16 | 128M | RING SIMPLE | DDA LL128 | 348.99 | 473.12 | 721.1 us | 531.9 us | 1.36x | | 16 | AllReduce fp8 e4m3 | 256K | TREE LL128 | TREE LL | 3.02 | 4.89 | 162.3 us | 100.4 us | 1.62x | | 16 | AllReduce fp8 e4m3 | 512K | TREE LL128 | TREE LL | 6.08 | 9.34 | 161.7 us | 105.3 us | 1.54x | | 16 | AllReduce fp8 e4m3 | 32M | TREE SIMPLE | TREE LL128 | 61.97 | 170.18 | 1015.3 us | 369.7 us | 2.75x | | 16 | AllReduce fp8 e4m3 | 64M | TREE SIMPLE | TREE LL128 | 102.75 | 220.62 | 1224.5 us | 570.4 us | 2.15x | | 16 | AllReduce fp8 e4m3 | 128M | TREE SIMPLE | TREE LL128 | 129.63 | 259.69 | 1941.4 us | 969.1 us | 2.00x | | 16 | AllReduce fp8 e4m3 | 256M | TREE SIMPLE | TREE LL128 | 152.57 | 313.91 | 3298.8 us | 1603.4 us | 2.06x | | 16 | AllReduce fp8 e4m3 | 512M | TREE SIMPLE | TREE LL128 | 203.81 | 303.29 | 4938.9 us | 3319.1 us | 1.49x | | 16 | AllReduce fp8 e4m3 | 1G | TREE SIMPLE | TREE LL128 | 226.92 | 355.06 | 8872.4 us | 5670.3 us | 1.56x | | 16 | AllReduce fp8 e4m3 | 2G | TREE SIMPLE | TREE LL128 | 253.85 | 366.41 | 15.86 ms | 10.99 ms | 1.44x | | 16 | AllReduce fp8 e4m3 | 4G | TREE SIMPLE | TREE LL128 | 290.73 | 371.31 | 27.70 ms | 21.69 ms | 1.28x | | 16 | AllReduce fp8 e4m3 | 8G | TREE SIMPLE | TREE LL128 | 308.42 | 367.52 | 52.22 ms | 43.82 ms | 1.19x | | 16 | AllReduce fp8 e4m3 | 16G | TREE SIMPLE | TREE LL128 | 314.48 | 357.79 | 102.43 ms | 90.06 ms | 1.14x | | 16 | AllReduce fp8 e5m2 | 256K | TREE LL128 | TREE LL | 2.98 | 4.97 | 165.1 us | 98.9 us | 1.67x | | 16 | AllReduce fp8 e5m2 | 512K | TREE LL128 | TREE LL | 5.92 | 9.50 | 166.2 us | 103.4 us | 1.61x | | 16 | AllReduce fp8 e5m2 | 32M | TREE SIMPLE | TREE LL128 | 61.64 | 175.80 | 1020.7 us | 357.9 us | 2.85x | | 16 | AllReduce fp8 e5m2 | 64M | TREE SIMPLE | TREE LL128 | 103.28 | 221.03 | 1218.3 us | 569.3 us | 2.14x | | 16 | AllReduce fp8 e5m2 | 128M | TREE SIMPLE | TREE LL128 | 131.32 | 263.46 | 1916.4 us | 955.2 us | 2.01x | | 16 | AllReduce fp8 e5m2 | 256M | TREE SIMPLE | TREE LL128 | 153.84 | 308.48 | 3271.6 us | 1631.7 us | 2.01x | | 16 | AllReduce fp8 e5m2 | 512M | TREE SIMPLE | TREE LL128 | 204.20 | 305.37 | 4929.5 us | 3296.5 us | 1.50x | | 16 | AllReduce fp8 e5m2 | 1G | TREE SIMPLE | TREE LL128 | 227.17 | 357.25 | 8862.4 us | 5635.4 us | 1.57x | | 16 | AllReduce fp8 e5m2 | 2G | TREE SIMPLE | TREE LL128 | 254.86 | 368.46 | 15.80 ms | 10.93 ms | 1.45x | | 16 | AllReduce fp8 e5m2 | 4G | TREE SIMPLE | TREE LL128 | 291.60 | 374.08 | 27.62 ms | 21.53 ms | 1.28x | | 16 | AllReduce fp8 e5m2 | 8G | TREE SIMPLE | TREE LL128 | 309.65 | 372.68 | 52.01 ms | 43.22 ms | 1.20x | | 16 | AllReduce fp8 e5m2 | 16G | TREE SIMPLE | TREE LL128 | 316.22 | 369.93 | 101.87 ms | 87.08 ms | 1.17x | | 16 | ReduceScatter float | 32M | RING LL128 | DDA SIMPLE | 161.54 | 208.00 | 194.8 us | 151.3 us | 1.29x | | 16 | ReduceScatter float | 128M | RING SIMPLE | RING LL128 | 330.13 | 390.68 | 381.2 us | 322.2 us | 1.18x | | 16 | ReduceScatter bfloat16 | 32M | RING LL128 | DDA SIMPLE | 164.09 | 208.36 | 191.7 us | 151.0 us | 1.27x | | 16 | ReduceScatter bfloat16 | 128M | RING SIMPLE | RING LL128 | 327.03 | 388.49 | 384.8 us | 324.0 us | 1.19x | | 16 | ReduceScatter fp8 e4m3 | 256M | RING SIMPLE | RING LL128 | 368.14 | 411.19 | 683.8 us | 612.2 us | 1.12x | | 16 | AllGather float | 256K | DDA LL128 | DDA LL | 15.20 | 24.86 | 16.2 us | 9.9 us | 1.64x | | 16 | AllGather float | 128M | CE-Scratch | DDA LL128 | 370.12 | 492.19 | 340.0 us | 255.7 us | 1.33x | | 16 | AllGather float | 256M | CE-Scratch | DDA LL128 | 441.38 | 522.77 | 570.2 us | 481.4 us | 1.18x | | 16 | AllGather float | 512M | CE-Scratch | RING LL128 | 477.76 | 621.03 | 1053.5 us | 811.2 us | 1.30x | | 16 | AllGather float | 1G | CE-Scratch | RING LL128 | 500.23 | 638.51 | 2012.4 us | 1576.7 us | 1.28x | | 16 | AllToAll float | 2M | CE-Scratch | Direct SIMPLE | 36.20 | 50.44 | 54.3 us | 39.0 us | 1.39x | | 16 | AllToAll float | 4M | CE-Scratch | Direct SIMPLE | 66.95 | 95.71 | 58.7 us | 41.1 us | 1.43x | | 16 | AllToAll float | 8M | CE-Scratch | Direct SIMPLE | 109.91 | 170.43 | 71.6 us | 46.1 us | 1.55x | | 16 | AllToAll float | 16M | CE-Scratch | Direct SIMPLE | 167.31 | 266.40 | 94.0 us | 59.0 us | 1.59x | | 16 | AllToAll float | 32M | CE-Scratch | Direct SIMPLE | 119.53 | 349.04 | 263.3 us | 90.1 us | 2.92x | | 16 | AllToAll float | 64M | CE-Scratch | Direct SIMPLE | 159.03 | 421.15 | 395.6 us | 149.4 us | 2.65x | | 16 | AllToAll float | 128M | CE-Scratch | Direct SIMPLE | 225.28 | 478.93 | 558.6 us | 262.8 us | 2.13x | | 16 | AllToAll float | 256M | CE-Scratch | Direct SIMPLE | 266.93 | 530.88 | 942.8 us | 474.1 us | 1.99x | | 16 | AllToAll float | 512M | CE-Scratch | Direct SIMPLE | 478.78 | 562.65 | 1051.3 us | 894.6 us | 1.18x | | 16 | AllToAll float | 1G | CE-Scratch | Direct SIMPLE | 503.10 | 594.06 | 2000.8 us | 1694.5 us | 1.18x | | 16 | Gather float | 1K | CE-Scratch* | P2P kernel* | 0.02 | 0.04 | 45.0 us | 23.8 us | 2.00x | | 16 | Gather float | 2K | CE-Scratch* | P2P kernel* | 0.04 | 0.11 | 45.9 us | 17.2 us | 2.87x | | 16 | Gather float | 4K | CE-Scratch* | P2P kernel* | 0.08 | 0.24 | 46.0 us | 15.7 us | 3.06x | | 16 | Gather float | 8K | CE-Scratch* | P2P kernel* | 0.17 | 0.45 | 45.3 us | 17.0 us | 2.68x | | 16 | Gather float | 16K | CE-Scratch* | P2P kernel* | 0.34 | 0.97 | 45.3 us | 16.2 us | 2.84x | | 16 | Gather float | 32K | CE-Scratch* | P2P kernel* | 0.68 | 1.81 | 45.4 us | 17.0 us | 2.69x | | 16 | Gather float | 64K | CE-Scratch* | P2P kernel* | 1.36 | 3.73 | 45.2 us | 16.7 us | 2.74x | | 16 | Gather float | 128K | CE-Scratch* | P2P kernel* | 2.71 | 5.87 | 45.4 us | 21.1 us | 2.16x | | 16 | Gather float | 256K | CE-Scratch* | P2P kernel* | 5.30 | 11.48 | 46.3 us | 21.6 us | 2.16x | | 16 | Gather float | 512K | CE-Scratch* | P2P kernel* | 10.63 | 20.85 | 46.2 us | 23.8 us | 1.96x | | 16 | Gather float | 1M | CE-Scratch* | P2P kernel* | 21.15 | 46.30 | 46.5 us | 21.3 us | 2.19x | | 16 | Gather float | 2M | CE-Scratch* | P2P kernel* | 42.49 | 83.68 | 46.3 us | 23.5 us | 1.97x | | 16 | Gather float | 4M | CE-Scratch* | P2P kernel* | 84.13 | 162.68 | 46.7 us | 24.2 us | 1.93x | | 16 | Gather float | 8M | CE-Scratch* | P2P kernel* | 163.54 | 302.44 | 48.1 us | 26.1 us | 1.85x | | 16 | Gather float | 16M | CE-Scratch* | P2P kernel* | 264.16 | 497.50 | 59.6 us | 31.7 us | 1.88x | | 16 | Gather float | 32M | CE-Scratch* | P2P kernel* | 217.53 | 592.82 | 144.6 us | 53.4 us | 2.73x | | 16 | Gather float | 64M | CE-Scratch* | P2P kernel* | 272.05 | 674.32 | 231.3 us | 93.3 us | 2.48x | | 16 | Gather float | 128M | CE-Scratch* | P2P kernel* | 334.03 | 710.39 | 376.7 us | 177.2 us | 2.13x | | 16 | Gather float | 256M | CE-Scratch* | P2P kernel* | 459.98 | 720.77 | 547.1 us | 349.2 us | 1.57x | | 16 | Gather float | 512M | CE-Scratch* | P2P kernel* | 522.46 | 723.37 | 963.4 us | 695.8 us | 1.38x | | 16 | Gather float | 1G | CE-Scratch* | P2P kernel* | 551.57 | 730.95 | 1825.0 us | 1377.1 us | 1.33x | | 16 | Scatter float | 1K | CE-Scratch* | P2P kernel* | 0.02 | 0.05 | 42.7 us | 19.9 us | 2.50x | | 16 | Scatter float | 2K | CE-Scratch* | P2P kernel* | 0.04 | 0.12 | 42.9 us | 15.2 us | 2.78x | | 16 | Scatter float | 4K | CE-Scratch* | P2P kernel* | 0.09 | 0.24 | 43.2 us | 15.7 us | 2.72x | | 16 | Scatter float | 8K | CE-Scratch* | P2P kernel* | 0.17 | 0.50 | 43.6 us | 15.2 us | 2.86x | | 16 | Scatter float | 16K | CE-Scratch* | P2P kernel* | 0.35 | 1.00 | 43.3 us | 15.3 us | 2.83x | | 16 | Scatter float | 32K | CE-Scratch* | P2P kernel* | 0.70 | 2.05 | 43.5 us | 14.9 us | 2.91x | | 16 | Scatter float | 64K | CE-Scratch* | P2P kernel* | 1.42 | 4.01 | 43.1 us | 15.3 us | 2.81x | | 16 | Scatter float | 128K | CE-Scratch* | P2P kernel* | 2.87 | 5.97 | 42.9 us | 20.6 us | 2.08x | | 16 | Scatter float | 256K | CE-Scratch* | P2P kernel* | 5.84 | 12.06 | 42.1 us | 20.4 us | 2.07x | | 16 | Scatter float | 512K | CE-Scratch* | P2P kernel* | 11.59 | 22.02 | 42.4 us | 22.3 us | 1.90x | | 16 | Scatter float | 1M | CE-Scratch* | P2P kernel* | 22.77 | 44.12 | 43.2 us | 22.3 us | 1.94x | | 16 | Scatter float | 2M | CE-Scratch* | P2P kernel* | 40.95 | 84.75 | 48.0 us | 23.2 us | 2.07x | | 16 | Scatter float | 4M | CE-Scratch* | P2P kernel* | 74.37 | 146.88 | 52.9 us | 26.8 us | 1.97x | | 16 | Scatter float | 8M | CE-Scratch* | P2P kernel* | 122.50 | 246.18 | 64.2 us | 31.9 us | 2.01x | | 16 | Scatter float | 16M | CE-Scratch* | P2P kernel* | 196.52 | 370.55 | 80.0 us | 42.5 us | 1.89x | | 16 | Scatter float | 32M | CE-Scratch* | P2P kernel* | 239.56 | 448.07 | 131.3 us | 70.2 us | 1.87x | | 16 | Scatter float | 64M | CE-Scratch* | P2P kernel* | 352.81 | 540.71 | 178.3 us | 116.4 us | 1.53x | | 16 | Scatter float | 128M | CE-Scratch* | P2P kernel* | 486.11 | 606.51 | 258.9 us | 207.5 us | 1.25x | | 16 | Scatter float | 256M | CE-Scratch* | P2P kernel* | 584.59 | 650.20 | 430.5 us | 387.0 us | 1.11x | | 32 | AllReduce float | 4M | DDA LL128 | DDA SIMPLE | 31.34 | 73.44 | 259.3 us | 110.7 us | 2.34x | | 32 | AllReduce float | 8M | DDA LL128 | DDA SIMPLE | 48.62 | 125.58 | 334.3 us | 129.4 us | 2.58x | | 32 | AllReduce float | 16M | DDA LL128 | DDA SIMPLE | 93.79 | 175.18 | 346.6 us | 185.6 us | 1.87x | | 32 | AllReduce float | 32M | DDA LL128 | DDA LL | 180.88 | 256.34 | 359.4 us | 253.6 us | 1.42x | | 32 | AllReduce float | 64M | RING SIMPLE | DDA LL128 | 137.66 | 328.43 | 944.5 us | 395.9 us | 2.39x | | 32 | AllReduce float | 128M | RING SIMPLE | DDA LL128 | 244.69 | 459.11 | 1062.8 us | 566.5 us | 1.88x | | 32 | AllReduce float | 256M | RING SIMPLE | DDA LL128 | 355.84 | 485.71 | 1461.6 us | 1070.8 us | 1.36x | | 32 | AllReduce bfloat16 | 4M | DDA LL128 | DDA SIMPLE | 29.98 | 73.56 | 271.1 us | 110.5 us | 2.45x | | 32 | AllReduce bfloat16 | 8M | DDA LL128 | DDA SIMPLE | 48.45 | 125.09 | 335.4 us | 129.9 us | 2.58x | | 32 | AllReduce bfloat16 | 16M | DDA LL128 | DDA SIMPLE | 94.71 | 175.10 | 343.2 us | 185.6 us | 1.85x | | 32 | AllReduce bfloat16 | 32M | DDA LL128 | DDA LL | 181.96 | 255.94 | 357.3 us | 254.0 us | 1.41x | | 32 | AllReduce bfloat16 | 64M | RING SIMPLE | DDA LL128 | 138.31 | 327.41 | 940.1 us | 397.1 us | 2.37x | | 32 | AllReduce bfloat16 | 128M | RING SIMPLE | DDA LL128 | 245.81 | 451.27 | 1057.9 us | 576.3 us | 1.84x | | 32 | AllReduce bfloat16 | 256M | RING SIMPLE | DDA LL128 | 356.18 | 485.70 | 1460.2 us | 1070.8 us | 1.36x | | 32 | AllReduce fp8 e4m3 | 256K | TREE LL128 | TREE LL | 1.62 | 2.77 | 312.8 us | 183.5 us | 1.70x | | 32 | AllReduce fp8 e4m3 | 512K | TREE LL128 | TREE LL | 3.19 | 5.37 | 319.0 us | 189.1 us | 1.69x | | 32 | AllReduce fp8 e4m3 | 1M | TREE LL128 | TREE LL | 6.00 | 10.30 | 338.2 us | 197.2 us | 1.72x | | 32 | AllReduce fp8 e4m3 | 32M | TREE SIMPLE | TREE LL128 | 41.99 | 122.41 | 1548.2 us | 531.1 us | 2.91x | | 32 | AllReduce fp8 e4m3 | 64M | TREE SIMPLE | TREE LL128 | 61.30 | 172.03 | 2121.2 us | 755.8 us | 2.81x | | 32 | AllReduce fp8 e4m3 | 128M | TREE SIMPLE | TREE LL128 | 99.39 | 228.26 | 2616.4 us | 1139.3 us | 2.30x | | 32 | AllReduce fp8 e4m3 | 256M | TREE SIMPLE | TREE LL128 | 131.54 | 280.20 | 3953.8 us | 1856.1 us | 2.13x | | 32 | AllReduce fp8 e4m3 | 512M | TREE SIMPLE | TREE LL128 | 154.98 | 300.02 | 6711.7 us | 3467.0 us | 1.94x | | 32 | AllReduce fp8 e4m3 | 1G | TREE SIMPLE | TREE LL128 | 207.81 | 342.78 | 10.01 ms | 6069.1 us | 1.65x | | 32 | AllReduce fp8 e4m3 | 2G | TREE SIMPLE | TREE LL128 | 232.58 | 360.24 | 17.89 ms | 11.55 ms | 1.55x | | 32 | AllReduce fp8 e4m3 | 4G | TREE SIMPLE | TREE LL128 | 259.46 | 373.01 | 32.07 ms | 22.31 ms | 1.44x | | 32 | AllReduce fp8 e4m3 | 8G | TREE SIMPLE | TREE LL128 | 297.74 | 369.11 | 55.90 ms | 45.09 ms | 1.24x | | 32 | AllReduce fp8 e4m3 | 16G | TREE SIMPLE | TREE LL128 | 314.50 | 362.18 | 105.84 ms | 91.91 ms | 1.15x | | 32 | AllReduce fp8 e5m2 | 256K | TREE LL128 | TREE LL | 1.63 | 2.77 | 312.1 us | 183.3 us | 1.70x | | 32 | AllReduce fp8 e5m2 | 512K | TREE LL128 | TREE LL | 3.23 | 5.36 | 314.0 us | 189.5 us | 1.66x | | 32 | AllReduce fp8 e5m2 | 1M | TREE LL128 | TREE LL | 6.03 | 10.22 | 336.8 us | 198.7 us | 1.70x | | 32 | AllReduce fp8 e5m2 | 32M | TREE SIMPLE | TREE LL128 | 42.25 | 121.82 | 1538.6 us | 533.7 us | 2.88x | | 32 | AllReduce fp8 e5m2 | 64M | TREE SIMPLE | TREE LL128 | 61.36 | 167.12 | 2119.0 us | 778.0 us | 2.72x | | 32 | AllReduce fp8 e5m2 | 128M | TREE SIMPLE | TREE LL128 | 100.28 | 227.09 | 2593.4 us | 1145.1 us | 2.26x | | 32 | AllReduce fp8 e5m2 | 256M | TREE SIMPLE | TREE LL128 | 132.13 | 276.62 | 3936.1 us | 1880.2 us | 2.09x | | 32 | AllReduce fp8 e5m2 | 512M | TREE SIMPLE | TREE LL128 | 155.37 | 301.13 | 6694.8 us | 3454.2 us | 1.94x | | 32 | AllReduce fp8 e5m2 | 1G | TREE SIMPLE | TREE LL128 | 207.80 | 345.24 | 10.01 ms | 6026.0 us | 1.66x | | 32 | AllReduce fp8 e5m2 | 2G | TREE SIMPLE | TREE LL128 | 232.81 | 360.74 | 17.87 ms | 11.53 ms | 1.55x | | 32 | AllReduce fp8 e5m2 | 4G | TREE SIMPLE | TREE LL128 | 259.72 | 374.16 | 32.04 ms | 22.24 ms | 1.44x | | 32 | AllReduce fp8 e5m2 | 8G | TREE SIMPLE | TREE LL128 | 297.87 | 371.65 | 55.87 ms | 44.78 ms | 1.25x | | 32 | AllReduce fp8 e5m2 | 16G | TREE SIMPLE | TREE LL128 | 314.33 | 366.42 | 105.89 ms | 90.84 ms | 1.17x | | 32 | ReduceScatter float | 32M | RING LL | DDA LL | 100.25 | 211.01 | 324.3 us | 154.0 us | 2.10x | | 32 | ReduceScatter float | 64M | RING LL128 | DDA SIMPLE | 172.43 | 206.96 | 377.0 us | 314.1 us | 1.20x | | 32 | ReduceScatter bfloat16 | 32M | RING LL | DDA LL | 100.16 | 205.80 | 324.5 us | 158.0 us | 2.05x | | 32 | ReduceScatter bfloat16 | 64M | RING LL128 | DDA SIMPLE | 173.85 | 206.04 | 373.9 us | 315.5 us | 1.19x | | 32 | ReduceScatter fp8 e4m3 | 256M | RING SIMPLE | RING LL128 | 281.21 | 313.50 | 924.8 us | 829.5 us | 1.11x | | 32 | AllGather float | 128M | CE-Scratch | DDA LL128 | 354.72 | 479.30 | 366.6 us | 271.3 us | 1.35x | | 32 | AllGather float | 256M | CE-Scratch | DDA LL128 | 427.18 | 505.60 | 608.8 us | 514.3 us | 1.18x | | 32 | AllGather float | 512M | CE-Scratch | RING LL128 | 413.10 | 514.83 | 1259.0 us | 1010.2 us | 1.25x | | 32 | AllGather float | 1G | CE-Scratch | RING LL128 | 460.09 | 644.34 | 2260.8 us | 1614.4 us | 1.40x | | 32 | AllGather float | 2G | CE-Scratch | RING LL128 | 511.44 | 659.29 | 4067.7 us | 3155.5 us | 1.29x | | 32 | AllToAll float | 2M | CE-Scratch | Direct SIMPLE | 23.12 | 47.45 | 87.9 us | 42.8 us | 2.05x | | 32 | AllToAll float | 4M | CE-Scratch | Direct SIMPLE | 45.53 | 97.06 | 89.3 us | 41.9 us | 2.13x | | 32 | AllToAll float | 8M | CE-Scratch | Direct SIMPLE | 77.92 | 169.89 | 104.3 us | 47.8 us | 2.18x | | 32 | AllToAll float | 16M | CE-Scratch | Direct SIMPLE | 125.27 | 261.94 | 129.8 us | 62.1 us | 2.09x | | 32 | AllToAll float | 32M | CE-Scratch | Direct SIMPLE | 185.93 | 348.38 | 174.8 us | 93.3 us | 1.87x | | 32 | AllToAll float | 64M | CE-Scratch | Direct SIMPLE | 96.47 | 415.50 | 675.2 us | 156.5 us | 4.31x | | 32 | AllToAll float | 128M | CE-Scratch | Direct SIMPLE | 79.98 | 476.53 | 1626.0 us | 273.0 us | 5.96x | | 32 | AllToAll float | 256M | CE-Scratch | Direct SIMPLE | 97.37 | 519.89 | 2670.9 us | 500.2 us | 5.34x | | 32 | AllToAll float | 512M | CE-Scratch | Direct SIMPLE | 412.82 | 548.05 | 1259.9 us | 949.0 us | 1.33x | | 32 | AllToAll float | 1G | CE-Scratch | Direct SIMPLE | 459.59 | 571.77 | 2263.3 us | 1819.2 us | 1.24x | | 32 | AllToAll float | 2G | CE-Scratch | Direct SIMPLE | 512.45 | 586.47 | 4059.7 us | 3547.3 us | 1.14x | | 32 | Gather float | 1K | CE-Scratch* | P2P kernel* | 0.01 | 0.05 | 83.8 us | 18.8 us | 5.00x | | 32 | Gather float | 2K | CE-Scratch* | P2P kernel* | 0.02 | 0.12 | 82.9 us | 16.5 us | 6.00x | | 32 | Gather float | 4K | CE-Scratch* | P2P kernel* | 0.05 | 0.24 | 82.8 us | 16.4 us | 4.90x | | 32 | Gather float | 8K | CE-Scratch* | P2P kernel* | 0.10 | 0.48 | 83.2 us | 16.5 us | 5.11x | | 32 | Gather float | 16K | CE-Scratch* | P2P kernel* | 0.19 | 0.99 | 83.2 us | 16.1 us | 5.21x | | 32 | Gather float | 32K | CE-Scratch* | P2P kernel* | 0.39 | 1.99 | 83.1 us | 16.0 us | 5.17x | | 32 | Gather float | 64K | CE-Scratch* | P2P kernel* | 0.76 | 3.84 | 83.2 us | 16.6 us | 5.05x | | 32 | Gather float | 128K | CE-Scratch* | P2P kernel* | 1.54 | 7.50 | 82.2 us | 17.0 us | 4.86x | | 32 | Gather float | 256K | CE-Scratch* | P2P kernel* | 3.08 | 11.61 | 82.6 us | 21.9 us | 3.78x | | 32 | Gather float | 512K | CE-Scratch* | P2P kernel* | 6.10 | 22.52 | 83.4 us | 22.6 us | 3.70x | | 32 | Gather float | 1M | CE-Scratch* | P2P kernel* | 12.27 | 46.03 | 82.8 us | 22.1 us | 3.75x | | 32 | Gather float | 2M | CE-Scratch* | P2P kernel* | 24.37 | 85.27 | 83.4 us | 23.8 us | 3.50x | | 32 | Gather float | 4M | CE-Scratch* | P2P kernel* | 49.12 | 159.07 | 82.7 us | 25.6 us | 3.24x | | 32 | Gather float | 8M | CE-Scratch* | P2P kernel* | 97.28 | 301.85 | 83.6 us | 27.0 us | 3.10x | | 32 | Gather float | 16M | CE-Scratch* | P2P kernel* | 182.50 | 526.06 | 89.1 us | 30.9 us | 2.88x | | 32 | Gather float | 32M | CE-Scratch* | P2P kernel* | 302.38 | 583.54 | 107.5 us | 55.7 us | 1.93x | | 32 | Gather float | 64M | CE-Scratch* | P2P kernel* | 236.81 | 688.03 | 274.5 us | 94.5 us | 2.91x | | 32 | Gather float | 128M | CE-Scratch* | P2P kernel* | 282.24 | 714.64 | 460.7 us | 182.0 us | 2.53x | | 32 | Gather float | 256M | CE-Scratch* | P2P kernel* | 345.45 | 712.82 | 752.8 us | 364.8 us | 2.06x | | 32 | Gather float | 512M | CE-Scratch* | P2P kernel* | 470.07 | 703.76 | 1106.4 us | 739.1 us | 1.50x | | 32 | Gather float | 1G | CE-Scratch* | P2P kernel* | 523.25 | 691.90 | 1988.1 us | 1503.4 us | 1.32x | | 32 | Gather float | 2G | CE-Scratch* | P2P kernel* | 564.75 | 686.84 | 3683.7 us | 3029.0 us | 1.22x | | 32 | Scatter float | 1K | CE-Scratch* | P2P kernel* | 0.01 | 0.05 | 77.3 us | 19.0 us | 5.00x | | 32 | Scatter float | 2K | CE-Scratch* | P2P kernel* | 0.03 | 0.11 | 76.1 us | 18.9 us | 3.50x | | 32 | Scatter float | 4K | CE-Scratch* | P2P kernel* | 0.05 | 0.24 | 75.3 us | 16.5 us | 4.80x | | 32 | Scatter float | 8K | CE-Scratch* | P2P kernel* | 0.11 | 0.49 | 75.5 us | 16.3 us | 4.67x | | 32 | Scatter float | 16K | CE-Scratch* | P2P kernel* | 0.21 | 0.96 | 75.9 us | 16.5 us | 4.60x | | 32 | Scatter float | 32K | CE-Scratch* | P2P kernel* | 0.41 | 1.85 | 75.9 us | 17.1 us | 4.47x | | 32 | Scatter float | 64K | CE-Scratch* | P2P kernel* | 0.83 | 3.75 | 75.9 us | 16.9 us | 4.49x | | 32 | Scatter float | 128K | CE-Scratch* | P2P kernel* | 1.66 | 7.93 | 76.5 us | 16.0 us | 4.78x | | 32 | Scatter float | 256K | CE-Scratch* | P2P kernel* | 3.34 | 11.82 | 75.9 us | 21.5 us | 3.53x | | 32 | Scatter float | 512K | CE-Scratch* | P2P kernel* | 6.67 | 22.99 | 76.2 us | 22.1 us | 3.45x | | 32 | Scatter float | 1M | CE-Scratch* | P2P kernel* | 13.25 | 44.12 | 76.7 us | 23.0 us | 3.33x | | 32 | Scatter float | 2M | CE-Scratch* | P2P kernel* | 25.45 | 80.80 | 79.9 us | 25.2 us | 3.18x | | 32 | Scatter float | 4M | CE-Scratch* | P2P kernel* | 48.08 | 145.10 | 84.5 us | 28.0 us | 3.02x | | 32 | Scatter float | 8M | CE-Scratch* | P2P kernel* | 86.78 | 246.00 | 93.6 us | 33.0 us | 2.83x | | 32 | Scatter float | 16M | CE-Scratch* | P2P kernel* | 147.85 | 377.00 | 109.9 us | 43.1 us | 2.55x | | 32 | Scatter float | 32M | CE-Scratch* | P2P kernel* | 233.13 | 462.54 | 139.4 us | 70.3 us | 1.98x | | 32 | Scatter float | 64M | CE-Scratch* | P2P kernel* | 313.88 | 542.26 | 207.1 us | 119.9 us | 1.73x | | 32 | Scatter float | 128M | CE-Scratch* | P2P kernel* | 469.61 | 611.63 | 276.9 us | 212.6 us | 1.30x | | 32 | Scatter float | 256M | CE-Scratch* | P2P kernel* | 580.57 | 647.18 | 447.9 us | 401.8 us | 1.11x | Gain = this PR busbw / develop busbw (equal to develop time / this PR time). busbw and time are rccl-tests out-of-place values, median of 2 runs per library on the same nodes. `DDA SIMPLE` = DDA VMM tier; `Direct SIMPLE` = direct AllToAll kernel. \* Gather and Scatter print no path in rccl-tests; the path shown is from the code (develop forces CE-Scratch on gfx1250, this PR does not). Sizes where develop and this PR take the same path are not listed. `DDA SIMPLE` = DDA VMM tier; `Direct SIMPLE` = direct AllToAll kernel. \* Gather and Scatter print no path in rccl-tests; the path shown is from the code (develop forces CE-Scratch on gfx1250, this PR does not). Sizes where develop and this PR take the same path are not listed. **Sizes below 0.95x of develop (64K and up):** | Collective | Ranks | Sizes | vs develop | Why it is not a regression | |---|---|---|---|---| | Broadcast | 16 | 64K–4M | 0.87–0.92x | Same path and code in both libraries; develop alone varies down to 0.89x run to run; clean at 32 ranks | | AllToAll | 8, 16, 32 | 64K–512K | 0.92–0.94x | Same path and block count; develop alone swings 18–24 GB/s at 256K | | AllGather | 32 | 64K–128K, 16M | 0.92–0.94x | Same path and block count; develop alone varies down to 0.87x | | AllReduce float / bf16 | 8, 16 | 128K–256K | 0.92–0.95x | Same path, which this PR does not change | | Scatter | 32 | 2G | 0.93x | Not changed by this PR; develop alone varies down to 0.94x | ### Not in this PR - AllToAll mid-size DDA / Ring LL wins (AICOMRCCL-2877, dropped). - Copy Engine wins that need registered buffers (not a default-user path). - fp8 AllReduce Ring above 8 ranks (blocked by the precision guard). - Broadcast (the sweep has no out-of-box baseline). - Small-size wins that come only from a different CU count. ## Submission checklist - [x] CHANGELOG updated - [x] No env-variable default changes (the env-var CI check passes) - [x] Host microtests and GPU unit tests added - [x] H13 8/16/32-rank validation table filled in --------- Co-authored-by: Praneeth Vallem <Praneeth.Vallem@amd.com> (cherry picked from commit f33e360)

…TUNING_PARTITION_MODE) (ROCm#13416) The gfx1250 arch table (`rcclArchThresholds`) was tuned in SPX mode. The DPX sweep disagrees with it in several places: crossovers that are right for SPX are wrong in the other direction for DPX, and the reverse. One table cannot serve both modes, so this PR adds a way to pick the table by compute-partition mode. This PR is plumbing only. The DPX table is an exact copy of the SPX table, so tuning is unchanged. The DPX values come in a follow-up PR. **New env var `RCCL_TUNING_PARTITION_MODE`** (`auto` default | `SPX` | `DPX`, case-insensitive) - `auto` follows the compute-partition mode the GPU reports (sysfs `current_compute_partition` on the physical function). DPX picks the DPX table; SPX, CPX or no reported mode picks SPX. - `SPX` / `DPX` force a table. - An unknown value is reported once at ATTN and treated as `auto`. - It only selects tuning thresholds. It has no effect on other archs or under `RCCL_IGNORE_ARCH_TABLE=1`. **Table selection** - `rcclGetArchThresholds(gcn, partition)` takes an `rcclTuningPartition`, defaulting to SPX, so existing callers are unchanged. - gfx1250 has a full second table, `rcclArchThresholds_gfx1250_dpx`. gfx942 and gfx950 return the same table for both modes. **Every rank uses the same table** - Ranks choose algorithms independently, so a comm where ranks read different tables could pick different algorithms for one call. - `fillInfo` records each rank's request in a new `ncclPeerInfo::tuningPartition`, appended at the tail. - After AllGather1, `initTransportsRank` re-picks `comm->archThresholds`: DPX only when every rank asks for DPX, otherwise SPX on every rank. - Rank 0 logs a mixed comm at ATTN and the chosen table at INFO (`gfx1250 tuning table: SPX|DPX`). - The sysfs read is hoisted out of the MLOPart probe. One read now serves both uses, and MLOPart behaviour is unchanged. **Host microtests (`rccl-UnitTestsMicroInit`)** - Per-rank request: GPU mode SPX/DPX/CPX/none, env override, `auto` and empty value, invalid value reported once (isolated test), and a function above the MLOPart bound. - Per-comm rule: DPX only when every rank asks for it. - The table installed in `comm->archThresholds`, on rank 0 and other ranks, for gfx1250 and other archs, and its logging. - The `initTransportsRank` call site, including a comm where only one peer asks for SPX. - The `rcclGetArchThresholds` host fake can now return a separate table per partition, so tests check which table the comm got rather than only which partition was asked for. - Mutation-gated on `init.cc`. **GPU unit test** - `RcclDdaTierThresholds.TuningPartitionTablePerArch` checks: - the gfx1250 DPX table is a separate object with the same contents as SPX; - explicit SPX matches the default; - gfx942/gfx950 return one table for both modes. **On H13 nodes** - SPX node: INFO reports `SPX`, and the paths chosen are identical to develop. - DPX node: INFO reports `DPX`, and the paths chosen are identical to develop (the tables are equal). - `RCCL_TUNING_PARTITION_MODE=DPX` on an SPX node and `SPX` on a DPX node force the table. - An invalid value produces one ATTN. - [x] Host microtests: all 13 binaries pass. - [x] Mutation gate on `init.cc` (23 mutants) - [x] `rccl_check_default_behavior.py`: no undocumented env-var changes. - [x] GPU unit tests on gfx1250 - [x] SPX and DPX node checks on H13 - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md - [x] CHANGELOG.md updated (Added) - [x] docs/api-reference/env-variables.rst updated --------- Co-authored-by: Praneeth Vallem <Praneeth.Vallem@amd.com> Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit f710804)

The extra(rccl) topo_expl build fails to link on develop since ROCm#13336: rcclRingTreeTable() in rccl_wrap.cc now calls rcclGetArchThresholds(), which topo_expl never compiled. Fix: copy rccl_arch_thresholds.cc into hipify_rccl/ and build it with the other RCCL sources so topo_expl uses the real per-arch tables. AICOMRCCL-1756 (cherry picked from commit e45bca4)

akankjai · 2026-10-19T12:54:46Z

@pvallem @ankith117 This is a draft cherry-pick of your merged RCCL PRs into release/bkc/therock-10.2-20261013 for BKC MI455 26.13 (Jira ROCM-33119, cherry-pick cut-off is today, 10/19). It contains #12984, #13336, #13416 and #13466, one commit each, original authorship preserved.

Could you please review and approve? Two things to check:

In [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416's commit, CHANGELOG.md and MICROTEST_README.md had context conflicts with develop commits that are not on this branch. I resolved each to the same single-line addition your PR makes.
feat(rccl): embed CSV tuner configs in librccl.so #13038 (embedded CSV tuner configs) is not on this branch, so I left it out. [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416's code has no dependency on it. Please tell me if you disagree.

Also, should #13706 (follow-up to #13466) go in with this?

amd-argus · 2026-10-19T13:03:57Z

@cppccppccppc @sunguobao PTAL

pvallem · 2026-10-19T17:41:17Z

Validated all 4 PR's looks good.

akankjai · 2026-10-19T17:50:13Z

@ankith117 Could you please review and confirm the cherry-picks of your PRs #13336 and #13416 into release/bkc/therock-10.2-20261013? @pvallem has already approved. The cherry-pick cut-off for BKC MI455 26.13 (ROCM-33119) is today, 10/19. Two quick things to check:

In [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416's commit, CHANGELOG.md and MICROTEST_README.md had context conflicts with develop. I resolved each to the same single-line addition your PR makes.
feat(rccl): embed CSV tuner configs in librccl.so #13038 (embedded CSV tuner configs) is not on this branch, so I left it out. [RCCL] Pick the gfx1250 tuning table by compute-partition mode (RCCL_TUNING_PARTITION_MODE) #13416's code has no dependency on it. Please tell me if you disagree.

@cppccppccppc @sunguobao Could you please review this PR when you get a chance? It is a release-branch cherry-pick of already-merged RCCL changes and is blocked on required review. The PR will move out of draft once @ankith117 confirms.

@pvallem @ankith117 Should #13706 (the open follow-up to #13466) be included in this PR as well? It is currently not included.

pvallem and others added 4 commits October 19, 2026 07:42

github-actions Bot added documentation Improvements or additions to documentation project: rccl labels Oct 19, 2026

systems-assistant Bot added the external contribution label Oct 19, 2026

pvallem approved these changes Oct 19, 2026

View reviewed changes

sdarisala approved these changes Oct 19, 2026

View reviewed changes

astyrrian1 marked this pull request as ready for review October 19, 2026 18:29

astyrrian1 requested review from a team October 19, 2026 18:29

astyrrian1 requested review from a team as code owners October 19, 2026 18:29

astyrrian1 approved these changes Oct 19, 2026

View reviewed changes

ScottTodd approved these changes Oct 19, 2026

View reviewed changes

astyrrian1 merged commit f933973 into ROCm:release/bkc/therock-10.2-20261013 Oct 19, 2026
32 of 39 checks passed

Navigation Menu

Repository navigation

[RCCL] Cherry-pick gfx1250 tuning and AlltoAllv fixes (#12984, #13336, #13416, #13466) [bkc 20261013] - #13736

[RCCL] Cherry-pick gfx1250 tuning and AlltoAllv fixes (#12984, #13336, #13416, #13466) [bkc 20261013]#13736
astyrrian1 merged 4 commits into
ROCm:release/bkc/therock-10.2-20261013from
akankjai:users/akankjai/bkc-20261013-rccl-cherry-picks-ROCM-33119

akankjai commented Oct 19, 2026

Uh oh!

akankjai commented Oct 19, 2026

Uh oh!

amd-argus Bot commented Oct 19, 2026

Uh oh!

pvallem commented Oct 19, 2026

Uh oh!

akankjai commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

akankjai commented Oct 19, 2026

Notes

Uh oh!

akankjai commented Oct 19, 2026

Uh oh!

amd-argus Bot commented Oct 19, 2026

Uh oh!

pvallem commented Oct 19, 2026

Uh oh!

akankjai commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants