[aduforest]

Motivation

Improve AllReduce performance on MI350X, 2 nodes × 8 GPUs (16 ranks). The tested float cases take 5.9%-30.7% less time with Tree/LL128.

Technical Details

Adds this row to projects/rccl/tuner/rccl_tuner_gfx950.csv:

allreduce,33554432,150994943,tree,ll128,64,2,16,-1,-1

Uses Tree/LL128 with 64 channels for 32 MiB <= size < 144 MiB, only on two nodes with 16 ranks. It goes before the existing rows so it takes priority. Exactly 144 MiB and settings outside this scope are unchanged. The range follows #14722 without overlapping it.

Test Plan

Verified float logs from job 115389: six rounds around the changed range, three wider 1-512 MiB rounds, and 12 execution-mode combinations. Existing and new settings were compared in the same allocation, with order rotated and repeated existing-settings runs as controls. Both in-place and out-of-place operations were checked.

Test Result

• All 336 runs succeeded; all 6,936 result rows reported #wrong=0.
• All 288 tested float size/mode/sweep cases inside the range improved: 1.06x-1.44x speedup.
• Still unresolved: a 7.8% slowdown near 18 MiB, outside the rule, versus 0.6% variation in its matching control. This is not yet verified as regression-free.
• Separate bfloat16/half results are summary-only; their raw logs were not checked here.

Measurements and reproduction details

Median out-of-place float times, DEFAULT policy, registration 0, no graph:

Bytes	Before (us)	After (us)
33554432	380.65	270.91
48234480	396.46	372.98
62914528	595.44	465.49
77594576	687.65	562.27
92274624	764.99	652.38
106954672	861.02	743.91
121634720	1089.91	833.45
136314768	1104.94	921.53
150994816	1133.41	1011.18

The last sample is 128 bytes below 144 MiB. The baseline uses Ring/LL128 below 52 MiB, Tree/Simple below 112 MiB, then Ring/Simple. The new settings are Tree/LL128/64 throughout this interval.

The 12 tested modes span DEFAULT/EFFICIENCY/ZERO policies, registration modes and graph/non-graph runs—not every possible combination. The 18 MiB slowdown occurred with EFFICIENCY/R1/graph; another 18 MiB control varied by 9.2%.

The separate datatype summary reports 1.04x-1.8x speedups, except bfloat16 near 47 MiB took 0.7% longer. Range-finding showed slowdowns from 168 MiB; this rule stops before 144 MiB.

Command template; select one value per placeholder and add -G 1 for graphs:

mpirun -np 16 --map-by ppr:8:node -x NCCL_TUNER_CONFIG_FILE=<baseline|candidate>.csv \
  -x NCCL_CTA_POLICY=<DEFAULT|EFFICIENCY|ZERO> \
  all_reduce_perf -b 18874384 -e 165674864 -i 14680048 \
  -w 20 -n 100 -c 1 -A 1 -d float -R <0|1|2> [-G 1]

For the wider sweep, replace the size arguments with -b 1M -e 512M -f 2.

Both CSVs used RCCL 2.32.3 (091930df78f8d9402e53034fda2adbd53e3e3aec), ROCm 10.1.0 and Open MPI 5.0.10. The baseline excludes #14722. These are same-build CSV comparisons, not PR/base builds. Full logs are retained, not attached.

Issue Tracking

JIRA ID: AICOMRCCL-3532

Submission Checklist

• Reviewed the contributing guidelines.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[aduforest]

@argus review

[amd-argus]

ARGUS

No findings. The panel reviewed the changed files it was given and flagged nothing above Nit
level.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[aduforest]

Folded into #14722, which now contains the combined two-node tuning and boundary/fallback tests. The original benchmark evidence and limitations are preserved and linked there. Closing this PR as superseded; no changes were merged into develop.

[github-actions]

pr-rocm-systems-14969-oci-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status	Build time
pr-14969_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14969_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14969_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14969_rccl-perf_rocm-10.1-ga	rccl-pr-14969	468a39a	468a39a	cached
pr-14969_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	b203ee0	cached
pr-14969_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-14969_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	cached
pr-14969_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14969_rccl-perf_rocm-10.2-latest	rccl-pr-14969	468a39a	468a39a	cached
pr-14969_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	b203ee0	cached
pr-14969_rccl-smoke_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14969_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14969_rccl-smoke_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14969_rccl-smoke_rocm-10.1-ga	rccl-pr-14969-smoke	468a39a	468a39a	cached
pr-14969_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14969	468a39a	468a39a	cached
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop-2b3c00291b7d	2b3c002	2b3c002	cached
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	b203ee0	cached

Failures

No failing tests in this run.

15 tests skipped.

[github-actions]

pr-rocm-systems-14969-ruby64-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status	Build time
pr-14969_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14969_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14969_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14969_rccl-perf_rocm-10.1-ga	rccl-pr-14969	468a39a	468a39a	cached
pr-14969_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	b203ee0	cached
pr-14969_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-14969_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	cached
pr-14969_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14969_rccl-perf_rocm-10.2-latest	rccl-pr-14969	468a39a	468a39a	cached
pr-14969_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	b203ee0	cached
pr-14969_rccl-smoke_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14969_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14969_rccl-smoke_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14969_rccl-smoke_rocm-10.1-ga	rccl-pr-14969-smoke	468a39a	468a39a	cached
pr-14969_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14969	468a39a	468a39a	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop-2b3c00291b7d	2b3c002	2b3c002	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	b203ee0	cached

Failures

No failing tests in this run.

15 tests skipped.