[aduforest]

Motivation

Improve AllGather performance on MI350X, 2 nodes × 8 GPUs (16 ranks). Using 48 instead of 64 channels reduces time by 6.8%-13.5% in the samples below.

Technical Details

Adds this row to projects/rccl/tuner/rccl_tuner_gfx950.csv:

allgather,150994944,234881023,ring,simple,48,2,16,-1,-1

Uses Ring/Simple with 48 channels for 144 MiB <= size < 224 MiB, only on two nodes with 16 ranks. Exactly 224 MiB and settings outside this scope are unchanged.

Test Plan

Reported rccl-tests coverage: six rounds comparing existing/new settings with repeated controls, sizes around the changed range and a wider 1-512 MiB sweep. Coverage includes float/bfloat16/half, CTA policies, registration modes and graph/non-graph runs; it does not establish every combination was tested. Correctness checking was enabled (-c 1).

Test Result

• Reported speedup across tested modes: 1.05x-1.19x inside the range.
• Separate bfloat16/half summaries report 6%-14% lower times.
• Raw correctness counts, exit statuses and exact software versions are not attached. Unchanged settings outside the range do not prove unchanged performance.

Measurements and command template

Representative timings, Ring/Simple with 64 channels before and 48 after:

Bytes	Before (us)	After (us)
150994944	679.26	587.32
171966208	699.24	610.53
192937728	715.27	638.62
213909248	728.72	663.01
234880768	744.50	693.66

The last sample is 256 bytes below 224 MiB.

RCCL develop, 16 ranks (one per GPU). Select one value per placeholder and add -G 1 for graphs:

mpirun -np 16 --map-by ppr:8:node -x NCCL_TUNER_CONFIG_FILE=<baseline|candidate>.csv \
  -x NCCL_CTA_POLICY=<DEFAULT|EFFICIENCY|ZERO> \
  all_gather_perf -b 130023440 -e 255852464 -i 20971504 \
  -w 20 -n 100 -c 1 -A 1 -d <float|bfloat16|half> -R <0|1|2> [-G 1]

For the wider sweep, replace the size arguments with -b 1M -e 512M -f 2. These are templates, not a complete run transcript.

Previously investigated shared CI issues: GPU visibility in the test runner (#14944) and the policy bot's missing required check (#11156).

Issue Tracking

JIRA ID: AICOMRCCL-3447

Submission Checklist

• Reviewed the contributing guidelines.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[amd-argus]

ARGUS

No findings. The panel reviewed the changed files it was given and flagged nothing above Nit
level.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[github-actions]

pr-rocm-systems-14858-oci-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status	Build time
pr-14858_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14858_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14858_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14858_rccl-perf_rocm-10.1-ga	rccl-pr-14858	f74f44c	f74f44c	cached
pr-14858_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	3e8a0e6	cached
pr-14858_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-14858_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	cached
pr-14858_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14858_rccl-perf_rocm-10.2-latest	rccl-pr-14858	f74f44c	f74f44c	cached
pr-14858_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	3e8a0e6	cached
pr-14858_rccl-smoke_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14858_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14858_rccl-smoke_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14858_rccl-smoke_rocm-10.1-ga	rccl-pr-14858-smoke	f74f44c	f74f44c	cached
pr-14858_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14858	f74f44c	f74f44c	cached
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop-760177639d11	7601776	7601776	cached
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	3e8a0e6	cached

Failures

No failing tests in this run.

15 tests skipped.

[github-actions]

pr-rocm-systems-14858-ruby64-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status	Build time
pr-14858_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14858_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14858_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14858_rccl-perf_rocm-10.1-ga	rccl-pr-14858	f74f44c	f74f44c	cached
pr-14858_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	2624718	cached
pr-14858_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-14858_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	cached
pr-14858_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14858_rccl-perf_rocm-10.2-latest	rccl-pr-14858	f74f44c	f74f44c	cached
pr-14858_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	2624718	cached
pr-14858_rccl-smoke_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14858_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14858_rccl-smoke_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14858_rccl-smoke_rocm-10.1-ga	rccl-pr-14858-smoke	f74f44c	f74f44c	cached
pr-14858_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14858	f74f44c	f74f44c	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop-760177639d11	7601776	7601776	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	2624718	cached

Failures

No failing tests in this run.

15 tests skipped.

[aduforest]

Folded into #14722, which now contains the combined two-node tuning and boundary/fallback tests. The original benchmark evidence and limitations are preserved and linked there. Closing this PR as superseded; no changes were merged into develop.