perf(rccl): gfx950 tuner rows for 1-node Broadcast and Reduce by aduforest · Pull Request #14484 · ROCm/rocm-systems

aduforest · 2026-10-28T18:27:53Z

Motivation

Improve Broadcast and Reduce performance on MI350X, 1 node × 8 GPUs (8 ranks). Ring/LL outperforms the existing Ring/Simple settings at the tested sizes below 8 MiB.

Technical Details

Adds four rows to projects/rccl/tuner/rccl_tuner_gfx950.csv, only for one node with eight ranks:

Operations	Size range	New settings
Broadcast and Reduce	`512 KiB <= size < 2 MiB`	Ring/LL, 112 channels
Broadcast and Reduce	`2 MiB <= size < 8 MiB`	Ring/LL, default channel count

Exactly 2 MiB uses the second rule; exactly 8 MiB is excluded. Settings outside this scope are unchanged. Two-node tuning is excluded because Ring/LL was slower there.

Test Plan

Verified rccl-tests logs: repeated baseline/candidate/control comparisons, dense size sweeps and a wider 128 KiB–32 MiB sweep. The current-build check covers float/bfloat16/half with DEFAULT policy, registration 0 and no graphs. Earlier float tests cover 12 CTA/registration/graph combinations.

Test Result

Current-build check: 236 successful runs; all 3,352 result rows have #wrong=0.
Verified speedups: 1.37x-3.26x for Broadcast and 1.33x-3.07x for Reduce. Outside-range median differences stay below 0.7%.
Earlier 12-mode float tests: 1,104 successful runs, 16,032 rows with #wrong=0; outside-range differences below 0.9%.

Measurements and command template

Operation	Size	Before (us)	After (us)
Broadcast	512 KiB	40.17	16.86
Broadcast	1 MiB	40.99	19.72
Broadcast	2 MiB	61.08	25.48
Broadcast	4 MiB	67.93	37.34
Reduce	512 KiB	37.58	17.54
Reduce	1 MiB	39.15	20.50
Reduce	2 MiB	57.07	26.39
Reduce	4 MiB	67.69	38.79

Before: Ring/Simple with 32, 64, 112 and 112 channels respectively. After: Ring/LL with 112 channels. At 2 and 4 MiB, 112 is RCCL's default rather than a fixed value in the new rows (-1). Inclusive CSV byte ranges: 524288..2097151 and 2097152..8388607.

Current-build validation: job 103314, exact four-row candidate, RCCL 2.32.3 (091930df78f8d9402e53034fda2adbd53e3e3aec), ROCm 10.1.0 and Open MPI 5.0.10. Six dense rounds and three wider float rounds; datatype checks use DEFAULT/R0/no graph. Earlier jobs 100018 and 102856 use an older RCCL build and six lookup-equivalent rows; they are not a current-build 12-mode matrix.

The tables are compared using the same RCCL binary. Raw logs were audited and are retained, not attached. rccl-tests reports 2.19.6-HEAD:7ee494a5d4.

Eight ranks, one per GPU. Select one value per placeholder and add -G 1 for graphs:

mpirun -np 8 -x NCCL_TUNER_CONFIG_FILE=<baseline|candidate>.csv \
  -x NCCL_CTA_POLICY=<DEFAULT|EFFICIENCY|ZERO> \
  <broadcast_perf|reduce_perf> -b 131088 -e 2490288 -i 393200 \
  -w 20 -n 100 -c 1 -A 1 -d <float|bfloat16|half> -R <0|1|2> [-G 1]

Repeat with size arguments -b 524304 -e 9961392 -i 1572848, then -b 128K -e 32M -f 2. These are templates, not a complete run transcript.

Issue Tracking

JIRA ID: AICOMRCCL-3447

Submission Checklist

Reviewed the contributing guidelines.

Six rows for 1 node x 8 GPUs, Broadcast and Reduce: Ring/LL with 112 channels for [512K, 2M) and Ring/LL with RCCL's channel choice for [2M, 8M). The previous choice there, Ring/Simple, runs 1.5x to 3x slower. All other rows are unchanged. Co-Authored-By: Claude <noreply@anthropic.com>

therock-pr-bot · 2026-10-28T18:29:15Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

aduforest · 2026-10-28T19:40:35Z

@argus review

amd-argus

ARGUS

Findings

2 more finding(s), not posted inline

projects/rccl/tuner/rccl_tuner_gfx950.csv:9 (Nit) Broadcast rows 8-9 (and reduce rows 11-12) have identical ring/ll/112 settings over contiguous ranges; each pair can be one 524288-2097151 row.
projects/rccl/tuner/rccl_tuner_gfx950.csv:13 (Low) Follow-up: no csv-tuner test asserts every shipped row parses, so a misspelled row would be silently skipped (csv_tuner.cc:355-362); assert numConfigs == countConfigLines() in EmbeddedAndFileParseIdentically.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Broadcast and Reduce [512K, 1M) and [1M, 2M) had the same Ring/LL/112 choice; one row each covers [512K, 2M). Lookups are unchanged at every size. Co-Authored-By: Claude <noreply@anthropic.com>

Tuner-table-only changes don't carry a CHANGELOG entry. Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

github-actions · 2026-10-29T03:44:58Z

pr-rocm-systems-14484-oci-rccl-pr-build

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status
pr-14484_rccl-backtrace_rocm-10.1-ga	rccl-pr-14484-backtrace	`dd1f320`	`dd1f320`	build failed (exit 1)
pr-14484_rccl-perf_rocm-10.1-ga	rccl-pr-14484	`dd1f320`	`dd1f320`	build failed (exit 1)
pr-14484_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`da7f4d8`	build failed (exit 1)
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`da7f4d8`	build failed (exit 1)

Failures

No failing tests in this run.

116 tests skipped.

github-actions · 2026-10-29T03:44:59Z

pr-rocm-systems-14484-ruby64-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status
pr-14484_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14484_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14484_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14484_rccl-perf_rocm-10.1-ga	rccl-pr-14484	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	`7db73fb`	cached
pr-14484_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-14484_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-14484_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14484_rccl-perf_rocm-10.2-latest	rccl-pr-14484	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	`7db73fb`	cached
pr-14484_rccl-smoke_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14484_rccl-smoke_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rccl-pr-14484-smoke	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14484	`dd1f320`	`dd1f320`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`7db73fb`	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	`7db73fb`	cached

Failures

No failing tests in this run.

15 tests skipped.

PedramAlizadeh

LGTM!
Thank you @aduforest!

github-actions · 2026-11-03T19:29:46Z

pr-rocm-systems-14484-oci-rccl-pr-build

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status
pr-14484_rccl-backtrace_rocm-10.1-ga	rccl-pr-14484-backtrace	`dd1f320`	`dd1f320`	build failed (exit 1)
pr-14484_rccl-perf_rocm-10.1-ga	rccl-pr-14484	`dd1f320`	`dd1f320`	build failed (exit 1)
pr-14484_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`da7f4d8`	build failed (exit 1)
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`da7f4d8`	build failed (exit 1)

Failures

No failing tests in this run.

116 tests skipped.

github-actions · 2026-11-03T19:29:47Z

pr-rocm-systems-14484-oci-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status
pr-14484_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14484_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14484_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14484_rccl-perf_rocm-10.1-ga	rccl-pr-14484	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	`1e8c6de`	cached
pr-14484_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-14484_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	cached
pr-14484_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14484_rccl-perf_rocm-10.2-latest	rccl-pr-14484	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`1e8c6de`	cached
pr-14484_rccl-smoke_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14484_rccl-smoke_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rccl-pr-14484-smoke	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14484	`dd1f320`	`dd1f320`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`1e8c6de`	cached
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`1e8c6de`	cached

Failures

No failing tests in this run.

15 tests skipped.

github-actions · 2026-11-03T19:29:47Z

pr-rocm-systems-14484-ruby64-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status
pr-14484_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14484_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14484_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14484_rccl-perf_rocm-10.1-ga	rccl-pr-14484	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	`7db73fb`	cached
pr-14484_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-14484_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc0	10.2.0rc0	cached
pr-14484_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14484_rccl-perf_rocm-10.2-latest	rccl-pr-14484	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	`7db73fb`	cached
pr-14484_rccl-smoke_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14484_rccl-smoke_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rccl-pr-14484-smoke	`dd1f320`	`dd1f320`	cached
pr-14484_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14484	`dd1f320`	`dd1f320`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop	develop	`7db73fb`	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	`7db73fb`	cached

Failures

No failing tests in this run.

15 tests skipped.

aduforest · 2026-11-03T23:00:51Z

Validation follow-up for dd1f320:

Audited 1,340 successful benchmark launches and 19,384 result rows with #wrong=0 across the runs now listed in the description.
The exact four-row candidate was checked on RCCL 2.32.3 with float/bfloat16/half. Earlier 12-mode float runs use an older build and lookup-equivalent rows; that distinction is retained.
Range boundaries, default-channel handling and unrelated topology/collective selections check out. No source changes were needed.
All three required CI gates currently pass. The separate Math CI failure is the test-runner GPU visibility problem addressed by test(rccl): clamp isolated-test GPU pool to devices HIP can open #14944; the policy-bot failure is tracked by therock-pr-bot fails on every develop PR: required 'pre-commit' check is never published (pre-formatting.yml is gated to testbranch) #11156.

The description is shorter, with reproduction details collapsed. Required lead/team approval is still outstanding; this comment does not replace it.

aduforest · 2026-11-04T04:31:15Z

Final compatibility check with #14722 passed at the current PR heads: the seven new tuning rules do not overlap, and 1,139,712 CSV selection-model comparisons preserve each PR’s selections and the existing fallback behavior. This is a selection-model check, not an additional GPU benchmark. Details: #14722 (comment)

The existing benchmark evidence and this cross-check support merging once the remaining review requirements are satisfied; no further broad benchmark campaign is indicated. All three required CI checks on this PR are green. GitHub still reports it blocked under review requirements.

github-actions Bot added the project: rccl label Oct 28, 2026

systems-assistant Bot added the organization: ROCm label Oct 28, 2026

amd-argus Bot reviewed Oct 28, 2026

View reviewed changes

aduforest and others added 2 commits October 28, 2026 13:50

perf(rccl): merge the identical 112-channel rows

f17e277

Broadcast and Reduce [512K, 1M) and [1M, 2M) had the same Ring/LL/112 choice; one row each covers [512K, 2M). Lookups are unchanged at every size. Co-Authored-By: Claude <noreply@anthropic.com>

perf(rccl): drop the CHANGELOG line

dd1f320

Tuner-table-only changes don't carry a CHANGELOG entry. Co-Authored-By: Claude <noreply@anthropic.com>

aduforest force-pushed the users/aduforest/intellitune-gfx950-bcast-reduce-rows branch from 1b388ba to dd1f320 Compare October 28, 2026 22:42

aduforest marked this pull request as ready for review October 28, 2026 22:46

aduforest requested review from a team, PedramAlizadeh and mustafabar October 28, 2026 22:46

amd-argus Bot reviewed Oct 28, 2026

View reviewed changes

mustafabar approved these changes Oct 29, 2026

View reviewed changes

PedramAlizadeh approved these changes Nov 2, 2026

View reviewed changes

aduforest requested review from alex-breslow-amd and corey-derochie-amd November 3, 2026 18:55

aduforest mentioned this pull request Nov 3, 2026

perf(rccl): tune gfx950 2-node AllReduce, AllGather and ReduceScatter #14722

Open

1 task done

Navigation Menu

Repository navigation

perf(rccl): gfx950 tuner rows for 1-node Broadcast and Reduce - #14484

perf(rccl): gfx950 tuner rows for 1-node Broadcast and Reduce#14484
aduforest wants to merge 3 commits into
developfrom
users/aduforest/intellitune-gfx950-bcast-reduce-rows

aduforest commented Oct 28, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Oct 28, 2026

Uh oh!

aduforest commented Oct 28, 2026

Uh oh!

amd-argus Bot left a comment

Uh oh!

amd-argus Bot left a comment

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

PedramAlizadeh left a comment

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

aduforest commented Nov 3, 2026

Uh oh!

aduforest commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

aduforest commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Issue Tracking

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Oct 28, 2026

✅ All Policy Checks Passed

Uh oh!

aduforest commented Oct 28, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Findings

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 29, 2026

pr-rocm-systems-14484-oci-rccl-pr-build

Builds

Failures

Uh oh!

github-actions Bot commented Oct 29, 2026

pr-rocm-systems-14484-ruby64-rccl-pr-smoke

Builds

Failures

Uh oh!

PedramAlizadeh left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 3, 2026

pr-rocm-systems-14484-oci-rccl-pr-build

Builds

Failures

Uh oh!

github-actions Bot commented Nov 3, 2026

pr-rocm-systems-14484-oci-rccl-pr-smoke

Builds

Failures

Uh oh!

github-actions Bot commented Nov 3, 2026

pr-rocm-systems-14484-ruby64-rccl-pr-smoke

Builds

Failures

Uh oh!

aduforest commented Nov 3, 2026

Uh oh!

aduforest commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aduforest commented Oct 28, 2026 •

edited

Loading