perf(rccl): gfx950 tuner row for 2-node ReduceScatter 136M-168M by aduforest · Pull Request #14898 · ROCm/rocm-systems

aduforest · 2026-11-03T08:15:19Z

Motivation

Improve ReduceScatter performance on MI350X, 2 nodes × 8 GPUs (16 ranks). Using 32 instead of 64 channels reduces median time by 6.0%-16.9% in the verified float cases.

Technical Details

Adds this row to projects/rccl/tuner/rccl_tuner_gfx950.csv:

reducescatter,142606336,176160767,ring,simple,32,2,16,-1,-1

Uses Ring/Simple with 32 channels for 136 MiB <= size < 168 MiB, only on two nodes with 16 ranks. Exactly 168 MiB and settings outside this scope are unchanged. Smaller sizes are excluded because the benefit was inconsistent.

Test Plan

Verified float logs from job 113968: six rounds around the changed range, three wider 1-512 MiB rounds, and 12 execution-mode combinations. Existing/new settings and repeated controls ran in the same allocation, with order rotated. Both in-place and out-of-place operations were checked.

A separate 88-run follow-up repeated questionable controls and checked both rule boundaries.

Test Result

Original run: 336 successful runs, 5,208 rows with #wrong=0, and 1.06x-1.20x speedup across all 120 tested float cases inside the range.
Follow-up: 88 successful runs, 1,536 rows with #wrong=0; all 24 mode/size boundary checks passed.
Still unresolved: a 4.4% slowdown at 2 MiB outside the rule, versus 0.5% variation in its matching control. Larger earlier differences did not repeat, but this is not yet verified as regression-free.
Separate bfloat16/half summaries report 3%-14% lower times; their raw logs were not checked here.

Measurements and reproduction details

Median out-of-place float times, DEFAULT policy, registration 0, no graph:

Bytes	Before (us)	After (us)
142606336	680.09	572.44
150994688	682.06	595.20
159383296	693.05	618.04
167771904	692.92	637.45
176160512	706.43	658.65

The last sample is 256 bytes below 168 MiB. These samples use Ring/Simple, with 64 channels before and 32 after.

The 12 tested modes span DEFAULT/EFFICIENCY/ZERO policies, registration modes and graph/non-graph runs—not every possible combination.

Follow-up job 115790 used another two-node allocation, six rounds and all six orders of existing/repeated-existing/new settings. It tested EFFICIENCY/R2 without graphs, ZERO/R2 without graphs, and EFFICIENCY/R1 and R2 with graphs. Initial/final hardware checks passed. Reported settings matched at every control size. Probes at 256 bytes below, at and above both boundaries confirmed the rule's scope.

Original outside-range and repeated-control differences reached 11.3%; some 8%-9% differences had stable matching controls. Follow-up control variation peaked at 2.1%. The remaining 4.4% slowdown occurred at 2 MiB with EFFICIENCY/R2 without graphs.

Command template; select one value per placeholder and add -G 1 for graphs:

mpirun -np 16 --map-by ppr:8:node -x NCCL_TUNER_CONFIG_FILE=<baseline|candidate>.csv \
  -x NCCL_CTA_POLICY=<DEFAULT|EFFICIENCY|ZERO> \
  reduce_scatter_perf -b 134217744 -e 184549296 -i 8388592 \
  -w 20 -n 100 -c 1 -A 1 -d float -R <0|1|2> [-G 1]

For the wider sweep, replace the size arguments with -b 1M -e 512M -f 2.

Both CSVs used RCCL 2.32.3 (091930df78f8d9402e53034fda2adbd53e3e3aec), ROCm 10.1.0 and Open MPI 5.0.10. The rccl-tests binary reports 2.19.6-HEAD:7ee494a5d4, SHA-256 cafd89d0ba72aedbf84e49bc233b990d94931dfd555f661295297363247dcc91. These are same-build CSV comparisons, not PR/base builds. Full logs are retained, not attached.

Issue Tracking

JIRA ID: AICOMRCCL-3447

Submission Checklist

Reviewed the contributing guidelines.

One row for 2 nodes x 16 ranks, ReduceScatter [136M, 168M): Ring/Simple with 32 channels instead of the default 64. It runs 7% to 19% faster there in float, bfloat16 and half. All other rows are unchanged. Co-Authored-By: Claude <noreply@anthropic.com>

therock-pr-bot · 2026-11-03T08:16:01Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

aduforest · 2026-11-03T21:29:20Z

@argus review

amd-argus

ARGUS

Findings

1 more finding(s), not posted inline

projects/rccl/tuner/rccl_tuner_gfx950.csv:8 (Low) No test probes the 2-node/16-rank reducescatter row (AppliesEmbeddedConfigToCostTable uses 1 node/8 ranks); a follow-up asserting non-comment row count == numConfigs per embedded CSV in csv-tuner-test.cc would catch a dropped row.

Bottom line: Looks approval ready.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Fold #14969, #14858 and #14898 into #14722. Coalesce the adjacent AllReduce ranges while preserving unavailable-protocol fallbacks, and cover embedded row counts, size boundaries and topology selection.

aduforest · 2026-11-03T23:00:27Z

Folded into #14722, which now contains the combined two-node tuning and boundary/fallback tests. The original benchmark evidence and limitations are preserved and linked there. Closing this PR as superseded; no changes were merged into develop.

github-actions · 2026-11-03T23:23:18Z

pr-rocm-systems-14898-oci-rccl-pr-build

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status	Build time
pr-14898_rccl-backtrace_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-backtrace_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14898_rccl-backtrace_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-backtrace_rocm-10.1-ga	rccl-pr-14898-backtrace	`c1acb47`	`c1acb47`	built	134s
pr-14898_rccl-backtrace_rocm-10.1-ga	rccl_gtest-pr-14898	`c1acb47`	`c1acb47`	built
pr-14898_rccl-debug_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-debug_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14898_rccl-debug_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-debug_rocm-10.1-ga	rccl-pr-14898-debug	`c1acb47`	`c1acb47`	built	120s
pr-14898_rccl-debug_rocm-10.1-ga	rccl-tests-develop-perf	develop	`b203ee0`	built	11s
pr-14898_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	built	72s
pr-14898_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-perf_rocm-10.1-ga	rccl-pr-14898	`c1acb47`	`c1acb47`	built	87s
pr-14898_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	`b203ee0`	built	13s
pr-14898_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	built	78s
pr-14898_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-perf_rocm-10.2-latest	rccl-pr-14898	`c1acb47`	`c1acb47`	built	87s
pr-14898_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`b203ee0`	built	12s
pr-14898_rccl-smoke_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14898_rccl-smoke_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rccl-pr-14898-smoke	`c1acb47`	`c1acb47`	built	137s
pr-14898_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14898	`c1acb47`	`c1acb47`	built
pr-14898_rccl_rocm-7.0.2	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl_rocm-7.0.2	rocm-7.0.2	7.0.2	7.0.2	cached
pr-14898_rccl_rocm-7.0.2	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl_rocm-7.0.2	rccl-pr-14898	`c1acb47`	`c1acb47`	built	145s
pr-14898_rccl_rocm-7.0.2	rccl_gtest-pr-14898	`c1acb47`	`c1acb47`	built
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop-2a45136e12c5	`2a45136`	`2a45136`	built	88s
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`b203ee0`	built	11s

Failures

No failing tests in this run.

189 tests skipped.

pr-rocm-systems-14898-oci-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status
pr-14898_rccl-perf_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14898_rccl-perf_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-perf_rocm-10.1-ga	rccl-pr-14898	`c1acb47`	`c1acb47`	cached
pr-14898_rccl-perf_rocm-10.1-ga	rccl-tests-develop-perf	develop	`b203ee0`	cached
pr-14898_rccl-perf_rocm-10.2-latest	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	cached
pr-14898_rccl-perf_rocm-10.2-latest	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-perf_rocm-10.2-latest	rccl-pr-14898	`c1acb47`	`c1acb47`	cached
pr-14898_rccl-perf_rocm-10.2-latest	rccl-tests-develop-perf	develop	`b203ee0`	cached
pr-14898_rccl-smoke_rocm-10.1-ga	gcc-13.2	13.2.0	13.2.0	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14898_rccl-smoke_rocm-10.1-ga	openmpi-5.0.8	5.0.8	5.0.8	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rccl-pr-14898-smoke	`c1acb47`	`c1acb47`	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14898	`c1acb47`	`c1acb47`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-13.2	13.2.0	13.2.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-5.0.8	5.0.8	5.0.8	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop-2a45136e12c5	`2a45136`	`2a45136`	cached
rocm-10.1-ga_rccl-develop-perf	rccl-tests-develop-perf	develop	`b203ee0`	cached

Failures

Stack	Type	Subtype	Tag	Test	Topology	Case	Status	Message
rocm-10.1-ga_rccl-develop-perf	rccl_test		regression-bf16-sum	all_reduce_perf	4n_8ppn_amd-rccl		Timeout	Timed out (exit code 124)

15 tests skipped.

github-actions · 2026-11-03T23:23:18Z

pr-rocm-systems-14898-ruby64-rccl-pr-smoke

View this job on CoCo

Builds

Stack	Component	Version	Commit	Status
pr-14898_rccl-perf_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14898_rccl-perf_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14898_rccl-perf_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14898_rccl-perf_rocm-10.1-ga	rccl-pr-14898	`c1acb47`	`c1acb47`	cached
pr-14898_rccl-perf_rocm-10.1-ga	rcclt-develop-perf	develop	`3e8a0e6`	cached
pr-14898_rccl-perf_rocm-10.2-latest	gcc-11.5	11.5.0	11.5.0	cached
pr-14898_rccl-perf_rocm-10.2-latest	rocm-10.2-latest	10.2.0rc1	10.2.0rc1	cached
pr-14898_rccl-perf_rocm-10.2-latest	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14898_rccl-perf_rocm-10.2-latest	rccl-pr-14898	`c1acb47`	`c1acb47`	cached
pr-14898_rccl-perf_rocm-10.2-latest	rcclt-develop-perf	develop	`3e8a0e6`	cached
pr-14898_rccl-smoke_rocm-10.1-ga	gcc-11.5	11.5.0	11.5.0	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rocm-10.1-ga	10.1.0	10.1.0	cached
pr-14898_rccl-smoke_rocm-10.1-ga	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rccl-pr-14898-smoke	`c1acb47`	`c1acb47`	cached
pr-14898_rccl-smoke_rocm-10.1-ga	rccl_gtest-pr-14898	`c1acb47`	`c1acb47`	cached
rocm-10.1-ga_rccl-develop-perf	gcc-11.5	11.5.0	11.5.0	cached
rocm-10.1-ga_rccl-develop-perf	rocm-10.1-ga	10.1.0	10.1.0	cached
rocm-10.1-ga_rccl-develop-perf	openmpi-4.0.2	4.0.2rc3	4.0.2rc3	cached
rocm-10.1-ga_rccl-develop-perf	rccl-develop-2a45136e12c5	`2a45136`	`2a45136`	cached
rocm-10.1-ga_rccl-develop-perf	rcclt-develop-perf	develop	`3e8a0e6`	cached

Failures

No failing tests in this run.

15 tests skipped.

github-actions Bot added the project: rccl label Nov 3, 2026

systems-assistant Bot added the organization: ROCm label Nov 3, 2026

aduforest marked this pull request as ready for review November 3, 2026 21:27

aduforest requested review from a team November 3, 2026 21:27

aduforest marked this pull request as draft November 3, 2026 21:29

aduforest marked this pull request as ready for review November 3, 2026 21:29

amd-argus Bot reviewed Nov 3, 2026

View reviewed changes

amd-argus Bot requested review from corey-derochie-amd and mustafabar November 3, 2026 21:54

aduforest mentioned this pull request Nov 3, 2026

perf(rccl): tune gfx950 2-node AllReduce, AllGather and ReduceScatter #14722

Open

1 task done

aduforest closed this Nov 3, 2026

Navigation Menu

Repository navigation

perf(rccl): gfx950 tuner row for 2-node ReduceScatter 136M-168M - #14898

perf(rccl): gfx950 tuner row for 2-node ReduceScatter 136M-168M#14898
aduforest wants to merge 1 commit into
developfrom
users/aduforest/intellitune-gfx950-2n-reducescatter-row

aduforest commented Nov 3, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Nov 3, 2026

Uh oh!

aduforest commented Nov 3, 2026

Uh oh!

amd-argus Bot left a comment

Uh oh!

aduforest commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

aduforest commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Issue Tracking

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Nov 3, 2026

✅ All Policy Checks Passed

Uh oh!

aduforest commented Nov 3, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Findings

Uh oh!

aduforest commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

pr-rocm-systems-14898-oci-rccl-pr-build

Builds

Failures

pr-rocm-systems-14898-oci-rccl-pr-smoke

Builds

Failures

Uh oh!

github-actions Bot commented Nov 3, 2026

pr-rocm-systems-14898-ruby64-rccl-pr-smoke

Builds

Failures

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aduforest commented Nov 3, 2026 •

edited

Loading