Repository navigation
Conversation
One row for 2 nodes x 16 ranks, AllReduce [16M, 32M): Tree/LL128 with 64 channels. Without it the choice switches to Ring/LL128 just above 16M and the time doubles; Tree/LL128 runs 1.4x to 2.2x faster from 20M to 32M. All other rows are unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
@argus review |
![amd-argus[bot]](https://avatars.githubusercontent.com/in/4740020?s=60&v=4){kind=small}
pr-rocm-systems-14722-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
pr-rocm-systems-14722-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
aduforest
changed the title
|
Consolidated the related two-node work here: #14969, #14858 and #14898. AllReduce is now one 16–144 MiB row; AllGather and ReduceScatter keep their original ranges. Added embedded-row, boundary, topology and unavailable-protocol fallback tests. The original benchmark limitations remain documented. Please review the expanded change at 18a3ef1; #14484 remains separate with its existing approvals. Nothing was merged into develop. |
aduforest
changed the title
|
@argus re-review |
pr-rocm-systems-14722-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
pr-rocm-systems-14722-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
Validation update for
|
|
Final cross-check with the companion one-node tuning PR #14484:
The remaining merge requirements are external to that validation: the required multi-arch GPU test is queued without an assigned runner, and required human approvals are outstanding. Host tests and Ruby/OCI smoke CI passed; Argus gave LGTM. No CI gate or approval has been bypassed. |
Motivation
Improve AllReduce, AllGather and ReduceScatter performance on MI350X (gfx950), 2 nodes × 8 GPUs (16 ranks), for these message-size ranges:
Lower limits are included; upper limits are excluded.
Technical Details
Updates
projects/rccl/tuner/rccl_tuner_gfx950.csvfor MI350X, 2 nodes × 8 GPUs (16 ranks):Exactly 16 MiB AllReduce stays on Tree/LL128 with 64 channels. Other sizes and topologies keep their settings. Existing AllReduce rows remain as fallbacks when the preferred algorithm/protocol is unavailable.
Results below use float, out-of-place, DEFAULT CTA policy,
-R 0, no graphs; median of six runs per configuration. Reduction operation: sum for AllReduce/ReduceScatter, none for AllGather. AllGather sizes are total receive-buffer bytes; ReduceScatter sizes are total send-buffer bytes.Before:
After:
The small difference at 16 MiB AllReduce is run-to-run variation.
Issue Tracking
JIRA ID: AICOMRCCL-3447
Keep open for further tuning improvements.
Test Plan
Two-node rccl-tests runs with repeated before/after measurements and baseline controls, range-boundary checks, 1–512 MiB sweeps and correctness checking (
-c 1). Tested in-place/out-of-place operations, CTA policies, registration modes and graphs; not every combination.Host tests check CSV loading, message-size boundaries, node/rank matching and fallbacks.
Test Result
The results above show performance improvements of up to 2.01x for AllReduce, 1.16x for AllGather and 1.19x for ReduceScatter.
#wrong=0, testing all three tuning rows together.Follow-up checks completed: with the combined rule, the previously flagged 18 MiB AllReduce case was 48.7% faster. Longer-warmup checks at 2 MiB ReduceScatter were within 0.91% of baseline. Additional outside-range 8–9% differences did not persist in focused reruns: the original-sweep follow-ups still showed differences up to 3.46%, while longer-warmup probes were within 0.7%. These results do not establish that every timing difference is noise or guarantee regression-free performance in every environment.
Test details
The combined matrix covers float in 12 CTA/registration/graph combinations, both in-place and out-of-place, with three balanced baseline/repeated-baseline/candidate rounds. Focused controls use eight four-arm rounds, including an identical baseline and an equal-length placebo, followed by eight longer-warmup rounds.
These comparisons used the same RCCL 2.32.3 build (
091930df78f8d9402e53034fda2adbd53e3e3aec), ROCm 10.1.0 and Open MPI 5.0.10, changing only the CSV—not separate PR/base builds. The rccl-tests monorepo revision was7ee494a5d42c370916257b469584933c1a72716a.Full per-round results, commands, input/binary hashes and independent checkers are retained. See the combined validation update for the results and CI coverage limitations. The before/after tables above are from the earlier individual-rule campaigns; the combined validation is additional evidence.
Submission Checklist