Repository navigation
WarpSpeed Refactor and Gating to SPX - #8743
Conversation
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
There was a problem hiding this comment.
Pull request overview
Restores single-node performance for MI350 when WarpSpeed is enabled by moving the single-node MAXCHANNELS/2 clamp to occur after the gfx950 8-rank single-node halving, avoiding the effective “cap-at-32” behavior introduced by the earlier cap placement.
Changes:
- Removes the early single-node
MAXCHANNELS/2cap when computingncunder WarpSpeed. - Adds a final single-node clamp (
nc = min(nc, MAXCHANNELS/2)) after the gfx950 single-node halving logic to preserve higher channel budgets while still protectingcomm->channels[MAXCHANNELS]from overflow.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
LGTM. I think SPX should be used as gate for other resource intensive algorithm as well, like direct algorithms. |
I'll add a follow up task for this. I think that makes sense at least on gfx942 and gfx950. |
|
Merging in develop because CI failed due to ROCK issue. Fix and bump have landed: ROCm/TheRock@2e7c190 and #9056 |
Cherry pick commit of #8743 ## Motivation Restore performance on single node for MI350 when WarpSpeed enabled. ## Technical Details **New as of 7/22/2026** -Address @Kapil-Shyam-Pawar's feedback regarding changes getting picked up by the testrunner. Thanks for the pointers! :-) **New as of 7/17/2026** - Synced with team. We disable WarpSpeed on non-SPX modes (e.g., DPX, QPX, CPX). Thanks @mustafabar and @wenkaidu for the suggestion. - Move WarpSpeed enablement logic into separate functions that we can unit test - Add unit tests for enablement logic - Fixes regression in single node performance from PR#8077 **Old as of 7/16/2026** Drop in performance due to max channels being capped at 32 for large message sizes on single node. This is one possible fix. Need to verify that it plays nice with CPX and still prevents the issue mentioned in PR#8077. We need to decide if we should just revert to not using WarpSpeed and go back to the original logic. Will consult with team tomorrow. Another option is checking which mode we're in e.g., DPX or CPX and having custom max channel counts for that. ## Issue Tracking JIRA ID: AICOMRCCL-1553 ## Test Plan Ran with this PR on develop and without it. Verified that performance returns and channel count is again 56 for 1GiB. ## Test Result See above ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. (cherry picked from commit 4076c9b) ## Motivation <!-- Explain the purpose of this PR and the goals it aims to achieve. --> ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Issue Tracking <!-- Include a GitHub Issue and/or JIRA ID. Do not post any full JIRA links here. --> <!-- GitHub issue: #1234 --> <!-- JIRA ID: EXAMPLECOMPONENT-1234 --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. Co-authored-by: Corey Derochie <161367113+corey-derochie-amd@users.noreply.github.com>
Motivation
Restore performance on single node for MI350 when WarpSpeed enabled.
Technical Details
New as of 7/22/2026
-Address @Kapil-Shyam-Pawar's feedback regarding changes getting picked up by the testrunner. Thanks for the pointers! :-)
New as of 7/17/2026
Old as of 7/16/2026
Drop in performance due to max channels being capped at 32 for large message sizes on single node. This is one possible fix. Need to verify that it plays nice with CPX and still prevents the issue mentioned in PR#8077.
We need to decide if we should just revert to not using WarpSpeed and go back to the original logic. Will consult with team tomorrow.
Another option is checking which mode we're in e.g., DPX or CPX and having custom max channel counts for that.
Issue Tracking
JIRA ID: AICOMRCCL-1553
Test Plan
Ran with this PR on develop and without it. Verified that performance returns and channel count is again 56 for 1GiB.
Test Result
See above
Submission Checklist