Repository navigation
fix(rccl): CPX 64-rank comm-init crashes (bootstrap AllGather + WarpSpeed) (#8077) - #8194
Merged
marbre merged 1 commit intoJul 7, 2026
Merged
Conversation
…peed) (#8077) Cherry-picks commit 831fad2 into the 7.14 release branch. --- ## Motivation On gfx950 CPX+NPS2 (64 ranks), all_gather_perf aborts during comm init even after #7766. Issues identified: - A bad-pointer `Bad address` in the topology bootstrap AllGather ``` NCCL WARN ncclOsSocketProgressOpt: Call to send to 10.235.24.26<60509> failed : Bad address ``` - A segfault from WarpSpeed over-allocating channels. ``` NCCL INFO WarpSpeed enabled: warpSpeedChannelMultiplier 4, maxNchannels 1760, nc 1760 ``` ## Technical Details - bootstrap.cc: ring slice offset slice * size overflowed 32-bit int with the ~102 MB per-rank topology buffer at 64 ranks - connect.cc: single-node WarpSpeed channel count was unclamped (nc=1760), overrunning comm->channels[MAXCHANNELS] ## JIRA ID JIRA ID : ROCM-27078 ## Test Plan - Run all_gather_perf on gfx950 CPX+NPS2 (64 GPUs) - Add host unit tests for the 64-bit offset and the channel clamp ## Test Result - AllGather on CPX+NPS2 passes end-to-end with no bad address/segfault - New unit tests (`BootstrapBidir.SliceOffsetIsComputedIn64Bit`, `Rcclwrap.WarpSpeedChannelClampCapsAtHalfMaxChannels`) pass ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
araravik-psd
approved these changes
Jul 7, 2026
araravik-psd
left a comment
araravik-psd
left a comment
Contributor
There was a problem hiding this comment.
CI builds and single node tests have passed and the Cherry-pick seems to have applied cleanly. Approving this PR
astyrrian1
approved these changes
Jul 7, 2026
marbre
added a commit
to ROCm/TheRock
that referenced
this pull request
Jul 8, 2026
Bumps rocm-libraries and rocm-systems to include the following cherry-picks: * ROCm/rocm-libraries#9149, JIRA ID ROCM-27477 * ROCm/rocm-systems#8194, JIRA ID ROCM-27078 * ROCm/rocm-systems#8212, JIRA ID ROCM-27116
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks commit 831fad2 into the 7.14 release branch.
Motivation
On gfx950 CPX+NPS2 (64 ranks), all_gather_perf aborts during comm init even after #7766.
Issues identified:
Bad addressin the topology bootstrap AllGatherTechnical Details
JIRA ID
JIRA ID : ROCM-27078
Test Plan
Test Result
BootstrapBidir.SliceOffsetIsComputedIn64Bit,Rcclwrap.WarpSpeedChannelClampCapsAtHalfMaxChannels) passSubmission Checklist