Repository navigation
fix(rccl): disable Rome model matching for gfx1250 - #10161
Merged
nileshnegi merged 4 commits intoAug 25, 2026
Merged
Conversation
Skip preset topology matching for gfx1250 GPUs by setting skipPresetTopoMatching=true when the architecture is detected. This causes the generic topology search algorithm to be used instead of the hardcoded Rome model patterns, which are not appropriate for gfx1250. Add unit test that initializes a real communicator and verifies skipPresetTopoMatching and romeTopoModelIdx on gfx1250 hardware. Co-Authored-By: Claude <noreply@anthropic.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates RCCL’s topology initialization to bypass Rome preset topology model matching on gfx1250 so the generic topology search is used immediately, and adds a unit test intended to validate the new behavior on real gfx1250 hardware.
Changes:
- Update
initTransportsRank()to setcomm->topo->skipPresetTopoMatchingwhen the detected GPU arch matchesgfx1250. - Add a new isolated gtest that initializes a communicator and checks
skipPresetTopoMatching/romeTopoModelIdxon gfx1250.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| projects/rccl/src/init.cc | Extends skipPresetTopoMatching logic to force skipping Rome preset matching on gfx1250. |
| projects/rccl/test/RcclWrapTests.cpp | Adds a new isolated runtime test for gfx1250 preset-matching skip behavior. |
Suppressed comments (1)
projects/rccl/test/RcclWrapTests.cpp:2420
- After skipping on non-gfx1250, the remaining assertions can be unconditional. Also, destroy should not use NCCLCHECK here for the same reason; prefer ASSERT_EQ to ensure failures are surfaced and the test doesn't continue with invalid state.
// Cast to internal struct to access topo
ncclComm* comm = commHandle;
if (isGfx1250) {
// gfx1250 should skip preset topo matching
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Remove StandaloneUtils.hpp include to avoid NCCLCHECK macro conflict - Replace HIPCALL/NCCLCHECK macros with gtest ASSERT_EQ checks - Skip test early on non-gfx1250 hardware instead of conditional assertions Addresses Copilot review comments on PR #10161. Co-Authored-By: Claude <noreply@anthropic.com>
Wrap the gfx1250 check in a block scope to avoid goto-over-initialization errors from NCCLCHECKGOTO macros earlier in the function. Co-Authored-By: Claude <noreply@anthropic.com>
nileshnegi
force-pushed
the
users/nileshnegi/rccl/gfx1250-disable-model-match
branch
from
August 14, 2026 16:04
5b31872 to
802564e
Compare
atulkulk
approved these changes
Aug 15, 2026
wenkaidu
approved these changes
Aug 25, 2026
nileshnegi
deleted the
users/nileshnegi/rccl/gfx1250-disable-model-match
branch
August 25, 2026 16:06
cmcknigh
pushed a commit
that referenced
this pull request
Aug 25, 2026
## Motivation The Rome model-matching logic in `rome_models.cc` contains hardcoded topology patterns for specific AMD GPUs (MI2XX and MI3XX series). These patterns define optimized ring/tree topologies based on known GPU-to-GPU connectivity matrices. gfx1250 has a different interconnect topology that doesn't match any existing Rome model. When the matcher attempts to find a matching model and fails, it falls back to generic search anyway, but only after wasting cycles on futile pattern comparisons. More critically, a partial or incorrect match could select a suboptimal topology. This change proactively skips Rome model matching for gfx1250, ensuring the generic topology search algorithm is used from the start. ## Technical Details - Modified the `skipPresetTopoMatching` assignment in `initTransportsRank()` to include an architecture check - When `comm->topo->nodes[GPU].nodes[0].gpu.gcn` matches "gfx1250", preset matching is skipped. Rome model parsers (parseChordalRing, parseA2a8P, parseRome4P2H, parse1H16P, parse4H4P, parseGIOTopos) are bypassed. - Control falls through to the generic `ncclTopoSearchRec()` algorithm ## Issue Tracking JIRA ID: AICOMRCCL-1932 --------- Co-authored-by: Claude <noreply@anthropic.com>
ajanicijamd
pushed a commit
that referenced
this pull request
Aug 25, 2026
## Motivation The Rome model-matching logic in `rome_models.cc` contains hardcoded topology patterns for specific AMD GPUs (MI2XX and MI3XX series). These patterns define optimized ring/tree topologies based on known GPU-to-GPU connectivity matrices. gfx1250 has a different interconnect topology that doesn't match any existing Rome model. When the matcher attempts to find a matching model and fails, it falls back to generic search anyway, but only after wasting cycles on futile pattern comparisons. More critically, a partial or incorrect match could select a suboptimal topology. This change proactively skips Rome model matching for gfx1250, ensuring the generic topology search algorithm is used from the start. ## Technical Details - Modified the `skipPresetTopoMatching` assignment in `initTransportsRank()` to include an architecture check - When `comm->topo->nodes[GPU].nodes[0].gpu.gcn` matches "gfx1250", preset matching is skipped. Rome model parsers (parseChordalRing, parseA2a8P, parseRome4P2H, parse1H16P, parse4H4P, parseGIOTopos) are bypassed. - Control falls through to the generic `ncclTopoSearchRec()` algorithm ## Issue Tracking JIRA ID: AICOMRCCL-1932 --------- Co-authored-by: Claude <noreply@anthropic.com>
This was referenced Aug 28, 2026
5 of 9 tasks
sdarisala
pushed a commit
that referenced
this pull request
Aug 28, 2026
## Motivation The Rome model-matching logic in `rome_models.cc` contains hardcoded topology patterns for specific AMD GPUs (MI2XX and MI3XX series). These patterns define optimized ring/tree topologies based on known GPU-to-GPU connectivity matrices. gfx1250 has a different interconnect topology that doesn't match any existing Rome model. When the matcher attempts to find a matching model and fails, it falls back to generic search anyway, but only after wasting cycles on futile pattern comparisons. More critically, a partial or incorrect match could select a suboptimal topology. This change proactively skips Rome model matching for gfx1250, ensuring the generic topology search algorithm is used from the start. ## Technical Details - Modified the `skipPresetTopoMatching` assignment in `initTransportsRank()` to include an architecture check - When `comm->topo->nodes[GPU].nodes[0].gpu.gcn` matches "gfx1250", preset matching is skipped. Rome model parsers (parseChordalRing, parseA2a8P, parseRome4P2H, parse1H16P, parse4H4P, parseGIOTopos) are bypassed. - Control falls through to the generic `ncclTopoSearchRec()` algorithm ## Issue Tracking JIRA ID: AICOMRCCL-1932 --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The Rome model-matching logic in
rome_models.cccontains hardcoded topology patterns for specific AMD GPUs (MI2XX and MI3XX series). These patterns define optimized ring/tree topologies based on known GPU-to-GPU connectivity matrices.gfx1250 has a different interconnect topology that doesn't match any existing Rome model. When the matcher attempts to find a matching model and fails, it falls back to generic search anyway, but only after wasting cycles on futile pattern comparisons. More critically, a partial or incorrect match could select a suboptimal topology.
This change proactively skips Rome model matching for gfx1250, ensuring the generic topology search algorithm is used from the start.
Technical Details
skipPresetTopoMatchingassignment ininitTransportsRank()to include an architecture checkcomm->topo->nodes[GPU].nodes[0].gpu.gcnmatches "gfx1250", preset matching is skipped. Rome model parsers (parseChordalRing, parseA2a8P, parseRome4P2H, parse1H16P, parse4H4P, parseGIOTopos) are bypassed.ncclTopoSearchRec()algorithmIssue Tracking
JIRA ID: AICOMRCCL-1932
Test Plan
Added
TEST(SkipPresetTopoMatching, Gfx1250_SkipsRomeModelMatching)totest/RcclWrapTests.cppTest Result
Submission Checklist