Repository navigation
test(rccl): pin the NIC gate on the graph search even-channel preference - #14002
Draft
thomas-huber wants to merge 1 commit into
Draft
thomas-huber wants to merge 1 commit into
thomas-huber wants to merge 1 commit into
Conversation
Since the 2.32.3 sync, ncclTopoCompareGraphs keeps an even-channel reference over an odd-channel graph with no more aggregate bandwidth, but only while the graph has fewer channels than the system has NICs. CompareGraphs_OddCountTiesGoToTheEvenReference covers the preference; nothing covered the NIC gate. Two SearchMicrotest cases tie an odd graph that has fewer hops against an even reference, on a system without NICs and with as many channels as NICs, and expect the hop count to decide. Upstream: NVIDIA/nccl@24264d1499 ("graph/search: update compareGraphs to favor even channels")
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
NCCL 2.32.3 made
ncclTopoCompareGraphskeep an even-channel reference over an odd-channel graph when the reference has at least the same aggregate bandwidth, but only while the graph has fewer channels than the system has NICs (NVIDIA/nccl@24264d1499). RCCL picked it up with the 2.32.3 sync in #12321.CompareGraphs_OddCountTiesGoToTheEvenReference(#13605) covers the preference; nothing covered the NIC condition, so dropping or loosening it would go unnoticed.Technical Details
SearchMicrotestcases inrccl-UnitTestsMicroTopotie an odd-channel graph that has fewer hops against an even-channel reference with the same aggregate bandwidth, where the NIC condition does not hold, and expect the hop count to decide, as it did before the sync:CompareGraphs_WithoutNicsOddCountTiesGoToFewerHops: two GPUs and no NICs; 1 channel at 40 GB/s against 2 at 20 GB/s.CompareGraphs_OddCountAtTheNicCountTiesGoToFewerHops: three GPUs with a NIC each; 3 channels at 40 GB/s against 4 at 30 GB/s."Search*"intest_categories_micro_topo.yamlalready covers them.Issue Tracking
Test Plan
Search*cases ofrccl-UnitTestsMicroTopofrom the standalone build (test/host/run_host_tests.sh build) and the in-tree build, ondevelop(98f6a72dcc49): the fiveCompareGraphs_*cases three runs each, the other 74 once.ncclTopoCompareGraphschanged three ways, one at a time: the even-channel check disabled, the NIC condition removed, and its<relaxed to<=.install.sh -l -t).Test Result
develop: all 79Search*cases pass in both builds, theCompareGraphs_*cases 3/3.CompareGraphs_OddCountTiesGoToTheEvenReferencefails 3/3; the two new cases pass.CompareGraphs_OddCountTiesGoToTheEvenReferencestill passes.<=instead of<: onlyCompareGraphs_OddCountAtTheNicCountTiesGoToFewerHopsfails 3/3.Search*cases pass with every change, so none of the full searches on the test topologies depends on the preference.Submission Checklist