Repository navigation
Conversation
… name KernelIdIsLL_OnlyTheLLKernels covered 3 of the 5 ids in kernelMask_LL, and both tests listed only a few non-LL and non-Tma ids, so dropping an id from either mask, or adding a vector id to it, could stay green. Both now loop over every kernel id and compare against its name: exactly the kernels named *LL* are LL and exactly those named *Tma* are TDM. Also move the float64 comment back above the test it describes.
SymKGetInfo_TmaKernelReportsSymmetricTdm stubbed rcclSymkKernelIdIsTma to return true for any id, so rcclSymkQuery asking about the wrong kernel id stayed green. OnlyKernelIsTma answers true for the expected id only and replaces the three different stubs.
…electors Only the AllReduce selector had a test asserting RCCL_SYMMETRIC_TDM, so reverting decision->algo to RCCL_SYMMETRIC in rcclSelectAllGather or rcclSelectReduceScatter passed every host test. Each new case fakes a TDM kernel id, asserts SymTDM in query mode and RCCL_SYMMETRIC on the live path.
SelectSymkTuning ignored its input, so passing the wrong collective to rcclSymkQuery from rcclSelectAllReduce, rcclSelectAllGather or rcclSelectReduceScatter passed every wrap test. The stub now takes the expected collective and checks in->func.
The in-place cases used ncclInt8 only, where count equals bytes, so dropping ncclTypeSize from the slice size stayed green. Float32 slices of 6 and 8 elements make the element-count rule disagree on ranks 2 and 0.
ReduceScatter.SymmetricInPlaceSlices and AllGather.SymmetricInPlaceSlices were listed in no test_runner config, and every filter there is exact, so the gfx1250 lane never ran them.
CollImplInfoTests decodes "<Func> [Symmetric]: ... Kernel <name>" and hands a child skip to its parent through a file, since an isolated child's skip scores as a pass. Move both into test/common so the in-place slice tests reuse them instead of carrying a second parser.
…l that ran The cases only checked values, so a fallback to ring passed, and both skip paths ran inside the isolated child, where a skip scores as a pass. Each case now queries rcclGetCollImplInfo on every rank and requires one answer, and requires rank 0's [Symmetric] TUNING line to name the kernel the report claims. On gfx1250 it requires a symmetric kernel, and for the aligned slice the TDM kernel whenever the build's SDK ships the TDM header; the unaligned slice may take either tier. A failed window registration there fails instead of skipping. The GPU-count gate moves to the parent, a child skip is handed back to it, and NCCL_ALGO is cleared in the child. The unaligned slice is 4 MiB + 8 B, so with 3 or more ranks rank 2 sits on the 16 B grid while rank 1 does not. Sizing the slice in elements instead of bytes then splits the ranks between kernels.
pvallem
merged commit Oct 20, 2026
662f3fc
into
users/abreslow/symmetricMemoryTDMEnable
23 of 25 checks passed
alex-breslow-amd
pushed a commit
that referenced
this pull request
Oct 20, 2026
…#13787) The tests in #13357 miss AllGather/ReduceScatter SymTDM reporting, the queried collective, the byte-sized slice and the full LL/TDM kernel sets, and its in-place GPU tests run in no config. Fix: add those host tests, make the GPU tests check the kernel that ran, and list them in the MI455 configs. JIRA ID: AICOMRCCL-2008
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tests in #13357 miss AllGather/ReduceScatter SymTDM reporting, the queried collective, the byte-sized slice and the full LL/TDM kernel sets, and its in-place GPU tests run in no config.
Fix: add those host tests, make the GPU tests check the kernel that ran, and list them in the MI455 configs.
JIRA ID: AICOMRCCL-2008