Repository navigation
fix(rccl): drop the GDR flush case the NCCL sync never wired up - #12062
Conversation
ForcedScratchpadWrite_ReproducesFault calls gdrSupported and a four-argument RunRecvFlushBurst, neither of which exists, so rccl-UnitTestsMPI does not build when fault injection is on. Co-authored-by: Cursor <cursoragent@cursor.com>
…k lands ForcedScratchpadWrite_ReproducesFault was merge-introduced and never callable; keep the intent next to the remaining flush tests. Co-authored-by: Cursor <cursoragent@cursor.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
There was a problem hiding this comment.
ARGUS
Bottom line: LGTM once the comment at GdrFlushTests.cpp:221 drops the "removed" framing for the scratchpad RDMA_WRITE. That one is a wording fix, so it does not need another pass from me.
Comment @argus review to re-run, or @argus re-review after pushing.
Co-authored-by: amd-argus[bot] <321830313+amd-argus[bot]@users.noreply.github.com>
There was a problem hiding this comment.
ARGUS
Bottom line: LGTM once the comment at GdrFlushTests.cpp:220-225 is collapsed onto the suggested two lines: the round-1 one-liner was appended rather than substituted, so line 224 breaks off mid-sentence. That is a comment fix, so it does not need another pass from me.
Comment @argus review to re-run, or @argus re-review after pushing.
TheRock Submodule Bump ActivityNewest first
|
Motivation
The NCCL v2.31.2-1 sync (
b71e057a71c) addedGdrFlushTest.ForcedScratchpadWrite_ReproducesFaultunderENABLE_FAULT_INJECTION. That case callsgdrSupported()andRunRecvFlushBurst(..., /*forceWrite=*/true, &forced). There is nogdrSupportedsymbol, andRunRecvFlushBursttakes three arguments.FAULT_INJECTIONdefaults toON, sorccl-UnitTestsMPIfails to compile. The nightly then reports return code 183 because OpenMPI cannot start the missing binary. The coco PR arm does not build this target, so the break only shows up inoci-rccl-unit-nightly.#10450wrapped the case in#if 0. The calls were still the merge-introduced ones, and the case cannot be restored without aforceWritehook that was never added.Technical Details
Remove
ForcedScratchpadWrite_ReproducesFaultfromprojects/rccl/test/transport/NetIbMPI/GdrFlushTests.cpp. It was not in the tree before the sync. The other GDR flush cases keep the three-argumentRunRecvFlushBurstandgdrPtrSupport().Issue Tracking
JIRA ID: AICOMRCCL-2403
Test Plan
On a compute node with ROCm 7.14.0, from a tree that contains
b71e057a71c, rebuildrccl-UnitTestsMPIwithENABLE_MPI_TESTS=ONandFAULT_INJECTION=ON.Confirm
GdrFlushTests.cppcompiles, the binary is produced, andForcedScratchpadWriteis not linked.Test Result
The debug cache had
FAULT_INJECTION=ONandENABLE_MPI_TESTS=ON.GdrFlushTests.cppcompiled andrccl-UnitTestsMPIlinked.Submission Checklist