Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 6 additions & 28 deletions projects/rccl/test/transport/NetIbMPI/GdrFlushTests.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -217,34 +217,12 @@ TEST_F(GdrFlushTest, RepeatedFlush_NoFaultBurst) {
EXPECT_EQ(flush, ncclSuccess) << "no flush in the burst may raise a QP async-fatal";
}

// Regression guard - force the removed scratchpad RDMA_WRITE back. On a dma-buf
// scratchpad this must reproduce the fault (flush no longer succeeds), proving
// the WRITE is the culprit and that removing it is the fix.
//
// Disabled: the NCCL v2.31.2-1 sync (#10785) added this case but not the two
// things it calls. RunRecvFlushBurst has no forceWrite parameter and there is no
// gdrSupported(), so a --debug --tests_build --enable-mpi-tests build of develop
// does not compile once ENABLE_FAULT_INJECTION is on. Restoring the forceWrite
// hook is a decision for whoever owns the GDR flush fence, not something to
// guess at here, so the case is guarded off rather than half-fixed.
#if 0
TEST_F(GdrFlushTest, ForcedScratchpadWrite_ReproducesFault) {
SKIP_UNLESS_MPI_PREREQS(kExactTwoProcesses, kExactTwoProcesses,
false, kMinGpusPerNode, kNoNodeLimit);
if (!cuMemEnabledEnv()) GTEST_SKIP() << "Requires NCCL_CUMEM_ENABLE=1 (dma-buf scratchpad target)";
if (!scratchpadFlushEnabled())
GTEST_SKIP() << "Requires the scratchpad flush enabled (RCCL_GDR_FLUSH_GPU_MEM_NO_RELAXED_ORDERING=1)";
AssertInitAndGetDevices(nullptr);
if (!(gdrPtrSupport() & NCCL_PTR_CUDA)) GTEST_SKIP() << "GDR (NCCL_PTR_CUDA) not supported on this device";

ncclResult_t forced = ncclSuccess;
RunRecvFlushBurst(/*iterations=*/1, /*verifyData=*/false,
&forced);
if (MPIEnvironment::world_rank == 0)
EXPECT_NE(forced, ncclSuccess)
<< "forced scratchpad RDMA_WRITE on a dma-buf buffer should fault the flush QP";
}
#endif // disabled pending the forceWrite hook
// The NCCL v2.31.2-1 sync added ForcedScratchpadWrite_ReproducesFault: force the
// removed scratchpad RDMA_WRITE on a dma-buf flush QP and expect the flush to
// fault. That case never compiled: gdrSupported() does not exist, and
// RunRecvFlushBurst has no forceWrite parameter. If a production forceWrite
// hook is added, restore the case (skip with gdrPtrSupport(), three-arg burst
// plus the hook) rather than leaving the regression untested.
Comment thread
amd-jiali marked this conversation as resolved.
Resolved

} // namespace

Expand Down
Loading