Skip to content

CI: enable GPU tests for nixlbench on DL cluster #1981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dpressle wants to merge 2 commits into from
Closed

CI: enable GPU tests for nixlbench on DL cluster #1981

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aaac113
CI: enable GPU tests for nixlbench on DL cluster
dpressle Jul 22, 2026
0dfeeeb
tests/ucx: skip SingleWriteTest when GPUDirect RDMA is unavailable
dpressle Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .ci/jenkins/lib/test-dl-matrix.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,9 +198,6 @@ steps:
dockerImage: "${registry_host}#${ARTIFACTORY_PATH}/pr/${arch}/nixl-ci-dl-gpu-test-${ucx_version}:${BUILD_NUMBER}"
credentialsId: "${SSH_CREDENTIALS_ID}"
containerName: "nixl-ci-${ucx_version}-${BUILD_NUMBER}"
slurmEnv: [
'HAS_GPU=false'
]

pipeline_stop:
containerSelector: "{name: 'build_helper_dl'}"
Expand Down
3 changes: 3 additions & 0 deletions test/gtest/device_api/single_write_test.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,9 @@ protected:
EXPECT_NE(backend_handle, nullptr);
backend_handles.push_back(backend_handle);
}
if (lig_->getIgnoredCount() > 0) {
GTEST_SKIP() << "UCX accelerated IB support not found; GPUDirect RDMA unavailable";
}
Comment on lines +202 to +204

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and inspect the relevant test file sections.
git ls-files 'test/gtest/device_api/single_write_test.cu'
echo '---'
nl -ba test/gtest/device_api/single_write_test.cu | sed -n '1,280p'
echo '--- SEARCH ---'
rg -n "getIgnoredCount|LogIgnoreGuard|createBackend|GPUDirect|ibv_reg_mr_iova2|Bad address|accelerated-IB" test/gtest/device_api/single_write_test.cu test/gtest/device_api -S

Repository: ai-dynamo/nixl

Length of output: 239

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map related symbols to understand whether the skip only covers backend creation.
rg -n "class LogIgnoreGuard|struct LogIgnoreGuard|getIgnoredCount|createBackend\(" -S .

Repository: ai-dynamo/nixl

Length of output: 10157

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Search for the specific failure mode in repo tests/comments/docs.
rg -n "ibv_reg_mr_iova2|Bad address|GPUDirect RDMA unavailable|accelerated IB support not found" -S .

Repository: ai-dynamo/nixl

Length of output: 297

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the test body and the warning guard implementation.
sed -n '150,230p' test/gtest/device_api/single_write_test.cu
echo '---'
sed -n '120,190p' test/gtest/common.h
echo '---'
sed -n '140,185p' test/gtest/common.cpp

Repository: ai-dynamo/nixl

Length of output: 5708

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the rest of the test to see where GPU memory registration happens relative to the skip.
sed -n '230,420p' test/gtest/device_api/single_write_test.cu
echo '---'
rg -n "register|MemBuffer|cudaMalloc|cudaHostAlloc|queryMem|makeDescList|getBuffer" test/gtest/device_api/single_write_test.cu test/gtest/device_api/utils.cu test/gtest/device_api -S

Repository: ai-dynamo/nixl

Length of output: 14417

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the UCX createBackend implementation and any log messages about accelerated IB / GDR registration.
rg -n "accelerated IB support was not found|accelerated IB support not found|IB device\\(s\\) were detected|ibv_reg_mr_iova2|Bad address|GPUDirect RDMA|gdr|registration" src test -S

Repository: ai-dynamo/nixl

Length of output: 15333

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the UCX helper around the VRAM registration / accelerated IB warning path.
sed -n '560,670p' src/plugins/ucx/ucx_utils.cpp
echo '---'
sed -n '120,210p' test/gtest/hw_warning_test.cpp

Repository: ai-dynamo/nixl

Length of output: 6193

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect whether the device test has any later skip/guard around registration.
sed -n '420,560p' test/gtest/device_api/single_write_test.cu

Repository: ai-dynamo/nixl

Length of output: 5748

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Confirm if any other tests use the same warning regex as a skip signal.
rg -n "IB device\\(s\\) were detected, but accelerated IB support was not found|registerMem: registration failed|GPUDirect RDMA unavailable" test src -S

Repository: ai-dynamo/nixl

Length of output: 666

Broaden the skip to cover registerMem() failures
This only skips when the backend-creation warning is emitted. The test still reaches registerMem() and can fail later on GPUDirect RDMA registration, so affected nodes will fail instead of skipping. Add a probe for the registration path or skip on registerMem() failure.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/gtest/device_api/single_write_test.cu` around lines 202 - 204, Broaden
the setup logic around getIgnoredCount() and registerMem() so the test skips
when GPUDirect RDMA memory registration is unavailable, not only when backend
creation reports a warning. Probe the registration path or handle registerMem()
failure by issuing GTEST_SKIP(), while preserving normal assertions for other
failures.

}

void
Expand Down
Loading