Repository navigation
CI: Run the DL cluster job on the GB200 partition - #11885
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
Stale GB300 description: The JJB job description for Coverage note: CI-config changes like this are validated by the pipeline actually running against the new partition/account. Confirm the |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
🤖 Starting review — findings will be posted here when done. |
|
Slurm allocation may be too short for sequential Build + Test. SLURM_JOB_TIMEOUT: '02:45:00' = 165 minutes. The Build step (parallel: false, runs first alone) has BUILD_TIMEOUT: 45, and the four worker steps (parallel: true) have TEST_TIMEOUT: 140. Worst-case wall time is 45 + 140 = 185 minutes, exceeding the 165-minute allocation. If a build runs near its budget, Slurm can reclaim the node partway through the test phase and fail the run non-deterministically. Consider raising SLURM_JOB_TIMEOUT to cover BUILD_TIMEOUT + TEST_TIMEOUT plus scheduling slack, or confirm the two timeouts are not additive in this framework.
The 'Build' step actually runs the MPI and configure test suites under a 45-minute budget. With BUILD_ONLY_TARBALL set and ASAN_CHECK=no, dispatch calls build_for_tests, which runs run_mpi_tests (perftest-with-MPI, malloc-hook MPI, MPI CUDA tests) and run_configure_tests before the devel build, then packs the tarball. So the step named 'Build' is doing substantial testing, not just compiling, yet it is constrained by BUILD_TIMEOUT: 45 rather than the test timeout. These tests now run only on the single build node. A slow MPI phase can abort the build before any worker starts. Consider renaming the step / raising its timeout, or confirming the MPI+configure phase reliably fits in 45 minutes on the new partition. |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
621fa06 to
c69b2ea
Compare
|
🤖 Starting review — findings will be posted here when done. |
c69b2ea to
eff2c6f
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
🤖 CI Triage Agent — TL;DR: The gtest suite on "roce on worker 1" failed with a single test failure — Full analysisSummary: Root cause: The failing test allocates the target buffer in Implicated commit: none in this PR; the flaky test originates from File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:33 (buffer alloc) and test/gtest/uct/uct_test.cc:985 (retry only on Suggested fix: Re-run the "roce on worker 1" job to confirm it is transient — the PR is CI-config-only and should not be blocked by it. To stop the recurrence, make the RDMA/MEMIC path resilient in Related: #11885 (the PR under test, CI-only), #10475 ("TEST/GTEST/UCT: Increase retries and add random sleep when MEMIC allocation fails") — prior mitigation for the same flakiness.
|
eff2c6f to
35ff2a7
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: this is a CI-config change with no dry-run artifact in the diff — merge should be gated on the |
35ff2a7 to
a2b6a5a
Compare
|
🤖 Starting review — findings will be posted here when done. |
a2b6a5a to
321703f
Compare
|
BUILD_JOBS=32 may be silently capped: the gpu matrix passes
Non-blocking observation: Residual coverage gap: this is a CI-config change with no dry-run artifact in the diff. Merge should be gated on |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
321703f to
b2c8e61
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
BUILD_JOBS=32 can be silently clamped: parallel_jobs is capped by NPROC in buildlib/tools/common.sh, and NPROC is not passed in slurmEnv; if the pyxis container hits the container-detection path, NPROC becomes min(16, mem_GB). CI-config change with no dry-run artifact — merge should be gated on ucx-test-dl-gpu-oss actually scheduling and passing on gb200nvl72_ci under oberon-gb-ci across all four shards. |
b2c8e61 to
160b91d
Compare
|
BUILD_JOBS=32 silently clamped by NPROC: this file passes parallel_jobs=${BUILD_JOBS} but not NPROC in slurmEnv. In buildlib/tools/common.sh, parallel_jobs is clamped to num_pinned_threads=${NPROC}, and if the pyxis container triggers the container-detection path (/.dockerenv, /run/.containerenv, or KUBERNETES_SERVICE_HOST), NPROC becomes min(16, mem_GB). test_gpu_matrix.yaml passes NPROC explicitly; this one does not. pipeline_stop scans the whole jobIdDir rather than filtering by _${BUILD_NUMBER}. CI-config coverage gate: no dry-run artifact; merge should be gated on ucx-test-dl-gpu-oss actually scheduling and passing on gb200nvl72_ci under oberon-gb-ci across all four shards. |
b30777b to
90148e3
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
90148e3 to
6ceba53
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
/build |
Alexey-Rivkin
changed the title
6ceba53 to
16c2fb2
Compare
|
🤖 Starting review — findings will be posted here when done. |
The job never completed a run. It sat on gb300nvl72_ci, where 36 nodes serve a queue that stays around 30 deep, so salloc burned its 3600s --immediate window every time. Slurm hands out these nodes FIFO by queue age with no account carrying priority, so nothing short of moving helps. gb200nvl72_ci has 99 nodes and a short queue; svcnbu-swx-hpcx already belongs to its account, and the cluster's cli_filter ties account and partition together. Once a node arrives, one worker cannot finish the suite inside the step budget. Shard four ways like ucx-test-gpu-oss, one allocation each; build with make -j32 instead of common.sh's default of 4; give the test step 180 minutes, which the slowest shards need. Checkout is its own step, so a clone failure shows up as one. Build dl-base on ucx/aarch64/ubuntu24.04-mofed24.10-cuda13, the image the Azure release pipeline already uses, instead of the hpcx builder: same OS and MOFED, plus the CUDA 13 toolkit, the NVIDIA driver hook and ibv_devinfo. On top go gdrcopy, which dlcluster has no environment module for, and python3-setuptools for test_ucx_tls.py. enroot injects the invoking user into the container itself, so the useradd goes. try_load_cuda_env gave up before looking for a GPU whenever the nvidia_peermem sysfs file was missing. dlcluster nodes register GPU memory through dmabuf and do not load that module, so the CUDA test paths never ran there. Require peermem only on Azure, where it is how GPUDirect RDMA is provided. The gdrcopy module probe at the end of the same function leaked its failure status out of the function, which under errexit ended the whole run on any host without environment modules; test it in the if condition. Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
16c2fb2 to
e5e428a
Compare
Not clamped. pyxis sets none of
It does filter, by
#27 is that run, from |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
/build |
|
🤖 CI Triage Agent — TL;DR: Two of four gtest shards (funk13, funk15) aborted because large-message IB/RC RMA operations stopped completing entirely — the requests sat for the full 180 s test deadline and the 900 s watchdog then SIGABRT'ed the run; this is a node/fabric-level hang, not a defect in PR #11885's CI-only diff (the same commit passed all 4 shards in build #17 on funk05/06/07/11). Full analysisSummary: Root cause: A hang, not slowness. Timestamp analysis of the stage logs shows the runtime is dominated by silent gaps of exactly the request deadline (~3 min each) followed by a final ~12 min gap up to the watchdog kill:
In both cases small transfers in the same test completed normally and then the large zero-copy RC/IB put/get stalled forever with no error CQE, no peer-failure or endpoint-timeout log — i.e. the IB QP silently stopped completing work requests. Evidence that this is environment-specific rather than a code regression: the same branch/commit passed all four shards in build #17 (nodes funk05, funk06, funk07, funk11), and in build #18 the two shards that ran on other nodes (incl. funk16) passed 9108/9108 tests. Only funk13 and funk15 — nodes not used by build #17 — hung. The head commit Implicated commit: none in UCX source. Head commit File: Suggested fix:
Related: PR #11885 (branch |
|
🤖 CI Triage Agent — TL;DR: The new GB200 ( Full analysisSummary: All four GB200 test workers failed: Root cause: Two distinct environment defects on the newly added
Corroborating node misconfiguration in the same logs: Implicated commit: [REDACTED:Hex High Entropy String] (branch File: Suggested fix:
Related: PR #11885 (this branch). No existing issue found for
|
|
🤖 CI Triage Agent — TL;DR: Build #19 never ran any tests — it was a Replay of build #18 and therefore tried to check out the stale commit Full analysisSummary: The "Checkout source code" stage failed 6 times in a row with Root cause: Not a UCX code or test failure — a SCM/job-configuration issue:
Implicated commit: No code commit is at fault. The branch update that orphaned the replayed SHA is File: Jenkins job SCM configuration for Suggested fix:
Related: PR #11885 "CI: Run the DL cluster job on the GB200 partition" (#11885); predecessor build #18 of
|
|
🤖 CI Triage Agent — TL;DR: Build #29 never ran any tests — it was a Replay of build #28 and therefore tried to check out the pinned SHA Full analysisSummary: The "Checkout source code" stage failed all 6 retry attempts with Root cause: Pure SCM/checkout problem, not a UCX code or test failure. The console log shows
The trigger commit for this event is Two aggravating configuration details visible in the log:
Implicated commit: No faulty code commit. The precipitating event is the force-push of PR #11885 ( File: Jenkins job SCM configuration for Suggested fix:
Related: PR #11885 "CI: Run the DL cluster job on the GB200 partition" (#11885) — the PR under test; build #28 is the replayed parent build. No existing issue found for this checkout signature.
|
What?
Get
ucx-test-dl-gpurunning on the GB200 DL cluster.Why?
The job never passed: it timed out waiting for a node on
gb300nvl72_ci, and even with a node one worker could not finish the suite in time.How?
gb200nvl72_ci/ accountoberon-gb-ci(99 nodes vs 36, tied together by the cluster'scli_filter).workeraxis), separate checkout and test steps,make -j32, 180-min test step.dl-baserebased on the CUDA 13 release image (buildlib/dockers) plus gdrcopy, so CUDA tests build and run; no localuseraddneeded.try_load_cuda_envonly needsnvidia_peermemon Azure; dlcluster uses dmabuf.UCX issues surfaced (not fixed here)
GB200: aarch64, GPUDirect via dmabuf, no
nvidia_peermem.dcx/test_ucp_mmap.rndv_mpool_mdesc_no_rcache/2(ucp_mm.c:84EIO ->mpool.c:246-> SIGSEGV). Same crash-after-failure pattern aftercuda_ipc/test_cuda_ipc_md.mkey_pack_mempool/0anddc_mlx5/uct_p2p_rma_test.{put_short,put_zcopy,get_bcopy}.ucp_client_server -c stream -s 1(intermittent, both fleets),dcx/test_ucp_tag_mt.send_recv.test_ucp_devices_config_mlx5.range_negate*assumes contiguousmlx5_0..N; nodes havemlx5_6,mlx5_16.shm/test_ucp_wait_mem.envelope/0:ucp_worker_wait_mem()~40x slower than reference put on aarch64.Endpoint timeoutover RoCE:test_ucp_tag_nbx.fallback/2(rc/rcx/dcx/shm_ib),rcx/test_ucp_rma_mt.put_get/0.rc_verbs/uct_p2p_rma_test_inlresp.get_*_inlresp*,all/test_ucp_ep_based_fence.test_ep_based_fence_get/0.Logs (internal link):
https://nbuprod.blsm.nvidia.com/nbu-swx-ucx-master/job/UCX-OSS/job/ucx-test-dl-gpu-oss/26/