CI: add VLLM+nixl_ep test to EP CI job (first version)

[lishapira]

Add vLLM + NIXL EP tests to the existing NIXL EP CI job. vLLM runs test_elastic_ep_scaling with nixl_ep as the all-to-all backend.

• prepare vLLM and the backend-selection patch in the EP base image
• create one Torch 2.13.0 CUDA 13 venv; build NIXL EP and run both elastic.py and vLLM from it to keep their Torch versions aligned
• update vLLM from its configured release branch only when the EP base image rebuilds; the cached base image does not update it on every PR
• print the resolved vLLM ref and commit before the vLLM Elastic EP test
• run the vLLM Elastic EP test after native elastic.py on the same allocation
• extend EP CI timeouts, model-download configuration, cache invalidation, and docs

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi lishapira! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 93baa21c

TL;DR: The nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04 image build failed during the vLLM elastic-test step because import torch loaded HPC-X's /opt/hpcx/ucc/lib/libucc.so.1, which needs the UCX symbol ucs_config_doc_nop that the resolved libucs no longer provides — a UCX/UCC (HPC-X) version mismatch in the base image. Realign the UCX used with the HPC-X UCC (rebuild/pin matching UCX or drop the mismatched libucs from the loader path).

Full analysis

Summary: Setup Image stage #37 failed while building the vLLM editable install for the DL/EP elastic test; import torch aborts with an undefined-symbol error from libucc.so.1.

Root cause: Not a compile error in NIXL itself — the NIXL/Mooncake/Azure builds all succeeded (PRE_INSTALLED_UCX_ENV/PRE_INSTALLED_NIXL_ENV are set, so UCX and NIXL are provided by the base image). The Dockerfile step gated by BUILD_VLLM_ELASTIC_TEST=true runs uv pip install --editable vllm, whose setup.py does import torch. torch transitively loads /opt/hpcx/ucc/lib/libucc.so.1, which fails:

ImportError: /opt/hpcx/ucc/lib/libucc.so.1: undefined symbol: ucs_config_doc_nop

ucs_config_doc_nop is exported by UCX's libucs. HPC-X's UCC was built against a UCX that exports it, but the libucs actually resolved at load time (the pre-installed/rebuilt UCX in the image) does not — i.e. the UCC in /opt/hpcx and the effective UCX are из different, incompatible versions. This is an environment/library-ABI mismatch, not a code defect.

Implicated commit: Most likely the base-image UCX change — d815856f (NirWolfer, 2026-08-19, "CI: build the UCX spcx plugin in the verification container", PR #2064), which lands the same day as this build and alters the UCX shipped in the container. The build-script default UCX pin is v1.22.x (.gitlab/build.sh:32, from 3ab92dc9, Mikhail Brinskiy). Cannot be certain without the image-build manifest, so treat as probable rather than confirmed.

File: .gitlab/build.sh:382-419 (UCX handling / PRE_INSTALLED_UCX_ENV) — the runtime symptom surfaces at the Dockerfile BUILD_VLLM_ELASTIC_TEST RUN step, on import torch → /opt/hpcx/ucc/lib/libucc.so.1.

Suggested fix: Make the UCX that ends up on the loader path match the HPC-X UCC in the image. Concretely, either (a) don't shadow HPC-X's UCX/UCS with a separately built UCX for this base image (ensure /opt/hpcx/.../libucs.so is the one resolved, e.g. via LD_LIBRARY_PATH/ldconfig ordering), or (b) rebuild/point UCC at the same UCX version the container's UCX provides, or (c) if the spcx-plugin/UCX rebuild from #2064 introduced the mismatched libucs, rebuild UCC against it or bump/pin UCX so ucs_config_doc_nop is present. As a quick check, run ldd /opt/hpcx/ucc/lib/libucc.so.1 and nm -D <resolved libucs> | grep ucs_config_doc_nop in the image to confirm which libucs is picked up.

Related: PR #2064 (d815856, UCX spcx plugin in verification container); UCX-version bump PR #1868 (3ab92dc9). No existing issue found for ucs_config_doc_nop.

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 065a923a

TL;DR: The nixl-ci-dl-gpu-ep-base Docker image build failed while building vLLM because torch fails to load HPC-X's libucc.so.1 (undefined symbol: ucs_config_doc_nop) — a UCC/UCX ABI mismatch in the base image. Fix the HPC-X UCC/UCX pairing (or preload the matching libucs) before running the vLLM install step.

Full analysis

Summary: Stage 37 ("Setup Image aarch64/nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04/0") failed building the vLLM elastic-test image; import torch aborts with an unresolved UCX symbol from HPC-X's UCC.

Root cause: The new BUILD_VLLM_ELASTIC_TEST Docker step runs an editable vLLM install; setuptools imports torch, which loads /opt/hpcx/ucc/lib/libucc.so.1, failing with ImportError: /opt/hpcx/ucc/lib/libucc.so.1: undefined symbol: ucs_config_doc_nop. The build's own diagnostics show UCC resolving libucs.so.0 out of the bundled HPC-X UCX (reported by ucx_info -v as v1.21.0 / HPC-X v2.50), but ucs_config_doc_nop is not present in that libucs — i.e. the libucc.so.1 shipped in the pytorch26.06-cuda13.3 base image was built against a different UCX ABI than the UCX actually loaded at runtime. This is an environment/image mismatch, not a NIXL code defect.

Implicated commit: 33606ee9 (Lior Shapira, "CI: add VLLM+nixl_ep test to EP CI job"), with 0c37adda (same author) adding the diagnostics — both from PR #2123. These introduced the vLLM build step that first exercises the broken UCC/UCX pairing.

File: The RUN if [ "${BUILD_VLLM_ELASTIC_TEST}" = "true" ]; then ... uv pip install --editable "${VLLM_ELASTIC_TEST_DIR}" ... step in the DL EP Dockerfile added under .gitlab/ (step 36 of the image build).

Suggested fix: Resolve the HPC-X UCC/UCX ABI mismatch in the base image rather than papering over it. Concretely: ensure the UCC and UCX from the same HPC-X package are used together — e.g. export LD_PRELOAD=/opt/hpcx/ucx/lib/libucs.so.0 (or add /opt/hpcx/ucx/lib to LD_LIBRARY_PATH) so torch loads the matching UCX that exports ucs_config_doc_nop, or rebuild/align the HPC-X (v2.50) UCC against UCX 1.21.0 in the pytorch26.06-cuda13.3 image. If the EP tests don't need UCC/HPC-X during the torch import, set UCX_TLS/disable UCC loading for the vLLM build step. The ldd/ucx_info diagnostics already added in 0c37adda confirm the mismatch and should guide picking the correct HPC-X component paths.

Related: PR #2123 (#2123) — the PR under test that introduced this step.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit 065a923a

TL;DR: The TSAN sanitizer gtest suite failed on a single deterministic test, MDManagerEtcdFixture.InvalidateLocalRemovesRemote: after invalidateLocalMD() returned success over the ETCD metadata-manager path, the peer still resolved the source agent's metadata for the full 3 s poll window instead of getting NIXL_ERR_NOT_FOUND. The fix is in the metadata manager's ETCD/KV invalidation path — invalidation must actually remove the published/remote-resolvable key, not in the test or the CI time limit.

Full analysis

Summary: nixl-ci-test-sanitizers #811 (TSAN variant) failed in the "Test Sanitizer" stage because the meson gtest suite had 1 failing test, MDManagerEtcdFixture.InvalidateLocalRemovesRemote.

Root cause: In the ETCD/KV metadata-manager path, invalidateLocalMD(nullptr) returns NIXL_SUCCESS but the destination agent's checkRemoteMD keeps returning NIXL_SUCCESS (metadata still resolvable) rather than NIXL_ERR_NOT_FOUND. waitForRemoteMD polls for 3 s at 25 ms intervals (md_manager.cpp:78-92), so this is a genuine functional bug, not a race or a wall-clock kill (the suite ran to completion in ~89 s). The near-identical P2P test MDManagerFixture.SendLocalAndInvalidateLocal passed, so the removal-on-invalidate semantics are correct for the P2P/listener backend but broken for the ETCD KV backend. The test was newly added by the metadata-manager work landed two days prior.

Implicated commit: 2ff0105f — "Metadata manager pr5 (#1930)", author aschwartz12 (the metadata-manager ETCD path, PRs #1745/#1907/#1930).

File: test/gtest/md_manager.cpp:298 (assertion); root cause in the ETCD KV backend invalidation logic behind nixlMDManager::invalidateLocalMD (not the test).

Suggested fix: Make the ETCD/KV metadata-manager invalidation path delete the agent's published metadata key (and any locally cached remote resolution) so a subsequent checkRemoteMD returns NIXL_ERR_NOT_FOUND, matching the P2P path's behavior. If the deletion is genuinely asynchronous, ensure invalidateLocalMD does not report NIXL_SUCCESS until the key removal is committed to ETCD. Verify against a live ETCD store (NIXL_ETCD_ENDPOINTS set), which is how CI exercises this fixture.

Related: PR #1930 (Metadata manager pr5), PR #1907 (make the manager the single metadata-exchange path), PR #1745 (add etcd backend for nixlMDManager).

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit c5cb6c59

TL;DR: The Setup Image stage failed while building the vLLM elastic-test Docker layer: VLLM_USE_PRECOMPILED=1 uv pip install --editable vllm --torch-backend=auto aborted with ImportError: cannot import name '_TensorIterator' from 'torch._C'. The precompiled torch wheel that uv resolved has a Python layer newer than its compiled C extension; pin the vLLM/torch versions instead of using floating precompiled/auto resolution.

Full analysis

Summary: Docker image build (stage 37, nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04) failed at the BUILD_VLLM_ELASTIC_TEST step while installing vLLM.

Root cause: During the editable install of vLLM (VLLM_USE_PRECOMPILED=1 ... --torch-backend=auto), vLLM's setup.py imports torch. The resolved PyTorch is internally inconsistent: torch/_native/ops/scatter_add/cutedsl_impl.py → torch/_tensor_iterator.py does from torch._C import _TensorIterator, but the installed torch._C.cpython-312-aarch64-linux-gnu.so does not export that symbol — the Python source and the compiled _C extension come from mismatched builds. This is not a NIXL code defect; it is a bad/mismatched upstream torch resolved by the unpinned --torch-backend=auto + VLLM_USE_PRECOMPILED=1 path against a floating VLLM_REF. (The Failed to initialize NumPy: No module named 'numpy' warning just before confirms the build venv is also missing numpy, but the fatal error is the torch _C import.)

Implicated commit: 33606ee9 (Lior Shapira, "CI: add VLLM+nixl_ep test to EP CI job"), reached via PR #2123 merge c5cb6c59; the vLLM install step was last touched by 04c2a44a (Lior Shapira, "CI: fix UCX version mismatch for vLLM EP").

File: The RUN if [ "${BUILD_VLLM_ELASTIC_TEST}" = "true" ]; ... layer in the nixl_ep EP Dockerfile under .ci/dockerfiles/ (STEP 36 of the image build) — the VLLM_USE_PRECOMPILED=1 uv pip install --editable ... --torch-backend=auto command and its VLLM_REF build arg.

Suggested fix: Pin the torch/vLLM combination rather than resolving them freshly at build time:

• Pin VLLM_REF to a specific released tag/commit known to work with the base image's PyTorch, and pin the torch version (e.g. install a fixed torch==<version> matching the vLLM release, or drop --torch-backend=auto in favor of an explicit index/version) so the Python layer and torch._C can't diverge.
• Also install numpy in the vLLM .venv before the editable install to clear the NumPy-init warning.
• As a quick unblock, re-pin to the previously-passing torch wheel; the current auto resolution picked up an inconsistent nightly-style torch build exporting no _TensorIterator.

Related: PR #2123 (this build); prior CI changes 33606ee, 0c37add, 04c2a44. No existing issue tracks this specific _TensorIterator import failure.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit c5cb6c59

TL;DR: The DL Python test stage failed on the single test test_nixl_api.py::test_prep_mem_view; the UCX GPU device-API prepMemView local overload rejected the VRAM memory handle (invalid memh for md_index), returning NIXL_ERR_BACKEND. This is a real defect in how NIXL builds the local device mem-list against UCX 1.22, not an infra or timeout issue.

Full analysis

Summary: nixl-ci-dl-gpu #1904 failed in stage 219 "Run DL Python tests" — pytest reported 1 failed, 24 passed, 2 skipped, the failure being test/python/test_nixl_api.py::test_prep_mem_view.

Root cause: In the spawned worker (rank 1), agent.prep_mem_view(local VRAM descs) (test line 293) reached nixlUcxEngine::prepMemView(nixl_meta_dlist_t...) → nixl::ucx::createMemList → ucp_device_local_mem_list_create, which failed. UCX logged:

• ucp_device.c:248 invalid memh for md_index=7 / md_index=6
• ucp_device.c:340 failed to pack local mem list element for element=0
• ucx_backend.cpp:1502 Failed to prepare local memory view: ... Invalid parameter

The local memListElement::create (mem_list.cpp:117-121) sets only MEMH | LOCAL_ADDR and passes md->getMem().getMemh(). On this build's UCX 1.22.0 GPU Device API (built in stage 166) on the GB200 node, that memh does not carry a valid device-capable md entry for the selected transport, so UCX rejects it as invalid memh. The later cuDevicePrimaryCtxGetState ... error code 4 (CUDA_ERROR_DEINITIALIZED) is a teardown symptom after the worker already errored, not the cause. The test's own comment documents this UCX-version sensitivity ("UCX v1.21.x tolerated it, newer UCX rejects it").

Implicated commit: The test + Python binding was added in [REDACTED:Hex High Entropy String] "BINDINGS/PYTHON: Expose prepMemView (local + remote overloads)" (#1715, x41lakazam). The local device-mem-list construction lives in mem_list.cpp (last substantively touched by Raul Akhmetshin, 1f64b71c/8181319e). The failing behavior surfaced for PR #2123 under UCX 1.22.

File: src/plugins/ucx/mem_list.cpp:117-121 (local memListElement::create, packing element.memh); triggered via src/plugins/ucx/ucx_backend.cpp:1493-1505 and test/python/test_nixl_api.py:293.

Suggested fix: Investigate why the local VRAM memh produces invalid memh for md_index under UCX 1.22 — the registered memory handle needs a device-capable md entry (matching the cuda_ipc lane) for ucp_device_local_mem_list_create. Confirm the buffer is registered with the correct backend/device flags before prepMemView, and ensure the memh md index passed to the device mem-list is one supported by the device API. If this is a known UCX-1.22 incompatibility with no NIXL-side fix yet, guard/skip test_prep_mem_view's local overload on UCX ≥ 1.22 (extend the existing skipif around HAVE_UCX_GPU_DEVICE_API) until the backend supports it, rather than letting it hard-fail CI.

Related: PR #1715 (adds prep_mem_view + this test); no existing issue matches the invalid memh signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a7b11119-111c-45f2-9b56-af9084ce30ae in the triage console for the audit trail.

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 52d8dde7

TL;DR: The Setup Image stage failed while building the DL EP Docker image because uv pip install --editable vllm --no-build-isolation couldn't build vLLM — its setup.py imports setuptools_rust, which isn't installed in the build environment. Add setuptools_rust (and setuptools) to the environment before the --no-build-isolation install.

Full analysis

Summary: Docker image build for the DL EP tests failed installing vLLM from source (stage 37 – Setup Image .../nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04).

Root cause: In the Dockerfile step guarded by BUILD_VLLM_ELASTIC_TEST, vLLM is installed with uv pip install --editable ... --no-build-isolation. With build isolation disabled, uv does not provision the PEP 518 build requirements, so vLLM's setup.py fails at import time:

ModuleNotFoundError: No module named 'setuptools_rust'
...
× Failed to build `vllm @ file:///home/svc-nixl/vllm`
  ╰─▶ Call to `setuptools.build_meta.prepare_metadata_for_build_editable` failed
subprocess exited with status 1
Error: building at STEP "RUN if [ "${BUILD_VLLM_ELASTIC_TEST}" = "true" ]; then ...": exit status 1
script returned exit code 1

The newly pinned VLLM_REF requires the Rust toolchain integration (setuptools_rust) at build time, but the base image's Python environment doesn't have it, and --no-build-isolation means it must be pre-installed. This is a build-recipe gap, not a code defect in nixl and not an infra failure.

Implicated commit: unknown — the failing RUN step lives in the DL EP Dockerfile/CI recipe (the .ci/scripts / build-helper image tree). The commit under test (52d8dde) triggered the rebuild; the missing dependency is inherent to the vLLM install step + the pinned VLLM_REF. No single nixl commit in the searched history clearly introduced it.

File: The Dockerfile RUN block containing VLLM_USE_PRECOMPILED=1 "${HOME}/.local/bin/uv" pip install --python .../python --editable "${VLLM_ELASTIC_TEST_DIR}" --torch-backend=auto --no-build-isolation (DL EP base image Dockerfile; the exact path was removed on-image via sudo rm -rf /.gitlab/build.sh /.ci/scripts/ at STEP 35).

Suggested fix: Pre-install the vLLM build requirements into the venv before the --no-build-isolation editable install. Concretely, add a step right after uv venv:

"${HOME}/.local/bin/uv" pip install --python "${VLLM_ELASTIC_TEST_DIR}/.venv/bin/python" \
    setuptools setuptools_rust wheel

Alternatively, use uv's extra-build-dependencies ([tool.uv.extra-build-dependencies] vllm = ["setuptools_rust"]) or drop --no-build-isolation so PEP 518 requirements are resolved automatically. Pinning VLLM_REF to a commit whose build-system.requires already lists setuptools_rust would also resolve it.

Related: none found (search_issues_prs returned no matches for the setuptools_rust/vLLM build error).

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 9a4c9bea

TL;DR: The nixl-ci-dl-gpu-ep-base Docker image build failed while setting up the new vLLM elastic test: uv pip install --editable vllm aborts because the installed torch==2.13.0 is broken (ImportError: cannot import name '_TensorIterator' from 'torch._C'). Pin torch to a real, working release (the pinned 2.13.0 / --torch-backend=cu130 combination does not resolve to a valid PyTorch build).

Full analysis

Summary: Stage #37 "Setup Image … base-pytorch26.06-cuda13.3" failed with exit status 1 during the BUILD_VLLM_ELASTIC_TEST container build step.

Root cause: The new vLLM EP test step installs torch==2.13.0 (--torch-backend=cu130), then runs uv pip install --editable vllm. vLLM's editable build invokes setup.py, which imports torch; that import fails: ImportError: cannot import name '_TensorIterator' from 'torch._C' (…/torch/_C.cpython-312-aarch64-linux-gnu.so), preceded by Failed to initialize NumPy: No module named 'numpy'. The torch 2.13.0 wheel that was resolved is inconsistent — its Python layer (torch/_native/…) references a C-extension symbol _TensorIterator the compiled .so doesn't export — so the whole build backend call fails and the Docker build (and thus the image stage) aborts. This is a hard build error, not a wall-clock/hang issue (activity is continuous right up to the failure).

Implicated commit: 9d4faf11 — "CI: build NIXL EP with the vLLM Torch" by Lior Shapira (part of PR #2123, merge 9a4c9bea).

File: The BUILD_VLLM_ELASTIC_TEST RUN step in the DL/EP Dockerfile (.gitlab/), specifically the lines uv pip install "torch==2.13.0" --torch-backend=cu130 followed by uv pip install --editable "${VLLM_ELASTIC_TEST_DIR}".

Suggested fix:

1. Pin torch to a valid, released version compatible with the target CUDA (e.g. a real torch==2.x cu130 wheel that actually exists and imports cleanly) rather than 2.13.0, which resolves to a broken/inconsistent wheel here. Verify with a quick python -c "import torch" before the editable install.
2. Add numpy to the venv before building vLLM (the Failed to initialize NumPy warning shows it's missing from the fresh uv venv), and use vLLM's own requirements/pinned torch instead of an independent torch pin so the versions stay consistent.
3. As a guard, run git -C "$VLLM_ELASTIC_TEST_DIR" apply and the torch install, then a smoke import torch step, so the failure surfaces earlier with a clearer message.

Related: PR #2123 (#2123) — "CI: add VLLM+nixl_ep test to EP CI job"; predecessor CI commits 04c2a44a (UCX version fix) and 0c37adda (diagnose vLLM EP image dependencies).

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 9a4c9bea

TL;DR: The Run DL Python tests stage failed because test/python/test_nixl_api.py::test_prep_mem_view errored — UCX rejected the local device mem-list with "invalid memh for md_index=6 … Invalid parameter" on the GB200 node. This is a failure in the UCX GPU-device-API path (recently added test PR #1715), not in PR #2123's CI-only change.

Full analysis

Summary: test_prep_mem_view failed in the DL Python test stage; a spawned worker raised nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND from prep_mem_view.

Root cause: In the worker, the local agent.prep_mem_view(...) (test line 293) called nixlUcxEngine::prepMemView → nixl::ucx::createMemList, which UCX rejected: ucp_device.c:248 invalid memh for md_index=6 → failed to pack local mem list element for element=0 → failed to create local mem list handle: Invalid parameter, surfaced at ucx_backend.cpp:1502 as NIXL_ERR_BACKEND. The VRAM memory handle registered for the buffer had no valid device memh for the memory-domain index UCX tried to pack, and a follow-on cuda_ctx.c:23 cuDevicePrimaryCtxGetState … unrecognized error code 4 shows the CUDA context on the GB200 node was in a bad/unloading state. This is a defect/environment mismatch in the UCX GPU-device-API prep_mem_view path, not in the CI change under test (#2123, a CI-only YAML/test addition). The test is skipped unless built against a UCX with HAVE_UCX_GPU_DEVICE_API and ≥2 GPUs, so it only exercises on nodes like this one.

Implicated commit: [REDACTED:Hex High Entropy String] (x41lakazam, PR #1715 "BINDINGS/PYTHON: Expose prepMemView (local + remote overloads)") — introduced the test and API path; the failure is not attributable to PR #2123's commit 9a4c9bea.

File: test/python/test_nixl_api.py:293 (local prep_mem_view call) → src/plugins/ucx/ucx_backend.cpp:1498-1502 (prepMemView local overload / nixl::ucx::createMemList).

Suggested fix: Investigate nixl::ucx::createMemList handling of the VRAM registration when packing the local device mem-list — the "invalid memh for md_index=6" means the registered nixlUcxMem lacks a device-registered handle for that MD index; ensure the VRAM buffer is registered with a UCX memh that carries a valid device (GDR/cuda) memh before prepMemView packs it, and confirm the node's CUDA/UCX GPU-device stack is healthy (the cuDevicePrimaryCtxGetState … error code 4 indicates a broken CUDA context). Since PR #2123 doesn't touch this code, re-run to check for flakiness/node-specific state; if it reproduces, file against the prepMemView path owner rather than blocking #2123.

Related: PR #1715 (introduced prep_mem_view + the test); PR #2123 (the CI-only change under test).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id ac476014-0fbd-4218-a24d-071f181b5f99 in the triage console for the audit trail.

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit d74ea039

TL;DR: The test_prep_mem_view Python test failed because UCX rejected the local memory handle (invalid memh for md_index=6 → NIXL_ERR_BACKEND) when building a device mem-list via prep_mem_view on the GB200 node; PR #2123 (a CI-only change) doesn't touch this code path, so this is a pre-existing/environmental failure in the UCX GPU device API, not caused by the PR.

Full analysis

Summary: Stage "Run DL Python tests" failed — test/python/test_nixl_api.py::test_prep_mem_view raised nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND from the spawned worker (rank 0).

Root cause: In the worker, agent.prep_mem_view(...) for the local VRAM buffer (test_nixl_api.py:293) called into the UCX backend's local overload (nixlUcxEngine::prepMemView → nixl::ucx::createMemList, ucx_backend.cpp:1493-1504). UCX failed to build the device memory list: ucp_device.c:248 invalid memh for md_index=6 → failed to pack local mem list element for element=0 → failed to create local mem list handle: Invalid parameter, which createMemList turns into NIXL_ERR_BACKEND. A related cuDevicePrimaryCtxGetState ... unrecognized error code 4 appeared just before, indicating the memory handle registered for VRAM was not valid/usable for the UCX GPU device API on this GB200 node. The test completed in 23.53s with continuous output (no hang/timeout). Critically, PR #2123 is a CI-only change ("add VLLM+nixl_ep test to EP CI job") and does not modify the UCX backend, the Python bindings, or this test — so the PR's changes did not introduce this failure; it is a pre-existing bug/environment issue in the UCX GPU device-API path.

Implicated commit: Not the PR's commit (d74ea03 is CI-only). The affected feature was introduced by [REDACTED:Hex High Entropy String] (x41lakazam, "BINDINGS/PYTHON: Expose prepMemView (local + remote overloads)", #1715); the UCX backend path lives in ucx_backend.cpp (recent changes: 79cc687f, e72eb50b by Ilia Yastrebov). Root-cause commit unconfirmed.

File: src/plugins/ucx/ucx_backend.cpp:1493-1504 (local prepMemView overload) via nixl::ucx::createMemList; triggered from test/python/test_nixl_api.py:293.

Suggested fix: This is a UCX-backend / device-API failure unrelated to PR #2123, so re-run CI to confirm it's not a transient node/environment issue on gb200-nvl4-ts2-102. If it reproduces, investigate createMemList and how the VRAM registration handle is passed to the UCX device mem-list (the invalid memh for md_index=6 plus cuDevicePrimaryCtxGetState error code 4 suggests a stale/invalid CUDA context or memory handle for the local buffer). As an interim measure to unblock unrelated PRs, consider gating test_prep_mem_view more strictly (e.g. verify the CUDA context/GPU device-API readiness) or marking it xfail on this hardware until the UCX device-API handle bug is fixed. Ping the UCX-backend owners (Ilia Yastrebov) and the prepMemView author (x41lakazam).

Related: PR #1715 (introduced prepMemView bindings/test); PR #2123 (build under test, CI-only). No existing issue found for the invalid memh/NIXL_ERR_BACKEND failure.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 47e50df9-e6bf-4f0a-b14b-1dcb1885822d in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit d74ea039

TL;DR: The DL-EP Docker build fails compiling the UCCL p2p bindings because nanobind (Python module + headers) is missing; it's only pip-installed in the PRE_INSTALLED_ENV=false path, which this PRE_INSTALLED_ENV=true job skips, while the UCCL build (gated by the separate PRE_INSTALLED_UCX_ENV) still runs. Ensure nanobind is available in this path — install it before the UCCL build or bake it into the base image.

Full analysis

Summary: Stage "Compiling NIXL EP Docker Image for DL" (build_helper_dl_ep/aarch64) failed at STEP 15 while running .gitlab/build.sh — the UCCL p2p make aborted with ModuleNotFoundError: No module named 'nanobind' and fatal error: nanobind/nanobind.h: No such file or directory.

Root cause: .gitlab/build.sh pip-installs nanobind only inside the if [ -z "$PRE_INSTALLED_ENV" ] package-install block (lines 165–170). This job is invoked with --build-arg PRE_INSTALLED_ENV=true (confirmed in the log: "PRE_INSTALLED_ENV is set, skipping package installation"), so that block is skipped. However the UCCL p2p build at lines 392–401 is gated by a different flag (PRE_INSTALLED_UCX_ENV, which is unset) and runs anyway. The pre-installed base image nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04:[REDACTED:Hex High Entropy String] does not provide nanobind, so UCCL's make -j16 fails immediately. Not a hang/timeout — the log shows continuous activity and the error appears seconds after the clone.

Implicated commit: unknown — this is a pre-existing flag-guard mismatch in .gitlab/build.sh (UCCL build introduced independently of the nanobind install), not a specific recent regression. The most recent touch, 90cce46 (ovidiusm, "Disable cmake install skip based on mtime"), is unrelated.

File: .gitlab/build.sh:170 (nanobind install, skipped) vs .gitlab/build.sh:392-401 (UCCL p2p build that requires it)

Suggested fix: Make nanobind available on the PRE_INSTALLED_ENV=true path that still builds UCCL. Either (a) move the nanobind pip install out of the PRE_INSTALLED_ENV block so it always runs before the UCCL build (e.g. pip3 install --break-system-packages nanobind just inside the if $HAS_GPU && test -d "$CUDA_HOME" block before cd p2p), or (b) add nanobind to the DL-EP base image so pre-installed environments ship it. Option (a) is the smaller, self-contained fix.

Related: none found.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a3fa61a3-86ab-4f3b-9632-0e9230c93110 in the triage console for the audit trail.

[lishapira]

/build

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 9c13a7a7

TL;DR: The DL-EP Docker build fails because .gitlab/build.sh deletes the meson native-file immediately after meson setup, but ninja's automatic "Regenerating build files" step re-reads that path and dies with FileNotFoundError: /tmp/tmp.EijRzxSc7w. Keep the native file alive until the build finishes (or make it stable/non-temporary).

Full analysis

Summary: Stage "Compiling NIXL EP Docker Image for DL" (#184) failed: ninja -C nixl_build aborted during its "Regenerating build files" reconfigure because the meson native-file was already removed.

Root cause: In build.sh, NIXL_PYTHON_NATIVE_FILE is created and passed to meson setup --native-file, then deleted on lines 456–458 before ninja runs. Meson persists the native-file path into the build directory and re-reads it whenever it regenerates build files (ninja's [0/1] Regenerating build files step). Because the file no longer exists, meson's reconfigure throws FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp.EijRzxSc7w', ninja fails with error: rebuilding 'build.ninja': subcommand failed, and the subsequent wheel copy (cp nixl_build/.../nixl-*.whl dist/) also fails.

Implicated commit: 361e8608 — "CI: isolate vLLM Python to the DL EP Meson build" by Lior Shapira (merged as 9c13a7a7), part of PR #2123.

File: .gitlab/build.sh:456-459 (native-file deleted at 456–458, then used again implicitly by ninja at 459).

Suggested fix: Do not delete the native file until after the build/install completes. Either move the rm -f "${NIXL_PYTHON_NATIVE_FILE}" to after line 459 (ninja ... build && ninja ... install), or better, use a stable path inside the build tree (e.g. ${NIXL_BUILD_DIR}/nixl-python-native.ini) that persists for the lifetime of the build directory so meson reconfigures succeed. Optionally add --reconfigure-safe handling by keeping the native file for the whole build.sh run and cleaning it in a trap at exit.

Related: PR #2123 (this change); prior touch commit #2097 (mtime install change) is unrelated.

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 878c7d1b

TL;DR: The C++ gtest suite in the "Run CPP tests" stage died with websocket: close 1006 (abnormal closure): unexpected EOF right after the etcd-backed test MetadataExchangeTestFixture.EtcdSendLocalAndFetchRemote (43 s), an infrastructure/flaky exec-stream drop unrelated to PR #2123. Re-run the job; if it recurs, harden the etcd MetadataExchange tests (isolate the etcd namespace/endpoint so concurrent CI builds on the same node don't collide).

Full analysis

Summary: "Run CPP tests" stage (node 219) failed — the gtest process's container exec stream closed abnormally during the etcd metadata-exchange tests.

Root cause: The gtest run was progressing normally (continuous output, no >2-min gaps) through the MetadataExchangeTestFixture cases. Immediately after [15/237] EtcdSendLocalAndFetchRemote (43464 ms) completed at 09:05:13, the container streaming session dropped at 09:05:35 with Error: error streaming command in container: ... websocket: close 1006 (abnormal closure): unexpected EOF and the script exited 1. This is an abnormal loss of the exec/websocket stream to the container, not a test assertion failure and not a wall-clock timeout. Infra telemetry rules out a node problem: slurm job 83965 stayed RUNNING on mizu01 (no CANCELLED/FAILED, no OOM), and slurm_nodes_local shows mizu01 healthy across 09:00–09:10 (state mix, ~1 TB free of 2 TB, CPU load ~4–6, empty drain reason). Notably a second CI build (job 83943, nixl-ci-gpu-3275) was co-scheduled on the same node mizu01 during the exact window — two builds sharing the host and the same etcd store, a common source of flakiness in the etcd-dependent tests. The change under test, PR #2123, only edits EP-CI-job config and cannot cause a C++ gtest failure, so this is an environmental/flaky failure rather than a code defect.

Implicated commit: None in PR #2123 (878c7d1) — unrelated CI-config change. The heavy etcd tests were most recently reworked in #1930 "Metadata manager pr5" (aschwartz12) and touched by #2130 (kerrg, 2026-08-22), but no commit is proven at fault.

File: test/gtest/metadata_exchange.cpp (MetadataExchangeTestFixture.EtcdSendLocalAndFetchRemote); failure surfaced in the "Run CPP tests" stage (node 219).

Suggested fix: Re-run the "Run CPP tests" stage — this is a transient container exec-stream drop, not a defect in PR #2123. If it recurs: (1) ensure each CI build gets an isolated etcd namespace/endpoint (per #1408) so concurrent builds on the same node (3275 and 3276 were both on mizu01) don't interfere; and (2) consider not co-scheduling two nixl-ci-gpu builds on the same node, or giving the etcd tests their own etcd instance, to remove the cross-build contention.

Related: PR #2123 (under test, unrelated); #2130 "test: skip centralized metadata cleanup without a store"; #1930 "Metadata manager pr5"; #1408 "Allow any ETCD namespace".

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 878c7d1b

TL;DR: The nixl-ci-dl-gpu-ep-base image-build stage was force-aborted (exit 143) after hanging for ~3.5 hours during the vLLM editable/source compile inside the new BUILD_VLLM_ELASTIC_TEST Docker step; the root cause is that step building vLLM from source with MAX_JOBS=1 stalling with no output, not a legitimately slow build.

Full analysis

Summary: Stage "Setup Image aarch64/nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04/0" (ID 37) ran 3.8 h and was ABORTED with exit code 143 while building the vLLM elastic-test image.

Root cause: A hang, not a wall-clock overrun. The stage log shows continuous progress up to 08:46:05 ("Downloaded nvidia-cutlass-dsl-libs-cu12"), which is the last package download of the uv pip install --editable "${VLLM_ELASTIC_TEST_DIR}" --no-build-isolation step (run with TORCH_CUDA_ARCH_LIST=10.0 MAX_JOBS=1). After that line there is no application output for ~3.5 hours — a brief agent channel blip at 09:23:29 ("seems to be removed or offline … is back online") and then nothing until Jenkins sent SIGTERM at 12:04:54 ("Sending interrupt signal … script returned exit code 143 / Aborted by danielpr"). The largest inter-line gap is essentially the entire runtime, so the vLLM source build (single-job compile of the vLLM C++/CUDA extensions on aarch64) hung/stalled rather than making progress. Exit 143 and the abort are symptoms of that hang.

Implicated commit: The BUILD_VLLM_ELASTIC_TEST image step was introduced/iterated in a recent cluster by Lior Shapira — 33606ee9 "CI: add VLLM+nixl_ep test to EP CI job", 9d4faf11 "CI: build NIXL EP with the vLLM Torch", 361e8608, 79badca9. This build ran on merge commit 878c7d1b (lishapira). One of these introduced the long, output-silent source-build step that hangs.

File: The Dockerfile RUN if [ "${BUILD_VLLM_ELASTIC_TEST}" = "true" ]; then … uv pip install --editable "${VLLM_ELASTIC_TEST_DIR}" … --no-build-isolation … fi step under .gitlab/ (path shown inline in the stage log for build image STEP 36/39).

Suggested fix: Treat this as a hung build step, not a timeout to be raised. Concretely: (1) add progress/verbose output and a per-step timeout to the vLLM editable build so a stall fails fast instead of consuming the whole stage (e.g. wrap in timeout and set VERBOSE=1/uv … -v); (2) revisit MAX_JOBS=1 for the aarch64 vLLM compile — a single-job source build of vLLM+CUDA extensions is extremely long and appears to have stalled (possible OOM-thrash or deadlock during compilation); bump job count within the node's memory budget or use a prebuilt vLLM wheel; (3) confirm on a node whether the compile genuinely deadlocks or is thrashing. Do not simply increase the stage time limit.

Related: PR #2123 (this build); CI commits 33606ee9, 9d4faf11, 361e8608, 79badca9, 0c37adda "CI: diagnose vLLM EP image dependencies", 04c2a44a "CI: fix UCX version mismatch for vLLM EP".

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 878c7d1b

TL;DR: The Setup Image stage for the new EP CI base image was killed by Jenkins after hitting the pipeline timeout (exit 143); the real cause is that the vLLM editable build added by PR #2123 hung for ~5.5 hours (no output from 12:35:17 to 18:07:27) while compiling vLLM with MAX_JOBS=1. Fix the vLLM build step so it makes progress within the timeout (raise MAX_JOBS, add build logging/timeout) rather than raising the pipeline limit.

Full analysis

Summary: Docker base-image build stage nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04/0 was ABORTED by a Jenkins timeout during the new vLLM elastic-test image build (Dockerfile STEP 36/39).

Root cause: A hang, not slow progress. Timestamps show continuous activity through 12:35:17.961Z (uv finished downloading vLLM's build dependencies, Building vllm @ file:///home/svc-nixl/vllm), then a single ~5.5-hour gap with zero output until 18:07:27.988Z Sending interrupt signal to process / script returned exit code 143 / Timeout has been exceeded. The process silently stalled at the very start of the vLLM editable compile step — TORCH_CUDA_ARCH_LIST=10.0 MAX_JOBS=1 uv pip install --editable ${VLLM_ELASTIC_TEST_DIR} --torch-backend=cu130 --no-build-isolation. Serializing the entire vLLM CUDA compile with MAX_JOBS=1 makes it exceed the wall clock (and it produces no incremental output, so it presents as a hang). This step was newly introduced by PR #2123 (BUILD_VLLM_ELASTIC_TEST=true block).

Implicated commit: [REDACTED:Hex High Entropy String] (PR #2123, lishapira) — "CI: add VLLM+nixl_ep test to EP CI job", which adds the vLLM elastic-test image build.

File: The BUILD_VLLM_ELASTIC_TEST RUN block in the EP CI base Dockerfile (STEP 36/39 in the build log), containing the MAX_JOBS=1 uv pip install --editable "${VLLM_ELASTIC_TEST_DIR}" line. (Exact path is in PR #2123's Dockerfile under the EP CI contrib/docker directory; not resolvable from the truncated log path.)

Suggested fix: In the new vLLM build step, stop serializing the compile: raise MAX_JOBS to the available core count (the build node reports 128 CPUs) or a bounded value like 16–32, so the compile finishes within the stage timeout. Additionally, add periodic build output (e.g. VERBOSE=1/uv --verbose, or a timeout/heartbeat wrapper) so a genuine stall is distinguishable from slow compilation, and consider a dedicated, longer timeout for this image-build stage only if the parallel build still legitimately needs more time. Do not simply raise the pipeline timeout — the step produced no progress for 5.5 hours, indicating a stall.

Related: PR #2123 — #2123

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 4778f44c-ef87-447a-bd5b-2b2c72129aca in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 4c121565

TL;DR: The "Run DL NIXL EP tests" stage was killed by an outer pipeline timeout after ~2.5 min of total silence; the last line was pyxis: importing docker image: ...nixl-ci-dl-gpu-ep-test:825, so the srun/pyxis container-image import hung and never started the test — this is a hang, not a slow test.

Full analysis

Summary: Stage 219 (Run DL NIXL EP tests) aborted with exit code 143 while srun was still importing the freshly-pushed test container image via pyxis; the process produced no output for the entire window before being terminated.

Root cause: The srun --container-image=".../nixl-ci-dl-gpu-ep-test:825" invocation hung inside the pyxis image import on the compute node (gb-nvl-118-compute02). Evidence: the image was pushed to Artifactory at 18:14:28, srun launched at 18:14:44, and the last application line is pyxis: importing docker image at 18:14:47 — after which there is zero output for ~2.5 minutes until the outer timeout fired ("Cancelling nested steps due to timeout" → SIGTERM → exit 143). The test script itself (.gitlab/test_ep.sh) never ran — none of its set -x output, "Show system info", or env dump appears — so this is a hang during container import (pyxis pulling/unpacking the ~80-layer image from Artifactory), not the elastic/vLLM tests being slow. Note the stage's own 3-hour timeout did not trip; the kill came from a shorter outer timeout wrapping the parallel branch, so the import stall was fatal well before the intended budget.

Implicated commit: unknown — this is an infrastructure/environment hang (pyxis import from Artifactory onto the GB200 node), not a code defect in commit 4c12156.

File: .gitlab/test_ep.sh:1 (never reached); hang is in the Jenkins slurm.run step's srun --container-image=... pyxis import, invoked from the Run DL NIXL EP tests stage.

Suggested fix:

1. This is a transient pyxis/registry import hang — retry the build; the image is already built and pushed, so a re-run will likely import cleanly.
2. To make it self-healing, wrap the pyxis import so a stall fails fast instead of eating the whole outer timeout: pre-import the image with a bounded step (e.g. srun … --container-image=… true under a timeout 600 with retries) before the test srun, or add --container-save/enclave pre-pull, so a hung import surfaces as a quick, clear failure rather than a wall-clock abort.
3. If it recurs on the same node, have infra check gb-nvl-118-compute02's pyxis/enroot cache and its Artifactory connectivity (large multi-layer import stalling mid-pull).
   Do not raise the timeout — the log shows no progress at all during the silence, so this was a hang, not a job that needed more time.

Related: none (no matching issue/PR found).

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 05f7f908

TL;DR: The nixl-ci-dl-gpu-ep #825 pipeline hit its overall wall-clock timeout (exit 143) 2.5 min into the test stage; the real cause is the "Setup Image" stage spending 292 minutes building vLLM from source, exhausting the pipeline's time budget before the tests could run.

Full analysis

Summary: The pipeline was killed by its top-level timeout ("Timeout has been exceeded", exit 143) during pyxis image import in the "Run DL NIXL EP tests" stage — but that stage had barely started; the budget was consumed earlier.

Root cause: Legitimate but excessively long setup work, not a hang. The "Setup Image …pytorch26.06-cuda13.3…" stage ran ~5.58 hrs, dominated by a single step: Prepared 163 packages without build isolation in 292m 29s (building vLLM from source with its CUDA/CUTLASS kernels), followed by a multi-GB base+PR image build and push (~17:39→17:52). The log shows continuous activity with no multi-minute silent gaps, so this is genuine slowness. By the time the allocation (slurm job 1928292 on gb-nvl-118-compute02) and test stage began at 18:14, the outer pipeline timeout had almost no time left and fired at 18:17 while pyxis was still importing the test image — the test script never produced its first line (==== Show system info ====).

Implicated commit: PR #2123 — the vLLM-from-source EP integration; 33606ee9 "CI: add VLLM+nixl_ep test to EP CI job" and 9d4faf11 "CI: build NIXL EP with the vLLM Torch" (both Lior Shapira).

File: .gitlab/test_ep.sh (the vLLM-based test path) and the nixl-ci-dl-gpu-ep pipeline setup that builds vLLM from source; the 292-minute build step is the cost driver.

Suggested fix: Don't build vLLM from source inside the pipeline's timed window. Prebuild/cache the vLLM+CUDA/CUTLASS wheel (or the base image with vLLM already installed) so setup is a fast install rather than a ~5-hour compile, and/or move the Setup Image stage outside the wall-clock timeout that also covers allocation and tests. Raising the timeout alone is not advised — a ~5 hr from-source vLLM build on every run is the underlying problem to eliminate. As a guard, have test_ep.sh fail fast if setup consumed the budget so failures surface in the test stage rather than as a pyxis-import kill.

Related: PR #2123 (#2123); related setup commits 361e8608, 9d4faf11, 0c37adda, 33606ee9.

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 9c52ce79

TL;DR: The build succeeded; the "Run DL Python tests" stage failed on a single test, test_prep_mem_view, when UCX's ucp_device_local_mem_list_create rejected the local memory handle ("invalid memh for md_index=7"). This is a UCX GPU-device-API / memory-registration issue on the GB200 node, not a defect introduced by the PR under test.

Full analysis

Summary: pytest test/python/test_nixl_api.py::test_prep_mem_view failed (1 failed, 24 passed, 2 skipped) — a spawned worker raised nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND from agent.prep_mem_view(...).

Root cause: In the local prep_mem_view path, nixl::ucx::createMemList(nixl_meta_dlist_t...) builds a ucp_device_mem_list_elem_t from the registered buffer's memory handle (mem_list.cpp:119, element.memh = md->getMem().getMemh()), then calls ucp_device_local_mem_list_create. UCX rejected it: ucp_device.c:248 UCX ERROR invalid memh for md_index=7 / md_index=6 → failed to create local mem list handle: Invalid parameter, surfaced as Failed to prepare local memory view: ... Invalid parameter (ucx_backend.cpp:1502) and mapped to NIXL_ERR_BACKEND (line 1503). Both ranks then hit cuDevicePrimaryCtxGetState ... unrecognized error code 4 (CUDA_ERROR_DEINITIALIZED) during teardown. This is a UCX-1.22.0 / GPU-device-API memory-handle interaction on the GB200 node (gb200-nvl4-ts2-69) — the registered VRAM memh lacks a device key for the MD index the cuda_ipc device path expects. It is not related to the change under test: PR #2123 only adds a VLLM+nixl_ep CI test and touches no UCX/device-API code.

Implicated commit: Not a code regression from this PR. The test itself was introduced by [REDACTED:Hex High Entropy String] (x41lakazam, PR #1715 "Expose prepMemView"); the UCX device mem-list path is in mem_list.cpp (last touched 6ad0cea). The failure is environmental (UCX/driver on the GB200 test node), not caused by commit 9c52ce7.

File: src/plugins/ucx/mem_list.cpp:119 (local memListElement::create, element.memh) → createMemList at line 184-195; error surfaced at src/plugins/ucx/ucx_backend.cpp:1502.

Suggested fix: This should not gate PR #2123. Investigate the UCX GPU-device-API local mem-list on the GB200 node: confirm the UCX 1.22.0 build's ucp_device_local_mem_list_create accepts the memh produced for a cuda/VRAM registration (the "invalid memh for md_index" points to the memh not exposing the MD the device path selects). As a short-term measure, quarantine/skip test_prep_mem_view on this node (or add a UCX-version/MD-capability guard alongside the existing HAVE_UCX_GPU_DEVICE_API and device_count()<2 skips) and file a UCX-side ticket; re-run to confirm the rest of the suite passes. Do not treat this as a change-under-test failure.

Related: PR #1715 (introduced the test), PR #1410 / #1342 (UCX device mem-list API). No existing issue found for the "invalid memh" signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 72245942-2678-419f-8d0b-a82e69c881ca in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit d1ac16eb

TL;DR: The test_prep_mem_view Python test failed because UCX's device mem-list creation rejected the memory handle (invalid memh for md_index=7, "Invalid parameter") when building the local memory view via agent.prep_mem_view(), which NIXL surfaces as NIXL_ERR_BACKEND. This is a defect/incompatibility in the prep_mem_view (UCX GPU device API) path exercised on GB200 hardware, not an infra or timeout issue.

Full analysis

Summary: Single Python test test/python/test_nixl_api.py::test_prep_mem_view failed in the "Run DL Python tests" stage (build and all other 24 tests passed).

Root cause: In the spawned worker, agent.prep_mem_view(...) for the local VRAM buffer failed. UCX reported ucp_device.c:248 invalid memh for md_index=7 / element=0 → failed to create local mem list handle: Invalid parameter; NIXL caught the exception in nixlUcxEngine::prepMemView and returned NIXL_ERR_BACKEND (ucx_backend.cpp:1502), which the binding raised as nixl_cu13._bindings.nixlBackendError at test_nixl_api.py:293. The memory handle passed into ucp_device_mem_list lacked a valid registration for the required md_index — i.e. the device mem-list element built in mem_list.cpp (memListElement::create) is not selecting/exporting a memh valid for the GPU device API on this UCX/GB200 configuration. The subsequent cuDevicePrimaryCtxGetState ... unrecognized error code 4 lines are teardown noise after the primary failure. Build ran on gb200-nvl4-ts2-69 (4 GPUs, so the device_count() < 2 skip did not apply). This is a genuine backend/test-code defect, not a hang (test finished in 24.29s) and not infrastructure.

Implicated commit: [REDACTED:Hex High Entropy String] — "BINDINGS/PYTHON: Expose prepMemView (local + remote overloads) and re… (#1715)" by x41lakazam (introduced this test and the prep_mem_view path).

File: src/plugins/ucx/mem_list.cpp (memListElement::create, invoked via nixl::ucx::createMemList) surfacing at src/plugins/ucx/ucx_backend.cpp:1502; test at test/python/test_nixl_api.py:293.

Suggested fix: Investigate how the local mem-list element derives the UCX memh/md_index in mem_list.cpp — the memh handed to ucp_device_mem_list_create has no valid entry for the selected md_index on the GB200 NVL4 UCX build (UCX 1.22.0 with GPU device API). Verify the buffer's registration exports a device-capable memh (correct md_index for the cuda_ipc/GPU device transport) before building the list, and add a clear guard/error if it does not. As an immediate CI unblock, gate test_prep_mem_view more strictly (e.g. skip unless the specific UCX device-API mem-list capability is confirmed for the platform) so PR #2123's unrelated vLLM/EP CI change isn't blocked by this pre-existing GB200 failure.

Related: PR #1715 (introduced prep_mem_view + this test); current build is on PR #2123. No existing issue tracks this specific UCX "invalid memh" failure.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id bdb617a0-0ff1-40f6-9292-cc9fafe1fd67 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 9c52ce79

TL;DR: The "Setup Image" stage failed while building vLLM (with SM100 NVFP4/CUTLASS kernels) inside the EP-test Docker image; the cmake --build subprocess was killed by SIGKILL (returned non-zero exit status 9) — a host-OOM kill caused by too many parallel, memory-hungry nvcc/CUTLASS compilations. Lower MAX_JOBS/-j (e.g. to 2–3) or build on a higher-RAM node.

Full analysis

Summary: Stage "Setup Image …nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04/0" (#37) failed building the vLLM editable wheel during the new EP elastic-test image build.

Root cause: The Dockerfile RUN that builds vLLM runs cmake --build . -j=6 (MAX_JOBS=6) compiling extremely heavy SM100 NVFP4 CUTLASS GEMM templates (sm100_fp4_config_*, SM100_MMA_MXF4_SS, nvfp4_scaled_mm_kernels.cu). The build ends with subprocess.CalledProcessError: Command '['cmake','--build','.','-j=6', …]' returned non-zero exit status 9. Exit status 9 = the child compiler was terminated by SIGKILL, the classic host-OOM signature — six concurrent CUTLASS nvcc instantiations exhausted RAM and the OOM killer reaped a compiler. The log shows only deprecation warnings (#2908-D), no error: diagnostics, and continuous output up to the kill, so this is not a source compile error and not a hang/timeout. (A secondary latent issue: nvcc was invoked with -gencode arch=compute_89 despite TORCH_CUDA_ARCH_LIST=10.0, but that is not what killed the build.)

Implicated commit: PR #2123 "CI: add VLLM+nixl_ep test to EP CI job" (commit [REDACTED:Hex High Entropy String]) — this PR introduces the failing image-build RUN step.

File: The EP-test Dockerfile RUN block guarded by BUILD_VLLM_ELASTIC_TEST (the uv pip install --editable … MAX_JOBS=6 command shown in the log). Path is added by PR #2123 (the nixl_ep_vllm_release_test.patch / EP Dockerfile).

Suggested fix: Reduce build parallelism to fit available RAM: set MAX_JOBS=2 (and drop -j=6) in the vLLM uv pip install --editable RUN, or schedule this image build on a node/container with substantially more memory (SM100 CUTLASS/NVFP4 compiles need several GB per nvcc job). Also confirm the arch flags actually reflect TORCH_CUDA_ARCH_LIST=10.0 so you're not additionally compiling SM89 kernels you don't need. Re-run after lowering job count to verify the OOM is resolved.

Related: #2123

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 964c92d5-c88a-4840-9f57-45b01a828a32 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit d1ac16eb

TL;DR: The nixl-ci-dl-gpu-ep image build fails while compiling vLLM's CUTLASS SM100 fp4 CUDA kernels — nvcc is SIGKILLed (exit status 9), i.e. the OOM killer struck under MAX_JOBS=6/-j=6. Reduce build parallelism (and/or give the builder more RAM) for the new vLLM elastic-test compile added by PR #2123.

Full analysis

Summary: Stage "Setup Image aarch64/nixl-ci-dl-gpu-ep-base-pytorch26.06-cuda13.3-ubuntu24.04/0" failed; the podman/docker build RUN step that builds vLLM's editable wheel died with subprocess exited with status 1 after cmake --build . -j=6 ... returned non-zero exit status 9.

Root cause: Out-of-memory during parallel CUDA compilation. The vLLM build was launched with MAX_JOBS=6 and cmake --build . -j=6, compiling very memory-hungry CUTLASS SM100 NVFP4 GEMM kernels (csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_kernels.cu, .../w8a8/cutlass/scaled_mm_c2x.cu, with -DENABLE_NVFP4_SM100=1, sm100_fp4_config_*). nvcc exited with status 9 (SIGKILL) — the OOM-killer signature — while ninja was still making progress ([71/428]), so this is a memory exhaustion, not a hang or wall-clock timeout. (Note also the log shows nvcc being invoked with -gencode arch=compute_89,code=sm_89 even though TORCH_CUDA_ARCH_LIST=10.0 was requested — an arch-list mismatch worth checking, but the fatal event is the SIGKILL/OOM.)

Implicated commit: [REDACTED:Hex High Entropy String] (PR #2123, "CI: add VLLM+nixl_ep test to EP CI job") — this PR introduces the BUILD_VLLM_ELASTIC_TEST RUN step that performs this compile.

File: The RUN if [ "${BUILD_VLLM_ELASTIC_TEST}" = "true" ]; then ... TORCH_CUDA_ARCH_LIST=10.0 MAX_JOBS=6 ... uv pip install --editable "${VLLM_ELASTIC_TEST_DIR}" ... step in the Dockerfile added by PR #2123 (the base image nixl-ci-dl-gpu-ep-base-... build recipe).

Suggested fix: Lower the compile parallelism so peak RAM stays within the builder's limits — set MAX_JOBS to 2–3 (and drop --threads accordingly) for the vLLM editable build, or run the build on a builder with substantially more memory. Because CUTLASS SM100 fp4 units are the heaviest, even -j=6 on an aarch64 build node can exceed available RAM; start at MAX_JOBS=2 and raise only if it stays under memory. Separately, verify the CUDA arch actually passed to nvcc (compute_89 seen despite TORCH_CUDA_ARCH_LIST=10.0), since compiling SM100 kernels for the wrong target both wastes memory/time and may not produce usable code.

Related: #2123

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id fb3abfed-0654-452c-b7a8-a5b644855de8 in the triage console for the audit trail.

[lishapira]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit f9000e8e

TL;DR: The container-build/nixlbench/aarch64 "Setup docker" step failed because authenticating to the GitLab container registry (gitlab-master.nvidia.com:5005) timed out with a TLS handshake timeout — a transient network/infra issue, not a code defect. Re-run the job; if it persists, investigate registry/network connectivity from the aarch64 agent.

Full analysis

Summary: The aarch64 nixlbench container "Setup docker" step aborted with exit code 125 during registry authentication.

Root cause: Transient network failure reaching the GitLab container registry: Error: authenticating creds for "gitlab-master.nvidia.com:5005": ... net/http: TLS handshake timeout. Podman logged in successfully moments earlier, then the auth/pull request to the registry could not complete its TLS handshake within the timeout. The three other build variants succeeded, confirming this is environmental to that agent/attempt rather than caused by the PR.

Implicated commit: unknown — not a code issue; the PR commit f9000e8 is not implicated (parallel variants of the same code built successfully).

File: unknown (infrastructure failure in the "Setup docker" step, not a source file).

Suggested fix: Retry the build. If the TLS handshake timeouts to gitlab-master.nvidia.com:5005 recur, investigate network/DNS/proxy connectivity and registry availability from the aarch64 CI agent, and consider adding a retry with backoff around the docker/podman login+pull in the "Setup docker" step to tolerate transient registry hiccups.

Related: none

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit dad3e90f

TL;DR: The container-build/nixlbench/aarch64 build failed in its "Setup docker" step because podman couldn't authenticate to the GitLab container registry — a TLS handshake timeout. This is a transient network/infra issue, not a code defect; retry the build.

Full analysis

Summary: Parallel stage container-build/nixlbench/aarch64 → "Setup docker" (node 111) failed with podman exit code 125.

Root cause: Podman failed to authenticate credentials for gitlab-master.nvidia.com:5005 — the auth request to https://gitlab-master.nvidia.com/jwt/auth returned net/http: TLS handshake timeout. This is a transient network/registry-reachability problem from the aarch64 build agent, not related to the PR's code. The other three parallel container builds (nixl x86_64/aarch64 and nixlbench x86_64) all completed successfully with the same commit, confirming the source is fine.

Implicated commit: none — failure is infrastructural (registry TLS handshake timeout), unrelated to commit dad3e90.

File: N/A (no source file implicated); failure is in the "Setup docker" pipeline step, log node 111.

Suggested fix: Retry the build. If aarch64 agents repeatedly hit TLS handshake timeouts against gitlab-master.nvidia.com:5005, escalate to infra to check network/proxy connectivity from those agents to the GitLab registry, and consider adding a retry with backoff around the registry login in the "Setup docker" step to tolerate transient handshake failures.

Related: none

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 4c121565

TL;DR: The Setup Image aarch64 stage was aborted after ~6h by the pipeline timeout while compiling vLLM's vllm-flash-attn CUDA kernels from source; the build wasn't hung — it was making steady progress but the newly-added from-source vLLM build on aarch64 simply doesn't fit in the wall-clock budget. Either build _vllm_fa3_C with more parallelism / a prebuilt wheel, or raise the timeout for this specific stage.

Full analysis

Summary: Jenkins job nixl-ci-dl-gpu-ep #832 — the Setup Image aarch64/...pytorch26.06-cuda13.3-ubuntu24.04 stage (ID 37) ran 21,602,858 ms (~6h) and was ABORTED with exit code 143 / "Timeout has been exceeded".

Root cause: Not a hang. The stage log shows nvcc compiling flash-attn instantiations advancing continuously — [344/428] at 10:58:07 through [400/428] at 11:38:27, each unit taking ~20–90 s, with the kill signal at 11:39:21 (largest inter-line gap well under 2 min). This is genuine, sequential CUDA compilation (--threads=1, one Hopper sm90 instantiation at a time) that overran the pipeline wall-clock limit. The overrun was introduced by the PR's own CI changes that switched to building vLLM (and its vllm-flash-attn C++/CUDA extension — 428 translation units) from source on aarch64; before these commits the image did not compile vLLM from source.

Implicated commit: daad0188 "CI: build vLLM from source on aarch64" by lishapira, plus follow-ups 8d12e125 (reduce build memory) and 05f7f908 (speed up targeted vLLM source build) — all rolled into the tested merge 4c121565 (PR #2123).

File: The vLLM source-build step in the aarch64 base-image build config under .ci/ (Dockerfile/build script added in daad0188). Exact path not resolvable via the tools here; it is whatever daad0188 added to drive the vllm from-source build seen at /home/svc-nixl/vllm.

Suggested fix: Treat this as a build-cost problem, not a flaky hang:

• Fastest: don't compile vllm-flash-attn (_vllm_fa3_C) from source in the image build — use a prebuilt vLLM/flash-attn aarch64 wheel, or set VLLM_FLASH_ATTN_SRC/FLASH_ATTENTION_SKIP_CUDA_BUILD / limit FLASH_ATTN_CUDA_ARCHS to only the arch the tests need, so hundreds of sm90 instantiations aren't built.
• Or add parallelism: the nvcc lines use --threads=1 and appear serialized; raise CMake/ninja job count (MAX_JOBS/-j) and --threads so the 428 units build concurrently.
• Only if the above are infeasible, raise the timeout for stage ID 37 specifically (the base-image build), since the remainder of the pipeline is fast.

Related: PR #2123; CI commits daad0188, 8d12e125, 05f7f908, 54d7d748 (all "build/tune vLLM source build" changes). No pre-existing issue found via search.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit f9000e8e

TL;DR: The nixl_ep native elastic tests all passed; the build failed only in the vLLM Elastic EP pytest, where all 4 test_elastic_ep_scaling cases errored at setup because the ray executable is missing from the vLLM .venv (FileNotFoundError: 'ray'). Fix: install ray into the vLLM test environment (or make the cleanup_ray_between_tests fixture tolerate a missing ray binary).

Full analysis

Summary: Stage #219 "Run DL NIXL EP tests" failed — the vLLM tests/distributed/test_elastic_ep.py::test_elastic_ep_scaling suite errored during fixture setup for all 4 parametrizations.

Root cause: The pytest fixture cleanup_ray_between_tests (test_elastic_ep.py:20) unconditionally runs subprocess.run(["ray", "stop", "--force"], ...), but the ray binary is not present in the vLLM venv (/home/svc-nixl/vllm/.venv). This raises FileNotFoundError: [Errno 2] No such file or directory: 'ray' at setup, so every test errors before running. This is an environment/provisioning gap, not a code defect in nixl itself — the native elastic.py EP tests completed successfully and the pre-check import (nixl, nixl_ep, vllm, CUDA/GPU asserts) passed. The ray dependency was almost certainly dropped by the immediately-preceding change to the venv provisioning, b430ce54 "CI: simplify vLLM EP environment setup".

Implicated commit: b430ce54 — lishapira, 2026-08-23 "CI: simplify vLLM EP environment setup" (this build runs PR #2123 merge f9000e8e on top of it).

File: vLLM tests/distributed/test_elastic_ep.py:20 (the ray stop --force call); provisioning to fix is in the DL EP image / vLLM venv setup driven by .gitlab/test_ep.sh (invokes ${VLLM_ELASTIC_TEST_DIR}/.venv/bin/python -m pytest, line 159).

Suggested fix: Install ray into the vLLM test venv when it's provisioned (e.g. uv pip install ray/add it to the EP test image), so ray stop --force in the fixture can run. The "simplify vLLM EP environment setup" change (b430ce5) evidently removed the dependency that pulled in ray; restore it. As a defensive measure, the fixture could also guard the call with shutil.which("ray"), but the primary fix is provisioning ray in the environment.

Related: PR #2150 "ci: add vLLM Elastic EP integration test"; implicated commit b430ce5.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit dad3e90f

TL;DR: The new vLLM Elastic EP pytest failed because the ray CLI isn't installed/on PATH in the test venv — the cleanup_ray_between_tests fixture raises FileNotFoundError: 'ray' at setup of all 4 parametrizations. Install ray in the vLLM venv (or guard the fixture when ray is absent).

Full analysis

Summary: Stage "Run DL NIXL EP tests" failed: tests/distributed/test_elastic_ep.py::test_elastic_ep_scaling errored (4/4 variants) during setup; native elastic.py tests all passed.

Root cause: The pytest fixture cleanup_ray_between_tests runs subprocess.run(["ray", "stop", "--force"], ...), but the ray executable does not exist in the vLLM .venv (/home/svc-nixl/vllm/.venv) / PATH used by the test. Python raises FileNotFoundError: [Errno 2] No such file or directory: 'ray' before any test body executes, so every parametrization ERRORs and srun exits with code 1. This is an environment/setup gap in the newly-added vLLM integration tests, not a NIXL runtime or hardware issue.

Implicated commit: [REDACTED:Hex High Entropy String] (branch nixl_ep_ci_add_vllm_integration_tests, PR #2123 — adds vLLM Elastic EP tests). The test file tests/distributed/test_elastic_ep.py lives in the vLLM tree (releases/v0.27.1, VLLM_COMMIT 6e448d0ea); the CI wiring that invokes it is what this PR added.

File: tests/distributed/test_elastic_ep.py:20 (fixture cleanup_ray_between_tests, subprocess.run(["ray", "stop", "--force"], ...)); invoked by .gitlab/test_ep.sh (the "Running vLLM Elastic EP test" block).

Suggested fix: Ensure ray is available to the vLLM venv the test uses. Either install it into /home/svc-nixl/vllm/.venv (e.g. add ray to the vLLM EP test container/venv install step, uv pip install ray or install vLLM with the appropriate extras), or make the fixture resilient by skipping the cleanup when ray isn't found — e.g. guard with shutil.which("ray") before calling, or wrap the subprocess.run in a try/except FileNotFoundError. Since PR #2123 is introducing these tests, the venv/image build for nixl-ci-dl-gpu-ep-test should add the ray dependency.

Related: PR #2123 (nixl_ep_ci_add_vllm_integration_tests); upstream vLLM tests/distributed/test_elastic_ep.py.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 76241301-8614-4d0d-951d-9738fffabd29 in the triage console for the audit trail.