CI: add per-PR container-build pipeline

[Alexey-Rivkin]

What?

• Adds a build-only container gate (nixl-ci-build-container-pr) to the PR CI.
• Builds the nixl (debug, EP on) and nixlbench images - one parallel cell each, x86_64 only.
• Pushes nothing - it just proves the images still build.
• Skips fast when a PR touches no build files, so it can be a required check without slowing unrelated PRs.

Why?

• Catch changes that break the container builds before they land.

How?

• build-container-pr-matrix.yaml - path-filter, then build per target, no push.
• proj-jjb.yaml - new leaf job + dispatcher fan-out entry.
• ci-overview.md - documents the new job.
• Path filter scope: contrib/, benchmark/, .ci/, meson.build/meson.options, pyproject.toml, setup.py.

Testing

• Green on a clean build (nixl EP+debug and nixlbench both built, nothing pushed).
• Correctly red on the nvlink -G register-count failure - so it blocks the real breakage, not just passes.

Follow-ups (not in this PR)

• After merge: JJB-update nixl-ci-dispatcher to turn the gate on for all PRs.
• Optional: add nixl-ci-build-container-pr to the Required PR Checks ruleset to make it block merges (like nixl-ci-gpu/non-gpu).

Summary by CodeRabbit

• New Features
  • Expanded the Jenkins CI dispatcher’s PR fan-out to include a new build-only container verification leaf job (nixl-ci-build-container-pr), triggered only after a /build comment.
  • The job runs a gated matrix for targets (nixl, nixlbench) across architectures (x86_64, aarch64), skipping when PR changes don’t affect build paths and enforcing per-cell time limits (no image pushes).
• Documentation
  • Updated CI overview and Jenkins job documentation: added the new leaf job, refreshed the flow diagram, and updated dispatcher job counts and descriptions.

[github-actions]

👋 Hi Alexey-Rivkin! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[Alexey-Rivkin]

/build

[Alexey-Rivkin]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 525d1e89

TL;DR: The Build sanity image stage failed because apt update/apt install curl git in contrib/Dockerfile.vllm couldn't reach ports.ubuntu.com (network unreachable / connection timed out), so git had no installation candidate and the build exited 100 — this cascaded into the build_helper_vllm pipeline abort. It's a network/infra failure, not a bug in the PR.

Full analysis

Summary: Docker build of the vLLM sanity image failed at the apt install -y curl git step with E: Package 'git' has no installation candidate, aborting the vLLM branch of the pipeline.

Root cause: During RUN apt update && apt install -y curl git in contrib/Dockerfile.vllm, the aarch64 build host could not connect to the Ubuntu ports mirror: repeated Cannot initiate the connection to ports.ubuntu.com:80 ... Network is unreachable and connection timed out for both IPv6 and IPv4. Because the jammy package index failed to download, apt fell back to stale/empty indexes and had no candidate for git, causing exit code 100. Stage 332 (build_helper_vllm) FAILURE is the downstream "pipline stop" reaction to this same sanity-image build failure — not an independent error. The x86_64 and SGLang paths succeeded, so this is environmental (transient ports.ubuntu.com reachability / IPv6 routing on the aarch64 node), not caused by commit 525d1e8.

Implicated commit: No code commit implicated — infra/network failure. (The Dockerfile step originates from b594eb0 / 32f3104, but those are not the fault.)

File: contrib/Dockerfile.vllm:22-25

Suggested fix:

• Primary: this is a transient network failure — re-run the build; it should pass once ports.ubuntu.com is reachable from the aarch64 node.
• To make the step resilient (recommended), harden the apt step in contrib/Dockerfile.vllm, e.g.:
  • Add retries and force IPv4: RUN apt-get -o Acquire::Retries=5 -o Acquire::ForceIPv4=true update && apt-get install -y --no-install-recommends curl git && apt-get clean && rm -rf /var/lib/apt/lists/*
  • And/or point apt at an internal/artifactory Ubuntu mirror instead of the public ports.ubuntu.com, since the CI network clearly restricts egress to it (note IPv6 routes are unreachable and public IPv4 times out).

Related: PR #1863 (this build); sanity-image pipeline introduced in #1777 and #1664. No matching existing issue found.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 525d1e89

TL;DR: The "Run Python tests" stage died mid-pytest with websocket: close 1006 (abnormal closure): unexpected EOF — the srun/enroot container stream dropped, not a NIXL test failure or hang. This is transient CI infrastructure flakiness unrelated to PR #1863 (a CI-only change); rerun the job.

Full analysis

Summary: Jenkins stage 337 "Run Python tests" failed when the container command stream closed abnormally in the middle of pytest -s test/python.

Root cause: The scctl ... srun ... bash -c '.gitlab/test_python.sh' streaming connection was severed: Error: error streaming command in container: error reading from error stream: next reader: websocket: close 1006 (abnormal closure): unexpected EOF, followed by script returned exit code 1. pytest was progressing normally (NIXL agents initializing every ~0.5s, no gaps >1s, last agent 82a58349... at 13:11:15), so this is an abrupt transport-level container/websocket drop — not a hang (no long silent gap) and not a test assertion failure (no Python traceback, no FAILED, no NIXL core dump). The node stayed alive: the subsequent enroot export (added by PR #1800 for crash debugging) completed successfully. Note the build 124 ucx-master "failure" is the expected optional build_helper stop; it retried and succeeded in stage 142.

Implicated commit: none — infrastructure flake. PR #1863 (525d1e8) is a CI-only change ("add per-PR container-build pipeline") and does not touch test/python/, .gitlab/test_python.sh, or the C++/Python bindings.

File: N/A (Jenkins/Slurm/enroot websocket stream in stage 337; no source file implicated)

Suggested fix: Re-run the "Run Python tests" stage — a websocket 1006 mid-stream is a transient Slurm/enroot session drop. If it recurs across builds, investigate the scctl/srun keep-alive and enroot container stability on the GPU test node (check node dmesg/OOM and Slurm step logs for StepId 65353.1 around 13:11:15 UTC), and consider adding stream reconnection/retry to the slurm.run wrapper. Do not raise any time limit — the run was actively making progress when the stream dropped.

Related: PR #1863 (#1863); PR #1800 (container export on failure, explains the trailing enroot export step).

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 525d1e89

TL;DR: The Run Nixlbench tests stage crashed with SIGSEGV (exit code 139) in the UCCL backend during "Destroying Engine" teardown, right after the first UCCL READ VRAM→VRAM benchmark completed; this is a pre-existing UCCL crash (already being fixed in open PR #1724), not a regression from this PR.

Full analysis

Summary: Jenkins job nixl-ci-gpu #2793 failed in the Run Nixlbench tests stage (node 337) when the UCCL backend benchmark segfaulted on engine destruction.

Root cause: The UCCL nixlbench worker completed its measurement (printed 16384 4 0.035242 ...) and then, during Destroying Engine..., hit an RDMA connection error ([WARN ... event_loop rdma/epoll_client.h:207] Error/HUP on connection: 1.1.101.2:41061) and crashed with SIGSEGV. GNU parallel reported This job failed, and slurm reported task 0: Exited with exit code 139. The crash is in the UCCL plugin's engine/connection teardown, not in the code changed by this PR — all UCX and POSIX benchmarks, plus the CPP/Python/Rust stages, passed. The concurrent stage-125 "ucx-v1.22.x build_helper" failure was transient: the retried step (node 142) succeeded, so it is not the terminal failure.

Implicated commit: unknown for the exact regression; the crash lives in the UCCL backend (last significant changes: 63c494e1 "UCCL: Simplify and Optimize for batch transfers" and f8e2221f "Enable local xfer for UCCL backend", both by Pravein Govindan Kannan). It is a known issue tracked by open PR #1724.

File: UCCL RDMA teardown path — epoll_client.h:207 (Error/HUP handler) reached during Destroying Engine in src/plugins/uccl/uccl_backend.cpp engine-destroy logic.

Suggested fix: This is a known UCCL teardown crash — land/rebase onto PR #1724 ("UCCL crash Fix + GDB trap") which addresses the same segfault. In the meantime, since the crash is unrelated to this PR's changes, re-run the build (the failure is not caused by commit 525d1e8); if UCCL remains unstable in CI, gate the UCCL nixlbench case behind a flag or mark it non-fatal (e.g. don't run it under parallel --halt now,fail=1) until #1724 merges. The crash core dump was exported to /enroot_images/nixl-ci-v1.22.x-2793.sqsh (see docs/DebugCoreDumps.md) for a backtrace confirming the exact frame.

Related: #1724 (UCCL crash Fix + GDB trap)

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 525d1e89

TL;DR: The GPU CI on node mizu02 failed because its InfiniBand devices (ibp75s0f0/f1) are missing/non-functional — HardwareWarningTest.NoWarningWhenIbAndCudaSupported threw Failed to create UCX context: No such device (exit 1), and on the ucx-master stage ucx_backend_test aborted with failed to arm iface … Input/output error (SIGABRT/core dump). This is an infrastructure/hardware problem on the runner, not a code regression in commit 525d1e8.

Full analysis

Summary: Two "Run CPP tests" stages failed: stage 242 (ucx-v1.22.x) had a genuine test failure in HardwareWarningTest.NoWarningWhenIbAndCudaSupported, and stage 203 (ucx-master) core-dumped in ucx_backend_test.

Root cause: The test node mizu02 has no working IB fabric. The log repeatedly shows UCX ERROR failed to get interface index for ibp75s0f0/f1: No such device and 4 IB device(s) were detected, but accelerated IB support was not found. Consequently:

• NoWarningWhenIbAndCudaSupported explicitly asks UCX for ib,cuda transports (no usable transports/devices (asked ib,cuda …)) and throws Failed to create UCX context: No such device — a hard failure (exit code 1).
• ucx_backend_test (ucx-master) crashed mid-run with ucp_worker.c:669 Fatal: failed to arm iface … Input/output error, aborting with a core dump (exit 134).
  The remaining 68 "failures" in stage 242 are exit-code-42 harness trips on the benign "UCX CUDA support was not found" warning, not real test logic failures. Both stages ran with continuous output up to the crash (no multi-minute gaps), so these are crashes/assertions, not hangs or timeouts.

Implicated commit: unknown — no evidence the PR #1863 commit 525d1e8 touched UCX context creation or these tests; the errors stem from the runner's missing IB devices.

File: stage 203: .gitlab/test_cpp.sh:87 invoking ./bin/ucx_backend_test (crash at UCX ucp_worker.c:669); stage 242: HardwareWarningTest.NoWarningWhenIbAndCudaSupported (UCX context creation with ib,cuda transports).

Suggested fix: Treat as a CI infrastructure issue: repair/re-enable the IB devices on mizu02 (interfaces ibp75s0f0/ibp75s0f1 report "No such device"), or drain/reschedule the job onto a node with a healthy IB fabric, then re-run the build. Separately, the codebase should make HardwareWarningTest.NoWarningWhenIbAndCudaSupported (and ucx_backend_test) skip gracefully when no IB transport is available instead of throwing/aborting, so a fabric outage produces a SKIP rather than a hard failure. This is not fixed by increasing the time limit.

Related: #1909 ("Fail on unexpectedly skipped tests and fix device tests") is the closest related PR touching device-test skip behavior; no existing issue matches the failed to arm iface … Input/output error signature.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 525d1e89

TL;DR: The "Run DL CPP tests" stage failed because 8 ucx_threadpool/ucx_threadpool_no_pt TestTransfer cases could not create a UCX backend — the GB200 node's UCX GPUDirect-Async (GDAKI) transport failed with mlx5dv_devx_umem_reg ... Input/output error / Failed to allocate memory pool (gdaki_channel_pool), a device/driver-level fault, not a code bug in the PR.

Full analysis

Summary: Stage 313 (Run DL CPP tests) exited 250; gtest-parallel reported 8 failed tests, all in the multi-worker UCX threadpool suites, at test/gtest/test_transfer.cpp:143 (ASSERT_EQ(status, NIXL_SUCCESS)), plus SIGABRT crashes.

Root cause: UCX's GDAKI (GPUDirect Async KI) transport on node gb200-nvl4-ts2-72 repeatedly failed to register device memory: mlx5dv_devx_umem_reg(...size=8407040) failed: Input/output error and mlx5dv_devx_obj_modify(opcode=0x502/0x507/0x50a) failed, syndrome 0x0: Input/output error, leading to Failed to allocate memory pool (name=gdaki_channel_pool) → UCX endpoint create failed → createBackend: backend 'UCX' ... NIXL_ERR_BACKEND (-3). Some runs also hit a fatal gdaki.c:438 failed to reset gdaki qp and cuDeviceGet(...) invalid device ordinal, killing the process with SIGABRT. This is a low-level RDMA/mlx5 devx + GPU device fault (bad HCA/GPU state, driver/firmware, or resource exhaustion) on the test server — the single-worker ucx/ucx_no_pt suites that ran earlier passed, so it is environmental/hardware, not the PR's source.

Implicated commit: unknown — evidence points to node/driver state, not commit 525d1e8; git-history lookup returned HTTP 503 so no code change could be tied to this.

File: test/gtest/test_transfer.cpp:143 (assertion that surfaces the failure); underlying fault is in UCX GDAKI transport (gdaki.c, ib_mlx5_dv.c), external to the repo.

Suggested fix: Treat this as a GPU/RDMA node health issue: drain and check gb200-nvl4-ts2-72 (dmesg/mlx5 errors, ibstat, GPU health via nvidia-smi, HCA firmware, and available locked/pinned memory), then re-run the build on a healthy node. If GDAKI is not intended for CI on this hardware, disable the GDAKI transport for the threadpool UCX tests (e.g. set UCX_TLS/UCX_NET_DEVICES to exclude the failing device, or gate the gdaki transport) rather than changing code. Do not treat the SIGABRTs/ASSERT_EQ failures as a test-logic regression until the node is confirmed healthy.

Related: Issue/PR search returned HTTP 503 and could not be completed — none found.

Note: Two IB device names appear in the log (mlx5_1/mlx5_2/mlx5_3); no secrets were present in the fetched logs.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 0edebda0

TL;DR: The CPP test stages failed because test node mizu01 has no usable InfiniBand devices and a non-CUDA UCX build; HardwareWarningTest.NoWarningWhenIbAndCudaSupported threw Failed to create UCX context: No such device (exit 42) and ucx_backend_test crashed with failed to arm iface: Input/output error (exit 134). Fix by repairing/relabeling the IB fabric on mizu01 (or gating these transport-requiring tests on device availability) — not a code bug in the PR.

Full analysis

Summary: Two "Run CPP tests" stages failed — one gtest threw on UCX context creation (exit 42) and ucx_backend_test aborted with a UCX fatal error (exit 134).

Root cause: The environment on node mizu01 is broken/misconfigured: the log repeatedly shows UCX ERROR failed to get interface index for ibp75s0f0/ibp75s0f1: No such device and transports 'ib','cuda' are not available. Consequently: (a) HardwareWarningTest.NoWarningWhenIbAndCudaSupported requested ib,cuda transports and UCX returned no usable transports/devices (asked ib,cuda ...) → C++ exception "Failed to create UCX context: No such device"; and (b) during the P-Thr=ON VRAM→VRAM inter-agent transfer, UCX hit ucp_worker.c:669 Fatal: failed to arm iface ...: Input/output error, aborting ucx_backend_test (SIGABRT, exit 134). The remaining 68 gtest "failures" in stage 203 are false positives: the harness's "ATTENTION: Unexpected NIXL warning" check flags benign warnings (8 NVIDIA GPU(s) detected, but UCX CUDA support was not found, getXferTelemetry: cannot return values when telemetry is not enabled) and returns exit 42 even though every case printed [ OK ].

Implicated commit: unknown — no PR #1863 code change is implicated; the failure is environmental (missing IB devices + non-CUDA UCX on mizu01).

File: N/A (infrastructure); the genuine assertion originates in the UCX-context-creation path of the HardwareWarningTest.NoWarningWhenIbAndCudaSupported gtest and inside UCX ucp_worker.c:669.

Suggested fix: Repair the test node: bring up/relabel the InfiniBand interfaces ibp75s0f0/ibp75s0f1 on mizu01 (they report "No such device") and/or schedule these UCX/IB+CUDA tests on a node with a CUDA-enabled UCX build. If the node is intentionally IB/CUDA-less, make the tests skip (GTEST_SKIP) when ib/cuda transports are unavailable instead of asserting, and stop treating the "GPU present but UCX CUDA not found" / "telemetry not enabled" warnings as fatal in the test harness. Re-running on healthy hardware should clear both stages. This is not a code regression from PR #1863.

Related: #1909 ("Fail on unexpectedly skipped tests and fix device tests") is directly relevant to hardening these device-dependent gtests; #1323/#1294 introduced the warning-based fail-reliably harness that produces the exit-42 false positives.

[dpressle]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 3ca2e6d9

TL;DR: The Run Nixlbench tests stage was killed (exit 143 / SIGTERM) after the UCCL backend hung — the process produced its last output at 11:32:39 ("Engine destroyed" after the UCCL READ/DRAM→DRAM test) and then went completely silent for ~16.5 minutes until Jenkins killed it at 11:49:18. The root cause is a hang in the UCCL backend engine setup/teardown, not a timeout that needs raising.

Full analysis

Summary: Jenkins stage "Run Nixlbench tests" (node 337) was ABORTED by a wall-clock kill (script returned exit code 143) because a nixlbench UCCL run hung.

Root cause: A UCCL backend transfer hung. Scanning log timestamps, the largest gap by far is between the last application line — [2026-07-19T11:32:39.099Z] Engine destroyed (end of the UCCL --op_type READ --initiator_seg_type DRAM --target_seg_type DRAM ETCD run) — and [2026-07-19T11:49:18.315Z] Sending interrupt signal to process. That's ~16.5 minutes of total silence, i.e. a hang, not slow-but-progressing work. The next scheduled iteration (the first UCCL --op_type WRITE ETCD test) never printed even its "Connecting to ETCD"/"Creating Engine" banner, so the UCCL worker deadlocked during engine creation/connection setup or during the previous run's asymmetric engine teardown (note the recurring one-sided Destroying Engine... / Stop background accept... / Error/HUP on connection pattern that precedes it). This is consistent with the known-open UCCL stability issue (PR #1724 "UCCL crash Fix + GDB trap"). Exit code 143 is the symptom of the kill, not the cause.

Implicated commit: Not the PR under test. PR #1863 ("CI: add per-PR container-build pipeline", commit 3ca2e6d) only modifies CI plumbing and does not touch UCCL or nixlbench. The hang originates in the UCCL backend (src/plugins/uccl, last substantive change f8e2221 "Enable local xfer for UCCL backend" by Pravein Govindan Kannan, and 63c494e "UCCL: Simplify and Optimize for batch transfers").

File: UCCL backend engine setup/teardown path in src/plugins/uccl/ (engine create/connect/destroy). Last completed test before the hang logged from benchmark/nixlbench UCCL WRITE loop; the hang is on the C++ UCCL side, not the shell driver.

Suggested fix: Do not raise the stage time limit — this is a deadlock. Investigate the UCCL backend's connection/engine teardown for a missing wakeup or blocking wait (the asymmetric Stop background accept... / Error/HUP on connection sequence suggests one worker blocks waiting on a peer that has already torn down). Add a per-nixlbench-invocation timeout wrapper (e.g. wrap the parallel command in timeout) so a single UCCL hang fails fast with a diagnostic instead of consuming the whole build budget, and capture a backtrace on timeout (as proposed in PR #1724). Since PR #1863 is CI-only and unrelated, re-run to confirm the UCCL hang is intermittent/flaky and track it against the UCCL stability work rather than blocking this PR on it.

Related: PR #1863 (the CI change under test); PR #1724 "UCCL crash Fix + GDB trap" (#1724); UCCL backend PRs #1428, #1271, #1151, #895.