ci: add vLLM/SGLang NIXL disaggregation sanity testing

[NirWolfer]

Add vLLM / SGLang NIXL disaggregation sanity to the build-wheel pipeline

What

Adds a per-PR GPU sanity test that proves a freshly-built NIXL wheel performs a real prefill → decode KV transfer inside the vLLM and SGLang runtime images. It is folded into the existing nixl-ci-build-wheel job and runs on x86_64 / the mizu SLURM cluster, reusing the wheels that job already builds (no second wheel build).

Why

Until now the wheels were only smoke-tested for import/version. This validates that the wheel actually works end-to-end in the disaggregated inference path that consumes NIXL.
HPCINFRA-4419

How it works

Everything runs sequentially in the single podman container (ci-demo runs each runs_on_dockers entry as a parallel branch with no cross-branch ordering, so a split would race):

1. Build wheels (existing, both arches).
2. Build per-PR images (x86_64): extract the wheels and layer them onto the weights-base images; tag by commit SHA and push.
3. Install scctl (dedicated x86_64-only step, scoped to the sanity).
4. Allocate a mizu GPU node, run the vLLM and SGLang sanity, then release the allocation.

All sanity steps are gated to x86_64 via containerSelector.

Components

• Weights-base images — contrib/Dockerfile.{vllm,sglang}-base = pinned upstream image (docker.io/...) + baked-in test model (Qwen/Qwen3-1.7B) + iproute2. Built by the ci-demo runs_on_dockers mechanic (category: tool), rebuilt only when the Dockerfile changes or CI_IMAGE_TAG is bumped. sglang-base also installs sglang-router and bakes the GSM8K test set, so nothing is fetched at test time.
• Per-PR images — reuse the wheels just built; the sanity script + .ci/scripts/common.sh are baked in and the image is tagged by commit SHA so the SLURM run pulls the exact image.
• .gitlab/test_vllm_sglang_sanity.sh:
  • vLLM — launches a 1-prefill / 1-decode pair through the baked toy_proxy_server and asserts a completion comes back.
  • SGLang — routes through sglang-router and asserts gsm8k accuracy ≥ GSM8K_MIN_ACCURACY (default 0.55); a broken transfer collapses accuracy toward 0.
  • All host-bound ports (HTTP, vLLM NIXL side-channel, SGLang bootstrap) are allocated via get_next_tcp_port, so concurrent jobs sharing a node (8 GPUs, host networking) don't collide.

Other changes

• cidemo-init.sh tracks the new base Dockerfiles.
• CI_IMAGE_TAG bumped across the matrices so the bases rebuild.

Notes for reviewers

• The sanity model and accuracy threshold are env-overridable (SANITY_MODEL_ID, GSM8K_MIN_ACCURACY, GSM8K_NUM_QUESTIONS).

Summary by CodeRabbit

• New Features
  • Added end-to-end CI sanity checks for both vLLM and SGLang (prefill→decode validation with GSM8K scoring).
  • Introduced per-PR vLLM/SGLang images with preloaded model artifacts and GSM8K test data for faster, more reliable runs.
  • Added dedicated pinned “base” images for vLLM and SGLang with baked-in models.
• Bug Fixes
  • Improved test readiness gating to verify a real completion request (not just health).
  • Standardized the proxy script install location used during image builds.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[NirWolfer]

/build

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu build 1360…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit ca294b98

This confirms the nixlbench branch in my log passed all its DRAM tests and skipped VRAM ("Worker without GPU, skipping VRAM tests") because HAS_GPU=false. That branch exited 0.

I've now thoroughly reviewed the one log I could fetch. The captured branch — including the full C++ gtest suite, the shell test binaries, test_azure.sh, and all nixlbench runs — completed successfully end to end. The pipeline's FAILURE comes from parallel task failed with msg: ... script returned exit code 1, referring to a sibling parallel branch (a second SLURM job, 1483396, was cancelled at cleanup) whose console output is not present in the log text returned to me.

I must not speculate about a branch whose log I don't have. Here is what the evidence supports:

Summary: A parallel branch of the nixl-ci-dl-gpu #1360 build failed with exit code 1; the branch captured in the fetched console log passed all its stages successfully.

Root cause: Not determinable from the available log. The branch present in the fetched console (SLURM job 1483397, the "master"/DL nixlbench + C++ tests branch) ran to completion with all tests passing — every gtest case passed, test_azure.sh printed "Done.", and all nixlbench POSIX/UCX DRAM runs reported results and exited cleanly. The pipeline-level hudson.AbortException: parallel task failed with msg: ... script returned exit code 1 originates from a different parallel branch (a second SLURM job, 1483396, was scanceled during cleanup) whose console output was not included in the returned log. The only anomaly in the captured branch was benign: the container reported HAS_GPU=false with cuDeviceGetCount(&num_devices) failed: initialization error and UCX CUDA support was not found despite nvidia-smi topo -m showing 4 GPUs — but that branch still passed by degrading to DRAM-only tests, so it is not the failing task.

Implicated commit: unknown (the failing branch's log was not retrievable; the CUDA-init anomaly, if it proves fatal in the other branch, would point to a base-image/CUDA-driver mismatch rather than a specific NIXL source commit).

File: unknown — the failing branch's step/log is not in the fetched output.

Suggested fix: Retrieve the console log for the failing parallel branch specifically (the sibling SLURM job 1483396 / the non-"master" branch of build #1360) — e.g. open the branch's "Restart from stage" / branch-specific log in Blue Ocean, or the flowNode/branch log URL — since the top-level log only interleaved the passing branch. Separately, investigate the cuDeviceGetCount ... initialization error / HAS_GPU=false GPU-init anomaly on gb-nvl-059-compute07: CUDA cannot initialize inside the container even though the GPUs are visible, which is consistent with a CUDA driver/toolkit or base-image mismatch (compare with the CUDA base-image PRs #1869 / #1893). If the other branch runs GPU/VRAM tests, that init failure would make it fail hard rather than skip.

Related: #1869 (Switch CI base image to pytorch + CUDA 13.3), #1893 (CI: pin CUDA-ambiguous LLM base image defaults) — both concern CUDA base-image init issues that could explain the GPU-init anomaly seen in the captured log.

[NirWolfer]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 8aa1f4d7

TL;DR: The "Allocate DL EP Environment" stage failed because a Slurm salloc on the gb200nvl72_cx8 partition sat queued for the full 1-hour --immediate=3600 window and timed out ("Unable to allocate resources: Connection timed out") — a cluster capacity/infra problem, not a code bug. Re-run the build once GB200 nodes are free; no source change is needed.

Full analysis

Summary: Stage 245 ("Allocate DL EP Environment", master-UCX variant) failed at the Slurm resource-allocation step; the NIXL build, EP compile, and Docker image all succeeded beforehand.

Root cause: salloc -N 1 -p gb200nvl72_cx8 --immediate=3600 --time=01:30:00 ... was issued at 09:29:40 and returned salloc: job 1550183 queued and waiting for resources / error: Unable to allocate resources: Connection timed out at 10:29:48 — exactly ~60 minutes (the --immediate=3600 limit) later. No GB200 node became available in that window, so Slurm gave up and the script exited 1. The ~1-hour silent gap is Slurm legitimately waiting in queue for scarce GPU capacity (not a hung process); the sibling allocation for the ucx-v1.22.x path (stage 183) succeeded in 98s, confirming this is capacity contention specific to this allocation, not PR #1777's code.

Implicated commit: none — the failure is in CI infrastructure/cluster scheduling, not in commit 8aa1f4d (PR #1777). All build/compile/install steps passed.

File: n/a (CI Slurm allocation step: salloc ... -p gb200nvl72_cx8 --immediate=3600)

Suggested fix: Retry the build when the gb200nvl72_cx8 partition has free nodes — this is a transient capacity timeout, not a regression. To reduce recurrence, consider: (1) raising --immediate / adding a retry-with-backoff loop around salloc in the allocation stage so a transient queue wait doesn't fail the whole pipeline, or (2) surfacing a clearer "cluster busy, retrying" message. Do not treat this as a bug in the PR.

Related: none