ci: import the Slurm container image before the timed test stages

[dpressle]

What

The first srun --container-image=... of a Slurm pipeline makes pyxis pull and unpack the whole CI image inside that test stage's own timeout. A slow registry or node then kills the stage before any test runs (nixl-ci-gpu-vr build 176: 15 min of pyxis import, exit 143).

Fix

Add a Sync Container Image step right after the allocation in every Slurm matrix: test, vr, dl, dl-ep, llm-container and build-wheel.

• It runs a cheap environment check (hostname, nvidia-smi -L, df -h /) with the same dockerImage and containerName the test steps use, so pyxis creates the named container once and later steps attach to it.
• It has its own 30 min timeout; the per-test timeouts are unchanged.
• A slow or failed import now shows up as its own stage instead of a test timeout.
• The GPU check is non-fatal (prints no-gpu-visible), so a node without nvidia-smi does not turn the pipeline red.

Also documents the step in .ci/docs/ci-overview.md.

Testing

Not testable locally; the slurmCI module comes from ci-demo. The YAML parses and the step order was checked in all six matrices. The nixl-ci-gpu-vr run on this PR is the real check.

Refs HPCINFRA-4925

Summary by CodeRabbit

• CI Improvements

  • Slurm-based test workflows now synchronize container images after resource allocation and before running tests.
  • Image synchronization has a separate 30-minute timeout, keeping image import time outside test-stage timeouts.
  • Pre-test checks report node, GPU visibility, and available disk space.
  • Slurm job timeouts have been extended to accommodate the synchronization step.

• Documentation

  • Added guidance on configuring image synchronization in Slurm pipelines, including container settings and GPU visibility checks.

[dpressle]

/build

[github-actions]

👋 Hi dpressle! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/nixl/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: 1f2dbdfe-90c2-41ac-b301-3869308357d8

---

📥 Commits

Reviewing files that changed from the base of the PR and between 5e0caa2 and a29ca52.

---

📒 Files selected for processing (4)

• .ci/jenkins/lib/build-wheel-matrix.yaml
• .ci/jenkins/lib/test-dl-ep-matrix.yaml
• .ci/jenkins/lib/test-matrix.yaml
• .ci/jenkins/lib/test-vr-matrix.yaml

---

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

---

📝 Walkthrough

Walkthrough

Slurm pipelines now synchronize container images before test stages and run host, GPU-visibility, and disk checks. Job timeouts increase in several pipeline definitions. The CI overview documents the synchronization step’s configuration and behavior.

Changes

Slurm image synchronization

Layer / File(s)	Summary
Add image synchronization before Slurm test stages .ci/jenkins/lib/build-wheel-matrix.yaml, .ci/jenkins/lib/test-*-matrix.yaml, .ci/docs/ci-overview.md	Slurm pipelines add a 30-minute image synchronization step before test stages. The step runs environment checks. Several Slurm job timeouts also increase. The documentation describes container reuse, GPU visibility behavior, and placement guidance.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Suggested reviewers: nirwolfer

---

Merge Risk: ⚪ Minimal · up to a29ca

No confirmed issue blocks merging. The DL and LLM allocation time budgets remain worth watching in CI.

Pre-merge checks | 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: importing the Slurm container image before timed test stages.
Description check		The description explains what changed, why it was needed, how it works, testing performed, and the related issue reference. It provides the required information despite using "Fix" instead of the temp…
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[dpressle]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit a29ca521

TL;DR: Two of the four x86_64 "Build" variants failed at meson setup because the taskflow subproject is a wrap-git that must clone https://github.com/taskflow/taskflow.git at configure time, and git on those agents got an auth prompt instead of the repo (fatal: could not read Username for 'https://github.com'). Fix by making taskflow a hash-pinned wrap-file tarball (with a wrapdb fallback URL) or pre-installing the header-only taskflow into the CI image, so configure no longer depends on live GitHub git access.

Full analysis

Summary: nixl-ci-non-gpu #3426 failed in the Parallel → Build stages (node IDs 414 and 420) during meson setup nixl_build, with ERROR: Subproject taskflow is buildable: NO / Git command failed: git clone --depth 1 --branch v3.10.0 https://github.com/taskflow/taskflow.git.

Root cause: subprojects/taskflow.wrap is a [wrap-git] entry, so dependency('taskflow', fallback: ['taskflow', 'taskflow_dep']) at meson.build:233 triggers a live git clone of GitHub during configure. On the two failing (x86_64) build containers that clone was answered with a credential challenge rather than the repository:

Cloning into 'taskflow'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: expected flush after ref listing
...
meson.build:233:16: ERROR: Git command failed: ['/usr/bin/git', '-c', 'advice.detachedHead=false', 'clone', '--depth', '1', '--branch', 'v3.10.0', 'https://github.com/taskflow/taskflow.git', 'taskflow']

This is a build-time dependency-fetch failure, not a code defect and not a hang: the stages died in 5.6 s and 13.4 s with continuous log output. Note every other dependency in the same configure run resolved fine — liburing downloaded over HTTPS ([wrap-file] with source_fallback_url to wrapdb), asio and tomlplusplus likewise, and the aarch64 variants (e.g. node 422) completed the whole build. So plain HTTPS tarball fetches worked on the same agents; only the git-protocol clone to GitHub failed. taskflow is the sole wrap-git dependency and therefore the single point of failure.

Implicated commit: 76275cfc — "Use tagged Taskflow git wrap for NIXL builds (#2121)", bzsuni (introduced the git-based wrap). The branch under test (ci/slurm-warm-container-image, commit a29ca521) did not cause this; it merely ran on agents where the clone is blocked.

File: subprojects/taskflow.wrap:1-4 (the [wrap-git] url/revision), consumed at meson.build:233

Suggested fix: Pick one, in order of preference:

1. Convert subprojects/taskflow.wrap to a [wrap-file] mirroring the liburing pattern — source_url = https://github.com/taskflow/taskflow/archive/refs/tags/v3.10.0.tar.gz, a pinned source_hash, and a source_fallback_url pointing at the wrapdb/internal mirror, keeping patch_directory = taskflow and the [provide] block. This removes git from the configure path and gains a mirror fallback, matching every other subproject that succeeded in this very log.
2. Additionally, pre-install taskflow (header-only) into the CI container images so dependency('taskflow') is satisfied by pkg-config/CMake and the fallback is never exercised — consistent with the PRE_INSTALLED_ENV / PRE_INSTALLED_UCX_ENV approach already used for Abseil and UCX in these images.
3. Investigate why outbound git-over-HTTPS to github.com is credential-challenged on the x86_64 agents while HTTPS tarball downloads succeed (likely a proxy that intercepts the git smart-HTTP endpoint); also set GIT_TERMINAL_PROMPT=0 in the build environment so such failures fail fast with a clear error instead of an interactive-prompt message.

Related: PR #2291 "CI: retry git fetches and trim gRPC submodules in dependency builds" — same class of problem (flaky git fetches during dependency builds) and a good place to land the retry/fallback hardening; PR #2378 is the branch under test and is not implicated.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit a29ca521

TL;DR: The container build fails in meson setup because the taskflow Meson git-wrap clone of https://github.com/taskflow/taskflow.git is unauthenticated (no ghconfig secret mounted on that RUN) and GIT_TERMINAL_PROMPT=0 turns the resulting credential prompt into a hard failure; mount the ghconfig secret on the meson setup step (or convert taskflow.wrap to a wrap-file tarball like liburing.wrap).

Full analysis

Summary: Two of the four parallel "Build image" branches of nixl-ci-build-container-pr #1012 failed at the RUN ... meson setup ... build/ Dockerfile step with ERROR: Subproject taskflow is buildable: NO.

Root cause: meson setup resolves the taskflow dependency through a [wrap-git] subproject, so it shells out to git clone --depth 1 --branch v3.10.0 https://github.com/taskflow/taskflow.git. The log shows that clone failing:

Cloning into 'taskflow'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
fatal: expected flush after ref listing
...
Run-time dependency taskflow found: NO  (tried pkg-config and cmake)
Looking for a fallback subproject for the dependency taskflow
ERROR: Subproject taskflow is buildable: NO
meson.build:233:16: ERROR: Git command failed: ['/usr/bin/git', '-c', 'advice.detachedHead=false', 'clone', '--depth', '1', '--branch', 'v3.10.0', 'https://github.com/taskflow/taskflow.git', 'taskflow']
Error: building at STEP "RUN rm -rf build && mkdir build && ... meson setup ...": exit status 1

GitHub refused the anonymous clone, git fell back to asking for credentials, and ARG GIT_TERMINAL_PROMPT=0 (contrib/Dockerfile:165) made that fatal instead of interactive. Every explicit git clone in the Dockerfile is wrapped in RUN --mount=type=secret,id=ghconfig,target=/root/.gitconfig (abseil, grpc, etcd, aws-sdk, gusli, azure, ucx, gtest-parallel), but the meson setup step at contrib/Dockerfile:419-430 — which clones taskflow implicitly via the wrap — has no such mount, so it runs with an empty /root/.gitconfig and no credential helper. The same defect is present in benchmark/nixlbench/contrib/Dockerfile (the [5/5] STEP 37/54 branch failed identically). Note that liburing resolves fine in the same run because subprojects/liburing.wrap is a [wrap-file] HTTPS tarball, not a git clone.

Implicated commit: [REDACTED:Hex High Entropy String] — "ci: authenticate the Dockerfiles' github.com clones in CI (#2310)", NirWolfer (2026-10-05). It introduced GIT_TERMINAL_PROMPT=0 plus the ghconfig secret mounts but missed the implicit clone performed by meson setup. Contributing: 76275cfc (bzsuni) which made subprojects/taskflow.wrap a tagged [wrap-git].

File: contrib/Dockerfile:419 (and the equivalent meson setup RUN in benchmark/nixlbench/contrib/Dockerfile); wrap definition at subprojects/taskflow.wrap:1-5; consumed at meson.build:233.

Suggested fix: Pick one:

1. Minimal — add the secret mount to the meson steps in both Dockerfiles:
   RUN --mount=type=secret,id=ghconfig,target=/root/.gitconfig rm -rf build && mkdir build && ... meson setup ...
2. Preferred/robust — stop cloning at configure time: convert subprojects/taskflow.wrap to a [wrap-file] pointing at the v3.10.0 release tarball with a source_hash and a source_fallback_url, exactly as subprojects/liburing.wrap does. This removes the auth dependency, is reproducible, and is cacheable.
3. Alternatively, run meson subprojects download in an earlier RUN that does mount ghconfig, so configure-time resolution hits the already-populated subprojects/taskflow directory.

Also worth auditing for any other [wrap-git] entries under subprojects/ that would hit the same path.

Related: PR #2310 (introduced the auth/prompt change), PR #2121 (76275cfc, taskflow git wrap), PR #2378 / branch ci/slurm-warm-container-image (the failing PR — unrelated to the cause; it is just the first build to hit it on this path).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8cae0b3a-7ee3-461f-9dcf-fdef82722d01 in the triage console for the audit trail.