ci: split Dockerfile.manylinux into wheel_base/wheel stages to cache deps

[NirWolfer]

What?

Add a wheel_base Docker build target covering all dependency compilation (hwloc, OpenSSL, Abseil, gRPC, AWS/Azure SDKs, Rust, DOCA, libfabric, UCX, etc.). The wheel stage starts FROM wheel_base and only runs the NIXL build and wheel creation steps.

In CI the wheel_base image is built and pushed to Artifactory by ci-demo when Dockerfile.manylinux changes (via the new runs_on_dockers entry). The build-wheel pipeline then passes --wheel-base-image to build-container.sh which pulls the cached image and runs --target wheel, skipping the expensive dep compilation on every PR.

Local docker builds are unaffected: wheel_base ARG defaults to the local stage name so docker build ./contrib/Dockerfile.manylinux builds the full image as before.

Also removes the now-redundant manylinux matrix axis (was single-valued 2_28, hardcoded in the image name and wheel-base arg), and wires cidemo-init.sh to enforce a CI_IMAGE_TAG bump in build-wheel-matrix.yaml whenever Dockerfile.manylinux changes.

Why?

Lower nixl-ci-build-wheel pipeline times, and to justify adding #1777 later on which adds time to this pipeline

Summary by CodeRabbit

• New Features
  • Wheel builds now use a single cached wheel-base (manylinux_2_28) and support selecting a prebuilt dependencies image, alongside optional Torch version selection.
  • The wheel build matrix is simplified to vary only by Python version and architecture for more consistent builds.
• Bug Fixes
  • CI validation now more reliably enforces required image tag updates for both standard CI changes and wheel Dockerfile changes, with clearer guidance when updates are missing.
• Documentation
  • Updated CI docs to reflect the new cached wheel-base flow and how to refresh it via the wheel cache image tag.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[svc-nixl]

👀 Investigating nixl-ci-test-sanitizers build 225…

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds a cached wheel_base image flow for manylinux wheel builds, updates the wheel build script and Jenkins matrix to use it, extends CI tag validation for wheel-specific changes, and refreshes the related CI documentation.

Changes

Wheel base image support

Layer / File(s)	Summary
Dockerfile wheel_base stage and wheel build args contrib/Dockerfile.manylinux	Names the base image stage wheel_base and adds a new wheel stage built from $wheel_base with stage-scoped build args.
build-container.sh wheel-base-image and torch versions support contrib/build-container.sh	Adds --wheel-base-image and --torch-versions options, documents them, and wires them into docker build selection and build args.
build-wheel-matrix.yaml wheel-base job and matrix simplification .ci/jenkins/lib/build-wheel-matrix.yaml	Adds a cached wheel_base image build job, removes the manylinux matrix axis, updates env values, and runs the build inside the manylinux container with the cached base image.
CI change-gate wheel tag validation .ci/cidemo-init.sh	Adds wheel-specific constants, generalizes YAML tag extraction, expands the skip condition, and validates CI_WHEEL_IMAGE_TAG when wheel-related files change.
Wheel build CI documentation updates .ci/docs/build-wheel-matrix-ci.md, .ci/docs/ci-overview.md	Updates the wheel build docs and overview to describe the cached base image flow, the wheel-stage build, and the CI_WHEEL_IMAGE_TAG maintenance check.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• ai-dynamo/nixl#1775: Also adds --torch-versions handling in the wheel build plumbing and related Docker build arg wiring.
• ai-dynamo/nixl#1866: Updates the wheel build path with WHL_TORCH_VERSIONS propagation in the Docker build flow.
• ai-dynamo/nixl#1871: Also expands .ci/docs/ci-overview.md with a Jenkins job description and CI flow details.

Suggested reviewers: ovidiusm, dpressle

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: splitting Dockerfile.manylinux into cached wheel_base and wheel stages.
Description check	✅ Passed	The description matches the required What/Why template and includes the key design details for CI and local builds.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[NirWolfer]

/build

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

.ci/docs/build-wheel-matrix-ci.md (1)

46-58: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Remove the stale Manylinux axis from this section. manylinux_2_28 is fixed in .ci/jenkins/lib/build-wheel-matrix.yaml, so the docs should list only the Python and architecture axes here.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.ci/docs/build-wheel-matrix-ci.md around lines 46 - 58, The Matrix Axes
section still lists Manylinux Versions even though manylinux_2_28 is fixed in
build-wheel-matrix.yaml. Update this documentation block to describe only the
Python Versions and Architectures axes, and remove the stale Manylinux axis from
the build-wheel-matrix-ci.md section.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.ci/cidemo-init.sh:
- Around line 56-62: The wheel_dockerfile_changed detection repeats the same
three-way git diff pattern already used for files_changed, so factor that logic
into a shared helper and reuse it here. Add or extend a helper in cidemo-init.sh
that checks HEAD~1..HEAD, working tree, and staged changes for a given path,
then call it for WHEEL_DOCKERFILE instead of duplicating the grep chain. Keep
the existing wheel_dockerfile_changed flag assignment and echo behavior, but
make the change through the shared helper so future file-gating updates stay
consistent.

In `@contrib/build-container.sh`:
- Around line 168-175: The option names in build-container.sh are too easy to
mix up because --wheel-base and --wheel-base-image control different values in
separate argument cases. Rename the manylinux tag option handled alongside
WHL_BASE to a more distinct name such as --wheel-platform, and update the
related parser branches and any callers that reference the current name so the
argument handling around WHEEL_BASE_IMAGE and WHL_BASE is unambiguous.

---

Outside diff comments:
In @.ci/docs/build-wheel-matrix-ci.md:
- Around line 46-58: The Matrix Axes section still lists Manylinux Versions even
though manylinux_2_28 is fixed in build-wheel-matrix.yaml. Update this
documentation block to describe only the Python Versions and Architectures axes,
and remove the stale Manylinux axis from the build-wheel-matrix-ci.md section.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: c641ee91-2db0-4771-b293-209365ebd236

📥 Commits

Reviewing files that changed from the base of the PR and between 48179b9 and a42637a.

📒 Files selected for processing (6)

• .ci/cidemo-init.sh
• .ci/docs/build-wheel-matrix-ci.md
• .ci/docs/ci-overview.md
• .ci/jenkins/lib/build-wheel-matrix.yaml
• contrib/Dockerfile.manylinux
• contrib/build-container.sh

[NirWolfer]

/ok to test a42637a

[NirWolfer]

/build