build/ci: backport wheel-build CI and UCX spcx + Infinia plugin bundling to release/1.4.0 - #1968
Conversation
|
👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
f8564c3 to
26de09f
Compare
|
/build |
| NPROC: 32 | ||
| GRPC_NPROC: 10 | ||
| BASE_IMAGE: "${registry_host}/sw-nbu-swx-nixl-docker-local/base/cuda" | ||
| BASE_TAG: '13.0-devel-manylinux--25.09' |
There was a problem hiding this comment.
I don't see the ubi8 used as base image as discussed
There was a problem hiding this comment.
This is not part of the changes i made to allow building infinia + ucx plugin, im pretty sure its part of nates work at #1923
There was a problem hiding this comment.
i will fast follow with the ubi8 changes here
|
🤖 CI Triage Agent — TL;DR: The "Checkout source code" stage failed because PR #1968 modified Full analysisSummary: Jenkins Root cause: The commit touched Implicated commit: [REDACTED:Hex High Entropy String] (merge of [REDACTED:Hex High Entropy String]; PR #1968). Recent Dockerfile.manylinux changes by NirWolfer (e.g. b212c12). File: Suggested fix: Increase Related: PR #1968; prior Dockerfile.manylinux PRs #1918, #1941, #1870.
|
|
🤖 CI Triage Agent — TL;DR: The "Checkout source code" stage failed because PR #1968 modified Full analysisSummary: The Root cause: PR #1968 changed Implicated commit: PR #1968 merge commit File: Suggested fix: In PR #1968, bump
Keep all six tags consistent, then re-run the build. Related: PR #1968 (this build); tag-gate mechanism from PR #1933; the file's recent history includes PRs #1918, #1941, #1870.
|
|
🤖 CI Triage Agent — TL;DR: The "Checkout source code" stage failed because PR #1968 modified Full analysisSummary: Root cause: The build detected a change in Implicated commit: PR #1968 (merge File: Suggested fix: Increase Related: PR #1968 (this build); prior manylinux change PR #1870 (split Dockerfile.manylinux, NirWolfer).
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: The Root cause: The commit modifies
Since the tags are inconsistent, the guard raised Implicated commit: PR #1968 (merge commit File: Suggested fix: In PR #1968, update Related: Recent
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Jenkins Root cause: PR #1968 (commit Implicated commit: File: Suggested fix: In PR #1968, bump Related: PR #1933 (
|
|
🤖 CI Triage Agent — TL;DR: The Jenkins build failed in the "Checkout source code" stage because Full analysisSummary: The Root cause: PR #1968 modified the tracked CI file Implicated commit: [REDACTED:Hex High Entropy String] (merge commit for PR #1968) — the change to File: Suggested fix: In PR #1968, increment Related: PR #1968 (the triggering PR). none other found.
|
|
🤖 CI Triage Agent — TL;DR: The "Checkout source code" stage failed because PR #1968 modified the CI file Full analysisSummary: The Root cause: PR #1968 edited Implicated commit: [REDACTED:Hex High Entropy String] (PR #1968 merge; author not shown in log). The relevant change is the edit to File: Suggested fix: Increase Related: PR #1933 (
|
|
🤖 CI Triage Agent — TL;DR: The "Checkout source code" stage failed because PR #1968 modified Full analysisSummary: Stage 13 "Checkout source code" aborted — the Root cause: The guard in Implicated commit: The head commit of PR #1968 (merge commit File:
Suggested fix: In PR #1968, increase Related: PR #1968 (this build); prior manylinux CI changes in #1918, #1941, #1870. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: The Root cause: Implicated commit: PR #1968, merge commit [REDACTED:Hex High Entropy String] (the change to File: Suggested fix: Increment Related: PR #1968 (this build); related manylinux changes: PR #1918, PR #1941, PR #1870.
|
|
🤖 CI Triage Agent — TL;DR: The "Checkout source code" stage failed because PR #1968 modified Full analysisSummary: Jenkins build #1141 aborted in the "Checkout source code" stage when the Root cause: The commit changed Implicated commit: [REDACTED:Hex High Entropy String] (PR #1968 merge head). Note the CI-file change to File: Suggested fix: Update
so all six match the value already set in Related: PR #1968 (this build); PR #1918 / #1870 (619f09e) — recent
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Build #429 aborted in the "Checkout source code" stage when Root cause: PR #1968 (a backport to Implicated commit: PR #1968 base/backport changes to File: Suggested fix: In PR #1968, bump Related: PR #1968 (#1968); auto-derive rewrite #1933; manylinux split #1870.
|
|
🤖 CI Triage Agent — TL;DR: The "Checkout source code" stage failed because PR #1968 modified Full analysisSummary: The Root cause: PR #1968 changed Implicated commit: [REDACTED:Hex High Entropy String] (merge for PR #1968, "build: backport UCX spcx plugin and Infinia DDN plugin wheel support to release/1.4.0"; wheel-related Dockerfile.manylinux changes trace to NirWolfer's #1918/#1941). File: Suggested fix: Increase
Then re-run the build. (Note: the checked-in Related: PR #1968 (#1968); Dockerfile.manylinux history: #1918, #1941, #1870.
|
…deps (ai-dynamo#1870) ## What? Add a wheel_base Docker build target covering all dependency compilation (hwloc, OpenSSL, Abseil, gRPC, AWS/Azure SDKs, Rust, DOCA, libfabric, UCX, etc.). The wheel stage starts FROM wheel_base and only runs the NIXL build and wheel creation steps. In CI the wheel_base image is built and pushed to Artifactory by ci-demo when Dockerfile.manylinux changes (via the new runs_on_dockers entry). The build-wheel pipeline then passes --wheel-base-image to build-container.sh which pulls the cached image and runs --target wheel, skipping the expensive dep compilation on every PR. Local docker builds are unaffected: wheel_base ARG defaults to the local stage name so docker build ./contrib/Dockerfile.manylinux builds the full image as before. Also removes the now-redundant manylinux matrix axis (was single-valued 2_28, hardcoded in the image name and wheel-base arg), and wires cidemo-init.sh to enforce a CI_IMAGE_TAG bump in build-wheel-matrix.yaml whenever Dockerfile.manylinux changes. ## Why? Lower nixl-ci-build-wheel pipeline times, and to justify adding ai-dynamo#1777 later on which adds time to this pipeline <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Wheel builds now use a single cached wheel-base (manylinux_2_28) and support selecting a prebuilt dependencies image, alongside optional Torch version selection. * The wheel build matrix is simplified to vary only by Python version and architecture for more consistent builds. * **Bug Fixes** * CI validation now more reliably enforces required image tag updates for both standard CI changes and wheel Dockerfile changes, with clearer guidance when updates are missing. * **Documentation** * Updated CI docs to reflect the new cached wheel-base flow and how to refresh it via the wheel cache image tag. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Nir Wolfson <nwolfer@nvidia.com> Signed-off-by: NirWolfer <nwolfer@nvidia.com>
Replace the manual CI_IMAGE_TAG bump requirement with automatic
workspace-level derivation. When CI source files (Dockerfiles, build.sh,
common.sh) change, the tag now updates automatically with no developer
action required.
How it works:
- cidemo-init.sh derives CI_IMAGE_TAG as the short git hash of the most
recent commit touching any CI source file, using: git log -1 --format=%h
-- "${CI_FILES[@]}"
- It then patches all five matrix YAML files in the Jenkins workspace
via sed before the matrix library reads them. No commit or push is made.
- The YAML files now carry CI_IMAGE_TAG: "CI_MANAGED" as a placeholder
that signals the value is managed automatically and must not be
hand-edited.
Caching is preserved: identical CI file history produces the same tag
across PRs, so Artifactory base images are reused when nothing changed.
A PR that modifies a Dockerfile or build script gets a new tag and
triggers a rebuild automatically.
Removes the old validation gate that failed CI when developers forgot to
bump the tag manually.
[HPCINFRA-4578](https://jirasw.nvidia.com/browse/HPCINFRA-4578)
_It is optional, but for complex PRs, please provide information about
the design,
architecture, approach, etc._
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
* **New Features**
* CI Docker image tags are now managed automatically: Jenkins matrix
configs use a `CI_MANAGED` placeholder that’s replaced at the start of
each CI run with a derived tag.
* Improves CI efficiency by keeping the derived tag stable when CI
inputs haven’t changed.
* **Bug Fixes**
* Ensures cached Docker image rebuilds correctly when CI-related
scripts/configs are updated, without requiring manual tag bumps.
* **Documentation**
* Added/updated a “CI_IMAGE_TAG management” section explaining how the
derived tag is computed and how the placeholder is applied across the
Jenkins matrices.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: NirWolfer <nwolfer@nvidia.com>
…whl (ai-dynamo#1941) ## What? Add an opt-in --build-infinia flag to contrib/build-container.sh that pulls the DDN Infinia libs from harbor.mellanox.com (reachable only from the internal network) on the host, stages them flat into the build context, and passes BUILD_INFINIA=true to the manylinux wheel build. Dockerfile.manylinux copies the staged libs to /opt/ddn/red before meson runs so libplugin_INFINIA.so is built into the wheel; libred_client/libred_async remain excluded by auditwheel. No harbor reference exists in the Dockerfile, so external/default builds (BUILD_INFINIA=false) are byte-for-byte unaffected ## Why? needed for the release ## How? _It is optional, but for complex PRs, please provide information about the design, architecture, approach, etc._ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional Infinia DDN plugin support to manylinux wheel builds. * Introduced build flags to enable bundling and to select a custom Infinia libraries image. * When enabled, the build workflow automatically stages the required Infinia libraries for packaging. * **Chores** * Updated CI Docker image tags across wheel and test matrices to the latest version. * Updated ignore rules to prevent staged Infinia libraries from being committed. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: NirWolfer <nwolfer@nvidia.com>
## What Adds opt-in support for building the internal `ucx-spcx-plugin` (UCX external plugin, `libuct_ib_mlx5_ext.so`) as part of the NIXL wheel image build and bundling it into the `nixl-cuXX` wheel. ## How - **`contrib/Dockerfile.manylinux`**: a dedicated RUN right after the UCX build (same pattern as the UCX build itself) clones the plugin from internal GitLab at `UCX_SPCX_PLUGIN_REF` (branch, tag, or full sha; default `main`), builds it against the just-built UCX (`--with-ucx=/usr --with-ucx-src=/usr/local/src/ucx`, shared-only like UCX), and make-installs it directly into `/usr/lib64/ucx` via `--libdir`. Gated by `BUILD_UCX_SPCX_PLUGIN` (default `false`). - **Wheel bundling — no changes**: `build-wheel.sh` and `wheel_add_ucx_plugins.py` are untouched; the plugin sits in the UCX plugins dir and rides the existing bundling/RPATH-patching like any other UCX module. - **`contrib/build-container.sh`**: new `--build-ucx-spcx-plugin` / `--ucx-spcx-plugin-ref` options; the GitLab token is taken from `NIXL_GITLAB_TOKEN` and handed to the docker build as a file-based `--secret` (always supplied, empty when the feature is off, so the unconditional secret mount works identically on docker and podman). - **DPA/FlexIO readiness**: the image pre-installs the FlexIO SDK (`libflexio`) and `dpacc` from the already-configured doca-host repo; when the plugin's upcoming DPA transport lands and links `libflexio`, the Dockerfile bundles it next to the plugin automatically so the wheel ships it — no further changes needed here. ## Token hygiene The token never appears in image layers, `docker history`, argv, or git error output: it travels env → 0600 temp file → build secret → env inside the plugin RUN only, and the clone uses a git credential helper reading it from the environment (credential-free URL, `GIT_TERMINAL_PROMPT=0`). ## Behavior when disabled Flag off ⇒ the wheel build is byte-identical to today; the plugin RUN is a no-op and the wheel RUN is unchanged from main. ## Testing - Feature leg built end-to-end locally (x86_64, CUDA 13, Python 3.12): plugin cloned/built/installed by the inline RUN, wheel contains `nixl_cu13.libs/ucx/libuct_ib_mlx5_ext.so.0.0.0` (shared-only, no static archive) alongside `libplugin_UCX.so`; clean-venv `pip install` verified; `docker history` clean of the token. - Regression: with the flag off the wheel RUN is byte-identical to main (verified by diff); default builds don't reach any new code path. - Build fixes discovered during E2E are baked in: UCX installs to `/usr` in this image (its configure default), `ACLOCAL_PATH` so the plugin's autogen finds `pkg.m4` with the image's source-built autotools, and explicit `-L/usr/lib64/ucx -L/usr/lib64` link dirs (the plugin's configure assumes a `lib/` layout). ## Why? [HPCINFRA-4539](https://jirasw.nvidia.com/browse/HPCINFRA-4539) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an opt-in build flow to include the UCX SPX plugin in manylinux container builds. * Introduced build options/flags to enable the plugin and select the plugin revision. * Updated build help/output to show plugin status and required prerequisites when enabled. * **Chores** * Prevented the temporary plugin source directory from being committed. * Ensured the downloaded plugin sources are cleaned up after the build. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: NirWolfer <nwolfer@nvidia.com>
…mo#1956) ## What? Rename the local `BUILD_TARGET` in `contrib/build-container.sh` to `DOCKER_BUILD_TARGET`. ## Why? The container job exports a `BUILD_TARGET` param (nixl/nixlbench), which leaked into the `docker build` line via `${BUILD_TARGET:-}` as a stray positional arg. Every nixl-target run has failed since ai-dynamo#1870 with `accepts at most 1 arg(s), received 2`. ## How? Rename the local var so it no longer collides with the param. The `--wheel-base-image` path still passes `--target wheel`. ### Note This highlights the need for the per-PR container check ai-dynamo#1863 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved container builds when a wheel base image is configured. * Prevented build parameters from being unintentionally passed to Docker, resulting in more reliable build behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Adds a build-only container gate (`nixl-ci-build-container-pr`) to the PR CI. - Builds the `nixl` (debug, EP on) and `nixlbench` images - one parallel cell each, x86_64 only. - Pushes nothing - it just proves the images still build. - Skips fast when a PR touches no build files, so it can be a required check without slowing unrelated PRs. - Catch changes that break the container builds before they land. - `build-container-pr-matrix.yaml` - path-filter, then build per target, no push. - `proj-jjb.yaml` - new leaf job + dispatcher fan-out entry. - `ci-overview.md` - documents the new job. - Path filter scope: `contrib/`, `benchmark/`, `.ci/`, `meson.build`/`meson.options`, `pyproject.toml`, `setup.py`. - Green on a clean build (nixl EP+debug and nixlbench both built, nothing pushed). - Correctly red on the nvlink `-G` register-count failure - so it blocks the real breakage, not just passes. - After merge: JJB-update `nixl-ci-dispatcher` to turn the gate on for all PRs. - Optional: add `nixl-ci-build-container-pr` to the Required PR Checks ruleset to make it block merges (like `nixl-ci-gpu`/`non-gpu`). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> * **New Features** * Expanded the Jenkins CI dispatcher’s PR fan-out to include a new build-only container verification leaf job (`nixl-ci-build-container-pr`), triggered only after a `/build` comment. * The job runs a gated matrix for targets (`nixl`, `nixlbench`) across architectures (`x86_64`, `aarch64`), skipping when PR changes don’t affect build paths and enforcing per-cell time limits (no image pushes). * **Documentation** * Updated CI overview and Jenkins job documentation: added the new leaf job, refreshed the flow diagram, and updated dispatcher job counts and descriptions. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
26de09f to
e1d7996
Compare
|
/build |
NirWolfer
changed the title
|
🤖 CI Triage Agent — TL;DR: The sanitizer image setup failed because the CI runner couldn't reach the Ubuntu apt mirrors ( Full analysisSummary: Both "Setup Image" stages (tsan #37 FAILURE, asan #38 ABORTED) failed while building the base Docker image; the Root cause: Network connectivity to the Ubuntu package mirrors was down during the build. The tsan stage log shows Implicated commit: None — not caused by commit e03a158 / PR #1968. The apt step in File: Suggested fix: Re-run the build — this is a transient mirror outage. To harden against it: (1) ensure the prebuilt base image tag is available in Harbor so the Related: none |
|
🤖 CI Triage Agent — TL;DR: Both the ASAN and TSAN "Setup Image" stages failed at the Full analysisSummary: Root cause: The prebuilt base image tag Implicated commit: none — not caused by PR #1968 or commit e03a158. The failure is environmental (registry cache miss + archive.ubuntu.com unreachable from the build node). File: Suggested fix:
Related: none found. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: Network connectivity to the Ubuntu package mirrors was down. The log shows repeated Implicated commit: none — the failure is environmental (mirror/network outage), unrelated to PR #1968 / commit e03a158. File: Suggested fix:
Related: none found. |
|
🤖 CI Triage Agent — TL;DR: The two x86_64 container image builds failed at the Full analysisSummary: Stages Root cause: Network connectivity failure from the x86_64 build agent to Implicated commit: none — not caused by the PR commit (e03a158); infrastructure/network issue. File: N/A (failure is in Suggested fix: Retry the build — the connectivity issue appears transient (parallel/retry stages already succeeded). To harden against recurrence: (1) point the container builds at a reliable internal apt mirror or caching proxy (e.g. apt-cacher-ng) instead of Related: none found. |
…C CUDA) Incorporates the "ubi8" base image discussed on #1968: the wheel_base deps stage now builds on the public PyPA manylinux_2_28 image and COPYs CUDA from an el8/ubi8 NGC CUDA devel image (nvcr.io/nvidia/cuda:*-devel-ubi8), instead of a combined internal manylinux+CUDA base. - Dockerfile.manylinux: add the Option-B header (cuda donor stage + quay manylinux wheel_base + COPY --from=cuda + gcc-toolset-14 + CMake 4.x policy). Keeps #1968's two-stage wheel_base/wheel caching split and the opt-in --build-infinia / --build-ucx-spcx-plugin blocks untouched. INFINIA stays opt-in (host-pull COPY) — not the bundled-default FROM stage. - build-container.sh: restore --cuda-version (MAJOR.MINOR) so local/release full builds drive the torch cuXXX index + cu12/cu13 meta-wheel split. - build-wheel-matrix.yaml: point BASE_IMAGE/BASE_TAG at the ubi8 CUDA image, add CUDA_VERSION + MANYLINUX_VERSION, pass --cuda-version, and forward CUDA_VERSION/MANYLINUX_VERSION to the wheel_base cache build. - Bump CI_IMAGE_TAG in all six matrix YAMLs (cidemo-init.sh requires it when a CI file such as Dockerfile.manylinux changes). Note: only the ci-demo wheel_base cache build pulls nvcr.io/quay.io; per-PR wheel builds pull the cached wheel_base from Artifactory. If the blossom runner cannot reach nvcr.io/quay.io, repoint BASE_IMAGE at an Artifactory ubi8 mirror (same ubi8 tag). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🤖 CI Triage Agent — TL;DR: The sanitizer image-setup stages failed because the pre-built base image tag Full analysisSummary: "Setup Image" stages (nodes 37 tsan, 38 asan) failed building Root cause: Two compounding infra failures: (1) base image Implicated commit: unknown — no application commit is implicated; failure is environmental (missing registry tag + apt mirror unreachable). File: Suggested fix: This is an infra issue — no code change will fix it. Recommended actions: (1) re-run the build once mirror connectivity is restored; (2) investigate why base tag Related: none found. |
|
🤖 CI Triage Agent — TL;DR: The "Run DL CPP tests" stage failed because three Full analysisSummary: Stage Root cause: A segfault in the single-worker UCX backend engine's error/disconnect handling. All three crashing tests are the Implicated commit: File: Suggested fix: Review PR #1906's refactor of the base Related: PR #1906 (implicated); the ASan/UBSan/TSan CI build added in PR #1709 ( Note: the earlier
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Jenkins stage "Allocate DL EP Environment" (node 252/235) failed at the SSH Root cause: The Slurm allocation could not be satisfied within the 3600s immediate window — job 1632472 was "queued and waiting for resources" and salloc ultimately failed with Implicated commit: none — not a code regression; e03a158 built and packaged cleanly. File: CI pipeline Slurm allocation step ( Suggested fix: Retry the allocation stage (this is a transient cluster-capacity failure). To make CI more resilient, wrap the Related: none found. |
|
🤖 CI Triage Agent — TL;DR: The two "Setup Image" stages failed because the Docker base-image build could not reach Full analysisSummary: Root cause: Network failure reaching the Ubuntu APT mirrors from the x86_64 build node. Logs show Implicated commit: None — not caused by commit File: Suggested fix:
Related: none found. Note: no time-limit change is warranted — the stages failed with a hard apt error (exit 100), not a wall-clock kill. |
…C CUDA) Incorporates the "ubi8" base image discussed on #1968: the wheel_base deps stage now builds on the public PyPA manylinux_2_28 image and COPYs CUDA from an el8/ubi8 NGC CUDA devel image (nvcr.io/nvidia/cuda:*-devel-ubi8), instead of a combined internal manylinux+CUDA base. - Dockerfile.manylinux: add the Option-B header (cuda donor stage + quay manylinux wheel_base + COPY --from=cuda + gcc-toolset-14 + CMake 4.x policy). Keeps #1968's two-stage wheel_base/wheel caching split and the opt-in --build-infinia / --build-ucx-spcx-plugin blocks untouched. INFINIA stays opt-in (host-pull COPY) — not the bundled-default FROM stage. - build-container.sh: restore --cuda-version (MAJOR.MINOR) so local/release full builds drive the torch cuXXX index + cu12/cu13 meta-wheel split. - build-wheel-matrix.yaml: point BASE_IMAGE/BASE_TAG at the ubi8 CUDA image, add CUDA_VERSION + MANYLINUX_VERSION, pass --cuda-version, and forward CUDA_VERSION/MANYLINUX_VERSION to the wheel_base cache build. - Bump CI_IMAGE_TAG in all six matrix YAMLs (cidemo-init.sh requires it when a CI file such as Dockerfile.manylinux changes). Note: only the ci-demo wheel_base cache build pulls nvcr.io/quay.io; per-PR wheel builds pull the cached wheel_base from Artifactory. If the blossom runner cannot reach nvcr.io/quay.io, repoint BASE_IMAGE at an Artifactory ubi8 mirror (same ubi8 tag). ## What? _Describe what this PR is doing._ ## Why? _Justification for the PR. If there is an existing issue/bug, please reference it. For bug fixes, the 'Why?' and 'What?' can be merged into a single item._ ## How? _It is optional, but for complex PRs, please provide information about the design, architecture, approach, etc._ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated wheel builds to use pinned CUDA and manylinux versions for more consistent, repeatable packaging. * Added a selectable CUDA version option for containerized wheel builds (defaulting to CUDA 13.0). * Improved build compatibility by aligning the toolchain/CMake behavior with newer environments. * Refreshed the wheel build base image approach while preserving CUDA support for the build. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#2009) …C CUDA) Incorporates the "ubi8" base image discussed on #1968: the wheel_base deps stage now builds on the public PyPA manylinux_2_28 image and COPYs CUDA from an el8/ubi8 NGC CUDA devel image (nvcr.io/nvidia/cuda:*-devel-ubi8), instead of a combined internal manylinux+CUDA base. - Dockerfile.manylinux: add the Option-B header (cuda donor stage + quay manylinux wheel_base + COPY --from=cuda + gcc-toolset-14 + CMake 4.x policy). Keeps #1968's two-stage wheel_base/wheel caching split and the opt-in --build-infinia / --build-ucx-spcx-plugin blocks untouched. INFINIA stays opt-in (host-pull COPY) — not the bundled-default FROM stage. - build-container.sh: restore --cuda-version (MAJOR.MINOR) so local/release full builds drive the torch cuXXX index + cu12/cu13 meta-wheel split. - build-wheel-matrix.yaml: point BASE_IMAGE/BASE_TAG at the ubi8 CUDA image, add CUDA_VERSION + MANYLINUX_VERSION, pass --cuda-version, and forward CUDA_VERSION/MANYLINUX_VERSION to the wheel_base cache build. - Bump CI_IMAGE_TAG in all six matrix YAMLs (cidemo-init.sh requires it when a CI file such as Dockerfile.manylinux changes). Note: only the ci-demo wheel_base cache build pulls nvcr.io/quay.io; per-PR wheel builds pull the cached wheel_base from Artifactory. If the blossom runner cannot reach nvcr.io/quay.io, repoint BASE_IMAGE at an Artifactory ubi8 mirror (same ubi8 tag). ## What? _Describe what this PR is doing._ ## Why? _Justification for the PR. If there is an existing issue/bug, please reference it. For bug fixes, the 'Why?' and 'What?' can be merged into a single item._ ## How? _It is optional, but for complex PRs, please provide information about the design, architecture, approach, etc._ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated wheel builds to use pinned CUDA and manylinux versions for more consistent, repeatable packaging. * Added a selectable CUDA version option for containerized wheel builds (defaulting to CUDA 13.0). * Improved build compatibility by aligning the toolchain/CMake behavior with newer environments. * Refreshed the wheel build base image approach while preserving CUDA support for the build. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- ## What? _Describe what this PR is doing._ ## Why? _Justification for the PR. If there is an existing issue/bug, please reference it. For bug fixes, the 'Why?' and 'What?' can be merged into a single item._ ## How? _It is optional, but for complex PRs, please provide information about the design, architecture, approach, etc._ Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
6 participants
What
Backports the wheel-build and CI infrastructure from
maintorelease/1.4.0— six commits, cherry-picked in main's chronological order:Dockerfile.manylinuxintowheel_base/wheelstages so CI caches the expensive deps image; adds--wheel-base-imagetobuild-container.shCI_IMAGE_TAGincidemo-init.sh(matrix YAMLs carry aCI_MANAGEDplaceholder patched at CI time; no manual tag bumps)--build-infinia: bundle the Infinia DDN plugin (libplugin_INFINIA.so) into the wheel (opt-in)--build-ucx-spcx-plugin: build and bundle the UCX spcx external plugin (libuct_ib_mlx5_ext.so) into the wheel (opt-in)BUILD_TARGET→DOCKER_BUILD_TARGETinbuild-container.sh(Jenkins exports aBUILD_TARGETjob param that leaked into the docker command line)nixl-ci-build-container-pr) and dispatcher fan-outAfter this PR,
Jenkinsfile.dispatcher,cidemo-init.sh,build-container.sh,Dockerfile.manylinux, andbuild-container-pr-matrix.yamlare byte-identical tomain.Conflict resolutions (all mechanical):
CI_IMAGE_TAGvalues replaced by theCI_MANAGEDplaceholder (ci: auto-derive CI_IMAGE_TAG in cidemo-init.sh #1933)ci-overview.md: kept release-accurate wording where main's text references features not backported (vLLM/SGLang sanity ci: add vLLM/SGLang NIXL disaggregation sanity testing #1777, cleanup job CI: Add job for CI images cleanup #1785)Intentionally not backported (not needed for release CI): vLLM/SGLang sanity testing (#1777), Artifactory cleanup job (#1785, #1964), CI timeout right-sizing (#1932), LLM base image pins (#1893), port pool cap (#1685).
Why
Needed for the 1.4.0 release: internal wheels must bundle the UCX spcx and Infinia plugins, and the release-branch CI needs the same wheel-build pipeline as
mainto build them.Both plugin features are opt-in (off by default); default builds are byte-identical to the current release branch.