ci: build wheel_base on the ubi8 base (public manylinux + NG… (#1991) - #2009
Merged
Merged
Conversation
…C CUDA) Incorporates the "ubi8" base image discussed on #1968: the wheel_base deps stage now builds on the public PyPA manylinux_2_28 image and COPYs CUDA from an el8/ubi8 NGC CUDA devel image (nvcr.io/nvidia/cuda:*-devel-ubi8), instead of a combined internal manylinux+CUDA base. - Dockerfile.manylinux: add the Option-B header (cuda donor stage + quay manylinux wheel_base + COPY --from=cuda + gcc-toolset-14 + CMake 4.x policy). Keeps #1968's two-stage wheel_base/wheel caching split and the opt-in --build-infinia / --build-ucx-spcx-plugin blocks untouched. INFINIA stays opt-in (host-pull COPY) — not the bundled-default FROM stage. - build-container.sh: restore --cuda-version (MAJOR.MINOR) so local/release full builds drive the torch cuXXX index + cu12/cu13 meta-wheel split. - build-wheel-matrix.yaml: point BASE_IMAGE/BASE_TAG at the ubi8 CUDA image, add CUDA_VERSION + MANYLINUX_VERSION, pass --cuda-version, and forward CUDA_VERSION/MANYLINUX_VERSION to the wheel_base cache build. - Bump CI_IMAGE_TAG in all six matrix YAMLs (cidemo-init.sh requires it when a CI file such as Dockerfile.manylinux changes). Note: only the ci-demo wheel_base cache build pulls nvcr.io/quay.io; per-PR wheel builds pull the cached wheel_base from Artifactory. If the blossom runner cannot reach nvcr.io/quay.io, repoint BASE_IMAGE at an Artifactory ubi8 mirror (same ubi8 tag). ## What? _Describe what this PR is doing._ ## Why? _Justification for the PR. If there is an existing issue/bug, please reference it. For bug fixes, the 'Why?' and 'What?' can be merged into a single item._ ## How? _It is optional, but for complex PRs, please provide information about the design, architecture, approach, etc._ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated wheel builds to use pinned CUDA and manylinux versions for more consistent, repeatable packaging. * Added a selectable CUDA version option for containerized wheel builds (defaulting to CUDA 13.0). * Improved build compatibility by aligning the toolchain/CMake behavior with newer environments. * Refreshed the wheel build base image approach while preserving CUDA support for the build. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
NirWolfer
added a commit
that referenced
this pull request
Jul 30, 2026
## Summary The release wheel build fails in the `--build-ucx-spcx-plugin` step with `dpacc error : Unknown option '-std=gnu23'` (exit 255), breaking every python/arch cell. Since #2009 pinned `gcc-toolset-14` ahead of the system gcc on `PATH`, the plugin's `AC_PROG_CC` detects C23 support and resolves `CC` to `gcc -std=gnu23`. The plugin's `src/gdp/Makefile.am` forwards the compiler unsplit as `-hostcc=$(CC)`, so dpacc receives a bare `-std=gnu23` argument and rejects it. - Preset `ac_cv_prog_cc_c23=no` before the plugin's `configure`, so autoconf falls back to C11 — already covered by gcc 14's default gnu17 — and `CC` stays plain `gcc`. This is a workaround on the nixl side. The underlying bug is in the plugin, which should split the compiler from its flags (`-hostcc=$(firstword $(CC))`, remainder into `--hostcc-options`); the comment notes the revert condition. Signed-off-by: NirWolfer <nwolfer@nvidia.com>
copy-pr-bot Bot
pushed a commit
that referenced
this pull request
Aug 2, 2026
Since #2009 CUDA_VERSION is a Dockerfile ARG defaulting to 13.0, not an ENV from the old combined base image. The nightly never passes it, so the cu12 run builds as cu13: no nixl meta wheel published, torch +cu130 in a 12.9 build. Builds stay green, so the missing artifact goes unnoticed. Derive it from BASE_TAG and move the nightly + release poller to the same nvcr.io ubi8 CUDA base the per-PR wheel job already uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…C CUDA)
Incorporates the "ubi8" base image discussed on #1968: the wheel_base deps stage now builds on the public PyPA manylinux_2_28 image and COPYs CUDA from an el8/ubi8 NGC CUDA devel image
(nvcr.io/nvidia/cuda:*-devel-ubi8), instead of a combined internal manylinux+CUDA base.
Note: only the ci-demo wheel_base cache build pulls nvcr.io/quay.io; per-PR wheel builds pull the cached wheel_base from Artifactory. If the blossom runner cannot reach nvcr.io/quay.io, repoint BASE_IMAGE at an Artifactory ubi8 mirror (same ubi8 tag).
What?
Describe what this PR is doing.
Why?
Justification for the PR. If there is an existing issue/bug, please reference it. For
bug fixes, the 'Why?' and 'What?' can be merged into a single item.
How?
It is optional, but for complex PRs, please provide information about the design,
architecture, approach, etc.
Summary by CodeRabbit
What?
Describe what this PR is doing.
Why?
Justification for the PR. If there is an existing issue/bug, please reference it. For
bug fixes, the 'Why?' and 'What?' can be merged into a single item.
How?
It is optional, but for complex PRs, please provide information about the design,
architecture, approach, etc.