[Build] Switch CUDA 13.0 wheel builds to PyTorch manylinux_2_28 base by mgoin · Pull Request #41416 · vllm-project/vllm

mgoin · 2026-04-30T21:04:33Z

Purpose

Lowers the glibc floor of the published CUDA 13.0 wheels from 2.34 (Ubuntu 22.04) to 2.28 (AlmaLinux 8 via pytorch/manylinux2_28-builder:cuda13.0), matching what PyTorch ships and restoring compatibility with older OSes (RHEL 8, Ubuntu 20.04, etc.).

Context: see slack discussion and #26118.

Changes:

docker/Dockerfile: new BUILD_OS arg (ubuntu default, manylinux opt-in) branches the system-deps install (apt → dnf), Python bootstrap (uv-fetched → /opt/python/cpXY-cpXY), and the dev stage's libnuma install. Both paths converge on /opt/venv so downstream stages (csrc-build, extensions-build, build) are unchanged.
.buildkite/release-pipeline.yaml: cu130 wheel builds (x86 + aarch64) now pass BUILD_OS=manylinux + the pytorch base image.
FINAL_BASE_IMAGE, cu129 builds, and the release Docker image build are unchanged — runtime container stays on Ubuntu.

TL;DR

  ┌─────────────────────────┬────────────────┬─────────────┐                                                                                       
  │          Wheel          │  Detected tag  │ glibc floor │
  ├─────────────────────────┼────────────────┼─────────────┤                                                                                       
  │ v0.19.1 cu12            │ manylinux_2_31 │ 2.29        │                                                                                     
  ├─────────────────────────┼────────────────┼─────────────┤
  │ v0.20.0 cu130 (current) │ manylinux_2_34 │ 2.34        │
  ├─────────────────────────┼────────────────┼─────────────┤                                                                                       
  │ PR build cu130          │ manylinux_2_24 │ 2.17        │
  └─────────────────────────┴────────────────┴─────────────┘

Test Plan

Local manylinux build of --target build succeeds for x86_64 cu130
auditwheel show on the produced wheel reports manylinux_2_28_x86_64

Validated in cu130 release build with
arm: https://buildkite.com/vllm/release-v2/builds/1150/canvas?sid=019de0f0-babe-4073-b87a-ee91de535bd5&tab=output
x86 https://buildkite.com/vllm/release-v2/builds/1150/canvas?sid=019de0f0-bac5-4dd5-93d7-3c7ef554f632&tab=output

Test Result

Pending validation.

AI assistance (Claude) was used to draft the Dockerfile branching.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

Signed-off-by: mgoin <mgoin64@gmail.com>

claude

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

gemini-code-assist

Code Review

This pull request introduces support for manylinux-based builds to ensure compatibility with a glibc 2.28 floor, matching PyTorch's published wheels. Key changes include the addition of a BUILD_OS build argument in the Dockerfile and Buildkite pipeline, conditional logic for package management (switching between apt and dnf), and specialized Python bootstrapping for manylinux environments using uv. I have no feedback to provide.

pytorch/manylinux2_28_aarch64-builder only ships CPU tags. The CUDA aarch64 variant lives at pytorch/manylinuxaarch64-builder (despite the asymmetric naming, AUDITWHEEL_POLICY=manylinux_2_28 is set inside the image). Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

mgoin · 2026-04-30T22:38:09Z

Validation building the docker image locally

TL;DR

  ┌─────────────────────────┬────────────────┬─────────────┐                                                                                       
  │          Wheel          │  Detected tag  │ glibc floor │
  ├─────────────────────────┼────────────────┼─────────────┤                                                                                       
  │ v0.19.1 cu12            │ manylinux_2_31 │ 2.29        │                                                                                     
  ├─────────────────────────┼────────────────┼─────────────┤
  │ v0.20.0 cu130 (current) │ manylinux_2_34 │ 2.34        │
  ├─────────────────────────┼────────────────┼─────────────┤                                                                                       
  │ PR build cu130          │ manylinux_2_24 │ 2.17        │
  └─────────────────────────┴────────────────┴─────────────┘

sudo docker buildx build --build-arg CUDA_VERSION=13.0.2 --build-arg torch_cuda_arch_list="8.0" --build-arg max_jobs=32 --build-arg BUILD_OS=manylinux --build-arg BUILD_BASE_IMAGE=pytorch/manylinux2_28-builder:cuda13.0 --target build -f docker/Dockerfile -t vllm-manylinux-test .

(vllm) ➜  vllm git:(manylinux-image-cu130) ✗ auditwheel show vllm-0.19.1-cp38-abi3-manylinux_2_31_x86_64.whl 

vllm-0.19.1-cp38-abi3-manylinux_2_31_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libgcc_s.so.1 with versions
{'GCC_3.0'}, libc.so.6 with versions {'GLIBC_2.3.3', 'GLIBC_2.14',
'GLIBC_2.2.5', 'GLIBC_2.3.4', 'GLIBC_2.4', 'GLIBC_2.3', 'GLIBC_2.3.2',
'GLIBC_2.9', 'GLIBC_2.7'}, libstdc++.so.6 with versions
{'GLIBCXX_3.4.18', 'CXXABI_1.3', 'GLIBCXX_3.4.11', 'GLIBCXX_3.4.21',
'GLIBCXX_3.4.9', 'CXXABI_1.3.9', 'GLIBCXX_3.4', 'GLIBCXX_3.4.20'},
libcudart.so.12 with versions {'libcudart.so.12'}, libdl.so.2 with
versions {'GLIBC_2.2.5', 'GLIBC_2.3.4'}, libm.so.6 with versions
{'GLIBC_2.2.5', 'GLIBC_2.29'}, librt.so.1 with versions
{'GLIBC_2.2.5'}, libpthread.so.0 with versions {'GLIBC_2.2.5'}

This constrains the platform tag to "manylinux_2_31_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.


(vllm) ➜  vllm git:(manylinux-image-cu130) ✗ auditwheel show vllm-0.20.0-cp38-abi3-manylinux_2_35_x86_64.whl 

vllm-0.20.0-cp38-abi3-manylinux_2_35_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libgcc_s.so.1 with versions
{'GCC_3.0'}, libc.so.6 with versions {'GLIBC_2.3.3', 'GLIBC_2.6',
'GLIBC_2.3.2', 'GLIBC_2.7', 'GLIBC_2.2.5', 'GLIBC_2.4', 'GLIBC_2.14',
'GLIBC_2.3', 'GLIBC_2.34', 'GLIBC_2.9', 'GLIBC_2.3.4', 'GLIBC_2.17'},
libcudart.so.13 with versions {'libcudart.so.13'}, libstdc++.so.6 with
versions {'GLIBCXX_3.4', 'GLIBCXX_3.4.18', 'CXXABI_1.3.2',
'CXXABI_1.3.3', 'CXXABI_1.3', 'GLIBCXX_3.4.9', 'GLIBCXX_3.4.20',
'GLIBCXX_3.4.14', 'GLIBCXX_3.4.15', 'CXXABI_1.3.5', 'CXXABI_1.3.9',
'GLIBCXX_3.4.21', 'GLIBCXX_3.4.19', 'GLIBCXX_3.4.11'}, libm.so.6 with
versions {'GLIBC_2.2.5', 'GLIBC_2.29'}, libnvrtc.so.13 with versions
{'libnvrtc.so.13'}, librt.so.1 with versions {'GLIBC_2.2.5'},
libdl.so.2 with versions {'GLIBC_2.2.5', 'GLIBC_2.3.4'},
libpthread.so.0 with versions {'GLIBC_2.2.5', 'GLIBC_2.3.2'}

This constrains the platform tag to "manylinux_2_34_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.


(vllm) ➜  vllm git:(manylinux-image-cu130) ✗ sudo docker run --rm vllm-manylinux-test bash -c 'uv pip install auditwheel; auditwheel show dist/vllm-0.20.1rc1.dev107+g5aa7eb21b-cp38-abi3-linux_x86_64.whl'
Using Python 3.12.13 environment at: /opt/venv
Resolved 3 packages in 100ms
Prepared 2 packages in 10ms
Installed 2 packages in 38ms
 + auditwheel==6.6.0
 + pyelftools==0.32
INFO:auditwheel.wheel_abi:setting libc to glibc

vllm-0.20.1rc1.dev107+g5aa7eb21b-cp38-abi3-linux_x86_64.whl is
consistent with the following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libgcc_s.so.1 with versions
{'GCC_3.0'}, libc.so.6 with versions {'GLIBC_2.2.5', 'GLIBC_2.14'},
libcudart.so.13 with versions {'libcudart.so.13'}, libstdc++.so.6 with
versions {'CXXABI_1.3.9', 'GLIBCXX_3.4.21', 'GLIBCXX_3.4.11',
'CXXABI_1.3', 'GLIBCXX_3.4.18', 'GLIBCXX_3.4.9', 'GLIBCXX_3.4'},
libdl.so.2 with versions {'GLIBC_2.2.5'}, libm.so.6 with versions
{'GLIBC_2.2.5'}

This constrains the platform tag to "manylinux_2_24_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

mgoin · 2026-05-01T00:27:49Z

https://buildkite.com/vllm/release-v2/builds/1150/canvas

mgoin · 2026-05-01T14:19:30Z

I believe CI failures are unrelated and the release build passed, so I think we are good

tlrmchlsmth · 2026-05-01T18:15:22Z

+            # Upgrade to GCC 10 to avoid https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92519
+            # as it was causing spam when compiling the CUTLASS kernels
+            gcc-10 \
+            g++-10 \
+        && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 110 --slave /usr/bin/g++ g++ /usr/bin/g++-10 \


I think we can remove this now! This is actually a downgrade since we're not building on 20.04 anymore

I can remove this in a separate PR as it is unrelated, this is just keeping the previous state

…llm-project#41416) Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Joachim Studnia <joachim@mistral.ai>

jikunshang · 2026-05-06T04:38:26Z

          queue: cpu_queue_release
        commands:
-          - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=13.0.2 --build-arg torch_cuda_arch_list=\"${CUDA_ARCH_X86}\" --build-arg BUILD_BASE_IMAGE=nvidia/cuda:13.0.2-devel-ubuntu22.04 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
+          - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=13.0.2 --build-arg torch_cuda_arch_list=\"${CUDA_ARCH_X86}\" --build-arg BUILD_OS=manylinux --build-arg BUILD_BASE_IMAGE=pytorch/manylinux2_28-builder:cuda13.0 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."


images like pytorch/manylinux2_28-builder:cuda13.0 will be regular updated from torch side, this may cause some potential regression.
Should we use a stable image version like pytorch/manylinux2_28-builder:cuda13.0-v2.11.0-rc6 instead?

…llm-project#41416) Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

…llm-project#41416) Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: hongbolv <33214277+hongbolv@users.noreply.github.com>

Switch cu130 wheel builds to pytorch/manylinux2_28-builder

5aa7eb2

Signed-off-by: mgoin <mgoin64@gmail.com>

claude Bot reviewed Apr 30, 2026

View reviewed changes

mgoin changed the title ~~[Build] Switch cu130 wheel builds to pytorch/manylinux2_28-builder~~ [Build] Switch CUDA 13.0 wheel builds to PyTorch manylinux_2_28 base Apr 30, 2026

mergify Bot added ci/build nvidia labels Apr 30, 2026

github-project-automation Bot added this to NVIDIA Apr 30, 2026

gemini-code-assist Bot reviewed Apr 30, 2026

View reviewed changes

mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 30, 2026

Merge branch 'main' into manylinux-image-cu130

420cd49

mgoin mentioned this pull request May 1, 2026

[Bug]: H100 CUDA 13.0 nightly wheels require glibc 2.35 (manylinux_2_35), failing on glibc 2.34 systems #38908

Closed

1 task

tlrmchlsmth reviewed May 1, 2026

View reviewed changes

vllm-bot merged commit cfd2573 into main May 2, 2026
149 of 154 checks passed

vllm-bot deleted the manylinux-image-cu130 branch May 2, 2026 12:51

github-project-automation Bot moved this to Done in NVIDIA May 2, 2026

mgoin mentioned this pull request May 4, 2026

[Build] Switch CUDA 12.9 wheel builds to PyTorch manylinux_2_28 base #41668

Open

3 tasks

jikunshang reviewed May 6, 2026

View reviewed changes

chaojun-zhang pushed a commit to chaojun-zhang/vllm that referenced this pull request May 6, 2026

[Build] Switch CUDA 13.0 wheel builds to PyTorch manylinux_2_28 base (v…

197afcc

…llm-project#41416) Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Build] Switch CUDA 13.0 wheel builds to PyTorch manylinux_2_28 base#41416

[Build] Switch CUDA 13.0 wheel builds to PyTorch manylinux_2_28 base#41416
vllm-bot merged 3 commits intomainfrom
manylinux-image-cu130

mgoin commented Apr 30, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

mgoin commented Apr 30, 2026

Uh oh!

mgoin commented May 1, 2026

Uh oh!

mgoin commented May 1, 2026

Uh oh!

tlrmchlsmth May 1, 2026 •

edited

Loading

Uh oh!

mgoin May 1, 2026

Uh oh!

Uh oh!

jikunshang May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mgoin commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

mgoin commented Apr 30, 2026

Uh oh!

mgoin commented May 1, 2026

Uh oh!

mgoin commented May 1, 2026

Uh oh!

tlrmchlsmth May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgoin May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jikunshang May 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mgoin commented Apr 30, 2026 •

edited

Loading

tlrmchlsmth May 1, 2026 •

edited

Loading