Skip to content

[Bugfix] Build CUDA release/nightly images on vLLM 0.27 base - #6612

Closed
panpan0000 wants to merge 1 commit into
vllm-project:mainfrom
panpan0000:fix/cuda-release-image-vllm-0.27
Closed

panpan0000 wants to merge 1 commit into
vllm-project:mainfrom
panpan0000:fix/cuda-release-image-vllm-0.27

Conversation

@panpan0000

@panpan0000 panpan0000 commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

When I was digging which image works best for Minimax-H3, I found this issue on Dockerfile...

This PR are created with AI assistant...

Summary

  • docker/Dockerfile.cuda (the Dockerfile built by .buildkite/release/release-pipeline.yml for release and nightly CUDA images) still defaults to ARG BASE_IMAGE=vllm/vllm-openai:v0.26.0, while the codebase was rebased onto vLLM 0.27.0 in [Rebase] Rebase to vllm 0.27.0 #5976.
  • Result: the published vllm/vllm-omni:v0.27.0rc1, latest, and daily nightly CUDA images install current vLLM-Omni — which uses vLLM 0.27-only APIs — over a vLLM 0.26.0 runtime, a combination CI never validates (docker/Dockerfile.ci uses VLLM_BASE_TAG=v0.27.0).
  • This one-line change aligns the release image base with the CI image base, matching the rebase rule noted in Dockerfile.ci ("Keep VLLM_BASE_TAG aligned with the target vLLM release on every rebase"). The ROCm release Dockerfile was already fixed the same way in [ROCm] [CI] Migrate CI to mi300x for v0.27.x #5886.

Evidence

  • Image label inspection of the published vllm/vllm-omni:v0.27.0rc1 via the Docker Hub registry API shows ai.vllm.image.tag = vllm/vllm-openai:v0.26.0. Labels are inherited from the actual build base; Dockerfile.cuda has no LABEL overrides and does not reinstall vLLM.
  • Timeline: upstream vllm/vllm-openai:v0.27.0 was published 2026-08-10, ~20h before the v0.27.0rc1 image build (2026-08-11), so a correct base was available.
  • The release pipeline builds with -f docker/Dockerfile.cuda and passes no --build-arg BASE_IMAGE override, so the stale default is what ships.
  • Prior attempt: [Bugfix] Build CUDA images on vLLM 0.27 #6378 (same fix, closed unmerged by its author; no maintainer objection on record).

Test plan

  • CI builds the release image with the v0.27.0 base
  • Maintainers may want to rebuild/republish the nightly CUDA image after merge so it stops shipping the mismatched runtime; the v0.27.0rc1 tag remains affected (immutable)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@vllm-omni-review-bot

Copy link
Copy Markdown

This PR was classified as CI work.

CI owner: @yenuo26

@panpan0000, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

@panpan0000

Copy link
Copy Markdown
Contributor Author

Self-review: one-line change, ARG BASE_IMAGE=vllm/vllm-openai:v0.26.0 → v0.27.0 in docker/Dockerfile.cuda.

What I checked:

  • docker/Dockerfile.cuda is a 22-line file with no LABEL overrides and no vLLM reinstall step, so the image's vLLM version is entirely determined by the base image.
  • .buildkite/release/release-pipeline.yml builds this Dockerfile (x86_64 and aarch64) with no --build-arg BASE_IMAGE override, so the default is what ships in release and nightly CUDA images.
  • docker/Dockerfile.ci already targets VLLM_BASE_TAG=v0.27.0 (kept aligned per its own rebase comment), and docker/Dockerfile.rocm was bumped to vllm-openai-rocm:v0.27.0 in [ROCm] [CI] Migrate CI to mi300x for v0.27.x #5886 — this change makes the CUDA release image consistent with both.
  • The published vllm/vllm-omni:v0.27.0rc1 image's inherited label ai.vllm.image.tag=vllm/vllm-openai:v0.26.0 confirms it was built on the 0.26.0 base. Upstream vllm/vllm-openai:v0.27.0 was published 2026-08-10, before the rc1 image build.
  • Grepped docker/ and .buildkite/ on main: this line is the only remaining vllm-openai:v0.26.0 pin.

Not verified: I did not run a full image build or boot test locally (no GPU host); relying on CI for that. The already-published v0.27.0rc1 tag stays as-is (immutable); this fix takes effect for nightlies and the v0.27.0 final.

@vllm-omni-review-bot automated review welcome.

Since vllm-project#5976 rebased vLLM-Omni onto vLLM 0.27.0, the release pipeline
(.buildkite/release/release-pipeline.yml) keeps building
docker/Dockerfile.cuda with its default BASE_IMAGE pinned at
vllm/vllm-openai:v0.26.0. The published v0.27.0rc1 and nightly CUDA
images therefore install current vLLM-Omni (which requires vLLM 0.27
APIs) over an incompatible vLLM 0.26.0 runtime, a combination CI never
validates (Dockerfile.ci uses VLLM_BASE_TAG=v0.27.0).

Confirmed on the published v0.27.0rc1 image: the inherited label
ai.vllm.image.tag=vllm/vllm-openai:v0.26.0 shows the actual build base
(Dockerfile.cuda has no LABEL overrides and does not reinstall vLLM).
Align the release image base with the CI image base. Prior attempt in
vllm-project#6378 was closed unmerged by its author without maintainer objection.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
@panpan0000
panpan0000 force-pushed the fix/cuda-release-image-vllm-0.27 branch from 2cf1b75 to 9412a06 Compare August 25, 2026 09:45
@panpan0000

Copy link
Copy Markdown
Contributor Author

@lengrongfu PTAL

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical version alignment fix. The CUDA release images were built on vLLM 0.26.0 base while codebase uses vLLM 0.27.0 APIs - this prevents runtime compatibility issues. Simple but essential change.

@hsliuustc0106 hsliuustc0106 added bug Something isn't working CI/CD codes related to changes to CI/CD labels Aug 26, 2026
@panpan0000

Copy link
Copy Markdown
Contributor Author

Thank you @hsliuustc0106

@panpan0000

Copy link
Copy Markdown
Contributor Author

hmm.. my pr was late..

@NickCao

NickCao commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

hmm.. my pr was late..

Yeah, no longer applicable as we are now on 0.28:

ARG BASE_IMAGE=vllm/vllm-openai:v0.28.0

@NickCao NickCao closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD codes related to changes to CI/CD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants