Skip to content

[docker] add vllm-openai-efa target #58768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
khluu merged 2 commits into from
Oct 26, 2026
Merged

[docker] add vllm-openai-efa target #58768

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1205391
[docker] Add AWS EFA image builds via docker/Dockerfile.efa
snadampal Oct 24, 2026
bda1f6f
Merge branch 'main' into efa_install
vllm-agent Oct 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions .buildkite/release-pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,80 @@ steps:
- "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "$$BUILDKITE_LABEL" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134" release-images'


# ---- AWS EFA variants: docker/Dockerfile.efa on the images built above ----
- label: "Build release image - x86_64 - CUDA 13.0 - EFA"
depends_on:
- build-release-image-x86
id: build-release-image-x86-efa
agents:
queue: cpu_queue_release
commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
- |
DOCKER_BUILDKIT=1 docker build \
$(bash .buildkite/scripts/docker-build-metadata-args.sh efa) \
--build-arg BASE_IMAGE=public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m) \
--progress plain \
-f docker/Dockerfile.efa docker/
- "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-efa"
# re-tag to default image tag and push, just in case arm64 build fails
- "docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-efa public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-efa"
- "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-efa"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "$$BUILDKITE_LABEL" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-efa" release-images'

- label: "Build release image - aarch64 - CUDA 13.0 - EFA"
depends_on:
- build-release-image-arm64
id: build-release-image-arm64-efa
agents:
queue: arm64_cpu_queue_release
commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
- |
DOCKER_BUILDKIT=1 docker build \
$(bash .buildkite/scripts/docker-build-metadata-args.sh efa) \
--build-arg BASE_IMAGE=public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m) \
--progress plain \
-f docker/Dockerfile.efa docker/
- "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-efa"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "$$BUILDKITE_LABEL" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-efa" release-images'

- label: "Build release image - x86_64 - Rubin CUDA 13.4 - EFA"
depends_on:
- build-release-image-x86-rubin-cuda-13-4
id: build-release-image-x86-rubin-cuda-13-4-efa
if: build.env("NIGHTLY") == "1"
agents:
queue: cpu_queue_release
commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
- |
DOCKER_BUILDKIT=1 docker build \
$(bash .buildkite/scripts/docker-build-metadata-args.sh cu134-efa) \
--build-arg BASE_IMAGE=public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134 \
--progress plain \
-f docker/Dockerfile.efa docker/
- "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134-efa"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "$$BUILDKITE_LABEL" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134-efa" release-images'

- label: "Build release image - aarch64 - Rubin CUDA 13.4 - EFA"
depends_on:
- build-release-image-arm64-rubin-cuda-13-4
id: build-release-image-arm64-rubin-cuda-13-4-efa
if: build.env("NIGHTLY") == "1"
agents:
queue: arm64_cpu_queue_release
commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
- |
DOCKER_BUILDKIT=1 docker build \
$(bash .buildkite/scripts/docker-build-metadata-args.sh cu134-efa) \
--build-arg BASE_IMAGE=public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134 \
--progress plain \
-f docker/Dockerfile.efa docker/
- "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134-efa"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "$$BUILDKITE_LABEL" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-$(uname -m)-cu134-efa" release-images'
- label: ":docker: Build release image - x86_64 - XPU"
depends_on: ~
id: build-xpu-release-image
Expand Down Expand Up @@ -394,6 +468,35 @@ steps:
- "docker manifest push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-cu134"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "Manifest: Rubin CUDA 13.4" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-cu134" release-manifests'



# ---- AWS EFA variant manifests ----
- label: "Create multi-arch manifest - CUDA 13.0 - EFA"
depends_on:
- build-release-image-x86-efa
- build-release-image-arm64-efa
id: create-multi-arch-manifest-efa
agents:
queue: small_cpu_queue_release
commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
- "docker manifest create public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-efa public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-x86_64-efa public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-aarch64-efa --amend"
- "docker manifest push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-efa"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "Manifest: CUDA 13.0 - EFA" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-efa" release-manifests'

- label: "Create multi-arch manifest - Rubin CUDA 13.4 - EFA"
depends_on:
- build-release-image-x86-rubin-cuda-13-4-efa
- build-release-image-arm64-rubin-cuda-13-4-efa
id: create-multi-arch-manifest-rubin-cuda-13-4-efa
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
- "docker manifest create public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-cu134-efa public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-x86_64-cu134-efa public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-aarch64-cu134-efa --amend"
- "docker manifest push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-cu134-efa"
- 'bash .buildkite/scripts/annotate-build-artifact.sh "Manifest: Rubin CUDA 13.4 - EFA" "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-cu134-efa" release-manifests'
- label: "Create manifest - XPU"
depends_on:
- build-xpu-release-image
Expand Down Expand Up @@ -761,6 +864,10 @@ steps:
allow_failure: true
- step: build-cpu-release-image-arm64
allow_failure: true
# Wait for the EFA images, but don't let an EFA failure block publishing
# the others; the cuda-13-0-efa publish job fails on a partial build.
- step: create-multi-arch-manifest-efa
allow_failure: true

- label: "Publish {{matrix}} release images to DockerHub"
depends_on:
Expand All @@ -772,6 +879,8 @@ steps:
- "bash .buildkite/scripts/publish-release-images.sh {{matrix}}"
matrix:
- "cuda-13-0"
# AWS EFA variant; skipped if no EFA image was built, fails on a partial build.
- "cuda-13-0-efa"
- "rocm"
- "rocm72"
- "xpu"
Expand Down
89 changes: 68 additions & 21 deletions .buildkite/scripts/publish-release-images.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ set -euo pipefail

TARGET="${1:-all}"
case "${TARGET}" in
cuda-13-0 | rocm | rocm72 | xpu | cpu | all) ;;
cuda-13-0 | cuda-13-0-efa | rocm | rocm72 | xpu | cpu | all) ;;
*)
echo "Usage: $0 {cuda-13-0|rocm|rocm72|xpu|cpu|all}"
echo "Usage: $0 {cuda-13-0|cuda-13-0-efa|rocm|rocm72|xpu|cpu|all}"
exit 2
;;
esac
Expand All @@ -21,6 +21,63 @@ target_enabled() {
[ "${TARGET}" = "all" ] || [ "${TARGET}" = "$1" ]
}

# Promote one image family from ECR staging to DockerHub: per-arch tags plus the
# latest and versioned multi-arch manifests. $1 is the tag suffix shared by both
# registries: "" for the default CUDA family, else e.g. "-efa".
publish_image_family() {
local suffix="$1" arch
local staging="public.ecr.aws/q9t5s3a7/vllm-release-repo"
local latest="vllm/vllm-openai:latest${suffix}"
local versioned="vllm/vllm-openai:v${RELEASE_VERSION}${suffix}"

for arch in x86_64 aarch64; do
docker pull "${staging}:${COMMIT}-${arch}${suffix}"
done

for arch in x86_64 aarch64; do
docker tag "${staging}:${COMMIT}-${arch}${suffix}" "vllm/vllm-openai:latest-${arch}${suffix}"
docker tag "${staging}:${COMMIT}-${arch}${suffix}" "vllm/vllm-openai:v${RELEASE_VERSION}-${arch}${suffix}"
docker push "vllm/vllm-openai:latest-${arch}${suffix}"
docker push "vllm/vllm-openai:v${RELEASE_VERSION}-${arch}${suffix}"
done

docker manifest rm "${latest}" || true
docker manifest rm "${versioned}" || true
docker manifest create "${latest}" \
"vllm/vllm-openai:latest-x86_64${suffix}" \
"vllm/vllm-openai:latest-aarch64${suffix}"
docker manifest create "${versioned}" \
"vllm/vllm-openai:v${RELEASE_VERSION}-x86_64${suffix}" \
"vllm/vllm-openai:v${RELEASE_VERSION}-aarch64${suffix}"
docker manifest push "${latest}"
docker manifest push "${versioned}"
}

# Publish an opt-in family all-or-nothing, like the CPU images below: skip when
# neither arch was built, fail on a partial build so it is not silenced.
publish_optional_family() {
local suffix="$1" arch missing=() present=()
local staging="public.ecr.aws/q9t5s3a7/vllm-release-repo"

for arch in x86_64 aarch64; do
if docker manifest inspect "${staging}:${COMMIT}-${arch}${suffix}" >/dev/null 2>&1; then
present+=("${arch}")
else
missing+=("${arch}")
fi
done

if [ "${#present[@]}" -eq 0 ]; then
echo "WARNING: no ${suffix} images found in ECR for ${COMMIT}, skipping the ${suffix} publish"
elif [ "${#missing[@]}" -eq 0 ]; then
publish_image_family "${suffix}"
else
echo "ERROR: partial ${suffix} build (present: ${present[*]}; missing: ${missing[*]})." >&2
echo " Refusing to publish to avoid per-arch and multi-arch tags disagreeing." >&2
exit 1
fi
}

RELEASE_VERSION=$(buildkite-agent meta-data get release-version --default "" | sed 's/^v//')
if [ -z "${RELEASE_VERSION}" ]; then
echo "ERROR: release-version metadata not set"
Expand All @@ -41,25 +98,7 @@ aws ecr-public get-login-password --region us-east-1 | \
# ---- CUDA (default: 13.0) ----

if target_enabled cuda-13-0; then
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-x86_64"
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-aarch64"

docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-x86_64" vllm/vllm-openai:latest-x86_64
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-x86_64" "vllm/vllm-openai:v${RELEASE_VERSION}-x86_64"
docker push vllm/vllm-openai:latest-x86_64
docker push "vllm/vllm-openai:v${RELEASE_VERSION}-x86_64"

docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-aarch64" vllm/vllm-openai:latest-aarch64
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-aarch64" "vllm/vllm-openai:v${RELEASE_VERSION}-aarch64"
docker push vllm/vllm-openai:latest-aarch64
docker push "vllm/vllm-openai:v${RELEASE_VERSION}-aarch64"

docker manifest rm vllm/vllm-openai:latest || true
docker manifest rm "vllm/vllm-openai:v${RELEASE_VERSION}" || true
docker manifest create vllm/vllm-openai:latest vllm/vllm-openai:latest-x86_64 vllm/vllm-openai:latest-aarch64
docker manifest create "vllm/vllm-openai:v${RELEASE_VERSION}" "vllm/vllm-openai:v${RELEASE_VERSION}-x86_64" "vllm/vllm-openai:v${RELEASE_VERSION}-aarch64"
docker manifest push vllm/vllm-openai:latest
docker manifest push "vllm/vllm-openai:v${RELEASE_VERSION}"
publish_image_family ""

ZSTD_DIGEST=$(docker buildx imagetools inspect \
"public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-x86_64" \
Expand All @@ -69,6 +108,14 @@ if target_enabled cuda-13-0; then
"vllm/vllm-openai:v${RELEASE_VERSION}-x86_64-zstd"
fi

# ---- AWS EFA variant (docker/Dockerfile.efa) ----
# No cu134 entry: Rubin CUDA 13.4 is nightly-only and is not promoted to
# DockerHub for the base variant either.

if target_enabled cuda-13-0-efa; then
publish_optional_family "-efa"
fi

# ---- ROCm ----
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
# default stack also owns the plain :latest / :v<ver> tags.
Expand Down
3 changes: 2 additions & 1 deletion docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ ARG TRITON_INSTALL_FROM_SOURCE_REVISION=
# DeepEPv2 requires NCCL >= 2.30.4 (GIN backend).
# This version is used for CUDA 12+ builds to keep the DeepEP build and
# runtime NCCL versions compatible.
ARG NCCL_VERSION=2.30.7
# DeepEPv2 GDA over EFA requires NCCL 2.31.x or later.
ARG NCCL_VERSION=2.31.2

@robertgshaw2-redhat robertgshaw2-redhat Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tlrmchlsmth is this okay to change globally?

@tlrmchlsmth tlrmchlsmth Oct 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, good to change. I doublechecked that the ABI mismatch we hit going from 2.30.4 to 2.30.7 won't happen here

@tlrmchlsmth tlrmchlsmth Oct 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to upgrade to 2.32 for DeepEP 2.5 (see #62033 (comment))


# By parameterizing the base images, we allow third-party to use their own
# base images. One use case is hermetic builds with base images stored in
Expand Down
Loading
Loading