Skip to content

[CI] Stamp provenance labels on the torch-nightly image so Initialized Snapshot E2E can run - #59487

Merged
khluu merged 1 commit into
vllm-project:mainfrom
Thangnguyenvn98:agent/torch-nightly-image-provenance-labels
Oct 7, 2026
Merged

khluu merged 1 commit into
vllm-project:mainfrom
Thangnguyenvn98:agent/torch-nightly-image-provenance-labels

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What

The torch-nightly image build (.buildkite/image_build/image_build_torch_nightly.sh) pushes a --target test image with no provenance labels. The regular test-ci bake target inherits the _labels block in docker/docker-bake.hcl, so its images carry org.opencontainers.image.revision and ai.vllm.build.commit; the nightly image does not (the Dockerfile's LABEL instructions live on the vllm-openai-base stage, which the test target does not descend from).

The (H200) Initialized Snapshot E2E job (introduced in #51360) refuses to run against a candidate image whose commit labels differ from BUILDKITE_COMMIT, so every torch-nightly run of main has failed that job a few seconds after pulling the image — before any snapshot logic runs.

Fix

Stamp the same provenance labels (revision, ai.vllm.build.commit, ai.vllm.build.pipeline, ai.vllm.build.url) on the nightly image at build time.

Why this is not duplicating an existing PR

Searched open PRs for "snapshot e2e", "torch nightly image label", and "image_build_torch_nightly" — no open PR addresses this.

Validation

AI assistance was used to prepare this change; every changed line was reviewed by the submitter.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the ci/build label Sep 30, 2026
@Thangnguyenvn98

Thangnguyenvn98 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Validation

Scoped build: https://buildkite.com/vllm/ci/builds/92230 on 593df82f33 (TORCH_NIGHTLY=1, steps image-build + initialized-snapshot-e2e).

  • :docker: Build image — passed (31m; the first attempt failed a package-index fetch while an apt mirror was mid-sync, and passed on a manual retry; unrelated to this change).
  • Label gate — fixed. The candidate image commit labels do not match failure no longer occurs: the E2E proceeded past image pull and provenance verification through container start, snapshot-runtime verification, CUDA device selection, and model prefetch.
  • The E2E then fails later, in vllm snapshot create at the criu dump step (exit 1, ~11 min in). This is unrelated to this PR, which only adds image labels: the label bug has failed the job seconds after image pull on every torch-nightly run since the job was introduced in [Frontend] Add reusable TP1 initialized-engine snapshots #51360, so nothing past the pull stage has ever executed on this lane and the criu failure was masked. The same job passes on the regular (non-nightly) lane as of build 92059 today, so this looks specific to the torch-nightly image and needs separate investigation. Note the criu dump.log lives inside the container and is not surfaced in the job log, which will make that investigation harder than it should be.

@vllm-agent

vllm-agent commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

CI selector (shadow): 19 test steps (30 jobs) instead of 0 (0 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs) Today's rules Selector Would skip Would add
NVIDIA, CPU and others 0 (0) 19 (30) 0 (0) 19 (30)
AMD mirrors 0 (0) 0 (0) 0 (0) 0 (0)
Selector would run (19)
  • arm-cpu-test ×3
  • basic-models-test-other-cpu
  • cpu-distributed-tests-dp-tp
  • cpu-distributed-tests-pp-tp
  • cpu-kernel-tests ×2
  • cpu-language-generation-and-pooling-model-tests ×3
  • cpu-multi-modal-model-tests-n ×4
  • cpu-multimodal-config
  • cpu-params-env-tokenizers-parser
  • cpu-quantization-model-tests
  • cpu-qwen2-5-vl-multimodal-tests
  • cpu-reasoning-renderers
  • cpu-spec-decode-tests
  • cpu-tool-parsers
  • crcr-report
  • docker-build-metadata
  • multi-modal-processor-cpu ×4
  • qwen4-exp-unit-tests-cpu
  • v1-others-cpu
Would skip (today's rules run them) (0)

none

Would add (today's rules do not run them) (19)
  • arm-cpu-test ×3 (code map)
  • basic-models-test-other-cpu (code map)
  • cpu-distributed-tests-dp-tp (code map)
  • cpu-distributed-tests-pp-tp (code map)
  • cpu-kernel-tests ×2 (code map)
  • cpu-language-generation-and-pooling-model-tests ×3 (code map)
  • cpu-multi-modal-model-tests-n ×4 (code map)
  • cpu-multimodal-config (code map)
  • cpu-params-env-tokenizers-parser (code map)
  • cpu-quantization-model-tests (code map)
  • cpu-qwen2-5-vl-multimodal-tests (code map)
  • cpu-reasoning-renderers (code map)
  • cpu-spec-decode-tests (code map)
  • cpu-tool-parsers (code map)
  • crcr-report (code map)
  • docker-build-metadata (code map)
  • multi-modal-processor-cpu ×4 (code map)
  • qwen4-exp-unit-tests-cpu (code map)
  • v1-others-cpu (code map)
AMD mirrors: would skip (0)

none

AMD mirrors: would add (0)

none

1 changed files · base a982c81e3e · head b47ec440bc · Python record: build 93039 at 1e5d0ea888 · kernel record: table 43b4aae (build 93244), map 43b4aae · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run

The regular test-ci bake target inherits the _labels block in
docker/docker-bake.hcl, so its image carries
org.opencontainers.image.revision and ai.vllm.build.commit. The
torch-nightly lane builds --target test with a plain docker buildx
build and sets no labels, so the (H200) Initialized Snapshot E2E job
fails its candidate-image commit check seconds after pulling the
image on every torch-nightly run of main.

Stamp the same provenance labels on the nightly image.

Co-authored-by: Kimi <noreply@moonshot.cn>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/torch-nightly-image-provenance-labels branch from 593df82 to 907037c Compare October 7, 2026 20:02
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

Validation (rebased head)

Scoped build https://buildkite.com/vllm/ci/builds/93400 on 907037cdb0 (rebased onto main, includes #59699 and #59661), TORCH_NIGHTLY=1, steps image-build + initialized-snapshot-e2e.

  • :docker: Build image passed. A fresh torch-nightly image was built and pushed as …:907037cdb0…-torch-nightly.
  • Initialized Snapshot E2E passed. It passed the label gate, then pull, runtime and device checks, model prefetch, vllm snapshot create, inspect, and restores 1 and 2 across a container stop/start, ending in result=PASS. Both restores returned token 12095 ( Paris) at logprob -0.42218855023384094, matching the captured value.
  • No automatic or manual retries.

Together with #59699, this unblocks the torch-nightly lane for this job: the label fix gets the job past the gate, and #59699 gets it through criu dump.

@khluu
khluu merged commit b47ec44 into vllm-project:main Oct 7, 2026
13 checks passed
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
…d Snapshot E2E can run (vllm-project#59487)

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Kimi <noreply@moonshot.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants