Repository navigation
[CI] Stamp provenance labels on the torch-nightly image so Initialized Snapshot E2E can run - #59487
Conversation
ValidationScoped build: https://buildkite.com/vllm/ci/builds/92230 on 593df82f33 (
|
CI selector (shadow): 19 test steps (30 jobs) instead of 0 (0 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (19)
Would skip (today's rules run them) (0)none Would add (today's rules do not run them) (19)
AMD mirrors: would skip (0)none AMD mirrors: would add (0)none 1 changed files · base |
The regular test-ci bake target inherits the _labels block in docker/docker-bake.hcl, so its image carries org.opencontainers.image.revision and ai.vllm.build.commit. The torch-nightly lane builds --target test with a plain docker buildx build and sets no labels, so the (H200) Initialized Snapshot E2E job fails its candidate-image commit check seconds after pulling the image on every torch-nightly run of main. Stamp the same provenance labels on the nightly image. Co-authored-by: Kimi <noreply@moonshot.cn> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
593df82 to
907037c
Compare
Validation (rebased head)Scoped build https://buildkite.com/vllm/ci/builds/93400 on
Together with #59699, this unblocks the torch-nightly lane for this job: the label fix gets the job past the gate, and #59699 gets it through |
…d Snapshot E2E can run (vllm-project#59487) Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com> Co-authored-by: Kimi <noreply@moonshot.cn>
What
The torch-nightly image build (
.buildkite/image_build/image_build_torch_nightly.sh) pushes a--target testimage with no provenance labels. The regulartest-cibake target inherits the_labelsblock indocker/docker-bake.hcl, so its images carryorg.opencontainers.image.revisionandai.vllm.build.commit; the nightly image does not (the Dockerfile's LABEL instructions live on thevllm-openai-basestage, which thetesttarget does not descend from).The
(H200) Initialized Snapshot E2Ejob (introduced in #51360) refuses to run against a candidate image whose commit labels differ fromBUILDKITE_COMMIT, so every torch-nightly run ofmainhas failed that job a few seconds after pulling the image — before any snapshot logic runs.Fix
Stamp the same provenance labels (
revision,ai.vllm.build.commit,ai.vllm.build.pipeline,ai.vllm.build.url) on the nightly image at build time.Why this is not duplicating an existing PR
Searched open PRs for "snapshot e2e", "torch nightly image label", and "image_build_torch_nightly" — no open PR addresses this.
Validation
bash -nand pre-commit (incl. shellcheck) pass on the edited script; GitHubpre-commitis green on the head commit.ai.vllm.build.commitlabel while bake-built images do.TORCH_NIGHTLY=1,image-build+initialized-snapshot-e2e) on head907037cdb0, rebased ontomainwith [Bugfix] Avoid InfiniBand state in TP1 snapshots #59699 and [Bugfix] Log CRIU failure details before snapshot cleanup #59661: both jobs passed with no retries. The freshly built nightly image passed the commit-label gate, and the E2E ran snapshot creation and both restores through toresult=PASS(token12095,Paris, logprob-0.42218855023384094on both restores).criu dump. That failure had been hidden behind the label bug, and [Bugfix] Avoid InfiniBand state in TP1 snapshots #59699 has since fixed it (it disables NCCL InfiniBand in the snapshot donor process).AI assistance was used to prepare this change; every changed line was reviewed by the submitter.