[josh-schell-amd]

Purpose

Make AITER_NIGHTLY=1 work in the ci pipeline (the Python pipeline generator), not just in amd-ci (bootstrap-amd.sh + test-template-amd.j2).

Since vllm-project/vllm#61939 and #820, vLLM's build.sh installs the AITER nightly wheel in image-build-amd. It picks the newest wheel that matches the image's ROCm, torch and Python, or uses the pip spec in AITER_SPEC. rocm_ci_repo() already sends AITER nightly builds to rocm/vllm-dev, never rocm/vllm-ci. So with AITER_NIGHTLY=1, the ci pipeline already builds the right image. What it lacked was step selection: it still ran a normal main build, with CUDA and MI250 jobs, test-selector and source-file gating, NOAUTO blocks and the docs-only skip.

What this PR does

With AITER_NIGHTLY=1, the generator runs the build as a build of named steps, the same mechanism as a retry build (VLLM_CI_ONLY_STEP_KEYS). aiter_nightly_step_keys() names the steps:

• Only the AMD lane. The AMD build steps (amd_cpu) and the GPU jobs on the archs vLLM builds AITER for (AITER_ROCM_ARCH in Dockerfile.rocm_base): gfx942 (MI300, MI325) and gfx950 (MI355). Native AMD steps and amd- mirrors are both included. Everything else, CUDA and MI250 included, is dropped. amd-ci picks by amdgfx942nightly/amdgfx950nightly tags; the generator picks by device arch, using a new arch field on AmdDeviceConfig.
• Nothing is gated. Every named step runs, even with NOAUTO=1, optional steps included. The test selector is skipped, and a docs-only last commit does not skip the build.
• Retry builds. With VLLM_CI_ONLY_STEP_KEYS set, the named steps must be AITER nightly steps.
• Guards. Generation fails with an error annotation, before anything is uploaded:
  • when the nightly's steps, or their prerequisites, include any other step (a kept job depending on a dropped one, or a retry naming a dropped step);
  • when TORCH_NIGHTLY and AITER_NIGHTLY are both set;
  • when the vLLM checkout has no .buildkite/scripts/rocm/build.sh, since an older checkout would otherwise test the stock AITER.
• No recording. The test-selector recorders (VLLM_CI_KERNREC, VLLM_CI_KERNREC_AMD, VLLM_CI_FNREC) are off on the nightly, even if set: its AMD-only runs on an unreleased AITER would mislead the selector's records.

vLLM's build steps are used as they are: refresh-rocm-base-amd → image-build-amd (build.sh rocm ci). Both get ROCM_CI_REPO=rocm/vllm-dev, and GPU pods run rocm/vllm-dev:build-$BUILDKITE_BUILD_ID. As with any named-steps build, the image steps are not uploaded early (#864); they go up with the pipeline. On a PR build, pre-commit still runs and gates the AMD jobs, as on any PR build.

If AITER_NIGHTLY is unset, the pipeline is exactly as before.

Also fixes a retry-build bug on main: a retried AMD GPU job whose depends_on named image-build (e.g. from a group default) was uploaded without image-build-amd and refresh-rocm-base-amd, waiting on a step not in the build. Retry builds now expand AMD dependencies the way the uploaded job declares them.

Changes since the first revision

Rebased onto main after #820, #864, #737 and #871. The earlier aiter-nightly-amd overlay step is gone: it ran aiter-nightly-overlay.sh after ensure-ci-base-amd, and vLLM main has neither anymore, so build.sh now does that job.

Verification

• buildkite/tests/pipeline_generator/test_aiter_nightly.py runs generate() end to end on one step list with every step shape (unkeyed, A100 mirror, dind mirror, mirror without a device, AMD job with a mirror, inherited image-build, sharded jobs):

  • the nightly's exact job set, with and without NOAUTO: nothing deferred, no early upload, no dangling dependency, vLLM's build steps into rocm/vllm-dev, runtime sharding (Fix automatic sharding for standalone AMD jobs and selected tests #871) only for kept jobs, a docs-only diff not skipping it, recorders off even when set, and the selector record;
  • the stock pipeline with AITER_NIGHTLY unset or not 1;
  • a retry build of a kept job;
  • every guard above: an error annotation, no upload, no pipeline file.

  Each guard and filter was mutation-checked: removing it fails a test.

• pytest buildkite/tests: the same results as main plus the new tests.

• Rendering the real vLLM ci pipeline: with AITER_NIGHTLY unset, the output is byte-identical to main across 9 variants (default, NOAUTO, PR build, TORCH_NIGHTLY, AITER_NIGHTLY=0/true, early upload, retry builds of a native AMD job and a mirror). With AITER_NIGHTLY=1: 210 jobs, none deferred, no dangling dependencies, with or without NOAUTO.

• Buildkite ci run with AITER_NIGHTLY=1: pending

🤖 Generated with Claude Code

[josh-schell-amd]

blocking comment

[micah-wil]

This test looks trivial, we can remove it

[josh-schell-amd]

ack

[micah-wil]

Can consolidate this test with test_generate_fails_on_a_dropped_dependency

[micah-wil]

https://buildkite.com/vllm/ci/builds/96238/list <- AITER_NIGHTLY=1 (the TG failures there are expected and unrelated)

https://buildkite.com/vllm/ci/builds/96253/list <- AITER_NIGHTLY unset

cc @AndreasKaratzas can you take a look?

[josh-schell-amd]

Setting back to draft with changes to rocm base image and AITER_NIGHTLY being moved in there from vllm-project/vllm#61939 being merged, need to reevaluate this PR.