Skip to content

[CI] Remove the retired A100 queue's steps; keep their live AMD mirrors standalone - #64602

Merged
AndreasKaratzas merged 2 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/retire-a100-queue-steps
Oct 30, 2026
Merged

AndreasKaratzas merged 2 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/retire-a100-queue-steps

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Remove every remaining device: a100 declaration — the fleet is retired and the generator never emits those steps, but they still render on ci.vllm.ai/sharding as never-run jobs that count against automatic-shard enrollment. The two steps with live AMD mirrors keep those mirrors as standalone AMD steps. Supersedes #60596, which went stale and now conflicts with main.

Claims

  • Removes the (A100) LM Eval Humming Default/Act INT8 TEMPORARY steps, the (A100) Batch Invariance and (A100) Distributed declarations, the commented-out 4xA100 block in lm_eval.yaml, and the five humming/config-a100-*.txt shard lists only the Humming steps referenced.
  • The two live AMD mirrors keep running as standalone steps, under the same emitted keys, labels, queues, commands and timeouts: amd-distributed-tests-4xa100 (:amd: (MI355) Distributed) and amd-batch-invariance-a100 (:amd: (MI250) Batch Invariance).
  • Under a diff touching the three changed test-area files, the generated pipeline is unchanged job-for-job except one benign prefix (Details): no job is added, removed or altered. Selection under a docker/Dockerfile.rocm_base diff was validated separately (Details, behavior change 2).
  • Difference from [CI] Remove A100 steps that the pipeline generator no longer emits #60596, why this is not a plain duplicate: it predates the now-live :amd: (MI250) Batch Invariance mirror on batch-invariance-a100, so merging it as-is would delete a running job; this PR preserves both mirrors as standalone steps instead (its own description names this as the needed follow-up).

Validation

  • Generated the full Buildkite pipeline with ci-infra's pipeline generator (ci-infra@9d2935b5) for main's YAML and for this change, per-key diffed, under three triggering diffs: (a) a touch of the three changed test-area files — every job identical except the prefix below; (b) VLLM_CI_ONLY_STEP_KEYS=["amd-..."] to force each converted job to emit — same result; (c) a docker/Dockerfile.rocm_base touch — amd-batch-invariance-a100 still runs in both layouts, amd-distributed-tests-4xa100 moves to the deferred list (behavior change 2).
  • Repro: run pipeline-generator --pipeline_config_path .buildkite/ci_config.yaml from a checkout at this PR's head and at main, with get_ecr_cache_registry stubbed, and diff per-key.
  • tools/pre_commit/check_buildkite_step_keys.py and tools/pre_commit/check_test_tethering.py pass; no test loses its CI job.
  • Repo-wide grep: no remaining reference to any deleted config-a100-*.txt.
  • No GPU run locally; a /ci run plus a scoped build of the two amd- keys exercises the converted steps for real. Recorder builds were code-read only: _kernrec_applies/_fnrec_applies see the same outcome because KERNREC_SKIP_STEPS names neither key.

Details

  • The converted steps keep their amd--prefixed keys rather than getting fresh ones: it preserves job identity for retries, scoped builds and dashboard history, and native AMD steps with amd- keys already exist (e.g. amd-kernels-mi355).
  • Each standalone step carries the fields its mirror inherited from the base step explicitly: optional: true and num_devices: 4 (Distributed), and the base's commands for Batch Invariance, whose mirror had none of its own. source_file_dependencies are the union the generator applied to the mirror (base + mirror lists).
  • Behavior change 1 (commands): the native AMD path prepends export VLLM_TEST_GROUP_NAME=<key> to the command string; the mirror path didn't. Harmless: nothing on main reads that variable.
  • Behavior change 2 (selection): mirrors auto-run on PRs that change docker/Dockerfile.rocm_base even when optional (_amd_mirror_should_run); native steps use plain dependency matching. For the (MI250) Batch Invariance step this PR restores the old behavior by adding docker/Dockerfile.rocm_base to its source_file_dependencies, matching the other native AMD steps (misc.yaml). For the optional (MI355) Distributed step it is accepted instead of emulated — optional: true short-circuits file-dependency selection, so such PRs get it at the nightly or via manual unblock, same as every other native optional AMD step.
  • Reviewer call (carried over from [CI] Remove A100 steps that the pipeline generator no longer emits #60596): humming/Qwen3-4B-mixed-quant-RTN-humming.yaml is kept although only the deleted A100 shard lists referenced it, in case it should move to an H100 list.
  • Out of scope: the commented-out, already-disabled NixlConnector A100 block in disaggregated.yaml (never parsed), and ci-infra's now-unused A100 queue constants.

AI assistance: this change was written by an agent (Kimi Code) and peer-reviewed by a second agent; the human submitter has reviewed the diff and the pipeline-equality evidence above.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

@mergify mergify Bot added the ci/build label Oct 29, 2026
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/retire-a100-queue-steps branch from 4c8b833 to 2c38457 Compare October 29, 2026 22:21
@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI] Retire the A100 queue's last declared step; keep its MI355 mirror standalone [CI] Remove the retired A100 queue's steps; keep their live AMD mirrors standalone Oct 30, 2026
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/retire-a100-queue-steps branch from 2c38457 to 799e989 Compare October 30, 2026 17:40
@mergify mergify Bot added the rocm Related to AMD ROCm label Oct 30, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Oct 30, 2026
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/retire-a100-queue-steps branch from 799e989 to 180541b Compare October 30, 2026 17:46
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98763 for commit 251e5a867b5a.

ℹ️ This PR is 5 commits behind upstream main. Merged upstream main into this branch as 251e5a867b5a before starting CI. Pull the branch before pushing more changes.

…ndalone

The A100 fleet is retired and the pipeline generator never emits steps
whose device is a100 (ci-infra's buildkite_step.py retains them only
for their AMD mirrors), yet the declarations still render on the
sharding dashboard and count against automatic-shard enrollment.

- Drop the two lm-eval-humming-a100 steps, the commented-out 4xA100
  block, and the five config-a100-*.txt shard lists only they used.
- batch-invariance-a100 and distributed-tests-4xa100 have live AMD
  mirrors ((MI250) and (MI355)); declare what their mirror.amd emitted
  as standalone AMD steps under the same amd- keys, labels, queues,
  commands and selection rules.

Generated pipeline is unchanged job-for-job except a benign
`export VLLM_TEST_GROUP_NAME=<key>` prefix the native AMD path adds.
Supersedes vllm-project#60596.

Co-authored-by: Kimi Code <noreply@moonshot.ai>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/retire-a100-queue-steps branch from 251e5a8 to 26d7762 Compare October 30, 2026 20:19
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98799 for commit fe3a6901a5c3.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as fe3a6901a5c3 before starting CI. Pull the branch before pushing more changes.

@Thangnguyenvn98
Thangnguyenvn98 marked this pull request as ready for review October 30, 2026 22:39
@Thangnguyenvn98
Thangnguyenvn98 requested a review from a team October 30, 2026 22:39

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@AndreasKaratzas
AndreasKaratzas merged commit d2d550c into vllm-project:main Oct 30, 2026
76 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants