Repository navigation
[CI] Remove the retired A100 queue's steps; keep their live AMD mirrors standalone - #64602
Merged
AndreasKaratzas merged 2 commits intoOct 30, 2026
Conversation
Thangnguyenvn98
force-pushed
the
agent/retire-a100-queue-steps
branch
from
October 29, 2026 22:21
4c8b833 to
2c38457
Compare
Thangnguyenvn98
force-pushed
the
agent/retire-a100-queue-steps
branch
from
October 30, 2026 17:40
2c38457 to
799e989
Compare
6 tasks done
Thangnguyenvn98
force-pushed
the
agent/retire-a100-queue-steps
branch
from
October 30, 2026 17:46
799e989 to
180541b
Compare
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #98763 for commit ℹ️ This PR is 5 commits behind upstream |
…ndalone The A100 fleet is retired and the pipeline generator never emits steps whose device is a100 (ci-infra's buildkite_step.py retains them only for their AMD mirrors), yet the declarations still render on the sharding dashboard and count against automatic-shard enrollment. - Drop the two lm-eval-humming-a100 steps, the commented-out 4xA100 block, and the five config-a100-*.txt shard lists only they used. - batch-invariance-a100 and distributed-tests-4xa100 have live AMD mirrors ((MI250) and (MI355)); declare what their mirror.amd emitted as standalone AMD steps under the same amd- keys, labels, queues, commands and selection rules. Generated pipeline is unchanged job-for-job except a benign `export VLLM_TEST_GROUP_NAME=<key>` prefix the native AMD path adds. Supersedes vllm-project#60596. Co-authored-by: Kimi Code <noreply@moonshot.ai> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Thangnguyenvn98
force-pushed
the
agent/retire-a100-queue-steps
branch
from
October 30, 2026 20:19
251e5a8 to
26d7762
Compare
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #98799 for commit ℹ️ This PR is 1 commit behind upstream |
Thangnguyenvn98
requested review from
AndreasKaratzas,
Harry-Chen,
ZJY0516,
khluu and
vadiklyutiy
as code owners
October 30, 2026 22:39
AndreasKaratzas
approved these changes
Oct 30, 2026
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Remove every remaining
device: a100declaration — the fleet is retired and the generator never emits those steps, but they still render on ci.vllm.ai/sharding as never-run jobs that count against automatic-shard enrollment. The two steps with live AMD mirrors keep those mirrors as standalone AMD steps. Supersedes #60596, which went stale and now conflicts with main.Claims
(A100) LM Eval Humming Default/Act INT8 TEMPORARYsteps, the(A100) Batch Invarianceand(A100) Distributeddeclarations, the commented-out 4xA100 block inlm_eval.yaml, and the fivehumming/config-a100-*.txtshard lists only the Humming steps referenced.amd-distributed-tests-4xa100(:amd: (MI355) Distributed) andamd-batch-invariance-a100(:amd: (MI250) Batch Invariance).docker/Dockerfile.rocm_basediff was validated separately (Details, behavior change 2).:amd: (MI250) Batch Invariancemirror onbatch-invariance-a100, so merging it as-is would delete a running job; this PR preserves both mirrors as standalone steps instead (its own description names this as the needed follow-up).Validation
main's YAML and for this change, per-key diffed, under three triggering diffs: (a) a touch of the three changed test-area files — every job identical except the prefix below; (b)VLLM_CI_ONLY_STEP_KEYS=["amd-..."]to force each converted job to emit — same result; (c) adocker/Dockerfile.rocm_basetouch —amd-batch-invariance-a100still runs in both layouts,amd-distributed-tests-4xa100moves to the deferred list (behavior change 2).pipeline-generator --pipeline_config_path .buildkite/ci_config.yamlfrom a checkout at this PR's head and atmain, withget_ecr_cache_registrystubbed, and diff per-key.tools/pre_commit/check_buildkite_step_keys.pyandtools/pre_commit/check_test_tethering.pypass; no test loses its CI job.config-a100-*.txt./ci runplus a scoped build of the twoamd-keys exercises the converted steps for real. Recorder builds were code-read only:_kernrec_applies/_fnrec_appliessee the same outcome becauseKERNREC_SKIP_STEPSnames neither key.Details
amd--prefixed keys rather than getting fresh ones: it preserves job identity for retries, scoped builds and dashboard history, and native AMD steps withamd-keys already exist (e.g.amd-kernels-mi355).optional: trueandnum_devices: 4(Distributed), and the base's commands for Batch Invariance, whose mirror had none of its own.source_file_dependenciesare the union the generator applied to the mirror (base + mirror lists).export VLLM_TEST_GROUP_NAME=<key>to the command string; the mirror path didn't. Harmless: nothing on main reads that variable.docker/Dockerfile.rocm_baseeven when optional (_amd_mirror_should_run); native steps use plain dependency matching. For the (MI250) Batch Invariance step this PR restores the old behavior by addingdocker/Dockerfile.rocm_baseto itssource_file_dependencies, matching the other native AMD steps (misc.yaml). For the optional (MI355) Distributed step it is accepted instead of emulated —optional: trueshort-circuits file-dependency selection, so such PRs get it at the nightly or via manual unblock, same as every other native optional AMD step.humming/Qwen3-4B-mixed-quant-RTN-humming.yamlis kept although only the deleted A100 shard lists referenced it, in case it should move to an H100 list.disaggregated.yaml(never parsed), and ci-infra's now-unused A100 queue constants.AI assistance: this change was written by an agent (Kimi Code) and peer-reviewed by a second agent; the human submitter has reviewed the diff and the pipeline-equality evidence above.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)