[Thangnguyenvn98]

Overview

Turns on automatic_shard for two long NVIDIA steps and six AMD mirrors that run longer than 20 minutes as one job. Picked by p90 duration over the last day on main. Only the flag changes; no commands or tests change.

Claims

• Each step below runs as parallel jobs planned from main's per-file timings, and each job runs only its share of the tests:

Step	Command shape	Planned shards
e2e-scheduling-1-gpu	one pytest file, split by test	2
amd-e2e-scheduling-1-gpu (mirror)	same	2
benchmarks-cli-test	one pytest over benchmarks/	2
amd-kernels-root-misc-test-b200 (mirror)	one pytest over a list of kernel test files, with --deselect	2
amd-spec-decode-eagle-nightly-b200 (optional mirror)	one pytest over a directory; its large file is split by test	2
amd-spec-decode-speculators-mtp-nightly-b200 (optional mirror)	two pytest commands over directories	2
amd-lm-eval-large-models-ep-2xb200 (optional mirror)	export setup, then pytest on the gsm8k eval with --config-list-file, split by test	2
amd-lm-eval-large-models-4xh100 (optional mirror)	export setup, then pytest on the lm-eval harness test with --config-list-file and --tp-size, split by test	2

• kernels-root-misc-test-b200, spec-decode-eagle-nightly-b200 and spec-decode-speculators-mtp-nightly-b200 are enrolled so their mirrors can shard (a mirror shards only when its base step is enrolled). Their own runs plan to one shard, so they stay one job.
• Every test runs exactly once across the shards, with the same environment as the single job. If planning fails, the step runs as one job, as it does today.

Validation

• Planned each step on this branch with ci-infra's runtime_shard.plan_step, using main's timings on the step's own queue.
• Rendered each step with the pipeline generator and ran every shard's commands against a mock test tree, as a PR build and with main's pytest -p ci_otel wrapper. Every test ran in exactly one shard, with the same env. The three single-shard base steps were confirmed to stay one job.
• /ci run exercises E2E Scheduling, Benchmarks CLI and the E2E Scheduling and Miscellaneous Kernels mirrors. The four optional mirrors need a build scoped to their keys.

Details

• Uneven or tight plans. amd-lm-eval-large-models-ep-2xb200's planner output is marked over budget: one eval test takes most of 20 minutes by itself, so its slowest shard sits near the target.
• Left out.
  • language-models-test-extended-generation and its mirror: each job first builds mamba-ssm and causal-conv1d from source, which every shard would repeat. They wait for [CI] Prebuild mamba-ssm and causal-conv1d in the CUDA test image for (H200) Language Models (Hybrid), (L4) Granite and Extended Generation #58795, which prebuilds both in the test image.
  • entrypoints-integration-api-server-generate's mirror: two of its files assert on caplog, the pattern that took entrypoints-integration-multimodal out of sharding.
  • benchmarks-cli-test's mirror runs under 20 minutes, so it stays one job.
  • Long steps whose commands aren't plain pytest (PyTorch Fullgraph, NixlConnector PD accuracy, Examples, Spec Decode MTP + Other) can't be planned.
• Where to look. The automatic_shard: true lines in .buildkite/test_areas/{engine,benchmarks,kernels,spec_decode,lm_eval}.yaml.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Thangnguyenvn98]

/ci run

[github-actions]

✅ Triggered Buildkite CI #95430 for commit 7cee01b4b627.