Skip to content

[CI] Enroll five steps and four AMD mirrors in automatic sharding - #64355

Merged
khluu merged 2 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/automatic-shard-batch-14
Nov 2, 2026
Merged

khluu merged 2 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/automatic-shard-batch-14

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Enrolls five steps and four of their AMD mirrors in automatic_shard, continuing the earlier enrollment batches (#64115, #64174, #64181). Each runs as one job today, unchanged, and splits without another CI change once its tests outgrow one job. One command changes: DeepSeek V4 Kernels (H100) names the single file its glob matched, so the shard planner can find it.

Claims

Step Plan AMD mirror
language-models-test-ppl 1 shard 1 shard
multi-modal-models-standard-1-qwen2 1 shard 1 shard
engine 1 shard 1 shard
deepseek-v4-kernel-test-h100 1 shard 1 shard
kernels-deepgemm-test-h100 1 shard none
  • Each job runs the same tests, environment and timeout as on main.

Validation

  • Planned each step and mirror with ci-infra main's planner against main's timings: every plan is one job, within main's machine time.
  • Every step and mirror passes on recent main runs, nightly included.
  • kernels/test_fused_deepseek_v4_*.py matches only test_fused_deepseek_v4_qnorm_rope_kv_insert.py today, the one test file the step's source_file_dependencies lists. In build 98080 that command collected and passed the same 215 tests as on main.
  • /ci run on the previous head (98766) plus a build scoped to the AMD mirrors it leaves behind the block step (98793): every job passed. A fresh /ci run follows on this head.

Details

  • Left out: steps the generator can't shard. It only shards a step whose commands after the first pytest are all plain pytest. An earlier revision also enrolled Python 3.14 Smoke, Python-only Installation (shell scripts) and Plugin Integration (pip install between pytest commands), with their mirrors; the generator ignored the flag on them, so this PR no longer touches them.
  • DeepSeek V4 glob. The planner reads the files a pytest command names from the checkout and doesn't expand shell globs, so with the glob the step got no plan. The B200 DeepSeek V4 step keeps its glob; it isn't in this batch.
  • Engine Core caplog. test_arg_utils.py, test_config.py and test_logger.py assert on logs; none starts an engine (test_logger.py's LLM(...) raises before engine init), and the step stays one job in its current order.
  • Timeouts unchanged. timeout_in_minutes stays per job. Fallback-aware timeouts belong in a central timeout setting.

🤖 Generated with Claude Code


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98080 for commit 1367a9781048.

@mergify mergify Bot added ci/build rocm Related to AMD ROCm labels Oct 28, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Oct 28, 2026
@Thangnguyenvn98
Thangnguyenvn98 marked this pull request as ready for review October 28, 2026 21:31
@Thangnguyenvn98
Thangnguyenvn98 requested a review from a team October 28, 2026 21:31

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98766 for commit 73207f4fd6fd.

Thangnguyenvn98 and others added 2 commits October 30, 2026 20:37
Also name the one file that DeepSeek V4 Kernels' glob matches, so the
shard planner can find it in the checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The pipeline generator only shards a step whose commands after the first
pytest are all plain pytest, so on script, pip-install, export and
find|xargs steps the flag is ignored and the step runs as one job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/automatic-shard-batch-14 branch from 73207f4 to 7799a54 Compare October 31, 2026 00:38
@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI] Enroll eight steps and six AMD mirrors in automatic sharding [CI] Enroll five steps and four AMD mirrors in automatic sharding Oct 31, 2026
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98860 for commit 7799a545eb62.

@khluu
khluu merged commit b2fcbdd into vllm-project:main Nov 2, 2026
92 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants