Skip to content

[CI] Enroll four Nightly steps and three AMD mirrors in automatic sharding - #64174

Merged
khluu merged 3 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/automatic-shard-batch-11
Nov 2, 2026
Merged

khluu merged 3 commits into
vllm-project:mainfrom
Thangnguyenvn98:agent/automatic-shard-batch-11

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Enrolls four Nightly steps and three of their AMD mirrors in automatic_shard, continuing the earlier enrollment batches (#61999, #62039, #64115). This PR only sets the flag; it changes no tests or commands. A step that fits in one job runs as one job, unchanged, and splits without another CI change once its tests outgrow it.

Claims

Step Planned shards AMD mirror
multi-modal-models-extended-pooling 3 2
lm-eval-deepseek-v32-tp-8xh200 1 1
lm-eval-deepseek-r1-tp-8xh200 1 1
lm-eval-hisparse-nightly-4xb200 1 none
  • Each job runs the same tests, environment and timeout as on main. One gsm8k config takes about as long as a whole shard, so the planner keeps the lm-eval steps whole for now.

Validation

  • Planned each step and mirror with ci-infra main's runtime_shard.plan_step against main's timings, and rendered each with the pipeline generator as a PR build and in main's OTel mode.
  • /ci run (98071) on this head, plus a build scoped to the three AMD mirrors it leaves behind the block step (98079): every job passed. Extended Pooling ran its planned 3 shards (mirror: 2) with the same pass and skip counts as main; the lm-eval steps ran as one job each, as on main.

Details

  • Extended Pooling is back. An earlier build of this PR failed it on the ColUltraVBERT checkpoint; main fixed that in [CI][Bugfix] Pin the ColUltraVBERT test checkpoint to the revision vLLM supports #64189 and its nightly passes the step again.
  • Cost. Extended Pooling's machine time rises by about a third: each extra shard repeats job setup and the model loads its test files share. In exchange its wall time halves. It runs nightly, so we keep the shorter wall time.
  • Dropped from this batch. lm-eval-humming-b200-act fails on main from a regression there. It comes back in a later batch once main passes it.
  • Timeouts unchanged (human). timeout_in_minutes stays per job. Fallback-aware timeouts belong in a central timeout setting.
  • AI assistance. Claude Code prepared the change and this description; the submitter reviews every changed line.

🤖 Generated with Claude Code


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #97804 for commit 4756f6bab452.

@mergify mergify Bot added ci/build rocm Related to AMD ROCm labels Oct 28, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Oct 28, 2026
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/automatic-shard-batch-11 branch 2 times, most recently from a187428 to 9779570 Compare October 28, 2026 02:26
@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI] Enroll five Nightly steps and three AMD mirrors in automatic sharding [CI] Enroll three Nightly steps and two AMD mirrors in automatic sharding Oct 28, 2026
@Thangnguyenvn98 Thangnguyenvn98 changed the title [CI] Enroll three Nightly steps and two AMD mirrors in automatic sharding [CI] Enroll four Nightly steps and three AMD mirrors in automatic sharding Oct 28, 2026
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/automatic-shard-batch-11 branch from 9779570 to 55b6a87 Compare October 28, 2026 19:34
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98071 for commit 55b6a8728932.

@Thangnguyenvn98
Thangnguyenvn98 marked this pull request as ready for review October 28, 2026 21:33
@Thangnguyenvn98
Thangnguyenvn98 requested a review from a team October 28, 2026 21:33

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Thangnguyenvn98 and others added 3 commits October 30, 2026 14:40
…rding

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Both fail on main right now, so they wait until main is green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
main fixed the ColUltraVBERT failure that dropped it (vllm-project#64189), and main's
nightly passes the step again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/automatic-shard-batch-11 branch from 55b6a87 to 6dc1ef0 Compare October 30, 2026 18:41
@Thangnguyenvn98

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98767 for commit 6dc1ef0f957f.

@khluu
khluu merged commit 5da6f2e into vllm-project:main Nov 2, 2026
51 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants