[CI][ROCm] Add non-blocking engine and model executor GPU coverage

[andyluo7]

Summary

• add the R2-01 AMD ready job for GPU-backed tests/engine/ and tests/model_executor/ coverage on mi300_1
• keep the new job NonBlocking, bound it to 20 minutes, and explicitly exclude cards_2 through cards_8
• run collection and execution once; fail closed on zero collection and publish node IDs through verbose pytest output and JUnit, together with skips, per-test durations, and a concise summary
• exclude the CUDA/L4-only CosyVoice3 ragged-flow parity test from this ROCm lane after it exceeded the R2-01 runtime bound while cold-JITing AITER kernels
• leave CUDA pipeline definitions unchanged

The CUDA ready job also includes tests/worker/test_batched_omni_output.py. This first ROCm burn-in is intentionally limited to the two R2-01 directories; the worker test remains a documented follow-up rather than an implicit parity claim.

Validation

Base: 58adeec05f151e18542323cb4644b009b83cfef3 (includes merged #6978 and #7395)

Exact head: 6d8615f7e51af8e8e5b2468d4ecf059e4124b978

• git range-diff confirms that the original three-commit R2-01 series is unchanged by the rebase onto current main
• reviewer follow-up c978fb70 removes the separate --collect-only invocation, so the GPU-backed tree is collected only once
• runtime follow-up 6d8615f7 excludes only test_code2wav_streaming_batch_matches_ragged_flow_numerics; that test declares CUDA/L4 hardware and remains covered by the CUDA ready Engine&Model Executor Test
• the single pytest -s -v -ra run retains node IDs in pytest.log and JUnit while preserving zero-collection failure, skip reporting, per-test durations, and the 20-minute step bound
• focused Buildkite contract test: 2 passed
• broader tests/buildkite suite with the Torch-dependent package-discovery test excluded: 137 passed
• changed-file pre-commit: passed, including YAML, Ruff, mypy, marker, SPDX, and Buildkite schema checks
• git diff --check: passed
• the normal local pytest startup remains unavailable on this Mac because repository fixtures require Torch; the isolated contract test was run with the conftest boundary restricted to tests/buildkite

Exact-head ReviewBot re-review completed on 6d8615f7 with no actionable findings.

AMD hardware burn-in

The following builds are retained as historical evidence but do not count for the new exact head:

• #11716 on 6efb4aed: 578 selected tests passed, but native-pod artifacts were not uploaded.
• #11720 and #11724 on 818221ed: 578 passed with 2,814 deselected; required artifacts uploaded.
• #11736 and #11751 on a3d17b0e: two clean passes, 578 passed with 2,814 deselected, within the 20-minute bound.
• #11797 on c978fb70: the aggregate build passed because R2-01 is non-blocking, but R2-01 itself timed out after selecting 579 tests. It logged 372 passes and no test failures before a newly introduced CUDA/L4-only CosyVoice3 test spent the remaining budget cold-JITing AITER kernels; only pytest.log was uploaded.

Current burn-in status: 2 of 2 exact-head passes complete.

• #11843 on exact head 6d8615f7e51af8e8e5b2468d4ecf059e4124b978: R2-01 exited 0 with no timeout or soft failure; 578 passed, 2,851 deselected, 18 warnings in 501.52s (8m21s); JUnit reports 578 tests with zero failures, errors, or skips.
• #11849 on the same exact head: R2-01 exited 0 with no timeout or soft failure; 578 passed, 2,851 deselected, 18 warnings in 493.07s (8m13s); JUnit reports 578 tests with zero failures, errors, or skips.

Both aggregate builds passed. Each target retained readable pytest.log, pytest.xml, and pytest-summary.txt artifacts; the verbose log and JUnit carry the selected node IDs without a duplicate collection pass.

Rollout gate

Promotion-ready evidence is complete:

1. exact-head ReviewBot re-review had no actionable findings;
2. two terminal AMD ready builds on 6d8615f7e51af8e8e5b2468d4ecf059e4124b978 passed the R2-01 target;
3. each run had non-zero collection, completed within 20 minutes, and uploaded the three readable artifacts.

The job remains NonBlocking; any later blocking-grade promotion should follow the broader ROCm CI burn-in policy.

Tracking: #5731

[hsliuustc0106]

This PR touches .buildkite/amd/, tests/buildkite/ (2 files). Based on CODEOWNERS coverage of the changed files, the most-related reviewers appear to be:

@yenuo26 @NickCao @congw729

Could one of you take a look when you get a chance? Thanks!

[andyluo7]

R2-01's hardware promotion gate is satisfied on exact head 818221ed5dcf6df310b2b838a1f6bc633096e38c:

• AMD #11720: 578 passed, 2,814 deselected, 18 warnings, 490.45 seconds.
• AMD #11724: 578 passed, 2,814 deselected, 18 warnings, 477.45 seconds; the full build passed 20/20 jobs with no failures or soft failures.

Both runs uploaded the four required artifacts: collected-nodeids.txt, pytest.log, pytest.xml, and pytest-summary.txt. The PR remains AMD-only; no CUDA path changed.

The visible Intel failure is from accidentally launched Intel #8115, not from this AMD-only change. Its XPU job exited during collection against stale Intel vLLM APIs (including missing compute_layout_strides, vllm.v1.kv_cache_layout, and vllm.entrypoints.generate.base.protocol). It is unrelated to the R2-01 diff and AMD validation above.

Marking the PR ready and requesting review from @yenuo26, @NickCao, and @congw729.

[vllm-omni-review-bot]

This PR was classified as CI work.

CI owner: @yenuo26 @congw729 @NickCao

Routing: @yenuo26 via semantic router, CI owner, CODEOWNERS; @congw729 via CODEOWNERS; @NickCao via CODEOWNERS

@andyluo7, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[andyluo7]

Author self-review completed on exact head 818221ed5dcf6df310b2b838a1f6bc633096e38c.

• Scope is limited to the AMD ready pipeline and its Buildkite invariant test; no CUDA path is modified.
• Both collection and execution use the same tests/engine/ and tests/model_executor/ selection, explicitly excluding cards_2 through cards_8.
• The native AMD runner uses pipefail and preserves pytest's nonzero/no-tests exit status, so artifact logging through tee does not mask failures.
• Artifact paths, non-blocking grade, mi300_1 assignment, and single-GPU selection are covered by the added test.
• AMD #11720 and #11724 each passed 578 tests on this exact head and uploaded all four required artifacts.
• The CUDA worker test remains an intentional, documented follow-up.

No blocking concerns found in self-review.

@vllm-omni-review-bot, please review.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Direct under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This PR adds a NonBlocking AMD ready job (R2-01) on mi300_1 that runs GPU-marked tests/engine/ and tests/model_executor/ coverage while excluding multi-GPU cards_2–cards_8 cases. It writes collection/result artifacts under the Buildkite checkout path so native-pod runs can upload them, and fails closed when pytest collects zero tests (no VLLM_CI_ALLOW_NO_TESTS). CUDA pipelines are untouched; omitting tests/worker/test_batched_omni_output.py is an explicit first-slice scope choice, pinned by a new Buildkite invariant test.

Change flow

flowchart TD
  A["[EXISTING] AMD ready pipeline<br/>mi300_1 + run-amd-test.sh"]:::existing --> B["[CHANGED] test-amd-ready.yml<br/>R2-01 NonBlocking job"]:::changed
  B --> C["[NEW] collect+run pytest<br/>engine + model_executor<br/>exclude cards_2..8"]:::new
  C --> D["[EXISTING] pipefail runner<br/>preserves pytest exit / exit 5"]:::existing
  B --> E["[NEW] test_amd_ready_r2_01.py<br/>grade/pool/marker/artifact pins"]:::new
  C --> F["[CHANGED] checkout artifacts<br/>nodeids/log/junit/summary"]:::changed
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

See inline comments below.

[andyluo7]

Addressed the runtime-bound finding in a3d17b0e and resolved the thread after focused validation. @vllm-omni-review-bot, please re-review the current head.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This PR adds a NonBlocking AMD ready burn-in job (R2-01) on mi300_1 that collects and runs GPU-marked tests/engine/ and tests/model_executor/ coverage while excluding multi-GPU cards_2–cards_8. Artifacts are written under BUILDKITE_BUILD_CHECKOUT_PATH so native-pod uploads work, zero collection stays fail-closed via unset VLLM_CI_ALLOW_NO_TESTS plus the existing pipefail runner, and the step is bounded with timeout_in_minutes: 20. CUDA pipelines are unchanged; omitting the CUDA worker batched-output test is an explicit first-slice choice pinned by a new Buildkite invariant test.

Change flow

flowchart TD
  A["[EXISTING] AMD ready upload<br/>run-amd-test.sh pipefail"]:::existing --> B["[CHANGED] test-amd-ready.yml"]:::changed
  B --> C["[NEW] R2-01 NonBlocking job<br/>mi300_1 + timeout_in_minutes 20"]:::new
  C --> D["[NEW] collect+run pytest<br/>engine + model_executor<br/>exclude cards_2..8"]:::new
  C --> E["[NEW] checkout artifacts<br/>nodeids/log/junit/summary"]:::new
  B --> F["[NEW] test_amd_ready_r2_01.py<br/>grade/marker/artifact pins"]:::new
  D --> G["[EXISTING] exit 5 fail-closed<br/>without ALLOW_NO_TESTS"]:::existing
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

No actionable findings.

[andyluo7]

R2-01's promotion gate is satisfied on exact head a3d17b0e668bffd9a56ee1b6ea4ea86bf1c68954 after the 20-minute timeout follow-up:

• AMD #11736: 578 passed, 2,814 deselected, 18 warnings in 500.70 seconds; full build passed 20/20 jobs.
• AMD #11751: 578 passed, 2,814 deselected, 18 warnings in 478.42 seconds; full build passed 20/20 jobs.

Both exact-head runs completed without failures, timeouts, or soft failures, and each uploaded four readable artifacts: collected-nodeids.txt, pytest.log, pytest.xml, and pytest-summary.txt.

ReviewBot's exact-head re-review reported no actionable findings. The diff remains AMD-only with no CUDA-path change. The remaining gate is maintainer review.

[andyluo7]

Refresh complete on exact head c978fb70e510129323d1e11ca8c104b5411b45d2, rebased onto post-#6978 main at 58adeec05f151e18542323cb4644b009b83cfef3.

• The original three R2-01 commits are patch-equivalent under git range-diff.
• The maintainer follow-up removes the duplicate --collect-only pass and leaves one verbose pytest execution.
• Zero collection still fails closed; pytest.log and JUnit retain node IDs, with durations and the concise summary preserved.
• Focused contract test: 2 passed.
• Changed-file pre-commit, Buildkite schema validation, DCO, and git diff --check: passed.

The prior AMD runs are now historical evidence. The promotion gate is reset to 0/2 for this head.

@vllm-omni-review-bot, please re-review the current exact head.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

Adds a NonBlocking AMD ready burn-in step (R2-01) on mi300_1 that runs one GPU-scoped pytest pass over tests/engine/ and tests/model_executor/, excluding multi-GPU cards_2–cards_8, with a 20-minute step timeout. The job writes checkout-relative pytest.log, JUnit, and a short summary under artifacts/rocm-r2-01/, and relies on the existing native AMD pipefail runner plus unset VLLM_CI_ALLOW_NO_TESTS to fail closed on zero collection. CUDA pipelines are untouched; a new Buildkite contract test pins grade, pool, timeout, single invocation, marker selection, and artifact paths.

Change flow

flowchart TD
  A["[EXISTING] run-amd-test.sh<br/>pipefail + exit-5 gate"]:::existing --> B["[CHANGED] test-amd-ready.yml"]:::changed
  B --> C["[NEW] R2-01 NonBlocking job<br/>mi300_1 timeout_in_minutes 20"]:::new
  C --> D["[NEW] single pytest -s -v -ra<br/>engine + model_executor<br/>exclude cards_2..8"]:::new
  C --> E["[NEW] artifacts/rocm-r2-01<br/>log + junit + summary"]:::new
  B --> F["[NEW] test_amd_ready_r2_01.py<br/>contract pins"]:::new
  D --> G["[EXISTING] soft_fail NonBlocking<br/>AMD ready upload"]:::existing
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

No actionable findings.

[andyluo7]

Follow-up published on exact head 6d8615f7e51af8e8e5b2468d4ecf059e4124b978.

AMD #11797 completed successfully at the aggregate level because R2-01 is non-blocking, but the R2-01 target itself timed out after 20 minutes:

• 3,429 collected / 2,850 deselected / 579 selected
• 372 passes and no test failures logged before termination
• timeout occurred in test_code2wav_streaming_batch_matches_ragged_flow_numerics while cold-JITing multiple AITER kernels
• only pytest.log was produced, so #11797 does not count toward the R2-01 promotion gate

That test was the sole newly selected item relative to the prior 578-test passes, came from upstream commit 9edc1b46, and declares CUDA/L4 hardware. Commit 6d8615f7 now excludes only that test from the ROCm R2-01 lane; the CUDA ready Engine&Model Executor Test remains its owner.

Local validation on the new head:

• focused R2-01 contract: 2 passed
• broader Buildkite contracts: 137 passed
• changed-file pre-commit and Buildkite schema validation: passed
• git diff --check: passed

The exact-head hardware gate is reset to 0/2.

@vllm-omni-review-bot, please review the current exact head.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This PR adds a NonBlocking AMD ready burn-in job (R2-01) on mi300_1 that runs a single GPU-scoped pytest over tests/engine/ and tests/model_executor/, excluding multi-GPU cards_2–cards_8 and one CUDA/L4 CosyVoice3 ragged-flow parity test that previously blew the 20-minute bound while cold-JITing AITER kernels. Results go to checkout-relative artifacts/rocm-r2-01/ (pytest.log, JUnit, summary); zero collection stays fail-closed via the native AMD pipefail runner and unset VLLM_CI_ALLOW_NO_TESTS. CUDA pipelines are unchanged; a new Buildkite contract test pins grade, pool, timeout, single invocation, selection, and artifacts.

Change flow

flowchart TD
  A["[EXISTING] run-amd-test.sh<br/>pipefail + exit-5 gate"]:::existing --> B["[CHANGED] test-amd-ready.yml"]:::changed
  B --> C["[NEW] R2-01 NonBlocking job<br/>mi300_1 timeout_in_minutes 20"]:::new
  C --> D["[NEW] single pytest -s -v -ra<br/>engine + model_executor<br/>exclude cards_2..8 + CosyVoice3 -k"]:::new
  C --> E["[NEW] artifacts/rocm-r2-01<br/>log + junit + summary"]:::new
  B --> F["[NEW] test_amd_ready_r2_01.py<br/>contract pins"]:::new
  D --> G["[EXISTING] soft_fail NonBlocking<br/>AMD ready upload"]:::existing
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

No actionable findings.

[andyluo7]

Exact-head AMD promotion evidence

Exact head: 6d8615f7e51af8e8e5b2468d4ecf059e4124b978

• ReviewBot exact-head review: completed with no actionable findings.
• AMD #11843: aggregate passed. R2-01 job 01a0a0dc-7ec8-4b67-ab36-1a7ed6ea8991 exited 0, with no timeout or soft failure; 578 passed, 2,851 deselected, 18 warnings in 501.52s (8m21s). JUnit: 578 tests, zero failures/errors/skips.
• AMD #11849: aggregate passed. R2-01 job 01a0a10e-b02e-4b99-9dc5-e6bab403ccd6 exited 0, with no timeout or soft failure; 578 passed, 2,851 deselected, 18 warnings in 493.07s (8m13s). JUnit: 578 tests, zero failures/errors/skips.
• Both target jobs selected the intended 578 tests and retained readable pytest.log, pytest.xml, and pytest-summary.txt artifacts.
• All current GitHub checks are green.

Gate: 2 of 2 exact-head passes complete. R2-01 remains NonBlocking as designed; this evidence supports promotion readiness without changing the blocking policy.

@congw729 @NickCao, the exact-head gate is complete and all current checks are green. Please provide human approval when ready.