[andyluo7]

Purpose

Add a small, experimental AMD/ROCm MI300 full_model nightly vertical slice while AMD ready/merge coverage continues to burn in.

This PR:

• adds .buildkite/amd/test-amd-nightly.yml with four MI300/Qwen3-Omni jobs:
  • full-function expansion
  • Daily-Omni / Seed-TTS accuracy
  • documentation examples
  • an AITER-on targeted smoke
• keeps every new nightly leaf non-blocking during burn-in
• maps repository-side PR label nightly-test and scheduled main builds with NIGHTLY=1 to the AMD nightly suite
• allows ready, merge-test, and nightly-test suites to be combined behind one AMD image build
• registers the AMD nightly YAML with shared skip-CI classification and documents the external trigger boundary

This is intentionally not the CUDA L4 configuration copied wholesale. Performance is deferred until an MI300 baseline and measurement protocol exist; this PR does not import H100 thresholds. AMD coverage publication and broader model/hardware expansion remain follow-ups.

Rebase and trigger status

This PR is based directly on main at d3493384cfc6c4b9ba1fc4e534e91fe588ecb4f1; the exact head is aa5b535c6927ea2946c8c5c72faaa17c493b7dd5. This ancestry includes the corrected ROCm vLLM 0.29 image from #7395.

AMD label handling has two layers:

• amd-test admits the external AMD Buildkite pipeline.
• nightly-test makes the repository bootstrap select only NIGHTLY_TESTS:test-amd-nightly.yml.

For AMD-only nightly validation, the external Buildkite condition must accept a label event when the complete PR label set contains both amd-test and nightly-test. The event should be accepted when either member of the pair is applied last, making the trigger order-independent. A manual exact-head build with DEBUG_TEST_YAML=nightly validated the repository lane independently of that external trigger change.

Related to #5731.

Test Plan

vLLM Version: N/A (CI configuration only)

vLLM-Omni Commit: aa5b535c6927ea2946c8c5c72faaa17c493b7dd5

• focused AMD selector/bootstrap and artifact-staging pytest coverage
• complete changed-file pre-commit suite
• shellcheck .buildkite/amd/scripts/bootstrap-amd-omni.sh
• bash -n .buildkite/amd/scripts/bootstrap-amd-omni.sh
• Buildkite schema validation for .buildkite/amd/test-amd-nightly.yml
• git diff --check
• exact-head external AMD Buildkite bootstrap and child-pipeline inspection
• verify that both Accuracy JSON files are retained by Buildkite's artifact hook

Test Result

• On exact head aa5b535c, 77 focused tests passed. Changed-file pre-commit, YAML validation, the artifact-copy simulation, and git diff --check passed.
• AMD Buildkite #11712 ran on exact head aa5b535c with DEBUG_TEST_YAML=nightly:
  • Bootstrap selected only NIGHTLY_TESTS:test-amd-nightly.yml and rendered exactly four non-blocking amd_mi300_2 leaves.
  • The image build passed with vLLM 0.29.0, TorchCodec 0.10.0a0, a successful final VideoDecoder import, and digest sha256:09e50771c1cee2cef8c6ae0cd8e41db9c775ba8c128499b79a3c19b6c63ad80d.
  • Function Expansion: 26 passed; the earlier test_speaker_002[async_chunk] mismatch did not reproduce.
  • Accuracy: 2 passed. Daily-Omni evaluated all 1,197 samples with zero request/parse failures and accuracy 0.7184628237259816. Seed-TTS evaluated all 1,088 samples with zero request, PCM, or ASR/WER failures; mean WER was 0.33326678996457526, below the unchanged 0.35 maximum.
  • Documentation Examples: 7 passed.
  • AITER-on Smoke: 3 passed, with VLLM_ROCM_USE_AITER=1 and module_aiter_core.so loaded.
  • The post-command hook copied both generated result files into the checkout and Buildkite uploaded both successfully: Daily-Omni JSON and Seed-TTS JSON.
  • Bootstrap, image build, and all four GPU jobs finished with exit status 0.
• Build #11711 exposed Buildkite upload-time interpolation of the artifact-staging runtime path. Exact head aa5b535c defers those variables to job runtime and adds regression coverage; #11712 confirms the corrected pipeline uploads and runs end to end.
• Earlier burn-in observations remain recorded rather than normalized away: #11709 measured Seed-TTS mean WER 0.3379, while #11646 measured 0.376480. The 0.35 threshold is unchanged, and the four AMD nightly leaves remain non-blocking during burn-in.

[andyluo7]

Self-review completed on exact head e415518197e3e51e75b7b520fe0d46c1aee2d660.

• Scope is limited to the AMD selector/bootstrap, the new AMD nightly data file, shared skip-CI registration, focused tests, and CI documentation. No CUDA or NPU execution path is changed.
• main remains AMD L3 by default; main with NIGHTLY=1 selects L4 only. PR labels map ready → L2, merge-test → L3, and nightly-test → L4, with multiple labels sharing one image build. The legacy unlabeled/amd-test L2 fallback remains intact.
• The initial L4 vertical slice has four explicit Qwen3-Omni jobs: function expansion, accuracy, documentation examples, and an AITER-on smoke. All render as soft_fail: true on amd_mi300_2 during burn-in.
• Performance is intentionally deferred rather than importing CUDA/H100 baselines without MI300 measurements. Coverage publication and broader model/hardware expansion also remain follow-ups.
• The legacy MI325 pytest marker is retained only for current ROCm test collection; the AMD template independently selects the current MI300 queue.
• Diff-scoped precheck found no new production code, dependency, example-policy, code-quality, or simplification issue. Merge simulation with current main is clean. The branch is intentionally stacked on [CI][ROCm] Match CUDA/NPU L2/L3 label routing #6966, so review of this PR should focus on the final commit until [CI][ROCm] Match CUDA/NPU L2/L3 label routing #6966 merges.

Validation:

• targeted AMD selector/nightly/skip-CI tests: 56 passed
• changed-file pre-commit: all hooks passed
• Bash syntax and changed AMD bootstrap ShellCheck: passed
• MiniJinja 2.3.1 render: four non-blocking amd_mi300_2 jobs with the expected commands, timeouts, GPU count, and accuracy artifact path
• DCO: passed

@yenuo26 @congw729, I cannot add repository labels with this account (AddLabelsToLabelable permission was denied). Could one of you apply amd-test and nightly-test together for exact-head runtime proof? amd-test admits the external AMD Buildkite build under the current condition; nightly-test makes the repository selector upload only the new L4 suite. Please keep both labels present for the triggering event.

Exact-head MI300 results are still pending and are not claimed by this self-review.

[andyluo7]

Restacked the draft onto #6966 review follow-up e830e422; the new exact head is 57062ede.

The rebase preserves the fail-closed PR-label lookup and removal of the two unused legacy label branches while extending suite selection to L4. Validation is green locally: AMD selector/nightly/skip-CI tests 56 passed, Bash syntax, git diff --check, and the complete changed-file pre-commit suite all pass.

The prior runtime request is now superseded by this head. @yenuo26 @congw729, when the static checks finish, please apply amd-test and nightly-test together to start the external AMD pipeline and select only the experimental L4 suite. Exact-head MI300 results remain pending.

[andyluo7]

Self-review update for rebased exact head f9d5cd4ced130ceee53c3cdb56f989b6caede852:

• Rebased directly onto current main (4b8dcac6) and reduced the PR to one commit; the merged [CI][ROCm] Match CUDA/NPU L2/L3 label routing #6966 stack is no longer duplicated.
• Scope is limited to AMD L4 selection/bootstrap integration, the four-job Qwen3-Omni nightly definition, skip-CI registration, focused tests, and CI documentation. CUDA/NPU execution paths are unchanged.
• The four rendered pilot leaves are non-blocking amd_mi300_2 jobs with positive timeouts. Accuracy publishes tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc/*.json; the AITER smoke explicitly exports VLLM_ROCM_USE_AITER=1.
• Focused selector/bootstrap/nightly/skip-CI tests: 78 passed. Changed-file pre-commit, ShellCheck, Bash syntax, and git diff --check: passed.
• Live external-trigger check: nightly-test alone did not register AMD CI; neither did an amd-test label event or a synchronize event with amd-test + nightly-test. A temporary ready event registered AMD #11646; ready was removed before bootstrap selection. The exact-head bootstrap selected only NIGHTLY_TESTS:test-amd-nightly.yml and uploaded the expected four pilot jobs.

The external Buildkite condition still needs a separate update to admit nightly-test directly. The image build and accelerator jobs in #11646 are in progress, so terminal MI300 results are not claimed yet.

[vllm-omni-review-bot]

This PR was classified as CI work.

CI owner: @yenuo26 @hsliuustc0106

Routing: @yenuo26 via semantic router, CI owner, CODEOWNERS; @hsliuustc0106 via CODEOWNERS

@andyluo7, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[andyluo7]

AMD L4 burn-in follow-up for exact head f9d5cd4ced130ceee53c3cdb56f989b6caede852:

Build #11646 exercised all four intended mi300_2 pilot jobs. They remain non-blocking as designed.

Baseline observation (run 1):

• Function Expansion: 1 failed, 25 passed; test_speaker_002[async_chunk] had one audio/text mismatch (similarity=0.1893656507) while the other samples were accurate.
• Accuracy: Daily-Omni passed; Seed-TTS evaluated all 1,088 samples with zero request, PCM, or ASR failures, but mean WER was 0.376480 versus the current maximum 0.35.
• Documentation Examples: passed.
• AITER-on Smoke: passed.

@yenuo26 @congw729, could one of you please run two additional isolated retries of these two jobs on the same exact head?

1. mi300_2: Qwen3-Omni Function Expansion (01a0896a-d610-4823-ad60-8b2f291403d5)
2. mi300_2: Qwen3-Omni Accuracy (01a0896a-d613-462a-81a8-836c9b8cafd0)

If per-job retry is unavailable, two AMD-nightly-only runs are sufficient. I do not have Buildkite retry credentials in this environment, and a GitHub ready label pulse would unnecessarily retrigger the unrelated CUDA/NPU/Intel pipelines.

Please keep the Seed-TTS artifact for each run. We should collect the additional ROCm observations before deciding whether a platform baseline is warranted; this request does not propose changing the 0.35 threshold.

[andyluo7]

Rebase and merge-readiness update:

• New exact head: 17af998e8f30013503735e95980586af30b2bfa2, based directly on current main ffcaaa9439823d9b927d201397bb9d4db7218f14.
• git range-diff confirms the L4 patch is unchanged from f9d5cd4c.
• Focused selector/bootstrap/nightly/skip-CI tests: 78 passed.
• Changed-file pre-commit, Buildkite schema validation, ShellCheck, Bash syntax, and git diff --check: passed.
• The PR description now records the terminal results from historical AMD build #11646 and makes clear that its results predate this rebase.

The earlier retry request on f9d5cd4c is superseded. @yenuo26 @congw729, could one of you please start one AMD-nightly-only build on exact head 17af998e—either via an authorized manual build with DEBUG_TEST_YAML=nightly or after admitting nightly-test in the external pipeline condition? Please do not use a ready label pulse, since that would schedule unrelated platform pipelines.

For merge readiness, the immediate goal is to verify that the rebased head selects and runs the four intended non-blocking L4 jobs. Two further isolated retries of Function Expansion and Accuracy, with each Seed-TTS artifact retained, remain useful burn-in evidence and would give three observations on the new SHA; they are not a proposal to change the current 0.35 threshold in this PR.

@hsliuustc0106, please review the CI configuration when the fresh exact-head checks have registered.

[andyluo7]

Review fixes are now on exact head f015caf75bbe14770aa062d7bcdcfb150765d309:

• renamed the AMD lane/group to MI300 nightly terminology
• removed the hardcoded nightly inventory unit test
• retained general AMD label-to-suite coverage
• documented the paired amd-test + nightly-test contract
• annotated the two historical #11646 burn-in failures
• resolved the four addressed review threads

Local validation: 128 focused/general Buildkite tests passed, all changed-file pre-commit hooks passed, and Bash syntax, ShellCheck, Buildkite schema, DCO, and git diff --check are clean.

@yenuo26 @congw729, the remaining prerequisite for exact-head AMD validation is the external vllm-omni-amd-ci PR build condition. Under the existing build.source_action == "labeled" branch, please add an order-independent paired-label condition equivalent to:

(
  (build.pull_request.label == "nightly-test" ||
   build.pull_request.label == "amd-test") &&
  build.pull_request.labels includes "nightly-test" &&
  build.pull_request.labels includes "amd-test"
)

The PR already carries nightly-test. After the condition is saved, applying amd-test should trigger AMD CI for this exact head and the repository bootstrap should select only NIGHTLY_TESTS:test-amd-nightly.yml. Please avoid a ready pulse because it would schedule unrelated platform CI.

[andyluo7]

Exact-head AMD validation is complete for aa5b535c6927ea2946c8c5c72faaa17c493b7dd5.

• AMD Buildkite #11712 selected only NIGHTLY_TESTS:test-amd-nightly.yml; bootstrap, image build, and all four non-blocking amd_mi300_2 jobs finished with exit status 0.
• Function Expansion: 26 passed; Documentation Examples: 7 passed; AITER-on Smoke: 3 passed with VLLM_ROCM_USE_AITER=1 and module_aiter_core.so loaded.
• Accuracy: 2 passed. Daily-Omni evaluated 1,197/1,197 with accuracy 0.7184628237259816. Seed-TTS evaluated 1,088/1,088 with zero request, PCM, or ASR/WER failures and mean WER 0.33326678996457526, below the unchanged 0.35 threshold.
• Both generated Accuracy JSON files were copied into the checkout and uploaded successfully as Buildkite artifacts.
• All current GitHub checks are green, and all review threads are resolved. The changes remain AMD CI scoped; CUDA/NPU paths are unchanged, and the new AMD leaves remain non-blocking during burn-in.

The PR description now contains the full terminal evidence and artifact links. The external order-independent amd-test + nightly-test admission rule remains a Buildkite pipeline configuration item outside this repository PR.

@yenuo26 @hsliuustc0106, could you please take a final look and approve if there are no further issues?

[andyluo7]

Second exact-head AMD nightly observation is complete for aa5b535c6927ea2946c8c5c72faaa17c493b7dd5.

• AMD Buildkite #11747 passed all 6 jobs with no failures or soft failures. The paired-label PR event registered AMD CI and produced the intended four MI300 nightly leaves.
• Function Expansion: 26 passed.
• Documentation Examples: 7 passed.
• AITER-on Smoke: 3 passed, with VLLM_ROCM_USE_AITER=1 and module_aiter_core.so loaded.
• Accuracy: 2 passed.
  • Daily-Omni evaluated 1,197/1,197 samples with 867 correct, accuracy 0.7243107769423559, and zero request or parse failures.
  • Seed-TTS evaluated 1,088/1,088 samples with mean WER 0.324273148846563, below the unchanged 0.35 maximum, and zero request, PCM, or ASR/WER failures.
• Both generated result artifacts are directly readable: Daily-Omni JSON and Seed-TTS JSON.

Together with #11712, this gives two successful observations on the current exact head. All current GitHub checks are green, the new AMD jobs remain non-blocking, and the Seed-TTS threshold remains unchanged.

@congw729 @NickCao @Gaohan123 @david6666666, could one of you please take a final look and approve if there are no remaining concerns?

[yenuo26]

LGTM