Skip to content

[CI/Build][ROCm] Add Qwen3-Omni and BAGEL full_model MI300 nightlies - #9435

Merged
andyluo7 merged 2 commits into
vllm-project:mainfrom
haic0:ci/amd-entrypoints-full-model-nightly
Oct 28, 2026
Merged

andyluo7 merged 2 commits into
vllm-project:mainfrom
haic0:ci/amd-entrypoints-full-model-nightly

Conversation

@haic0

@haic0 haic0 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Rebuild the PR on current main to resolve the prior merge conflict.
  • Split the former combined entrypoints lane into two focused, non-blocking mi300_2 nightly jobs:
    • Qwen3-Omni Duplex Playback: exactly 40 runnable full_model cases.
    • BAGEL Sleep Mode: exactly three runnable full_model cases.
  • Extend only the BAGEL fixture total startup budget from 600 to 1,200 seconds after AMD showed a 553.2-second cold AITER compile exhausting the old budget.
  • Keep TestBagelCoordinatedSleepMode::test_coordinated_cross_device in the source tree but intentionally exclude it because it remains explicitly skipped.
  • Add exact collection guards, zero-skip JUnit guards, timeout/pytest exit propagation, and collection, JUnit, execution-log, and summary artifacts for both jobs.
  • Add focused source dependency groups so PR nightly routing runs only the relevant lanes.
  • Add native ROCm MI325 coverage only to Qwen3-Omni's full_model playback method while preserving the existing CUDA H100/B200 coverage on the class.

Contributor intent

1. What outcome do you want to achieve?

Run the CUDA nightly full_model entrypoint cases on MI300.

2. What is the highest-leverage variable for that outcome?

Whether the duplex playback repeats and BAGEL sleep checks are collected on two MI300 GPUs.

3. How will this work move that variable?

Focused non-blocking mi300_2 jobs fail collection or execution unless their exact runnable node sets are present and complete.

4. Whose customer or user needs does this address, and what are those needs?

ROCm users need the same nightly entrypoint gate CUDA already runs.

Purpose

Before this change, the PR combined Qwen3-Omni and BAGEL into one broad job and included the deliberately skipped coordinated BAGEL case in its collection expectation. The branch also conflicted with current main.

After this change, Qwen3-Omni and BAGEL run as independent nightly jobs with exact runnable counts, zero accepted skips, separate artifacts, and focused source routing. The Qwen marker is method-scoped, so advanced-model methods retain their existing CUDA-only selection while the 40 full-model playback variants gain ROCm MI325 selection.

This changes CI coverage and test selection only. It does not change runtime behavior, public configuration, supported APIs, or an active design contract, so no user or design documentation update is required.

Test Plan

vLLM Version: Hardware CI image version selected by the AMD pipeline; not installed in the local macOS environment.

vLLM-Omni Commit: a477577570c732837dc3de97b5189459299fbe5e

Test Result

  • python3 -m pytest tests/buildkite/test_amd_pipeline.py -q --override-ini=addopts= --confcutdir=tests/buildkite — 122 passed at the reported commit.
  • Focused contract selection (-k 'entrypoints_full or qwen_duplex_rocm_mark or bagel_rocm_nightly') — 14 passed, 108 deselected.
  • An exact-diff source-filter dry run retained only Qwen3-Omni Duplex Playback and BAGEL Sleep Mode.
  • pre-commit run --files .buildkite/amd/test-amd-nightly.yml .buildkite/common/ci_source_file_dependencies.yml tests/buildkite/test_amd_pipeline.py tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py tests/entrypoints/test_omni_sleep_mode.py — all applicable hooks passed, including Buildkite schema, test marks, SPDX, forbidden imports, torch.cuda, Ruff, and mypy-3.10.
  • pre-commit run --hook-stage manual mypy-3.12 --files tests/buildkite/test_amd_pipeline.py tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py tests/entrypoints/test_omni_sleep_mode.py — passed.
  • git diff --check origin/main...HEAD — passed.

AMD #14879 at the superseded head 43f5d2f5b459aaa6a7477bd8a2fb48e499d208c4 supplied the first hardware evidence:

  • Qwen3-Omni collected and executed 40/40, with 0 skipped, no retry, and no soft failure.
  • BAGEL collected the intended 3 runnable cases, then all three errored during shared fixture setup because cold AITER compilation took 553.2 seconds and exhausted the 600-second total orchestrator startup budget.
  • The current head raises only that BAGEL fixture budget to 1,200 seconds and adds a regression contract. Fresh exact-head AMD qualification is required.

The local Mac has no Torch/ROCm runtime or model weights, so it cannot provide the requested real collection/execution evidence. Exact-head AMD nightly CI remains the hardware qualification gate:

  • Qwen3-Omni must collect and execute 40/40, with 0 skipped.
  • BAGEL must collect and execute 3/3, with 0 skipped.
  • The coordinated BAGEL case must remain excluded from both commands.
  • Both individual non-blocking jobs must pass without retries or soft failures; aggregate build green alone is not sufficient.

AI assistance disclosure

  • Cursor co-authored the original nightly job, ROCm mark, and contract test.
  • Codex rebuilt the branch on current main, split and hardened the jobs, added focused routing and regression contracts, ran the reported local validation, and drafted this description.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-26T18:45:10.535807Z
We couldn’t complete this review. Comment @codex review to try again.
5f010ea PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@vllm-omni-review-bot

Copy link
Copy Markdown

This PR appears to belong to: docs/design/module/entrypoints.md.

Module owners: @NickCao @alex-jw-brooks @linyueqian

Routing: @NickCao via module named in the PR description, CODEOWNERS; @alex-jw-brooks via module named in the PR description; @linyueqian via module named in the PR description

@haic0, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

@haic0

haic0 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Self-review of 5f010ea25.

Checked the CUDA nightly selector full_model and cuda and H100 and B200 and cards_2. The BAGEL sleep classes already had rocm: MI325. I added that mark on TestQwen3OmniDuplexPlayback so the 20 full_model repeats are selected. Advanced-model methods on that class are not run, because the job marker includes full_model. The node list is 40 playback ids ([0-generating] through [19-queued-playback]) plus the three BAGEL diffusion methods and the skipped coordinated method. Collection exits 5 if that count is wrong.

test_entrypoints_full_model_is_one_nonblocking_nightly_job passed. pre-commit and mypy-3.12 passed. I could not collect the nodes here because this environment cannot import validate_profile_iteration_bounds from vllm.config.profiler.

@haic0 haic0 closed this Oct 26, 2026
@haic0 haic0 reopened this Oct 27, 2026
@haic0
haic0 force-pushed the ci/amd-entrypoints-full-model-nightly branch from 5f010ea to 7b95c7d Compare October 27, 2026 17:31

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At reviewed head 7b95c7d:

  • Update this branch against current main to resolve the reported merge conflict, then record the fix SHA. Acceptance: the updated head merges cleanly with the target and receives fresh applicable checks.
  • Supply the nightly selector's actual collection output and execution/JUnit result at the updated head with the matching vLLM/ROCm environment. The self-review says collection could not run because of an upstream import mismatch. Distinguish the 40 playback cases and three runnable BAGEL cases from the deliberately skipped coordinated case: acceptance is the intended node set collected, runnable cases exercised, and skips accounted for. If MI325 hardware is unavailable, state that limitation so maintainers can choose validation.

Owner: haic0 (PR author). Status: awaiting branch update and evidence. My inspection was source-only, not a hardware run.

@andyluo7
andyluo7 force-pushed the ci/amd-entrypoints-full-model-nightly branch from 7b95c7d to 8ac0463 Compare October 28, 2026 00:09
@andyluo7 andyluo7 changed the title [CI/Build][ROCm] Add full_model entrypoints to MI300 nightly [CI/Build][ROCm] Add Qwen3-Omni and BAGEL full_model MI300 nightlies Oct 28, 2026
@andyluo7 andyluo7 added ready label to trigger buildkite CI nightly-test label to trigger buildkite nightly test CI amd-test Used to trigger AMD CI separately. labels Oct 28, 2026
@vllm-omni-review-bot

vllm-omni-review-bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Omni ReviewBot triage note

Automated triage of commit a477577570c7 produced:

  • Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

@andyluo7
andyluo7 force-pushed the ci/amd-entrypoints-full-model-nightly branch from 8ac0463 to 43f5d2f Compare October 28, 2026 00:16
@andyluo7 andyluo7 added ready label to trigger buildkite CI amd-test Used to trigger AMD CI separately. and removed ready label to trigger buildkite CI amd-test Used to trigger AMD CI separately. labels Oct 28, 2026
haic0 and others added 2 commits October 27, 2026 19:46
Run Qwen3-Omni duplex playback and BAGEL sleep coverage as separate non-blocking MI300 jobs with exact collection and zero-skip JUnit contracts.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Signed-off-by: andyluo7 <43718156+andyluo7@users.noreply.github.com>
@andyluo7
andyluo7 force-pushed the ci/amd-entrypoints-full-model-nightly branch from 43f5d2f to a477577 Compare October 28, 2026 00:50
@andyluo7

Copy link
Copy Markdown
Collaborator

Updated the branch on current main at head a477577570c732837dc3de97b5189459299fbe5e and addressed the first AMD hardware result.

AMD #14879 at the superseded head 43f5d2f5b459aaa6a7477bd8a2fb48e499d208c4 showed:

  • Qwen3-Omni: exact collection and 40/40 passed, zero skipped, no retry or soft failure.
  • BAGEL: exact collection of the intended three runnable cases, then shared fixture setup timed out. Cold AITER compilation took 553.2 seconds, exhausting the fixture’s 600-second total orchestrator startup budget before the stage became ready.

The current head raises only that BAGEL fixture budget to 1,200 seconds and adds a regression contract. Post-rebase validation is 122 passed for the full AMD pipeline-contract suite, 14 passed / 108 deselected for the focused contracts, with pre-commit, mypy-3.12, and git diff --check all passing. Fresh exact-head READY CI remains the merge gate; the superseded BAGEL failure is not being treated as qualification evidence.

@andyluo7 andyluo7 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI amd-test Used to trigger AMD CI separately. labels Oct 28, 2026
@andyluo7 andyluo7 added the ready label to trigger buildkite CI label Oct 28, 2026
@andyluo7

Copy link
Copy Markdown
Collaborator

Exact-head READY qualification is complete at a477577570c732837dc3de97b5189459299fbe5e.

  • AMD #14881: all 37 job records finished with exit 0; no retries or soft failures.
    • Qwen3-Omni Duplex Playback: exact collection 40, execution 40 passed / 0 skipped, and four diagnostic artifacts uploaded.
    • BAGEL Sleep Mode: exact collection 3, execution 3 passed / 0 skipped. Cold AITER compilation took 551.1 seconds, worker cleanup reported still_alive=[], and four diagnostic artifacts uploaded.
  • CUDA #18338: all 20 job records finished with exit 0; no retries or soft failures.
  • NPU #10087: all 22 job records finished with exit 0; no retries or soft failures.
  • GitHub Actions, DCO, CodeQL, pre-commit, wheel builds, and docs are green.
  • Intel #11048 remains queued/pending and is explicitly treated as non-blocking for this merge.

The head and base have not drifted, GitHub reports the PR mergeable, and there are no unresolved review threads.

@andyluo7 andyluo7 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at exact head a477577 after reviewing the focused timeout change and regression contract. AMD #14881 passed the exact Qwen3-Omni 40/40 and BAGEL 3/3 workloads with zero skips, retries, soft failures, or leaked workers; CUDA #18338 and NPU #10087 are also fully green. Intel #11048 is explicitly waived as non-blocking.

@andyluo7
andyluo7 merged commit 4c51b23 into vllm-project:main Oct 28, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nightly-test label to trigger buildkite nightly test CI ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants