[Test] Add qwen3-omni tests for audio_in_video and one word prompt by yenuo26 · Pull Request #2097 · vllm-project/vllm-omni

yenuo26 · 2026-03-23T11:20:31Z

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Add qwen3-omni tests for audio_in_video and one word prompt

Test Plan

1.run in local env

/workspace/.venv/bin/python -m pytest -s -v tests/e2e/online_serving/test_qwen3_omni_expansion.py -k "test_audio_in_video_001"  -m "advanced_model" --run-level "advanced_model"

2.run in ci

Test Result

1.local

==================================================================== 1 passed, 18 warnings in 132.98s (0:02:12) ====================================================================

2.ci

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 · 2026-03-23T11:21:21Z

@Shirley125 @amy-why-3459 PTAL

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 42387a32de

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…d increasing max tokens in CI configuration Signed-off-by: yenuo26 <410167048@qq.com>

amy-why-3459 · 2026-03-23T12:10:50Z

Please add a multi-concurrency precision test case for audio_in_video.

Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 · 2026-03-23T12:37:06Z

Please add a multi-concurrency precision test case for audio_in_video.

done

yenuo26 · 2026-03-23T12:37:49Z

@Gaohan123 @tzhouam please help to add a ready label

hsliuustc0106

BLOCKER scan:

Correctness: ISSUES: test_audio_in_video_001/002 still do not assert on audio-derived content, so the new use_audio_in_video path can break while the tests continue to pass by describing only the video.
Reliability/Safety: ISSUES: .buildkite/test-ready.yml now hard-switches the H100 job to an advanced expansion test with an in-file note saying it is for debug and should be removed before merge. That changes the repo's normal CI signal rather than adding stable coverage.
Breaking Changes: PASS
Test Coverage: needs tests/evidence that specifically fail when embedded audio is ignored.
Documentation: PASS (test-only PR)
Security: PASS

OVERALL: 2 BLOCKERS FOUND

VERDICT: REQUEST_CHANGES

I validated the new use_audio_in_video wiring in send_omni_request() and the new expansion tests. The remaining blockers are that the assertions still don't depend on extracted audio content, and the Buildkite change is explicitly checked in as a temporary debug path instead of stable CI coverage.

hsliuustc0106 · 2026-03-23T14:09:25Z

+        timeout 20m bash -c '
+          export VLLM_WORKER_MULTIPROC_METHOD=spawn
+          export VLLM_TEST_CLEAN_GPU_MEMORY="1"
+          #pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m "core_model" --run-level "core_model"


This swaps the normal H100 omni test job over to test_qwen3_omni_expansion.py and even leaves a for debug, will be removed before merging note in the committed pipeline. That means the PR is changing repository CI coverage in a temporary/debug-only way instead of adding a stable test signal. Could you revert the debug pipeline override and keep the new coverage in a dedicated test/job if it needs CI coverage?

hsliuustc0106 · 2026-03-23T22:17:18Z

Review Summary

Verdict: ✅ LGTM (No Blockers)

Test-only PR for Qwen3-omni edge cases. Good coverage expansion!

Test Quality Check

✅ Test correctness: Tests use standard omni_server + openai_client fixtures
✅ Coverage: Tests 3 scenarios:

test_audio_in_video_001 - audio-in-video input
test_audio_in_video_002 - different audio-in-video config
test_one_word_prompt_001 - minimal text input edge case

✅ Test evidence: PR shows 1 passed in 132.98s

Non-blocking Observations

conftest.py: Large refactoring (121 additions, 32 deletions) - test infrastructure improvements. Not reviewing in detail since it's test infrastructure.

test-ready.yml: Modified CI config (39 additions, 38 deletions) - appears to be test scheduling changes.

Ready to merge.

…ken configuration Signed-off-by: yenuo26 <410167048@qq.com>

…into audio_in_video

Signed-off-by: yenuo26 <410167048@qq.com>

…to include audio in the output MP4. Update tests to utilize the new feature and adjust CI configuration to reduce `max_tokens` for improved performance. Signed-off-by: yenuo26 <410167048@qq.com>

Signed-off-by: wangyu <410167048@qq.com>

Gaohan123 · 2026-03-25T11:18:21Z

Please fix docs

yenuo26 · 2026-03-25T14:14:30Z

Please fix docs

Actually, I didn't modify any document. I will retry it.

Gaohan123 · 2026-03-25T15:35:23Z

@yenuo26 Please resolve conflicts

yenuo26 · 2026-03-25T16:24:02Z

@yenuo26 Please resolve conflicts
There are currently no conflicts to resolve.

amy-why-3459 · 2026-03-26T03:34:45Z

+    openai_client.send_omni_request(request_config, request_num=get_max_batch_size())
+
+
+@pytest.mark.skip(reason="There is a known issue: https://github.com/vllm-project/vllm-omni/pull/2019")


PR #2019 has been merged. Please remove the skip option and test whether the test cases pass.

During local testing, I found that when called concurrently, this error is not resolved.

OK, I will continue to follow up on this issue and will remove the skip flag once it is resolved.

yenuo26 · 2026-03-26T08:32:11Z

@Gaohan123 @david6666666 please help to add nightly-test

yenuo26 · 2026-03-26T11:55:40Z

Can this be merged? I've already run L2+L4. The vllm-omni CI failed because the Diffusion L4 performance test didn't pass. @Gaohan123

…ecision for unique timestamps, preventing file overwrites during concurrent calls. Signed-off-by: wangyu <410167048@qq.com>

…into audio_in_video

…d checks for similarity and normalized matching against audio_transcript_key_words. Updated test cases to reflect new requirements. Signed-off-by: wangyu <410167048@qq.com>

…ness. Signed-off-by: wangyu <410167048@qq.com>

yenuo26 · 2026-03-28T01:15:51Z

I just modify test case in "Omni Model Test with H100", and it is success in nightly test.
Please review whether this PR can be merged. @Gaohan123 @david6666666

gcanlin

LGTM

Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>

…llm-project#2097) Signed-off-by: yenuo26 <410167048@qq.com> Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com> Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>

add qwen3-omni tests

42387a3

Signed-off-by: yenuo26 <410167048@qq.com>

chatgpt-codex-connector Bot reviewed Mar 23, 2026

View reviewed changes

Comment thread tests/e2e/online_serving/test_qwen3_omni_expansion.py Outdated

Enhance qwen3-omni tests by adding support for audio-video prompts an…

040100c

…d increasing max tokens in CI configuration Signed-off-by: yenuo26 <410167048@qq.com>

Update Omni Model Test configuration and enhance audio-video test cases

16e87f6

Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 requested a review from hsliuustc0106 as a code owner March 23, 2026 12:36

hsliuustc0106 added the ready label to trigger buildkite CI label Mar 23, 2026

hsliuustc0106 requested changes Mar 23, 2026

View reviewed changes

yenuo26 and others added 6 commits March 24, 2026 09:35

Merge branch 'vllm-project:main' into audio_in_video

3665991

Update CI timeout and enhance Omni model test parameters for batch to…

95aa645

…ken configuration Signed-off-by: yenuo26 <410167048@qq.com>

Merge branch 'audio_in_video' of https://github.com/yenuo26/vllm-omni …

8fe2c41

…into audio_in_video

debug

f205c5d

Signed-off-by: yenuo26 <410167048@qq.com>

Enhance synthetic video generation by adding embed_audio parameter …

22458c8

…to include audio in the output MP4. Update tests to utilize the new feature and adjust CI configuration to reduce `max_tokens` for improved performance. Signed-off-by: yenuo26 <410167048@qq.com>

remove debug

95ca2b3

Signed-off-by: wangyu <410167048@qq.com>

yenuo26 force-pushed the audio_in_video branch from 589ceb8 to 95ca2b3 Compare March 24, 2026 14:53

Merge branch 'main' into audio_in_video

a699b37

amy-why-3459 reviewed Mar 26, 2026

View reviewed changes

yenuo26 added 2 commits March 26, 2026 14:24

Merge branch 'vllm-project:main' into audio_in_video

f52ece8

Merge branch 'vllm-project:main' into audio_in_video

453f551

Merge branch 'main' into audio_in_video

d1fc6f3

congw729 added the nightly-test label to trigger buildkite nightly test CI label Mar 26, 2026

amy-why-3459 mentioned this pull request Mar 27, 2026

[BugFix][Qwen3-Omni]Fixed the issue of incorrect answers for single words. #2239

Merged

5 tasks

yenuo26 added 3 commits March 27, 2026 10:18

Update output file naming in modify_stage_config to use nanosecond pr…

51d4fbb

…ecision for unique timestamps, preventing file overwrites during concurrent calls. Signed-off-by: wangyu <410167048@qq.com>

Merge branch 'audio_in_video' of https://github.com/yenuo26/vllm-omni …

8f4eb6e

…into audio_in_video

Improve audio transcript validation in OmniResponse assertions. Adde…

2b7e886

…d checks for similarity and normalized matching against audio_transcript_key_words. Updated test cases to reflect new requirements. Signed-off-by: wangyu <410167048@qq.com>

yenuo26 force-pushed the audio_in_video branch from 8d3af9f to 2b7e886 Compare March 27, 2026 04:08

yenuo26 and others added 2 commits March 27, 2026 12:21

Merge branch 'main' into audio_in_video

c125fe7

Updated test cases to retry on assertion failures for improved robust…

975f197

…ness. Signed-off-by: wangyu <410167048@qq.com>

gcanlin approved these changes Mar 28, 2026

View reviewed changes

gcanlin removed the nightly-test label to trigger buildkite nightly test CI label Mar 28, 2026

gcanlin requested a review from hsliuustc0106 March 28, 2026 11:10

Merge branch 'main' into audio_in_video

073bfee

Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>

hsliuustc0106 merged commit c1a978a into vllm-project:main Mar 31, 2026
7 of 8 checks passed

yenuo26 deleted the audio_in_video branch March 31, 2026 11:33

		openai_client.send_omni_request(request_config, request_num=get_max_batch_size())


		@pytest.mark.skip(reason="There is a known issue: https://github.com/vllm-project/vllm-omni/pull/2019")

Conversation

yenuo26 commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

yenuo26 commented Mar 23, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

amy-why-3459 commented Mar 23, 2026

Uh oh!

yenuo26 commented Mar 23, 2026

Uh oh!

yenuo26 commented Mar 23, 2026

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

yenuo26 Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 commented Mar 23, 2026

Review Summary

Test Quality Check

Non-blocking Observations

Uh oh!

Gaohan123 commented Mar 25, 2026

Uh oh!

yenuo26 commented Mar 25, 2026

Uh oh!

Gaohan123 commented Mar 25, 2026

Uh oh!

yenuo26 commented Mar 25, 2026

Uh oh!

amy-why-3459 Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

yenuo26 Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

amy-why-3459 Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

yenuo26 commented Mar 26, 2026

Uh oh!

yenuo26 commented Mar 26, 2026

Uh oh!

yenuo26 commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gcanlin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yenuo26 commented Mar 23, 2026 •

edited

Loading

yenuo26 commented Mar 28, 2026 •

edited

Loading