[psv666]

Purpose

• Consolidate redundant Qwen3-Omni online-serving coverage into longer or broader cases:
  • combine uppercase CHELSIE speaker lookup and Chinese-language output in test_speaker_001;
  • preserve the reduced-token text batching path on test_text_to_text_audio_001, in addition to the stronger long-audio stress case;
  • remove the skipped mixed-modality case because it contributes neither runtime nor executable coverage.
• Move unsupported chat audio-format regression coverage from the L4 expansion suite to the weekly invalid-parameter suite.
• Keep audio-only as suite-level coverage: the multi-replica streaming test still sends modalities=["audio"], while the expansion suite focuses on broader output cases.
• Keep NPU, CUDA nightly, and CUDA weekly Async Chunk entries in one JSON and select them by hardware plus the existing slow scheduling marker.
• Remove CUDA Async Chunk default text+audio perf cases already covered by multi-replica; retain Async Chunk text-only cases because multi-replica does not cover that output contract.
• Move CUDA Async Chunk random text-only coverage to weekly at concurrency 1/8/32. Keep nightly random-mm text-only endpoints at 10/100 requests.
• Reduce every Async Chunk random workload from input/output lengths 2500/900 to 100/100 to lower test pressure. Remove the affected weekly H100 latency baseline because it must be remeasured for the new workload.
• Apply the same concurrency/request-count trimming to the weekly vLLM-text comparison: 1/8/32 for random and 10/100 for random-mm. The multi-replica suite retains its 40-request image+video point.
• Remove H100 baselines from the A3-only NPU entry; NPU remains collect-only until A3 baselines are measured.

Test Plan

• Run the consolidated speaker case against both default and async-chunk serving modes on GPU.
• Validate expansion, invalid-parameter, and perf marker collection after cleanup.
• Run audio-format resolver and perf metadata unit tests.
• Run repository pre-commit checks, including Ruff, mypy, test markers, SPDX, Markdown, and Buildkite validation.

vLLM Version: 0.27.0
vLLM-Omni Commit: e710314

Test Result

• Previous GPU inference:
  • test_speaker_001[default]: passed
  • test_speaker_001[async_chunk]: passed
• Unit tests:
  • audio-format resolver and perf metadata: 40 passed
  • post-adjustment perf metadata: 21 passed
• Post-cleanup collection:
  • expansion suite: 26 tests
  • test_text_to_text_audio_001: default, async_chunk, and batch_token_2048 selected
  • unsupported audio format weekly case: 1 selected
  • Async Chunk combined JSON: 11 tests
  • Async Chunk CUDA nightly: 2 selected
  • Async Chunk CUDA weekly: 1 selected
  • Async Chunk NPU: 8 selected
• JSON validation and git diff --check: passed
• pre-commit on all changed files: passed

Previous GPU model inference pytest log

This log validates the unchanged consolidated speaker/language request path. The final cleanup changes other cases, so its module-level collection count is intentionally superseded by the post-cleanup collect-only result above.

Reserved 2 GPU(s): [0 4] for command execution (timeout: 1h 0m 0s)

INFO 08-24 19:25:05 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
INFO 08-24 19:25:06 [patch.py:503] inductor factorable-divisibility patch: installed.
INFO 08-24 19:25:06 [patch.py:568] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /data/pengsiv/vllm-omni/.venv/bin/python
cachedir: .pytest_cache
rootdir: /data/pengsiv/vllm-omni-main
configfile: pyproject.toml
plugins: anyio-4.14.2, shard-0.1.2, asyncio-1.4.0, mock-3.15.1, typeguard-4.5.2, hydra-core-1.3.4, cov-7.1.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ...
collecting 28 items
collected 28 items / 26 deselected / 2 selected
Running 2 items in this shard: tests/e2e/online_serving/test_qwen3_omni_expansion.py::test_speaker_001[default], tests/e2e/online_serving/test_qwen3_omni_expansion.py::test_speaker_001[async_chunk]

tests/e2e/online_serving/test_qwen3_omni_expansion.py::test_speaker_001[default]
OmniServerStageCli ready on 127.0.0.1:53257 (stage-CLI startup took 508.062s)
OmniServer started successfully
--- Running test: test_speaker_001[default]
audio data is saved: /tmp/test_ix5bd3yf.wav
OmniServer stopped

tests/e2e/online_serving/test_qwen3_omni_expansion.py::test_speaker_001[async_chunk]
OmniServerStageCli ready on 127.0.0.1:37379 (stage-CLI startup took 452.088s)
OmniServer started successfully
--- Running test: test_speaker_001[async_chunk]
audio data is saved: /tmp/test_uuq75t1i.wav
OmniServer stopped

=============================== warnings summary ===============================
../vllm-omni/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /data/pengsiv/vllm-omni/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
--- Running Summary
========== 2 passed, 26 deselected, 14 warnings in 1288.39s (0:21:28) ==========

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to be related to model: qwen-omni.

Model owners: @amy-why-3459 @psv666

@psv666, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[psv666]

@yenuo26 @amy-why-3459 PTAL

[amy-why-3459]

Thanks for the cleanup — the CUDA nightly/weekly split is the right
shape, and the weekly 1/8/32 points are a clean slice of the current
5-point H100 matrix on main. A few coverage and config issues before
this can land.

1. test_text_to_text_audio_001 lost reduced_token_params when
   test_text_to_audio_001 was deleted. The @parametrize(..., test_params) that sat on the deleted function is now the only
   decorator on _001, and the original test_params + test_token_params went with it. get_batch_token_config() exists
   specifically for 5-way concurrency; _001 is the
   request_num=get_max_batch_size() text case. Reduced-token now
   remains only on long-audio _002 and the skipped mix test.
   large_image / text_video_to_text_audio look intentionally
   narrowed to test_params — please confirm _001 was meant to
   drop it too, and restore test_params + reduced_token_params
   if not.

2. test_qwen3_omni_async_chunk.json is now NPU-only (npu: A3)
   but every baseline is still {"H100": {...}}.
   resolve_baseline_for_sweep() will write an H100 bucket into
   NPU result files. Please add an A3 bucket (or drop the H100
   numbers and say NPU is collect-only until measured). Leaving
   H100 thresholds on an A3-only file is misleading.

3. test_invalid_audio_format_rejected is not obsolete.
   serving_chat.py::_resolve_audio_format() still 400s formats
   outside SUPPORTED_CHAT_AUDIO_FORMATS, and aac is still
   outside that set. Deleting it from expansion is the right
   location (it was a raw SDK call in an L4 file), but the
   contract should move to
   tests/dfx/reliability/invalid_param_test/test_invalid_omni_chat.py
   via send_chat_completions_http_request + err_code=400.
   MiniCPM already has the sibling weekly case; Qwen3-Omni chat
   does not.

4. test_text_to_audio_001 was the only expansion case with
   modalities=["audio"]. Remaining speaker / long-output cases
   use the default ["text", "audio"], and L2 test_qwen3_omni.py
   does not cover audio-only either. That is a dropped output
   modality, not a longer case absorbing a shorter one. Please
   keep audio-only on one remaining streaming case, or say
   explicitly that L2/L3 owns it.

5. Folding CHELSIE and text_chinese into test_speaker_001
   mixes two intents. Talker speaker lookup is already
   {k.lower(): v}, so uppercase vs Chelsie is the same path;
   the failure mode is now "voice or Chinese?" and the
   title-case English path is gone (test_speaker_002 only
   keeps Ethan). Please keep one speaker case and a separate
   Chinese case.

Smaller: the 40-request random-mm case was the only
image+video / no-audio / request_rate=0.5 point — if dropping
it is intentional to save nightly time, please say so in the
Purpose. l4_performance_tests.inc.md still documents
test_qwen3_omni_async_chunk.json as the CUDA nightly file.
The GPU log's collected 28 is the pre-cleanup module count
and only test_speaker_001 was executed; a post-cleanup
--collect-only for the expansion file would match the Test
Plan. Also please leave the author self-review the review-bot
asked for.

The SPDX copyright-line tweak and the extra parametrize on the
already-skipped mix test can wait.

[yenuo26]

Why add a new JSON? Can't we just use Mark to distinguish it?

[psv666]

Good point. I checked the perf loader: marks are attached per top-level config object, so separate files do not provide additional isolation. I merged the NPU, CUDA nightly, and CUDA weekly entries into test_qwen3_omni_async_chunk.json. The existing slow marker selects the weekly entry, nightly uses not slow, and both temporary CUDA JSON files were deleted. CI commands and the L4 perf documentation were updated in a0051ea.

[yenuo26]

I recall that we said this test case could be deleted?

[psv666]

Yes. It was initially retained because the skipped case consumed no runtime, but that also means it provides no executable coverage. Since this PR is cleaning up dead and redundant cases, keeping the no-op case is not useful; it is deleted in a0051ea.

[yenuo26]

We may need to check whether it is covered in xxx. If not, we should add this test case there.

[psv666]

I checked the exact contract rather than treating it as obsolete. The existing MiniCPM case validates an invalid input-audio container, while this Qwen3-Omni regression validates unsupported output audio.format in chat completions. The same aac request is now in tests/dfx/reliability/invalid_param_test/test_invalid_omni_chat.py, using the HTTP helper and asserting 400 plus the supported-format message. The L4 copy is removed only after that migration.

[yenuo26]

I recall that we mentioned some multi-replica performance test scenarios could also be removed. Additionally, if the omni test scenarios have already been trimmed, we may want to check whether the corresponding scenarios in vllm-text are redundant as well.

[psv666]

I compared the input/output contracts. Multi-replica already covers Async Chunk default text+audio for random and random-mm at 10/40/100, so those default-output cases were removed from the single-replica Async Chunk CUDA entries. The text-only output cases remain because multi-replica does not cover that contract: random 1/8/32 runs weekly and random-mm 10/100 runs nightly. I also trimmed vllm-text to the same 1/8/32 and 10/100 comparison points. Multi-replica keeps the 40-request image+video point.

[congw729]

Thanks for working on reducing the Qwen3-Omni nightly workload. I left two suggestions about preserving regression coverage while consolidating these cases.

[congw729]

This test was added as regression coverage when #4718 fixed #4716, where audio.format in chat completion requests was incorrectly handled. If we remove it directly from this expansion suite, Qwen3-Omni will no longer have equivalent coverage for rejecting unsupported audio formats.

Moving the test out of this L4 suite makes sense, but could we first migrate it to tests/dfx/reliability/invalid_param_test/test_invalid_omni_chat.py, and then remove it from here?

[psv666]

Agreed on preserving the regression contract. I verified that _resolve_audio_format still rejects aac, then migrated the request to tests/dfx/reliability/invalid_param_test/test_invalid_omni_chat.py with the HTTP helper and 400/message assertions before removing the expansion copy. The target collects under the existing weekly H100 invalid-parameter job, and the audio-format resolver plus perf metadata unit tests pass (40 passed).

[congw729]

One personal observation: after removing test_text_to_audio_001, test_text_to_text_audio_001 is now parametrized with only test_params, so the reduced-token configuration previously attached to this test is no longer covered.

Since this case sends five concurrent text requests and get_batch_token_config() is intended to exercise the smaller max_num_batched_tokens path, I suggest keeping test_params + reduced_token_params on test_text_to_text_audio_001.

[psv666]

Accepted in e710314. I initially kept the reduced-token configuration only on the 120-second audio stress case, but after reconsidering the coverage boundary, the five-request text-only path can exercise token batching behavior that is distinct from long multimodal preprocessing. Preserving that exact reduced max_num_batched_tokens regression path is worth the additional configuration. test_text_to_text_audio_001 now uses test_params + reduced_token_params; collect-only confirms default, async_chunk, and batch_token_2048.

[psv666]

@amy-why-3459 I addressed the review points, with the latest adjustment in e710314:

1. Accepted the reduced-token coverage suggestion. test_text_to_text_audio_001 now uses test_params + reduced_token_params. Although the 120-second audio case is stronger overall, the five-request text-only path can exercise token batching separately from multimodal preprocessing.
2. Removed every H100 baseline from the A3-only NPU entry. It remains collect-only until A3 measurements exist.
3. Migrated the unsupported output audio-format regression to the weekly Qwen3-Omni invalid-parameter suite using the HTTP helper and explicit 400/message checks.
4. The deleted expansion audio-only case is not directly covered by the long case. Its exact modalities=["audio"] streaming contract remains in test_qwen3_omni_multi_replicas.py, so this is suite-level coverage rather than single-case subsumption.
5. The CHELSIE + Chinese fusion is intentional: reducing model-serving runtime is the goal of this cleanup, and one request exercises both paths. test_speaker_002 still independently covers another preset speaker.

Also: the 40-request image+video point remains in multi-replica; single-replica Async Chunk and vLLM-text retain the 10/100 endpoints. The Async Chunk entries are now in one JSON selected by H100/NPU plus slow/not slow. Every Async Chunk random input/output length is reduced from 2500/900 to 100/100, and the affected H100 baseline was removed pending remeasurement. The perf documentation is updated, and post-cleanup expansion collection is 26 tests.

[psv666]

Author self-review:

• Compared the final diff with the requested nightly cleanup and each reviewer concern.
• Verified suite-level coverage before deleting cases: multi-replica owns default text+audio perf and streaming audio-only; Async Chunk retains text-only perf; unsupported output audio format moved to weekly reliability.
• Restored the reduced-token text batching case after reconsidering that pure-text batching can fail independently of long multimodal preprocessing.
• Checked the single-JSON schedule split: CUDA nightly selects 2 cases, CUDA weekly selects 1, and NPU selects 8 with no overlap.
• Checked expansion collection after dead-case removal and reduced-token restoration: 26 tests; test_text_to_text_audio_001 selects default, async_chunk, and batch_token_2048.
• Checked all 11 Async Chunk configs after reducing random input/output lengths to 100/100; removed the now-invalid weekly H100 latency baseline.
• The unsupported-audio-format target is uniquely collected by the weekly H100 selector.
• Ran audio-format resolver and perf metadata tests: 40 passed previously; the post-adjustment perf metadata run passed 21 tests.
• Ran pre-commit on every changed file: all hooks passed, including Ruff, mypy, SPDX, Markdown, and Buildkite validation.

I did not find an additional correctness issue after these checks.

[amy-why-3459]

Thanks for the follow-up in a0051ea — the first-round items look addressed:
one Async Chunk JSON + marks, aac moved to the weekly invalid-param suite,
H100 numbers stripped from the A3 object, audio-only kept on multi-replica,
and reduced-token left on the 120 s / 5-way case.

Two things before this can land:

1. Please rebase onto current main. GitHub reports mergeable_state=dirty.
   Nightly test-nightly.yml on main has grown (MiniCPM-o perf / duplex
   Seed-TTS, etc.). Keep those jobs and only add
   -m "H100 and full_model and not slow" on the Async Chunk step.
   After the JSON split, running the file without -m starts all three
   test_name objects (NPU + CUDA nightly + CUDA weekly).

2. NPU still runs the full 8-point matrix every nightly while CUDA was
   trimmed to 10/100 + weekly 1/8/32. If A3 is collect-only until
   baselines exist, please say so in the Purpose. If the goal is less
   nightly time, trim NPU to the same slice.

Non-blocking: the CHELSIE+Chinese fold and dropping the 40-req
image+video point from single-replica / vLLM-text are fine given
multi-replica still has that contract. GPU log (28 collected) is
stale vs the post-cleanup collect-only 25 — no need to re-run the
full expansion module.

[yenuo26]

LGTM