[perf]Qwen3-Omni performance optimization by amy-why-3459 · Pull Request #3203 · vllm-project/vllm-omni

amy-why-3459 · 2026-04-28T05:26:10Z

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd9b3f4ffd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-04-28T05:31:41Z

+            extra_cli_args,
+            use_omni,
+        )


Keep reliability tuple unpacking compatible

Adding use_omni to server_param makes create_unique_server_params emit 6-tuples, but create_reliability_omni_server_params in the same file still unpacks each entry as 5 values. When reliability tests initialize QWEN_PARAMS/WAN_PARAMS, this now raises ValueError: too many values to unpack, so those suites fail before running any test logic.

Useful? React with 👍 / 👎.

hsliuustc0106 · 2026-04-28T05:53:11Z

many ci failed

gcanlin

LGTM

amy-why-3459 · 2026-04-28T08:06:46Z

In the main branch, this test case also appears to fail.

ZeldaHuang · 2026-04-28T08:29:12Z

+    def _warmup_single_request_prefill_compile_path(self) -> None:
+        # Warm up with a single long prefill request to avoid first-token
+        # compile spikes from wide prefill shapes.
+        warmup_tokens = min(int(self.max_num_tokens), 3072)


Why 3072? Is it intermediate_size of qwen omni?

Why 3072? Is it intermediate_size of qwen omni?

Fixed

hsliuustc0106 · 2026-04-28T09:44:07Z

+
+    def _maybe_prune_downstream_payload_cache(self) -> None:
+        # Keep cache size bounded under long-lived serving workloads.
+        if len(self._downstream_payload_cache) <= max(4096, len(self.requests) * 2):


hardcoded 4096

hardcoded 4096

Fixed

amy-why-3459 · 2026-04-28T09:45:28Z

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

hsliuustc0106 · 2026-04-30T03:49:20Z

retest it now under vllm 0.20.0

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: sphinxkkkbc <binchengkang8@gmail.com>

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

amy-why-3459 requested a review from hsliuustc0106 as a code owner April 28, 2026 05:26

hsliuustc0106 added ready label to trigger buildkite CI merge-test label to trigger buildkite merge test CI labels Apr 28, 2026

chatgpt-codex-connector Bot reviewed Apr 28, 2026

View reviewed changes

amy-why-3459 force-pushed the perf branch from bd9b3f4 to b2e102d Compare April 28, 2026 05:53

gcanlin reviewed Apr 28, 2026

View reviewed changes

Comment thread vllm_omni/worker/gpu_ar_model_runner.py Outdated

amy-why-3459 force-pushed the perf branch from 13f676a to da109a6 Compare April 28, 2026 06:42

gcanlin approved these changes Apr 28, 2026

View reviewed changes

ZeldaHuang reviewed Apr 28, 2026

View reviewed changes

amy-why-3459 force-pushed the perf branch from 64e987a to 2ff2989 Compare April 28, 2026 08:52

hsliuustc0106 reviewed Apr 28, 2026

View reviewed changes

amy-why-3459 force-pushed the perf branch 3 times, most recently from f1acfe7 to 04f92d8 Compare April 28, 2026 11:26

hsliuustc0106 removed ready label to trigger buildkite CI merge-test label to trigger buildkite merge test CI labels Apr 29, 2026

Gaohan123 added this to the v0.20.0 milestone Apr 29, 2026

amy-why-3459 added 2 commits April 30, 2026 09:08

Add text-only performance test cases for vllm

b6192d2

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

[perf]Qwen3-Omni performance optimization (vllm-project#3164)

4b8df56

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

amy-why-3459 force-pushed the perf branch from 40e5a5b to 4b8df56 Compare April 30, 2026 01:09

Merge branch 'main' into perf

77f1f27

hsliuustc0106 added ready label to trigger buildkite CI omni-test label to trigger buildkite omni model test in nightly CI labels Apr 30, 2026

hsliuustc0106 mentioned this pull request Apr 30, 2026

[FEAT] support multi-stage deployment #2396

Merged

5 tasks

Merge branch 'main' into perf

e56af85

hsliuustc0106 removed the omni-test label to trigger buildkite omni model test in nightly CI label Apr 30, 2026

Merge branch 'main' into perf

b434cc6

Gaohan123 added the high priority high priority issue, needs to be done asap label Apr 30, 2026

Merge branch 'main' into perf

1665329

hsliuustc0106 added omni-test label to trigger buildkite omni model test in nightly CI labels Apr 30, 2026

Merge branch 'main' into perf

e0ca2ca

hsliuustc0106 merged commit 01f500a into vllm-project:main May 1, 2026
7 of 8 checks passed

lengrongfu pushed a commit to lengrongfu/vllm-omni that referenced this pull request May 1, 2026

[perf]Qwen3-Omni performance optimization (vllm-project#3203)

8b1e1ae

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

alex-jw-brooks mentioned this pull request May 2, 2026

[Core] Support Async & Sync AutoRegressive Scheduling #3306

Merged

Shirley125 mentioned this pull request May 8, 2026

[Bug]: [CI failure] Audio Streaming Input Test with H100 fails occasionally #3418

Closed

1 task

Sy0307 mentioned this pull request May 10, 2026

[Perf] Fix Qwen3-TTS latency regression #3485

Merged

10 tasks

Yadan-Wei mentioned this pull request May 12, 2026

feat(vllm-omni): align chat-omni token counting + 3 new benchmarks + ICE workaround for qwen2.5-omni-3b aws/deep-learning-containers#6079

Merged

6 tasks

clodaghwalsh17 pushed a commit to clodaghwalsh17/nm-vllm-omni-ent that referenced this pull request May 12, 2026

[perf]Qwen3-Omni performance optimization (vllm-project#3203)

b33b4f6

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

Yadan-Wei mentioned this pull request May 13, 2026

[Docs Update] vLLM-Omni 0.20.0 aws/deep-learning-containers#6088

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[perf]Qwen3-Omni performance optimization#3203

[perf]Qwen3-Omni performance optimization#3203
hsliuustc0106 merged 7 commits into
vllm-project:mainfrom
amy-why-3459:perf

amy-why-3459 commented Apr 28, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 28, 2026

Uh oh!

hsliuustc0106 commented Apr 28, 2026

Uh oh!

Uh oh!

gcanlin left a comment

Uh oh!

amy-why-3459 commented Apr 28, 2026

Uh oh!

ZeldaHuang Apr 28, 2026

Uh oh!

amy-why-3459 Apr 28, 2026

Uh oh!

hsliuustc0106 Apr 28, 2026

Uh oh!

amy-why-3459 Apr 28, 2026

Uh oh!

amy-why-3459 commented Apr 28, 2026 •

edited

Loading

Uh oh!

hsliuustc0106 commented Apr 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

amy-why-3459 commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 commented Apr 28, 2026

Uh oh!

Uh oh!

gcanlin left a comment

Choose a reason for hiding this comment

Uh oh!

amy-why-3459 commented Apr 28, 2026

Uh oh!

ZeldaHuang Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

amy-why-3459 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

amy-why-3459 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

amy-why-3459 commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsliuustc0106 commented Apr 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

amy-why-3459 commented Apr 28, 2026 •

edited

Loading

amy-why-3459 commented Apr 28, 2026 •

edited

Loading