[perf]Qwen3-Omni performance optimization

[amy-why-3459]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Fix: #3093

Qwen3-Omni performance optimization

Test Plan

vllm bench serve \
    --omni \
  --dataset-name random \
  --port 28889 \
  --max-concurrency 1 \
  --model /home/wuhaiyan/Qwen3-Omni-30B-A3B-Instruct \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --num-prompts 10 \
  --random-input-len 2500 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf \
  --random-output-len 100 \
  --extra_body '{"modalities": ["text"]}'

Test Result

Before:

============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  9.40      
Request throughput (req/s):              1.06      
Peak concurrent requests:                2.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          939.33    
Median E2EL (ms):                        935.98    
P99 E2EL (ms):                           959.29    
================== Text Result ===================
Total input tokens:                      25480     
Total generated tokens:                  876       
Output token throughput (tok/s):         93.22     
Peak output token throughput (tok/s):    111.00    
Peak concurrent requests:                2.00      
Total Token throughput (tok/s):          2804.75   
---------------Time to First Token----------------
Mean TTFT (ms):                          124.68    
Median TTFT (ms):                        119.81    
P99 TTFT (ms):                           150.19    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.69      
Median TPOT (ms):                        8.92      
P99 TPOT (ms):                           13.25     
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.15      
Median ITL (ms):                         7.50      
P99 ITL (ms):                            18.69     
================== Audio Result ==================

After:

============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  5.43      
Request throughput (req/s):              1.84      
Peak concurrent requests:                3.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          543.07    
Median E2EL (ms):                        531.94    
P99 E2EL (ms):                           590.33    
================== Text Result ===================
Total input tokens:                      25480     
Total generated tokens:                  876       
Output token throughput (tok/s):         161.21    
Peak output token throughput (tok/s):    190.00    
Peak concurrent requests:                3.00      
Total Token throughput (tok/s):          4850.25   
---------------Time to First Token----------------
Mean TTFT (ms):                          79.55     
Median TTFT (ms):                        73.47     
P99 TTFT (ms):                           135.16    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.50      
Median TPOT (ms):                        5.29      
P99 TPOT (ms):                           7.47      
---------------Inter-token Latency----------------
Mean ITL (ms):                           4.64      
Median ITL (ms):                         3.29      
P99 ITL (ms):                            7.61      
================== Audio Result ==================

Qwen3-Omni:
text+audio:

test_name	max_concurrency	RTF（Before)	RTF(After)	Delta	%Delta
qwen3_omni	1	0.24	0.24	0.00	+0.00%
qwen3_omni	4	0.34	0.21	-0.13	-38.24%
qwen3_omni	8	0.40	0.25	-0.15	-37.50%
qwen3_omni	16	0.56	0.39	-0.17	-30.36%
qwen3_omni	32	0.82	0.54	-0.28	-34.15%
qwen3_omni_chunk	1	0.16	0.15	-0.01	-6.25%
qwen3_omni_chunk	4	0.31	0.25	-0.06	-19.35%
qwen3_omni_chunk	8	0.40	0.38	-0.02	-5.00%
qwen3_omni_chunk	16	0.78	0.80	+0.02	+2.56%

text-only:

test_name	max_concurrency	request_rate	E2E(Before)	E2E(After)	Delta	%Delta
qwen3_omni_chunk	1	-	7383.12	4741.55	-2641.57	-35.78%
qwen3_omni_chunk	4	-	17646.13	7597.20	-10048.93	-56.95%
qwen3_omni_chunk	8	-	20778.37	9882.98	-10895.39	-52.44%
qwen3_omni_chunk	16	-	30034.60	13530.12	-16504.48	-54.95%
qwen3_omni_chunk	32	-	43792.69	19114.88	-24677.81	-56.35%
qwen3_omni_chunk	-	0.1	1201.87	952.74	-249.13	-20.73%
qwen3_omni_chunk	-	0.5	1190.62	747.09	-443.53	-37.25%
qwen3_omni_chunk	-	1	2117.33	1354.09	-763.24	-36.05%

Qwen3-TTS:

test_name	max_concurrency	RTF（Before)	RTF(After)	Delta	%Delta
qwen3_tts_customvoice	1	0.16	0.15	-0.01	-6.25%
qwen3_tts_customvoice	1	0.48	0.43	-0.05	-10.42%
qwen3_tts_customvoice	8	0.36	0.24	-0.12	-33.33%
qwen3_tts_customvoice	8	0.37	0.26	-0.11	-29.73%

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5d8b019d2d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Clone hidden-state CPU buffer before exposing to async paths

Returning dst directly here leaks a view into self._hidden_states_cpu_pinned, which is reused and overwritten on the next step. In chunk-transfer mode, omni_ar_scheduler.update_from_output passes pooler_output to OmniChunkTransferAdapter.save_async, and _send_single_request consumes it later on a background thread, so enqueued hidden tensors can be mutated before send and carry wrong step data. The previous code allocated a fresh CPU tensor each step, so this introduces a new correctness regression.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Avoid returning reusable MM pinned buffers to output payloads

This returns cached pinned tensors by reference, but downstream payload construction (to_payload_element) often keeps slices/views of those tensors. Because GPUARModelRunner.sample_tokens now enables tensor_buffer_cache, the same buffers are rewritten on subsequent steps; with async chunk sending (save_async + background _send_single_request), previously queued multimodal payloads can be overwritten before transmission. That can corrupt inter-stage multimodal data for active requests.

Useful? React with 👍 / 👎.

[hsliuustc0106]

is there any reason why async chunk scenarios did not benifit from the modifications?

[amy-why-3459]

is there any reason why async chunk scenarios did not benifit from the modifications?

is there any reason why async chunk scenarios did not benifit from the modifications?

The main benefit of this modification comes from async_scheduling. However, asynchronous scheduling is already implemented to some extent in async_chunk mode, so the improvement to async_chunk is not as significant. There will be some benefit for text-only output. I will update the performance comparison results for text-only output after the test cases are completed.

[amy-why-3459]

is there any reason why async chunk scenarios did not benifit from the modifications?

is there any reason why async chunk scenarios did not benifit from the modifications?

The main benefit of this modification comes from async_scheduling. However, asynchronous scheduling is already implemented to some extent in async_chunk mode, so the improvement to async_chunk is not as significant. There will be some benefit for text-only output. I will update the performance comparison results for text-only output after the test cases are completed.

Updated

[amy-why-3459]

is there any reason why async chunk scenarios did not benifit from the modifications?

Performance data for the Qwen3-TTS has been updated.