Talker CUDA graph for Qwen-3 TTS

[evezhier]

Purpose

This PR enables CUDA graph for Qwen3 TTS with batching support.

Test Plan

[Done] Performance benchmark
[Done] Numerical correctness tests
[Done] Batching support
[In progress] Accuracy issue with increased concurrencies
[TODO] Benchmark update for batching

Test Result

Performance (L40S)

vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml --omni --port 8000 --trust-remote-code

vllm-omni bench serve --backend openai-audio-speech --endpoint /v1/audio/speech --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --dataset-name hf --dataset-path philschmid/mt-bench --skip-chat-template --num-prompts 10 --max_concurrency $MAX_CONCURRENCY --percentile-metrics e2el,audio_rtf,audio_duration --extra-body '{"voice": "Vivian", "instructions": "Speak with great enthusiasm", "max_new_tokens": 1024}'

Main

============ Serving Benchmark Result ============                                                                                                                                        
Successful requests:                     10                                                                                                                                               
Failed requests:                         0                                                                                                                                                
Maximum request concurrency:             1                                                                                                                                                
Benchmark duration (s):                  117.77                                                                                                                                           
Request throughput (req/s):              0.08                                                                                                                                             
Peak concurrent requests:                2.00                                                                                                                                             
----------------End-to-end Latency----------------                                                                                                                                        
Mean E2EL (ms):                          11776.92                                                                                                                                         
Median E2EL (ms):                        7764.31                                                                                                                                          
P99 E2EL (ms):                           32342.57                                                                                                                                         
================== Text Result ===================                                                                                                                                        
Total input tokens:                      785                                                                                                                                              
Total generated tokens:                  0                                                                                                                                                
Output token throughput (tok/s):         0.00      
Peak output token throughput (tok/s):    1.00      
Peak concurrent requests:                2.00      
Total Token throughput (tok/s):          6.67       
================== Audio Result ==================  
Total audio duration generated(s):       238.16     
Total audio frames generated:            5715840    
Audio throughput(audio duration/s):      2.02       
-----------------Real Time Factor-----------------  
Mean AUDIO_RTF:                          0.48      
Median AUDIO_RTF:                        0.47       
P99 AUDIO_RTF:                           0.58      
------------------Audio Duration------------------                                                                                                                                        
Mean AUDIO_DURATION (s):                 23.82                                                                                                                                            
Median AUDIO_DURATION (s):               16.72                                                                                                                                            
P99 AUDIO_DURATION (s):                  60.49      
==================================================


============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Maximum request concurrency:             8         
Benchmark duration (s):                  59.90     
Request throughput (req/s):              0.17      
Peak concurrent requests:                9.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          29868.76  
Median E2EL (ms):                        26344.04  
P99 E2EL (ms):                           59847.30  
================== Text Result ===================
Total input tokens:                      785       
Total generated tokens:                  0         
Output token throughput (tok/s):         0.00      
Peak output token throughput (tok/s):    8.00      
Peak concurrent requests:                9.00      
Total Token throughput (tok/s):          13.11     
================== Audio Result ==================
Total audio duration generated(s):       237.60    
Total audio frames generated:            5702400   
Audio throughput(audio duration/s):      3.97      
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          1.40      
Median AUDIO_RTF:                        1.35      
P99 AUDIO_RTF:                           2.34      
------------------Audio Duration------------------
Mean AUDIO_DURATION (s):                 23.76     
Median AUDIO_DURATION (s):               18.48     
P99 AUDIO_DURATION (s):                  55.49     
==================================================

This PR

============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  40.97     
Request throughput (req/s):              0.24      
Peak concurrent requests:                2.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          4096.87   
Median E2EL (ms):                        2811.94   
P99 E2EL (ms):                           10299.14  
================== Text Result ===================
Total input tokens:                      785       
Total generated tokens:                  0         
Output token throughput (tok/s):         0.00      
Peak output token throughput (tok/s):    1.00      
Peak concurrent requests:                2.00      
Total Token throughput (tok/s):          19.16     
================== Audio Result ==================
Total audio duration generated(s):       238.16    
Total audio frames generated:            5715840   
Audio throughput(audio duration/s):      5.81      
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.17      
Median AUDIO_RTF:                        0.17      
P99 AUDIO_RTF:                           0.18      
------------------Audio Duration------------------
Mean AUDIO_DURATION (s):                 23.82     
Median AUDIO_DURATION (s):               16.72     
P99 AUDIO_DURATION (s):                  60.49     
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 229bee61af

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Fall back when CUDA-graph talker receives batched decode

This hard-fails any decode batch larger than 1, but load_model now enables the TTS CUDA-graph path whenever full cudagraph mode is on, and _talker_mtp_forward invokes talker_mtp on the whole decode batch. With two concurrent decode requests, this raises ValueError and aborts the step instead of degrading to eager execution, which is a production-facing regression from the previous batched eager path.

Useful? React with 👍 / 👎.

[hsliuustc0106]

why audio results are the same?

------------------Audio Duration------------------
Mean AUDIO_DURATION (s): 23.82
Median AUDIO_DURATION (s): 16.72
P99 AUDIO_DURATION (s): 60.49

[evezhier]

why audio results are the same?

------------------Audio Duration------------------ Mean AUDIO_DURATION (s): 23.82 Median AUDIO_DURATION (s): 16.72 P99 AUDIO_DURATION (s): 60.49

@hsliuustc0106 because I'm in the middle of updating the benchmarks after the rebase and batching support update :) I'll mark this draft for now, but the code is here, the numbers are arriving shortly.

[evezhier]

Closing this as incompatible with #1913