[Optimize][Qwen3-Omni] Reduce inter-packet latency in async chunk

[ZeldaHuang]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Reduce inter-packet latency by per-token decode embedding passing in async chunk，previous torch.cat is time costing in large context(video request etc)

vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py

Line 192 in 0c6a10e

payload_data[key] = torch.cat([origin_payload[key], value], dim=0)

Test Plan

vllm bench serve \
    --omni \
    --dataset-name random \
    --port 8091 \
    --max-concurrency 1 \
    --model /mnt/data/models/Qwen3-Omni-30B-A3B-Instruct \
    --endpoint /v1/chat/completions \
    --backend openai-chat-omni \
    --num-prompts 1 \
    --random-input-len 8000 \
    --ignore-eos \
    --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf \
    --random-output-len 100 \
    --extra_body '{"modalities": ["text", "audio"]}'

Test Result

before this PR

============ Serving Benchmark Result ============
Successful requests:                     1         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  21.28     
Request throughput (req/s):              0.05      
Peak concurrent requests:                1.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          21277.44  
Median E2EL (ms):                        21277.44  
P99 E2EL (ms):                           21277.44  
================== Text Result ===================
Total input tokens:                      8000      
Total generated tokens:                  100       
Output token throughput (tok/s):         4.70      
Peak output token throughput (tok/s):    94.00     
Peak concurrent requests:                1.00      
Total Token throughput (tok/s):          380.68    
---------------Time to First Token----------------
Mean TTFT (ms):                          908.48    
Median TTFT (ms):                        908.48    
P99 TTFT (ms):                           908.48    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          13.05     
Median TPOT (ms):                        13.05     
P99 TPOT (ms):                           13.05     
---------------Inter-token Latency----------------
Mean ITL (ms):                           13.18     
Median ITL (ms):                         9.40      
P99 ITL (ms):                            55.72     
================== Audio Result ==================
Total audio duration generated(s):       29.65     
Total audio frames generated:            711675    
Audio throughput(audio duration/s):      1.39      
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    3197.30   
Median AUDIO_TTFP (ms):                  3197.30   
P99 AUDIO_TTFP (ms):                     3197.30   
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.71      
Median AUDIO_RTF:                        0.71      
P99 AUDIO_RTF:                           0.71      
==================================================

after this PR

============ Serving Benchmark Result ============
Successful requests:                     1         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  7.37      
Request throughput (req/s):              0.14      
Peak concurrent requests:                1.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          7368.84   
Median E2EL (ms):                        7368.84   
P99 E2EL (ms):                           7368.84   
================== Text Result ===================
Total input tokens:                      8000      
Total generated tokens:                  100       
Output token throughput (tok/s):         13.57     
Peak output token throughput (tok/s):    86.00     
Peak concurrent requests:                1.00      
Total Token throughput (tok/s):          1099.15   
---------------Time to First Token----------------
Mean TTFT (ms):                          824.10    
Median TTFT (ms):                        824.10    
P99 TTFT (ms):                           824.10    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          13.43     
Median TPOT (ms):                        13.43     
P99 TPOT (ms):                           13.43     
---------------Inter-token Latency----------------
Mean ITL (ms):                           13.71     
Median ITL (ms):                         9.42      
P99 ITL (ms):                            137.64    
================== Audio Result ==================
Total audio duration generated(s):       25.46     
Total audio frames generated:            611025    
Audio throughput(audio duration/s):      3.45      
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    1796.20   
Median AUDIO_TTFP (ms):                  1796.20   
P99 AUDIO_TTFP (ms):                     1796.20   
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.28      
Median AUDIO_RTF:                        0.28      
P99 AUDIO_RTF:                           0.28      
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b66dbf09f7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Preserve thinker embeddings on no-token async chunks

When output_token_ids is empty (the code comment says this happens while chunked thinker prefill is still running), this branch updates only thinker_hidden_states and never refreshes thinker_embeddings. The new decode path later computes start_index from num_processed_tokens - thinker_embeddings.shape[0] in _thinker_decode_to_talker_decode, so if prefill spans additional chunks after chunk 0, thinker_embeddings.shape[0] becomes stale and the talker can jump past available decode embeddings and emit EOS/pad early.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Append decode embeddings per token, not per chunk

This stores the entire decode embedding tensor as a single list element regardless of how many tokens were produced in the chunk. The consumer indexes thinker_decode_embeddings_list by token position (start_index), so any chunk containing multiple generated tokens will look like length 1 and all but the first token embedding in that chunk become unreachable, causing premature EOS/pad behavior for those requests.

Useful? React with 👍 / 👎.

[amy-why-3459]

Thank you very much for your contribution, LGTM

[amy-why-3459]

How will this PR affect E2E in a high-concurrency scenario?

[amy-why-3459]

`vllm bench serve     --omni   --dataset-name random   
--port 28889   --max-concurrency 10   
--model /home/models/Qwen3-Omni-30B-A3B-Instruct   
--endpoint /v1/chat/completions   
--backend openai-chat-omni   
--num-prompts 100   
--random-input-len 100   
--ignore-eos   
--percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf  
 --random-output-len 100   
--extra_body '{"modalities": ["text", "audio"]}'`

[hsliuustc0106]

`vllm bench serve     --omni   --dataset-name random   
--port 28889   --max-concurrency 10   
--model /home/models/Qwen3-Omni-30B-A3B-Instruct   
--endpoint /v1/chat/completions   
--backend openai-chat-omni   
--num-prompts 100   
--random-input-len 100   
--ignore-eos   
--percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf  
 --random-output-len 100   
--extra_body '{"modalities": ["text", "audio"]}'`

any comparison for this case?

[amy-why-3459]

`vllm bench serve     --omni   --dataset-name random   
--port 28889   --max-concurrency 10   
--model /home/models/Qwen3-Omni-30B-A3B-Instruct   
--endpoint /v1/chat/completions   
--backend openai-chat-omni   
--num-prompts 100   
--random-input-len 100   
--ignore-eos   
--percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf  
 --random-output-len 100   
--extra_body '{"modalities": ["text", "audio"]}'`

any comparison for this case?

In this scenario, the E2E latency degradation is quite significant and requires further analysis.

[ZeldaHuang]

`vllm bench serve     --omni   --dataset-name random   
--port 28889   --max-concurrency 10   
--model /home/models/Qwen3-Omni-30B-A3B-Instruct   
--endpoint /v1/chat/completions   
--backend openai-chat-omni   
--num-prompts 100   
--random-input-len 100   
--ignore-eos   
--percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf  
 --random-output-len 100   
--extra_body '{"modalities": ["text", "audio"]}'`

decode_embedding tensor list serialization cause performance degradation,
future optimization fix this
before pr:

============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Maximum request concurrency:             10        
Benchmark duration (s):                  17.21     
Request throughput (req/s):              0.58      
Peak concurrent requests:                10.00     
----------------End-to-end Latency----------------
Mean E2EL (ms):                          15702.55  
Median E2EL (ms):                        16458.21  
P99 E2EL (ms):                           17198.42  
================== Text Result ===================
Total input tokens:                      1000      
Total generated tokens:                  1000      
Output token throughput (tok/s):         58.11     
Peak output token throughput (tok/s):    321.00    
Peak concurrent requests:                10.00     
Total Token throughput (tok/s):          116.21    
---------------Time to First Token----------------
Mean TTFT (ms):                          419.02    
Median TTFT (ms):                        452.20    
P99 TTFT (ms):                           453.57    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          34.18     
Median TPOT (ms):                        34.38     
P99 TPOT (ms):                           39.06     
---------------Inter-token Latency----------------
Mean ITL (ms):                           33.84     
Median ITL (ms):                         18.96     
P99 ITL (ms):                            985.66    
================== Audio Result ==================
Total audio duration generated(s):       251.69    
Total audio frames generated:            6040575   
Audio throughput(audio duration/s):      14.62     
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    2224.78   
Median AUDIO_TTFP (ms):                  2232.00   
P99 AUDIO_TTFP (ms):                     2768.72   
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.64      
Median AUDIO_RTF:                        0.61      
P99 AUDIO_RTF:                           0.86      
==================================================

after pr:

============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Maximum request concurrency:             10        
Benchmark duration (s):                  17.57     
Request throughput (req/s):              0.57      
Peak concurrent requests:                10.00     
----------------End-to-end Latency----------------
Mean E2EL (ms):                          15568.83  
Median E2EL (ms):                        16396.14  
P99 E2EL (ms):                           17552.55  
================== Text Result ===================
Total input tokens:                      1000      
Total generated tokens:                  1000      
Output token throughput (tok/s):         56.91     
Peak output token throughput (tok/s):    328.00    
Peak concurrent requests:                10.00     
Total Token throughput (tok/s):          113.81    
---------------Time to First Token----------------
Mean TTFT (ms):                          270.63    
Median TTFT (ms):                        294.26    
P99 TTFT (ms):                           295.23    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          44.25     
Median TPOT (ms):                        44.38     
P99 TPOT (ms):                           54.63     
---------------Inter-token Latency----------------
Mean ITL (ms):                           43.80     
Median ITL (ms):                         0.01      
P99 ITL (ms):                            1308.06   
================== Audio Result ==================
Total audio duration generated(s):       264.38    
Total audio frames generated:            6345255   
Audio throughput(audio duration/s):      15.05     
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    1828.64   
Median AUDIO_TTFP (ms):                  1848.61   
P99 AUDIO_TTFP (ms):                     2365.43   
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.59      
Median AUDIO_RTF:                        0.58      
P99 AUDIO_RTF:                           0.64      
==================================================

[amy-why-3459]

The override_keys are the same for each step, so we don't need to accumulate them or transmit them every time.

[amy-why-3459]

Thank you very much for your contribution, nice work, LGTM.

[hsliuustc0106]

lgtm