[Fix] Fix slow hasattr in CUDAGraphWrapper.__getattr__

[ZeldaHuang]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Notice that hasattr(self.model, "flush_pending_metadata") cost 6ms per decode step when profiling Qwen3 Omni.

The original CUDAGraphWrapper.__getattr__ raises:

raise AttributeError(f"... cudagraph wrapper: {self.runnable}")

When hasattr() is called for a non-existent attribute, Python internally calls getattr which constructs this AttributeError. The {self.runnable} triggers __repr__() on the underlying model (e.g., Qwen3OmniMoeForConditionalGeneration), which recursivelytraverses the entire nn.Module tree to generate an 18,000+ character string. This takes ~6-7ms per call.

Since hasattr(self.model, "flush_pending_metadata") is called every decode step in the Talker forward path, this adds ~6ms overhead per step, severely impacting audio inter-chunk latency (ICL).

Test Plan

vllm bench serve \
  --omni \
  --dataset-name random \
  --port 8091 \
  --max-concurrency 1 \
  --model /mnt/data/models/Qwen3-Omni-30B-A3B-Instruct/ \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --num-prompts 1 \
  --random-input-len 8000 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf \
  --random-output-len 100 \
  --extra_body '{"modalities": ["text", "audio"]}'

Test Result

before：

============ Serving Benchmark Result ============
Successful requests:                     1         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  7.85      
Request throughput (req/s):              0.13      
Peak concurrent requests:                1.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          7849.57   
Median E2EL (ms):                        7849.57   
P99 E2EL (ms):                           7849.57   
================== Text Result ===================
Total input tokens:                      8000      
Total generated tokens:                  100       
Output token throughput (tok/s):         12.74     
Peak output token throughput (tok/s):    84.00     
Peak concurrent requests:                1.00      
Total Token throughput (tok/s):          1031.83   
---------------Time to First Token----------------
Mean TTFT (ms):                          1029.55   
Median TTFT (ms):                        1029.55   
P99 TTFT (ms):                           1029.55   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.92     
Median TPOT (ms):                        11.92     
P99 TPOT (ms):                           11.92     
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.80     
Median ITL (ms):                         11.16     
P99 ITL (ms):                            50.24     
================== Audio Result ==================
Total audio duration generated(s):       22.52     
Total audio frames generated:            540540    
Audio throughput(audio duration/s):      2.87      
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    2155.17   
Median AUDIO_TTFP (ms):                  2155.17   
P99 AUDIO_TTFP (ms):                     2155.17   
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.35      
Median AUDIO_RTF:                        0.35      
P99 AUDIO_RTF:                           0.35      
==================================================

after:

============ Serving Benchmark Result ============
Successful requests:                     1         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  4.75      
Request throughput (req/s):              0.21      
Peak concurrent requests:                1.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          4751.33   
Median E2EL (ms):                        4751.33   
P99 E2EL (ms):                           4751.33   
================== Text Result ===================
Total input tokens:                      8000      
Total generated tokens:                  100       
Output token throughput (tok/s):         21.04     
Peak output token throughput (tok/s):    99.00     
Peak concurrent requests:                1.00      
Total Token throughput (tok/s):          1704.60   
---------------Time to First Token----------------
Mean TTFT (ms):                          1001.51   
Median TTFT (ms):                        1001.51   
P99 TTFT (ms):                           1001.51   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.37     
Median TPOT (ms):                        10.37     
P99 TPOT (ms):                           10.37     
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.26     
Median ITL (ms):                         10.14     
P99 ITL (ms):                            46.88     
================== Audio Result ==================
Total audio duration generated(s):       22.52     
Total audio frames generated:            540540    
Audio throughput(audio duration/s):      4.74      
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    1837.54   
Median AUDIO_TTFP (ms):                  1837.54   
P99 AUDIO_TTFP (ms):                     1837.54   
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.21      
Median AUDIO_RTF:                        0.21      
P99 AUDIO_RTF:                           0.21      
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[ZeldaHuang]

Better commit to vllm repo if its works.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09ec759ef3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Patch UBatchWrapper too for ubatched decode paths

In configs that enable DBO/ubatching, self.model is not a CUDAGraphWrapper at the top level anymore: upstream GPUModelRunner wraps it in UBatchWrapper, and UBatchWrapper.__getattr__ has the same expensive AttributeError(... {self.runnable}) pattern. Because this monkey-patch only rewrites CUDAGraphWrapper, the hot hasattr(self.model, "flush_pending_metadata") call in gpu_ar_model_runner.py still pays the full repr cost whenever ubatching is on, so the latency fix silently disappears in that deployment mode.

Useful? React with 👍 / 👎.

[Gaohan123]

Great catch!

[Gaohan123]

Please supplement a simple test to protect the optimization. Thanks

[ZeldaHuang]

Please supplement a simple test to protect the optimization. Thanks

Done

[hsliuustc0106]

it seems the upstream is going to merge the changes, shall we close this PR after it's fixed in upstream?

[ZeldaHuang]

@hsliuustc0106 If vllm-project/vllm#37425 will be included in v0.18.0, I think we can close this PR.

[hsliuustc0106]

@hsliuustc0106 If vllm-project/vllm#37425 will be included in v0.18.0, I think we can close this PR.

it's merged

[hsliuustc0106]

we will revert this PR in vllm-omni 0.19.0rc1