Repository navigation
[Perf][Qwen3-Omni] Capture the thinker image encoder with CUDA graphs (part of #7452) - #8215
Conversation
Implement vLLM's encoder CUDA graph protocol for the Qwen3-Omni thinker's image tower, including the DeepStack outputs. It is enabled by setting cudagraph_mm_encoder on the thinker stage; video and audio keep the eager path. The vision metadata is built from the host grid, so preparing a replay reads nothing back from the device. A step's images replay only when one captured graph holds them all; otherwise the group goes back to the runner for one eager call, which for this launch-bound tower is faster than several replays or per-image eager calls in every group measured except five or more small images. The OmniGPUModelRunner installs that manager for models that set encoder_cudagraph_single_replay. Auto-inferred budgets stop at 256 image tokens, beyond which replay no longer beats the eager tower. Co-authored-by: zijianc2 <157244773+zijianc2@users.noreply.github.com> Co-authored-by: Yifan Tian <221685243+Sendoh-code@users.noreply.github.com> Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
This PR appears to belong to: docs/design/module/ar_runtime.md. Module owners: @Gaohan123 @tzhouam @fake0fan Routing: @Gaohan123 via module of the changed files, CODEOWNERS; @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files @MrlixiangWE, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
hsliuustc0106
added
high priority
0z5a
left a comment
0z5a
left a comment
There was a problem hiding this comment.
Supplemental cross-GPU-architecture validation on RTX 5090 (Blackwell), complementing the PR's A800 results. I tested the unchanged head with BF16 and FLASH_ATTN. The focused CPU suite passed 26/26, the CUDA graph suite passed 6/6, and two runner integration cases passed for graph replay and whole-group eager fallback. I also instantiated the full Qwen3-Omni vision-tower dimensions (depth 27, hidden size 1152) with random weights, captured budgets [64, 128, 256], and compared eager and graph outputs. All tested output groups passed the existing BF16 closeness check.
| Image input | Eager median | Graph median | Speedup |
|---|---|---|---|
| One 49-token image | 17.17 ms | 5.79 ms | 2.97× |
| Two 49-token images | 19.49 ms | 6.58 ms | 2.96× |
| One 98-token image | 17.29 ms | 6.19 ms | 2.79× |
The measured graph capture reduced free GPU memory by 186 MiB in this configuration. These runs use the real tower architecture with random weights; they supplement cross-architecture coverage and do not represent a full checkpoint serving benchmark. I found no actionable issue in the tested path.
LGTM
Omni ReviewBot: resolvedThe CI failure noted on |
Omni ReviewBot routing recordAssigned Strict under experiment |
vllm-omni-review-bot
left a comment
vllm-omni-review-bot
left a comment
There was a problem hiding this comment.
Omni ReviewBot review
PR description
This PR opts the Qwen3-Omni thinker image tower into vLLM’s encoder CUDA-graph path, off by default behind cudagraph_mm_encoder. The vision forward is split so position/attention metadata is built on the host before capture or replay, and a SingleReplayEncoderCudaGraphManager replays a step only when one captured budget holds every image; otherwise the runner runs one eager embed_multimodal call. Stage wrappers expose the protocol only on the thinker instance, auto budgets stop at 256 image tokens, and the recipe documents how to enable it.
Change flow
flowchart TD
A["[EXISTING] Thinker image encode<br/>embed_multimodal / runner"]:::existing
B["[CHANGED] Vision forward split<br/>prepare_encoder_metadata"]:::changed
C["[NEW] Qwen3OmniVisionEncoderCudaGraphMixin<br/>SupportsEncoderCudaGraph"]:::new
D["[CHANGED] OmniGPUModelRunner<br/>_create_encoder_cudagraph_manager"]:::changed
E["[NEW] SingleReplayEncoderCudaGraphManager<br/>one-graph or whole-group eager"]:::new
F["[EXISTING] Cached encoder outputs<br/>optional TTFT win"]:::existing
A --> B
B --> C
C --> D
D --> E
E --> F
classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px
CI at
6a6dc516842a(2026-09-29T20:02:21.939691+00:00): verification incomplete; required-check status is unknown.
See inline comments below.
🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!
| metadata["sequence_lengths"] = sequence_lengths | ||
| return metadata | ||
|
|
||
| def forward( |
There was a problem hiding this comment.
[P1] Path-gated Qwen3-Omni e2e guard not reported in Test Result
.buildkite/cuda/test-ready.yml step Omni · Qwen3-Omni Test declares source_file_dependencies: omni_qwen3_omni_function, which includes vllm_omni/model_executor/models/qwen3_omni/ (see .buildkite/common/ci_source_file_dependencies.yml). That step runs pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m 'core_model' --run-level 'core_model'. The PR Test Plan/Result cover tests/model_executor CPU/CUDA selectors matching Simple · Model Executor and the model_executor portion of Engine&Model Executor, plus manual serving timings, but do not report that e2e selector and do not state a deliberate skip. The always-on vision forward split in this file is what default (graph-off) e2e would exercise; please run that command or explicitly document the uncovered guard.
Omni ReviewBot: finding feedback[p1] Path-gated Qwen3-Omni e2e guard not reported in Test Result —
This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR. |
Preserve current wrapper and runner behavior while carrying the image encoder graph protocol. Use real nested Qwen HF configuration in the wrapper contract tests, including the current speculative/MTP fields. Co-authored-by: zijianc2 <157244773+zijianc2@users.noreply.github.com> Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
@hsliuustc0106 Ready for review. |
…raph gate vllm-project#8461 constructs the thinker without a pipeline group and with a stand-in vision tower. Check the tower's attention backend before asking for the pipeline rank, so a thinker without a real tower never reaches the distributed query. The encoder-graph test config gains the device_config that vllm-project#8461 now reads. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: zijianc2 <157244773+zijianc2@users.noreply.github.com> Co-authored-by: Yifan Tian <221685243+Sendoh-code@users.noreply.github.com>
amy-why-3459
left a comment
amy-why-3459
left a comment
There was a problem hiding this comment.
Reviewed head: 0ce25b37821b825ddab25b3d0659ff54c28b6ec9.
I checked the thinker-only protocol exposure, eligibility gate, host metadata preparation, padded sequence boundaries, DeepStack output ordering/ownership, and runner dispatch for graph hits, whole-group eager fallback, and audio/video. I found no confirmed functional defect in the inspected and tested paths.
Independent focused verification on this exact head:
python -m pytest -q -o addopts= --strict-markers --strict-config \
tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph.py \
-m 'core_model and cpu'
CUDA_VISIBLE_DEVICES=2 python -m pytest -q -o addopts= --strict-markers --strict-config \
tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph_cuda.py \
-m 'core_model and cuda' --run-level core_modelResults: 26 CPU tests passed; 6 CUDA tests passed, including native FLASH_ATTN/TRITON_ATTN capture and replay. Environment: NVIDIA L20X, Python 3.12, vLLM 0.31.0, PyTorch 2.13.0+cu130, Transformers 5.14.1, pytest 9.1.1. These are focused checks, not a full-suite run or full-checkpoint serving validation; my vLLM version also differs from the author's reported 0.30.0 environment.
Validation requests before merge:
-
Provide complete repository UT evidence on the current head. The PR description identifies
23a6ab2/ baseb65f97b; those results do not establish coverage for this merged head. The currently visible head checks cover pre-commit, package builds, DCO and docs, not the complete UT suite. In particular,tests/model_executor -m 'core_model and cpu'is only one of the four CPU Simple Test groups in the current CUDA ready pipeline. Please also provide results for the Diffusion, Engine&Entrypoints, and Other CPU groups, the accelerator UT jobs/shards defined by the repository CI, and required backend-specific UT jobs from the supported CUDA/AMD/NPU matrix. Use the pipeline's actual selectors, shard settings and exclusions rather than treating alltests/as unit tests. Attach exact commands, environment/dependency versions, passed/failed/skipped/deselected/collection-error counts, and CI links or complete logs tied to this SHA. Explain skips and unavailable jobs; obtain an explicit maintainer disposition for any remaining gap. If failures are pre-existing, attach a same-environment base/head comparison. Existing complete CI on this SHA would satisfy this request without another rerun. Please also refresh the path-gated Qwen3-Omni online ready-selector evidence for this head; the earlier bot concern has been addressed in the description for the older revision and should not be presented as a new code defect. -
Extend graph coverage for the enabled configurations. The new CUDA fixture initializes TP=1 and tests FLASH_ATTN/TRITON_ATTN, while the capability gate also enables ROCM_AITER_FA and permits TP>1. Please add or attach TP=2 weights-mode eager/replay comparisons covering multiple images, changing replay budgets, all DeepStack slices, and whole-group eager fallback; and provide native ROCM_AITER_FA capture/replay evidence on supported AMD hardware. This is a coverage request, not a claim that those configurations are broken.
The focused path looks sound, but merge readiness is not established until the full-suite evidence and remaining validation gaps are resolved or explicitly dispositioned by a maintainer.
amy-why-3459
added
ready
|
Could you please add a Daily-Omni benchmark at concurrency 10 comparing the Qwen3-Omni image encoder CUDA graph enabled versus disabled? Please use the Stage-0 switch introduced by this PR, Please report end-to-end TTFT and first-audio latency (p50/p95), throughput, failed requests, and GPU memory usage. Include encoder graph hit/miss or eager-fallback statistics and the input image-token distribution so we can understand how often the default capture budgets are exercised. Please attach the tested head SHA, exact commands/configurations, warmup and measurement settings, and result/log links. |
The ready and merge pipelines both ran on this head: Buildkite build 16785 on
The merge pipeline's Simple groups report the same counts as the ready ones. The skips are platform conditions: the 14 in the Qwen3-Omni job are its ROCm and NPU variants, and the rest need NPU, Hopper, more GPUs or optional packages such as
These pipelines fail the same way on other ready PRs from the same day, for example #7947, #8462 and #8339. Could you confirm them as pre-existing for this PR?
TP=2 on 2x A800 80GB PCIe at Per group on FLASH_ATTN, in the order the test runs them; TRITON_ATTN gives the same maxima. Grids are
The manager counts 8 graph hits and 3 misses, one per image. Shifting
|
Head
The two arms are equal within the spread between repeated runs. The Daily-Omni loader sends each clip as one
The same pairs with text-only output, where the talker stays idle, give +17.60% req/s and -14.89% TTFT p50.
Image encoder dispatch, counted in one graph-on pass per workload:
At concurrency 10 the scheduler sometimes puts several single images into one step, and those groups exceed the 256-token default cap. Raising the budgets replays half of them, for a further -2.09% TTFT p50 and 178 MiB more graph memory. The single-image p95 values vary more between repeated runs than between arms; they are in the attached results with everything else. Peaks are sampled once per second. The failed requests in the frame runs, here and in the text-only pairs (graph off 1 and 1, graph on 1 and 0), are all prompt 161, rejected in stage 0 with pr8215-daily-omni-results.zip has each run's vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--stage-overrides '{"0": {"compilation_config": {"cudagraph_mm_encoder": true}}}' # false for the other arm
vllm bench serve --omni --backend openai-chat-omni --endpoint /v1/chat/completions \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct --dataset-name daily-omni \
--daily-omni-qa-json qa.json --daily-omni-video-dir Videos --daily-omni-inline-local-video \
--daily-omni-input-mode all --num-prompts 1197 --max-concurrency 10 --num-warmups 10 --seed 0 \
--percentile-metrics ttft,e2el,audio_ttfp --metric-percentiles 50,95 \
--extra-body '{"modalities":["text","audio"]}' |
Purpose
Part of #7452: capture and replay the Qwen3-Omni thinker's image encoder, including DeepStack outputs, through vLLM's encoder CUDA graph protocol. Enable it with
cudagraph_mm_encoderon stage 0; it defaults to off.Propose
SupportsEncoderCudaGraphfor images and expose it through the thinker stage of the three-stage wrapper. Audio and video retain eager encoding.cu_seqlensentries.SingleReplayEncoderCudaGraphManagerfor models requesting it. Replay a group only when all its images fit one captured graph; otherwise send the whole group through one eager encoding call.Test Plan
vLLM-Omni Commit:
23a6ab2, merged with mainb65f97b.Environment: vLLM 0.30.0, PyTorch 2.13.0+cu130; NVIDIA A800 80GB.
CUDA protocol tests use a small real vision transformer with BF16 weights and native graph capture on FLASH_ATTN and TRITON_ATTN. They cover replay versus eager outputs, multiple images, retained outputs and metadata errors, plus a synchronization guard on host metadata preparation.
Additional online validation runs the existing
test_thinker_prefix_caching_async_chunk_shared_image_prefixcase with stage 0'scompilation_config.cudagraph_mm_encoder=true. The online tests use the CI deploy with dummy weights for the full Qwen3-Omni architecture. The encoder-enabled case sends two streamed text requests sharing an image and system prefix but using different user text, and retains the original response and block-aligned partial-prefix-cache assertions. The stage-0 process captures three encoder graphs at 64/128/256 tokens before serving the requests.The component benchmark loads the full image tower's real Qwen3-Omni-30B-A3B-Instruct weights. It compares this commit's eager and single-replay paths on 1x A800 80GB, BF16, FLASH_ATTN, automatically inferred budgets 64/128/256, up to 4 images per replay, one image per call, 10 warm-ups and 30 measured CUDA-event samples per size. These are image-tower timings.
Test Result
b65f97bBoth initial online failures occur while generating test audio, before the API request, because
pyttsx3is absent from the local test environment. The focused rerun providespyttsx3andespeak-ng, keeps the original test assertions, and exercises the five mixed text/audio/video/image requests. The skipped online cases target ROCm and NPU.Current component performance, median of 30 samples:
This replaces the earlier performance tables measured on development prototypes with measurements tied to the current production source. Serving first-token latency, TP=2 output comparisons and stage memory figures from those prototypes are excluded from these current results.