[amy-why-3459]

Purpose

Extend Qwen3-Omni Thinker encoder graphs from images to video and audio, as an explicit opt-in on the V1 CUDA and supported Ascend NPU runners (excluding 310P). Shipped deployment profiles keep encoder graphs disabled to preserve default KV-cache capacity. MRv2 rejects encoder-graph opt-in before tower construction because its manager does not enforce the required replay bounds.

• Capture one full-width audio CNN/transformer graph per chunk count (1–64). Short-tail padding occupies isolated attention sequences and is discarded. Padding can change floating-point rounding; validate accuracy on the target backend and workload.
• Share the audio feature arena across graphs and reuse graph-local device metadata for unchanged clip lengths. Cache one host replay plan for consecutive identical lengths; invalidate it on clear/recapture. Changed lengths refresh the plan and required device metadata. Outputs remain cloned for encoder-cache ownership.
• Support video and DeepStack outputs, with eager fallback outside spatial/frame/item/token bounds. Automatic vision budgets stop at 4096; larger explicit budgets trade KV capacity for graph coverage.
• Use CUDA's unified graph-memory profiler without a second encoder reservation. Skip encoder profiling when graph mode is NONE; retain the guarded NPU reservation path and per-replay FIA updates.

Daily-Omni full set: graphs off vs frame-chunk replay

Measured on NVIDIA L20X with Qwen/Qwen3-Omni-30B-A3B-Instruct. Both runs used the full Daily-Omni set (1197 prompts, --no-oversample), seed 0, concurrency 10, temperature 0, max output 128, text-only modalities, and input mode all. Graphs off ran on GPUs 6 and 7. Frame-chunk replay (6f694be94d) ran later the same day on GPUs 1 and 3. These are not a simultaneous same-GPU pair, so part of the delta can be run-to-run or device noise.

Both runs completed 1197 requests with 0 failures. Input tokens 11,979,637 and output tokens 2,394 were identical. Accuracy was 857/1197 = 71.60% on both sides.

Metric	Graphs off	Frame-chunk replay	Change
Benchmark duration (s)	554.88	420.42	-24.2%
Request throughput (req/s)	2.16	2.85	+31.9%
Mean E2EL (ms)	4615.06	3494.15	-24.3%
Median E2EL (ms)	4047.06	3193.12	-21.1%
P99 E2EL (ms)	20024.83	7203.72	-64.0%
Mean TTFT (ms)	4332.12	3136.95	-27.6%
Median TTFT (ms)	3797.11	2910.19	-23.4%
P99 TTFT (ms)	19442.68	6693.63	-65.6%
Mean TPOT (ms)	282.88	357.13	+26.2%
P99 TPOT (ms)	1098.35	1080.27	-1.6%

Frame-chunk replay logged vision hits/misses 2752/0. The hit count is chunk replays, not requests: a video longer than one captured budget replays that same graph on successive frames. Audio hits/misses reached 896/2. The two audio misses were clips of 122 chunks, above the captured 1–64 range, and stayed eager.

Mean output length is 2 tokens, so TPOT is one interval and is not evidence of a decode regression. The serving difference is in TTFT and end-to-end latency. P99 improved more than the mean; that tail was not characterized separately. This comparison does not establish the same gain on other GPUs, concurrencies, or NPU.

Current validation

Reviewed 56a731a9c4072345ec412e993b155ce768c312ff; the host-plan optimization and regressions are in a8bb9468bb8d6cb35f80a36ac01eb8d405e08147.

Environment: Python 3.12.13, vLLM 0.31.0, PyTorch 2.13.0+cu130, Transformers 5.14.1, NVIDIA L20X. All commands sourced /data/why/scripts/env.sh first. Tests imported this checkout; installed vLLM-Omni version metadata still emits a mismatch warning.

CUDA_VISIBLE_DEVICES=0 python -m pytest \
  tests/model_executor/models/qwen3_omni/test_audio_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph_cuda.py \
  --run-level core_model -q -ra
# 14 passed; exit 0

CUDA_VISIBLE_DEVICES=6 python -m pytest \
  tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_encoder_graphs_npu.py \
  tests/worker/test_worker_base.py --run-level core_model -q -ra
# 70 passed, 5 skipped; exit 0

CUDA_VISIBLE_DEVICES=6 python -m pytest tests/config/test_config_factory.py -vv -ra
# 468 passed; exit 0

python -m pre_commit run --files \
  vllm_omni/model_executor/models/qwen3_omni/audio_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_audio_encoder_cudagraph.py \
  recipes/Qwen/Qwen3-Omni.md
git diff --check
# All applicable hooks passed, including mypy; exit 0

CUDA checks cover FLASH_ATTN and TRITON_ATTN, contiguous/strided inputs, eager/replay equivalence, retained outputs, same lengths with new feature values, changed tails, and capture lifecycle. Five NPU tests skip because torch_npu/vllm_ascend are unavailable. There were no failures or deselections in the successful runs above.

Earlier combined attempts are not clean-suite evidence: one was interrupted after 340 passes and 5 skips; another printed 84 passes and 5 skips but exited 130. The separate runs above all exited 0. A standalone benchmark also emitted its measurements and completed assertions but exited 130; its timings are diagnostic evidence only. No full-checkpoint serving, startup-memory A/B, or NPU run was performed for this update.

Host-plan performance measurement

Compare the original 56a731a9 execute method against this commit using the same captured two-layer BF16 audio tower (d_model=128), graph buffers, and inputs on L20X. Synchronize before each measured call; time host submission without waiting for GPU completion. Alternate before/after order over 300 pairs and report median microseconds:

Backend	Feature lengths	Before	After
FLASH_ATTN	100	28.67	24.80
FLASH_ATTN	101	61.80	52.22
FLASH_ATTN	1500	38.55	27.40
TRITON_ATTN	100	32.21	26.87
TRITON_ATTN	101	63.06	53.80
TRITON_ATTN	1500	39.48	28.09

Repeated-length host submission decreased about 13–29% in these samples. Synchronized full-forward timing was noisy and does not establish a stable encoder or serving speedup. This cache does not change captured GPU work, and no global optimality claim is made.

Required before merge

Please supply complete repository unit-test evidence on a8bb9468bb8d6cb35f80a36ac01eb8d405e08147: the four CPU Simple Test shards in .buildkite/cuda/test-ready.yml (model_executor, diffusion, engine/entrypoints, and remaining tests), plus the required hardware/backend UT jobs in the supported CUDA/AMD/NPU CI matrix. Include exact commands, dependency/hardware versions, passed/failed/skipped/deselected/collection-error counts, CI links or complete logs, and explanations for skipped or unavailable jobs. Use a same-environment base/head comparison for suspected pre-existing failures. Missing jobs require completion or explicit maintainer disposition. The focused results above do not establish merge readiness.

Historical results on older revisions

Everything below refers to its stated older revision, not validation or performance of the current head. Earlier exact-tail capture/default-on descriptions have been superseded by the opt-in, full-width strategy above. The historical NPU shutdown abort remains unresolved. Review measurements on 2x A800 80GB found startup-memory limits and a workload-dependent throughput regression; the earlier large-memory Daily-Omni gain below cannot be generalized to other deployments.

Original CUDA Test Plan

Environment: Python 3.12, vLLM 0.31.0, PyTorch 2.13.0+cu130, Transformers 5.14.1. CUDA tests ran on a single NVIDIA L20X (143771 MiB reported memory).

python -m pytest \
  tests/model_executor/models/qwen3_omni/test_audio_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph.py \
  tests/worker/test_worker_base.py \
  -m 'core_model and cpu' --run-level core_model -q -ra

python -m pytest \
  tests/model_executor/models/qwen3_omni/test_audio_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph_cuda.py \
  --run-level core_model -q -ra

python -m pre_commit run --from-ref upstream/main --to-ref HEAD
git diff --check

Original CUDA Test Result

• CPU-selected tests: 43 passed, 3 deselected. The deselected audio tests were covered by the second invocation.
• Audio/vision CUDA tests: 9 passed, including FLASH_ATTN and TRITON_ATTN coverage, eager/replay numerical comparisons, and retained-output checks.
• All applicable pre-commit hooks passed, including Ruff, mypy, test marks, import checks, and CUDA API checks. git diff --check passed.
• The environment emitted an installed vLLM-Omni version-metadata mismatch warning; the tests imported this checkout. No full repository suite was run.

The small-tower CUDA tests do not by themselves establish production-model performance. A same-head Daily-Omni serving comparison on this head is recorded below. Startup-time and peak-memory A/B were not measured; the graph-enabled run reserved 6.06 GiB for encoder graphs before KV allocation.

NPU support

• Reuse Ascend's vision encoder ACL graph manager while retaining the single-replay and eager-fallback rules; NPU vision budgets are 256–8192.
• Capture the audio CNN and transformer with NPUGraph. Use ACLNN convolution during capture and restore the prior internal-format setting afterward.
• Update FIA sequence boundaries on each audio replay, with distinct registry keys for audio shapes and vision budgets. Preserve replay output ownership and support audio-only capture/cleanup.
• Profile encoder graph allocations before automatic NPU KV-cache allocation; leave explicit KV-cache budgets unchanged.

NPU validation (commit 1a409f8f6)

Environment: Python 3.12, vLLM 0.31.0, torch_npu 2.10.0.post4, CANN 9.1. Tests used small real audio and vision towers on an Ascend NPU, registering the Ascend MMEncoderAttention implementation (other layers used their portable implementations).

python -m pytest \
  tests/model_executor/models/qwen3_omni/test_encoder_graphs_npu.py \
  tests/model_executor/models/qwen3_omni/test_audio_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph.py \
  tests/worker/test_worker_base.py \
  --run-level core_model -q -ra

• Combined test run: 48 passed, 2 skipped (CUDA unavailable).
• After strengthening the boundary-sensitivity check, reran the NPU test file: 4 passed. Coverage includes audio/vision interleaved replay, same-shape audio with different clip boundaries, images/videos/DeepStack, retained outputs, audio-only recapture, runner manager selection, and mocked KV-memory reservation.
• Both pytest runs completed their assertions/fixture teardown but subsequently aborted during interpreter shutdown with corrupted size vs. prev_size (exit 134). This container teardown failure remains unresolved; these are not clean process exits.
• Ruff lint/format, SPDX, test marks, forbidden imports, CUDA API checks, and git diff --check passed. Mypy and CUDA hardware tests were not rerun for this update.
• No full-checkpoint serving, end-to-end quality, or performance benchmark was run. Shared NPU load makes latency comparisons unsuitable; no NPU performance gain is claimed.

Review follow-up (2026-10-11, 3113f419d3a56baef27a184f580961ce3732727a)

• Remove the single-chunk partial-token captures: inputs narrower than a full CNN window always fall back, so these graphs cannot be used (12 redundant captures with a 100-frame window).
• Stage audio directly into the graph input. Full-window inputs use one view/copy; clips with tails copy full-window runs together and zero only their tail padding. This removes the padded temporary allocation and per-window copies without changing CNN/GEMM dimensions.
• Extend eager/replay regression coverage to strided inputs, changed clip boundaries with the same graph shape, and reordered tails. Keep cloned outputs to protect cached embeddings from later replay.

Validation on NVIDIA L20X, Python 3.12, PyTorch 2.13.0+cu130, vLLM 0.31.0, Transformers 5.14.1:

python -m pytest \
  tests/model_executor/models/qwen3_omni/test_audio_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph.py \
  tests/model_executor/models/qwen3_omni/test_vision_encoder_cudagraph_cuda.py \
  tests/model_executor/models/qwen3_omni/test_encoder_graphs_npu.py \
  tests/worker/test_worker_base.py --run-level core_model -q -ra

54 passed, 4 skipped. NPU tests skipped because torch_npu/vllm_ascend are unavailable in this environment. CUDA tests exercised FLASH_ATTN and TRITON_ATTN; audio outputs matched eager exactly for the tested small towers. All applicable pre-commit hooks, including mypy, and git diff --check passed.

Input-staging measurements below were collected before tests/benchmarks/benchmark_qwen3_audio_staging.py was removed in 320c8cccf9. They are BF16, 128 mel bins, 100-frame windows; 20 warmups per method, five alternating rounds of 100 calls, synchronized wall time. Contiguous-input results (microseconds per call, median [min–max]):

Feature lengths	Previous staging	Direct staging
3000	169.73 [168.68–191.27]	6.91 [6.89–7.00]
6000	318.81 [317.92–319.83]	6.91 [6.77–6.97]
3001	173.66 [172.73–173.99]	26.37 [26.27–26.63]
6001	327.35 [324.94–337.30]	26.48 [25.93–29.29]
4 × 3001	635.72 [635.30–667.16]	103.21 [102.87–103.45]

Strided inputs also matched exactly and improved in every measured case. These numbers isolate input staging; they are not encoder-forward or serving speedups and do not establish globally optimal performance.

CUDA serving comparison (commit 3113f419d3a56baef27a184f580961ce3732727a)

Same commit, same deploy config, and the same 200 requests. Decode CUDA graphs stayed enabled. The only change was stage-0 cudagraph_mm_encoder (false vs the branch default true).

• Model: Qwen/Qwen3-Omni-30B-A3B-Instruct
• Hardware: two GPUs, 143771 MiB each, CUDA_VISIBLE_DEVICES=4,7
• Workload: Daily-Omni, first 200 samples, --no-oversample, seed 0, concurrency 10, temperature 0, max output 128, text-only modalities, input mode all
• Both runs: 200 succeeded, 0 failed. Input tokens 2,023,684 and output tokens 400 were identical. Accuracy was 137/200 = 68.5% on both sides.

Metric	Encoder off	Encoder on	Change
Benchmark duration (s)	169.24	117.90	-30.3%
Request throughput (req/s)	1.18	1.70	+44.1%
Mean E2EL (ms)	8277.77	5776.85	-30.2%
Median E2EL (ms)	8274.56	5780.81	-30.1%
P99 E2EL (ms)	11227.54	8279.52	-26.3%
Mean TTFT (ms)	8225.96	5676.04	-31.0%
Median TTFT (ms)	8150.37	5648.01	-30.7%
P99 TTFT (ms)	11181.33	8112.02	-27.5%
Mean TPOT (ms)	51.75	100.75	+94.7%
P99 TPOT (ms)	456.17	681.82	+49.5%

The graph-enabled server captured 134 audio encoder graphs and reported 141 encoder graphs in total, which is those audio shapes plus the configured vision budgets. It reserved 6.06 GiB for the encoder graphs. Logged replay counters, printed on a fixed cadence rather than at shutdown, reached audio hits/misses 128/7 and vision hits/misses 192/0. No vision fallback line was logged.

Mean output length is 2 tokens, so TPOT is a single interval and is not evidence of a decode regression. The serving gain is in TTFT and end-to-end latency. This is one run and does not separate a small delta from run-to-run noise; the observed TTFT/E2EL change is about 30%.

Remaining validation: an attempted broader Qwen3-Omni component run was interrupted by KeyboardInterrupt after 29 passes, so it is not a passing-suite result. Startup-time and peak-memory A/B, and clean NPU process exits, remain unverified; the previously recorded NPU shutdown abort is not resolved by this change. Before merge, complete the repository UT matrix on this head (CPU shards and required hardware/backend jobs), attach commands/environment/result counts and CI logs, and explain skipped/deselected/unavailable jobs. Existing same-head complete CI evidence is sufficient; do not infer full coverage from these focused tests.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/ar_runtime.md.

Module owners: @Gaohan123 @tzhouam @fake0fan

Routing: @Gaohan123 via module of the changed files, CODEOWNERS; @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files

@amy-why-3459, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 18fa0ea67bd0 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[amy-why-3459]

CUDA Daily-Omni serving comparison

Measured on head 3113f419d3a56baef27a184f580961ce3732727a. Same commit and deploy config; decode CUDA graphs stayed enabled. The only difference was stage-0 cudagraph_mm_encoder (false vs the default true).

• Model: Qwen/Qwen3-Omni-30B-A3B-Instruct
• Two GPUs, 143771 MiB each (CUDA_VISIBLE_DEVICES=4,7)
• Daily-Omni, 200 samples, --no-oversample, seed 0, concurrency 10, temperature 0, max output 128, text-only modalities, input mode all

Metric	Encoder off	Encoder on	Change
Benchmark duration (s)	169.24	117.90	-30.3%
Request throughput (req/s)	1.18	1.70	+44.1%
Mean E2EL (ms)	8277.77	5776.85	-30.2%
Median E2EL (ms)	8274.56	5780.81	-30.1%
P99 E2EL (ms)	11227.54	8279.52	-26.3%
Mean TTFT (ms)	8225.96	5676.04	-31.0%
Median TTFT (ms)	8150.37	5648.01	-30.7%
P99 TTFT (ms)	11181.33	8112.02	-27.5%
Mean TPOT (ms)	51.75	100.75	+94.7%
P99 TPOT (ms)	456.17	681.82	+49.5%

Both runs: 200 succeeded, 0 failed. Input tokens 2,023,684 and output tokens 400 were identical. Accuracy was 137/200 = 68.5% on both sides.

The graph-enabled server captured 134 audio encoder graphs and reported 141 encoder graphs in total (audio shapes plus the vision budgets). It reserved 6.06 GiB for those graphs. Logged replay counters, printed on a cadence rather than at shutdown, reached audio hits/misses 128/7 and vision hits/misses 192/0. No vision fallback was logged.

Mean output length is 2 tokens, so the TPOT increase is one interval and is not evidence of a decode regression. The gain is TTFT and end-to-end latency. This is a single run. Startup-time and peak-memory A/B were not measured.

These results are also in the PR description.

[MrlixiangWE]

Ran 2a69aba on 2x A800 80GB with vLLM 0.31.0 and Qwen3-Omni-30B-A3B-Instruct, from the shipped deploy YAMLs.

The shipped CUDA deploys no longer start on 80 GB cards. With cudagraph_mm_encoder on by default, stage 0's graph reserve leaves less KV than one max_model_len request needs (ValueError: To serve at least one request with the model's max seq len (65536), (6.0 GiB KV cache is needed, ...):

Deploy	Stage-0 graph reserve	Thinker KV	Starts
qwen3_omni_moe.yaml, main 5217267	0.34 GiB	7.58 GiB	yes
qwen3_omni_moe.yaml, this PR	6.71 GiB	1.20 GiB	no
qwen3_omni_moe_mrv2.yaml, main 5217267	0.32 GiB	7.64 GiB	yes
qwen3_omni_moe_mrv2.yaml, this PR	4.51 GiB	3.45 GiB	no
CI overlay qwen3_omni_moe (max_model_len 32768), main / this PR	0.10 / 6.47 GiB	7.87 / 1.50 GiB	yes / no

The 134 audio graphs take about 2.3 GiB of that: on 3113f41, stage 0 with image and video limits at 0 reserved 2.30 GiB, and budgets [256] with audio still failed at 5.57 GiB of KV. The YAML header lists 2x H100, also 80 GB per card.

Your Daily-Omni run had room for the reserve on 143 GB. On 80 GB, with stage 0 at gpu_memory_utilization: 0.95 and max_model_len: 32768 so both arms start (3113f41, off/on/on/off, concurrency 10, two runs per arm averaged, no failed requests), the reserve cuts the thinker's KV from 130,400 to 60,624 tokens, and Daily-Omni prompts average about 10,000 tokens:

Workload	Metric	Graphs off	Graphs on	Change
Daily-Omni, 1197 video+audio questions	Throughput	1.015 req/s	0.975 req/s	-3.98%
	TTFT p50	7548.73 ms	7919.09 ms	+4.91%
	Audio TTFP p50	9835.64 ms	10110.31 ms	+2.79%
	Accuracy	0.7243 / 0.7235	0.7243 / 0.7243
random-mm, one 448x448 image (196 tokens), 300 requests	TTFT p50	153.37 ms	144.00 ms	-6.11%
random-mm, one 896x896 image (784 tokens), 300 requests	TTFT p50	258.03 ms	261.66 ms	+1.41%

I'd keep cudagraph_mm_encoder off in the shipped CUDA and NPU stage-0 sections (the NPU one turns it on for its TP=2 stage 0) and document it in the recipe as opt-in with its memory cost.

With cudagraph_mode: NONE, the separate encoder reserve covers graphs that are never captured. vLLM's GPUModelRunner.capture_model returns before creating the encoder manager when cudagraph_mode is NONE. Starting stage 0 with cudagraph_mode: NONE and max_model_len: 16384, the profile captures 141 graphs and logs Reserved 6.39 GiB for encoder CUDA graphs, KV then holds 17,328 tokens, and no encoder capture follows the KV allocation. test_encoder_graph_memory_is_reserved_before_kv_allocation builds its worker with CUDAGraphMode.NONE, so it asserts this reserve.

No test goes through the thinker's _process_audio_input dispatch or the real profile_encoder_cudagraph_memory.

[MrlixiangWE]

qwen3_omni_moe_mrv2.yaml inherits this through base_config. MRv2 builds upstream's plain EncoderCudaGraphManager (v1/worker/gpu/model_states/interface.py), which never calls encoder_cudagraph_replay_supported, while the capture bakes max_seqlen = min(patches, 4096). With the tower and _tol from test_vision_encoder_cudagraph_cuda.py, budgets [2048]:

Grid through the plain manager	FLASH_ATTN	TRITON_ATTN
1x40x40 (1600 patches)	matches eager	matches eager
1x80x80 (6400 patches)	outside _tol, max abs 0.0076	576/1600 tokens under 0.99 cosine

SingleReplayEncoderCudaGraphManager rejects the 80x80 grid. With encoder_cudagraph_max_frames_per_batch: 128 as in the YAML, a 140-frame video (grid 140x2x2) through the plain manager raises RuntimeError: The size of tensor a (129) must match the size of tensor b (141) in _pad_cu_seqlens, and SingleReplayEncoderCudaGraphManager falls back. On the MRv2 deploy at 2a69aba with the real model (stage-0 max_model_len 32768 so it starts), a 1280x1280 daylight photo of a dog on a porch comes back as "taken at night, likely from a security camera or a doorbell camera" with graphs on, and correctly with them off; a 448x448 copy of the same photo is described correctly both ways.

Turning graphs off in the MRv2 profile covers the default only; opting in on MRv2 still reaches both cases, since the 4096 cap and video capture are new here (main captures images with max_seqlen = patches). The capture inputs need to hold for the plain manager too, or MRv2 should refuse cudagraph_mm_encoder. For the profile itself, the override has to sit under platforms.cuda.stages[0].compilation_config and repeat custom_ops: ["+rotary_embedding"]: an overlay replaces compilation_config whole, since it is not in _DEEP_MERGE_KEYS.

[amy-why-3459]

Addressed in dc87ddd.

MRv2 no longer inherits the enabled flag. qwen3_omni_moe_mrv2.yaml sets cudagraph_mm_encoder: false under both platforms.cuda.stages[0].compilation_config and platforms.npu.stages[0].compilation_config, and repeats custom_ops: ["+rotary_embedding"] and cudagraph_mode: PIECEWISE, since an overlay replaces compilation_config whole.

The thinker also rejects the opt-in unless it is the vLLM-Omni V1 runner. Upstream's plain EncoderCudaGraphManager (MRv2, and vllm serve without --omni) never calls encoder_cudagraph_replay_supported, so those paths no longer reach a capture whose max_seqlen is capped at 4096.

[MrlixiangWE]

Re-checked 0d7aab6 on 2x A800 80GB (vLLM 0.31.0). The shipped deploy now starts like main: stage 0 reserves 0.34 GiB and keeps 7.58 GiB of KV on both, and the 448/1280 image replies match. Four things on the opt-in path and its tests, inline.

[MrlixiangWE]

This check does not fire for a plain vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct (no --omni) with vllm-omni installed: the plugin still builds this thinker, but upstream's ModelConfig has no use_v2_model_runner, so getattr returns False. The run then uses upstream's plain EncoderCudaGraphManager, which never calls encoder_cudagraph_replay_supported, and an image over 4096 patches replays with max_seqlen capped at 4096 (vision_encoder_cudagraph.py:155). Measured with --compilation-config '{"cudagraph_mm_encoder": true}': a 1280x1280 daylight photo is described as "taken at night" on both MRv2 and VLLM_USE_V2_MODEL_RUNNER=0, while graphs off (MRv2) and a 448x448 image are correct. On main (d9110c1) the same command captures budgets up to 256 with max_seqlen = patches, so this image runs eager and the reply is correct. Could the capture inputs stay valid for any manager, or the thinker only expose the graph interface under the omni V1 runner?

[amy-why-3459]

Correct. use_v2_model_runner exists only on OmniModelConfig, so getattr(..., False) treated an upstream ModelConfig as V1 and then built the plain manager.

In dc87ddd the guard defaults that attribute to true. cudagraph_mm_encoder: true is accepted only when the field is present and false (--omni with model_runner=v1). Plain vllm serve on V1 or MRv2, and the MRv2 deploy, raise before tower construction. test_non_omni_v1_runners_reject_encoder_graph_opt_in_before_tower_construction covers both.

[MrlixiangWE]

This command does not start on A800 80GB: stage 0 captures 5 vision and 64 audio graphs (69 in total), reserves 2.51 GiB, and is left with 5.41 GiB of KV against the 6.0 GiB that max_model_len 65536 needs (ValueError: To serve at least one request ...). Could the example also lower max_model_len or raise stage 0's gpu_memory_utilization (the log suggests 0.9317), or state the memory it needs?

[amy-why-3459]

The current capture (vision budgets through 4096, one full-width audio graph per chunk count 1..64) no longer leaves stage 0 short of the KV the default context length needs, so the example keeps max_model_len. The shipped CUDA and NPU V1 profiles enable these budgets. MRv2 stays off, as noted on the deploy comment.

[MrlixiangWE]

This fails at the head on A800: all four parametrizations, 415-437 of 896 elements differ, max abs 4.88e-4. The file is collected by "Engine&Model Executor Test". With the real Qwen3-Omni audio tower and the default 64 full-width graphs (FLASH_ATTN, random bf16 features), most lengths are bit-identical to eager (100, 150, 199, 2950, 3000, 3001, 6000 frames and 1500+3000), but [101] differs by relative L2 2.2%, [205, 150] by up to 5.7% with min per-token cosine 0.974, and [801, 100] by 1.5-1.6%. A tolerance for those cases, and an accuracy check on such audio, would show whether that difference matters.

[amy-why-3459]

The exact compare is too tight for this replay. Full-width graphs pad the GEMM rows, so another GPU can pick a different kernel and round differently from eager. The 4.88e-4 max abs on A800 is that gap. In dc87ddd the check is rtol=1.6e-2, atol=2e-3. The retained-output check is still exact, so a later replay cannot overwrite a cached embedding.

The larger gap on the real tower ([101] relative L2 2.2%, [205, 150] up to 5.7% with min cosine 0.974) is the same padded-GEMM effect. This small random tower does not reproduce that magnitude, and the earlier Daily-Omni accuracy match was not rerun on the real tower for these padded lengths.

[MrlixiangWE]

No test covers this guard: with it replaced by if False, the 45 tests in test_vision_encoder_cudagraph.py and test_vision_encoder_cudagraph_cuda.py still pass, and the only failures in that run are the four test_audio_graph_matches_eager_and_preserves_cached_outputs cases. A case with an 80x80 grid that expects the eager path would protect it, and the 4096 here and at line 155 could share one constant.

[amy-why-3459]

Added test_frame_above_the_spatial_attention_bound_stays_eager in dc87ddd. With budget 2048, a 64x64 image replays, and an 80x80 image (1600 tokens, 6400 patches) stays eager. A 2-frame 80x80 video stays eager too, so the frame-chunk path cannot split it. Replacing either h * w > MAX_GRAPH_FRAME_PATCHES check with False fails this test.

The capture max_seqlen and the replay guard now share MAX_GRAPH_FRAME_PATCHES (4096). The 4096 in get_encoder_cudagraph_budget_range is a token cap, not a patch cap, so it stays separate.

[MrlixiangWE]

Re-checked 18fa0ea on CPU (vLLM 0.31.0, and merged with main aa3fd1d on vLLM 0.32.0). The non-omni guard and the spatial bound are now protected: with the use_v2_model_runner default set back to False, both upstream cases of test_non_omni_v1_runners_reject_encoder_graph_opt_in_before_tower_construction fail, and replacing either MAX_GRAPH_FRAME_PATCHES check fails test_frame_above_the_spatial_attention_bound_stays_eager. The branch conflicts with main only in vllm_omni/platforms/npu/worker/base.py, where #9066 changed the profile() signature; with both sides kept, the PR's CPU tests pass on vLLM 0.32.0. It also conflicts with #9010 in 7 of the 9 files both change; both add Qwen3-Omni audio encoder graphs and change SingleReplayEncoderCudaGraphManager.execute. Three things inline.

[MrlixiangWE]

This makes encoder graphs the default for vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni. At 0d7aab6 the same stage-0 block, passed as a --stage-overrides block on 2x A800 80GB (vLLM 0.31.0), captured 69 graphs, reserved 2.51 GiB and left 5.41 GiB of KV against the 6.0 GiB that max_model_len 65536 needs, so stage 0 stopped with ValueError: To serve at least one request with the model's max seq len (65536). This head captures the same 5 vision budgets and 64 full-width audio graphs, plus 99 single-clip short-width graphs (audio num_graphs_to_capture() is 64 + 99). The file header says this profile is verified on 2x H100, and the PR description still says the shipped profiles keep encoder graphs disabled. Could the default stay off until a startup log at this head on an 80 GB card shows stage 0 fits, or the profile set a stage-0 max_model_len or gpu_memory_utilization that the reserve fits?

[MrlixiangWE]

This also turns graphs on for the NPU stage 0 (TP=2). The latest NPU run in the description is at 1a409f8, before the full-width tail replay, the short-width graphs and the FIA rebind in this head, and it lists the corrupted size vs. prev_size shutdown abort as unresolved. test_encoder_graphs_npu.py is not in any NPU lane; the only Qwen3-Omni entry in .buildkite/npu/test-npu-ready.yml is tests/e2e/online_serving/test_qwen3_omni.py. Could you attach an NPU run at this head, or keep the NPU default off until then?

[MrlixiangWE]

offset += spec.input_size is what moves each video to its own pixels when a batch holds more than one video, but test_video_reuses_one_graph_across_frame_chunks uses a single video, so the offset is always 0. With offset += 0 both vision suites still pass (47 passed, 6 skipped), and a batch of a 6-frame and a 1-frame video (encoder_cudagraph_max_frames_per_batch 2) returns outputs that differ from encoder_eager_forward by max abs 384. Could the test add a second video, e.g. grids [[6, 2, 2], [1, 2, 2]], and compare each output with eager?