[MrlixiangWE]

Purpose

• While working on [Perf][Qwen3-Omni] Capture the thinker image encoder with CUDA graphs (part of #7452) #8215, I found that for a request with a 6600-token video and 750 audio tokens, one call of vLLM's check_interleaved_audio_video in the Qwen3-Omni thinker calls .item() 14,698 times, each a CPU sync on the CUDA masks, and takes 171.35 ms. The same result needs only two host reads.

Propose

1. Add vllm_omni/model_executor/models/common/audio_in_video.py: the same per-span result computed with nonzero, cumsum and scatter_reduce, with two host reads per call.
2. Use it from the Qwen3-Omni and Qwen2.5-Omni thinkers, and point the existing Qwen2.5-Omni interleave tests at it.

Test Plan

vLLM-Omni Commit: aadb0b4, on main 3e705fc. Environment: vLLM 0.31.0, PyTorch 2.13.0+cu130, 2x A800 80GB (PCIe).

pytest -sv tests/model_executor -m 'core_model and cpu'
pytest -sv tests/engine/ tests/model_executor/ tests/worker/test_batched_omni_output.py \
  tests/worker/test_gpu_generation_async_output.py tests/worker_v2/ -m 'core_model and cuda' --run-level 'core_model'
SKIP=actionlint,markdownlint-cli2 pre-commit run --files <changed files>
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni
vllm bench serve --omni --backend openai-chat-omni --endpoint /v1/chat/completions \
  --max-concurrency 10 --num-warmups 10 --seed 0 \
  --percentile-metrics ttft,e2el,audio_ttfp --metric-percentiles 50,95 \
  --extra-body '{"modalities":["text","audio"]}' \
  --dataset-name daily-omni --daily-omni-input-mode all <dataset paths> \
  --num-prompts 300                  # short answers
  # --num-prompts 100 --ignore-eos   # 256-token answers

Default deploy configs, fresh server per run, order main / this PR / this PR / main. The check alone: median of 20 calls after 3 warmups, synchronized. Answers: Daily-Omni videos with mm_processor_kwargs={"use_audio_in_video": true}, temperature=0, max_tokens=16, text output, concurrency 10. Probe: a wrapper that also runs vLLM's original on every call with both video and audio tokens and logs both results. Serving, answer and probe runs used earlier commits of this branch whose production code differs from this head only in a variable name, comments and formatting.

Test Result

Local run of the Buildkite lane command	main	This PR
Model Executor Test (CPU)	4003 passed, 4 skipped	4038 passed, 4 skipped
Engine & Model Executor Test (CUDA)	1388 passed, 14 skipped, 1 failed	1390 passed, 14 skipped, same 1 failed (missing offline test checkpoint)

New tests	Result
CPU: 1000 random batches and span rules vs vLLM, host reads constant in prompt length	35 passed
CUDA: 201 batches on CPU and CUDA vs vLLM; 2 device syncs per call at 3,250 and 13,000 tokens (vLLM's loop: 6,507 and 13,009)	2 passed

The check alone, one call (video, then audio as a separate item):

Device	Video + audio tokens	vLLM	This PR
CUDA (Qwen3-Omni)	1980 + 400	52.37 ms	0.37 ms (-99.29%)
CUDA (Qwen3-Omni)	6600 + 750	171.35 ms	0.41 ms (-99.76%)
CUDA (Qwen3-Omni)	13200 + 1500	342.08 ms	0.47 ms (-99.86%)
CPU (Qwen2.5-Omni)	6600 + 750	35.15 ms	0.50 ms (-98.57%)

Qwen3-Omni-30B-A3B, Daily-Omni, concurrency 10, mean of two runs each:

Metric	Short answers (300 requests), main	This PR	256-token answers (100 requests), main	This PR
Duration	358.00 s	297.03 s (-17.03%)	266.45 s	263.54 s (-1.09%)
TTFT p50 / p95	8636 / 13855 ms	7602 / 11868 ms (-11.98% / -14.34%)	4456 / 9189 ms	3975 / 8532 ms (-10.80% / -7.15%)
First audio p50 / p95	11832 / 17813 ms	9866 / 14827 ms (-16.61% / -16.76%)	6568 / 13016 ms	5279 / 10698 ms (-19.63% / -17.81%)
Thinker stage time p50	8.71 s	5.54 s (-36.45%)	12.92 s	10.91 s (-15.54%)
Thinker GPU utilization	74.3%	85.9%	64.6%	68.5%
Audio generated	202.7 s	190.9 s	4523 s	4707 s
Failed requests	0	0	0	0

Qwen2.5-Omni-7B, short answers, 300 requests, mean of two runs	main	This PR
Duration	944.41 s	934.60 s (-1.04%)
TTFT p50	23109 ms	22843 ms (-1.15%)

Greedy answers, use_audio_in_video=True, 100 Daily-Omni videos:

Qwen3-Omni	Identical answers	Accuracy
main run 1 vs main run 2	98 / 100	0.75 / 0.75
This PR run 1 vs run 2	100 / 100	0.75 / 0.75
main vs this PR, run 1	99 / 100	0.75 / 0.75
main vs this PR, run 2	99 / 100	0.75 / 0.75

Probe: every call with both video and audio tokens	Calls	Device	Interleaved / not	Mismatches
Qwen3-Omni, audio in video, 50 requests	33	cuda:0	33 / 0	0
Qwen3-Omni, Daily-Omni, 100 requests	52	cuda:0	0 / 52	0
Qwen2.5-Omni, audio in video, 30 requests	30	cpu	30 / 0	0

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md.

Module owners: @tzhouam @fake0fan @Gaohan123

Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files

@MrlixiangWE, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on cursor (cursor-grok-4.6-high) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[MrlixiangWE]

Self-review of aadb0b4:

• Same result as vLLM's per-span loop: test_matches_vllm_on_random_batches (1000 batches, both outcomes asserted), test_span_rules, test_request_sized_inputs and test_cuda_matches_cpu compare against vLLM's function on CPU and CUDA masks.
• Host syncs: test_device_syncs_do_not_grow_with_the_prompt counts 2 CUDA syncs per call at 3,250 and 13,000 placeholder tokens (vLLM's loop: 6,507 and 13,009); test_host_reads_do_not_grow_with_the_prompt covers CPU masks.
• Wiring: test_thinkers_use_this_check fails if either thinker imports vLLM's version again.
• The function holds no state between calls.

[MrlixiangWE]

@amy-why-3459 PTAL ，thanks