[Multimodal][Qwen3 Omni] Make Qwen3 Omni work with audio-in-video inputs in V1 engine.

[huachenheli]

Purpose

(EngineCore_DP0 pid=2834739) INFO 10-29 13:21:24 [core.py:237] init engine (profile, create kv cache, warmup model) took 18.56 seconds
INFO 10-29 13:21:25 [llm.py:345] Supported tasks: ['generate']
Adding requests: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:10<00:00, 10.90s/it]
Processed prompts: 100%|███████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.31s/it, est. speed input: 1492.83 toks/s, output: 102.27 toks/s]
The
The video shows a baby sitting on a bed, wearing glasses and intently focused on a book they are holding. The baby is dressed in a light blue shirt and appears to be deeply engaged in reading, turning the pages of the book. The room around them is somewhat cluttered, with clothes and other items scattered on the bed. In the background, there is a crib and a wall with a picture frame. The baby occasionally looks up from the book, seemingly curious about something off-camera, but quickly returns their attention to the book. The baby's glasses are large in proportion to their face, adding a humorous and endearing touch to the scene.

FIX #22268
FIX #22364
CLOSE #23888
CLOSE #25473
CLOSE #28046

Test Plan

HF_HUB_DISABLE_XET=1 VLLM_ATTENTION_BACKEND=TORCH_SDPA python examples/offline_inference/qwen3_omni/only_thinker.py -q use_audio_in_video

pytest tests/model_executor/test_qwen3_omni.py

Sanity checked with https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/draw.mp4

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--27721.org.readthedocs.build/en/27721/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @huachenheli.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jeremyteboul]

2 questions:

1. What is the reason of using Torch_SDPA ?
2. if we return immediately should the if section be removed ?

[huachenheli]

This will be removed when I clean up this PR. I was having issue with flash-attn on my devgpu so this is a local hack to get qwen3 running.

[jeremyteboul]

it feels the intent was good to have the validation of the placeholders , should we just have one doing the verification we need ?

[huachenheli]

This is being removed already in #26334

[jeremyteboul]

nice, that can be very useful

[jzou-dev]

Hope you folks can see this: when I try https://github.com/QwenLM/Qwen3-Omni/blob/main/web_demo.py with this fix, and use vllm as backend, and put system prompt as "what does this man say?"
python web_demo.py -c ../Qwen3-Omni-30B-A3B-Instruct --server-name localhost
I still cannot get correct result.

However, if I switch to transformers as backend:
python web_demo.py -c ../Qwen3-Omni-30B-A3B-Instruct --server-name localhost --use-transformers --flash-attn2
I get reasonable output.

So it seems there is still more work to do.

[chatgpt-codex-connector]

💡 Codex Review

vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py

Lines 395 to 413 in 9e6f9f0

	if is_update_applied:
	mm_placeholders = self._find_mm_placeholders(
	prompt_ids,
	mm_prompt_updates,
	)
	self._validate_mm_placeholders(
	mm_placeholders,
	mm_item_counts,
	use_audio_in_video=use_audio_in_video,
	)
	else:
	prompt_ids, mm_placeholders = self._apply_prompt_updates(
	prompt_ids,
	mm_prompt_updates,
	)
	self._validate_mm_placeholders(
	mm_placeholders,
	mm_item_counts,
	use_audio_in_video=use_audio_in_video,

Remove unsupported keyword from placeholder validation call

Qwen2_5OmniThinkerMultiModalProcessor._maybe_apply_prompt_updates still invokes _validate_mm_placeholders(..., use_audio_in_video=use_audio_in_video) even though this class’s override of _validate_mm_placeholders was deleted in this change. The inherited implementation only accepts two positional parameters, so this call will now raise TypeError: _validate_mm_placeholders() got an unexpected keyword argument 'use_audio_in_video' whenever multimodal inputs are processed, crashing Qwen2.5 Omni inference before any work is done.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[jeremyteboul]

thanks !

[wangruohui]

will this PR automatically support online inference?

[ywang96]

@huachenheli Thanks for working on this - I'll review this PR and push it to the finishing line.

[jeremyteboul]

Curious if we could land a version for V1 @ywang96 ; thanks in advance !

[wwxs123W]

thank u for doing it! But can it support qwen2.5 omni?