Repository navigation
[EPD] Support metadata-only audio-in-video requests - #62573
Merged
DarkLight1337 merged 5 commits intoOct 21, 2026
Merged
Conversation
Assisted-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Contributor
|
Documentation preview: https://vllm--62573.org.readthedocs.build/en/62573/ |
mergify
Bot
added
documentation
Assisted-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tianyu Guo <guoty@inferact.ai>
gty111
marked this pull request as ready for review
October 20, 2026 02:04
gty111
requested review from
AndreasKaratzas,
DarkLight1337,
njhill,
vadiklyutiy and
ywang96
as code owners
October 20, 2026 02:04
Member
|
@claude review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Split supplied vision embeddings per media item, preserve per-occurrence push transfer IDs, and accept video-only encoder replies when use_audio_in_video is ignored. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Member
|
@claude review |
Include modality in producer metadata expectations and pass per-item feature lists to rewrite_for_decode in the failure-isolation helper. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Member
|
@claude review |
DarkLight1337
approved these changes
Oct 21, 2026
DarkLight1337
enabled auto-merge (squash)
October 21, 2026 06:42
Member
|
/ci run |
github-actions
Bot
added
the
ready
|
✅ Triggered Buildkite CI #96347 for commit |
Collaborator
Author
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #96347. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Support metadata-only
use_audio_in_videorequests for Qwen2.5-Omni and Qwen3-Omni in the Python EPD frontend/proxy.Claims
Validation
Original validation before the review fixes, GB200, 1E + 1PD, Model Runner V2: 48/48 functional smoke requests succeeded across ECExample, Mooncake TCP, and CPU/NIXL with
Qwen/Qwen2.5-Omni-3BandQwen/Qwen3-Omni-30B-A3B-Instruct.Covered one/two videos with audio, image + video, and audio disabled, each repeated twice. Audio-enabled requests use
mm_processor_kwargs={"use_audio_in_video": true}; both instances use--trust-request-mm-kwargs.Review-fix validation:
enable_mm_embeds=True, no EC connector: 4/4 synthetic embedding requests completed (one/two images and one/two videos), followed by a successful text request on the same engine. Inputs used BF16 embeddings shaped(4 * n, 8192)withgrid_thw=[[1,4,4]] * n; this checks request completion and item handling, not semantic accuracy. Runtime: Torch 2.13.0+cu130, Transformers 5.15.0, existing precompiled extensions.VLLM_GPU_SYNC_CHECK=errorpasses the changed embedding-split path, then stops at the pre-existinginput_ids.cpu()in Qwen3-Omni'sembed_input_ids; the successful smoke above ran without that strict check. The full EPD connector matrix was not rerun for this follow-up.Details
A video with audio produces two encoder features. Carry both in one
video_embedscontent part with a companionaudio_embedspayload and shared source UUID, preserving the chat template's single video item and interleaved placeholders.Review fixes keep precomputed vision embeddings split by
grid_thwin the shared Omni mixin. For push transfers, proxy-assigned IDs remain per occurrence even when encoder responses aggregate repeated hashes. Encoder response modalities select the audiovisual path; a single video feature uses the existing matching and raw-fallback behavior. Non-push connector-provided transfer handles remain supported.Related: #52409. Duplicate checks found no equivalent PR; #57275 fixes raw-fallback identity and transfer lifecycle, not audiovisual metadata-only inputs.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)