[Dong1017]

Purpose

MiniMax-H3 Ref2VA extracts reference audio after decoding and preparing its video frames. Start audio extraction in one background CPU worker after transcoding, then collect results in reference order while assembling the video conditions. Isolated encoder-input preparation improves from 11.423 s to 10.374 s, saving 1.049 s (9.2%), with exactly equal prepared inputs.

This PR changes reference-audio scheduling and adds a 120-second extraction timeout in two production files. Each reference's audio is extracted once; audio failures propagate through future.result(). Queued jobs are cancelled when preparation fails, and running work finishes before its enclosing temporary directory is removed. Reference order and videos without audio retain their existing behavior. Frame conversion is separate in #8129. Related performance tracker: #8228.

Test plan and correctness

vLLM: 0.30.0. vLLM-Omni: baseline c8a1419e0 plus this PR's 6cc83ecef.

• Run real CPU-only prepare_encoder_inputs on an H200 host with a 14.79-second video containing audio, a 15-second output request, 512×512, 24 fps, one warmup per arm, and three alternating baseline/candidate pairs. Both arms use the same model files and /usr/bin/ffmpeg; no GPU is allocated for this measurement.
• Compare media tensors and waveforms, metadata, Qwen vision tensors, and presentation inputs in every pair. All three pairs match exactly.
• Applicable pre-commit hooks passed, including Ruff, mypy, SPDX, and CI marks. 40 targeted CPU tests passed, including real ffmpeg waveform equivalence, video-failure cancellation, subprocess timeout propagation, and temporary-directory cleanup.
• Audio extraction uses per-call temporary files and performs no random sampling or global seed changes. Text-only and image-only preparation continue through their existing paths.

Performance

CPU encoder preparation, hot runs	Baseline	This PR
Pair 1	11.396 s	10.374 s
Pair 2	11.538 s	10.313 s
Pair 3	11.423 s	10.419 s
Median	11.423 s	10.374 s

This is an isolated CPU-preparation comparison. Audio extraction overlaps video decode/preparation, accounting for the approximately one-second improvement on this host.

Eight-device scheduling evidence

A separate matched capture uses 8×Ascend 910B1 (64 GiB/device), source 9678c60a, a fixed 15-second reference video, seed 2101, 1344×768 output, 362 frames at 24 fps with audio, 50 executed DiT steps, DiT USP8, text TP8, and VAE tile/patch8. Both arms enable synchronized component timing and detailed CPU/NPU collection. The candidate includes this PR, #8129, VAE residency, and explicit rank-local text weight transfer.

Host start/end records confirm background audio extraction overlaps frame loading/preparation. Audio extraction itself takes 0.196 s in the baseline and 0.193 s in the candidate on this workload, so the scheduling change overlaps approximately 0.2 s of audio work here. These records establish the background overlap; the native trace does not expose that worker's named function scope.

The combination reduces pre-DiT preparation from 50.181 s to 42.226 s, common eight-rank time without any device task from 5.723 s to 3.448 s, and instrumented E2E from 2254.917 s to 2247.298 s. These are combination results. The isolated performance result for this PR is the CPU-preparation table above.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md.

Module owners: @gcanlin @Sy0307 @tzhouam

Routing: @gcanlin via module of the changed files, CODEOWNERS; @Sy0307 via module of the changed files, CODEOWNERS; @tzhouam via module of the changed files, CODEOWNERS

@Dong1017, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[Dong1017]

Author self-review: checked reference ordering, one audio extraction per reference, exception propagation, and executor completion before temporary-directory cleanup. The background task does not change random state. Three real-input CPU preparation pairs matched exactly for media, metadata, Qwen vision, and presentation inputs; the median improves from 11.423 s to 10.374 s. Frame conversion remains separate in #8129, and the description distinguishes isolated CPU preparation from the combined NPU diagnosis. All current CI checks pass.

@vllm-omni-review-bot Please review this PR.

[Dong1017]

@vllm-omni-review-bot Please review this PR.

@david6666666 Could you review this change as the MiniMax-H3 model owner, especially its model semantics, correctness evidence and performance attribution?

Scope: Reference-audio extraction overlapped with video preparation, preserving the existing media output.

The five changes have separate scopes: #8129, #8229, #8577, #8578 and #8579. The shared eight-rank Ref2VA evidence and per-change attribution are summarized in the Issue #8228 update. The existing Draft/Ready status is retained.

[david6666666]

If frame decoding fails, this context still waits for all submitted audio jobs, including queued ones. Since ffmpeg has no timeout, a failed request can remain stuck here. Could we cancel pending jobs on failure and bound the subprocess runtime, with a regression test?

[Dong1017]

Fixed in a3a33b6.

The request now shuts down its executor with wait=True, cancel_futures=True: queued extraction jobs are cancelled after a frame failure, and an already-running job is joined before the enclosing temporary directory is removed. Reference-audio ffmpeg extraction has a 120-second timeout, with the existing subprocess exception propagated to the caller.

Regression tests use a real executor and ffmpeg. They verify queued-job cancellation and worker completion on a frame failure, a real blocked ffmpeg process timing out and cleaning its temporary output, and exactly equal extracted waveforms with reference ordering and a no-audio reference preserved. 40 targeted CPU tests passed, and all applicable pre-commit checks passed.

The checks reported on the previous head were successful; the follow-up needs its own CI results. This change does not include unrelated VAE fixture or dependency-API repairs.

[david6666666]

LGTM now