Skip to content

fix(vllm-omni): harden generated video output - #14691

Closed
GuanLuo wants to merge 4 commits into
qiwa/minimax-h3-omnifrom
codex/dis-2745-universal-video-fixes
Closed

GuanLuo wants to merge 4 commits into
qiwa/minimax-h3-omnifrom
codex/dis-2745-universal-video-fixes

Conversation

@GuanLuo

@GuanLuo GuanLuo commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Harden vLLM-Omni video output handling for every video model, independent of MiniMax-H3.
  • Reject unsupported output formats instead of returning MP4 under a mismatched format contract.
  • Reject ambiguous channel-first tensors and mismatched audio batches instead of silently guessing or dropping audio.
  • Resolve audio sample rate from emitted output metadata, with a documented 24-kHz fallback.
  • Make the opt-in H.264/AAC codec overlay build against the current Dynamo vLLM runtime layout.

Validation

  • pre-commit run --files passed for all files in this PR.
  • Python compile validation passed.
  • The final stack passed a four-B200 MiniMax-H3 T2VA Kubernetes qualification using image digest sha256:5b820b31f5a4b9b14eed29952623787756a38b283e2e73e35b82f96c6af86f5a.
  • The qualification produced a 10.125-second 448x256 H.264 video with synchronized, non-silent 32-kHz stereo AAC in 24.206 seconds.
  • Local pytest was unavailable in the workspace venv; the image build and end-to-end qualification exercised the runtime path.

Stack

Review from bottom to top:

  1. feat(vllm-omni): preserve generated video audio #13707 — preserve generated video audio
  2. feat(vllm-omni): pass model-specific video parameters #13708 — pass model-specific video parameters
  3. feat(vllm-omni): qualify MiniMax-H3 T2VA on B200 #13589 — qualify MiniMax-H3 T2VA on B200
  4. fix(vllm-omni): harden generated video output #14691 — universal video-output hardening
  5. fix(vllm-omni): strengthen MiniMax-H3 qualification #14692 — H3-specific qualification corrections

furionw and others added 4 commits September 10, 2026 19:34
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: Guan Luo <gluo@nvidia.com>
)
is_cfhw = video.shape[0] in (1, 3, 4)
is_fchw = video.shape[1] in (1, 3, 4)
if is_cfhw and is_fchw:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vLLM-Omni's existing channel-first convention is CFHW, so valid CFHW outputs with 3 or 4 frames now satisfy both dimension checks and are rejected. This turns normal short generated videos into failed responses; the modified test explicitly drops those frame counts.

🤖 AI Fix

Preserve the established CFHW interpretation for channel-first tensors, or obtain an explicit layout indicator before rejecting a tensor as ambiguous.

return [None] * expected_count
if isinstance(audio, (np.ndarray, torch.Tensor)):
if audio.ndim > 1 and audio.shape[0] == expected_count:
if audio.ndim >= 3 and audio.shape[0] == expected_count:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A rank-2 (batch, samples) tensor is the standard representation for a batch of mono waveforms. For two generated videos with audio shaped (2, samples), this condition no longer splits the batch and the formatter returns a failed video response despite the audio batch matching the video batch.

🤖 AI Fix

Accept rank-2 tensors whose leading dimension equals expected_count as batched mono audio outputs and split them along axis zero.

return sample_rate

return None
return DEFAULT_AUDIO_SAMPLE_RATE

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change removes the model-config sample-rate fallback and always labels/encodes metadata-less audio as 24 kHz. Models whose vocoder configuration specifies another rate, such as the previously covered 48-kHz configuration, will have their raw waveform muxed at the wrong rate, causing incorrect duration, pitch, and video/audio synchronization.

🤖 AI Fix

Retain the model-config-derived sample-rate fallback after checking emitted metadata, using 24 kHz only when neither source provides a valid rate.

assert frames.min() == 0
assert frames.max() == 255

@pytest.mark.parametrize("shape", [(4, 3, 8, 10), (3, 4, 8, 10)])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The (3, 4, 8, 10) parameter is equivalent to the retained (4, 3, 8, 10) case: both only prove that a 4-D tensor whose first two dimensions are both channel-sized hits the same is_cfhw and is_fchw rejection path. Either case would catch the previous silent CFHW preference, so the second parameter adds no distinct contract coverage.

🤖 AI Fix

Collapse the shape parametrization to a single representative ambiguous shape, for example [(4, 3, 8, 10)].

@GuanLuo
GuanLuo force-pushed the qiwa/minimax-h3-omni branch from 033769b to 8e27b85 Compare September 11, 2026 03:03
@GuanLuo
GuanLuo requested review from a team as code owners September 11, 2026 03:03
@GuanLuo

GuanLuo commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Superseded: this universal hardening commit has been folded into #13707 so the base video/audio feature is internally complete.

@GuanLuo GuanLuo closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants