feat(vllm-omni): qualify MiniMax-H3 T2VA on B200

components/src/dynamo/vllm/tests/omni/test_omni_base_handler.py

@@ -195,14 +195,14 @@ def test_model_defined_diffusion_fields_forwarded_to_async_omni(self):
         config = _make_config()
         config.diffusion = dataclasses.replace(
             OmniDiffusionKwargs(),
-            task_type="t2va",
+            task_type="fl2va",
             lora_path=["/models/fasth3/adapter_model.safetensors"],
             diffusion_attention_backend="TRTLLM_ATTN",
         )
 
         kwargs = _build_kwargs(config)
 
-        assert kwargs["task_type"] == "t2va"
+        assert kwargs["task_type"] == "fl2va"
         assert kwargs["lora_path"] == ["/models/fasth3/adapter_model.safetensors"]
         assert kwargs["diffusion_attention_backend"] == "TRTLLM_ATTN"
 

docs/fern/pages/reference/backends/vllm-omni-configuration.mdx

@@ -31,6 +31,16 @@ These are the vLLM-Omni orchestration flags. The omni worker also accepts the sa
 <ParamField path="--default-video-fps" type="int" default="16">
   Default frames per second for generated videos.
 </ParamField>
+<ParamField path="--task-type" type="string">
+  Model-defined startup task or checkpoint partition.
+</ParamField>
+<ParamField path="--lora-path" type="path ...">
+  Diffusion checkpoint adapter path(s) fused into the model at startup. This is
+  separate from request-time LoRA loading.
+</ParamField>
+<ParamField path="--diffusion-attention-backend" type="string">
+  Default diffusion attention backend, such as `TRTLLM_ATTN` or `FLASH_ATTN`.
+</ParamField>
 
 ## Memory & Performance
 
@@ -76,6 +86,12 @@ These are the vLLM-Omni orchestration flags. The omni worker also accepts the sa
 <ParamField path="--cfg-parallel-size" type="int" default="1">
   GPUs for classifier-free guidance parallelism (1 or 2).
 </ParamField>
+<ParamField path="--vae-patch-parallel-size" type="int" default="1">
+  Ranks used for VAE patch or tile parallelism.
+</ParamField>
+<ParamField path="--text-encoder-tp-size" type="int" default="1">
+  Tensor-parallel ranks used by the diffusion text encoder.
+</ParamField>
 
 ## Disaggregated Mode
 

docs/fern/pages/use-cases/diffusion/workflows/text-to-video.md

@@ -20,6 +20,7 @@ Text-to-video generation runs a vLLM-Omni worker with `--output-modalities video
 |---|---|
 | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | Default model (1 GPU) |
 | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | |
+| `MiniMaxAI/MiniMax-H3` | Joint 24-FPS video and 32-kHz stereo audio; requires the dedicated [MiniMax-H3 workflow](#minimax-h3) and 4 B200 GPUs |
 
 To run a non-default model, pass `--model` to the launch script:
 
@@ -53,6 +54,121 @@ curl -s http://localhost:8000/v1/videos \
 
 The response returns a video URL or base64 data depending on `response_format` (e.g. `{"object": "video", "status": "completed", "data": [{"url": "file:///tmp/dynamo_media/videos/req-abc123.mp4"}]}`).
 
+## MiniMax-H3
+
+The initial MiniMax-H3 qualification serves text-to-video-and-audio (T2VA) with
+one aggregated diffusion worker. The launcher passes `--task-type fl2va`, which
+loads only H3's FL2VA checkpoint partition. Dynamo's standard image stays on
+its VP9-only media stack, so build the opt-in video-audio overlay to mux H.264
+video and AAC audio:
+
+```bash
+export DYN_H3_DYNAMO_REVISION="$(git rev-parse HEAD)"
+export DYN_H3_BASE_IMAGE="dynamo:${DYN_H3_DYNAMO_REVISION}-vllm-runtime"
+export DYN_H3_IMAGE="dynamo:minimax-h3-${DYN_H3_DYNAMO_REVISION}"
+
+python container/render.py --framework vllm --target runtime --output-short-filename
+docker build -t "$DYN_H3_BASE_IMAGE" -f container/rendered.Dockerfile .
+docker build \
+  --build-arg BASE_IMAGE="$DYN_H3_BASE_IMAGE" \
+  -f examples/backends/vllm/omni/video_audio.Dockerfile \
+  -t "$DYN_H3_IMAGE" .
+export DYN_H3_IMAGE_REF="$(docker image inspect --format '{{.Id}}' "$DYN_H3_IMAGE")"
+```
+
+Start a shell with exactly four B200 GPUs visible. The commands pass the tested
+source revision and image ID into the qualification record:
+
+```bash
+container/run.sh \
+  --image "$DYN_H3_IMAGE" \
+  --gpus device=0,1,2,3 \
+  --mount-workspace \
+  -e DYN_H3_DYNAMO_REVISION \
+  -e DYN_H3_IMAGE_REF \
+  -e DYN_H3_MODEL \
+  -e DYN_H3_MODEL_PATH \
+  -e DYN_H3_MODEL_REVISION \
+  -e DYN_H3_FASTH3_LORA_PATH \
+  -e HF_TOKEN \
+  -it
+```
+
+Launch the profile inside the container. It uses four-way Ulysses and text-encoder tensor
+parallelism while keeping VAE patch parallelism at one, so small supported
+resolutions such as 448x256 still have at least one tile per participating
+rank. Larger resolutions can opt into four-way VAE patch parallelism with
+`DYN_H3_VAE_PATCH_PARALLEL_SIZE=4`:
+
+```bash
+bash examples/backends/vllm/launch/agg_omni_minimax_h3.sh \
+  > /tmp/dynamo-minimax-h3.log 2>&1 &
+```
+
+When an offline cache contains only H3's `FL2VA/` partition, set
+`DYN_H3_MODEL_PATH` to that partition directory. The launcher keeps
+`DYN_H3_MODEL` as the public API model name and passes the local partition as
+the engine path.
+
+To use the four-step Dense FastH3 adapter, download and pin the startup adapter:
+
+```bash
+export DYN_H3_FASTH3_LORA_PATH="$(hf download \
+  FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA \
+  dense-datafree/adapter_model.safetensors \
+  --revision bcf40ca6f457ed66f8badf13514943e390205fca)"
+```
+
+The launcher fuses this adapter into the checkpoint at load time. It is not a
+request-time LoRA and cannot be combined with another request-time adapter.
+The qualification script verifies the adapter SHA-256, requests four inference
+steps, omits request scheduler-shift overrides, and records request duration and
+adapter provenance. FastH3 v1 is T2VA-only and does not support CPU or layerwise
+offload.
+
+Run the supplied end-to-end qualification against the same worker. It requests
+a 10-second clip, then verifies a non-empty H.264 stream at 24 FPS and a
+non-silent 32-kHz stereo AAC stream:
+
+```bash
+export DYN_H3_QUAL_DIR=/tmp/dynamo_minimax_h3_qualification
+bash examples/backends/vllm/launch/validate_omni_minimax_h3.sh
+```
+
+```json
+{
+  "model": "MiniMaxAI/MiniMax-H3",
+  "prompt": "A cat playing Canon in D on a grand piano.",
+  "size": "448x256",
+  "response_format": "url",
+  "output_format": "mp4",
+  "nvext": {
+    "fps": 24,
+    "num_inference_steps": 50,
+    "seed": 42
+  },
+  "task": "t2va",
+  "duration": 10.0,
+  "flow_shift": 12.0,
+  "audio_flow_shift": 3.0,
+  "aspect_ratio": "16:9"
+}
+```
+
+The launcher pins model revision `42ed227ee7df40d41602854ae760620d6eb651fe`.
+Set `DYN_H3_MODEL_REVISION` to test another revision. Dynamo preserves
+unrecognized top-level video fields and the vLLM-Omni adapter merges them into
+the pipeline's `extra_args`, where the upstream model validates them. Client
+libraries that expose an `extra_body` option can use it to add these top-level
+fields; do not send a literal nested `extra_body` object. Generated responses
+report `fps` and `audio_sample_rate` for each MP4. The qualification output
+records the exact Dynamo revision, container image ID, model revision, package
+versions, visible GPU models, request, media checksum, and stream probe data.
+
+First/last-frame FL2VA and reference-driven Ref2VA are not part of this initial
+qualification. They additionally require typed image, video, and audio input
+references.
+
 ## Request Parameters (`nvext`)
 
 The `/v1/videos` endpoint also accepts NVIDIA extensions via the `nvext` field for fine-grained control:

examples/backends/vllm/launch/agg_omni_minimax_h3.sh

@@ -0,0 +1,150 @@
+#!/bin/bash
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Launch one aggregated vLLM-Omni worker that serves MiniMax-H3 T2VA.
+set -euo pipefail
+trap 'echo Cleaning up...; kill 0' EXIT
+
+SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
+source "$SCRIPT_DIR/../../../common/launch_utils.sh"
+
+# MiniMax-H3 uses the four visible B200s for diffusion parallelism. The
+# text-generation KV-cache flags from gpu_utils.sh do not apply to this worker.
+
+MODEL="${DYN_H3_MODEL:-MiniMaxAI/MiniMax-H3}"
+MODEL_PATH="${DYN_H3_MODEL_PATH:-}"
+MODEL_REVISION="${DYN_H3_MODEL_REVISION:-42ed227ee7df40d41602854ae760620d6eb651fe}"
+ULYSSES_DEGREE="${DYN_H3_ULYSSES_DEGREE:-4}"
+TEXT_ENCODER_TP_SIZE="${DYN_H3_TEXT_ENCODER_TP_SIZE:-4}"
+# H3's native patch-parallel VAE requires at least one spatial tile per rank.
+# Keep it single-rank by default so small supported resolutions (for example,
+# 448x256) do not leave ranks with empty tile lists. Larger workloads may opt
+# into the full DiT group size explicitly.
+VAE_PATCH_PARALLEL_SIZE="${DYN_H3_VAE_PATCH_PARALLEL_SIZE:-1}"
+ATTENTION_BACKEND="${DYN_H3_ATTENTION_BACKEND:-TRTLLM_ATTN}"
+FASTH3_LORA_PATH="${DYN_H3_FASTH3_LORA_PATH:-}"
+EXTRA_ARGS=()
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --model)
+            if [[ $# -lt 2 || "$2" == --* ]]; then
+                echo "Error: --model requires a value" >&2
+                exit 1
+            fi
+            MODEL="$2"
+            shift 2
+            ;;
+        --model-path)
+            if [[ $# -lt 2 || "$2" == --* ]]; then
+                echo "Error: --model-path requires a value" >&2
+                exit 1
+            fi
+            MODEL_PATH="$2"
+            shift 2
+            ;;
+        --model-revision)
+            if [[ $# -lt 2 || "$2" == --* ]]; then
+                echo "Error: --model-revision requires a value" >&2
+                exit 1
+            fi
+            MODEL_REVISION="$2"
+            shift 2
+            ;;
+        *)
+            EXTRA_ARGS+=("$1")
+            shift
+            ;;
+    esac
+done
+
+MODEL_PATH="${MODEL_PATH:-$MODEL}"
+FASTH3_ARGS=()
+if [[ -n "$FASTH3_LORA_PATH" ]]; then
+    if [[ ! -f "$FASTH3_LORA_PATH" ]]; then
+        echo "FastH3 adapter not found: $FASTH3_LORA_PATH" >&2
+        exit 1
+    fi
+    FASTH3_ARGS=(--lora-path "$FASTH3_LORA_PATH")
+    EXAMPLE_INFERENCE_STEPS=4
+    EXAMPLE_SCHEDULER_FIELDS=""
+else
+    EXAMPLE_INFERENCE_STEPS=50
+    EXAMPLE_SCHEDULER_FIELDS=$',\n    "flow_shift": 12.0,\n    "audio_flow_shift": 3.0'
+fi
+
+SERVED_MODEL_ARGS=()
+if [[ "$MODEL_PATH" != "$MODEL" ]]; then
+    SERVED_MODEL_ARGS=(--served-model-name "$MODEL")
+fi
+
+VIDEO_AUDIO_OVERLAY_HINT="Build the opt-in video-audio overlay image from examples/backends/vllm/omni/video_audio.Dockerfile."
+if ! command -v ffmpeg >/dev/null; then
+    echo "Error: ffmpeg was not found. $VIDEO_AUDIO_OVERLAY_HINT" >&2
+    exit 1
+fi
+if ! command -v ffprobe >/dev/null; then
+    echo "Error: ffprobe was not found. $VIDEO_AUDIO_OVERLAY_HINT" >&2
+    exit 1
+fi
+if ! ffmpeg -hide_banner -encoders 2>/dev/null | grep -E '(^| )libx264( |$)' >/dev/null; then
+    echo "Error: ffmpeg cannot encode H.264 because libx264 is unavailable. $VIDEO_AUDIO_OVERLAY_HINT" >&2
+    exit 1
+fi
+if ! python -c 'import av; av.codec.Codec("h264", "w"); av.codec.Codec("aac", "w")' >/dev/null 2>&1; then
+    echo "Error: PyAV cannot encode H.264 and AAC. $VIDEO_AUDIO_OVERLAY_HINT" >&2
+    exit 1
+fi
+
+export VLLM_WORKER_MULTIPROC_METHOD="${VLLM_WORKER_MULTIPROC_METHOD:-spawn}"
+export VLLM_OMNI_VIDEO_SYNC_TIMEOUT="${VLLM_OMNI_VIDEO_SYNC_TIMEOUT:-3600}"
+
+HTTP_PORT="${DYN_HTTP_PORT:-8000}"
+print_launch_banner --no-curl "Launching MiniMax-H3 T2VA (4 GPUs)" "$MODEL_PATH" "$HTTP_PORT"
+print_curl_footer <<CURL
+curl -sS http://localhost:${HTTP_PORT}/v1/videos \\
+  -H 'Content-Type: application/json' \\
+  -d '{
+    "model": "${MODEL}",
+    "prompt": "A cat playing Canon in D on a grand piano.",
+    "size": "448x256",
+    "response_format": "url",
+    "output_format": "mp4",
+    "nvext": {
+      "fps": 24,
+      "num_inference_steps": ${EXAMPLE_INFERENCE_STEPS},
+      "seed": 42
+    },
+    "task": "t2va",
+    "duration": 10.0,
+    "aspect_ratio": "16:9"${EXAMPLE_SCHEDULER_FIELDS}
+  }' | jq
+CURL
+
+python -m dynamo.frontend &
+
+if ! wait_for_ready "http://localhost:${HTTP_PORT}/health" 60; then
+    echo "Frontend failed to become ready; stopping launch." >&2
+    exit 1
+fi
+
+echo "Starting MiniMax-H3 T2VA Omni worker..."
+DYN_SYSTEM_PORT="${DYN_SYSTEM_PORT:-8081}" \
+    python -m dynamo.vllm.omni \
+    --model "$MODEL_PATH" \
+    "${SERVED_MODEL_ARGS[@]}" \
+    --revision "$MODEL_REVISION" \
+    --output-modalities video \
+    --media-output-fs-url file:///tmp/dynamo_media \
+    --trust-remote-code \
+    --task-type fl2va \
+    --ulysses-degree "$ULYSSES_DEGREE" \
+    --text-encoder-tp-size "$TEXT_ENCODER_TP_SIZE" \
+    --vae-patch-parallel-size "$VAE_PATCH_PARALLEL_SIZE" \
+    --vae-use-tiling \
+    --diffusion-attention-backend "$ATTENTION_BACKEND" \
+    "${FASTH3_ARGS[@]}" \
+    "${EXTRA_ARGS[@]}" &
+
+wait_any_exit