[david6666666]

Purpose

Address the remaining MiniMax-H3 I2VA reference-accuracy failure on Hopper without relaxing the existing SSIM/PSNR thresholds or changing the reference assets.

Main build 15686, at 9f5521351e4471d6538e74ca66cb63cde23b9140, fails I2VA on H100 with SSIM 0.947851 / PSNR 27.785708 dB (required ≥0.97 / ≥34 dB). Ref2VA passes. The keyframe cuDNN context is still unchanged in the tested latest main, ae3880f0e39df4ba7376481ae78d9269fc42e09c.

The tested base's green Build 15722 did not run the H3 I2VA/Ref2VA accuracy job (nightly upload was skipped). It is not evidence that this particular test now passes. The clean main local reproduction below is on B300, not H100.

Main advanced during validation to 1b87115c9ede2d29702c847b6da98f9e854fa593 (#7449). Its only changes are ComfyUI app integration and its feature test. vllm_omni/, the accuracy/diffusion tests, test helpers, CI configuration, and dependencies are unchanged from the tested base ae3880f0e. The recorded runs remain pinned to that base and this PR head rather than claiming a separate full run on the newer commit.

Diagnosis and scope

• 5d3e6dc1 / [Bugfix][MiniMax-H3] Use Hopper-safe modulation precision #7693 changes the fused DiT modulation precision, but leaves keyframe VAE convolution TF32 enabled. It does not address this separate precision-sensitive path, and introduces additional BF16 rounding in the fused RMSNorm/AdaLN calculation on Hopper.
• encode_image() explicitly uses FP32 tensors, but cudnn.flags(deterministic=True, allow_tf32=True) still permits reduced-precision TF32 convolution. Determinism and FP32 accuracy are independent controls. The resulting image latent becomes a conditioning anchor throughout denoising.
• The strongest existing controlled evidence is natureofnature's four-run H800 experiment: changing only the keyframe TF32 setting from True to False changes I2VA from 0.947329 / 27.533157 to 0.973072 / 34.587187. Reverting and reapplying reproduces both results, with byte-identical videos within each setting. This was 4×H800 at cb439f3e8439, not this PR or H100.
• Existing CI logs do not identify the exact cuDNN kernels selected on passing versus failing H100 nodes. This PR does not claim a proved NCCL/topology cause or that all Hopper configurations fail.
• The first revision of this PR, disabling only keyframe TF32, still fails H100: 0.969770 / 33.947107 dB in Build 15724. The current revision also restores FP32 intermediates in the fused modulation kernels. A precise rounding-boundary regression demonstrates why an intermediate BF16 cast changes the result (1.234375 instead of 1.2265625). This is not a threshold adjustment.

Fix

1. Disable cuDNN TF32 only during image/keyframe encoding on CUDA capability major 9. Preserve deterministic algorithm selection, restore the caller's cuDNN settings on both success and exceptions, and retain the existing math mode on other architectures. This covers the shared encode_image() path used by FL2VA keyframes and Ref2VA still-image references; video-reference encoding and decoding are unchanged.
2. Remove the Hopper-only BF16 RMSNorm/AdaLN branch introduced by [Bugfix][MiniMax-H3] Use Hopper-safe modulation precision #7693. Retain FP32 intermediates, as the fused kernels already do on other architectures. Preserve the existing BF16 residual boundary before RMSNorm. Only H3 calls these two shared wrappers; no seed, parallel-topology, reference-asset, or test-threshold changes.

Extend the CPU contract test for SM86, SM90, SM100, SM103, unavailable capability, both incoming TF32 settings, exceptional exits, and the CPU no-op path. Replace the Hopper approximate/dispatch checks with exact rounding-boundary tests for both modulation kernels, runnable on any CUDA GPU. No CI YAML change is needed: the existing H3 accuracy dependency already includes vae.py and indexed_modulation.py.

Test Plan

vLLM Version: 0.29.0

vLLM-Omni Commit: base ae3880f0e39df4ba7376481ae78d9269fc42e09c

Local environment: 4×B300 for full accuracy; Python 3.12.3, torch 2.13.0+cu130, cuDNN 92000, Triton 3.7.1, transformers 5.14.1, diffusers 0.40.0. The local editable distribution has stale 0.27 metadata and emits a version warning; source imports were verified against the test worktree. MiniMax-H3 snapshot: 42ed227ee7df40d41602854ae760620d6eb651fe.

L1 regression and H3 CPU suite, from the repository root:

python -m pytest -q tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py \
  -k keyframe_encode -m 'core_model and cpu' --run-level core_model
python -m pytest -q tests/diffusion/models/minimax_h3/ \
  -m 'core_model and cpu' --run-level core_model
python -m pytest -q tests/diffusion/models/minimax_h3/test_minimax_h3_modulation.py \
  --run-level core_model  # CUDA GPU required for the kernel cases
pre-commit run --files vllm_omni/diffusion/models/minimax_h3/vae.py \
  tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py \
  vllm_omni/diffusion/layers/indexed_modulation.py \
  tests/diffusion/models/minimax_h3/test_minimax_h3_modulation.py

Existing L4 Accuracy job / H100 reproduction (requires four GPUs, ffmpeg/ffprobe, and cached model weights or Hugging Face access):

export VLLM_TEST_MINIMAX_H3_FL2VA_MODEL=/path/to/42ed227ee7df40d41602854ae760620d6eb651fe/FL2VA
export VLLM_TEST_MINIMAX_H3_REF2VA_MODEL=/path/to/42ed227ee7df40d41602854ae760620d6eb651fe/Ref2VA
python -m pytest -s -v tests/e2e/accuracy/minimax_h3/test_minimax_h3_i2va_ref2va_similarity.py \
  -m 'full_model and H100 and B200 and cards_4' --run-level full_model

This retains the original seed 0, 50 inference steps, 4-way Ulysses/HSDP/text-encoder TP/VAE tiling, static compilation, fixture asset SHA256 checks, and accuracy thresholds. The B300 main baseline calls the two original test functions directly with an external artifact directory; the PR run uses the pytest command above with artifacts redirected outside the checkout. Neither is an H100 run. These are single full-fixture acceptance runs, not performance measurements.

Test Result

• Regression before the production fix: 4 failed, 19 passed; all four failures are the SM90 TF32 assertion.
• Regression after the fix: 23 passed.
• H3 CPU suite: 682 passed, 2 skipped, 46 deselected.
• Modulation suite at the current revision: 9 passed, 1 skipped on B300. Before the second fix, exercising the old Hopper BF16 math branch on B300 makes both exact rounding tests fail (2 failed); this is execution of that math branch, not an H100 hardware reproduction.
• Changed-file pre-commit: passed, including mypy, marks, forbidden imports, and the torch.cuda gate. git diff --check: passed.
• Latest-main B300: I2VA 0.974952 / 35.883204 dB; Ref2VA 0.980576 / 42.644924 dB; both pass.
• Current revision (66c500a71) B300, original pytest command: 2 passed, 15 warnings in 717.89s. I2VA 0.974952 / 35.883204 dB; Ref2VA 0.980576 / 42.644924 dB. Both thresholds pass; metrics match the main acceptance run and the first revision (a413d0ceb, 2 passed in 713.82s). Each generated MP4 is byte-identical to its main-baseline counterpart (SHA256 I2VA ffe44e473b6f3bccd78311b0a1df9042167c314de25251120cbd651198986274, Ref2VA 74debfc131da7a190766a04579393fdae59ea3c55f65a87129c569959d798de3). No speedup or repeated full-video A/B claim is made.
• Real B300 image-VAE smoke: loaded the FL2VA and Ref2VA encoder weights, used the fixture image with each task's production preprocessing, and alternated TF32 True/False three times per setting. All encoded rows were finite and bitwise-repeatable within each setting; the PR default matched TF32 True on B300. Normalized latent mean/max absolute differences between settings were 0.001208 / 0.441732 (FL2VA) and 0.001780 / 0.910092 (Ref2VA). This is an encoder smoke/precision check, not Ref2VA still-image end-to-end accuracy evidence.
• H100 I2VA fails at a413d0ceb in Build 15724, on inf-4x8h100-3: SSIM 0.969770 / PSNR 33.947107 dB, just below both unchanged thresholds. Ref2VA passes at 0.980572 / 42.650962 dB. Disabling keyframe TF32 alone is not a complete fix.
• Combined fix at 66c500a71, H100 Build 15725, again on inf-4x8h100-3: 2 passed, 14 warnings in 1293.17s; job exit 0. I2VA SSIM 0.976254 / PSNR 36.295646 dB; Ref2VA SSIM 0.980588 / PSNR 42.655355 dB. Both original accuracy thresholds pass. This is one full-fixture H100 acceptance run, not a repeated multi-node determinism claim. No local Hopper device is available, and the existing video/audio Ref2VA fixture does not establish still-image Ref2VA output quality.
• Additional H100 H3 FP8 Quality job at the same head: 1 passed, 2 deselected, exit 0.

Pre-submit full self-review: four directly related files; no threshold/reference changes, new public API, dependencies, broad catches, hot-path copies, or new examples. The obsolete precision branch and its selector are removed rather than left unreachable. The target H100 accuracy and FP8 quality jobs now pass. Both commits have DCO sign-off; GitHub DCO, pre-commit, Python 3.11/3.12 build, NPU CI, and docs checks have passed. The broader CUDA/AMD pipelines still have pending work as of this update; no performance claim is made.

AI assistance: Codex investigated CI/code history, implemented the scoped fix and tests, ran local validation, and drafted this description.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/offloader.md, docs/design/module/diffusion/diffusion_model_integration.md, docs/design/module/diffusion/index.md.

Module owners: @Bounty-hunter @fhfuih @wtomin

Routing: @Bounty-hunter via module of the changed files, CODEOWNERS; @fhfuih via module of the changed files, CODEOWNERS; @wtomin via module of the changed files, CODEOWNERS

@david6666666, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 66c500a71c2e produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[david6666666]

Author self-review for 66c500a71c2ecf1db53eec58f558353a18110fdc:

• Checked all four changed files and the H3 call sites. TF32 is disabled only for Hopper image/keyframe encoding; cuDNN settings are restored on success and exceptions. The fused modulation kernels keep FP32 norm/affine intermediates while preserving their existing BF16 residual boundary. Other architectures retain their existing math paths. No accuracy thresholds, reference assets, seeds, topology, dependencies, or public APIs changed.
• Regression evidence: the new context checks fail on the old SM90 policy; both exact affine-rounding tests fail when executing the old BF16 math branch. After the fix, the H3 CPU suite has 682 passed / 2 skipped, and the CUDA modulation suite has 9 passed / 1 skipped. Changed-file pre-commit and diff checks pass. Both commits are DCO signed.
• B300 original full accuracy fixture: 2 passed; both generated videos are byte-identical to the clean main baseline. This is not a local H100 reproduction.
• Real H100 validation: the TF32-only revision still failed I2VA in Build 15724. The combined fix in Build 15725, on the same inf-4x8h100-3 node, has 2 passed / exit 0: I2VA 0.976254 SSIM / 36.295646 dB, Ref2VA 0.980588 / 42.655355 dB. The separate H100 FP8 quality job also passes.
• Limits: no exact cuDNN kernel trace explaining the older pass/fail node difference, no multi-node repeatability or speedup claim, and no still-image Ref2VA end-to-end claim from the video/audio fixture. Main advanced from the tested base ae3880f0e to 1b87115c9 only in ComfyUI app integration and its feature test; the target inference/test paths are unchanged.

AI assistance is disclosed in the PR description. Other pending pipeline jobs are not being represented as passed.

[princepride]

LGTM