[Quant] Phase 1 (video-gen): ModelOpt FP8 for HunyuanVideo-1.5 and Wan2.2

[lishunyang12]

Purpose

Phase 1 of #2709 — extends ModelOpt FP8 support to video-gen models. #2913 covers Phase 1 for image-gen (Flux, Flux2-Klein, Qwen-Image, HunyuanImage-3); this PR adds the video-gen counterpart for both HunyuanVideo-1.5 and Wan2.2 TI2V-5B using the same loader infrastructure.

Builds on:

• （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913 (Phase 1, image-gen) — ModelOpt FP8 checkpoint auto-detect + diffusers loader adapter (cherry-picked into this branch; rebases away when （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913 merges)
• [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 (referenced, will not merge) — DiT quant_config wiring for HV-1.5 + Wan2.2 (extracted into this PR; [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 stays as online-FP8 ablation reference)

Changes

DiT wiring (extracted from #2920)

• hunyuan_video_15_transformer.py + pipelines — HunyuanVideo15Attention, HunyuanVideo15TransformerBlock, HunyuanVideo15Transformer3DModel accept quant_config / prefix; threaded to to_qkv, to_out[0], add_kv_proj, to_add_out, ff, ff_context.
• wan2_2_transformer.py + wan2_2_vace_transformer.py + 4 pipelines — WanSelfAttention, WanCrossAttention, WanFeedForward (+ ColumnParallelGELU), WanTransformerBlock, WanTransformer3DModel, VACE variant. Factories (create_transformer_from_config, create_vace_transformer_from_config) accept optional quant_config.
• Modulation (raw nn.Linear / scale_shift_table), patch embedders (Conv3d), time/text/image embedders, proj_out, and the HV-1.5 token refiner stay full precision.
• The aggressive skip patterns from [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 (attn1/attn2 quant_config=None on Wan2.2) are not applied here — that was an online-FP8 workaround; static calibration handles it.

ModelOpt FP8 helpers

• examples/quantization/quantize_hunyuanvideo_15_modelopt_fp8.py — HV-1.5 calibrator. Force-exports FP8 weights, patches quant_algo: FP8, hides quantizers during save. MHA quantizers off by default.
• examples/quantization/quantize_wan2_2_modelopt_fp8.py — Wan2.2 TI2V-5B calibrator. Same design.
• examples/quantization/check_modelopt_fp8_export.py — verifier. Reads safetensors header dtypes, checks quant_algo: FP8, classifies scale granularity (per-tensor / per-channel / per-block).
• vllm_omni/model_executor/stage_configs/hunyuan_video_15_dit_fp8.yaml + wan2_2_ti2v_dit_fp8.yaml — serving stage configs with auto-detect.

Adapter (this PR also fixes a general-purpose bug in #2913's adapter):

• modelopt_fp8.py:_get_weights_mapper now walks submodules to aggregate hf_to_vllm_mapper from whichever sub-module defines it. The adapter is instantiated with the whole Pipeline, so model-specific remaps (like Wan2.2's ffn.net.0. → ffn.net_0.) must be discovered on the transformer submodule, not the top-level Pipeline. Fixes silent-noise output that occurred on Wan2.2 ModelOpt FP8 before this change.
• WanTransformer3DModel.hf_to_vllm_mapper added with that remap.

Both calibrators share --weight-block-size 'M,N' for block-wise FP8, and the same fallback pattern: _force_export_quantized_weights + _patch_quant_config + hide_quantizers_from_state_dict — because ModelOpt's export_hf_checkpoint doesn't handle diffusers-video checkpoints natively.

Validation — HunyuanVideo-1.5 (1×H100 80GB, T2V 480×832, 33 frames, 30 steps, seed=42)

torch.compile enabled (default).

Metric	BF16 baseline	ModelOpt FP8 (this PR)	Delta
Model load	33.81 GiB	28.74 GiB	−15%
Peak GPU memory (allocated)	72.42 GiB	67.36 GiB	−7%
Total wall time	24.05 s	20.79 s	−14%
Throughput	1.44 it/s	1.67 it/s	+16%
On-disk transformer weights	31.02 GiB	10.45 GiB	−66%

Engine signals confirming the path is wired correctly:

• factory.py: Building quantization config: fp8 → Building quantization config: modelopt — auto-detect upgraded the user's --quantization fp8 flag to ModelOpt based on quant_algo: FP8 in transformer/config.json
• data.py: Auto-detected quantization 'modelopt' from model config
• __init__.py: Selected CutlassFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod — the ModelOpt FP8 kernel selected

Visual comparison — HunyuanVideo-1.5

BF16 baseline:

hv15_bf16_compiled.mp4

ModelOpt FP8 (this PR):

hv15_modelopt_fp8_compiled.mp4

Same prompt ("A dog running across a field of golden wheat."), same seed, same sampling params. Output is BF16-equivalent — no detail collapse or composition drift like the online FP8 path showed in #2920.

Validation — Wan2.2 TI2V-5B (1×H100 80GB, T2V 704×1280, 49 frames, 30 steps, seed=42)

torch.compile enabled (default).

Metric	BF16 baseline	ModelOpt FP8 (this PR)	Delta
Model load	21.22 GiB	16.68 GiB	−21%
Peak GPU memory (allocated)	38.28 GiB	33.74 GiB	−12%
Total wall time	19.75 s	16.63 s	−16%
Throughput	1.96 it/s	2.45 it/s	+25%
On-disk transformer weights	21.22 GiB	4.76 GiB	−77%

Engine signals:

• factory.py: Building quantization config: fp8 → modelopt (auto-detect fired)
• data.py: Auto-detected quantization 'modelopt' from model config
• __init__.py: Selected CutlassFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
• Zero unloaded weight_scale warnings after the hf_to_vllm_mapper fix for Wan2.2's ffn.net.0. → ffn.net_0. diffusers↔vllm-omni name remap.

Visual comparison — Wan2.2 TI2V-5B

BF16 baseline:

wan22_bf16_v4.mp4

ModelOpt FP8 (this PR):

wan22_modelopt_fp8_v4.mp4

Same prompt ("A dog running across a field of golden wheat."), same seed, same sampling params. Output is BF16-equivalent.

How to use

Pre-calibrated checkpoints are published on HF Hub so reviewers can test without recalibrating:

• shunyang90/HunyuanVideo-1.5-480p-ModelOpt-FP8
• shunyang90/Wan2.2-TI2V-5B-ModelOpt-FP8

Option A: use the published checkpoints (no calibration needed)

# HunyuanVideo-1.5
python examples/offline_inference/text_to_video/text_to_video.py \
    --model shunyang90/HunyuanVideo-1.5-480p-ModelOpt-FP8 \
    --quantization fp8 \
    --prompt "A dog running across a field of golden wheat." \
    --height 480 --width 832 --num-frames 33 \
    --num-inference-steps 30 --seed 42 --guidance-scale 6.0 \
    --output outputs/hv15_modelopt_fp8.mp4

# Wan2.2 TI2V-5B
python examples/offline_inference/text_to_video/text_to_video.py \
    --model shunyang90/Wan2.2-TI2V-5B-ModelOpt-FP8 \
    --quantization fp8 \
    --prompt "A dog running across a field of golden wheat." \
    --height 704 --width 1280 --num-frames 49 \
    --num-inference-steps 30 --seed 42 --guidance-scale 5.0 \
    --output outputs/wan22_modelopt_fp8.mp4

Option B: calibrate from BF16 yourself (reproducibility / custom prompts)

# 1. Install
pip install 'nvidia-modelopt[all]'

# 2a. Calibrate HV-1.5 (~10–15 min on 1×H100)
python examples/quantization/quantize_hunyuanvideo_15_modelopt_fp8.py \
    --model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v \
    --output ./hv15-480p-modelopt-fp8 --overwrite

# 2b. Calibrate Wan2.2 TI2V-5B (~10 min on 1×H100)
python examples/quantization/quantize_wan2_2_modelopt_fp8.py \
    --model Wan-AI/Wan2.2-TI2V-5B-Diffusers \
    --output ./wan22-ti2v-modelopt-fp8 --overwrite

# 3. (optional) Verify
python examples/quantization/check_modelopt_fp8_export.py --output ./hv15-480p-modelopt-fp8
python examples/quantization/check_modelopt_fp8_export.py --output ./wan22-ti2v-modelopt-fp8

# 4. Serve — auto-detect upgrades --quantization fp8 to ModelOpt FP8
# (same invocation as Option A, just pass the local output path as --model)

Test Plan

HunyuanVideo-1.5

• Calibration script completes on 1×H100 — 648 weights converted to FP8
• Checker reports quant_algo: FP8, 648 F8_E4M3 tensors, per-tensor scale granularity
• On-disk transformer 10.45 GiB (−66% vs 31.02 GiB BF16)
• Loads via （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913's adapter (Auto-detected quantization 'modelopt')
• End-to-end inference produces valid video; visual parity with BF16
• Memory −15%, wall-clock −14% vs BF16

Wan2.2 TI2V-5B

• Calibration script completes on 1×H100 — 300 weights converted to FP8
• Checker reports quant_algo: FP8, 300 F8_E4M3 tensors, per-tensor scale granularity
• Loads cleanly via （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913's adapter (after the hf_to_vllm_mapper fix — see adapter change below)
• End-to-end inference produces valid video; visual parity with BF16
• Memory −21%, wall-clock −16% vs BF16
• On-disk transformer 4.76 GiB (−77% vs 21.22 GiB BF16)

Both

• Pre-commit (ruff, format, typos) — passing
• torch.compile enabled (default) on both BF16 and FP8 for fair comparison
• HV-1.5 I2V variant + Wan2.2 I2V / T2V-A14B / VACE — wiring threaded, calibration untested

Known limitations

• Per-block static FP8 is calibrated-correct but not serveable yet. Upstream vLLM's ModelOptFp8Config / ModelOptFp8LinearMethod only dispatches per-tensor scales — a block-wise checkpoint crashes at load with a shape-mismatch assertion in parameter.py:_assert_and_load. Per-tensor serving is the shippable path; --weight-block-size is kept in the calibrator for when upstream gains block-wise dispatch.
• HV-1.5 and Wan2.2 aren't in ModelOpt's recognized-model registry — QKV fusion is skipped and we hand-roll the weight-export path. Works, but means fewer of ModelOpt's standard diffusion optimizations.
• MHA quantizers (K/V/softmax) off by default — attention numerics on long video sequences were sensitive even with static scales (empirically in [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 ablation).

Follow-ups (still Phase 1, other video/variant coverage)

• Wan2.2 T2V-A14B / I2V-A14B MoE variants (need 2×H100)
• Wan2.2 VACE variant (wiring threaded; calibration helper needs VACE-specific prompts)
• HunyuanVideo-1.5 720p + I2V variants
• Block-wise static FP8 serving once upstream vLLM dispatches on strategy: block
• Publish calibrated checkpoints to HF Hub under vllm-project-org/

Depends on #2913. References #2920 (online-FP8 ablation reference, will not merge).

cc @baonudesifeizhai @hsliuustc0106 @ArtificialRay

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[hsliuustc0106]

This PR is substantial (>1000 LOC / >10 files). Could you please run the L3 tests locally and paste the results here?

Once L3 test results are available, I will proceed with a full review of the ModelOpt FP8 video-gen implementation.