[diffusion] feat: support FP8 Qwen3-VL text encoder for MiniMax-H3

[decajoin]

Motivation

MiniMax-H3 loads its text encoder from a stock Qwen3-VL release, so a pre-quantized FP8 drop-in such as Qwen3-VL-32B-Instruct-FP8 should be usable by simply pointing --text-encoder-path at it. Today that silently fails to quantize: the encoder is built in BF16 regardless of what the checkpoint ships.

The encoder is Qwen3-VL-32B consumed at hidden_states[50] — 50 of the 64 language layers. Under --performance-mode memory those layers are CPU-resident, so an unquantized encoder costs a large amount of host RAM that a quantized checkpoint would halve.

The underlying machinery already existed (Qwen3VLTextModel accepts quant_config, and Ideogram4 uses the same path); only the wiring into the H3 encoder was missing.

Modifications

Three links were missing between the checkpoint and the quantized linear layers:

1. configs/models/encoders/minimax_h3_qwen3vl.py — post_diffusers_config_update() now resolves the checkpoint's own quantization_config into an Fp8Config. Only quant_method: fp8 is accepted; anything else raises instead of silently loading an unquantized model. BF16 checkpoints are unaffected (quant_config stays None).

2. runtime/models/encoders/qwen3vl.py — Qwen3VLModel.__init__ accepted quant_config from callers but dropped it instead of forwarding it to Qwen3VLTextModel. The vision tower is deliberately left unquantized: FP8 releases list every visual submodule under ignored_layers and ship no scales for them.

3. runtime/models/encoders/minimax_h3_qwen3vl.py — the encoder now passes quant_config through, and runs the post-load weight repack itself. The text-encoder loader calls load_weights() directly rather than going through the FSDP path that normally invokes process_weights_after_loading, so without this the quantized linear methods would never repack their weights and scales.

That repack is staged on the accelerator rather than run wherever the parameters happen to sit. Fp8LinearMethod selects the Marlin repack (auto-enabled on SM80–86) and the DeepGEMM ue8m0 requant from get_device_capability() — i.e. from the platform, not from the parameter device — and both are CUDA kernels. Under --layerwise-offload-components text_encoder the weights are still host-resident at that point, so they abort. Modules are staged one at a time, bounding the extra device memory to a single layer.

No new CLI flags: the checkpoint's config drives everything.

sglang serve --model-path /path/to/MiniMax-H3 \
  --text-encoder-path /path/to/Qwen3-VL-32B-Instruct-FP8 \
  --num-gpus 2 --tp-size 2 --performance-mode memory \
  --layerwise-offload-components dit,text_encoder,vae \
  --model-variant ref2va

Unit tests added in test/unit/test_minimax_h3_text_encoder_quant.py (8 tests): BF16 stays unquantized, FP8 builds a block-quant config, ignored_layers cover the vision tower, non-fp8 methods are rejected, object-form configs are accepted, and
three covering the device staging above.

Accuracy Tests

Environment: 2× RTX 5090 (sm_120), TP=2, --performance-mode memory, Ref2VA.

Three Ref2VA shots were rendered twice with identical seed, steps, references and prompt — only the text encoder differs — covering a single-image reference, a two-image first/last keyframe pair, and a portrait single-image shot. The outputs were reviewed frame by frame against the BF16 renders.

No visible degradation from the quantized encoder: subject identity, clothing, action sequence, lighting and camera motion are all preserved.

Speed Tests and Profiling

Same environment. Both variants were measured back to back in a single run. Memory is sampled per-PID from server launch until the post-warmup ready banner, so the peaks include the warmup request.

	BF16	FP8	delta
GPU peak (2 cards)	62.18 GiB	61.28 GiB	−0.89 GiB (−1.4 %)
GPU steady	21.87 GiB	21.60 GiB	−0.27 GiB (−1.2 %)
Host RSS peak	246.68 GiB	190.90 GiB	−55.78 GiB (−22.6 %)
Host RSS steady	193.75 GiB	172.14 GiB	−21.61 GiB (−11.2 %)

The saving lands on host RAM, not GPU. That is expected here: with --layerwise-offload-components text_encoder the 50 language layers are CPU-resident, so halving their size shows up as RSS rather than device memory.

Throughput is unchanged. The server warmup request (1344×768×124f, 2/50 steps) takes 52.66 s on BF16 and 51.18 s on FP8 — within run-to-run noise. None is expected either: text encoding is a low single-digit percentage of total runtime here, with the DiT denoise and VAE decode dominating. The point of this change is the host-memory headroom, not speed.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #31368078666
Latest PR Test (Extra): ❌ Run #31368078458

[mickqian]

Thank you for the earlier implementation and validation. #34986 now carries the shared-loader implementation, including the bounded per-layer accelerator staging from this PR in commit 28157eb, with Yiqi Yang credited as co-author. Closing this PR as superseded by #34986 so the two post-load paths cannot diverge or run twice.