[diffusion][feature] Add offline SVDQuant W4A4 support

[lishunyang12]

Summary

Add reusable offline SVDQuant W4A4 checkpoint loading and execution for
vLLM-Omni diffusion models. MiniMax-H3 FL2VA is the first validated consumer,
and the quantization config plus linear method remain model-independent.

This PR now also covers the MiniMax-H3 encoder fallback path needed to make
text_encoder: fp8 work on top of the existing checkpoint transformer
quantization, without overwriting the checkpoint's transformer-side svdquant
settings.

The loader:

• detects quant_method: svdquant from transformer checkpoint metadata;
• loads packed NVFP4 weights, block scales, smoothing factors, and a BF16
  low-rank correction;
• supports optional per-output scales, including fused Q/K/V projections;
• preserves vLLM Column, Row, and QKV tensor-parallel sharding;
• reuses compatible vLLM NVFP4 GEMM providers.

Runtime contract

• Offline checkpoint only; no calibration or conversion at model load.
• BF16 inputs with NVFP4 W4A4 base GEMM and BF16 rank correction.
• SM103 is the currently validated and enabled target.
• FlashInfer, CUTLASS, and FBGEMM NVFP4 tensor layouts are accepted.
• A forced backend with a different tensor ABI fails during model loading with
  an actionable error instead of failing on the first inference request.
• Each tensor-parallel input partition must be divisible by the NVFP4 block
  size of 16.

The checkpoint schema and tensor shapes are documented in
docs/user_guide/quantization/svdquant.md.

Validation

• 16 passed: MiniMax-H3 quantization regression suite, including the new
  transformer checkpoint fallback for text_encoder: fp8.
• 129 passed: MiniMax-H3 contract suite.
• git diff --check: clean.
• B300 e2e measurement on MiniMax-H3 FL2VA:
  • BF16 baseline: 36.84 s, 69,838 MiB peak
  • text_encoder: fp8 on top of checkpoint transformer svdquant:
    28.28 s, 48,158 MiB peak
  • speedup: 1.30x
  • peak memory reduction: 31.0%

This leaves the loader-only compatibility path intact, while also recording the
current encoder-FP8 result that is usable on B300.

[hsliuustc0106]

This PR touches vllm_omni/quantization/, tests/diffusion/, benchmarks/diffusion/, examples/offline_inference/, vllm_omni/diffusion/ (12 files). Based on CODEOWNERS coverage of the changed files, the most-related reviewers appear to be:

@Isotr0py @david6666666 @NickCao

Could one of you take a look when you get a chance? Thanks!

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to be related to model: MinimaxH3.

Model owners: @david6666666

@lishunyang12, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[lishunyang12]

Author self-review after the H3-only rewrite:\n\n- Rebased the branch onto current main and collapsed it to one self-contained MiniMax-H3 SVDQuant commit.\n- Audited both the local patch and GitHub file list: no Z-Image model, example, benchmark, converter branch, CLI option, or documentation entry remains in the PR diff. Generic SVDQuant runtime files are retained only because MiniMax-H3 exercises them.\n- Ran the complete changed-file pre-commit suite: passed.\n- Reran converter/config/FlashInfer/H3-contract/encoder-routing tests on the final commit: 212 passed, 4 skipped.\n- DCO, pre-commit, and Python 3.11/3.12 wheel builds are green; ReadTheDocs is still pending at the time of this review.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[hsliuustc0106]

update the feature design doc as well

[hsliuustc0106]

Reviewed c06c286b. I don't see a new code-level blocker: the implementation fits the current quantization architecture by adding an Omni factory override and vLLM QuantizationConfig/LinearMethod, while reusing the existing NVFP4 kernel ABI and preserving Column/Row/QKV TP loading.

I would still hold approval until the existing frozen-head BF16-vs-SVDQuant quality/performance request is addressed. Please also update both docs/design/feature/quantization.md and docs/user_guide/quantization/overview.md so the backend list, hardware matrix, model support, and component scope include SVDQuant/MiniMax-H3.

[hsliuustc0106]

lgtm