[yiliu30]

Contributor intent (required human answers)

Public-safe summary based on the contributor's explicit request to upstream the
serialized SVDQuant MXFP4 support.

1. What outcome do you want to achieve?

Enable vLLM-Omni users to load and run serialized SVDQuant MXFP4 diffusion
checkpoints directly, including the public MiniMax-H3 FL2VA checkpoint, without
fork-local runtime changes.

2. What is the highest-leverage variable for that outcome?

Correct end-to-end handling of the serialized MXFP4 checkpoint contract: kernel
selection, block-32 E8M0 scales, tensor-parallel loading, and the H3 grouped-QKV
row order.

3. How will this work move that variable?

This PR adds MXFP4 to the SVDQuant config and linear path and applies the same
grouped-QKV to Q/K/V permutation used by BF16 H3 weights to every serialized
quantized tensor before vLLM's fused QKV loader runs. It then validates both
FlashInfer W4A4 and Marlin W4A16 execution with coherent H3 video.

4. Whose customer or user needs does this address, and what are those needs?

Users with offline MXFP4 SVDQuant diffusion checkpoints on NVIDIA Blackwell need
those checkpoints to load without silent QKV corruption, preserve the low-rank
correction, and run with the best available MXFP4 kernel on one GPU.

Purpose

Before this PR, the serialized SVDQuant path accepted only NVFP4, and
MiniMax-H3 did not reorder serialized QKV tensors. The latter caused the fused
QKV loader to consume grouped rows as Q/K/V, producing noise.

After this PR:

• DiffusionSVDQuantConfig accepts precision="mxfp4"; NVFP4 behavior is
  unchanged.
• MXFP4 weights use qweight uint8 [N, K/2] and E8M0 wscales
uint8 [K/32, N]; wtscale and wcscales are not registered.
• process_weights_after_loading() delegates to
  init_mxfp4_linear_kernel(activation_quant_key=kMxfp4Dynamic) and retains the
  rank correction in BF16.
• MXFP4 is accepted on SM100/SM103; NVFP4 remains enabled only on SM103.
• MiniMax-H3 reorders serialized qkv_proj tensors before weight loading.
• The SVDQuant user guide documents both checkpoint layouts and runtime paths.

Affected documentation:
docs/user_guide/quantization/svdquant.md.

Test Plan

vLLM Version: 0.32.0

vLLM-Omni Commit: 2dc0772f552f080f84aee5710b007379e5f0dbd8

Hardware: NVIDIA B200, SM100, 183 GiB, one GPU

Software: PyTorch 2.13.0, FlashInfer 0.7.0.post1

pytest tests/diffusion/quantization/test_svdquant_config.py \
  tests/diffusion/quantization/test_svdquant_linear.py \
  tests/diffusion/quantization/test_svdquant_tp_loading.py -q

pytest tests/diffusion/models/minimax_h3/test_minimax_h3_quantization.py -q

# Bias-ownership follow-up
pytest tests/diffusion/quantization/test_svdquant_linear.py -q

CUDA_VISIBLE_DEVICES=4 pytest tests/diffusion/quantization/test_svdquant_marlin_gpu.py -q

uvx pre-commit run --files \
  docs/user_guide/quantization/svdquant.md \
  tests/diffusion/models/minimax_h3/test_minimax_h3_quantization.py \
  tests/diffusion/quantization/test_svdquant_config.py \
  tests/diffusion/quantization/test_svdquant_linear.py \
  tests/diffusion/quantization/test_svdquant_tp_loading.py \
  vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py \
  vllm_omni/quantization/svdquant_config.py

Test Result

• SVDQuant config, linear, and TP-loading tests: 25 passed.
• MiniMax-H3 quantization tests: 6 passed, including the serialized-QKV
  reorder regression.
• Pre-commit: passed for all changed files, including SPDX, test marks,
  mypy-3.10, ruff, and markdownlint.
• git diff --check: clean.
• Bias-ownership follow-up at 2dc0772: SVDQuant linear tests 14 passed,
  including TP=2 RowParallelLinear bias-once and skip_bias_add=True
  regressions.
• Combined SVDQuant config/linear/TP-loading and MiniMax-H3 quantization
  tests: 36 passed.
• Real Marlin replicated-linear regression on B200: 1 passed, verifying
  the restored logical (96,) bias.

End-to-end MiniMax-H3 FL2VA validation used the model-card request
(1344x768, 243 frames, 24 fps, 10 seconds, 50 steps, seed 0, Cache-DiT):

Kernel	Engine time	Cache-DiT skips	PSNR vs BF16	SSIM vs BF16
FlashInfer MXFP4 W4A4	382.9 s	0	10.85 dB	0.386
Marlin MXFP4 W4A16	449.9 s	20	11.12 dB	0.376

Both outputs are coherent and follow the BF16 scene, subject, and motion. The
PSNR/SSIM values are low because the diffusion trajectory diverges after
quantization; they are not reconstruction metrics.

Reproduction commands, inputs, and generated media are documented in the public
model card:
https://huggingface.co/Yi30/MiniMax-H3-FL2VA-SVD-MXFP4

sample_t2va_768p_mxfp4_w4a4

Checks not run:

• Full CUDA L2/Buildkite pipelines.
• Other model families and SM103 hardware.
• Tuned AutoRound residual scales; the export and validation target
  iters=0 RTN residuals.

AI assistance

Codex assisted with implementing the runtime changes, tests, documentation, and
drafting this description. All listed commands and results were executed. The
contributor requested this upstream submission and remains responsible for
review and responses to maintainer feedback.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-26T02:03:30.474391Z We couldn’t complete this review. Comment @codex review to try again.	51e925f	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/quantization.md, docs/design/module/diffusion/index.md, docs/design/module/diffusion/offloader.md.

Module owners: @david6666666 @Isotr0py @lishunyang12

Routing: @david6666666 via module of the changed files, module named in the PR description, model owner, CODEOWNERS; @Isotr0py via module of the changed files, module named in the PR description; @lishunyang12 via module of the changed files, module named in the PR description

@yiliu30, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 2dc0772f552f produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[yiliu30]

Hi @lishunyang12 @david6666666 @xuechendi PTAL, thx!

[david6666666]

Two Marlin compatibility issues in the new MXFP4 path.

[david6666666]

Marlin permutes and may pad layer.bias here, but apply() passes bias=None to the kernel and adds that packed bias directly to the output afterward. This gives wrong values for biased MXFP4 layers, or a shape error when N needs padding. Could we preserve the original bias for that final add, or let Marlin apply it?

[yiliu30]

Good catch. Fixed by preserving the logical bias before MXFP4 post-load processing, then using it for the final add after the rank correction. Added a CPU regression plus a B200 ReplicatedLinear test forced through Marlin with padded N (96 → 128) and a biased forward.

[Bounty-hunter]

[P2] Preserve the caller's bias suppression when using the saved Marlin bias

The original packed-bias problem is addressed, but at 573a73c67b0ed04a3fb621bb6221dfb65afe9e9c, svdquant_config.py:445-447 now adds _svdquant_bias even when the caller explicitly passes bias=None. In vLLM 0.32.0, RowParallelLinear.forward() passes None on every TP rank except rank 0 before all-reducing the outputs (linear.py:1775-1781). Marlin replaces the bias on every rank, so this path adds it on every rank and produces sum(partials) + TP * bias instead of sum(partials) + bias. This is reachable for biased quantized row-parallel projections such as FLUX to_out.0 (flux_transformer.py:165-172).

The same ownership issue breaks skip_bias_add=True: the method adds bias despite None, while the linear wrapper returns the still-permuted/padded layer.bias for the caller to add separately. Please preserve both the caller's decision whether to add bias and the logical bias returned by the wrapper. Add coverage for a biased RowParallelLinear at TP=2 and for skip_bias_add=True; the new TP=1 ReplicatedLinear test cannot exercise either case.

This is a source-verified follow-up to this thread; I did not execute PR code or claim a hardware reproduction.

[yiliu30]

Good catch. Fixed in 2dc0772: MXFP4 processing now restores the checkpoint bias onto layer.bias after Marlin repacking, and apply() adds bias only when the caller supplies one. This preserves TP>1 row-parallel behavior and skip_bias_add=True while keeping the logical bias shape. Added regressions for a biased TP=2 RowParallelLinear and skip_bias_add=True; 14 focused tests, 36 SVDQuant/H3 tests, and the B200 Marlin replicated-linear test pass. Pre-commit and mypy are clean.

[david6666666]

Please also set layer.input_size_per_partition. ReplicatedLinear doesn't define it, but Marlin's prepare_fp4_layer_for_marlin() reads it, so MXFP4 loading fails for replicated projections such as FLUX's proj_mlp. A loading test with a real ReplicatedLinear would catch this.

[yiliu30]

Good catch. create_weights() now sets layer.input_size_per_partition. The new B200 regression constructs a real ReplicatedLinear, forces Marlin, and exercises the loading and forward path that previously failed.

[yiliu30]

Re-requesting review on the current head 6df271096996a5751752565295efeac30d65491d from @david6666666 after the latest updates. Thanks!

[xuechendi]

Thanks @yiliu30. The MXFP4 layout, kernel dispatch, Marlin bias fix and H3 grouped-QKV reorder for the serialized tensors all look correct to me. Comments:

1. XPU works with this PR as-is, except for the gate (suggestion, fine as a follow-up). In vLLM 0.32, init_mxfp4_linear_kernel(kMxfp4Dynamic) selects XPUMxFp4LinearKernel (W4A4) on XPU, and its weight/scale ABI matches what process_weights_after_loading registers here. Only the CUDA SM100/103 check in _assert_supported blocks it.
   I bypassed only that check on an Intel Arc Pro B70 (torch 2.14.0+xpu). A ReplicatedLinear(256 → 96, bias=True) SVDQuant MXFP4 layer with rank 8 selects XPUMxFp4LinearKernel. Its output matches an exact dequantized reference including rank correction and bias: max abs err 4.0 on |ref| ≈ 1500, rel 2.7e-3, i.e. bf16 rounding. Could _assert_supported accept XPU for mxfp4, e.g. if precision == "mxfp4" and current_platform.is_xpu(): return?
2. svdquant_config.py:358: layer._svdquant_bias is a plain tensor attribute. module.to() and the offload hooks won't move it, and anything that walks parameters or buffers won't see it. With FlashInfer, which leaves layer.bias untouched, it also duplicates the bias. Consider layer.register_buffer("_svdquant_bias", ..., persistent=False), and keep it only when the kernel actually replaced layer.bias (shape or identity changed).
3. Nit, minimax_h3_transformer.py:1793: .attn.qkv_proj.bias is listed, but H3's qkv_proj is built with bias=False, so that suffix never matches. Harmless.
4. FYI, [AutoRound][Model][MiniMax-H3] support MXFP8-DiT and MXFP4-TE quantized checkpoint loading #9363 edits the same elif chain (it adds .attn.qkv_proj.weight_scale), so there's a trivial conflict for whichever PR lands second.

XPU regression check at 6df2710 against merge-base 81d8516, on the same B70 setup.

• Command: pytest tests/diffusion/models/minimax_h3 tests/diffusion/quantization -k "svdquant or minimax".
• Head: 1107 passed, 1 failed. Base: 1098 passed, the same 1 failed. That failure needs HF Hub access, which this node doesn't have.
• No new failures. The CUDA-only SVDQuant GPU tests skip as expected.

[yiliu30]

Good catches, thanks @xuechendi.

• _assert_supported now accepts XPU for mxfp4; NVFP4 remains CUDA SM103-only.
• _svdquant_bias is now a non-persistent buffer, registered only when MXFP4 processing replaces or changes layer.bias. Added coverage for module moves, state-dict exclusion, and the unchanged FlashInfer/XPU bias path.
• Removed the dead .attn.qkv_proj.bias suffix and updated the loader test.
• Ack on the [AutoRound][Model][MiniMax-H3] support MXFP8-DiT and MXFP4-TE quantized checkpoint loading #9363 conflict; I will rebase whichever PR lands second.

Updated in 573a73c67b0ed04a3fb621bb6221dfb65afe9e9c. Local targeted suite: 34 passed; pre-commit clean.

[xuechendi]

Thanks @yiliu30, 573a73c addresses all four of my points. I re-ran on Intel Arc (B70, torch 2.14.0+xpu, vLLM 0.32) at this head with no gate bypass: the SVDQuant MXFP4 ReplicatedLinear selects XPUMxFp4LinearKernel and matches the dequantized reference (rel err 2.7e-3), and the SVDQuant + H3 quantization test files give 34 passed / 1 skipped.

+1 to @Bounty-hunter's point on svdquant_config.py:445. I reproduced it with the fake Marlin kernel from test_svdquant_linear.py: method.apply(layer, x, None) still adds the bias. Restoring ... if bias is not None else None makes that case pass. LGTM once that's in.

[yiliu30]

Hi @xuechendi @Bounty-hunter @david6666666 @hsliuustc0106 PTAL, thanks!