[Fix] Restore online MXFP8 quantization for linear layers - #32953
Merged
Conversation
b8zhong
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng,
ch-wan and
mmangkad
as code owners
July 30, 2026 18:14
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
`--quantization mxfp8` on a non-fp8 checkpoint builds `Fp8Config(use_mxfp8=True, is_checkpoint_fp8_serialized=False)`, which `create_fp8_weight_` rejected with "MXFP8 requires fp8-serialized checkpoint for linear layers.". The guard used to sit inside the `if is_checkpoint_fp8_serialized:` branch, where it was unreachable; hoisting it out during the static-method refactor made it fire. Drop it and let `process_weights_after_loading` quantize the bf16 weights as it already does.
b8zhong
force-pushed
the
fix/online-mxfp8-linear
branch
from
July 30, 2026 18:25
5b08a4e to
ba20cb4
Compare
mmangkad
approved these changes
Jul 30, 2026
mmangkad
enabled auto-merge (squash)
July 30, 2026 23:22
fused_a_gemm_weight_eligible() inspects fc1_latent_proj.weight, but quant methods rewrite that weight in process_weights_after_loading (e.g. bf16 -> float8_e4m3fn), so the check at construction time sees a non-final dtype. Defer it to the first forward instead.
Contributor
|
Thanks @b8zhong |
This was referenced Jul 31, 2026
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Any bf16 checkpoint launched with
--quantization mxfp8fails at model init because AMD broke it again:But that is exactly the online quantization path — the weights are meant to be created in bf16 and quantized in
process_weights_after_loading.The guard was unreachable when added in #17449 (nested inside
if is_checkpoint_fp8_serialized:). Thecreate_fp8_weight_refactor in #28291 hoisted it out to anelif, making it fire. Only linear layers are affected; the MoE path never had this guard.Modifications
Drop the
elif use_mxfp8: raise ValueError(...)branch inFp8LinearMethod.create_fp8_weight_.Checklist
CI States
Latest PR Test (Base): ✅ Run #30595008130
Latest PR Test (Extra): ❌ Run #30595007887