Repository navigation
[Quant] Serve 32-wide-K ue8m0 block-FP8 linears through the FlashInfer MXFP8 GEMMs - #40039
Merged
Merged
Conversation
…ain_weight_layout
hnyls2002
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng,
b8zhong,
ch-wan and
mmangkad
as code owners
September 17, 2026 23:26
hnyls2002
added a commit
that referenced
this pull request
Sep 17, 2026
Collaborator
Author
|
/tag-and-rerun-ci |
Collaborator
Author
|
/rerun-test test_flashinfer_trtllm_fp8_fallback.py test_fp8_utils.py test_fp8_blockwise_linear_backends.py |
Contributor
|
Results for 🚀 🚀 🚀 🚀 |
Collaborator
Author
|
/rerun-test test_fp8_utils.py |
Contributor
|
Results for 🚀 |
Collaborator
Author
|
/rerun-test test_hicache_storage_lora.py |
Contributor
|
Results for 🚀 |
…k-fp8-mxfp8 # Conflicts: # python/sglang/srt/layers/quantization/fp8.py
hnyls2002
added a commit
that referenced
this pull request
Sep 18, 2026
This was referenced Sep 18, 2026
zozyo
pushed a commit
to Phala-Network/sglang
that referenced
this pull request
Sep 22, 2026
…r MXFP8 GEMMs (sgl-project#40039) (cherry picked from commit 1b200ff) (cherry picked from commit e533126)
mickqian
added a commit
to TyGu888/sglang
that referenced
this pull request
Oct 6, 2026
Resolve the conflict with the online MXFP8 branch from sgl-project#37903: unserialized checkpoints still use MXFP8OnlineLinearMethod, serialized ones now get ComfyMXFP8LinearMethod. Adapt the override to the scale_u8 argument that sgl-project#40039 added to Fp8LinearMethod._process_mxfp8_linear_weight_scale. Only scales read from a comfy checkpoint skip the interleave; an explicit scale_u8 is converted from block-FP8 in row-major order and still goes through SRT. Update the unit test expectations accordingly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Block-FP8 weights with a 32-wide K block and ue8m0 scales are MXFP8 operands;
Fp8LinearMethodserves them through the FlashInfer CUTLASS / CuTe-DSL MXFP8 GEMMs when that backend is selected, accepts a prequantizedMxfp8SwizzledInput, and dispatches the block kernel on the effective block size. No behavior change for existing FP8 checkpoints; consumer is #38798.CI States
Latest PR Test (Base): 🚫 Run #35330783063
Latest PR Test (Extra): ❌ Run #35330782851
Latest PR Test (AMD ROCm 10): 🚫 Run #35330783022