fix(modelopt): expand is_layer_excluded for fused and model.-prefixed names - #36761
Open
chakshu-dhannawat wants to merge 1 commit into
Open
chakshu-dhannawat wants to merge 1 commit into
chakshu-dhannawat wants to merge 1 commit into
Conversation
… names ModelOpt FP4/FP8 mixed-precision checkpoints list excluded modules by unfused projection names (e.g. q_proj) or model.visual.* prefixes, but is_layer_excluded only matched the literal prefix. Expand the candidate set using packed_modules_mapping and also test model.-prefixed variants so fused modules and vision towers are correctly excluded. Fixes sgl-project#36596
chakshu-dhannawat
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng,
b8zhong,
ch-wan and
mmangkad
as code owners
August 28, 2026 01:14
5 tasks
5 tasks
3 tasks
divyvasal
added a commit
to divyvasal/sglang
that referenced
this pull request
Sep 10, 2026
ModelOpt writes some vision-language checkpoints with the decoder nested under
`model.language_model.*` while sglang builds it as `model.*`. The strip in
`apply_weight_name_mapper` only fires for a *leading* `language_model.`, so an
interior one never matches: every layer the checkpoint keeps in BF16 is built as
an FP4 module and the load dies on a shape mismatch.
Seen on nvidia/GLM-5.3-Flash-NVFP4, whose exclusion list is written as
`model.language_model.layers.N.self_attn*`, `...mlp.gate` and
`...mlp.shared_experts*`. With TP8 on 8xB200 the first mismatch is:
AssertionError: target.shape=torch.Size([4096, 256]),
loaded_weight.shape=torch.Size([4096, 512])
the FP4-packed parameter against the BF16 tensor the checkpoint actually ships.
Expand the interior form alongside the existing leading-strip, at the same point
and in the same shape. Names without the segment are untouched, and the
expansion is exact rather than a wildcard, so it cannot widen a match: layer 10's
gate stays excluded while layer 11's is not.
This is the decoder half of sgl-project#36596. sgl-project#36761 covers the fused-module and
`model.`-prefixed vision-tower halves on the prefix side; the two are orthogonal
and touch different functions. On this checkpoint neither alone is sufficient:
sgl-project#36761 fixes `visual.*`, this fixes `model.layers.*`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes the
ModelOptQuantConfig.is_layer_excludedmatcher so mixed-precision ModelOpt FP4/FP8 checkpoints correctly honor their exclusion lists when SGLang builds fused modules or renamesmodel.visual.*tovisual.*during load (reported in #36596).Changes:
python/sglang/srt/layers/quantization/modelopt_quant.py:packed_modules_mappingbefore matching, so an exclude entry like*.self_attn.q_projalso matches a builtqkv_proj.model.-prefixed variants of every candidate prefix, somodel.visual.*exclusions still match after the vision tower is renamed tovisual.*.test/registered/unit/layers/quantization/test_modelopt_nvfp4.py: addTestModelOptIsLayerExcludedcovering exact match,language_model.prefix stripping, fused-module expansion,model.visual/visualrenaming, and wildcard patterns.The new unit tests pass:
All existing tests in
test_modelopt_nvfp4.pystill pass, andruff check --select=F401,F821,UP037andblackare clean on the changed files.CI States
Latest PR Test (Base): ❌ Run #33132248773
Latest PR Test (Extra): ❌ Run #33132248698
Latest PR Test (AMD ROCm 7.2): ❌ Run #33132248727