Skip to content

[Model] Support GLM-5.3 Flash NVFP4 loading - #38621

Merged
YAMY1234 merged 3 commits into
sgl-project:mainfrom
nvpohanh:codex/glm53-nvfp4-minimal-support
Sep 9, 2026
Merged

YAMY1234 merged 3 commits into
sgl-project:mainfrom
nvpohanh:codex/glm53-nvfp4-minimal-support

Conversation

@nvpohanh

@nvpohanh nvpohanh commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

[by Codex]

Motivation

nvidia/GLM-5.3-Flash-NVFP4 is a mixed-precision ModelOpt checkpoint: attention and shared-expert modules remain unquantized while routed experts use NVFP4.

Two GLM-5-specific mismatches currently prevent the checkpoint from loading:

  1. ModelOpt exclusion patterns use Hugging Face names under model.language_model.*, while the corresponding SGLang modules are under model.*. The weight loader already performs this rename, but precision selection did not.
  2. Shared-expert fusion cannot combine the checkpoint's unquantized shared experts with its NVFP4 routed experts.

This PR is intentionally narrower than #36761 and #37322. It does not change generic packed-module matching or redesign NextN quantization. Current main already keeps the GLM-5 NextN layer unquantized.

Modifications

  • Add the existing GLM-5 Hugging Face-to-SGLang name mapping to quantization-config matching.
  • Disable shared-expert fusion only when a ModelOpt FP4 checkpoint excludes shared experts but quantizes routed experts.
  • Add unit coverage for text/vision exclusion mapping, the mixed-precision fusion guard, and the uniform-FP4 case where fusion remains enabled.

Accuracy Tests

Validated an unchanged snapshot of nvidia/GLM-5.3-Flash-NVFP4 on 4x NVIDIA GB300 using lmsysorg/sglang:dev-cu13 and the requested launch configuration. The server became healthy without editing checkpoint metadata and without passing --disable-shared-experts-fusion:

sglang serve \
  --model-path nvidia/GLM-5.3-Flash-NVFP4 \
  --quantization modelopt_fp4 \
  --tp-size 4 \
  --dsa-prefill-backend trtllm \
  --dsa-decode-backend trtllm \
  --kv-cache-dtype fp8_e4m3 \
  --moe-runner-backend flashinfer_cutlass \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 6 \
  --speculative-adaptive \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --mem-fraction-static 0.85 \
  --cuda-graph-max-bs-decode 32 \
  --host 0.0.0.0 \
  --port 30000

Basic deterministic GSM8K subset:

  • Accuracy: 31/32 (96.9%)
  • Invalid answers: 0/32
  • Completion tokens: 2,596

Local regression tests:

44 passed, 2 subtests passed

The tested set includes test_modelopt_nvfp4.py, test_glm5_next_dflash_capture.py, test_shared_experts_fusion_gates.py, and the new GLM-5 ModelOpt tests.

Speed Tests and Profiling

Not applicable. This only changes checkpoint interpretation and prevents an invalid fusion combination; uniformly quantized configurations retain the existing fusion path.

Checklist

  • Ran pre-commit run --all-files --show-diff-on-failure successfully.
  • Added unit tests for the new behavior.
  • Provided an end-to-end accuracy smoke test.
  • Followed the existing SGLang model and quantization patterns.
  • Documentation is not required for this targeted compatibility fix.

CI States

Latest PR Test (Base): ❌ Run #34386744816
Latest PR Test (Extra): ❌ Run #34386744469
Latest PR Test (AMD ROCm 10): ❌ Run #34386744775

Comment thread python/sglang/srt/models/glm5_next.py
Comment thread test/registered/unit/models/test_glm5_next_modelopt.py
@nvpohanh
nvpohanh marked this pull request as ready for review September 9, 2026 05:49
@nvpohanh

nvpohanh commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-tests test/registered/e2e/models/test_glm53_flash_b200.py

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-tests test/registered/e2e/models/test_glm53_flash_b200.py:

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_glm53_flash_b200.py

@YAMY1234

YAMY1234 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/models/test_glm5_next_modelopt.py

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/models/test_glm5_next_modelopt.py:

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/models/test_glm5_next_modelopt.py

Comment thread test/registered/unit/models/test_glm5_next_modelopt.py
@mmangkad

mmangkad commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Wait for #38621 (comment) to pass

@YAMY1234 YAMY1234 added the run-ci label Sep 9, 2026
@YAMY1234

YAMY1234 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

@YAMY1234
YAMY1234 merged commit 96d91ef into sgl-project:main Sep 9, 2026
129 of 152 checks passed
pllimax added a commit to pllimax/sglang that referenced this pull request Sep 10, 2026
* origin/main: (27 commits)
  [Simulator] Give the OFFLINE/BLOCKING comparison tolerances real headroom (sgl-project#38732)
  [Config] msgspec.Struct for the config tier (sgl-project#38753)
  [AMD] ci: move the miles nightlies from rocm700 to rocm10 (sgl-project#37495)
  [Config] One writer for the declaration stash; no exception to the write seal (sgl-project#38752)
  docker(xpu): drop redundant setvars.sh from torch_memory_saver RUN (sgl-project#38665)
  [XPU][Fix] Pack device-pointer tables as uint64 to avoid 64-bit address overflow (sgl-project#35051)
  [CI] Temporarily disable GB300 tests (sgl-project#38770)
  [diffusion] feat: spill large tensors over shared memory like numpy arrays (sgl-project#38656)
  [diffusion] refactor: refactor utility ownership and document helper placement (sgl-project#38699)
  [NPU]Support GLM5.2 and FP8 DSA&Indexer kvcache for 950 (sgl-project#38250)
  [CI] Answer unrecognized slash commands instead of skipping silently (sgl-project#38736)
  [AMD] Parallelize aiter spec-decode KV index building over token blocks (sgl-project#37659)
  [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 (sgl-project#30805)
  Add Opt-In for GLM-5.3 Flash breakable prefill CUDA graphs (sgl-project#38522)
  [CI] Install helion 1.4.0 for the KDA Helion kernel tests (sgl-project#38688)
  [Rust] Gate health on startup warmup completion (sgl-project#37994)
  [HiCache] Replace skip_lock_node_ids with a segment lock protocol (sgl-project#36848)
  feat: add optimized Domino rollout to DFlash V2 (sgl-project#36899)
  [CI] Add /run-full-ci and /run-extra-ci slash commands (sgl-project#38734)
  [Model] Support GLM-5.3 Flash NVFP4 loading (sgl-project#38621)
  ...
Leoyzen pushed a commit to Leoyzen/sglang that referenced this pull request Sep 10, 2026
Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com>
(cherry picked from commit 96d91ef)
mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants