Skip to content

[Feature] Add Muse Glimmer model support - #34262

Merged
hnyls2002 merged 24 commits into
mainfrom
muse-glimmer
Aug 11, 2026
Merged

[Feature] Add Muse Glimmer model support#34262
hnyls2002 merged 24 commits into
mainfrom
muse-glimmer

Conversation

@sglang-bot

@sglang-bot sglang-bot commented Aug 10, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Member

Adds native Muse Glimmer model support across the CUDA and MLX backends.

CI States

Latest PR Test (Base): ❌ Run #31538429682
Latest PR Test (Extra): 🚫 Run #31543535341

@sglang-bot Brayden Zhong
@Jiminator @hnyls2002 @alexnails
Muse Glimmer native model support
ffd9d39 
Co-Authored-By: Brayden Zhong <brayden.zhong@radixark.ai>
Co-Authored-By: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-Authored-By: hnyls2002 <lsyincs@gmail.com>
Co-Authored-By: Alex Nails <alex.nails@radixark.ai>
@b8zhong b8zhong mentioned this pull request Aug 10, 2026
Merged
Brayden Zhong added 3 commits August 10, 2026 10:56
Brayden Zhong
Skip mxfp8_gemm in FlashInfer autotune
d3c5493
Brayden Zhong
Revert "Skip mxfp8_gemm in FlashInfer autotune"
561afe8 
This reverts commit d3c5493.
Brayden Zhong
Keep MXFP8 dense GEMM on cutlass for SM120
10c0a70 
flashinfer_mxfp8_blockscaled_linear swaps cutlass -> cute-dsl at M <= 64 for
the small-M speedup, but CuTe-DSL has no mm_mxfp8 kernel at capability 120, so
the swap raises BackendSupportedError. Prefill CUDA graph capture uses small
batch shapes, so an MXFP8-containing checkpoint (e.g. the MIXED_PRECISION NVFP4
build, whose down_proj is MXFP8) fails at startup on SM120.

Running FlashInfer autotune over mxfp8_gemm happened to mask this, so it only
surfaces when that op is skipped or autotune is off.

Gate the swap on SM100, which keeps the speedup where CuTe-DSL has the kernel.
@b8zhong
b8zhong requested review from BBuf and ch-wan as code owners August 10, 2026 11:11
@github-actions github-actions Bot added the quant LLM Quantization label Aug 11, 2026
@jasonge27

jasonge27 commented Aug 11, 2026

Copy link
Copy Markdown

I tested this PR on my M4 Pro with 48GB, it turns out that the PR can’t load the public mlx-community/Muse-Glimmer-30B-4bit checkpoint. muse_glimmer_mlx.py::sanitize() validates incoming weights against a raw, unquantized key schema and we got this error:

ValueError: not a complete raw Muse Glimmer HF checkpoint: 627 missing keys
['lm_head.weight', ...], 1463 unexpected keys
['language_model.lm_head.biases', 'language_model.lm_head.scales', ...]

@IwakuraRein IwakuraRein mentioned this pull request Aug 11, 2026
Closed
5 tasks
@hnyls2002
hnyls2002 merged commit fde9ad2 into main Aug 11, 2026
153 of 199 checks passed
@hnyls2002
hnyls2002 deleted the muse-glimmer branch August 11, 2026 22:41
@Fridge003 Fridge003 added the release-highlight Candidate PR for release note highlight label Aug 11, 2026
@Jiminator

Jiminator commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

I tested this PR on my M4 Pro with 48GB, it turns out that the PR can’t load the public mlx-community/Muse-Glimmer-30B-4bit checkpoint. muse_glimmer_mlx.py::sanitize() validates incoming weights against a raw, unquantized key schema and we got this error:

ValueError: not a complete raw Muse Glimmer HF checkpoint: 627 missing keys
['lm_head.weight', ...], 1463 unexpected keys
['language_model.lm_head.biases', 'language_model.lm_head.scales', ...]

Hey @jasonge27, for the MLX backend on Sglang, use one of the three MLX checkpoints we host under https://huggingface.co/RadixArk: Muse-Glimmer-q4-MLX, Muse-Glimmer-q4km-gs128-MLX, or Muse-Glimmer-q4k-dynamic-MLX. The two k-quant repacks preserve the exact quantization codes of the GGUFs in meta-models/Muse-Glimmer-30B-GGUF (bit-identical code), and q4 is the fastest variant. The mlx-community checkpoint's text weights are actually the same 4-bit quantization as our q4, but it is stored as a pre-quantized mlx-lm conversion of the full multimodal export, which isn't an input shape sanitize() supports. The loader accepts either the raw bf16 HF export, where it applies the fusion and norm folding itself, or our packaged artifacts, since those transforms can't be applied to already-quantized weights. Image input is also not supported on our MLX backend, so the vision tower it carries wouldn't be usable regardless.

@hnyls2002 hnyls2002 mentioned this pull request Aug 12, 2026
Open
vstone-w pushed a commit to ClownBin/sglang that referenced this pull request Aug 12, 2026
@sglang-bot Brayden Zhong
@Jiminator @hnyls2002 @alexnails @vstone-w
[Feature] Add Muse Glimmer model support (sgl-project#34262)
39ea08c 
Co-authored-by: sglang-bot <232288953+sglang-bot@users.noreply.github.com>
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
@mmangkad mmangkad mentioned this pull request Aug 12, 2026
Merged
5 tasks
@muthu-kumaravel muthu-kumaravel mentioned this pull request Aug 12, 2026
Merged
@amd-bot amd-bot mentioned this pull request Aug 13, 2026
Open
@KrishnanPrash KrishnanPrash mentioned this pull request Aug 13, 2026
Merged
@zixi-qi zixi-qi mentioned this pull request Aug 13, 2026
Merged
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Aug 14, 2026
@Leoyzen @hnyls2002
fix(parser): DSV4 tool-call streaming — infra + chunk-invariance + ha…
907448d 
…rdening

Cherry-pick and squash of three PRs:
- sgl-project#34262 infra: BaseFormatDetector.finish(), FunctionCallParser.parse_stream_end(),
  flush param in serving_chat._process_tool_call_stream
- sgl-project#34458: chunk-invariance — partial token holdback, _ends_with_partial_token,
  tool_start_token for DeepSeekV4Detector, MalformedJSON catch, preamble preservation
- sgl-project#34600: hardening — buffer retention on error, rstrip→removesuffix, narrow
  potentially_dsml to tool-call tags, finish() override on DeepSeekV32Detector,
  narrow tool_start_token to <｜DSML｜tool_calls, holdback at think_end boundary

Co-authored-by: hnyls2002 <hnyls2002@gmail.com>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Aug 14, 2026
@Leoyzen @hnyls2002
fix(parser): DSV4 tool-call streaming — infra + chunk-invariance + ha…
8a7dca4 
…rdening

Cherry-pick and squash of three PRs:
- sgl-project#34262 infra: BaseFormatDetector.finish(), FunctionCallParser.parse_stream_end(),
  flush param in serving_chat._process_tool_call_stream
- sgl-project#34458: chunk-invariance — partial token holdback, _ends_with_partial_token,
  tool_start_token for DeepSeekV4Detector, MalformedJSON catch, preamble preservation
- sgl-project#34600: hardening — buffer retention on error, rstrip→removesuffix, narrow
  potentially_dsml to tool-call tags, finish() override on DeepSeekV32Detector,
  narrow tool_start_token to <｜DSML｜tool_calls, holdback at think_end boundary

Co-authored-by: hnyls2002 <hnyls2002@gmail.com>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Aug 14, 2026
@Leoyzen @hnyls2002
fix(parser): DSV4 tool-call streaming — infra + chunk-invariance + ha…
86d8672 
…rdening

Cherry-pick and squash of three PRs:
- sgl-project#34262 infra: BaseFormatDetector.finish(), FunctionCallParser.parse_stream_end(),
  flush param in serving_chat._process_tool_call_stream
- sgl-project#34458: chunk-invariance — partial token holdback, _ends_with_partial_token,
  tool_start_token for DeepSeekV4Detector, MalformedJSON catch, preamble preservation
- sgl-project#34600: hardening — buffer retention on error, rstrip→removesuffix, narrow
  potentially_dsml to tool-call tags, finish() override on DeepSeekV32Detector,
  narrow tool_start_token to <｜DSML｜tool_calls, holdback at think_end boundary

Co-authored-by: hnyls2002 <hnyls2002@gmail.com>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 16, 2026
@sglang-bot Brayden Zhong
@Jiminator @hnyls2002 @alexnails @saturn-acc
[Feature] Add Muse Glimmer model support (sgl-project#34262)
ba81810 
Co-authored-by: sglang-bot <232288953+sglang-bot@users.noreply.github.com>
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
@hello-alexmcc hello-alexmcc mentioned this pull request Aug 22, 2026
Merged
4 tasks
@nbald nbald mentioned this pull request Sep 3, 2026
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JustinTong0323 JustinTong0323 Awaiting requested review from JustinTong0323 JustinTong0323 is a code owner
@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner
@Qiaolin-Yu Qiaolin-Yu Awaiting requested review from Qiaolin-Yu Qiaolin-Yu is a code owner
@mickqian mickqian Awaiting requested review from mickqian mickqian is a code owner
@yhyang201 yhyang201 Awaiting requested review from yhyang201 yhyang201 is a code owner
@yuan-luo yuan-luo Awaiting requested review from yuan-luo yuan-luo is a code owner
@b8zhong b8zhong Awaiting requested review from b8zhong b8zhong is a code owner
@mmangkad mmangkad Awaiting requested review from mmangkad mmangkad is a code owner
@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner
@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner
@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner
@hanming-lu hanming-lu Awaiting requested review from hanming-lu hanming-lu is a code owner
@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner
@hzh0425 hzh0425 Awaiting requested review from hzh0425 hzh0425 is a code owner
@alphabetc1 alphabetc1 Awaiting requested review from alphabetc1 alphabetc1 is a code owner
@yeahdongcn yeahdongcn Awaiting requested review from yeahdongcn yeahdongcn is a code owner
@CatherineSue CatherineSue Awaiting requested review from CatherineSue CatherineSue is a code owner
@slin1237 slin1237 Awaiting requested review from slin1237 slin1237 is a code owner
@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner
@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner
@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner
@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner
@AniZpZ AniZpZ Awaiting requested review from AniZpZ AniZpZ is a code owner
@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner
@OrangeRedeng OrangeRedeng Awaiting requested review from OrangeRedeng OrangeRedeng is a code owner
@Alisehen Alisehen Awaiting requested review from Alisehen Alisehen is a code owner

Assignees

No one assigned

Labels

apple-silicon bypass-fastfail high priority quant LLM Quantization release-highlight Candidate PR for release note highlight run-ci run-ci-extra

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sglang-bot @hnyls2002 @jasonge27 @Jiminator @Fridge003