[Perf][GLM-5.3-Flash] DeepGEMM FP8 MegaMoE with fused shared experts by gcanlin · Pull Request #60895 · vllm-project/vllm

gcanlin · 2026-10-09T17:24:26Z

Overview

Run GLM-5.3-Flash's MoE layers with DeepGEMM's FP8xFP8 MegaMoE (routed and shared experts in one kernel that also dispatches and combines across the EP group), using the official FP8 checkpoint. Enabled by --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' with --tensor-parallel-size 1 --data-parallel-size N --enable-expert-parallel.

Related: #55275 (BF16 / MXFP4 MegaMoE for GLM, open), #54049 (FlashInfer NVFP4 mega kernel).

Claims

On the AgentX replay at concurrency 96, DEP4 with MegaMoE gives 1.6x the output throughput of DEP4 on the generic MoE path.
Against TP4 it is about level on throughput (+6%, one run each) and has about a fifth of the TTFT.
It prefills 1.8x faster than DEP4 generic and 2.5x faster than TP4.
AIME 2026 Pass@1 matches the generic MoE path (0.934 vs 0.939 with a 131K output cap) and is 0.977 with the 1M context.
A weight reload, in layerwise or direct mode, restores byte-identical weights at the same addresses.
No change for any other moe_backend.

Validation

4x B200, nightly 8cbd5d03 with this change applied, MTP 5 with acceptance fixed by synthetic_acceptance_length 3.61. The three arms ran on the same node and in the same container, one run each; they differ only in parallel layout and moe_backend.

AgentX replay (InferenceX-pinned aiperf, semianalysis_cc_traces_weka_062126), concurrency 96, 1 hour:

metric	TP4	DEP4, generic MoE	DEP4, MegaMoE
output tok/s	3,152	2,063	3,332
input tok/s	380,126	234,727	399,297
total tok/s	383,278	236,790	402,628
requests completed (errors)	11,372 (4)	8,143 (1)	11,986 (5)
TTFT avg / p50 / p99	3.93 / 2.19 / 23.8 s	2.36 / 1.47 / 17.3 s	0.81 / 0.49 / 5.99 s
ITL avg	18.3 ms	45.6 ms	19.5 ms
output tok/s per user	64.0	26.2	64.1
prefix cache hit	93.9%	92.4%	94.0%
of which GPU / host reload	6.2% / 87.7%	50.9% / 41.5%	71.2% / 22.8%
preemptions	35	53	2

vllm bench serve, random prompts, concurrency 128, on the same servers:

workload (in/out)	metric	TP4	DEP4, generic MoE	DEP4, MegaMoE
8192 / 1	total tok/s	46,334	63,900	115,081
	mean TTFT	20.8 s	15.2 s	8.4 s
1024 / 1024	output tok/s	7,895	8,478	11,691
	mean TTFT	4.31 s	1.86 s	0.85 s

Accuracy:

eval	TP4	DEP4, generic MoE	DEP4, MegaMoE
GSM8K, lm-eval 5-shot, 1,319 questions	0.9757 ± 0.0042	–	0.9735 ± 0.0044
AIME 2026 Pass@1, 30 problems x 32 samples	–	0.939 ± 0.006	0.934 ± 0.006

GSM8K ran on nightly 43b4aaea (TP4) and the servers above (MegaMoE). AIME ran on nightly df3bd094 with this branch, both arms in the same container (MTP 3, fp8 KV, 131K output cap) with DeepInfer's eval/aime prompt and rule-based judge, temperature 1.0, top_p 0.95; the generic arm runs FLASHINFER_TRTLLM on the checkpoint's own scales. With the full 1M context MegaMoE reaches 0.977 ± 0.005; the remaining loss is mostly problem 15 (0.72), where 12 of 32 samples never stop and loop until the context is full.

Weight reload, 4x B200, nightly e0d00e4b8e with this branch, DEP4, MTP 3. The experts are zeroed before each reload, which breaks every greedy output; all results are compared with the cold start:

metric	layerwise	direct
target reload time	41.6 s	37.3 s
parameters and buffers differing (5,900 across ranks)	0	0
parameters and buffers moved	0	0
greedy outputs matching (4 short prompts)	4/4	4/4

The same holds for a second reload, a reload of the MTP drafter, and an update that sends expert weights and block scales in separate load_weights calls. Long prompts (about 14K tokens) already vary between identical runs without a reload, on both MoE backends, so they are not compared. No real NCCL or IPC trainer was run.

Unit tests: pytest tests/models/glm5next/test_mega_moe.py, 5 passed.

Commands

VLLM_USE_RUST_FRONTEND=1 PYTHONHASHSEED=42 vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel --max-num-seqs 96 \
  --language-model-only --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.61}' \
  --kernel-config '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe"}' \
  --compilation-config '{"cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' \
  --max-cudagraph-capture-size 8190 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.97 \
  --prefix-match-unit 16 \
  --kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":300000000000,"enable_cross_layers_blocks":"true","lazy_offload":false}}'

vllm-router --port 8000 --worker-urls http://127.0.0.1:8001 --intra-node-data-parallel-size 4 --policy consistent_hash

aiperf profile --scenario agentx --url http://127.0.0.1:8000 --endpoint /v1/chat/completions \
  --model zai-org/GLM-5.3-Flash --concurrency 96 --benchmark-duration 3600 \
  --public-dataset semianalysis_cc_traces_weka_062126

AIME 2026:

vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel --max-num-seqs 128 \
  --language-model-only --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser glm47 \
  --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' \
  --max-model-len 1048576 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.92

The client sends DeepInfer's eval/aime prompts (build_records) to /v1/chat/completions with temperature=1.0, top_p=0.95, 32 samples per problem, and scores reasoning + "</think>" + content with its score_rollout. Samples were drawn with a 131K output cap first; the 82 that hit it were continued from their own text to the 1M context, which samples the same distribution as the larger cap.

The baselines drop moe_backend (DEP4, generic) or use --tensor-parallel-size 4 alone with --intra-node-data-parallel-size 1 on the router (TP4). The generic path picked FLASHINFER_TRTLLM for the FP8 experts. The random-prompt rows were measured on the same servers before the replay with vllm bench serve --dataset-name random --max-concurrency 128.

Details

The checkpoint's experts are block FP8 (E4M3, one float32 scale per 128x128 block). DeepGEMM's SM100 MegaMoE feeds scales to the block-scaled MMA and needs UE8M0 scales per row and 32 columns, so the routed and shared expert weights are requantized once at load (requant_block_fp8_to_ue8m0).
This requantization is lossy: about 2.4% relative L2 per weight tensor and about 3.9% on one MoE layer's output (layer 15, real weights, random inputs). The generic TRTLLM path uses the checkpoint's scales as they are. The GSM8K gap above is within one standard error.
Routing stays outside the kernel: fused_topk_bias (sigmoid, score correction bias), with routed_scaling_factor folded into the top-k weights. DeepGEMM's Mega Gate only supports sqrtsoftplus.
Glm5NextMegaMoEExperts subclasses DeepGemmMegaMoEExperts, which is unchanged, with the MEGA_MOE_MXFP8 format. E4M3 shards load straight into the kernel layout; their float32 block scales are staged in *_weight_scale_inv, and the model's process_weights_after_loading calls requantize_block_fp8(), which rounds every newly loaded shard to UE8M0 1x32 scales in place (main model, multimodal wrapper and MTP drafter).
A weight update may send an expert weight and its block scale in separate load_weights calls; by the end of the load both must be in, or requantize_block_fp8() raises.
Reload and weight transfer finish with that hook ([Model Loader] Run the model-level post-load hook once at the end of every load and reload #64357, and direct mode from [RL][Model Loader] Direct weight reload: write checkpoint tensors into the live model #64336), so the kernel tensors keep their storage. The derived UE8M0 scales are marked weight_loader_numel = 0, so a layerwise reload replays a layer as soon as its checkpoint tensors are in.
The block-FP8 shared expert is fused the same way: the experts own its weights and the model builds no shared MLP.
--prefix-match-unit matters for this model independently of this PR: the KDA state page makes the KV block 8,960 tokens under DEP4, and without the flag prefix hits land on whole blocks.
Limitations: tensor_parallel_size=1 only; SM100 only; no sharded RDT or sparse NCCL updates. Under DEP4 each rank holds 8.8M KV tokens.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

With `moe_backend=deep_gemm_mega_moe` and expert parallel, GLM-5.3-Flash's MoE layers (MTP included) run DeepGEMM's FP8xFP8 MegaMoE: one kernel dispatches, runs the routed and the shared expert and combines across the EP group. The official checkpoint's experts are block FP8 with a float32 scale per 128x128 block; the SM100 kernel takes power-of-two scales per row and 32 columns. The weights are requantized once at load. Routing stays outside the kernel (`fused_topk_bias`, the routed scaling factor folded into the top-k weights). Requires tensor_parallel_size=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

Move the DeepGEMM-specific part into the shared backend abstraction: `DeepGemmSm100Fp8MegaMoEBackend` (`mma_type="fp8xfp8"`) requantizes block-FP8 routed and shared expert weights to the kernel's UE8M0 1x32 scales, and `get_deep_gemm_mega_moe_backend` selects it by `mma_type`. `Glm5NextMegaMoEExperts` becomes a thin adapter over `DeepseekV4MegaMoEExperts`, as Kimi K3's is: it only declares the checkpoint's loader parameters and picks the FP8 backend. The base weight loader now splits gate/up by half of the parameter's rows, so block scales shard like weights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

gcanlin · 2026-10-11T17:44:32Z

I'm not very sure about the regression for accuracy. But do we have another better choice for MegaMoE?

JaredforReal · 2026-10-13T02:36:19Z

@gcanlin thanks for the bring up, can u run a AA-LCR or AIME2026 to see the accuracy regression level
I've been looking at MegaMoE of GLM53Flash for a while, but did not find other solution orz..

gcanlin · 2026-10-13T03:47:25Z

@gcanlin thanks for the bring up, can u run a AA-LCR or AIME2026 to see the accuracy regression level I've been looking at MegaMoE of GLM53Flash for a while, but did not find other solution orz..

🥲I can't access to my machine recently. So if you have free machine, could you help run the accuracy benchmark? And I just noticed that we disabled MegaMoE for Qwen as the gsm8k had degraded. But currently, GLM's gsm8k doesn't happen this regression. So I think this approach looks promising.

mergify · 2026-10-13T19:47:27Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gcanlin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Port the GLM-5.3-Flash FP8 MegaMoE onto the unified DeepGEMM MegaMoE experts from vllm-project#61538 / vllm-project#60096, which replaced DeepseekV4MegaMoEExperts and the post-load MegaMoE backend transform: - Glm5NextMegaMoEExperts now subclasses DeepGemmMegaMoEExperts with the MEGA_MOE_MXFP8 (fp8xfp8) format. E4M3 shards load straight into the kernel layout; the float32 128x128 block scales are staged and every freshly loaded shard is requantized in place to UE8M0 1x32 scales after load_weights (requant_block_fp8_to_ue8m0 moves to the GLM adapter). - The block-FP8 shared expert is fused the same way: the experts own its weights and the separate shared MLP is dropped. - The SwiGLU clamp moves to MegaMoEActivation; the deep_gemm_mega_moe.py and DeepSeek V4 conflicts take main's version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ault mypy inferred tuple[int, ...] for weight_block_size; keep it as tuple[int, int]. ruff B008 flagged slice() as a default argument in the test; use a module constant for the local expert range. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

- A weight update can send an expert weight and its block scale in separate load_weights calls. Requantization ran at the end of every call and only weights marked a shard, so a weight arriving before its scale was rounded with the stale scale and the new scale was never applied. Track weights and scales separately and requantize a shard once both have arrived; a weight whose scale was never loaded still raises. - Fuse only a block-FP8 shared expert. The fallback to the base fusion could accept an MXFP8 shared expert that the GLM loader cannot place (gate_proj/up_proj names). - Dequantize with a broadcast block scale and quantize with the shared MXFP8 reference (_mxfp8_e4m3_quantize_torch) instead of a local copy; requant_block_fp8_to_ue8m0 now returns UE8M0 bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

Reloading weights in place (layerwise reload, weight transfer) produced garbage output: - Requantization ran at the end of load_weights. During a layerwise reload the layer is still on meta at that point, so it was skipped, and the replayed raw block-FP8 values were copied back with stale kernel scales. - The block-FP8 shared expert was written straight into .data instead of through param.weight_loader, so reload never saw those writes. - The kernel scales derived by requantization are never loaded, so the layer looked incomplete and was deferred to finalize with all of its weights buffered. Requantize from the model's process_weights_after_loading (text model, multimodal wrapper and MTP drafter), load the shared expert through its weight loader, and give the derived scales weight_loader_numel = 0. The finish step rejects any shard whose weight or block scale is still missing, instead of guessing from scale values. DeepGemmMegaMoEExperts gains small hooks so the subclass no longer copies base logic: _shared_linear_fits (shared-expert fusion check), _shared_param_name / _shared_shard_id (also accept gate_proj/up_proj), and _weight_shard_view (kernel-layout view of one shard). DeepSeek V4 behavior is unchanged. Cold load works as before. Reload and weight transfer additionally need vllm-project#57733, which calls the model-level hook at the end of finalize_layerwise_reload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

…7733 Layerwise reload and weight transfer replay the block-FP8 shards into the MegaMoE experts, but nothing requantizes them until the reload calls the model's process_weights_after_loading (vllm-project#57733), so the model would serve garbage. Like expert substitution, the text model, the multimodal wrapper and the MTP drafter now detect a reload at the start of load_weights, put the previous weights back and raise NotImplementedError. Models without MegaMoE are unaffected. Remove the check once vllm-project#57733 lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

Route the routed experts of DeepseekV2MoE through the block-FP8 MegaMoE experts from vllm-project#60895 when --moe-backend deep_gemm_mega_moe is set, for the DSA models (GLM-5.x, DeepSeek-V3.2; GlmMoeDsaForCausalLM / DeepseekV32ForCausalLM and their MTP). - Routing (scoring_func + e_score_correction_bias, renormalized, times routed_scaling_factor) runs before the kernel; the shared expert stays a separate MLP. - Requires EP, TP=1, one expert group and a block-FP8 checkpoint. Works with data parallel and prefill context parallel: every EP rank calls the kernel once per layer per step. - Weights load straight into the kernel layout (routed_experts_prefix ''), are requantized in process_weights_after_loading, and layerwise reload is refused until vllm-project#57733, as in vllm-project#60895. - Other moe backends and other DeepseekV2MoE users are unchanged; DeepSeek-V2/V3 models get a clear NotImplementedError for this backend. Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>

mergify · 2026-10-28T17:24:16Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gcanlin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Route the routed experts of DeepseekV2MoE through the block-FP8 MegaMoE experts from vllm-project#60895 when --moe-backend deep_gemm_mega_moe is set, for the DSA models (GLM-5.x, DeepSeek-V3.2; GlmMoeDsaForCausalLM / DeepseekV32ForCausalLM and their MTP). - Routing (scoring_func + e_score_correction_bias, renormalized, times routed_scaling_factor) runs before the kernel; the shared expert stays a separate MLP. - Requires EP, TP=1, one expert group and a block-FP8 checkpoint. Works with data parallel and prefill context parallel: every EP rank calls the kernel once per layer per step. - Weights load straight into the kernel layout (routed_experts_prefix ''), are requantized in process_weights_after_loading, and layerwise reload is refused until vllm-project#57733, as in vllm-project#60895. - Other moe backends and other DeepseekV2MoE users are unchanged; DeepSeek-V2/V3 models get a clear NotImplementedError for this backend. Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>

Resolve the loader conflicts with vllm-project#61457 (per-layer fused shared expert selection) in the GLM-5.3 model and MTP drafter. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vllm-project#64357 runs the model's process_weights_after_loading at the end of every reload, and vllm-project#64336's direct mode reruns it too, so the block-FP8 shards a reload writes are requantized in place. Remove the refusal added until that landed. Test both modes through start_reload / finish_reload. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Keep the requant math, split weight updates, and reload in both modes, which now also checks the cold load layout. Drop tests that repeat those or restate the shared-expert fit check. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

DeepGemmMegaMoEExperts needs no hooks for GLM: the adapter overrides the existing shared-expert fusion decision and loaders, and reads a shard's kernel-layout view through _kernel_layout_shard. The backend file is back to main. Under MegaMoE the model no longer builds a shared MLP only to drop it, which removes the name probing around it. Drop checks for states the block-FP8 entry check already rules out and helpers with one caller. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…kend Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

It has nothing specific to GLM, and it no longer imports a private function from another module. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Requantizing block-FP8 MoE weights to MXFP8 is exact when every block scale is a power of two, and rounds the weights otherwise, as with GLM-5.3-Flash's checkpoint. Warn once in that case. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gau-nernst

LGTM. just a minor nit, non-blocking. I think as long as we make it clear that requant takes place, it should be ok.

cc @JaredforReal for a quick check

github-actions · 2026-11-04T00:24:26Z

✅ @gcanlin, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

gcanlin · 2026-11-04T11:03:19Z

/ci run

github-actions · 2026-11-04T11:03:51Z

✅ Triggered Buildkite CI #99906 for commit 259d3658f47f.

ℹ️ This PR is 20 commits behind upstream main. Merged upstream main into this branch as 259d3658f47f before starting CI. Pull the branch before pushing more changes.

mergify Bot added the glm label Oct 9, 2026

mergify Bot added deepseek Related to DeepSeek models DSv4 labels Oct 9, 2026

gcanlin marked this pull request as ready for review October 9, 2026 17:50

gcanlin requested review from AndreasKaratzas, DarkLight1337, mgoin, pavanimajety, wzhao18, ywang96 and zyongye as code owners October 9, 2026 17:50

claude Bot reviewed Oct 9, 2026

View reviewed changes

This was referenced Oct 11, 2026

[Bugfix][GLM-5.3-Flash] Restore MTP index sharing across draft iterations #61079

Merged

[Bugfix][GLM-5.3-Flash] Honor --mamba-ssm-cache-dtype for the KDA recurrent state #59147

Merged

mergify Bot added the needs-rebase label Oct 13, 2026

zyongye mentioned this pull request Oct 14, 2026

[MoE] Unify DeepGEMM MegaMoE experts behind one feature-matrix layer; add NVFP4 #61538

Merged

5 of 8 tasks

qianlihuang mentioned this pull request Oct 20, 2026

[Perf][MoE] Enable SM90 block-FP8 MegaMoE via FlashInfer MoE-EP #62603

Open

6 tasks done

mergify Bot removed the needs-rebase label Oct 20, 2026

gcanlin and others added 3 commits October 20, 2026 23:38

gcanlin requested a review from bnellnm as a code owner October 21, 2026 15:27

aoshen02 mentioned this pull request Oct 23, 2026

[Model Loader][DSV41] Faster RL weight reload for MegaMoE experts #63429

Closed

mergify Bot added the needs-rebase label Oct 28, 2026

gcanlin and others added 3 commits November 2, 2026 12:14

mergify Bot removed the needs-rebase label Nov 2, 2026

gcanlin and others added 4 commits November 2, 2026 13:21

[GLM-5.3-Flash] MegaMoE: read the shared-expert shard id like the bac…

b80b925

…kend Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

[GLM-5.3-Flash] MegaMoE: make the shared-expert shard id a static method

4dcadcb

Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

[GLM-5.3-Flash] MegaMoE: move the block-FP8 requant to mxfp8_utils

3454a5f

It has nothing specific to GLM, and it no longer imports a private function from another module. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gcanlin requested review from robertgshaw2-redhat, tlrmchlsmth and yewentao256 as code owners November 2, 2026 05:40

mergify Bot added the quantization label Nov 2, 2026

yewentao256 mentioned this pull request Nov 2, 2026

[Feature]: GLM 5.3 Performance Optimization #57406

Open

26 of 44 tasks

fxmarty-amd mentioned this pull request Nov 2, 2026

[RFC]: Extended online quantization roadmap #52167

Open

1 task done

Merge branch 'main' into perf/glm53-fp8-megamoe

0d99737

gau-nernst reviewed Nov 3, 2026

View reviewed changes

Comment thread vllm/models/glm5next/nvidia/mega_moe.py Resolved

gau-nernst approved these changes Nov 4, 2026

View reviewed changes

Comment thread vllm/models/glm5next/common/model.py Resolved

gau-nernst added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 4, 2026

mergify Bot and others added 2 commits November 4, 2026 04:12

Merge branch 'main' into perf/glm53-fp8-megamoe

317550c

[GLM-5.3-Flash] MegaMoE: inline the moe_backend check

f308eb6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Canlin <canlinguosdu@gmail.com>

Merge branch 'main' into perf/glm53-fp8-megamoe

259d365

Navigation Menu

Repository navigation

[Perf][GLM-5.3-Flash] DeepGEMM FP8 MegaMoE with fused shared experts - #60895

[Perf][GLM-5.3-Flash] DeepGEMM FP8 MegaMoE with fused shared experts#60895
gcanlin wants to merge 19 commits into
vllm-project:mainfrom
gcanlin:perf/glm53-fp8-megamoe

gcanlin commented Oct 9, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

gcanlin commented Oct 11, 2026

Uh oh!

JaredforReal commented Oct 13, 2026

Uh oh!

gcanlin commented Oct 13, 2026

Uh oh!

mergify Bot commented Oct 13, 2026

Uh oh!

mergify Bot commented Oct 28, 2026

Uh oh!

Uh oh!

gau-nernst left a comment

Uh oh!

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

gcanlin commented Nov 4, 2026

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

gcanlin commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

Details

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

gcanlin commented Oct 11, 2026

Uh oh!

JaredforReal commented Oct 13, 2026

Uh oh!

gcanlin commented Oct 13, 2026

Uh oh!

mergify Bot commented Oct 13, 2026

Uh oh!

mergify Bot commented Oct 28, 2026

Uh oh!

Uh oh!

gau-nernst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

gcanlin commented Nov 4, 2026

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gcanlin commented Oct 9, 2026 •

edited

Loading