[TimRen01]

Overview

GLM-5.3-Flash decode TPOT -12% at concurrency 1 on 2x DGX Spark (SM121, TP=2). On nvidia/GLM-5.3-Flash-NVFP4 the attention projections (KDA and MLA), shared experts and LM head stay BF16 and take roughly 46 of a 68 ms decode step on cuBLAS gemv. glm5next has no low-latency GEMM module yet, so this adds one modelled on qwen4_exp/nvidia/low_latency_gemm.py (#59753, #59632, #61437) with measured SM121 plans for the TP=2 local shapes, installed from Glm5NextForCausalLM and Glm5NextMTP.

Claims

• 7 local (N, K) shapes at M = 1/2/4 (20 entries), routed only on capability (12, 1); other GPUs, dtypes, token counts and shapes keep the standard linear path.
• Kernel: 1.35x to 1.74x over cuBLAS gemv at M=1 (238-283 GB/s vs 162-176 GB/s), 1.04x to 1.36x at M=2/4.
• End to end, no speculation, concurrency 1: TPOT 64.9 -> 57.0 ms (-12%), 15.0 -> 17.0 tok/s. With MTP k=3 the target forward runs at M=4, where cuBLAS already uses a GEMM kernel: ITL -2.5%, TPOT and tok/s unchanged.
• GSM8K (n=100): 93 base, 94 plans.

Validation

Probe through the module path on GB10 (CUDA graph replay, weight rotated out of L2, nightly 8e2beaa), cosine 1.000000 vs F.linear for every entry:

local (N, K)	layer	calls/step	M=1 cuBLAS us	M=1 plan us	M=1	M=2	M=4
(12576, 4096)	KDA in_proj	34	608	429	1.42x	1.07x	1.05x
(4096, 4096)	KDA o_proj	34	204	137	1.48x	1.08x	1.06x
(2048, 4096)	shared gate_up, MLA qkv_a	53	103	64	1.61x	1.22x	1.24x
(4096, 1024)	shared down	42	51	30	1.74x	1.34x	1.36x
(8192, 1536)	MLA q_b	11	147	102	1.45x	1.13x	1.20x
(4096, 8192)	MLA o_proj	11	291	cuBLAS	1.01x (not listed)	1.04x	1.07x
(77440, 4096)	lm_head	1	3606	2663	1.35x	1.05x	1.06x

These are cold-L2 numbers, which is the serving case: decode reads each weight once per step and 45 layers do not fit the 24 MB L2. The M=1 entries add up to 12.8 ms per step; the server gains 8 ms because the GEMMs interleave with MoE and attention. M=8/16 are within 2% of cuBLAS and are not listed.

End to end, nvidia/GLM-5.3-Flash-NVFP4 TP=2 on two Sparks, FP8 KV, SPEED-Bench 2K in / 1K out, 4 requests at concurrency 1, two interleaved runs per arm, nightly 8e2beaa:

Arm	TPOT p50 (ms)	ITL p50 (ms)	Output tok/s	GSM8K (n=100)
base	64.83 / 64.98	64.53 / 64.64	15.06 / 14.87	93
plans	57.85 / 56.16 (-12%)	58.88 / 55.20	16.77 / 17.30	94

With MTP k=3 (--speculative-config '{"method":"mtp","num_speculative_tokens":3}', 2026-10-17 nightly):

Arm	ITL p50 (ms)	TPOT p50 (ms)	Output tok/s	Acceptance length
base	102.18 / 102.73	40.93 / 41.52	24.21 / 23.48	2.60 / 2.52
plans	99.56 / 100.27 (-2.5%)	41.76 / 42.01	23.85 / 23.71	2.48 / 2.48

An nsys census of the plans arm shows every listed shape on CuteSkinnyGemm in the server (45% of GPU time without speculation, 39.5% with MTP); what is left on cuBLAS is KDA f_b/g_b (4096, 128) (#57617), the fp32-output router gate (288, 4096) and MLA o_proj at M=1.

Tests: pytest tests/kernels/test_bf16_skinny_gemm.py -k glm5next, 26 passed on GB10 (table validity, capability routing, hook installation, 20 selected shapes with cosine > 0.999). The selected-shape test skips off capability (12, 1), which CI does not have.

Reproduction

# on each DGX Spark, --node-rank 0 / 1
vllm serve nvidia/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 2 --nnodes 2 --node-rank 0 \
  --master-addr 192.168.100.1 --master-port 29500 --distributed-executor-backend mp \
  --gpu-memory-utilization 0.84 --max-model-len 32768 --max-num-seqs 16 \
  --max-num-batched-tokens 8192 --enable-prefix-caching --disable-custom-all-reduce \
  --no-enable-flashinfer-autotune --reasoning-parser glm45 --kv-cache-dtype fp8
vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \
  --dataset-name speed_bench --speed-bench-dataset-subset throughput_2k \
  --speed-bench-output-len 1024 --ignore-eos --num-warmups 2 --num-prompts 4 --max-concurrency 1 --seed 0
pytest tests/kernels/test_bf16_skinny_gemm.py -k glm5next

Details

• Plans come from a CUDA-graph sweep over block_size, outputs_per_block, k_unroll, vector_width and static_k with the weight rotated out of L2, keeping entries that beat cuBLAS by more than 5%. (2048, 4096) serves both the shared-expert gate/up and the MLA fused q_a/kv_a projection.
• The replacement method follows the quant method the layer carries: a quant config that excludes lm_head gives ParallelLMHead an UnquantizedLinearMethod rather than an UnquantizedEmbeddingMethod, and the head stayed on cuBLAS until this was handled.
• Not covered: TP=1 (the model does not fit one Spark), KDA f_b/g_b (4096, 128) ([Perf][GLM-5.3-Flash] Batch paired KDA gate projections (f_b_proj/g_b_proj) into one bmm #57617), MTP eh_proj (a plain nn.Linear), and M=3, which uses cuBLAS.
• AI assistance (Claude Code) was used for the sweep scripts, the module and the tests; all code and measurements were reviewed and run by the author.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.