[haosdent]

Overview

GLM-5.x (the DeepseekV32 NVIDIA path) now leaves each decode MoE layer's top-k reduction open. The next RMSNorm's
fused all-reduce then finalizes it in one FlashInfer kernel (trtllm_moe_finalize_allreduce_fusion). This uses
the existing deferred-finalize contract (MoEOutput, defer_moe_finalize) that DeepSeek-V4.1 and Kimi-K3 already
use.

Needs flashinfer-ai/flashinfer#6630 (deadlock fix) and flashinfer-ai/flashinfer#6631 (speed); draft until a
FlashInfer release with both is pinned. Without the first the kernel can hang under back-to-back launches;
without the second, deferring is slower than the unfused path. flashinfer-ai/flashinfer#6632 (cluster sizing)
adds most of the gain at 5-7 running requests.

Claims

• InferenceX AgentX, GLM-5.3-NVFP4 on 4x GB200, with the FlashInfer PRs: decode time per turn drops 1.6%, 1.8%,
  3.3% and 2.1% at concurrency 1, 4, 8 and 16 (main vs main: within 0.7%), and e2e latency 1.4% to 2.7%.
• Per MoE layer, the GEMM2 tail is one kernel instead of three: finalize, shared + routed add, and all-reduce +
  residual + RMSNorm.
• KV cache capacity stays within 192 tokens of the baseline, and GSM8K accuracy is unchanged within run-to-run
  noise. The fused kernel accumulates in FP32.

Validation

• New tests/kernels/core/test_fused_allreduce_rms_norm.py: a MoEOutput holding an UnfinalizedMoEOutput
  through fused_allreduce_rms_norm must match finalize, shared add, then RMSNorm(all_reduce(.)), on TP 1, 2 and
  4, with 1, 4 and 64 tokens, hidden 2048 and 6144, on both the fused kernel (checking it was called) and the
  fallback. With the FlashInfer fixes, on GB200 and B300: 30 passed, 6 skipped (TP=1 has no fused path).

• InferenceX AgentX, GLM-5.3-NVFP4 on 4x GB200, this PR on top of [Perf] Emit the DSA indexer weights in BF16 on SM100 #62170, [Perf][Kernel] Size cooperative_topk's single pass by row length, not stride #62171 and [Perf][Kernel] Let cooperative_topk read the BF16 indexer logits #62172 (median per-turn change
  on the same turns, 95% CI; setup below):

  concurrency	1	4	8	16
  decode time	-1.6% [-1.8, -1.5]	-1.8% [-2.5, -0.9]	-3.3% [-3.9, -2.9]	-2.1% [-3.8, -0.9]
  e2e latency	-1.4% [-1.7, -1.3]	-1.4% [-2.0, +0.1]	-2.7% [-3.2, -2.2]	-1.6% [-2.9, -0.6]
  KV cache tokens	-192	-64	0	0
  main vs main, decode	-0.1%	-0.7%	+0.3%	-0.3%

• Target verify forward (vllm bench forward, GLM-5.2-NVFP4, TP4, B300, spread < 0.1%), us. "fix only" is FlashInfer
  with fix(comm): poll the local Lamport buffer in the MoE-finalize all-reduce flashinfer-ai/flashinfer#6630, "fix + perf" adds perf(comm): gather MoE-finalize routes together and accumulate in FP32 flashinfer-ai/flashinfer#6631:

  requests x 4 tokens @ KV	no deferral	deferral, fix only	deferral, fix + perf
  1 @25k	8570.1	8618.4 (+0.56%)	8464.4 (-1.23%)
  3 @25k	12075.2	12128.4 (+0.44%)	11891.7 (-1.52%)
  7 @25k	17472.5	17849.5 (+2.16%)	17337.3 (-0.77%)
  3 @94k	12267.1	12318.9 (+0.42%)	12081.3 (-1.51%)

  In an nsys trace of the batch-1 verify graph, finalizeKernel (297 us per step) and most of the shared + routed
  add (217 -> 70 us) are gone; finalize + all-reduce + norm goes from 1438 to 1265 us.

• GSM8K (lm_eval, 5-shot, 1319 questions, real MTP verification, concurrency 32), flexible / strict, two runs each:
  main 0.9439 / 0.9424 both times; with this PR 0.9401 / 0.9401 and 0.9447 / 0.9424 (stderr 0.0063). Greedy outputs
  differ from main because the finalize now rounds once instead of after each step.

InferenceX AgentX setup and results for all GLM-5.3 PRs (4x GB200)

InferenceX c801c9f2d AgentX scenario (pinned AIPerf), 20-minute fast mode, GLM-5.3-NVFP4 rev e3b8f9f2, TP4, MTP 3
with synthetic acceptance 2.99, native KV offload 135 GB per rank, Rust frontend, max-num-seqs = 2 x
concurrency. vLLM main dcd0796 with its default backends and CUDA graph config, plus the PRs cumulatively in
the order below (FlashInfer with flashinfer-ai/flashinfer#6630, flashinfer-ai/flashinfer#6631 and
flashinfer-ai/flashinfer#6632 from #62174 on). One FlashInfer MoE autotune per concurrency, shared by all
variants. A fresh server per point, with the page cache dropped before each, and main run again at the end of
each concurrency as an A/A check. SGLang a916d91cd3 with IMEX, TRT-LLM
all-reduce fusion and --mem-fraction-static 0.90.

Each cell is the median, over the turns both runs finished (150 to 520), of the per-turn ratio to the previous
variant, with a 95% paired bootstrap CI. Decode is request latency minus TTFT. The bootstrap does not include
run-to-run variance; the main-vs-main row does.

decode time per turn	c=1	c=4	c=8	c=16
+ #62170, #62171, #62172	-2.8% [-3.1, -2.6]	-2.5% [-3.1, -1.7]	-1.5% [-2.2, -1.1]	-1.5% [-2.4, +0.3]
+ #62174	-1.6% [-1.8, -1.5]	-1.8% [-2.5, -0.9]	-3.3% [-3.9, -2.9]	-2.1% [-3.8, -0.9]
+ #62173	+0.5% (no-op here)	-0.3% [-0.5, -0.1]	-0.1% [-0.4, +0.3]	-0.9% [-2.0, +0.2]
all PRs vs main	-3.7% [-3.9, -3.5]	-4.3% [-4.8, -4.0]	-5.3% [-6.4, -4.8]	-4.8% [-6.5, -3.8]
main vs main (noise)	-0.1%	-0.7%	+0.3%	-0.3%

e2e latency per turn	c=1	c=4	c=8	c=16
+ #62170, #62171, #62172	-3.3%	-1.8%	-1.2%	-0.6%
+ #62174	-1.4%	-1.4%	-2.7%	-1.6%
+ #62173	+0.9%	-0.3%	-0.7%	-1.1%
all PRs vs main	-4.1%	-3.4%	-4.7%	-3.8%
main vs main (noise)	-0.6%	-1.2%	+0.4%	-0.4%

KV cache tokens per GPU (c=1 / 4 / 8 / 16): main 1,176,832 / 1,162,880 / 1,143,872 / 1,105,984; with all PRs
1,178,496 / 1,168,512 / 1,157,824 / 1,103,936. Window throughput (tok/s/GPU) moves up to 2.4% between runs in fast
mode, since it counts whichever long-context turns finish inside the 20 minutes; with all PRs it is +3.0%, -1.4%,
+4.8% and +1.2% at c=1/4/8/16.

vLLM vs SGLang, decode time per turn on the same turns (positive = vLLM slower):

	c=1	c=4	c=8	c=16
main	+10.4%	+12.7%	+13.7%	-2.9%
all PRs	+6.2%	+7.5%	+5.7%	-6.9%

#62170, #62171 and #62172 were measured as one group. vllm bench forward (target verify forward, spread < 0.1%)
splits their gain about 1/6, 1/2 and 1/3. #62173 changes only decode steps of 3, 5 or 7 requests; steps with 3 or
more requests are 0%, 4%, 13% and 37% of all steps at c=1/4/8/16. At c=1 both runs capture the same graphs, so
its +0.5% there is noise. At c=16, three more runs each of + #62174 and + #62173 (4 runs each, compared as
runs): decode time -0.54% [-0.93, -0.16], with every + #62173 run faster than every + #62174 run (permutation
p = 2/70); e2e latency -0.29% [-1.41, +0.82].

Details

• fused_allreduce_rms_norm accepts a MoEOutput. If the routed part is unfinalized, and TP > 1 with the TRT-LLM
  workspace in use, it calls trtllm_moe_finalize_allreduce_fusion, which does the top-k sum, routed scaling,
  shared add, all-reduce, residual add and RMSNorm. Called without that kernel available (only possible from
  outside these layers, for example the unit test), it finalizes in torch and takes the normal path.
• moe_finalize_allreduce_max_num_tokens bounds deferral by the kernel's one-shot limit (128 tokens) and by the
  size up to which vLLM already picks the one-shot all-reduce. It also creates (or reuses) the shared TRT-LLM
  all-reduce workspace and returns 0 if that fails, so a layer only defers when the fused kernel will run. Above
  the bound, or when the backend isn't trtllm, the layer finalizes as before.
• A decoder layer defers only when its experts are TRT-LLM-Gen (NVFP4 or BF16 monolithic/modular, the ones that
  can stop after GEMM2), it isn't sequence parallel, and the router folds routed_scaling_factor into the routing
  weights (the deferred output carries no scaling of its own, so the experts' factor must be 1.0). Each
  local layer defers except the last one on a non-last PP rank, since its output leaves as IntermediateTensors.
  Layers that feed an aux hidden state (EAGLE-3) don't defer. The MTP block defers into its final norm.
• Once the fused kernel is launched, the consumer frees the deferred buffers. The model's layer loop still holds
  the MoEOutput until the consuming layer returns, and keeping the padded GEMM2 rows alive through that layer's
  attention split the indexer's logits block in the CUDA graph pool: one extra segment per graph, which cost
  1.0% of the KV cache with max_num_seqs 32.
• _forward_impl and _unpack are used directly, as in deepseek_v41/nvidia/model.py, because the runner's
  custom op returns tensors only.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.