[haosdent]

Overview

On SM100, fused_q now writes the DSA indexer weights directly in BF16, the dtype DeepGEMM's MQA logits kernels
take. This removes the FP32 to BF16 copy mqa_logits_mx_inputs launched on every indexer call.

Claims

• InferenceX AgentX, GLM-5.3-NVFP4 on 4x GB200: together with [Perf][Kernel] Size cooperative_topk's single pass by row length, not stride #62171 and [Perf][Kernel] Let cooperative_topk read the BF16 indexer logits #62172, decode time per turn
  drops 2.8%, 2.5%, 1.5% and 1.5% at concurrency 1, 4, 8 and 16 (main vs main: within 0.7%). This PR is
  about 1/6 of that.
• One fewer kernel per indexer call (prefill chunks and decode) on SM100/SM103 for DSA models using fused_q
  (DeepSeek-V3.2, GLM-5.x).
• No precision change. The SM100 logits kernels only take BF16 weights, so main already rounds them to BF16 with a
  separate .to(torch.bfloat16) in mqa_logits_mx_inputs. fused_q now
  does that rounding in its store, and the kernels get bit-identical weights.
• Other platforms are unchanged (FP32 weights).

Validation

• tests/kernels/test_fused_deepseek_v32_norm_rope.py on B300 (SM103): 214 passed, 8 skipped. The weight
  assertions now check BF16 output on SM100 and FP32 elsewhere.

• Bit-identity on B300: for 5.3M weights (Triton and CuTeDSL kernels, GLM-5.2 and DeepSeek-V3.2 shapes, 1 to 4096
  tokens), the BF16 output matches main's FP32 output followed by .to(torch.bfloat16) in every bit.

• InferenceX AgentX, GLM-5.3-NVFP4 on 4x GB200, main + [Perf] Emit the DSA indexer weights in BF16 on SM100 #62170, [Perf][Kernel] Size cooperative_topk's single pass by row length, not stride #62171 and [Perf][Kernel] Let cooperative_topk read the BF16 indexer logits #62172 vs main (median per-turn change
  on the same turns, 95% CI; setup below):

  concurrency	1	4	8	16
  decode time	-2.8% [-3.1, -2.6]	-2.5% [-3.1, -1.7]	-1.5% [-2.2, -1.1]	-1.5% [-2.4, +0.3]
  e2e latency	-3.3% [-3.9, -3.0]	-1.8% [-2.7, -0.9]	-1.2% [-2.0, -0.5]	-0.6% [-1.9, +2.4]
  KV cache tokens	+0.2%	+0.7%	+1.4%	0
  main vs main, decode	-0.1%	-0.7%	+0.3%	-0.3%

• Target verify forward (vllm bench forward, GLM-5.2-NVFP4, TP4, B300, MTP verify shapes, spread < 0.1%):
  -0.18% to -0.34% at 1, 3 and 7 requests, 25K and 94K context.

• Exactness: with real MTP verification on 32 GSM8K questions (greedy), the three changes together give the same
  outputs and the same acceptance length (3.4904) as main.

• MRCR (OpenAI, 8 needles, 594 samples), GLM-5.3-NVFP4 on 4x GB200, TP4, greedy, thinking off; main vs main +
  [Perf] Emit the DSA indexer weights in BF16 on SM100 #62170, [Perf][Kernel] Size cooperative_topk's single pass by row length, not stride #62171 and [Perf][Kernel] Let cooperative_topk read the BF16 indexer logits #62172. Difference is the paired per-sample mean with a 95% bootstrap CI:

  context	≤8K	≤16K	≤32K	≤64K	≤128K	≤256K	all
  main	0.925	0.904	0.934	0.898	0.904	0.852	0.903
  with the PRs	0.934	0.917	0.915	0.924	0.885	0.767	0.890
  difference	+0.009	+0.014	-0.019	+0.025	-0.018	-0.085	-0.013
  95% CI	[-0.029, +0.047]	[-0.035, +0.063]	[-0.068, +0.029]	[-0.033, +0.084]	[-0.085, +0.048]	[-0.174, +0.007]	[-0.038, +0.012]

  The 256K bin varies between runs, so it was run twice more for each:

  ≤256K	run 1	run 2	run 3
  main	0.852	0.732	0.732
  with the PRs	0.767	0.743	0.750

  Pooled over the three runs, the difference is -0.019 [-0.069, +0.031].

InferenceX AgentX setup and results for all GLM-5.3 PRs (4x GB200)

InferenceX c801c9f2d AgentX scenario (pinned AIPerf), 20-minute fast mode, GLM-5.3-NVFP4 rev e3b8f9f2, TP4, MTP 3
with synthetic acceptance 2.99, native KV offload 135 GB per rank, Rust frontend, max-num-seqs = 2 x
concurrency. vLLM main dcd0796 with its default backends and CUDA graph config, plus the PRs cumulatively in
the order below (FlashInfer with flashinfer-ai/flashinfer#6630, flashinfer-ai/flashinfer#6631 and
flashinfer-ai/flashinfer#6632 from #62174 on). One FlashInfer MoE autotune per concurrency, shared by all
variants. A fresh server per point, with the page cache dropped before each, and main run again at the end of
each concurrency as an A/A check. SGLang a916d91cd3 with IMEX, TRT-LLM
all-reduce fusion and --mem-fraction-static 0.90.

Each cell is the median, over the turns both runs finished (150 to 520), of the per-turn ratio to the previous
variant, with a 95% paired bootstrap CI. Decode is request latency minus TTFT. The bootstrap does not include
run-to-run variance; the main-vs-main row does.

decode time per turn	c=1	c=4	c=8	c=16
+ #62170, #62171, #62172	-2.8% [-3.1, -2.6]	-2.5% [-3.1, -1.7]	-1.5% [-2.2, -1.1]	-1.5% [-2.4, +0.3]
+ #62174	-1.6% [-1.8, -1.5]	-1.8% [-2.5, -0.9]	-3.3% [-3.9, -2.9]	-2.1% [-3.8, -0.9]
+ #62173	+0.5% (no-op here)	-0.3% [-0.5, -0.1]	-0.1% [-0.4, +0.3]	-0.9% [-2.0, +0.2]
all PRs vs main	-3.7% [-3.9, -3.5]	-4.3% [-4.8, -4.0]	-5.3% [-6.4, -4.8]	-4.8% [-6.5, -3.8]
main vs main (noise)	-0.1%	-0.7%	+0.3%	-0.3%

e2e latency per turn	c=1	c=4	c=8	c=16
+ #62170, #62171, #62172	-3.3%	-1.8%	-1.2%	-0.6%
+ #62174	-1.4%	-1.4%	-2.7%	-1.6%
+ #62173	+0.9%	-0.3%	-0.7%	-1.1%
all PRs vs main	-4.1%	-3.4%	-4.7%	-3.8%
main vs main (noise)	-0.6%	-1.2%	+0.4%	-0.4%

KV cache tokens per GPU (c=1 / 4 / 8 / 16): main 1,176,832 / 1,162,880 / 1,143,872 / 1,105,984; with all PRs
1,178,496 / 1,168,512 / 1,157,824 / 1,103,936. Window throughput (tok/s/GPU) moves up to 2.4% between runs in fast
mode, since it counts whichever long-context turns finish inside the 20 minutes; with all PRs it is +3.0%, -1.4%,
+4.8% and +1.2% at c=1/4/8/16.

vLLM vs SGLang, decode time per turn on the same turns (positive = vLLM slower):

	c=1	c=4	c=8	c=16
main	+10.4%	+12.7%	+13.7%	-2.9%
all PRs	+6.2%	+7.5%	+5.7%	-6.9%

#62170, #62171 and #62172 were measured as one group. vllm bench forward (target verify forward, spread < 0.1%)
splits their gain about 1/6, 1/2 and 1/3. #62173 changes only decode steps of 3, 5 or 7 requests; steps with 3 or
more requests are 0%, 4%, 13% and 37% of all steps at c=1/4/8/16. At c=1 both runs capture the same graphs, so
its +0.5% there is noise. At c=16, three more runs each of + #62174 and + #62173 (4 runs each, compared as
runs): decode time -0.54% [-0.93, -0.16], with every + #62173 run faster than every + #62174 run (permutation
p = 2/70); e2e latency -0.29% [-1.41, +0.82].

Details

mqa_logits_use_mx_scales() already decides that the SM100 logits kernels take BF16 weights, and
mqa_logits_mx_inputs returned weights.to(torch.bfloat16). That was a real copy kernel for every indexer call,
since fused_q (Triton and CuTeDSL) always wrote FP32. fused_q now allocates its weights output in the dtype the
logits kernels take, and both kernels cast on store. .to(torch.bfloat16) then returns the tensor as-is.

The CuTeDSL kernel's compile key gains the output dtype.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[LopezCastroRoberto]

Same comment as in #62172.

[haosdent]

Same answer as on #62172: main already rounds these weights to BF16 in mqa_logits_mx_inputs, since the SM100 logits kernels only take BF16. This PR just moves that rounding into fused_q. The output is bit-identical (5.3M weights checked, 0 mismatches), and the MRCR run in #62172 covers this PR too.