[Kernel] Add KDA NVFP4 GEMM for Qwen3.x on SM120

[BBuf]

Summary

• Add python/sglang/kernels/kda_kernels as the home for kernels generated by kernel-design agents and register their provenance as KernelBackend.KDA.
• Move the existing KDA diffusion implementations and their JIT CUDA sources into that implementation home while keeping sglang.kernels.ops as the stable public facade.
• Import the Qwen3.x ModelOpt NVFP4 SM120 GEMM generated by the Humanize2 / Kernel Design Agents workflow from BBuf/KDA-Pilot#195 at 516c976cee824a236679adf6eb525275a0a9a120.
• Dispatch it from ModelOpt when the E2E-qualified SM120 shape allowlist matches; every other call falls back to FlashInfer.

This supersedes #36043, which has been closed.

Production dispatch contract

The Qwen3.5 decode fast path accepts M in {1, 2, 4, 8} for these exact (K, N) pairs:

Model / role	K	N
Qwen3.5-4B gate/up	2560	18432
Qwen3.5-4B down	9216	2560
Qwen3.5-9B gate/up	4096	24576
Qwen3.5-9B down	12288	4096

Qwen3.5 M=9 remains callable through the low-level API for testing but is not production-dispatched. Qwen3.8 keeps its deliberately narrow production path: only the DSpark-qualified down projection (M, K, N) = (9, 17408, 5120) dispatches to KDA. The captured Qwen3.8 M=1/M=9 decode and M=4369 prefill shapes remain available through the low-level API.

All paths also retain the exact dtype, layout, device, and SM120 capability gates. Unsupported Cartesian combinations fall back to FlashInfer.

Why Qwen3.8 remains narrow

The original isolated-GEMM policy persisted each layer's 5.6--11 MiB weight-scale tensor in L2. That helped repeated single-kernel benchmarks but displaced attention/SSM state in the complete serving pipeline. Enabling every captured Qwen3.8 decode shape preserved acceptance but regressed end-to-end output throughput by 0.76%.

Streaming both the one-pass FP4 weights and their scales, then dispatching only (9, 17408, 5120), changed the three-round Qwen3.8-27B DSpark mean from 128.35 to 129.60 output tok/s (+0.98%) with unchanged accept length and fixed-prompt output.

The Qwen3.5 paths are not inferred from that microbenchmark. They have their own model-specific shape allowlist, CUDA Graph correctness coverage, and end-to-end concurrency sweep below.

Qwen3.5 SM120 validation

Hardware and software:

• NVIDIA RTX PRO 6000 Blackwell Server Edition, SM120, 96 GB
• PyTorch 2.13.0+cu130
• flashinfer-python==0.6.18; stale 0.6.17 cubin/JIT-cache packages were removed rather than bypassing FlashInfer's package-version guard
• AxionML/Qwen3.5-4B-NVFP4 at 4521f321dc8c46d255929203ae6d3062e51d52fa
• AxionML/Qwen3.5-9B-NVFP4 at 97aef92393f126bf649f310cd40861be8dad3279

GPU correctness and CUDA Graphs

GPU correctness on SM120 was checked against FlashInfer for Qwen3.5 M=1/2/4/8/9 decode shapes, Qwen3.8 decode/prefill, CUDA Graph replay, ModelOpt dispatch, and unsupported-shape fallback.

The multi-weight benchmark rotates eight distinct weights inside a CUDA Graph, alternates FlashInfer/KDA timing order across five trials, and uses the median per-call latency. All 20 Qwen3.5 rows passed rtol=1e-2, atol=2e-2.

Model shape	M	Gate/up speedup	Down speedup
Qwen3.5-4B	1	1.052x	2.023x
Qwen3.5-4B	2	1.031x	2.024x
Qwen3.5-4B	4	1.055x	2.026x
Qwen3.5-4B	8	1.052x	2.047x
Qwen3.5-4B	9	1.027x	2.039x
Qwen3.5-9B	1	1.135x	1.258x
Qwen3.5-9B	2	1.134x	1.259x
Qwen3.5-9B	4	1.139x	1.251x
Qwen3.5-9B	8	1.132x	1.252x
Qwen3.5-9B	9	1.124x	1.262x

The geometric-mean speedup is 1.319x across the 16 production-dispatched M=1/2/4/8 rows and 1.318x across all 20 rows.

End-to-end concurrency sweep

Protocol: 32 fixed-seed random-ids requests per round, 2048 input tokens, 512 output tokens, concurrency 1/2/4/8, cache flush before every round, three baseline rounds, three KDA rounds, and one adjacent baseline round. Every formal round completed all 32 requests (65,536 input and 16,384 output tokens).

Model	Concurrency	Baseline tok/s	KDA tok/s	Adjacent baseline	Throughput	TPOT	E2E latency
Qwen3.5-4B	1	266.218	289.452	265.647	+8.73%	+8.31%	+8.03%
Qwen3.5-4B	2	522.936	558.694	522.736	+6.84%	+6.85%	+6.41%
Qwen3.5-4B	4	960.507	1028.929	962.763	+7.12%	+7.02%	+6.65%
Qwen3.5-4B	8	1427.207	1520.322	1427.246	+6.52%	+6.63%	+6.10%
Qwen3.5-9B	1	184.624	190.493	184.476	+3.18%	+3.19%	+3.08%
Qwen3.5-9B	2	359.919	369.908	360.482	+2.78%	+2.82%	+2.71%
Qwen3.5-9B	4	678.651	697.802	680.757	+2.82%	+2.81%	+2.74%
Qwen3.5-9B	8	1022.864	1050.453	1024.721	+2.70%	+2.78%	+2.62%

Adjacent-baseline throughput stays within 0.32% of the original baseline mean. TTFT changes range from -1.93% to +3.57%, while TPOT and total E2E latency improve at every concurrency, consistent with a decode-focused kernel.

Validation summary

• Fused-op dispatch tests: 56/56 passed
• LayerNorm fused-op backend tests: 20 passed, 1 environment skip
• Kernel namespace tests: 61/61 passed
• All pre-commit hooks passed, including Python AST, isort, Ruff, codespell, and registered-test CI checks

KDA ownership and per-kernel provenance are documented in python/sglang/kernels/kda_kernels/README.md.

Related benchmark protocol: #34934.

---

CI States

Latest PR Test (Base): 🚫 Run #33539263034
Latest PR Test (Extra): ❌ Run #33539262840
Latest PR Test (AMD ROCm 7.2): ⏳ Run #33539263039

[BBuf]

/tag-and-rerun-ci