[ROCm][Perf][GLM-5] Fused decode with the MonoKernel in vLLM

[amd-sriram]

Overview

Same GLM-5 fused decode as vllm-project#60216, with the kernel carried in vLLM instead of AITER. This is an alternative to vllm-project#60216 + ROCm/aiter#6173, opened so the two layouts can be tested side by side before choosing one. It follows the layout of the other mono decode PRs, vllm-project#59705 (MiniMax-M3) and vllm-project#60397 (DeepSeek-V4.1).

What changes against vllm-project#60216

• vllm/models/deepseek_v32/amd/mono/: the GLM-5 MonoKernel from [FlyDSL] GLM-5 fused decode-layer MonoKernel ROCm/aiter#6173 at b02df0db8 (device code from [Kernel][Perf] Add GLM-5 and Kimi-K3 MonoKernels ROCm/FlyDSL#1204, host wrapper from [Perf] Add GLM-5.2 and Kimi-K3 small-concurrency MonoKernels ROCm/ATOM#2435, plus the paged indexer, dense-layer mode and fixes from [Bug]: As V100 does not support FlashAttention, it is not possible to run the gemma model, hopefully it can support the xformers way to run it vllm-project/vllm#6173). Each file keeps its origin and license in the header. Import paths are the only code change.
• mono_decode.py imports the kernel from vllm.models.deepseek_v32.amd.mono instead of aiter.ops.flydsl.glm5_mono.
• tests/models/glm_mono/: the three op tests from [Bug]: As V100 does not support FlashAttention, it is not possible to run the gemma model, hopefully it can support the xformers way to run it vllm-project/vllm#6173 (MoE layer, dense layer, paged indexer). Each has a pytest entry that runs at TP4 and skips unless four gfx950 GPUs are present.
• Remaining AITER dependencies: buffer_ops, dpp_utils and the MXFP4 weight shuffle, which the ROCm path already installs. FlyDSL comes from the flydsl package.
• Lint: vLLM's ruff settings, with file-level noqa for long lines in the adapted files and for B023 in the kernel, whose closures run while each loop iteration is traced.

Validation

MI355X, GLM-5.3 MXFP4, one build: vLLM 43b4aaea with vllm-project#60216, AITER 4bb520ba with vllm-project#6173. The in-tree arm copies this branch's mono/ and mono_decode.py over that build and deletes AITER's glm5_mono, so nothing can fall back to it.

Op tests: pytest tests/models/glm_mono passes at TP4 (3 passed), and the three tests pass at TP8 from the command line. Dense layer error against the quantized reference is 0.06 to 0.12%; the paged indexer picks are identical to the reference.

End to end at TP4, two servers at once on the same node (GPUs 0-3 and 4-7):

	in-tree kernel	AITER kernel
NaN probe, 1/2/4/8 concurrent, up to 61K context	all ok	all ok
NIAH 8K / 32K / 60K	15/15	15/15
GSM8K strict, 250 questions	0.916	0.880
TPOT concurrency 1 (8K in / 256 out)	6.26 ms	6.26 ms
TPOT concurrency 4	11.38 ms	11.43 ms

The kernel code is the same in both arms, so the GSM8K gap is run-to-run noise (about 2 standard errors at 250 questions; the earlier TP4 run of vllm-project#60216 scored 0.904).

The kernel's own validation (GPQA, TP8 end to end, TPOT against the unfused path) is in vllm-project#60216 and ROCm/aiter#6173.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[amd-sriram]

Moved upstream as vllm-project#60831.