[ROCm][fusion] add dual RMSNorm quant fusion pattern

[afriedri]

Purpose

Extends #49621 so that double RMSNorm fp8 quantization fusion works on group-sized quant. Distinct from open #44437 (MXFP4) and #45798 (RoPE-gate).

Validation

• Perf:
  becomes
• Accuracy (GSM8K, zero-shot, temp=0, max_gen_toks=32768, flexible-extract): baseline (fusion off) 0.8984 ±0.0083 vs fused 0.8946 ±0.0085 (Δ 0.0038, within noise). strict-match is 0.0 for both — expected, R1 wraps answers in <think> blocks, so flexible-extract is the correct metric.
• Unit tests: tests/compile/passes/test_fuse_mla_dual_rms_norm.py — 3 passed (plain, per-token, and new group-quant), on ROCm/AITER with VLLM_ROCM_USE_AITER=1.

End-to-end serving

DeepSeek-R1, TP8, MI350X (gfx950), ROCm + AITER (VLLM_ROCM_USE_AITER=1). Both
servers built from the same tree; the only difference is the
fuse_mla_dual_rms_norm pass. Each config ran a 3-prompt warmup that was
discarded, then the measured run below.

# fused (default)
vllm serve deepseek-ai/DeepSeek-R1 --tensor-parallel-size 8 --max-model-len 8192

# unfused baseline
vllm serve deepseek-ai/DeepSeek-R1 --tensor-parallel-size 8 --max-model-len 8192 \
  --compilation-config '{"pass_config":{"fuse_mla_dual_rms_norm":false}}'

vllm bench serve --backend vllm --model deepseek-ai/DeepSeek-R1 \
  --dataset-name random --random-input-len 1024 --random-output-len 1024 \
  --num-prompts 10 --max-concurrency 1 --ignore-eos \
  --percentile-metrics tpot,ttft,itl,e2el

Metric	Fused	Unfused	Δ
Mean TPOT (ms)	9.63	9.72	−0.93%
Median TPOT (ms)	9.63	9.72	−0.93%
P99 TPOT (ms)	9.64	9.73	−0.93%
Mean ITL (ms)	9.63	9.72	−0.93%
Mean E2EL (ms)	9965.8	10061.7	−0.95%
Output throughput (tok/s)	102.75	101.77	+0.96%
Mean TTFT (ms)	114.5	116.2	−1.5%

Update: rebased onto #53540 (2026-08-27)

#53540 landed the fused_mla_dual_rms_norm_group_quant custom op while this was in review, so the op registration this PR used to carry is dropped — it now calls upstream's. Keep compile pass.

vllm/_aiter_ops.py goes from +77 to +4. Total +354 → +284.

Re-validated against upstream's op

Unit tests: 3/3 pass (tests/compile/passes/test_fuse_mla_dual_rms_norm.py), including a bit-exactness check on the q latent vs the unfused reference.

Pass fires — refs in the AOT-compiled model, per rank:

	fused	fuse_mla_dual_rms_norm=false
fused_mla_dual_rms_norm_group_quant	93	9
rocm_aiter_rmsnorm_fp8_group_quant	17	83

---

Prepared with the help of AI (Claude). All changes looked over + verified by Andy.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[dllehr-amd]

Lets just see about 64 group size and it's efficacy here

[dllehr-amd]

Can you double check that group size 64 is used, and this pattern correctly runs it? The UT you provided is for 128 only

[afriedri]

All producers hard code 128. Removed 64

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @afriedri.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[dllehr-amd]

Thanks @afriedri looks good now

[github-actions]

✅ @afriedri, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[afriedri]

/ci run

[github-actions]

✅ Triggered Buildkite CI #87966 for commit 864384e532dd.