[ROCm][Perf] gfx942: use FlyDSL fp8 MQA logits kernel (ROCm/aiter#3913)

[akii96]

On gfx942, replace the vendored Triton fp8_mqa_logits with aiter.ops.flydsl.flydsl_fp8_mqa_logits from ROCm/aiter#3913. Drop-in replacement (identical args/semantics), gated behind _ON_GFX942.

gfx950 and other paths are untouched.

Results: GLM-5.2-FP8, 8× MI325X TP8

ISL=128K, OSL=1K, Conc=8

Metric	Baseline (Triton)	FlyDSL	Delta
Median TTFT (ms)	42,839	21,871	−49%
Median TPOT (ms)	109.11	68.76	−37%
Output tok/s	52.44	85.34	+63%

ISL=8K, OSL=1K, Conc=8

Metric	Baseline (Triton)	FlyDSL	Delta
Median TTFT (ms)	918	863	−6%
Median TPOT (ms)	19.79	19.68	−0.6%
Output tok/s	382.2	386.3	+1.1%

No regression at short context; gains scale with input length as seen above

Accuracy

Test	Filter	Result
GSM8K (5-shot, full 1319 samples)	flexible-extract	0.9416 ± 0.0065
	strict-match	0.9424 ± 0.0064
NIAH (128K tokens, 10 needles, 10 trials)	—	8/10 perfect, 2/10 miss one needle

Serve command

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export HSA_NO_SCRATCH_RECLAIM=1
vllm serve zai-org/GLM-5.2-FP8 --block-size 1 --compilation-config '{"pass_config": {"fuse_norm_quant": true, "fuse_act_quant": true}, "custom_ops": ["+rms_norm", "+silu_and_mul"]}' --disable-log-stats --enable-auto-tool-choice --no-enable-expert-parallel --gpu-memory-utilization 0.80 --hf-overrides '{"use_index_cache": true, "index_topk_freq": 8}' --host 0.0.0.0 --kv-cache-dtype fp8 --max-model-len 263168 --max-num-batched-tokens 16384 --max-num-seqs 128 --no-enable-prefix-caching --port 8000 --reasoning-parser glm45 --tensor-parallel-size 8 --tool-call-parser glm47 --trust-remote-code

Bench command

vllm bench serve --backend vllm --model zai-org/GLM-5.2-FP8 --host localhost --port 8000 --dataset-name random --random-input-len 131072 --random-output-len 1024 --max-concurrency 8 --num-prompts 20 --num-warmups 2 --seed 4 --ignore-eos --trust-remote-code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[tjtanaa]

@akii96 which aiter version is this validated on??

[tjtanaa]

As far as I know, I checked that our upstream v0.1.16.post5 aiter version does not contain the PR that you mentioned. So I will put this onhold.

[akii96]

I validated this on the nightly image from last Wednesday I believe and only cherry picked one vLLM PR #48886 which has the fix for the metadata issue.

The release notes are a bit of one thing pointing to another but maybe this helps:

The release notes of AITER v0.1.16.post5 mention it being on top of Post-release on top of the v0.1.16.post4 tag

So if we were to look at release notes of this AITER v0.1.16.post4

It mentions the aiter PR 3913

Thanks for looking into this! Hope this clears up the confusion about the release notes @tjtanaa

[akii96]

Hi @tjtanaa vLLM’s current docker/Dockerfile.rocm_base now pins AITER v0.1.19, and the FlyDSL FP8 MQA logits kernel used by this PR is present here -> https://github.com/ROCm/aiter/blob/v0.1.19/aiter/ops/flydsl/kernels/fp8_mqa_logits.py

I have not validated recently so if you need me to repeat against latest nightly happy to do so, just let me know! 😸

[akii96]

@tjtanaa validated the perf benefits of this PR on DSv4 Flash (averaged over 2 runs of baseline trition and flydsl arms). Can you please review this again now and see if we can get this merged ? Also benefits DSv4 Pro (not measured here) which got a weight refresh today!

Image: vllm/vllm-openai-rocm:nightly-3d204dfdaaf09d67d49c7855630ef949754e0f8f

Serving command

export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --no-enable-prefix-caching \
  --port 8000

Performance

ISL/OSL	Concurrency	Triton output tok/s	FlyDSL output tok/s	Change	Triton median TTFT	FlyDSL median TTFT	Change
8K/1K	4	153.33	152.72	-0.4%	726 ms	715 ms	-1.5%
8K/1K	8	282.81	283.12	+0.1%	732 ms	719 ms	-1.7%
64K/1K	4	103.92	108.49	+4.4%	6,767 ms	6,042 ms	-10.7%
64K/1K	8	150.88	160.85	+6.6%	6,764 ms	6,276 ms	-7.2%
128K/1K	4	67.15	75.48	+12.4%	16,262 ms	13,164 ms	-19.1%
128K/1K	8	84.43	98.49	+16.7%	16,148 ms	13,198 ms	-18.3%

The change is performance-neutral at 8K and provides increasing perf boost at larger ISL

GSM8K accuracy

• 1,319 questions
• 8-shot
• Temperature 0
• Seed 1234

Arm	Accuracy
Triton baseline	94.8%
FlyDSL	95.2%

[tjtanaa]

LGTM. Thanks for the evaluation and benchmark data.

[tjtanaa]

/ci run

[github-actions]

✅ @akii96, CI is now available for this PR.

• /ci run starts a CI build.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /ci cancel cancels scheduled or running CI builds for this PR branch.

[github-actions]

✅ Triggered Buildkite CI #84091 for commit df116bac6657.