[ROCm][Perf] Add bpreshuffled blockscaled fp8 GEMM by simondanielsson · Pull Request #51692 · vllm-project/vllm

simondanielsson · 2026-08-10T14:54:38Z

Purpose

Activated when shapes allow for it and configs are tuned.

Implications (DSv3 1k/1K):

TP8+DPA: +4-8% QPS
TP8+EP: +0-4% QPS

Test Plan

Bench serve & accuracy validation with DSv3 on (1) TP8+DPA (2) TP8+EP on 8xMI350.

(note need to run with VLLM_ROCM_USE_AITER_FP8BMM=0 until #51957 is resolved)

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_FP8BMM=0 \
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --data-parallel-size 8 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.8 \

Sweep concurrencies 1-128, 1k/1k

vllm bench serve \
    --backend vllm \
    --model deepseek-ai/DeepSeek-V3-0324 \
    --dataset-name random \
    --random-input-len 1024 \
    --random-output-len 1024 \
    --host localhost \
    --port 8000 \
    --ignore-eos \
    --ready-check-timeout-sec 6000 \
    --seed 5678 \
    --max-concurrency 128 \
    --num-prompts 1280 --num-warmups 256'

pip install --quiet 'lm_eval[api]' && \
  lm_eval \
    --model local-completions \
    --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \
    --tasks gsm8k \
    --num_fewshot 5 \
    --output_path /tmp/lm_eval_gsm8k

Test Result

Profiling

This branch: Using DPA, the MLA decode o-proj is using preshuffled weights. No other GEMM is changed. 21us in trace.

Main branch: ck-tile kernel for o-proj. 52us in trace, so more than 2x slower that of the bpreshuffled version!

TP8+DPA

Concurrency	Branch	TTFT (P50 ms)	TPOT (P50 ms)	QPS	QPS change (%)
1	This branch	156.9816	18.2976	0.0529	+7.30
1	Nightly	158.3445	19.7720	0.0493
2	This branch	161.0075	18.5311	0.1049	+8.14
2	Nightly	198.6506	19.9552	0.0970
4	This branch	220.7625	18.7688	0.2054	+7.48
4	Nightly	284.2663	20.1512	0.1911
8	This branch	557.7459	22.6129	0.3479	+7.61
8	Nightly	317.1527	23.8614	0.3233
16	This branch	581.6531	23.5224	0.6528	+5.96
16	Nightly	609.4239	24.8873	0.6161
32	This branch	767.6995	23.7180	1.2794	+4.61
32	Nightly	787.4866	24.8746	1.2230
64	This branch	1449.0858	24.3342	2.4326	+5.50
64	Nightly	1394.4616	25.7091	2.3057
128	This branch	2018.3187	25.9321	4.4837	+4.90
128	Nightly	2201.5545	27.2313	4.2742

TP8+EP

Concurrency	Branch	TTFT (P50 ms)	TPOT (P50 ms)	QPS	QPS change (%)
1	This branch	108.3783	12.6642	0.0765	-0.39
1	Nightly	113.4111	12.6244	0.0768
2	This branch	172.7154	13.6376	0.1416	+0.14
2	Nightly	178.4821	13.6537	0.1414
4	This branch	280.6493	13.3944	0.2861	+0.42
4	Nightly	287.1797	13.446	0.2849
8	This branch	424.7564	13.5723	0.5592	+0.81
8	Nightly	427.5746	13.6773	0.5547
16	This branch	479.1289	14.4931	1.0434	+0.69
16	Nightly	484.925	14.6155	1.0363
32	This branch	817.9601	15.6236	1.9015	+3.95
32	Nightly	821.0856	15.8782	1.8292
64	This branch	938.6096	21.6289	2.7676	+9.01
64	Nightly	932.3192	23.9109	2.5388
128	This branch	1064.6245	27.3408	4.4065	+4.04
128	Nightly	1079.7563	28.087	4.2355

GSM8k

This branch (TP8+DPA)

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9477	±	0.0061
		strict-match	5	exact_match	↑	0.9484	±	0.006

Nightly

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9439	±	0.0063
		strict-match	5	exact_match	↑	0.9431	±	0.0064

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

…oproj-bpreshuffle Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

simondanielsson · 2026-08-18T15:33:13Z

@Rohan138 Happy to get a re-review on this one whenever you have time, thanks 🙏

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

…lm into orchestrator/consolidate-gemm-loader Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

BadrBasowid

Small nit

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

tjtanaa

LGTM.

github-actions · 2026-09-08T08:49:26Z

✅ @simondanielsson, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

AndreasKaratzas · 2026-09-08T10:51:20Z

@simondanielsson I cancelled this build cause like the other 2 it creates a huge workload for our CI. When you relaunch CI, please wait for each PR to finish and then launch CI for you next PR.

simondanielsson · 2026-09-09T12:38:50Z

/ci run

github-actions · 2026-09-09T12:39:09Z

✅ Triggered Buildkite CI #87912 for commit fd1f9294da68.

simondanielsson · 2026-09-09T14:23:09Z

@AndreasKaratzas Same on this PR. Only failing tests are the same as in nightly. Can you check please? 🙏 Thanks

AndreasKaratzas · 2026-09-09T17:09:40Z

Indeed they are from nightly.

simondanielsson · 2026-09-10T13:01:16Z

/ci run

github-actions · 2026-09-10T13:01:55Z

✅ Triggered Buildkite CI #88123 for commit dacd2d15334e.

simondanielsson · 2026-09-10T15:47:42Z

/ci retry

github-actions · 2026-09-10T15:48:01Z

✅ Queued 8 failed job(s) for retry in Buildkite CI #88123.

mawong-amd · 2026-09-11T06:29:52Z

Was this PR tested on DeepSeek V3/V4 TP8? I'm seeing garbage output on the former, and it refuses to initialize on the latter (it hits the not implemented apply_block_scaled_mm).
Note: Testing on main now is slightly complicated because #56228 changed the signature of the sparse-attention indexer call (a separate breakage), but the results stand after patching that.

simondanielsson · 2026-09-11T07:36:39Z

Was this PR tested on DeepSeek V3/V4 TP8? I'm seeing garbage output on the former, and it refuses to initialize on the latter (it hits the not implemented apply_block_scaled_mm). Note: Testing on main now is slightly complicated because #56228 changed the signature of the sparse-attention indexer call (a separate breakage), but the results stand after patching that.

@mawong-amd Hi, yes tested on V3 TP8 (note: vanilla dense v3). Let me also test again

… N % 128 gate Follow-up to vllm-project#51692 / vllm-project#56433 (AITER preshuffled fp8 block-scale GEMM): * Fusion passes: register transposed-scale variants of the AITER RMSNorm(+add)+group-quant, AllReduce+RMSNorm+group-quant and SiLU*mul+ group-quant patterns, so the quant the preshuffled kernel emits (rocm_aiter_group_fp8_quant(x, 128, transpose_scale=True)) is matched and folded into the fused producer op instead of being left as a separate kernel behind a bf16 norm. MatcherQuantFP8 carries the flag. * Producers write the kernel's column-major scale layout directly: fused_allreduce_rmsnorm_quant_per_group via AITER's transpose_scale (re-viewed [G, M] -> [M, G], no copy), fused_rms_fp8_group_quant via its transpose_scale, act_mul via the HIP silu_and_mul_quant(shuffle_scale) kernel. No transpose copies remain. * AiterPreshuffledFp8BlockScaledMMKernel.can_implement: require N % 16 (the (16,16) preshuffle) and K % 128 instead of N % 128. DeepSeek's fused_qkv_a_proj (N = 2112) was falling back to the Triton GEMM (134.7 vs 44.8 us per layer at M = 1024 on MI355X). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

simondanielsson added 4 commits August 10, 2026 16:18

feat: first version of bpreshuffle gemm implementation

7895241

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

Merge remote-tracking branch 'upstream/main' into orchestrator/aiter-…

749d421

…oproj-bpreshuffle Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

fix: simplify

960def5

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

fix: remove envvar gate

89115b3

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

mergify Bot added the rocm Related to AMD ROCm label Aug 10, 2026

github-project-automation Bot added this to AMD Aug 10, 2026

github-project-automation Bot moved this to Todo in AMD Aug 10, 2026

simondanielsson added 6 commits August 11, 2026 14:45

fix: use aiter's group_fp8_quant(transpose_scale=True) to avoid segfault

579cb71

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

fix: remove redundant attribute assignment

f285864

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

fix: layout of A scales

af6f0ce

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

fix: only select bpreshuffle if tuned

80844c4

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

chore: improve comments

9d3508e

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

fix: remove redundant try-except

54f5a11

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

simondanielsson changed the title ~~[ROCm][Perf] Add bpreshuffled blockscaled GEMM~~ [ROCm][Perf] Add bpreshuffled blockscaled fp8 GEMM Aug 12, 2026

simondanielsson marked this pull request as ready for review August 12, 2026 14:31

simondanielsson requested review from AndreasKaratzas and tjtanaa as code owners August 12, 2026 14:31

claude Bot reviewed Aug 12, 2026

View reviewed changes

Rohan138 reviewed Aug 12, 2026

View reviewed changes

Comment thread vllm/_aiter_ops.py Outdated

Merge branch 'vllm-project:main' into feat/bpreshuffle-gemm

9acb005

simondanielsson requested a review from Rohan138 August 13, 2026 12:19

simondanielsson added 3 commits August 14, 2026 09:51

Merge branch 'main' into feat/bpreshuffle-gemm

29cd5cb

Merge branch 'main' into feat/bpreshuffle-gemm

0146384

Merge branch 'main' into feat/bpreshuffle-gemm

eedab7a

simondanielsson mentioned this pull request Aug 19, 2026

[Tracking][ROCm][Perf]: DP Attention performance #48255

Open

7 tasks

simondanielsson added 2 commits August 20, 2026 09:55

chore: generalize _load_gemm_tuned_configs

5d1610d

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

Merge branch 'feat/bpreshuffle-gemm' of github.com:simondanielsson/vl…

7752ecb

…lm into orchestrator/consolidate-gemm-loader Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

BadrBasowid reviewed Aug 20, 2026

View reviewed changes

Comment thread vllm/model_executor/kernels/linear/scaled_mm/aiter.py Outdated

refactor: into AiterPreshuffledFp8BlockScaledMMKernel

e37dbe6

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

Merge branch 'main' into feat/bpreshuffle-gemm

0e39079

tjtanaa approved these changes Sep 8, 2026

View reviewed changes

Merge branch 'main' into feat/bpreshuffle-gemm

5ee0170

tjtanaa added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 8, 2026

tjtanaa enabled auto-merge (squash) September 8, 2026 08:49

Merge branch 'main' into feat/bpreshuffle-gemm

fd1f929

simondanielsson mentioned this pull request Sep 9, 2026

[ROCm] AITER bpreshuffle blockscale GEMM #55253

Closed

4 tasks

Merge branch 'main' into feat/bpreshuffle-gemm

dacd2d1

tjtanaa merged commit 7470082 into vllm-project:main Sep 10, 2026
130 checks passed

github-project-automation Bot moved this from Todo to Done in AMD Sep 10, 2026

afriedri mentioned this pull request Sep 10, 2026

[ROCm] Refactor tuned gemms #55001

Open

mawong-amd mentioned this pull request Sep 11, 2026

[ROCm][Bugfix] Fix AITER preshuffled FP8 block-scale kernel #56433

Merged

4 tasks

eky-amd mentioned this pull request Sep 11, 2026

[ROCm] Fix fusion passes for AITER fp8 bpreshuffle blockscale GEMM #56502

Draft

4 tasks

ItsRoy69 pushed a commit to ItsRoy69/vllm that referenced this pull request Sep 15, 2026

[ROCm][Perf] Add bpreshuffled blockscaled fp8 GEMM (vllm-project#51692)

69a08ea

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

Uh oh!

Uh oh!

Conversation

simondanielsson commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Profiling

TP8+DPA

TP8+EP

GSM8k

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

simondanielsson commented Aug 18, 2026

Uh oh!

BadrBasowid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tjtanaa left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

AndreasKaratzas commented Sep 8, 2026

Uh oh!

simondanielsson commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

simondanielsson commented Sep 9, 2026

Uh oh!

AndreasKaratzas commented Sep 9, 2026

Uh oh!

simondanielsson commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

simondanielsson commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

Uh oh!

mawong-amd commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simondanielsson commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

simondanielsson commented Aug 10, 2026 •

edited

Loading

mawong-amd commented Sep 11, 2026 •

edited

Loading