[ROCm][Perf] Enable CSA multi-stream overlap for DeepSeek-V4 by shen-shanshan · Pull Request #51794 · vllm-project/vllm

shen-shanshan · 2026-08-11T09:48:40Z

Purpose

Enable kernel-level multi-stream overlap for DeepSeek-V4 CSA (Compressed Sparse Attention, compress_ratio=4) attention layers on ROCm. When enabled, the attention forward forks three HIP streams before the input GEMMs:

Default stream: fused wqa+wkv GEMM → q/kv RMSNorm → wq_b → qnorm/RoPE → SWA KV insert.
Aux stream 0: main-compressor wkv_gate GEMM → compress kernels (compressed KV cache write).
Aux stream 1: indexer-compressor wkv_gate GEMM → K-cache write.

After the join, the indexer weights GEMM, the indexer q-side (wq_b + fused q rope+quant), the sparse indexer op (top-k selection), and the MLA attention run serially on the main stream. Moving the two compressor wkv_gate GEMMs onto side streams takes them off the serial critical path; each stream runs a self-contained chain straight from
hidden_states, so no fine-grained cross-stream dependencies are required.

Trace Comparison (take `conc=4` as an example)

Before:

After (12.99% faster for CSA layer):

Test Plan

Server launch command:

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export AITER_LOG_LEVEL=WARNING

vllm serve deepseek-ai/DeepSeek-V4-Pro \
    --port 8333 \
    --tensor-parallel-size 8 \
    --max-model-len 9472 \
    --max-num-seqs 512 \
    --max-num-batched-tokens 16384 \
    --async-scheduling \
    --no-enable-prefix-caching \
    --distributed-executor-backend mp \
    --gpu-memory-utilization 0.9 \
    --kv-cache-dtype fp8 \
    --trust-remote-code \
    --moe-backend aiter \
    --tokenizer-mode deepseek_v4 \
    --reasoning-parser deepseek_v4 \
    --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'

Benchmark with SA InferenceX 8k1k workload.
Accuracy test with gsm8k dataset.

Test Result

Benchmark

Benchmark results based on nightly-rocm100 image:

Conc	Version	Output token throughput (tok/s)	Change vs. Main	Mean TTFT (ms)	Change vs. Main	Mean TPOT (ms)	Change vs. Main
1	Main	66.30	—	441.89	—	14.62	—
1	This PR	69.34	4.59% ↑	369.48	16.39% ↓	14.04	3.97% ↓
4	Main	219.20	—	460.16	—	17.33	—
4	This PR	227.58	3.82% ↑	448.85	2.46% ↓	16.69	3.69% ↓
16	Main	615.62	—	708.66	—	24.42	—
16	This PR	629.78	2.30% ↑	712.20	0.50% ↑	23.86	2.29% ↓
64	Main	1244.20	—	1614.50	—	48.81	—
64	This PR	1260.70	1.33% ↑	1595.88	1.15% ↓	48.17	1.31% ↓

In summary, the PR provides an overall positive performance impact: throughput improves at every concurrency level, TPOT consistently decreases, and TTFT is improved at three out of four concurrency levels. The only notable trade-off is the minor TTFT regression at conc=16.

Accuracy Test

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9439	±	0.0063
		strict-match	5	exact_match	↑	0.9439	±	0.0063

Full GSM8K evaluation on this change (1,319 examples) and both results exceed the required 94% threshold.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mergify · 2026-08-13T02:23:59Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @shen-shanshan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-08-16T04:32:59Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @shen-shanshan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-08-19T01:06:59Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @shen-shanshan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mawong-amd · 2026-09-11T22:17:33Z

+        Dict metadata marks piecewise cudagraph, whose eager breaks rebuild
+        the attention inputs on the owning stream. Forking side streams
+        there would rely on runtime HIP event sync, which is unreliable in
+        this overlap on ROCm (event waits can hang), so multi-stream only
+        runs where the fork/join becomes static graph edges: inside capture,
+        or with non-dict metadata (full cudagraph or the profile run), which
+        has no eager breaks.


QQ: do we still see this issue on the ROCm 10 nightly?

QQ: do we still see this issue on the ROCm 10 nightly?

Now we need to land this PR both on rocm-7.x and rocm-10.0 (even though the perf gain on rocm-7.x is much smaller), so we still need this guard, since launching multi-stream with eager mode will come across GPU hang issues on rocm-7.x. Haven't get time to test eager multi-stream on rocm-10.0 yet.

Fangzhou-Ai · 2026-09-14T02:16:38Z

LGTM!

Fangzhou-Ai · 2026-09-14T02:19:32Z

hi @zyongye @tjtanaa please take a look at this PR. We have test thoroughly on this PR and it shows decent perf gain.

shen-shanshan · 2026-09-14T02:32:40Z

/ci run

github-actions · 2026-09-14T02:32:57Z

✅ Triggered Buildkite CI #88708 for commit fb2e6824941e.

tjtanaa · 2026-09-14T07:02:03Z

+        attn_metadata = get_forward_context().attn_metadata
+        return (
+            self.aux_stream_list is not None
+            and envs.VLLM_ROCM_DSV4_CSA_MULTI_STREAM


Let's cache this envs.VLLM_ROCM_DSV4_CSA_MULTI_STREAM value before using it based on #17067

Let's cache this envs.VLLM_ROCM_DSV4_CSA_MULTI_STREAM value before using it based on #17067

Oh, in fact we plan to make this optimization default on and has removed the env gate. This line is the one that I forget to delete...

@tjtanaa Delete in c850e5f.

shen-shanshan · 2026-09-14T08:02:35Z

Full CI passed in https://buildkite.com/vllm/ci/builds/88708/list.

Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>

shen-shanshan · 2026-09-14T09:13:12Z

/ci run

github-actions · 2026-09-14T09:13:29Z

✅ Triggered Buildkite CI #88774 for commit c850e5f1d5b1.

tjtanaa

LGTM

shen-shanshan · 2026-09-14T09:43:15Z

/ci run

github-actions · 2026-09-14T09:43:33Z

✅ Triggered Buildkite CI #88781 for commit 395374a95a9a.

…oject#51794) Signed-off-by: shen-shanshan <467638484@qq.com> Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>

shen-shanshan requested review from dllehr-amd, hongxiayang, tjtanaa and zyongye as code owners August 11, 2026 09:48

claude Bot reviewed Aug 11, 2026

View reviewed changes

shen-shanshan marked this pull request as draft August 11, 2026 09:49

mergify Bot added deepseek Related to DeepSeek models nvidia rocm Related to AMD ROCm labels Aug 11, 2026

github-project-automation Bot added this to NVIDIA and AMD Aug 11, 2026

github-project-automation Bot moved this to Todo in AMD Aug 11, 2026

mergify Bot added the needs-rebase label Aug 13, 2026

shen-shanshan force-pushed the rocm-dsv4-csa-multi-stream branch from 796108d to 5f88b24 Compare August 13, 2026 03:24

mergify Bot removed the needs-rebase label Aug 13, 2026

shen-shanshan force-pushed the rocm-dsv4-csa-multi-stream branch from 48e4031 to 06d206b Compare August 13, 2026 06:35

mergify Bot added the needs-rebase label Aug 16, 2026

shen-shanshan force-pushed the rocm-dsv4-csa-multi-stream branch from 06d206b to d314c00 Compare August 17, 2026 08:06

mergify Bot removed the needs-rebase label Aug 17, 2026

mergify Bot added the needs-rebase label Aug 19, 2026

shen-shanshan mentioned this pull request Aug 19, 2026

[RFC]: DeepSeek-V4 Performance Optimization on ROCm (Phase Two) #52911

Open

17 tasks

mergify Bot added the DSv4 label Aug 19, 2026

Fangzhou-Ai mentioned this pull request Aug 26, 2026

[ROCm][DSV4][Perf] Fuse DeepSeek V4 C4 compressor GEMMs #53838

Merged

mawong-amd mentioned this pull request Sep 8, 2026

[ROCm][Perf][Bugfix] Multi-stream perf improvements; rocprofiler fixes #55099

Merged

4 tasks

shen-shanshan force-pushed the rocm-dsv4-csa-multi-stream branch from 79ffd28 to 9301d95 Compare September 9, 2026 13:58

mergify Bot removed the needs-rebase label Sep 9, 2026

shen-shanshan force-pushed the rocm-dsv4-csa-multi-stream branch from 0e4aade to ec961e9 Compare September 10, 2026 08:08

update

2dbe0b4

Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>

shen-shanshan force-pushed the rocm-dsv4-csa-multi-stream branch from 6a0acd4 to 1843b6a Compare September 11, 2026 04:42

remove env var gate

fb2e682

Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>

shen-shanshan force-pushed the rocm-dsv4-csa-multi-stream branch from 1843b6a to fb2e682 Compare September 11, 2026 08:22

shen-shanshan marked this pull request as ready for review September 11, 2026 08:24

claude Bot reviewed Sep 11, 2026

View reviewed changes

mawong-amd reviewed Sep 11, 2026

View reviewed changes

Fangzhou-Ai added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 14, 2026

Fangzhou-Ai approved these changes Sep 14, 2026

View reviewed changes

tjtanaa reviewed Sep 14, 2026

View reviewed changes

remove env var gate

c850e5f

Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>

tjtanaa approved these changes Sep 14, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Sep 14, 2026

tjtanaa enabled auto-merge (squash) September 14, 2026 09:39

Merge branch 'main' into rocm-dsv4-csa-multi-stream

395374a

tjtanaa merged commit a6c5d6d into vllm-project:main Sep 14, 2026
72 of 73 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Sep 14, 2026

github-project-automation Bot moved this from Todo to Done in AMD Sep 14, 2026

shen-shanshan mentioned this pull request Sep 14, 2026

[ROCm][Perf] Enable HCA dual-stream overlap for DeepSeek-V4 #56853

Open

4 tasks

Uh oh!

Uh oh!

Conversation

shen-shanshan commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Trace Comparison (take conc=4 as an example)

Test Plan

Test Result

Benchmark

Accuracy Test

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mergify Bot commented Aug 13, 2026

Uh oh!

mergify Bot commented Aug 16, 2026

Uh oh!

mergify Bot commented Aug 19, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mawong-amd Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

shen-shanshan Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fangzhou-Ai commented Sep 14, 2026

Uh oh!

Fangzhou-Ai commented Sep 14, 2026

Uh oh!

shen-shanshan commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

tjtanaa Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shen-shanshan Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

shen-shanshan Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

shen-shanshan commented Sep 14, 2026

Uh oh!

shen-shanshan commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

tjtanaa left a comment

Choose a reason for hiding this comment

Uh oh!

shen-shanshan commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shen-shanshan commented Aug 11, 2026 •

edited

Loading

Trace Comparison (take `conc=4` as an example)

shen-shanshan Sep 14, 2026 •

edited

Loading

tjtanaa Sep 14, 2026 •

edited

Loading