Skip to content

[ROCm][Perf] Allow DSA indexer native decode above next_n=2 - #59421

Draft
amd-sriram wants to merge 1 commit into
vllm-project:mainfrom
amd-sriram:rocm/indexer-native-decode-nextn-up
Draft

amd-sriram wants to merge 1 commit into
vllm-project:mainfrom
amd-sriram:rocm/indexer-native-decode-nextn-up

Conversation

@amd-sriram

@amd-sriram amd-sriram commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

On ROCm, _supports_native_decode is next_n in (1, 2), so MTP with 2+ speculative tokens is flattened to one row per draft token and each row re-reads its sequence's KV. The AITER paged kernels have no such limit. This adds VLLM_ROCM_INDEXER_NATIVE_DECODE_MAX_NEXT_N (default 2, today's behaviour) to raise the cap.

Needs #59420 above next_n = 2 (otherwise Memory access fault in warm-up).

Test Plan

GLM-5.3, TP4, MI355X, deepseek_mtp, cap = 8, Gluon decode, #59420 applied, CUDA graphs on. Throughput sweep, GPQA Diamond, RULER niah_single_2 (64k).

Test Result

next_n (spec tokens) GPQA NIAH Geomean tok/s/GPU
no MTP 0.879 1.00 2435.5
3 (2) 0.889 1.00 3309.1
5 (4) 0.904 1.00 3607.8

Throughput per GPU:

Conc 8k/1k no MTP 8k/1k n=3 8k/1k n=5 60k/600 no MTP 60k/600 n=3 60k/600 n=5
4 544.4 1105.3 1337.0 2925.1 4067.3 4422.3
8 897.5 1650.6 2015.3 3889.2 4658.8 4408.6
16 1415.4 2177.6 2648.5 4550.6 4948.6 5050.4
32 2058.7 3061.8 3638.4 4938.2 5175.0 5303.4
64 2853.3 4013.9 4447.8
128 3741.3 4715.5 4291.3
256 4603.2 4663.6 5208.6

The no-MTP run is the same setup without speculative decoding (decode paged MQA logits on the FlyDSL kernel instead of Gluon).

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @amd-sriram.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 7, 2026
@amd-sriram
amd-sriram force-pushed the rocm/indexer-native-decode-nextn-up branch from 6afd143 to f0de0a2 Compare October 7, 2026 09:55
@mergify mergify Bot removed the needs-rebase label Oct 7, 2026
@amd-sriram
amd-sriram force-pushed the rocm/indexer-native-decode-nextn-up branch from a82b92b to ea8059a Compare October 7, 2026 13:23
@@ -879,6 +879,8 @@ def _supports_native_decode(next_n: int) -> bool:
instead of flattening to one single-token row per query, which re-reads
the KV tile once per row.
"""
if current_platform.is_rocm():
return next_n <= envs.VLLM_ROCM_INDEXER_NATIVE_DECODE_MAX_NEXT_N

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets find out what next_n's work out of the box, and add any guards here, otherwise return True if we believe it works for all next_n's. (Check gfx942 and gfx950 for now)

@amd-sriram
amd-sriram force-pushed the rocm/indexer-native-decode-nextn-up branch from ea8059a to ff5a09d Compare October 8, 2026 13:29

@simondanielsson simondanielsson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @amd-sriram!

  1. Can we also make it clear in the perf table the effect of this PR? I.e. say for MTP=5, what is the performance when using VLLM_ROCM_INDEXER_NATIVE_DECODE_MAX_NEXT_N=5 vs the default
  2. I think AITER also dispatches to one of several different indexer kernels under the hood, would be good to validate that the reachable ones all work natively (as Doug said, at least on gfx942/950). Could worth checking if rnda also dispatches to these kernels, otherwise we might want to guard this on cdna>2

@amd-sriram
amd-sriram force-pushed the rocm/indexer-native-decode-nextn-up branch 3 times, most recently from 6330c2d to fd4669e Compare October 11, 2026 12:49
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
@amd-sriram
amd-sriram force-pushed the rocm/indexer-native-decode-nextn-up branch from fd4669e to ed643df Compare October 12, 2026 15:25
@amd-sriram
amd-sriram marked this pull request as draft October 14, 2026 09:40
@amd-sriram

Copy link
Copy Markdown
Contributor Author

@simondanielsson @dllehr-amd Thanks for the review. I am moving this PR to draft currently. I am waiting for aiter PRs - ROCm/aiter#6451 (Triton/Gluon) and ROCm/aiter#6450 (FlyDSL) to land in vLLM's AITER pin.
I need to reevaluate with an image which reverts the vllm pr - #59420 and adds these aiter prs.
The .contiguous()'s copy in vllm pr - #59420 (4us) cancels most of the native decode gain, so the delta is not significant.
I will rerun the end to end numbers with the aiter PRs and reopen this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

3 participants