Repository navigation
[ROCm][Perf] Allow DSA indexer native decode above next_n=2 - #59421
amd-sriram wants to merge 1 commit into
Conversation
b14d17b to
94fef76
Compare
e9b0252 to
6afd143
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
6afd143 to
f0de0a2
Compare
a82b92b to
ea8059a
Compare
| @@ -879,6 +879,8 @@ def _supports_native_decode(next_n: int) -> bool: | |||
| instead of flattening to one single-token row per query, which re-reads | |||
| the KV tile once per row. | |||
| """ | |||
| if current_platform.is_rocm(): | |||
| return next_n <= envs.VLLM_ROCM_INDEXER_NATIVE_DECODE_MAX_NEXT_N | |||
There was a problem hiding this comment.
Lets find out what next_n's work out of the box, and add any guards here, otherwise return True if we believe it works for all next_n's. (Check gfx942 and gfx950 for now)
ea8059a to
ff5a09d
Compare
simondanielsson
left a comment
There was a problem hiding this comment.
Thanks @amd-sriram!
- Can we also make it clear in the perf table the effect of this PR? I.e. say for MTP=5, what is the performance when using VLLM_ROCM_INDEXER_NATIVE_DECODE_MAX_NEXT_N=5 vs the default
- I think AITER also dispatches to one of several different indexer kernels under the hood, would be good to validate that the reachable ones all work natively (as Doug said, at least on gfx942/950). Could worth checking if rnda also dispatches to these kernels, otherwise we might want to guard this on cdna>2
6330c2d to
fd4669e
Compare
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
fd4669e to
ed643df
Compare
|
@simondanielsson @dllehr-amd Thanks for the review. I am moving this PR to draft currently. I am waiting for aiter PRs - ROCm/aiter#6451 (Triton/Gluon) and ROCm/aiter#6450 (FlyDSL) to land in vLLM's AITER pin. |
Purpose
On ROCm,
_supports_native_decodeisnext_n in (1, 2), so MTP with 2+ speculative tokens is flattened to one row per draft token and each row re-reads its sequence's KV. The AITER paged kernels have no such limit. This addsVLLM_ROCM_INDEXER_NATIVE_DECODE_MAX_NEXT_N(default 2, today's behaviour) to raise the cap.Needs #59420 above
next_n = 2(otherwiseMemory access faultin warm-up).Test Plan
GLM-5.3, TP4, MI355X,
deepseek_mtp, cap = 8, Gluon decode, #59420 applied, CUDA graphs on. Throughput sweep, GPQA Diamond, RULER niah_single_2 (64k).Test Result
Throughput per GPU:
The no-MTP run is the same setup without speculative decoding (decode paged MQA logits on the FlyDSL kernel instead of Gluon).