[ROCm][DSv4.1][Perf] One-pass dequantize and a gather-sized grid for the compressed K cache by JohnQinAMD · Pull Request #56720 · vllm-project/vllm

JohnQinAMD · 2026-09-13T20:47:34Z

Purpose

dequantize_and_gather_k_cache_triton walks a V4.1 prefill's whole compressed prefix once per layer, and it did so on a fixed (num_reqs, 128) grid with a per-token loop that issued twenty-two memory operations. Two changes here: the grid follows the length being gathered, and a token is dequantized in one pass. They compose — the first raises how many dependent-load chains are in flight, the second cuts the work each chain does.

Sizing the grid

Each worker walks its slice of the sequence one token at a time, and every step is a dependent load: block table entry, then the page it points at, then the token inside it. The loop is bound by that chain's latency rather than by bandwidth, so the grid has to hold enough chains in flight to cover it. At batch size 1 the whole grid was 128 workgroups on a 256-CU gfx950, while a V4.1 prefill gathers the entire compressed prefix once per layer: about 3.1k serial steps per worker at a 399k-token context.

A caller that knows a host-side bound on gather_len now passes it and the grid follows, aiming for eight steps per worker. The DeepSeek-V4.1 ROCm compressed-KV gather takes that bound from prefill_seq_lens_cpu, which the metadata already carries, so nothing new syncs with the device. Callers that pass no bound keep the previous width, so the CUDA paths are unchanged. The same file's sliding-window gather keeps the old width on purpose: it reads at most window_size plus the chunk's query tokens.

Dequantizing in one pass

The per-token loop walked the 448 fp8 bytes in seven 64-byte quantization blocks, loading a scale byte per block and storing per block, then copied the 64 bf16 tail in four chunks of sixteen. The seven blocks tile 448 exactly, so the qblock_start < fp8_dim branch and the offsets < fp8_dim masks never excluded anything; what they cost is twenty-two memory operations per token where four will do.

The MXFP8 record added for the FlashMLA V4.1 layout already reads its token in one load, scales it with a reshape, and stores it once. This gives the fp8_ds_mla kernel the same shape. scale_dim * quant_block is 512 against an fp8 region of 448, so the tail of the tile reads into the token's own bf16 bytes, is scaled by a scale nothing reads, and is dropped by the store mask; the token record is 576 bytes, so the read stays inside it, and a tl.static_assert holds that.

Kernel time

MI355X, one request, torch.profiler over ten calls after five warm-ups, same inputs on every arm:

context	main	grid by length	+ one-pass dequantize	vs main
8,192	95.9 us	18.9 us	6.6 us	14.5x
32,768	480.1 us	61.2 us	19.9 us	24.1x
131,072	1945.6 us	230.2 us	65.9 us	29.5x
399,360	7767.2 us	914.5 us	343.9 us	22.6x
1,048,576	20359.1 us	2385.1 us	912.0 us	22.3x

The grid's own gain settles at −88% from 32k up, because gather_num_workers caps at 8192 and a 1M gather wants more; the one-pass dequantize then takes a further 62–71% at every length, since it changes the work per token rather than the parallelism. The combined factor peaks at 131k and eases to 22x beyond it: past the cap the grid stops widening, so the longest contexts keep the per-token win but not the rest of the parallelism one.

A width of 8192 is where widening stops paying. Sweeping the grid by hand on a 1M-token gather, with the dequantize already in one pass, 2048 and 8192 workers are within 3% of each other and 65536 buys 2.5%; at 131k a width of 65536 is 12% slower than 2048. The cap is not what bounds the long-context number.

Correctness

The second commit adds a kernel rather than editing one: ROCm gets _dequantize_and_gather_k_one_pass_kernel, CUDA keeps _dequantize_and_gather_k_kernel. Neither had a test — the existing cases import the deepseek_v4 copy, and the two V4.1 cases cover the 528 B MXFP8 and 288 B NVFP4 records, both SM100-only. The 584 B record these two kernels decode is the only one a ROCm V4.1 server gathers.

The third commit covers both kernels, parametrized on the ROCm dispatch so each runs wherever the test lands. It checks that grid width cannot move the output — num_workers is only the per-token loop stride and each iteration writes its own row, so four widths must be byte-equal — and that the rows match a host-side UE8M0 dequantization, which is what catches a wrong tile or scale.

End-to-end

MI355X, TP 4, DeepSeek-V4.1-Flash, vLLM at af1c01499 (this branch's parent, so the arms differ only in this PR's two files), [official MI355X recipe][recipe] config minus its --speculative-config. Two things the recipe omits are needed to boot: --ulimit memlock=-1 on the container (the engram table pins 23.6 GiB per rank) and #57036 in every arm, without which the AITER fused-expert path indexes the expert weights with the -1 cudagraph padding sentinel and all ranks fault during the startup profile run.

vllm bench serve --random-output-len 8 so the number is prefill, 40 prompts per repetition at concurrency 4. A repetition replays the previous one's prompts, so the first is cold and the rest hit the prefix cache. Warm rows are the mean of eight repetitions ± their standard deviation; the whole table was measured twice with identical seeds and agrees to within 2 %.

shape	main	grid by length	+ one-pass dequantize
131k, warm cache	354.7 ± 4.3 ms	224.8 ± 1.4 ms	201.6 ± 1.8 ms	−43.2 %
131k, cold	15638.0 ± 2.9 ms	14409.9 ± 30.5 ms	13985.4 ± 11.2 ms	−10.6 %
32k, warm cache	158.7 ± 9.4 ms	137.2 ± 11.0 ms	129.1 ± 9.9 ms	−18.7 %
32k, cold	2223.7 ± 109.9 ms	2192.5 ± 121.9 ms	2096.3 ± 108.4 ms	−5.7 %

The changes compose at 131k: the grid takes 36.6 %, the one-pass dequantize a further 10.3 %. The 32k cold row is four fresh-server runs per arm and every arm's spread is ~5 %, so it should not be read as a gain — a microbenchmark puts the gather at ~1 % of a cold 32k prefill, against 11 % of a warm one.

Percentiles over the 320 pooled warm requests per cell (--save-detailed); a single repetition puts only four requests above p90 and swings by 25 %:

shape		main	grid by length	+ one-pass dequantize
131k	p50	389.6 ms	250.0 ms	219.8 ms
	p90	402.7 ms	259.7 ms	232.7 ms
	p99	549.1 ms	335.8 ms	313.5 ms
32k	p50	180.8 ms	146.3 ms	140.2 ms
	p90	188.8 ms	200.7 ms	165.6 ms
	p99	201.6 ms	216.8 ms	195.8 ms

GSM8K (1319 questions, 5-shot, temperature 0, tests/evals/gsm8k) with tests/evals/gsm8k/gsm8k_eval.py : 0.9022 on the merge base against 0.9045 with the branch. Both runs report a zero invalid rate.

GSM8K via InferenceX's lm-eval config (infx/evals/gsm8k.yaml, 5-shot, chat template, strict and flexible extraction): 0.9765 / 0.9757 on the unmodified nightly image against 0.9742 / 0.9735 with this PR's two files applied
Both arms carry the #57036 padding-sentinel fix, without which the server does not start on MI355X.

Notes

The grid width becomes data-dependent. This call site is prefill-only, and num_workers is read inside the kernel through tl.num_programs(1), so any width is correct — a replayed capture would use the width it recorded and only lose the speedup. #56686 goes the other way for the candidate-block kernels, which do run inside decode captures.

The sliding-window gather in the same file keeps the old width: 40 calls totalling 0.49 ms in the profile above. deepseek_v4 carries a copy of this kernel with the same fixed 128 and is left alone; that path deserves its own numbers.

#42504 attacks this kernel from a different angle — gather only the top-k slots the sparse attention reads, instead of dequantizing the whole prefix and discarding most of it. That is the larger win and this does not conflict with it: after this change the gather is still 32.4 ms of a 97.8 ms prefill kernel budget.

🤖 Generated with Claude Code

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mergify · 2026-09-14T07:39:22Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @JohnQinAMD.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Fangzhou-Ai · 2026-09-15T03:07:12Z

Hi @JohnQinAMD can you make your table covers all the way to context length limit 1M?

JohnQinAMD · 2026-09-17T04:08:02Z

Hi @JohnQinAMD can you make your table covers all the way to context length limit 1M?

Hi @Fangzhou-Ai — the kernel table already runs to the 1M limit: at a 1,048,576-token gather it is 20359.1 us → 912.0 us, 22.3x. That is the number that scales with context, and it is measured at every power-of-two step from 8k up. Thanks

Kernel time

MI355X, one request, torch.profiler over ten calls after five warm-ups, same inputs on every arm:

context	main	grid by length	+ one-pass dequantize	vs main
8,192	95.9 us	18.9 us	6.6 us	14.5x
32,768	480.1 us	61.2 us	19.9 us	24.1x
131,072	1945.6 us	230.2 us	65.9 us	29.5x
399,360	7767.2 us	914.5 us	343.9 us	22.6x
1,048,576	20359.1 us	2385.1 us	912.0 us	22.3x

github-actions · 2026-09-17T16:49:56Z

✅ @JohnQinAMD, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

…thers dequantize_and_gather_k_cache_triton launched a fixed (num_reqs, 128) grid. Each worker then walks its slice of the sequence one token at a time, and every step is a dependent load: block table entry, then the page it points at, then the token inside it. The loop is bound by that chain's latency rather than by bandwidth, so the grid has to hold enough of those chains in flight to cover it. At batch size 1 that whole grid is 128 workgroups on a 256-CU gfx950, while a V4.1 prefill gathers the entire compressed prefix once per layer: about 3.1k serial steps per worker at a 399k-token context. A caller that knows a host-side bound on gather_len now passes it and the grid follows, aiming for eight steps per worker. The DeepSeek-V4.1 ROCm compressed-KV gather takes that bound from prefill_seq_lens_cpu, which the metadata already carries, so nothing new syncs with the device. Callers that pass no bound keep the previous width, so the CUDA paths are unchanged. The same file's sliding-window gather keeps the old width on purpose: it reads at most window_size plus the chunk's query tokens, and in the profile below its 40 calls total 0.49 ms against the compressed gather's 261.7 ms. Measured on MI355X TP4, DeepSeek-V4.1-Flash, vllm bench serve with --random-output-len 8 so the number is prefill. Mean TTFT over 40 prompts at concurrency 4, reported as the mean of eight repetitions +- their standard deviation. A repetition replays the previous one's prompts, so the first is a cold prefill and the rest run against a populated prefix cache, which is the agentic shape this model serves. 128 workers grid by length 32k, warm cache 214.8 +- 8.5 186.3 +- 8.3 -13.3% 32k, cold 2672.9 2321.9 -13.1% 131k, warm cache 446.9 +- 8.0 320.8 +- 5.3 -28.2% 131k, cold 13453.0 12105.0 -10.0% The arms do not overlap at 131k: 437-460 ms against 311-328 ms across the eight repetitions. The gain grows with context because the gather walks the whole prefix once per layer, and it is larger on a warm cache because a cold prefill spends most of its time on the newly computed tokens instead. torch profiler on rank 0, one 399k-token request with 217 newly computed tokens, shows where it comes from: of 78 gather calls, the 38 that gather a full compressed prefix drop from a median of 8835 us to 1093 us, and the kernel falls from 262.2 ms to 32.4 ms of a prefill that goes from 403.7 ms to 195.5 ms. A width of 8192 is where this stops paying. Sweeping the grid by hand on a 1M-token gather, with the per-token dequantize already vectorized, 2048 and 8192 workers are within 3% of each other and 65536 buys 2.5%; at 131k a width of 65536 is 12% *slower* than 2048. The cap is not what bounds the long-context number. The grid only partitions tokens across workers: num_workers appears once, as the stride of the per-token loop, and every store addresses the row of the token the iteration is on. No atomics, no reduction, one writer per output row, so the result cannot depend on the width. Gathering the same paged cache at widths 128, 512, 4096 and 8192 gives byte-identical output, checked locally against the V4.1 op. Note that the dequantize_and_gather_k_cache cases in tests/kernels/test_compressor_kv_cache.py exercise the deepseek_v4 copy of this kernel, not the deepseek_v4_1 one changed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com>

_dequantize_and_gather_k_kernel walks each token's 448 fp8 bytes in seven 64-byte quantization blocks, loading a scale byte per block and storing the result per block, then copies the 64 bf16 tail in four chunks of sixteen. The seven blocks tile 448 exactly, so the `qblock_start < fp8_dim` branch and the `offsets < fp8_dim` masks never exclude anything; what they cost is twenty-two memory operations per token where four will do. The MXFP8 record added for the FlashMLA V4.1 layout already reads its token in one load, scales it with a reshape, and stores it once. This gives the fp8_ds_mla record a kernel of the same shape. scale_dim * quant_block is 512 against an fp8 region of 448, so the tail of the tile reads into the token's own bf16 bytes, is scaled by a scale nothing reads, and is dropped by the store mask; the token record is 576 bytes, so the read stays inside it, and a tl.static_assert holds that. It is a second kernel rather than a branch inside the first because the first is shared: deepseek_v41/nvidia/flashmla.py reaches it whenever the MXFP8 and NVFP4 records do not apply. Measuring and shipping a CUDA change is not in this PR's scope, and writing it this way leaves that kernel unedited -- the only line this commit removes is the launch it replaces with a choice between the two. The signatures are identical, so the choice is one expression and pointing CUDA at the one-pass kernel later is deleting it. The cost is a duplicated preamble of twenty-six lines. Output is bit-identical. Gathering the same paged cache both ways over three batch shapes, block sizes and gather lengths gives byte-equal bf16 out, with NaN payloads in the same positions -- random uint8 fill produces e4m3 NaN bit patterns at the expected rate and those compare unequal to themselves, so the comparison treats NaN positions separately. Kernel time on MI355X, measured with torch.profiler over ten calls after five warm-ups, both arms sized by the same grid: 1 request, 8192 tokens 19.2 us -> 7.2 us -63% 1 request, 65536 tokens 117.1 us -> 35.1 us -70% 4 requests, 32768 tokens 223.9 us -> 66.4 us -70% Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Nothing in the tree exercised `_dequantize_and_gather_k_kernel`: the `dequantize_and_gather_k_cache` cases in this file reach the deepseek_v4 copy, and the V4.1 case above it covers the MXFP8 record, which is a different kernel. The two changes ahead of this one therefore landed with no test. The new case asserts the two properties they rest on. The grid only partitions tokens across workers -- num_workers appears once, as the stride of the per-token loop, and every store addresses the row its iteration is on -- so four different widths over the same paged cache must give byte-equal output. And the rows themselves are checked against a host-side UE8M0 dequantization, which is what catches a wrong tile or a wrong scale in the one-pass form; the RoPE dims are stored verbatim, so those come back exactly. Both dequantization forms are covered on whatever device is present. The wrapper selects the one-pass form on ROCm alone, so without the parametrize each platform would only ever see the form it ships, and the two would never be held against the same reference. The marker is `is_cuda_alike`, not the `is_cuda` its neighbours use: this kernel runs on both, and `is_cuda` is false on ROCm, so the case would have skipped on the platform the two changes were measured on. Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JohnQinAMD · 2026-09-17T18:07:16Z

/ci run

github-actions · 2026-09-17T18:07:57Z

✅ Triggered Buildkite CI #89687 for commit e22d9b096694.

Fangzhou-Ai · 2026-09-17T21:29:17Z

We need upstream reviewers as it's touching the common path file @njhill @zyongye can you take a look at it?

JohnQinAMD · 2026-09-21T03:56:35Z

We need upstream reviewers as it's touching the common path file @njhill @zyongye can you take a look at it?

@Fangzhou-Ai this PR is modifying the common path file but mainly add optimized triton kernel that only called by ROCm and not by CUDA path. It has no impact on CUDA platform.

maeehart · 2026-09-25T14:44:07Z

@zyongye can you review the shared part of this PR? It changes vllm/models/deepseek_v41/common/ops/cache_utils.py, but the launcher only picks the new one-pass kernel when current_platform.is_rocm(), and only the ROCm prefill path passes max_gather_len. On CUDA the kernel and the 128-program grid stay the same. CI is green on e22d9b0.

We have seen the improvement on MI325X (gfx942) with deepseek-ai/DeepSeek-V4.1-Flash at TP4. In a 16k-token prefill step, the compressed-K gather goes from 141 ms to 9.2 ms. So we see this PR as important and would like to get it merged.

vllm-agent · 2026-10-03T16:27:59Z

CI selector (shadow): 7 test steps (15 jobs) instead of 33 (46 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs)	Today's rules	Selector	Would skip	Would add
NVIDIA, CPU and others	33 (46)	7 (15)	31 (42)	5 (11)
AMD mirrors	31 (39)	9 (9)	30 (38)	8 (8)

Selector would run (7)

arm-cpu-test ×3
cpu-kernel-tests ×2
cpu-language-generation-and-pooling-model-tests ×3
cpu-multi-modal-model-tests-n ×4
deepseek-v4-kernel-test-b200
deepseek-v4-kernel-test-h100
kernels-root-misc-test-b200

Would skip (today's rules run them) (31)

ascend-npu-test
basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-correctness-sleep-mode
basic-models-test-other-cpu
basic-models-tests-initialization
basic-models-tests-other
benchmarks-cli-test
distributed-dp-tests-4-gpus
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-generate
entrypoints-integration-api-server-openai-chat_completion
entrypoints-integration-api-server-openai-completion
entrypoints-integration-llm
entrypoints-integration-multimodal
entrypoints-integration-pooling
entrypoints-integration-responses-api
entrypoints-integration-speech_to_text
kernels-fla-ops-test-b200
kernels-mhc-test-b200
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4
multi-modal-processor-cpu ×4

Would add (today's rules do not run them) (5)

arm-cpu-test ×3 (code map)
cpu-kernel-tests ×2 (code map)
cpu-multi-modal-model-tests-n ×4 (code map)
deepseek-v4-kernel-test-b200 (code map)
deepseek-v4-kernel-test-h100 (code map)

AMD mirrors: would skip (30)

basic-correctness ×2
basic-correctness-cpu-offload
basic-correctness-cumem
basic-correctness-prefetch-offload
basic-correctness-sleep-mode
basic-models-tests-initialization
basic-models-tests-other
benchmarks-cli-test
distributed-dp-tests-4-gpus
entrypoints-integration-api-server ×4
entrypoints-integration-api-server-generate
entrypoints-integration-api-server-openai-chat_completion
entrypoints-integration-api-server-openai-completion
entrypoints-integration-llm
entrypoints-integration-multimodal
entrypoints-integration-pooling
entrypoints-integration-responses-api
entrypoints-integration-speech_to_text
kernels-fla-ops-test-b200
kernels-mhc-test-b200
language-models-tests-granite-l4-compatibility
language-models-tests-hybrid ×2
language-models-tests-standard
multi-modal-models-standard-1-qwen2
multi-modal-models-standard-2-qwen3-gemma
multi-modal-models-standard-3-llava-qwen2-vl
multi-modal-models-standard-4-other-whisper
multi-modal-processor ×4
platform-tests
pytorch-compilation-passes-unit-tests

AMD mirrors: would add (8)

batch-invariance-h100 (code map)
deepseek-v4-kernel-test-h100 (code map)
pytorch-nightly-dependency-override-check (code map)
ray-dependency-compatibility-check (code map)
rust-frontend-cargo-style-clippy (code map)
rust-frontend-cargo-tests (code map)
torch-stable-abi-audit (code map)
v1-sample (code map)

_{3 changed files · base d09ff77637 · head 5a22914e0d · Python record: build 92838 at 92044241a0 · kernel record: table 9204424 (build 92838), map 9204424 · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 10 optional steps the selector would also run}

Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

JohnQinAMD · 2026-10-05T18:14:08Z

/ci run

github-actions · 2026-10-05T18:14:31Z

✅ Triggered Buildkite CI #92926 for commit 5a22914e0de0.

JohnQinAMD · 2026-10-05T22:16:51Z

We need upstream reviewers as it's touching the common path file @njhill @zyongye can you take a look at it?

@njhill @zyongye gentle ping on the shared part. The common/ops/cache_utils.py change only takes effect on ROCm: dequantize_and_gather_k_cache gains an optional max_gather_len that defaults to None, which keeps the existing 128-program grid, and the new one-pass kernel is selected only under current_platform.is_rocm(), so CUDA keeps its kernel and grid. Upstream CI is green on the current head, and a re-run there on MI355X keeps the kernel times within 1 % of the description and takes 131k warm TTFT from 336 to 205 ms.

Move dsv41flash-fp4-mi355x-vllm-agentic-dspark from nightly-rocm100-0cbac6cd to nightly-rocm100-21d93d0d (published 2026-10-06T05:42:44Z, digest sha256:6367ca66a372c36ec0d00ad2f47c848b09620559f6748ab467b11be59488f143), 151 commits ahead. It carries vllm-project/vllm#56720, #57947, #58560, #59731 and #59794 (AITER 0.1.24.post1). Recipe container and master image stay equal; docs and perf-changelog record the new pin. 将 dsv41flash-fp4-mi355x-vllm-agentic-dspark 从 nightly-rocm100-0cbac6cd 重新固定到 nightly-rocm100-21d93d0d（2026-10-06T05:42:44Z 发布，digest 为 sha256:6367ca66a372c36ec0d00ad2f47c848b09620559f6748ab467b11be59488f143），领先 151 个提交，包含 vllm-project/vllm#56720、#57947、#58560、#59731 与 #59794（AITER 0.1.24.post1）。配方 container 与 master image 保持一致；文档与 perf-changelog 同步记录新镜像。 Co-authored-by: Cursor <cursoragent@cursor.com>

JohnQinAMD requested review from dllehr-amd, hongxiayang and tjtanaa as code owners September 13, 2026 20:47

claude Bot reviewed Sep 13, 2026

View reviewed changes

mergify Bot added deepseek Related to DeepSeek models DSv4.1 Related to DeepSeek-V4.1 models rocm Related to AMD ROCm labels Sep 13, 2026

github-project-automation Bot added this to AMD Sep 13, 2026

github-project-automation Bot moved this to Todo in AMD Sep 13, 2026

JohnQinAMD force-pushed the rocm-dsv41-gather-workers branch from 8e2b44a to 3578acb Compare September 13, 2026 22:56

mergify Bot added the needs-rebase label Sep 14, 2026

JohnQinAMD force-pushed the rocm-dsv41-gather-workers branch from 3578acb to e633e8a Compare September 16, 2026 19:03

JohnQinAMD requested review from AndreasKaratzas, WoosukKwon, mgoin, tlrmchlsmth, yewentao256 and zyongye as code owners September 16, 2026 19:03

JohnQinAMD force-pushed the rocm-dsv41-gather-workers branch from e633e8a to 436296f Compare September 17, 2026 04:03

mergify Bot removed the needs-rebase label Sep 17, 2026

JohnQinAMD force-pushed the rocm-dsv41-gather-workers branch from 436296f to 4069c92 Compare September 17, 2026 16:29

Fangzhou-Ai added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 17, 2026

JohnQinAMD force-pushed the rocm-dsv41-gather-workers branch 4 times, most recently from daefa9a to c082ed8 Compare September 17, 2026 17:21

JohnQinAMD force-pushed the rocm-dsv41-gather-workers branch from c082ed8 to 553640e Compare September 17, 2026 17:39

JohnQinAMD and others added 2 commits September 17, 2026 17:50

JohnQinAMD force-pushed the rocm-dsv41-gather-workers branch from 553640e to 44fe9e9 Compare September 17, 2026 17:50

Merge branch 'main' into rocm-dsv41-gather-workers

e22d9b0

Fangzhou-Ai mentioned this pull request Sep 17, 2026

[ROCm][DSv4.1][Perf] Fuse the inverse RoPE into the sparse decode reduce #57435

Merged

Fangzhou-Ai approved these changes Sep 17, 2026

View reviewed changes

Fangzhou-Ai self-requested a review September 17, 2026 21:28

WoosukKwon mentioned this pull request Sep 18, 2026

[Perf][DSV4.1] Optimize MegaMoE staging and NVFP4 cache gathers #57604

Merged

akii96 mentioned this pull request Sep 19, 2026

[HIP] [JIT] Add a DSv4 paged K gather+dequant kernel ROCm/aiter#5680

Merged

ZhengGong-amd mentioned this pull request Sep 24, 2026

[ROCm][DSv4.1][Perf] Use the shared prefill chunk plan in the ROCm sparse prefill #58539

Merged

mergify Bot added 3 commits October 1, 2026 00:17

Merge branch 'main' into rocm-dsv41-gather-workers

c4ccfb2

Merge branch 'main' into rocm-dsv41-gather-workers

ff03134

Merge branch 'main' into rocm-dsv41-gather-workers

9b3b334

Merge branch 'main' into rocm-dsv41-gather-workers

5a22914

Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zyongye approved these changes Oct 6, 2026

View reviewed changes

zyongye merged commit 97bd8c7 into vllm-project:main Oct 6, 2026
88 checks passed

Fangzhou-Ai mentioned this pull request Oct 6, 2026

[AMD][MI355X] DSv4.1 Flash: enable the vLLM mono decode kernels / [AMD][MI355X] DSv4.1 Flash：启用 vLLM mono decode 内核（镜像待定） SemiAnalysisAI/InferenceX#3690

Merged

JohnQinAMD mentioned this pull request Oct 7, 2026

[ROCm][DSv4.1][Perf] Replace the Torch prefill combine with the shared combine_topk_swa_indices kernel #56638

Merged

Conversation

JohnQinAMD commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Sizing the grid

Dequantizing in one pass

Kernel time

Correctness

End-to-end

Notes

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mergify Bot commented Sep 14, 2026

Uh oh!

Fangzhou-Ai commented Sep 15, 2026

Uh oh!

JohnQinAMD commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kernel time

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

JohnQinAMD commented Sep 17, 2026

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

Fangzhou-Ai commented Sep 17, 2026

Uh oh!

JohnQinAMD commented Sep 21, 2026

Uh oh!

maeehart commented Sep 25, 2026

Uh oh!

vllm-agent commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI selector (shadow): 7 test steps (15 jobs) instead of 33 (46 jobs)

Uh oh!

JohnQinAMD commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

JohnQinAMD commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JohnQinAMD commented Sep 13, 2026 •

edited

Loading

JohnQinAMD commented Sep 17, 2026 •

edited

Loading

vllm-agent commented Oct 3, 2026 •

edited

Loading