vulkan : copy strided f16 KV once on AMD so reads spread across memory channels by Nathanw1014 · Pull Request #27703 · ggml-org/llama.cpp

Nathanw1014 · 2026-08-25T13:37:39Z

Overview

f16 KV layout does not engage all memory channels. This PR engages all channels by copying each layers f16 KV once into a scratch so that the KV cache reads are sequential. Reuse of scratch introduced in PR #25494

Additional information

Gating set to AMD where effect was measured, RTX 3070 did not benefit, Intel and any other vendor have not been tested.

Consecutive KV rows of one head are read n_kv_heads × head_dim × 2 bytes apart. Typical model KV geometry spacing is an even multiple of the 256B channel-interleave block that Mesa documents, causing only a subset of memory channels to engage.

KV geometry (heads × head_dim)	example models	row spacing	÷ 256 B	channels engaged (of 16)
2 × 128	small GQA-heavy models	512 B	2	8
4 × 128	Qwen3-Coder-30B (benchmarked)	1024 B	4	4
2 × 256	Qwen3.6-35B class (benchmarked)	1024 B	4	4
4 × 256	Qwen3.8-27B (benchmarked)	2048 B	8	2
8 × 128	Llama-3.1-8B class	2048 B	8	2
32 × 128	MHA-era 7B models	8192 B	32	1

The tradeoff is scratch size, 1 layer of KV

KV geometry	example models	KV layers	scratch/token	@32k	@128k	% of f16 KV cache
2 × 128	small GQA-heavy (typ. 32L)	32	1 KiB	32 MiB	128 MiB	~3.1%
4 × 128	Qwen3-Coder-30B (measured)	48	2 KiB	64 MiB	256 MiB	2.1%
2 × 256	Qwen3.6-35B (hybrid, 10/40 attn)	10	2 KiB	64 MiB	256 MiB	10%
4 × 256	Qwen3.8-27B (hybrid, 16/65 attn)	16	4 KiB	128 MiB	512 MiB	6.25%
8 × 128	Llama-3.1-8B class	32	4 KiB	128 MiB	512 MiB	3.1%
32 × 128	MHA-era 7B models	32	16 KiB	512 MiB	2 GiB	3.1%

TG is measured unaffected, stridding tax increases at depth. Worst case model (hybrid, 10/40 attention layers) regress ~1% in prefill @ 32k, back within run variance @ 64k.
All runs: llama-bench -m <model> -fa 1 -p 512 -n 0 -d 0,16384,32768,65536 -r 1 on Radeon 8060S (RADV STRIX_HALO), f16 KV. Stock = master 3af988f.

Qwen3-Coder-30B-A3B UD-Q6_K_XL - stock:

model	size	params	backend	ngl	fa	test	t/s
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512	1147.43 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d16384	376.48 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d32768	203.10 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d65536	72.42 ± 0.00

Qwen3-Coder-30B-A3B UD-Q6_K_XL - patched:

model	size	params	backend	ngl	fa	test	t/s
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512	1145.28 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d16384	506.22 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d32768	317.94 ± 0.00
qwen3moe 30B.A3B Q6_K	24.53 GiB	30.53 B	Vulkan	-1	1	pp512 @ d65536	186.54 ± 0.00

Qwen3.8-27B Q8_0 - stock:

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512	339.37 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d16384	240.11 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d32768	133.87 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d65536	64.18 ± 0.00

Qwen3.8-27B Q8_0 - patched:

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512	338.04 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d16384	270.60 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d32768	227.04 ± 0.00
qwen35 27B Q8_0	27.04 GiB	27.32 B	Vulkan	-1	1	pp512 @ d65536	172.65 ± 0.00

Qwen3.6-35B-A3B UD-Q4_K_XL - stock:

model	size	params	backend	ngl	fa	test	t/s
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512	1114.82 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d16384	814.97 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d32768	663.33 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d65536	482.44 ± 0.00

Qwen3.6-35B-A3B UD-Q4_K_XL - patched:

model	size	params	backend	ngl	fa	test	t/s
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512	1113.00 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d16384	812.31 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d32768	656.72 ± 0.00
qwen35moe 35B.A3B Q4_K - Medium	20.81 GiB	34.66 B	Vulkan	-1	1	pp512 @ d65536	478.51 ± 0.00

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes : Claude (Opus 4.8 / Opus 5 / Fable 5)
Assisted with benchmarking, analysis, and review; design + implementation directed by me

Assisted-by: Claude (Opus 4.8 / Opus 5)

jeffbolznv · 2026-08-25T13:48:32Z

This seems fine. Might need broader testing to make sure it doesn't regress other AMD devices. Do you know what Qwen3.6-35B-A3B UD-Q4_K_XL didn't benefit?

0cc4m

Looks good overall and I can confirm the performance improvement. Great work.

0cc4m · 2026-08-26T12:29:50Z

+layout(local_size_x = 256, local_size_y = 1, local_size_z = 1) in;
+
+layout (binding = 0) readonly buffer A {f16vec4 data_a[];};
+layout (binding = 1) writeonly buffer D {f16vec4 data_b[];};


Since you're just copying memory you can probably just use regular ivec4. You don't need float16 explicit arithmetic types here.

0cc4m · 2026-08-26T12:34:33Z

        }
+        // Strided-copy counterpart for f16 KV (same scratch layout, no dequant).
+        if (tname == "f16") {
+            string_to_spv("dequant_f16_transpose", "dequant_f16_transpose.comp", {});


I don't think dequant is right, it's just a transpose copy shader. It's not immediately f16-specific either, e.g. bf16 kv could be handled by the same shader. It could also be written in a generic way to handle copies of any (reasonable) width.

I had also thought about commenting that this could just use the normal copy shader, no new shader needed. But this dedicated shader might perform a bit better? I'm not sure it will, it may be worth doing the experiment of just using the normal copy shader.

Thanks, the existing copy shader had the same performance as a dedicated shader.

hschougaard · 2026-08-31T06:22:59Z

Does this effort cover bf16 as well?

The f16 clause needs only a strided copy, not a dequant, so drop the dedicated shader and dispatch the existing copy pipeline instead. The copy is described in source memory order so the strided access lands on the writes, which stream, instead of the reads, which would concentrate on a few memory channels. Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)

0cc4m · 2026-09-03T11:07:17Z

No, but there probably isn't a reason why it couldn't.

0cc4m · 2026-09-04T08:22:18Z

@Nathanw1014 Please rebase to fix the test conflict and check bf16.

Nathanw1014 added 3 commits August 25, 2026 09:36

vulkan : copy strided f16 KV once so reads spread across memory channels

07c113e

Assisted-by: Claude (Opus 4.8 / Opus 5)

tests : cover the f16 KV copy to scratch

9f3e5fa

Assisted-by: Claude (Opus 4.8 / Opus 5)

vulkan : gate the f16 KV scratch copy to AMD

0e57398

Assisted-by: Claude (Opus 4.8 / Opus 5)

Nathanw1014 requested review from a team and ggerganov as code owners August 25, 2026 13:37

github-actions Bot added testing Everything test related Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 25, 2026

0cc4m approved these changes Aug 26, 2026

View reviewed changes

voidsurfer mentioned this pull request Sep 6, 2026

Strix Halo Vulkan stack: FA KV-quant, mmid, dense GEMM, delta-net prefill, DSv4 sparse attention halo-box/strix-llama.cpp#17

Merged

Navigation Menu

Uh oh!

vulkan : copy strided f16 KV once on AMD so reads spread across memory channels - #27703

vulkan : copy strided f16 KV once on AMD so reads spread across memory channels#27703
Nathanw1014 wants to merge 4 commits into
ggml-org:masterfrom
Nathanw1014:vulkan-fa-f16-kv-contig-v2

Nathanw1014 commented Aug 25, 2026

Uh oh!

jeffbolznv commented Aug 25, 2026

Uh oh!

0cc4m left a comment

Uh oh!

0cc4m Aug 26, 2026

Uh oh!

0cc4m Aug 26, 2026

Uh oh!

jeffbolznv Aug 26, 2026

Uh oh!

Nathanw1014 Aug 31, 2026

Uh oh!

hschougaard commented Aug 31, 2026

Uh oh!

0cc4m commented Sep 3, 2026

Uh oh!

0cc4m commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Nathanw1014 commented Aug 25, 2026

Overview

Additional information

Requirements

Uh oh!

jeffbolznv commented Aug 25, 2026

Uh oh!

0cc4m left a comment

Choose a reason for hiding this comment

Uh oh!

0cc4m Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

0cc4m Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

jeffbolznv Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Nathanw1014 Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

hschougaard commented Aug 31, 2026

Uh oh!

0cc4m commented Sep 3, 2026

Uh oh!

0cc4m commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants