vulkan: chunk batched mat-vec at the column counts that are slow on RDNA3.5 - #1
Merged
Merged
Conversation
…DNA3.5
On Strix Halo (RADV, Mesa 26.1) the mat-vec shader variants for 3, 5 and 6
columns -- and 4 for q4_K -- run several times slower per call than 1, 2 and
4. GGML_VK_PERF_LOGGER on Qwen3.8-Flash-Next: the q6_K LM head 2.24 ms at
n=1/2/4 but 12.8 at 3 and 18.5 at 5; q8_0 10240x2560 122 us at 1, 734 at 3,
1177 at 5; q4_K 12288x2560 85 us at 1, 180 at 4. Speculative verification and
multi-slot decode live exactly there.
Such batches are dispatched as chunks of column counts measured to scale
({4,2,1} for q8_0 and q6_K, pairs for the other quants), at the cost of
re-reading the weights once per chunk; 9-16 columns are kept on chunked
mat-vec instead of the mat-mat kernels, which are worse still at those widths
(a 3.3 MB q8_0 weight at n=21: 756 us against 17.5 per mat-vec column); and
q8_0 at two columns stays off the q8_1 integer-dot path, where it is 19x
slower than the f32 path. GGML_VK_MMV_NO_SPLIT=1 disables all of it.
Measured on the same tree: four bare slots decoding concurrently 53.5 t/s
aggregate (14.4 each) against 47.8 on ROCm; every shape in the census fixed;
single-stream within noise.
(cherry picked from commit d737d6e)
LaurentZuijdwijk
approved these changes
Aug 27, 2026
LaurentZuijdwijk
left a comment
LaurentZuijdwijk
left a comment
Member
There was a problem hiding this comment.
is this replicated for other models?
Collaborator
Author
|
I have not tried it but i'm assuming so because i don't think this change is model specific even if it was benchmarked with one. Happy to test it further before merging |
Collaborator
Author
|
this seems to bring no difference to 27B performance but affects Flash-Next version so worth merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Strix Halo (RADV, Mesa 26.1) the mat-vec shader variants for 3, 5 and 6 columns — and 4 for q4_K — run several times slower per call than 1, 2 and 4.
GGML_VK_PERF_LOGGERon Qwen3.8-Flash-Next: the q6_K LM head 2.24 ms at n=1/2/4 but 12.8 at 3 and 18.5 at 5; q8_0 10240x2560 122 us at 1, 734 at 3, 1177 at 5; q4_K 12288x2560 85 us at 1, 180 at 4. Speculative verification and multi-slot decode live exactly there.Such batches are dispatched as chunks of column counts measured to scale, at the cost of re-reading the weights once per chunk. 9–16 columns stay on chunked mat-vec rather than the mat-mat kernels, which are worse still at those widths. q8_0 at two columns stays off the q8_1 integer-dot path, where it is 19x slower than f32.
GGML_VK_MMV_NO_SPLIT=1disables all of it.Measured: four bare slots decoding concurrently 53.5 t/s aggregate (14.4 each) against 47.8 on ROCm; every shape in the census fixed; single-stream within noise.
Scope: gated on
VK_VENDOR_ID_AMDand env-disableable — behaviour is unchanged on every other vendor. It is a driver/hardware heuristic tuned for RDNA3.5, which is why it is here rather than inhalo-box/llama.cpp.Build-verified with
-DGGML_VULKAN=ONonfedora:44(vulkan-headers, glslc, glslang, spirv-tools — same toolchain asmimiron/llamacpp-vulkan's build stage).