vulkan: optimize iq3_xxs mat-vec shader - #28427
Closed
WakeUpMorty wants to merge 1 commit into
Closed
Conversation
|
Hi @WakeUpMorty, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
Contributor
Author
|
Rewrote the description. Could the check be re-run? |
Rewritten dequant: packed16 grid loads with bitfieldExtract signs, 8-thread workgroup. ~+0.5% token generation on xxs-heavy models. Assisted-by: Pi agent with Qwen3.8 27B
WakeUpMorty
force-pushed
the
perf-iq3xxs-dmmv
branch
from
September 5, 2026 08:53
ef8eb57 to
d748ad0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewritten mul_mat_vec_iq3_xxs mat-vec shader. Kernel goes ~65% → 74% of peak
bandwidth, total gain ~+0.5%. Related: #28426 (iq4_xs, filed together; no file
overlap)
Overview
Optimized the mul_mat_vec_iq3_xxs Vulkan compute shader for the dmmv path by mirroring the approach used in mul_mat_vec_iq3_s.
Instead of using two invocations per 32-value subblock and loading the grid via unaligned 1-byte accesses,
the new version assigns one invocation per subblock (totaling 8 per 256-value block). It now loads the grid using aligned u16 packed16 memory accesses.
The underlying math and accumulation order remain completely unchanged.
Benchmark (Qwen3.8-27B Q3_K_XL, llama-bench -r 3, tg leg d0, fresh context)
Reference = official prebuilt 10791.
Patched = 427291b + the iq4_xs PR + this change (this change sits on top of the iq4_xs PR).
Standalone effect (perf logger, 32 tokens):
= ~0.5% of total token time.
Per-shape iq3_xxs efficiency (GB/s, old → new):
Correctness: bit-identical output at temperature 0 (64 tokens, same prompt,
all three bench models) on the final combination build (both PRs applied).
Requirements