Skip to content

vulkan: optimize iq3_xxs mat-vec shader - #28427

Closed
WakeUpMorty wants to merge 1 commit into
ggml-org:masterfrom
WakeUpMorty:perf-iq3xxs-dmmv
Closed

vulkan: optimize iq3_xxs mat-vec shader#28427
WakeUpMorty wants to merge 1 commit into
ggml-org:masterfrom
WakeUpMorty:perf-iq3xxs-dmmv

Conversation

@WakeUpMorty

@WakeUpMorty WakeUpMorty commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Rewritten mul_mat_vec_iq3_xxs mat-vec shader. Kernel goes ~65% → 74% of peak
bandwidth, total gain ~+0.5%. Related: #28426 (iq4_xs, filed together; no file
overlap)

Overview

Optimized the mul_mat_vec_iq3_xxs Vulkan compute shader for the dmmv path by mirroring the approach used in mul_mat_vec_iq3_s.
Instead of using two invocations per 32-value subblock and loading the grid via unaligned 1-byte accesses,
the new version assigns one invocation per subblock (totaling 8 per 256-value block). It now loads the grid using aligned u16 packed16 memory accesses.
The underlying math and accumulation order remain completely unchanged.

Benchmark (Qwen3.8-27B Q3_K_XL, llama-bench -r 3, tg leg d0, fresh context)

Reference = official prebuilt 10791.
Patched = 427291b + the iq4_xs PR + this change (this change sits on top of the iq4_xs PR).

build tg256 tg512
reference 10791 (official prebuilt) 34.10 34.00
patched (iq4_xs PR + this change) 36.95 36.97
combined gain +8.4% +8.7%

Standalone effect (perf logger, 32 tokens):

build iq3_xxs eff iq3_xxs ms/tok
official 10791 451 GB/s 2.245
final build 471 GB/s 2.146

= ~0.5% of total token time.

Per-shape iq3_xxs efficiency (GB/s, old → new):

shape old new (iq3_s at same shape)
m=17408 k=5120 454 531 665
m=12288 k=5120 403 482 614
m=10240 k=5120 391 452 607
m=5120 k=17408 340 403 601
m=5120 k=6144 275 372 551
m=6144 k=5120 263 336 504

Correctness: bit-identical output at temperature 0 (64 tokens, same prompt,
all three bench models) on the final combination build (both PRs applied).

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - AI was used to help with shader development, benchmarking, and this PR description. All changes were reviewed and verified by the author.

@WakeUpMorty
WakeUpMorty requested a review from a team as a code owner September 5, 2026 07:33
@ggml-gh-bot

ggml-gh-bot Bot commented Sep 5, 2026

Copy link
Copy Markdown

Hi @WakeUpMorty, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

  • AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@WakeUpMorty

Copy link
Copy Markdown
Contributor Author

Rewrote the description. Could the check be re-run?

Rewritten dequant: packed16 grid loads with bitfieldExtract signs,
8-thread workgroup. ~+0.5% token generation on xxs-heavy models.

Assisted-by: Pi agent with Qwen3.8 27B
@WakeUpMorty WakeUpMorty closed this Sep 5, 2026
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant