vulkan: add IQ4_XS MMQ/MMV matmul kernels

[pwilkin]

Overview

Added dedicated MMV / MMQ IQ4_XS MAT_MUL kernels that avoid the generic path with float conversion which was slow.

Additional information

IQ4_XS MUL_MAT

Operation	Shape (m,n,k)	Before us	After us	Speedup	Before TFLOPS	After TFLOPS
Decode	4096, 1, 14336	69.55	26.87	2.59x	1.69	4.37
Prefill, full sweep	4096, 512, 14336	1290.84	1074.60	1.20x	46.58	55.96
Prefill, repeated median	4096, 512, 14336	1290.84	889.81	1.45x	46.58	67.58

IQ4_XS MUL_MAT_ID

Experts / used	(m,k)	Batch	Before us	After us	Speedup
128 / 8	768, 2048	1	8.88	11.03	0.81x
128 / 8	768, 2048	4	29.24	29.33	1.00x
128 / 8	768, 2048	8	56.47	56.51	1.00x
128 / 8	768, 2048	32	456.37	463.89	0.98x
128 / 8	768, 2048	64	613.56	438.14	1.40x
128 / 8	768, 2048	128	665.36	571.67	1.16x
128 / 8	768, 2048	256	729.47	681.96	1.07x
128 / 8	768, 2048	512	848.41	937.31	0.91x
32 / 4	1792, 2048	1	10.17	10.54	0.96x
32 / 4	1792, 2048	4	34.89	33.88	1.03x
32 / 4	1792, 2048	8	65.28	66.01	0.99x
32 / 4	1792, 2048	32	302.11	296.67	1.02x
32 / 4	1792, 2048	64	379.46	267.09	1.42x
32 / 4	1792, 2048	128	437.23	365.21	1.20x
32 / 4	1792, 2048	256	505.12	503.37	1.00x
32 / 4	1792, 2048	512	761.49	673.11	1.13x

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, Codex

[jeffbolznv]

As some general feedback, please avoid making ggml-vulkan codepaths that are unnecessarily vendor-specific. I don't think there's a reason for this to be limited to AMD, you can probe the layout in the shader and avoid hardcoding vendor- and compiler-version-specific layouts in the shader. I asked codex to fix this and it took all of five minutes and it runs fine on NV. Similarly, I don't think there's a reason this is iq4_xs-specific, though I didn't go through the exercise of enabling and testing the other types. From a skim it looks like the code is there, it just needs to be plugged in.

FWIW, on the n=512 iq4_xs perf test, I get 106 TFLOPS on the original coopmat1 path, 134 TFLOPS on this change's coopmat1 path, though still trailing coopmat2's 171 TFLOPS. There could still be room for improvement with some tile size tuning!

[0cc4m]

The vendor-specific part is an unmarked copy of #27952 ...

[0cc4m]

@jeffbolznv you can move that feedback to my PR, but the reason was simply that probing the layout costs registers that I do not have available. I'd rather wait to make it generic once maintenance1 is available in the main drivers (non-beta).

[pwilkin]

Yeah, this is my bad, I should've split it into the part stacked on top of #27952 and the separate one. Will do so.

[pwilkin]

All right, @jeffbolznv @0cc4m this branch now only has the standalone changes, the coopmat1 changes were moved to another PR that targets #27952

[pwomser]

See #28426 (comment) for a comparison of this PR to #28426 on unsloth/Qwen3.8-27B-UD-Q3_K_XL (40% of runtime is IQ4_XS). #28426 performs slightly better than this variant on my RX 9070 XT (radv)

[pwilkin]

@pwomser can you check with the coopmat kernels as well (#28440)?

[pwomser]

@pwomser can you check with the coopmat kernels as well (#28440)?

Here you go

llama-bench -m Qwen3.8-27B-UD-Q3_K_XL.gguf -ngl 99 -fa on -n 3 -r 4096 -p 0

#28426:

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q3_K - Large	12.23 GiB	27.32 B	Vulkan	99	1	tg3	37.30 ± 0.09

#28440:

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q3_K - Large	12.23 GiB	27.32 B	Vulkan	99	1	tg3	37.21 ± 0.10

#28415

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q3_K - Large	12.23 GiB	27.32 B	Vulkan	99	1	tg3	37.12 ± 0.10

But as @WakeUpMorty mentioned in #28426 (comment), it might not be an "either or" for the two approaches. Also they got slightly different numbers

#28415 is ~+2-3% ahead on tg here

so the two variants are probably pretty close in practice

[WakeUpMorty]

@pwomser @pwilkin @0cc4m

I am about to head out on vacation and will be away from my system. Since I wasn't expecting three PRs with overlapping or identical changes to arrive at the same time, I wanted to provide some initial data before I leave. These updates will definitely need more extensive testing across a broader range of hardware. However, I managed to run a few quick benchmarks on my setup. Here is what I found:

Models:

• Q3KXL = C:\Users\PC.lmstudio\models\Modellm\gguf\Qwen3.8-27B-UD-Q3_K_XL.gguf
• IQ4XS = C:\Users\PC.lmstudio\models\Modellm\gguf\Qwen3.8-27B-UD-IQ4_XS.gguf
• GAMMA = C:\Users\PC.lmstudio\models\Modellm\gguf\gemma-4-12b-it-IQ4_XS.gguf

Qwen3.8-27B-UD-Q3_K_XL.gguf

test	flag	Reference (BASE 10791)	PR 28426 (10816)	PR 28415 (10817)	Gain
tg256	-p 0 -n 256 -r 3 -fa on	34.50	37.02	38.10	+2.9%
tg512	-p 0 -n 512 -r 3 -fa on	34.46	37.06	38.09	+2.8%
pp512	-p 512 -n 0 -r 3 -fa on	782.39	773.66	838.52	+8.4%

Qwen3.8-27B-UD-IQ4_XS.gguf

test	flag	Reference (BASE 10791)	PR 28426 (10816)	PR 28415 (10817)	Gain
tg256	-p 0 -n 256 -r 3 -fa on	32.90	35.48	37.13	+4.7%
tg512	-p 0 -n 512 -r 3 -fa on	32.91	35.95	37.08	+3.1%
pp512	-p 512 -n 0 -r 3 -fa on	785.31	772.70	870.93	+12.7%

gemma-4-12b-it-IQ4_XS.gguf

test	flag	Reference (BASE 10791)	PR 28426 (10816)	PR 28415 (10817)	Gain
tg256	-p 0 -n 256 -r 3 -fa on	64.91	75.52	77.54	+2.7%
tg512	-p 0 -n 512 -r 3 -fa on	64.74	75.12	77.13	+2.7%
pp512	-p 512 -n 0 -r 3 -fa on	1853.46	1857.98	2117.18	+14.0%

Qwen3.8-27B-UD-Q3_K_XL.gguf

Source: C:\Users\PC\Desktop\llama-28415\bench-28415-3models fa.bat
Note: only tg3 + pp512

test	flag	Reference (BASE 10791)	PR 28426 (10816)	PR 28415 (10817)	Gain
tg3	-p 0 -n 3 -r 4096 -fa on	32.66	34.81	35.59	+2.2%
pp512	-p 512 -n 0 -r 4096 -fa on	774.51	776.27	834.94	+7.6%

Binaries:

• BASE = llama-b10791-bin-win-vulkan-x64\llama-bench.exe (build 95ef7fc / 10791)
• PR 28426 = Llama_PR1\build\bin\llama-bench.exe (build 427291b / 10816)
• PR 28415 = llama-28415\build\bin\llama-bench.exe (build 3eec6997b / 10817)

Environment InfoOS:
Windows 10Vulkan
SDK: 1.4.341.1Build Configuration:

• set VK_SDK_PATH=C:\VulkanSDK\1.4.341.1
• cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
• cmake --build build --target llama-server llama-bench llama-cli

[0cc4m]

Please rebase