Skip to content

[WIP] Custom iq4_xs vulkan kernel - #28417

Closed
pwomser wants to merge 3 commits into
ggml-org:masterfrom
pwomser:custom_iq4_xs
Closed

[WIP] Custom iq4_xs vulkan kernel#28417
pwomser wants to merge 3 commits into
ggml-org:masterfrom
pwomser:custom_iq4_xs

Conversation

@pwomser

@pwomser pwomser commented Sep 4, 2026

Copy link
Copy Markdown

Overview

This PR adds a custom kernel for the IQ4_XS quantization that increases token generation on unsloth/Qwen3.8-27B-UD-Q3_K_XL from ~36 t/s to ~37.2 t/s on my machine. The initial version was created by ShinkaEvolve with Qwen3.8-27b as LLM for the evolution and then reviewed and cleaned up by me.

Additional information

Benchmark on unsloth/Qwen3.8-27B-UD-Q3_K_XL (IQ4_XS takes up ~40% of total compute on that model) on my RX 9070 XT (radv):
./build/bin/llama-bench -m Qwen3.8-27B-UD-Q3_K_XL.gguf -ngl 99 -fa on -n 3 -r 4096 -p 0

Old:

model size params backend ngl fa test t/s
qwen35 27B Q3_K - Large 12.23 GiB 27.32 B Vulkan 99 1 tg3 36.03 ± 0.08

New:

model size params backend ngl fa test t/s
qwen35 27B Q3_K - Large 12.23 GiB 27.32 B Vulkan 99 1 tg3 37.19 ± 0.08

Perplexity on wikitext-2-raw didn't change:

lama-perplexity -m Qwen3.8-27B-UD-Q3_K_XL.gguf -c 512 --chunks 30 -ngl 99 --seed 1234:

PPL OLD PPL NEW
wiki.valid.raw PPL = 5.5262 +/- 0.15001 PPL = 5.5262 +/- 0.15001
wiki.train.raw PPL = 8.2900 +/- 0.24976 PPL = 8.2900 +/- 0.24976
wiki.test.raw PPL = 5.7448 +/- 0.15760 PPL = 5.7448 +/- 0.15760

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - Kernel was initially created with ShinkaEvolve with Qwen3.8-27b. Qwen3.8-27b and GPT5.6 were also used to review it before submission.

@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Sep 4, 2026
@pwomser

pwomser commented Sep 5, 2026

Copy link
Copy Markdown
Author

Closing in favor of #28426

@pwomser pwomser closed this Sep 5, 2026
@pwomser
pwomser deleted the custom_iq4_xs branch September 5, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant