vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 - #27471
vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1#27471fish-jiang wants to merge 2 commits into
Conversation
github-actions
Bot
added
Vulkan
jeffbolznv
left a comment
jeffbolznv
left a comment
There was a problem hiding this comment.
Haven't done a full review, just some initial comments.
| const bool x_non_contig = (ctx->device->coopmat2 && src0->type == GGML_TYPE_F32) || | ||
| !ggml_vk_dim01_contiguous(src0); | ||
| const bool y_non_contig = (ctx->device->coopmat2 && src1->type == GGML_TYPE_F32) || | ||
| // Intel coopmat1: force f32->f16 conversion so the f16-B-type pipeline is used. |
There was a problem hiding this comment.
Maybe we should do more testing and try to enable this across all devices? Reducing the load bandwidth for the B matrix seems like it should generally be a win. I've also thought about potentially fusing the f32->f16 conversion into the previous node to eliminate any overhead. But even without that, this is a win on NV and at least some Intel, so it seems likely it's a win everywhere.
There was a problem hiding this comment.
Yes, testing on more platforms would be good, but I don't have a non-Intel GPU on hand right now. May try to test later.
Fusing the f32->f16 conversion into the previous node is a great idea — it could reduce the overhead of that conversion.
| case GGML_TYPE_MXFP4: | ||
| case GGML_TYPE_NVFP4: | ||
| case GGML_TYPE_TQ2_0: | ||
| if (src1_type == GGML_TYPE_F16 && !ctx->device->coopmat2) { |
There was a problem hiding this comment.
Can this be merged into the logic at the end of the function? I think it's just a different selection of mmp.
2 participants
Overview
Performance (Windows OS)
Xe1-ARL_H
Before:
C:\kernel\llama.cpp\build_base\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf
After:
C:\kernel\llama.cpp\build\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf
Xe3-PTL
Before:
C:\Users\dungeon\Desktop\fish\llama.cpp\build_ d59d\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\qwen3-8b-q4_k_m\qwen3-8b-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\qwen2.5-1.5b-instruct-q4_k_m\qwen2.5-1.5b-instruct-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\Llama-3.1-8B-Instruct-Q4_K_M\Llama-3.1-8B-Instruct-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.8-27B-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf
After:
C:\Users\dungeon\Desktop\fish\llama.cpp\build\bin\Release>llama-bench.exe -p 512 -n 0 -r 3 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\qwen3-8b-q4_k_m\qwen3-8b-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\qwen2.5-1.5b-instruct-q4_k_m\qwen2.5-1.5b-instruct-q4_k_m.gguf,C:\Users\dungeon\Desktop\models\Llama-3.1-8B-Instruct-Q4_K_M\Llama-3.1-8B-Instruct-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.8-27B-Q4_K_M.gguf,C:\Users\dungeon\Downloads\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics i gfx-driver-ci-comp_igc-34261 DCH RI (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Xe2-B70
Before:
C:\kernel\llama.cpp\build_base\bin\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf
After:
C:\kernel\llama.cpp\build\bin\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\gemma-4-31B-it-Q4_K_M.gguf,C:\kernel\model\Qwen3-0.6B-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf,C:\kernel\model\Qwen3.8-27B-Q4_K_M.gguf,C:\kernel\model\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf
Requirements
I have read and agree with the contributing guidelines
AI usage disclosure: YES, used claude code, then lots of manual review/tweaking.