vulkan: Intel Xe flash attention, GEMM optimizations(Xe-LPG Plus/Xe2/Xe3) [MEGA PR]

[fish-jiang]

Overview

Co-authors: @jxia4intel, @sliu39

Draft / Evaluation only — not for merge. This mega PR exists solely to show the full feature set in one place and will remain as a draft. Please refer to the individual PRs below for review and merging.

Target platforms: Xe-LPG Plus (Arrow Lake-H iGPU), Xe2, Xe3

PR	Series	Description	Target
#24404	1/3	Xe-LPG Plus coopmat1 enable + INTEL_PRE_XE2 enum	Xe-LPG Plus
#24406	2/3	Intel Xe FA optimization kernels	Xe-LPG Plus, Xe2, Xe3
#24407	3/3	GEMM/Group GEMM optimizations	Xe-LPG Plus, Xe2, Xe3

Dependency graph:

#24404 (ARLH) ← #24406 (FA)
           ← #24407 (GEMM+CW)

PR6 and PR7 are independent of each other; both build on PR5.

Flash Attention (Intel Xe)

• New Vulkan shaders: single-phase prefill (flash_attn_hdim64/96/128) and two-phase split prefill/decode variants
• Pipelines keyed by (head_dim, gqa_ratio) for runtime dispatch across various GQA ratios without combinatorial pipeline proliferation
• Supports non-power-of-two GQA ratios via subgroup splitting (qk_groups)
• Intel Xe1 (integrated GPU, UMA, cooperative matrix) and Xe2 paths with separate warptile tuning
• Two-phase decode splits softmax reduction across subgroups; shared QK state copy (fa_copy_qstate) between prefill phases

GEMM kernel optimizations (Intel Xe)

• LOAD_A_OPT path: SLM-based A-matrix layout optimization for coopmat1
• MXFP4, Q4_K, Q5_K dequant via bitfieldExtract optimization
• Alt pipeline (l_alt/a_l_alt, BM=128 warptile) for runtime selection when problem dimensions are small
• f32→f16 activation conversion for Intel coopmat GEMM, scoped to Intel devices only
• vulkan-shaders-gen.cpp: registers all new pipeline variants

MoE optimizations (Intel Xe)

• mul_mm.comp shader optimization for MUL_MAT_ID: reduces unnecessary memory loads and matrix core operations for MoE models
• Separate warptile tuning for MoE expert GEMM
• Gemma4 MoE router: fuse rms_norm + mul into a single RMS_NORM_MUL kernel dispatch for the expert gate input calculation

Performance (Windows OS)

ARLH

LNL

B70 Arc Pro

PTL

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, used claude code, then lots of manual review/tweaking.

[ggml-gh-bot]

Hi @fish-jiang, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 4 open PRs.

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[SleepinDevil]

@fish-jiang, @jxia4intel, @sliu39, @virajwad, and the rest of the Intel team, amazing work on these PRs!

With these two PRs, Vulkan is MUCH faster than SYCL (even /w FP16) on non-quantised KV Cache when running both #24406 and #24407 on my Arc Pro B70 on Windows 11.

I am particularly interested in the pp8192 @ d65536 and tg128 @ d65536 results which have been absolutely stunning for lack of better words! Would be amazing to see if some of these improvements can be carried over to quantised KV cache (q8_0 and q4_1 in particular) as well.

I really hope you guys can work with the maintainers of llama.cpp to implement these in some way or form, OR work on an Intel GGUF patcher that can help patch custom GGUFs for Intel GPUs that implement the changes.

Also not sure if you guys are across the SYCL progress too, or if that's another team of devs from Intel. There's some interesting stuff happening there with #25025 for example.

GPU Tested On: Intel Arc Pro B70
OS: Windows 11

Test benches below for non-quantised KV cache across SYCL and Vulkan.

|----- Vulkan Baseline - Official Release for Windows -----|

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  --device Vulkan1  -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf
load_backend: loaded RPC backend from G:\Projects\AI\llama.cpp-Vulkan\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Arc(TM) Pro B70 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from G:\Projects\AI\llama.cpp-Vulkan\ggml-vulkan.dll
load_backend: loaded CPU backend from G:\Projects\AI\llama.cpp-Vulkan\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl |  fa | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |          pp8192 |        509.95 ± 1.07 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |           tg128 |         21.04 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |  pp8192 @ d8192 |        333.28 ± 0.12 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |   tg128 @ d8192 |         16.27 ± 0.00 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      | pp8192 @ d65536 |         92.40 ± 0.04 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |  tg128 @ d65536 |          6.28 ± 0.00 |

build: 0eca4d490 (9851) - Official Vulkan Release for Windows


|----- Vulkan PR#24406 only -----|

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  --device Vulkan1  -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Arc(TM) Pro B70 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |          pp8192 |        661.37 ± 1.64 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |           tg128 |         21.16 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |  pp8192 @ d8192 |        616.02 ± 4.80 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |   tg128 @ d8192 |         20.26 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      | pp8192 @ d65536 |       429.96 ± 12.89 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |  tg128 @ d65536 |         15.71 ± 0.01 |

build: 37bc472e6 (9591) - Vulkan Built with PR#24406 only


|----- Vulkan PR#24406 & PR#24407 -----|

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  --device Vulkan1  -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) Arc(TM) Pro B70 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |          pp8192 |       1013.48 ± 1.13 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |           tg128 |         21.19 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |  pp8192 @ d8192 |        910.56 ± 8.21 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |   tg128 @ d8192 |         20.27 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      | pp8192 @ d65536 |       554.22 ± 18.99 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     |  99 |   1 | Vulkan1      |  tg128 @ d65536 |         15.73 ± 0.00 |

build: 9da5214fd (9595) - Vulkan Built with PR#24406 & PR#24407


|----- SYCL Baseline - Official Release for Windows (FP32? without oneAPI?): -----|

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf
load_backend: loaded RPC backend from G:\Projects\AI\llama.cpp-SYCL\ggml-rpc.dll
load_backend: loaded SYCL backend from G:\Projects\AI\llama.cpp-SYCL\ggml-sycl.dll
load_backend: loaded CPU backend from G:\Projects\AI\llama.cpp-SYCL\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |          pp8192 |        281.83 ± 0.20 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |           tg128 |         20.47 ± 0.02 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |  pp8192 @ d8192 |        225.76 ± 0.04 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |   tg128 @ d8192 |         18.91 ± 0.00 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 | pp8192 @ d65536 |         96.12 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |  tg128 @ d65536 |         12.47 ± 0.00 |

build: 27c8bb4f6 (9829) - Official SYCL Release for Windows


|----- SYCL FP16 PR#25025 (with oneAPI, & #25025 - just a PR I had prebuilt for my testing that works faster than base SYCL build): -----|

call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64
set ONEAPI_DEVICE_SELECTOR=level_zero:gpu

llama-bench  -p 8192  -n 128  -d 0,8192,65536  -r 3  -fa 1  --delay 10  -ngl 99  -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |          pp8192 |        732.63 ± 2.74 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |           tg128 |         17.00 ± 1.40 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |  pp8192 @ d8192 |       493.16 ± 15.80 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |   tg128 @ d8192 |         18.99 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 | pp8192 @ d65536 |        233.11 ± 0.27 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |  tg128 @ d65536 |         12.47 ± 0.01 |

build: b5b58ae59 (9842) - SYCL Built with PR#25025