Skip to content

vulkan: add SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for coopmat1 on Intel Xe - #25380

Merged
0cc4m merged 5 commits into
ggml-org:masterfrom
fish-jiang:intel/xe-slm-a-reshape
Aug 15, 2026
Merged

vulkan: add SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for coopmat1 on Intel Xe#25380
0cc4m merged 5 commits into
ggml-org:masterfrom
fish-jiang:intel/xe-slm-a-reshape

Conversation

@fish-jiang

@fish-jiang fish-jiang commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

  • Adds two new spec constants to the coopmat mul_mm shader: SHMEM_STRIDE_PAD (id 12) and APPLY_SLM_A_RESHAPE (id 13).
  • Adds a_shmem_index()/a_shmem_stride() helper functions and a store_a() wrapper in mul_mm_funcs.glsl, used by every quant type's load_a_to_shmem path to write into shared memory.
  • ggml_vk_mul_mm_spec in ggml-vulkan.cpp now captures device by reference and pushes the two new constants only when vendor_id == VK_VENDOR_ID_INTEL && coopmat_support, enabling the reshaped layout with SHMEM_STRIDE_PAD=0 on those devices.
  • Scope note: this is split out of vulkan: GEMM/Group GEMM optimizations on Intel Xe (3/3, Xe-LPG Plus/Xe2/Xe3) #24407

Performance (Panther Lake B390 + Windows OS)

BEFORE:
C:\temp\base\Release>llama-bench.exe -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\Qwen3.5-35B-A3B-Q4_K_M\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gemma-4-26B-A4B-it-UD-Q4_K_M\gemma-4-26B-A4B-it-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) B390 GPU (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat

model size params backend ngl fa test t/s
qwen35moe 35B.A3B Q4_K - Medium 20.49 GiB 34.66 B Vulkan 99 0 pp8192 549.49 ± 6.23
qwen35moe 35B.A3B Q4_K - Medium 20.49 GiB 34.66 B Vulkan 99 1 pp8192 418.95 ± 1.33
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan 99 0 pp8192 419.09 ± 2.67
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan 99 1 pp8192 586.97 ± 2.38
gemma4 26B.A4B Q4_K - Medium 15.70 GiB 25.23 B Vulkan 99 0 pp8192 623.57 ± 1.15
gemma4 26B.A4B Q4_K - Medium 15.70 GiB 25.23 B Vulkan 99 1 pp8192 406.07 ± 1.44
qwen3moe 30B.A3B Q4_K - Medium 17.28 GiB 30.53 B Vulkan 99 0 pp8192 392.50 ± 0.74
qwen3moe 30B.A3B Q4_K - Medium 17.28 GiB 30.53 B Vulkan 99 1 pp8192 234.23 ± 2.96

build: ee445f9 (9892)

AFTER:
C:\temp\slma_pad\Release>llama-bench.exe -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\Qwen3.5-35B-A3B-Q4_K_M\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gemma-4-26B-A4B-it-UD-Q4_K_M\gemma-4-26B-A4B-it-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) B390 GPU (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat

model size params backend ngl fa test t/s
qwen35moe 35B.A3B Q4_K - Medium 20.49 GiB 34.66 B Vulkan 99 0 pp8192 617.56 ± 6.20
qwen35moe 35B.A3B Q4_K - Medium 20.49 GiB 34.66 B Vulkan 99 1 pp8192 515.63 ± 1.22
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan 99 0 pp8192 450.17 ± 0.23
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan 99 1 pp8192 624.28 ± 5.27
gemma4 26B.A4B Q4_K - Medium 15.70 GiB 25.23 B Vulkan 99 0 pp8192 707.97 ± 3.95
gemma4 26B.A4B Q4_K - Medium 15.70 GiB 25.23 B Vulkan 99 1 pp8192 441.34 ± 0.13
qwen3moe 30B.A3B Q4_K - Medium 17.28 GiB 30.53 B Vulkan 99 0 pp8192 422.60 ± 3.63
qwen3moe 30B.A3B Q4_K - Medium 17.28 GiB 30.53 B Vulkan 99 1 pp8192 247.01 ± 0.46

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, used claude code, then lots of manual review/tweaking.

@fish-jiang
fish-jiang requested a review from a team as a code owner July 7, 2026 03:26
@fish-jiang
fish-jiang marked this pull request as draft July 7, 2026 03:27
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Jul 7, 2026
@fish-jiang fish-jiang mentioned this pull request Jul 7, 2026
Draft
@jeffbolznv

jeffbolznv commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Would be good to keep ggml_vk_matmul_shmem_support up to date with the new SHMEM_STRIDE_PAD value for Intel.

I tested on NVIDIA (coopmat1) and the performance stays the same (as expected) with the change. But also, surprisingly, with the Intel spec constant values set and it was still more or less the same.

@fish-jiang

fish-jiang commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Would be good to keep ggml_vk_matmul_shmem_support up to date with the new SHMEM_STRIDE_PAD value for Intel.

I tested on NVIDIA (coopmat1) and the performance stays the same (as expected) with the change. But also, surprisingly, with the Intel spec constant values set and it was still more or less the same.

Updated the code in ggml_vk_matmul_shmem_support.

APPLY_SLM_A_RESHAPE reduces the number of SLM load messages generated for coopMatLoad in the ISA on Intel GPUs; NVIDIA's coopMatLoad likely doesn't hit the same bottleneck, which would explain why you saw no change there.

@fish-jiang
fish-jiang marked this pull request as ready for review July 8, 2026 04:16
@rillomas

rillomas commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Currently I see a slight regression when testing with Arc Pro B50 on Ubuntu 26.04, Mesa 26.1.4

Before

~/repo/llama.cpp_fish$ master-tree/build_vk/bin/llama-bench -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -m ~/models/gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   0 |          pp8192 |        697.77 ± 1.42 |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   1 |          pp8192 |        552.68 ± 0.38 |

build: ee445f93d (9892)

After

~/repo/llama.cpp_fish$ build_vk/bin/llama-bench -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -m ~/models/gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   0 |          pp8192 |        674.53 ± 0.58 |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   1 |          pp8192 |        531.50 ± 0.33 |

build: b38417ac4 (9894)

@fish-jiang

fish-jiang commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Currently I see a slight regression when testing with Arc Pro B50 on Ubuntu 26.04, Mesa 26.1.4

Before

~/repo/llama.cpp_fish$ master-tree/build_vk/bin/llama-bench -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -m ~/models/gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   0 |          pp8192 |        697.77 ± 1.42 |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   1 |          pp8192 |        552.68 ± 0.38 |

build: ee445f93d (9892)

After

~/repo/llama.cpp_fish$ build_vk/bin/llama-bench -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -m ~/models/gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   0 |          pp8192 |        674.53 ± 0.58 |
| gpt-oss 20B Q4_K - Medium      |  10.81 GiB |    20.91 B | Vulkan     |  -1 |   1 |          pp8192 |        531.50 ± 0.33 |

build: b38417ac4 (9894)

Thanks @rillomas for capturing this on B50 Linux. I found an issue with gpt-oss 20B Q4_K: the shared memory size for kvalues_mxfp4 causes SHMEM_STRIDE_PAD=0 to not work as expected. I submitted a new commit that makes the matrix shared memory address cacheline-aligned to fix this. Could you please double-check?

@rillomas

rillomas commented Jul 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Thanks, it seems the regression is fixed now. (Environment: Arc Pro B50 on Ubuntu 26.04, Mesa 26.1.4)

Before

$ master-tree/build_vk/bin/llama-bench -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -m ~/models/gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat

model size params backend ngl fa test t/s
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan -1 0 pp8192 701.41 ± 2.62
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan -1 1 pp8192 554.35 ± 0.60

build: ee445f9 (9892)

After

$ build_vk/bin/llama-bench -p 8192 -n 0 -r 2 -fa 0,1 --delay 10 -m ~/models/gpt-oss-20b-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat

model size params backend ngl fa test t/s
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan -1 0 pp8192 713.02 ± 1.98
gpt-oss 20B Q4_K - Medium 10.81 GiB 20.91 B Vulkan -1 1 pp8192 556.46 ± 0.41

build: 37c3ead (9895)

@fish-jiang

fish-jiang commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

@jeffbolznv I have a new commit in the PR: 37c3ead Could you please help review it? I'm not sure how much this impacts other vendors, but it helps with shared memory cacheline alignment for GPT-OSS on Intel GPUs.

jeffbolznv
jeffbolznv reviewed Jul 10, 2026
View reviewed changes
Comment thread ggml/src/ggml-vulkan/vulkan-shaders/types.glsl Outdated
};

shared int8_t kvalues_mxfp4[16];
shared int8_t kvalues_mxfp4[64];

@jeffbolznv jeffbolznv Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This at the very least needs a comment to say it's being padded to a cache line size for performance reasons. But now the loop at line 1796 fetches beyond the end of the source array, so that need to be fixed.

@jeffbolznv jeffbolznv Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, and the shared memory size calculation logic in ggml-vulkan.cpp also needs to be updated.

@fish-jiang fish-jiang Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching that. For the OOB read in init_iq_shmem, I've fixed the loop bound to use kvalues_mxfp4_const.length() instead of kvalues_mxfp4.length(), and added a comment explaining the cacheline padding.

For the shmem accounting in ggml_vk_matmul_shmem_support (

llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp

Lines 3633 to 3641 in 37c3ead

case GGML_TYPE_IQ4_NL:
case GGML_TYPE_IQ4_XS:
case GGML_TYPE_MXFP4:
lut_size = 4*16;
break;
case GGML_TYPE_NVFP4:
// Same kvalues budget as MXFP4 plus ue4m3_fp32_lut[128] (types.glsl, DATA_A_NVFP4).
lut_size = 4*16 + 128u * (uint32_t)sizeof(float);
break;
), I double-checked: lut_size = 4*16 already equals 64 bytes, matching the new padded kvalues_mxfp4[64] size exactly, so seems no logic change is needed there.

@0cc4m 0cc4m Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this change, shouldn't it already be aligned? Why does increasing the size without using any of the additional space help with alignment? That seems like something the driver should handle internally. I don't want to increased shared memory use without a very good reason.

@fish-jiang fish-jiang Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unexpected ISA generated by the Linux driver is causing a perf drop, so we disabled this optimization on Linux for now .
Also agree with your concern about the kvalues_mxfp4 alignment — I've reverted that change, since the SLM-A reshape path is not enabled on Linux right now.

@fish-jiang fish-jiang Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@0cc4m could you please review this PR again?

@fish-jiang
fish-jiang force-pushed the intel/xe-slm-a-reshape branch from a839129 to 4be81ee Compare July 14, 2026 02:01
@fish-jiang

fish-jiang commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

@jeffbolznv I rebased and resolved the conflicts with your USE_OCP_FP4 change. Let me know if there's anything else needed to move this PR forward.

@jeffbolznv

jeffbolznv commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

The change looks fine to me, just need @0cc4m to review. Are we confident this won't regress any other Intel configurations?

@fish-jiang

fish-jiang commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Thanks @jeffbolznv , I have re-verified the perf gains after the rebase on different platforms.
Xe3 - PTL
Before:
C:\temp\build_14d3ba45\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\Qwen3.5-35B-A3B-Q4_K_M\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gemma-4-26B-A4B-it-UD-Q4_K_M\gemma-4-26B-A4B-it-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf

model                                 size     params backend     ngl  fa            test                  t/s
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   0           pp512        772.08 ± 6.57
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   1           pp512       776.55 ± 15.10
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   0           pp512       1063.08 ± 9.62
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   1           pp512      1104.92 ± 22.71
gemma4 26B.A4B Q4_K - Medium    15.70 GiB    25.23 B Vulkan      99   0           pp512       813.54 ± 29.92
gemma4 26B.A4B Q4_K - Medium    15.70 GiB    25.23 B Vulkan      99   1           pp512       831.60 ± 44.12
qwen3moe 30B.A3B Q4_K - Medium  17.28 GiB    30.53 B Vulkan      99   0           pp512        705.81 ± 6.47
qwen3moe 30B.A3B Q4_K - Medium  17.28 GiB    30.53 B Vulkan      99   1           pp512        753.04 ± 9.86

After:
C:\temp\build_pr25380\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\Users\dungeon\Desktop\models\Qwen3.5-35B-A3B-Q4_K_M\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gpt-oss-20b-Q4_K_M\gpt-oss-20b-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\gemma-4-26B-A4B-it-UD-Q4_K_M\gemma-4-26B-A4B-it-UD-Q4_K_M.gguf,C:\Users\dungeon\Desktop\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf

model                                 size     params backend     ngl  fa            test                  t/s
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   0           pp512        867.79 ± 8.44
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   1           pp512       863.48 ± 12.40
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   0           pp512       1230.93 ± 7.10
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   1           pp512      1183.40 ± 35.74
gemma4 26B.A4B Q4_K - Medium    15.70 GiB    25.23 B Vulkan      99   0           pp512        913.45 ± 8.40
gemma4 26B.A4B Q4_K - Medium    15.70 GiB    25.23 B Vulkan      99   1           pp512       880.57 ± 63.30
qwen3moe 30B.A3B Q4_K - Medium  17.28 GiB    30.53 B Vulkan      99   0           pp512       730.57 ± 18.57
qwen3moe 30B.A3B Q4_K - Medium  17.28 GiB    30.53 B Vulkan      99   1           pp512        838.82 ± 8.31

Xe2- B70
Before:
C:\kernel\llama.cpp\build_14d3ba45\bin\Release>llama-bench.exe -p 512 -n 0 -r 10 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf,qwen3-8b-q4_k_m.gguf

model                                 size     params backend     ngl  fa            test                  t/s
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   0           pp512      1913.80 ± 28.19
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   1           pp512      1884.74 ± 22.66
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   0           pp512      2862.60 ± 61.58
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   1           pp512      2833.23 ± 46.67

After 
C:\kernel\llama.cpp\build_pr25380\bin\Release>llama-bench.exe -p 512 -n 0 -r 10 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf

model                                 size     params backend     ngl  fa            test                  t/s
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   0           pp512      2221.32 ± 24.92
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   1           pp512      2192.75 ± 32.68
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   0           pp512      3465.02 ± 47.80
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   1           pp512      3418.24 ± 39.37

Xe1 - ARL_H
Before:
C:\Users\gta\Desktop\test\build_14d3ba45\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf

model                                 size     params backend     ngl  fa            test                  t/s
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   0           pp512        349.77 ± 7.24
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   1           pp512        351.63 ± 5.75
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   0           pp512        469.46 ± 5.07
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   1           pp512        488.45 ± 8.13

After:
C:\Users\gta\Desktop\test\build_pr25380\Release>llama-bench.exe -p 512 -n 0 -r 5 -fa 0,1 --delay 10 -ngl 99 -m C:\kernel\model\Qwen3.5-35B-A3B-Q4_K_M.gguf,C:\kernel\model\gpt-oss-20b-Q4_K_M.gguf,C:\kernel\model\qwen3-8b-q4_k_m.gguf

model                                 size     params backend     ngl  fa            test                  t/s
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   0           pp512        359.03 ± 2.49
qwen35moe 35B.A3B Q4_K - Medium  20.49 GiB    34.66 B Vulkan      99   1           pp512        363.12 ± 3.59
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   0           pp512        531.80 ± 4.26
gpt-oss 20B Q4_K - Medium        10.81 GiB    20.91 B Vulkan      99   1           pp512       551.32 ± 10.25

@rillomas

rillomas commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

We're currently checking on some potential regressions on Linux.

@fish-jiang
fish-jiang force-pushed the intel/xe-slm-a-reshape branch from 69271a7 to a8ff697 Compare July 17, 2026 08:16
@rillomas

rillomas commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

We benchmarked two Battlemage GPUs on Linux, and while we see some gains we also saw many losses. Some pipelines like matmul_id_subgroup_f32_f32_aligned_l showed 9 -14% regression while matmul_id_subgroup_f16_f32_f16acc_aligned_l showed 5 - 17% regression. It seems better to keep the feature disabled for Linux at the moment.

Microbenchmark Results

Following results are from running test-backend-ops perf -o MUL_MAT,MUL_MAT_ID,MUL_MAT_HADAMARD for b9993 and 4be81ee and comparing the performance gains/losses.

Arc B570 (Mesa 26.1.4)

Best case +25%, worst case -15%. 5.7% of the cases show +5% or more improvement and 24% of the cases show -5% or more regression
image

Arc Pro B50 (Mesa 26.1.4)

Best case +25%, worst case -14%. 9.4% of the cases show +5% or more improvement and 10% of the cases show -5% or more regression.
image

Benchmark data
Benchmark test case B570 b9993 (GFLOPS) B570 slm-reshape 4be81ee (GFLOPS) B570 (Mesa 26.1.4) 4be81ee vs b9993 B50 b9993 (GFLOPS) B50 slm-reshape 4be81ee (GFLOPS) B50 (Mesa 26.1.4) 4be81ee vs b9993
MUL_MAT(type_a=f16,type_b=f32,m=16416,n=1,k=128,bs=[8,1],nr=[4,1],per=[0,2,1,3],k_v=0,o=1): 931.03 935.16 100% 541.05 543.98 101%
MUL_MAT(type_a=f16,type_b=f32,m=128,n=1,k=16416,bs=[8,1],nr=[4,1],per=[0,1,2,3],k_v=32832,o=1): 395.68 394.52 100% 329.46 330.63 100%
MUL_MAT_HADAMARD(type_a=f32,type_b=f32,m=128,n=1,k=128,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 11.26 11.13 99% 10.86 10.39 96%
MUL_MAT_HADAMARD(type_a=f32,type_b=f32,m=64,n=1,k=64,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 3.65 3.72 102% 3.65 3.46 95%
MUL_MAT_HADAMARD(type_a=f32,type_b=f32,m=256,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 20.57 20.21 98% 19.62 18.97 97%
MUL_MAT_HADAMARD(type_a=f32,type_b=f32,m=128,n=32,k=128,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 411.67 414.06 101% 401.1 381.51 95%
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 180.67 180.64 100% 108.82 108.8 100%
MUL_MAT(type_a=f16,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 351.88 351.94 100% 216 216 100%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 347.12 347.39 100% 216.92 216.93 100%
MUL_MAT(type_a=q4_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 836.57 837.25 100% 695.76 697.65 100%
MUL_MAT(type_a=q4_1,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 988.85 987.35 100% 642.1 642.36 100%
MUL_MAT(type_a=q5_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 825.72 826.4 100% 556.64 556.89 100%
MUL_MAT(type_a=q5_1,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 594.76 593.24 100% 441.51 440.28 100%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 604.4 603.99 100% 379.75 379.73 100%
MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1790 1800 101% 1250 1250 100%
MUL_MAT(type_a=mxfp4,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 463.52 464.03 100% 354.77 354.93 100%
MUL_MAT(type_a=nvfp4,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 444.37 444.18 100% 378.69 378.21 100%
MUL_MAT(type_a=q2_K,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1970 1960 99% 1380 1380 100%
MUL_MAT(type_a=q3_K,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 919.95 926.44 101% 598.16 598.95 100%
MUL_MAT(type_a=q4_K,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 964.64 967.42 100% 729.52 730.48 100%
MUL_MAT(type_a=q5_K,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 673.13 673.74 100% 538.63 538.73 100%
MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 638.31 638.72 100% 444.46 444.3 100%
MUL_MAT(type_a=iq2_xxs,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1080 1090 101% 843.43 847.02 100%
MUL_MAT(type_a=iq2_xs,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 827.92 826.38 100% 680.81 683.99 100%
MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 551.55 544.52 99% 542.71 550.53 101%
MUL_MAT(type_a=iq3_xxs,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 972.53 982.24 101% 667.67 665.96 100%
MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1010 1030 102% 718.18 734.92 102%
MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 728.27 695.42 95% 554.78 554.94 100%
MUL_MAT(type_a=iq4_nl,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 641.69 639.88 100% 479.53 480.29 100%
MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 873.89 869.15 99% 591.57 587.46 99%
MUL_MAT(type_a=iq4_xs,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 596.84 610.44 102% 416.94 414.73 99%
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 342.64 341.95 100% 213.84 213.76 100%
MUL_MAT(type_a=f16,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 701.02 702.6 100% 432.07 431.94 100%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 707.95 705.8 100% 431.47 431.63 100%
MUL_MAT(type_a=q4_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2020 2030 100% 1350 1340 99%
MUL_MAT(type_a=q4_1,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1800 1810 101% 1230 1230 100%
MUL_MAT(type_a=q5_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1510 1520 101% 1040 1050 101%
MUL_MAT(type_a=q5_1,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1540 1530 99% 992.53 992.86 100%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1180 1180 100% 754.49 754.76 100%
MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2620 2640 101% 1760 1780 101%
MUL_MAT(type_a=mxfp4,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 497.79 494.16 99% 544.08 544.43 100%
MUL_MAT(type_a=nvfp4,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 927.63 930.61 100% 696.17 695.34 100%
MUL_MAT(type_a=q2_K,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2440 2440 100% 1910 1900 99%
MUL_MAT(type_a=q3_K,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1610 1610 100% 1030 1040 101%
MUL_MAT(type_a=q4_K,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1680 1670 99% 1260 1270 101%
MUL_MAT(type_a=q5_K,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1290 1300 101% 1010 1010 100%
MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 970.35 971.19 100% 758.48 764.64 101%
MUL_MAT(type_a=iq2_xxs,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2010 2010 100% 1360 1350 99%
MUL_MAT(type_a=iq2_xs,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1460 1480 101% 1140 1140 100%
MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 986.44 987.45 100% 945.58 856.21 91%
MUL_MAT(type_a=iq3_xxs,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1680 1660 99% 1110 1120 101%
MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1360 1220 90% 1160 1150 99%
MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1070 1060 99% 859.46 871.57 101%
MUL_MAT(type_a=iq4_nl,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1210 1210 100% 876.28 876.36 100%
MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1420 1410 99% 938.7 942.83 100%
MUL_MAT(type_a=iq4_xs,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1100 1100 100% 750.86 753.85 100%
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 533.85 533.33 100% 323.94 323.91 100%
MUL_MAT(type_a=f16,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1040 1040 100% 644.25 644.01 100%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1040 1040 100% 643.03 643.3 100%
MUL_MAT(type_a=q4_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1760 1800 102% 1790 1790 100%
MUL_MAT(type_a=q4_1,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2450 2460 100% 1630 1630 100%
MUL_MAT(type_a=q5_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1780 1790 101% 1280 1270 99%
MUL_MAT(type_a=q5_1,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2030 2000 99% 1370 1370 100%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1660 1660 100% 952.09 957.81 101%
MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2870 2880 100% 2050 2040 100%
MUL_MAT(type_a=mxfp4,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 685.04 692.8 101% 779.92 780.99 100%
MUL_MAT(type_a=nvfp4,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1220 1240 102% 952.1 936.55 98%
MUL_MAT(type_a=q2_K,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 3340 3320 99% 2330 2330 100%
MUL_MAT(type_a=q3_K,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2190 2210 101% 1380 1410 102%
MUL_MAT(type_a=q4_K,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2100 2110 100% 1650 1640 99%
MUL_MAT(type_a=q5_K,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1900 1910 101% 1400 1400 100%
MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1490 1480 99% 1070 1080 101%
MUL_MAT(type_a=iq2_xxs,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1840 1830 99% 1350 1390 103%
MUL_MAT(type_a=iq2_xs,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2380 2400 101% 1610 1530 95%
MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1210 1220 101% 1120 1120 100%
MUL_MAT(type_a=iq3_xxs,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2130 2120 100% 1390 1390 100%
MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1530 1570 103% 1460 1460 100%
MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1290 1280 99% 1100 1090 99%
MUL_MAT(type_a=iq4_nl,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1620 1620 100% 1200 1210 101%
MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1190 1170 98% 950.21 945.59 100%
MUL_MAT(type_a=iq4_xs,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1470 1480 101% 1040 1030 99%
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 599.19 603 101% 400.66 401.21 100%
MUL_MAT(type_a=f16,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1320 1320 100% 828.75 828.83 100%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1300 1300 100% 826.16 826.24 100%
MUL_MAT(type_a=q4_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2920 2940 101% 2120 2110 100%
MUL_MAT(type_a=q4_1,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 3240 3250 100% 2080 2080 100%
MUL_MAT(type_a=q5_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2480 2490 100% 1700 1680 99%
MUL_MAT(type_a=q5_1,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2720 2730 100% 1770 1770 100%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2090 2090 100% 1490 1490 100%
MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2870 2890 101% 1950 1960 101%
MUL_MAT(type_a=mxfp4,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 783.08 777.83 99% 864.76 858.5 99%
MUL_MAT(type_a=nvfp4,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1200 1200 100% 1010 1000 99%
MUL_MAT(type_a=q2_K,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 3670 3670 100% 2700 2730 101%
MUL_MAT(type_a=q3_K,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2130 2120 100% 1640 1650 101%
MUL_MAT(type_a=q4_K,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2950 2950 100% 1930 1920 99%
MUL_MAT(type_a=q5_K,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2230 2210 99% 1590 1600 101%
MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1420 1420 100% 1200 1210 101%
MUL_MAT(type_a=iq2_xxs,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2210 2190 99% 1600 1610 101%
MUL_MAT(type_a=iq2_xs,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2580 2600 101% 1690 1660 98%
MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1300 1310 101% 1140 1030 90%
MUL_MAT(type_a=iq3_xxs,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1720 1690 98% 1430 1430 100%
MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2170 2130 98% 1730 1700 98%
MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1380 1370 99% 1260 1260 100%
MUL_MAT(type_a=iq4_nl,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1600 1640 103% 1370 1380 101%
MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 844.18 844.27 100% 702.34 695.58 99%
MUL_MAT(type_a=iq4_xs,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1310 1310 100% 1130 1130 100%
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 803.64 802.54 100% 508.76 508.8 100%
MUL_MAT(type_a=f16,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1600 1600 100% 1010 1020 101%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1610 1610 100% 1030 1030 100%
MUL_MAT(type_a=q4_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 3440 3470 101% 2340 2330 100%
MUL_MAT(type_a=q4_1,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 3530 3530 100% 2270 2280 100%
MUL_MAT(type_a=q5_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2160 2180 101% 1860 1860 100%
MUL_MAT(type_a=q5_1,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2050 2050 100% 1920 1910 99%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1870 1880 101% 1620 1610 99%
MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2080 2110 101% 1760 1770 101%
MUL_MAT(type_a=mxfp4,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 889.76 879.84 99% 1010 1020 101%
MUL_MAT(type_a=nvfp4,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1300 1310 101% 1100 1100 100%
MUL_MAT(type_a=q2_K,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 3970 3990 101% 2900 2900 100%
MUL_MAT(type_a=q3_K,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2410 2390 99% 1850 1850 100%
MUL_MAT(type_a=q4_K,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2960 2950 100% 2010 2020 100%
MUL_MAT(type_a=q5_K,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2420 2440 101% 1750 1730 99%
MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1660 1650 99% 1450 1440 99%
MUL_MAT(type_a=iq2_xxs,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2240 2280 102% 1600 1550 97%
MUL_MAT(type_a=iq2_xs,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1380 1370 99% 1140 1140 100%
MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1260 1250 99% 1070 1070 100%
MUL_MAT(type_a=iq3_xxs,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1630 1620 99% 1380 1390 101%
MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2030 2020 100% 1890 1920 102%
MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1320 1320 100% 1240 1260 102%
MUL_MAT(type_a=iq4_nl,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1700 1700 100% 1460 1470 101%
MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 589.61 586.82 100% 527.7 518.19 98%
MUL_MAT(type_a=iq4_xs,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1400 1400 100% 1230 1240 101%
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1100 1110 101% 680 676.62 100%
MUL_MAT(type_a=f16,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2240 2250 100% 1430 1430 100%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2310 2300 100% 1460 1460 100%
MUL_MAT(type_a=q4_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2300 2320 101% 2530 2510 99%
MUL_MAT(type_a=q4_1,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2030 2050 101% 2090 2100 100%
MUL_MAT(type_a=q5_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2360 2360 100% 2170 2170 100%
MUL_MAT(type_a=q5_1,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2030 2150 106% 2190 2160 99%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2360 2360 100% 2130 2130 100%
MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2810 2830 101% 1770 1940 110%
MUL_MAT(type_a=mxfp4,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 991.05 998.75 101% 1080 1090 101%
MUL_MAT(type_a=nvfp4,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1110 1100 99% 961.54 954.75 99%
MUL_MAT(type_a=q2_K,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 4240 4580 108% 3310 3320 100%
MUL_MAT(type_a=q3_K,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2790 2740 98% 2110 2110 100%
MUL_MAT(type_a=q4_K,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2950 2920 99% 2140 2120 99%
MUL_MAT(type_a=q5_K,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2440 2450 100% 1950 1960 101%
MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1890 1880 99% 1790 1760 98%
MUL_MAT(type_a=iq2_xxs,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 505.14 504.45 100% 413.4 413.86 100%
MUL_MAT(type_a=iq2_xs,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 509.01 505.92 99% 425.22 427.22 100%
MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 488.42 466.81 96% 407.58 410.74 101%
MUL_MAT(type_a=iq3_xxs,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 536.22 538.44 100% 439.15 438.77 100%
MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 2270 2300 101% 1900 1870 98%
MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1640 1650 101% 1530 1540 101%
MUL_MAT(type_a=iq4_nl,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1390 1390 100% 1200 1220 102%
MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 215.22 216.39 101% 227.67 226.26 99%
MUL_MAT(type_a=iq4_xs,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 1210 1220 101% 1060 1070 101%
MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 10210 9010 88% 6100 5900 97%
MUL_MAT(type_a=f16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 20630 18690 91% 15480 14130 91%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 7040 7030 100% 4880 4640 95%
MUL_MAT(type_a=q4_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 21350 19440 91% 15370 15320 100%
MUL_MAT(type_a=q4_1,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 22150 20600 93% 15840 15940 101%
MUL_MAT(type_a=q5_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 19230 18130 94% 14290 14820 104%
MUL_MAT(type_a=q5_1,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 21910 19900 91% 15170 15300 101%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 21100 19190 91% 13910 13850 100%
MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 22100 20300 92% 17050 16630 98%
MUL_MAT(type_a=mxfp4,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 20650 18750 91% 14790 15170 103%
MUL_MAT(type_a=nvfp4,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 18910 19030 101% 14570 14520 100%
MUL_MAT(type_a=q2_K,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 21730 20400 94% 14740 14940 101%
MUL_MAT(type_a=q3_K,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 17130 16440 96% 11470 11910 104%
MUL_MAT(type_a=q4_K,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 20150 18990 94% 13680 14070 103%
MUL_MAT(type_a=q5_K,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 19900 18140 91% 13190 13100 99%
MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 18010 16680 93% 11530 12030 104%
MUL_MAT(type_a=iq2_xxs,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 16780 17880 107% 14290 13990 98%
MUL_MAT(type_a=iq2_xs,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 21340 18170 85% 14440 14430 100%
MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 16760 18030 108% 13930 13930 100%
MUL_MAT(type_a=iq3_xxs,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 19200 17660 92% 13580 13900 102%
MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 19060 18080 95% 14850 14780 100%
MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 21840 18640 85% 14560 14400 99%
MUL_MAT(type_a=iq4_nl,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 20590 19290 94% 14930 15210 102%
MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 19430 17920 92% 13230 13400 101%
MUL_MAT(type_a=iq4_xs,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): 20390 18760 92% 13840 14040 101%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048): 167.35 167.42 100% 103.81 104.09 100%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048): 345.51 345.95 100% 246.27 252.86 103%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048): 1070 1160 108% 814.84 804.19 99%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048): 1180 1240 105% 841.27 873.45 104%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048): 1180 1210 103% 937.81 1010 108%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048): 882.7 930.33 105% 658.86 660.81 100%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048): 683.11 720.7 106% 543.7 540.77 99%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048): 173.85 173.63 100% 106.58 106.37 100%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048): 347.57 347.42 100% 210.48 214.28 102%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048): 1080 1060 98% 825.16 768.75 93%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048): 620.79 629.21 101% 399.16 404.42 101%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048): 1010 986.37 98% 720.29 742.53 103%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048): 555.24 587.6 106% 428.45 427.72 100%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048): 679.85 685.09 101% 533.72 547.19 103%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048): 175.05 176.77 101% 106.91 106.77 100%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048): 351.87 351.25 100% 218.8 214.32 98%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048): 1030 1020 99% 678.66 719.51 106%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048): 627.8 612.54 98% 397.86 386.38 97%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048): 984.12 968.78 98% 688.91 668.31 97%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048): 534.17 571.49 107% 404.45 405.82 100%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048): 636.83 611.87 96% 497.48 494.87 99%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048): 262.11 238.8 91% 167.03 159.46 95%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048): 382.14 349.68 92% 403.43 347.66 86%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048): 485.39 461.12 95% 402.17 398.93 99%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048): 304.72 296.93 97% 272.03 255.55 94%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048): 317.64 350.13 110% 264.01 307.54 116%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048): 199.3 249.49 125% 175.71 211.18 120%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048): 239.94 248.53 104% 208.24 209.54 101%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048): 500.94 447.81 89% 271.48 260.43 96%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048): 747.88 697.49 93% 664.42 600.75 90%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048): 872.01 827.43 95% 650.44 648.79 100%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048): 704.13 733.59 104% 537.34 547.29 102%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048): 702.86 733.72 104% 511.38 542.98 106%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048): 597.3 592.79 99% 406.88 421.21 104%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048): 775.64 758.41 98% 602.63 606 101%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048): 743.99 634.4 85% 397.9 356.93 90%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048): 1320 1160 88% 955.85 912.05 95%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048): 1310 1210 92% 952.69 986.97 104%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048): 1300 1180 91% 867.05 892.75 103%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048): 1250 1190 95% 862.26 922.32 107%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048): 1140 1120 98% 738.9 812.04 110%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048): 1290 1160 90% 959.31 953.18 99%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048): 1440 1240 86% 785.1 704.55 90%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048): 2560 2230 87% 1880 1770 94%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048): 2530 2370 94% 1860 1920 103%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048): 2490 2300 92% 1690 1740 103%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048): 2400 2290 95% 1680 1810 108%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048): 2210 2150 97% 1440 1580 110%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048): 2500 2240 90% 1850 1850 100%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048): 2820 2420 86% 1520 1380 91%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048): 5010 4380 87% 3590 3370 94%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048): 5070 4650 92% 3460 3780 109%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048): 4990 4480 90% 3220 3320 103%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048): 4860 4650 96% 3210 3460 108%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048): 3870 3910 101% 2760 3040 110%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048): 5070 4530 89% 3610 3600 100%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048): 169.18 168.92 100% 104.51 104.64 100%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048): 342.33 341.32 100% 219.13 227.67 104%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048): 1220 1210 99% 829.82 862.16 104%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048): 1070 992.06 93% 938.86 920.1 98%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048): 1470 1540 105% 1060 1040 98%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048): 918.96 925.94 101% 660.2 613.98 93%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048): 722.83 716.76 99% 537.09 540.69 101%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048): 180.28 179.51 100% 106.94 106.96 100%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048): 371.38 360.68 97% 212.16 218.82 103%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048): 1200 1130 94% 738.27 742.8 101%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048): 690.06 608.44 88% 399.89 391.09 98%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048): 1090 1020 94% 788.65 794.45 101%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048): 585.03 633.56 108% 440.59 474.22 108%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048): 669.89 685.48 102% 567.7 545.2 96%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048): 176.19 178.58 101% 109.94 107.47 98%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048): 353.88 361.06 102% 214.02 220.97 103%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048): 1080 1060 98% 784.29 743.62 95%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048): 615.7 625.59 102% 401.4 409.32 102%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048): 987.6 997.44 101% 717.51 754.27 105%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048): 592.2 585.63 99% 434.99 435.29 100%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048): 660.88 689.88 104% 524.54 524.08 100%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048): 439.1 397.46 91% 296.93 279.75 94%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048): 641.69 588.86 92% 640.2 583.96 91%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048): 810.87 760.01 94% 682.41 651.37 95%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048): 504.6 491.66 97% 462.76 442.62 96%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048): 528.15 592.69 112% 438.05 498.57 114%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048): 334.5 417 125% 328.15 349.62 107%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048): 418.34 420.73 101% 373.4 363.66 97%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048): 948.34 856.73 90% 526.44 502.99 96%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048): 1390 1330 96% 1230 1150 93%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048): 1640 1560 95% 1230 1240 101%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048): 1340 1340 100% 1010 1050 104%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048): 1170 1350 115% 976.27 1030 106%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048): 1120 1130 101% 774.16 816.11 105%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048): 1460 1450 99% 1150 1150 100%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048): 1420 1220 86% 791.7 709.19 90%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048): 2510 2180 87% 1880 1770 94%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048): 2490 2330 94% 1850 1940 105%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048): 2440 2250 92% 1700 1740 102%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048): 2370 2060 87% 1700 1840 108%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048): 2190 2150 98% 1450 1600 110%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048): 2450 2200 90% 1860 1850 99%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048): 2730 2350 86% 1550 1390 90%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048): 4820 4180 87% 3660 3420 93%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048): 4860 4500 93% 3570 3750 105%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048): 4760 4360 92% 3300 3360 102%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048): 4650 4510 97% 3080 3540 115%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048): 4200 4170 99% 2830 3080 109%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048): 4790 4290 90% 3580 3540 99%
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048): 5320 4600 86% 3000 2700 90%
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048): 9600 8500 89% 7110 6560 92%
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048): 10030 9290 93% 7010 7450 106%
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048): 9790 8900 91% 6350 6470 102%
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048): 9450 8660 92% 6330 6750 107%
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048): 8640 8340 97% 5400 5820 108%
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048): 10010 8960 90% 7030 6930 99%
MUL_MAT_ID(type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=1,k=2880): 429.49 429.17 100% 417.53 411.4 99%
MUL_MAT_ID(type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=4,k=2880): 404.87 401.77 99% 369.88 372.17 101%
MUL_MAT_ID(type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=8,k=2880): 404.46 413.38 102% 371.67 370.12 100%
MUL_MAT_ID(type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=512,k=2880): 8720 8320 95% 6830 6990 102%

@fish-jiang

fish-jiang commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

@0cc4m could you please review this PR again? Let me know if there's anything else needed to move this PR forward.

@0cc4m

0cc4m commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Sorry, I'm currently on vacation with limited time. Can you rebase this, please? I'll get to it soon.

@fish-jiang
fish-jiang force-pushed the intel/xe-slm-a-reshape branch from a8ff697 to 06cf97d Compare August 14, 2026 08:51
@fish-jiang

fish-jiang commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @0cc4m. Rebase done.

@0cc4m

0cc4m commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

I can't test any of the gains, but I can confirm that I see no regressions.

@0cc4m
0cc4m merged commit 9b0a2ce into ggml-org:master Aug 15, 2026
27 of 29 checks passed
CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Aug 16, 2026
@fish-jiang @CowboyTim
vulkan: add SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for coopmat1 on Inte…
487b89b 
…l Xe (ggml-org#25380)

* vulkan: add SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for coopmat mul_mm on Intel Xe

* vulkan: fix shmem estimate for Intel SHMEM_STRIDE_PAD=0 in matmul_shmem_support

* cacheline aligned for shared kvalues_mxfp4

* vulkan: fix OOB read in kvalues_mxfp4 init after cacheline padding

* vulkan: restrict SLM-A reshape to Intel Windows driver, revert mxfp4 cacheline padding
brittlewis12 pushed a commit to brittlewis12/llama.cpp that referenced this pull request Aug 17, 2026
@fish-jiang @brittlewis12
vulkan: add SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for coopmat1 on Inte…
e948a06 
…l Xe (ggml-org#25380)

* vulkan: add SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for coopmat mul_mm on Intel Xe

* vulkan: fix shmem estimate for Intel SHMEM_STRIDE_PAD=0 in matmul_shmem_support

* cacheline aligned for shared kvalues_mxfp4

* vulkan: fix OOB read in kvalues_mxfp4 init after cacheline padding

* vulkan: restrict SLM-A reshape to Intel Windows driver, revert mxfp4 cacheline padding
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jeffbolznv jeffbolznv jeffbolznv approved these changes
@0cc4m 0cc4m 0cc4m approved these changes

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@fish-jiang @jeffbolznv @rillomas @0cc4m