Add DMMV Q4_K and Q6_K ESIMD kernels

[malsbat]

Overview

Using optimized ESIMD kernels with the Qwen3-14B-Q4_K_M model gives a tg performance boost (on B70) of 1.05x against MMVQ on B70 and 1.46x against DMMV. On PVC, the gains are more significant: 1.31x against MMVQ and 1.76x against DMMV.

The ESIMD kernels are based on kernels used in the archived ipex-llm project. As measured with VTune, the average kernel time dropped 25-46%, read bandwidth rose 18-53%, and SEND instructions dropped 77-87% when using the ESIMD kernels instead of the existing DMMV ones.

I'd like to get feedback on:

1. Is the introduction of ESIMD kernels to the project acceptable?
2. The kernels are currently configured with a cmake build flag. I expect this is not the best approach and something more in-line with the GGML_SYCL_PRIORITIZE_DMMV flag may be preferred. Please advise.
3. This PR only covers the Q4_K and Q6_K kernels, the hot kernels of the Qwen3-14B-Q4_K_M model (from sycl: Battlemage (BMG) optimizations — AOT, Q5_K reorder, PAD stride fix, new ops, oneMKL routing #22066 (comment)). I can add wider support in additional PRs.

Additional information

Build recipe

export SYCL_PROGRAM_COMPILE_OPTIONS="-cl-fp32-correctly-rounded-divide-sqrt"
# baseline
cmake -B build -DGGML_NATIVE=OFF -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=ON -DGGML_SYCL_F16=ON
cmake --build build --config Release -j$(nproc)
# esimd
cmake -B build-esimd -DGGML_NATIVE=OFF -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_SYCL_ESIMD=ON -DLLAMA_BUILD_TESTS=ON -DGGML_SYCL_F16=ON
cmake --build build-esimd --config Release -j$(nproc)

Performance

The measurements were built using this branch, merged with aicss-genai/fattn-single-device-sync.

B70

MMVQ baseline

export GGML_SYCL_PRIORITIZE_DMMV=0
./build/bin/llama-bench -m /models/Qwen3-14B-Q4_K_M.gguf -ngl 100

model	size	params	backend	ngl	test	t/s
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	pp512	1752.26 ± 3.90
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	tg128	49.03 ± 0.18

DMMV baseline

export GGML_SYCL_PRIORITIZE_DMMV=1
./build/bin/llama-bench -m /models/Qwen3-14B-Q4_K_M.gguf -ngl 100

model	size	params	backend	ngl	test	t/s
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	pp512	1743.50 ± 7.20
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	tg128	35.40 ± 0.23

DMMV esimd

export GGML_SYCL_PRIORITIZE_DMMV=1
./build-esimd/bin/llama-bench -m /models/Qwen3-14B-Q4_K_M.gguf -ngl 100

model	size	params	backend	ngl	test	t/s
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	pp512	1752.81 ± 6.32
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	tg128	51.62 ± 0.14

PVC

MMVQ baseline

model	size	params	backend	ngl	test	t/s
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	pp512	2006.39 ± 11.89
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	tg128	35.64 ± 0.01

DMMV baseline

model	size	params	backend	ngl	test	t/s
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	pp512	1997.46 ± 24.01
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	tg128	26.67 ± 0.00

DMMV esimd

model	size	params	backend	ngl	test	t/s
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	pp512	2008.86 ± 15.43
qwen3 14B Q4_K - Medium	8.38 GiB	14.77 B	SYCL	100	tg128	46.81 ± 0.03

Perplexity

DMMV baseline vs esimd (B70)

# ./build/bin/llama-perplexity -m /models/Qwen3-14B-Q4_K_M.gguf -f wikitext-2-raw/wiki.test.raw
4.20.019.786 I Final estimate: PPL = 9.1146 +/- 0.07155
# ./build-esimd/bin/llama-perplexity -m /models/Qwen3-14B-Q4_K_M.gguf -f wikitext-2-raw/wiki.test.raw
4.02.712.891 I Final estimate: PPL = 9.1147 +/- 0.07155

# ./build/bin/llama-perplexity --hellaswag -f hellaswag_val_full.txt -m /models/Qwen3-14B-Q4_K_M.gguf
task    acc_norm        95% confidence interval
400     78.25000000%    [73.9487%, 82.0138%]
# ./build-esimd/bin/llama-perplexity --hellaswag -f hellaswag_val_full.txt -m /models/Qwen3-14B-Q4_K_M.gguf
task    acc_norm        95% confidence interval
400     78.25000000%    [73.9487%, 82.0138%]

# ./build/bin/llama-perplexity -f winogrande-debiased-eval.csv --winogrande -m /models/Qwen3-14B-Q4_K_M.gguf
2.19.350.501 I Final Winogrande score(1267 tasks): 71.9021 +/- 1.2633
# ./build-esimd/bin/llama-perplexity -f winogrande-debiased-eval.csv --winogrande -m /models/Qwen3-14B-Q4_K_M.gguf
2.19.452.176 I Final Winogrande score(1267 tasks): 71.9021 +/- 1.2633

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, used claude extensively to help analyze disassembled ipex-llm kernels and iterate on optimization strategies guided by that analysis.

[arthw]

I suggest adding a debug env var like GGML_SYCL_ENABLE_DNN for ESIMD.
So that we could easy to debug the new issue in the future.

[arthw]

Suggest setting it as ON as default

[malsbat]

I removed this and just use the __INTEL_LLVM_COMPILER define now to determine whether to build or not. I added GGML_SYCL_ENABLE_ESIMD env var to control on/off at runtime.

[Titaniumtown]

Very impressive performance uplift @malsbat! I'll try and test this locally sometime on my B70.

EDIT: I can reproduce the performance uplift. So cool!

[arthw]

Qwen3.5-4B-Q4_K_M.gguf on B60

Big perf increase with GGML_SYCL_PRIORITIZE_DMMV=1.

GGML_SYCL_PRIORITIZE_DMMV=1

Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
pp512	0	1046.83	1048.09	0.12%
pp512	1	1047.63	1048.46	0.08%
tg128	0	71.23	86.77	21.82%
tg128	1	71.60	87.24	21.84%

Qwen3.5-4B-Q4_K_M.gguf
GGML_SYCL_PRIORITIZE_DMMV=0

Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
pp512	0	1048.80	1047.96	-0.08%
pp512	1	1048.99	1048.52	-0.04%
tg128	0	80.04	80.05	0.01%
tg128	1	80.58	80.57	-0.01%

[arthw]

Since ESIMD depend on GGML_SYCL_PRIORITIZE_DMMV=1.
Based on the rule: we always set the default values of env vars to get the best performance for common cases.
I suggest set ESIMD enable as default, so GGML_SYCL_PRIORITIZE_DMMV can be enabled in same time.

Like,
GGML_SYCL_ENABLE_ESIMD=1 is default value when user not set it.
In initial, when GGML_SYCL_ENABLE_ESIMD=1, set the intel var of GGML_SYCL_PRIORITIZE_DMMV to 1 whenever user set it or not.
If user set GGML_SYCL_ENABLE_ESIMD=0, the value of GGML_SYCL_PRIORITIZE_DMMV depend on the env var as legacy.

[malsbat]

Since ESIMD depend on GGML_SYCL_PRIORITIZE_DMMV=1. Based on the rule: we always set the default values of env vars to get the best performance for common cases. I suggest set ESIMD enable as default, so GGML_SYCL_PRIORITIZE_DMMV can be enabled in same time.

Like, GGML_SYCL_ENABLE_ESIMD=1 is default value when user not set it. In initial, when GGML_SYCL_ENABLE_ESIMD=1, set the intel var of GGML_SYCL_PRIORITIZE_DMMV to 1 whenever user set it or not. If user set GGML_SYCL_ENABLE_ESIMD=0, the value of GGML_SYCL_PRIORITIZE_DMMV depend on the env var as legacy.

I'm not positive we want to do this just yet. I did some quick smoke tests across some more models and with ESIMD=1,DMMV=1 for example the decode performance of Qwen3.5-9B-Q8-latest measured only 81% of ESIMD=0,DMMV=0.

I suggest we wait until the rest of the kernels are supported via the ESIMD path and beat the MMVQ performance. The other alternative I can think of would be more complex logic in ggml_sycl_mul_mat to decide which one to use.

[malsbat]

@arthw, I added code to use ESIMD by default when support is available regardless of the DMMV setting. This removes the regression I mentioned earlier with Qwen3.5-9B-Q8-latest

I also noticed something else interesting: performance went down when DMMV is not enabled in some cases. It appears that the existing DMMV implementation, at least on B70, is better than MMVQ for some kernels (Q5_K at a minimum, I haven't investigated further).

For example with both ESIMD and DMMV:

Qwen3.5-27B-Q4-latest    tg   1024               24.30 ±0.00              26.12 ±0.00   1.075x

With only ESIMD:

Qwen3.5-27B-Q4-latest    tg   1024               24.28 ±0.00              25.00 ±0.00   1.030x

Still better, but less so. Adding Q5_K to the switch in the most recent commit restored the improvement from 1.030x back to 1.075x.

[arthw]

@malsbat
OK, I see.

Is it possible to update the above info to the guide of GGML_SYCL_ENABLE_ESIMD in SYCL.md?
Guide user to find the best feature pair.

[malsbat]

@arthw, I measured again with mutiple reps, I think I was just seeing measurement noise above. I'm going to mark this PR as ready for review now. I'll have the other QK kernels available soon in a separate PR.

[arthw]

It's good job!

ESIMD will bring surprise to all users.

Thank you!

[malsbat]

#26251 got merged ahead of this one, so the only difference is now the removal of the copyright header.

[malsbat]

I folded the copyright removal into #26376. I'm going to close this PR since the contents got merged as part of #26251.