Skip to content

Add Q5_K ESIMD kernel - #26376

Merged
ggerganov merged 10 commits into
ggml-org:masterfrom
aicss-genai:esimd-q5k
Aug 21, 2026
Merged

Add Q5_K ESIMD kernel#26376
ggerganov merged 10 commits into
ggml-org:masterfrom
aicss-genai:esimd-q5k

Conversation

@malsbat

@malsbat malsbat commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Overview

This completes the usage of ESIMD for DMMV QK kernels begun in #25942.

Measured tg improvement of 1.04x to 1.35x (depending on hardware) using Qwen35-27B-Q5_K_M.

Additional information

Note: all measurements were captured with aicss-genai/fix-row-calculation (already merged in master) and aicss-genai/fattn-single-device-sync (better solution already merged in master, will only improve pp results below) merged as the parent branch predates those necessary patches.

PVC performance

Baseline

export GGML_SYCL_PRIORITIZE_DMMV=0
export GGML_SYCL_ENABLE_ESIMD=0
model size params backend ngl test t/s
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 pp512 908.19 ± 3.04
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 tg128 17.25 ± 0.00

DMMV

export GGML_SYCL_PRIORITIZE_DMMV=1
export GGML_SYCL_ENABLE_ESIMD=0
model size params backend ngl test t/s
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 pp512 904.31 ± 7.70
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 tg128 11.84 ± 0.00

ESIMD

export GGML_SYCL_PRIORITIZE_DMMV=0
export GGML_SYCL_ENABLE_ESIMD=1
model size params backend ngl test t/s
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 pp512 901.58 ± 5.68
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 tg128 23.29 ± 0.03

B70 performance

Baseline

model size params backend ngl test t/s
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 pp512 826.10 ± 2.80
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 tg128 21.88 ± 0.04

DMMV

model size params backend ngl test t/s
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 pp512 829.38 ± 6.64
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 tg128 12.87 ± 0.05

ESIMD

model size params backend ngl test t/s
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 pp512 833.75 ± 3.07
qwen35 27B Q5_K - Medium 18.25 GiB 26.90 B SYCL 100 tg128 22.69 ± 0.05

Perplexity tests across wikitext, hellaswag, and winogrande showed no significant changes between baseline and ESIMD.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, used Claude heavily to author new kernel based on existing patterns.

malsbat added 8 commits July 20, 2026 21:43
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
@malsbat

malsbat commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Same as the others, I will rebase this once #25942 is merged.

@github-actions github-actions Bot added documentation Improvements or additions to documentation ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Jul 31, 2026
@malsbat malsbat changed the title Esimd q5k Add Q5_K ESIMD kernel Jul 31, 2026

@arthw arthw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With gemma-4-E2B-it-Q5_K_M.gguf on B60

Test fa Base t/s Primary t/s Increase Rate (Primary vs Base)
pp512 0 2154.07 2156.09 0.09%
pp512 1 2143.85 2133.25 -0.49%
tg128 0 99.42 104.56 5.17%
tg128 1 99.12 103.82 4.74%

@arthw arthw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good job!

Thank you!

@arthw arthw added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 4, 2026
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
@malsbat
malsbat marked this pull request as ready for review August 13, 2026 16:58
@malsbat
malsbat requested a review from a team as a code owner August 13, 2026 16:58
@ggerganov
ggerganov merged commit 9e89a19 into ggml-org:master Aug 21, 2026
29 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants