Skip to content

sycl: fix row calculation when K_QUANTS_PER_ITERATION is 1 - #25690

Merged
ggerganov merged 5 commits into
ggml-org:masterfrom
aicss-genai:fix-row-calculation
Jul 17, 2026
Merged

sycl: fix row calculation when K_QUANTS_PER_ITERATION is 1#25690
ggerganov merged 5 commits into
ggml-org:masterfrom
aicss-genai:fix-row-calculation

Conversation

@malsbat

@malsbat malsbat commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Overview

As noted by in @logari81 in #25063 (comment), multi-turn conversation returned corrupt results after the introduction of #25063.

The root cause is an incorrect row calculation that caused the kernel to overwrite the same row.

Additional issues found while validating this:

  • The reordered q5_K kernel only processed one half of each reordered block. Fix the reordered kernel to match the correct unordered kernel.
  • The unordered q5_k kernel did not use the K_QUANTS_PER_ITERATION define. This was the only K kernel to not do so.
  • The row guard check was incorrect in several of the kernels that could lead to an invalid row access.

Additional information

Validated with test-backend-ops test -o MUL_MAT for permutations of K_QUANTS_PER_ITERATION, GGML_SYCL_ENABLE_OPT, and GGML_SYCL_PRIORITIZE_DMMV.

Manual validation using multi-turn chat to confirm reasonable responses: llama-cli -m /models/Qwen3.6-35B-A3B-Q4.gguf -ngl 99.

Requirements

  • I have read and agree with the contributing guidelines: YES
  • AI usage disclosure: YES, used to help identify root cause of both the incorrect row calculation and q5_k reordered kernel error.

malsbat added 4 commits July 14, 2026 22:22
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
This is the only Q5_K kernel that was not using KQPI.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Error found while running

  GGML_SYCL_PRIORITIZE_DMMV=1 \
  build/bin/test-backend-ops test -o MUL_MAT

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
@malsbat
malsbat requested a review from a team as a code owner July 14, 2026 22:38
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Jul 14, 2026

@arthw arthw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you share the test method to trigger the issue to be fixed?
or
Let's wait for if it can fix issue: #25455

Thank you!

@malsbat

malsbat commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@arthw, of course:

Prior to the PR:

git checkout 00fa7cb284cbf133fc426733bd64238a3588a33e
# build
cmake -B build -DGGML_NATIVE=OFF -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=ON -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --config Release -j$(nproc)

First test, bad output on second prompt:

./build/bin/llama-cli -m /models/Qwen3.6-35B-A3B-Q4.gguf -ngl 99
> how do i bake a cake?
# good response
> customize it for chocolate
# incomplete response

Second test, MUL_MAT_OP:

GGML_SYCL_PRIORITIZE_DMMV=1 ./build/bin/test-backend-ops test -o MUL_MAT
# ...
Failing tests:
  MUL_MAT(type_a=q5_K,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)
  MUL_MAT(type_a=q5_K,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)
  Backend SYCL0: FAIL

After PR, good CLI responses and passing all MUL_MAT_OP tests.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

@arthw arthw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good job!

Thank you!

@arthw arthw added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Jul 17, 2026
@ggerganov
ggerganov merged commit 0bd0ec6 into ggml-org:master Jul 17, 2026
25 of 28 checks passed
CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Jul 21, 2026
…25690)

* sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel

This is the only Q5_K kernel that was not using KQPI.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: add missing second half processing to reordered q5_k

Error found while running

  GGML_SYCL_PRIORITIZE_DMMV=1 \
  build/bin/test-backend-ops test -o MUL_MAT

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix potential off-by-one error

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix missing row > nrows check

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
ggerganov pushed a commit to am17an/llama.cpp that referenced this pull request Jul 28, 2026
…25690)

* sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel

This is the only Q5_K kernel that was not using KQPI.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: add missing second half processing to reordered q5_k

Error found while running

  GGML_SYCL_PRIORITIZE_DMMV=1 \
  build/bin/test-backend-ops test -o MUL_MAT

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix potential off-by-one error

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix missing row > nrows check

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026
…25690)

* sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel

This is the only Q5_K kernel that was not using KQPI.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: add missing second half processing to reordered q5_k

Error found while running

  GGML_SYCL_PRIORITIZE_DMMV=1 \
  build/bin/test-backend-ops test -o MUL_MAT

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix potential off-by-one error

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix missing row > nrows check

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 12, 2026
…25690)

* sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel

This is the only Q5_K kernel that was not using KQPI.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: add missing second half processing to reordered q5_k

Error found while running

  GGML_SYCL_PRIORITIZE_DMMV=1 \
  build/bin/test-backend-ops test -o MUL_MAT

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix potential off-by-one error

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix missing row > nrows check

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants