Skip to content

CUDA: fix kernel selection logic for tile FA - #19686

Merged
JohannesGaessler merged 2 commits into
ggml-org:masterfrom
JohannesGaessler:cuda-fa-tile-fix-dkq576
Feb 19, 2026
Merged

CUDA: fix kernel selection logic for tile FA#19686
JohannesGaessler merged 2 commits into
ggml-org:masterfrom
JohannesGaessler:cuda-fa-tile-fix-dkq576

Conversation

@JohannesGaessler

Copy link
Copy Markdown
Contributor

Fixes #19652 .

The problem is that the kernel selection logic is slightly wrong so the host code runs into an abort.

@github-actions github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Feb 17, 2026
@pwilkin

pwilkin commented Feb 18, 2026

Copy link
Copy Markdown
Member

@JohannesGaessler CI seems to dislike something:

  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)
  Backend CUDA0: FAIL

@JohannesGaessler

Copy link
Copy Markdown
Contributor Author
2026-02-17T14:06:08.1940247Z [MUL_MAT] ERR = 0.000589984 > 0.000500000   MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): �[1;31mFAIL�[0m

The failing test is a matrix multiplication which is 100% unaffected by the changes in this PR. Since we are using random seeds in the tests it can happen from time to time that the small differences between the CUDA and CPU backends randomly exceed the tolerance we set, particularly if the input matrices are small and the differences are less likely to average out.

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a comment/reference of the DV limit would be useful.

@JohannesGaessler
JohannesGaessler merged commit c78e682 into ggml-org:master Feb 19, 2026
1 check passed
@pwilkin

pwilkin commented Feb 19, 2026

Copy link
Copy Markdown
Member

The failing test is a matrix multiplication which is 100% unaffected by the changes in this PR. Since we are using random seeds in the tests it can happen from time to time that the small differences between the CUDA and CPU backends randomly exceed the tolerance we set, particularly if the input matrices are small and the differences are less likely to average out.

Aight, just throwing out an idea: maybe it would be a good idea to vary the permitted NMSE divergence according to tensor size, to avoid false positives like this?

liparetejas pushed a commit to liparetejas/llama.cpp that referenced this pull request Feb 23, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
bartowski1182 pushed a commit to bartowski1182/llama.cpp that referenced this pull request Mar 2, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
ArberSephirotheca pushed a commit to ArberSephirotheca/llama.cpp that referenced this pull request Mar 3, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request May 30, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
MrLordCat referenced this pull request in MrLordCat/llama.cpp-rdna-lab Jul 16, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
* CUDA: fix kernel selection logic for tile FA

* add comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Misc. bug: GGML FlashAttn Crash on NVIDIA Card

3 participants