Skip to content

dont (immedietly) promote fa-auto -> fa-on - #26553

Open
JorisBolsens wants to merge 1 commit into
ggml-org:masterfrom
JorisBolsens:fix-fa-cpu
Open

dont (immedietly) promote fa-auto -> fa-on#26553
JorisBolsens wants to merge 1 commit into
ggml-org:masterfrom
JorisBolsens:fix-fa-cpu

Conversation

@JorisBolsens

Copy link
Copy Markdown

Overview

in #25871 if FA was set to AUTO and ctv is quantized, then it automatically sets FA to enabled. That causes the check on src/llama-context.cpp:550 to never run, and causes the guard on src/llama-context.cpp:459 to never trip. Which means flash attention automatically offloads to CPU in case it is not supported in backend (often when ctk != ctv)
There is no log line showing this is happening, which means performance is quietly degraded, and debugging is very difficult.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES
    • AI was used to identify why performance of the model was (much) lower than expected, and was used to identify the code responsible for the issue.

@JorisBolsens
JorisBolsens requested a review from ggerganov as a code owner August 4, 2026 06:10
@HermiG

HermiG commented Aug 4, 2026

Copy link
Copy Markdown

This was a gotcha for me when I didn't realize my llama-server was compiled without FA_ALL_QUANTS. Silently offloading to CPU should not be a default behavior; such a warning would have saved a lot of confusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants