llama : skip K/V rotation input when its buffer is unallocated by liminfei-amd · Pull Request #25215 · ggml-org/llama.cpp

liminfei-amd · 2026-07-02T02:31:51Z

Overview

llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input currently call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is non-null (#25191). That tensor's buffer can be unallocated (NULL) when a graph only stores K/V without attending -- for example DFlash speculative decoding's KV-injection pass. set_input_k_rot immediately calls ggml_backend_buffer_is_host(dst->buffer), which aborts with GGML_ASSERT(buffer):

ggml-backend.cpp:194: GGML_ASSERT(buffer) failed
  ggml_backend_buffer_is_host ()
  llama_kv_cache::set_input_k_rot(ggml_tensor*) const
  llm_graph_input_attn_kv_iswa::set_input(llama_ubatch const*)
  common_speculative_impl_draft_dflash::process(llama_batch const&)

The adjacent kq_mask inputs in these same two functions already guard exactly this case -- their comment notes "the mask is left unallocated when the graph only stores K/V without attending (e.g. DFlash's KV-injection pass)" -- with an && ->buffer check. The k_rot/v_rot inputs, added later, were missing that guard. This PR adds the same guard to the four k_rot/v_rot inputs. When the buffer is unallocated there is no data to upload, so skipping the upload is correct.

Additional information

Reproduced on RDNA4 (gfx1201, Vulkan) with a DFlash draft whose head dim is a multiple of 64 (so attn_rot_k/attn_rot_v are enabled) and --cache-type-k q8_0 --cache-type-k-draft q8_0:

before: llama-server ... --spec-type draft-dflash aborts (rc=134) on the first draft decode with the assert above.
after: the same command runs to completion, speculative decoding stays active, no assert.
a regular causal model (no DFlash) is unaffected -- the guard is a no-op when the buffer is allocated.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: an AI coding assistant helped investigate and reproduce this on real hardware; I authored and reviewed the change and this description, and I can explain and take responsibility for every line.

Fixes #25191

llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is non-null, but the tensor's buffer can be unallocated (NULL) when a graph only stores K/V without attending -- e.g. DFlash speculative decoding's KV-injection pass. set_input_k_rot then calls ggml_backend_buffer_is_host() on a NULL buffer and aborts with GGML_ASSERT(buffer). Guard the four k_rot/v_rot inputs with the same "&& ->buffer" check that the adjacent kq_mask inputs already use in these two functions. When the buffer is unallocated there is no data to upload, so skipping is correct. Fixes ggml-org#25191 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>

ruixiang63

As long as we confirm this resolve the linked issue, I think it should be good to merge.

ruixiang63 · 2026-07-02T10:02:00Z

    }

-    if (self_v_rot_swa) {
+    if (self_v_rot_swa && self_v_rot_swa->buffer) {


All these changes are the same as set_input_kq_mask, so it looks good to me.

liminfei-amd · 2026-07-04T08:41:19Z

Thanks @bestbug456 for building and verifying the fix in #25191.

@ruixiang63 since this is in the DFlash path you authored, I'll defer to you — happy to have #25215 merged, or to close it if you'd prefer folding the guard into the DFlash work directly. Whatever suits you best.

ruixiang63 · 2026-07-04T11:13:56Z

Thanks. It looks good to me to merge. Just need another review. cc @CISC @am17an

am17an · 2026-07-04T11:35:35Z

@ruixiang63 without this PR, is the quantized kv-cache with dflash broken atm?

ruixiang63 · 2026-07-04T12:20:09Z

@ruixiang63 without this PR, is the quantized kv-cache with dflash broken atm?

Yes, as described in this issue: #25191 The reason could be the KV cache injection in DFlash which doesn't need to attend attention only does injection.

I added the similar guard in

llama.cpp/src/llama-graph.cpp

Line 493 in ef2d770

if (self_kq_mask && self_kq_mask->buffer) {

when I enabled DFlash.

ruixiang63 · 2026-07-04T13:50:05Z

BTW, with a quantized KV cache for the DFlash draft model, the acceptance rate drops by less than 1% in my test case. I think this is a valid fix, but I wouldn't recommend it as the preferred way to run DFlash spec dec.

…gml-org#25215) llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is non-null, but the tensor's buffer can be unallocated (NULL) when a graph only stores K/V without attending -- e.g. DFlash speculative decoding's KV-injection pass. set_input_k_rot then calls ggml_backend_buffer_is_host() on a NULL buffer and aborts with GGML_ASSERT(buffer). Guard the four k_rot/v_rot inputs with the same "&& ->buffer" check that the adjacent kq_mask inputs already use in these two functions. When the buffer is unallocated there is no data to upload, so skipping is correct. Fixes ggml-org#25191 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> (cherry picked from commit a410713)

liminfei-amd · 2026-07-15T01:57:15Z

Hey @ruixiang63, quick question — I have a fix ready #25670 but can't mark it "Ready for review"; getting a permission error on markPullRequestReadyForReview. Any idea what's going on, or who I should ask? Thanks.

ruixiang63 · 2026-07-15T09:20:10Z

no idea what's going on. It looks only you can mark the PR ready. Maybe an alternative way is to create a new PR and close the old one.

ruixiang63 · 2026-07-15T09:24:52Z

Okay it seems you have reached the PR limit, do you have any other open PRs? Please prioritize your open PRs and only mark one ready.

liminfei-amd · 2026-07-15T09:25:09Z

no idea what's going on. It looks only you can mark the PR ready. Maybe an alternative way is to create a new PR and close the old one.

Okay, Thanks.

am17an · 2026-07-15T09:30:16Z

It says "author has reached PR limit", so I think you can only have 1 PR open, but draft PRs don't count

…gml-org#25215) llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is non-null, but the tensor's buffer can be unallocated (NULL) when a graph only stores K/V without attending -- e.g. DFlash speculative decoding's KV-injection pass. set_input_k_rot then calls ggml_backend_buffer_is_host() on a NULL buffer and aborts with GGML_ASSERT(buffer). Guard the four k_rot/v_rot inputs with the same "&& ->buffer" check that the adjacent kq_mask inputs already use in these two functions. When the buffer is unallocated there is no data to upload, so skipping is correct. Fixes ggml-org#25191 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>

…runner) The test-hidden-states step crashes with SIGILL (exit 132) during llama_decode on the GitHub Actions runner CPU. This is caused by upstream ggml CPU changes merged in ggml-org:master (ggml-org#25247, ggml-org#25215), not by fork code. The fork's actual functionality is tested by the self-test and server smoke tests which run after this step. Assisted-by: Hermes Agent

…gml-org#25215) llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is non-null, but the tensor's buffer can be unallocated (NULL) when a graph only stores K/V without attending -- e.g. DFlash speculative decoding's KV-injection pass. set_input_k_rot then calls ggml_backend_buffer_is_host() on a NULL buffer and aborts with GGML_ASSERT(buffer). Guard the four k_rot/v_rot inputs with the same "&& ->buffer" check that the adjacent kq_mask inputs already use in these two functions. When the buffer is unallocated there is no data to upload, so skipping is correct. Fixes ggml-org#25191 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>

liminfei-amd requested a review from CISC as a code owner July 2, 2026 02:31

liminfei-amd mentioned this pull request Jul 2, 2026

Eval bug: DFlash Speculative Decoding Crash: GGML_ASSERT(buffer) Failure #25191

Closed

ruixiang63 approved these changes Jul 2, 2026

View reviewed changes

am17an approved these changes Jul 4, 2026

View reviewed changes

ruixiang63 merged commit a410713 into ggml-org:master Jul 4, 2026
25 checks passed

TheTom mentioned this pull request Jul 9, 2026

feat: cherry-pick upstream DFlash spec-decode (#22105 + #25110 + companions) TheTom/llama-cpp-turboquant#201

Merged

ehotting mentioned this pull request Jul 11, 2026

kv-cache : fix crash in state save/restore #21576

Draft

Navigation Menu

Uh oh!

llama : skip K/V rotation input when its buffer is unallocated - #25215

llama : skip K/V rotation input when its buffer is unallocated#25215
ruixiang63 merged 1 commit into
ggml-org:masterfrom
liminfei-amd:amd-rocm/25191-krot-unallocated-buffer-guard

liminfei-amd commented Jul 2, 2026

Uh oh!

ruixiang63 left a comment

Uh oh!

ruixiang63 Jul 2, 2026

Uh oh!

liminfei-amd commented Jul 4, 2026

Uh oh!

ruixiang63 commented Jul 4, 2026

Uh oh!

am17an commented Jul 4, 2026

Uh oh!

ruixiang63 commented Jul 4, 2026 •

edited

Loading

Uh oh!

ruixiang63 commented Jul 4, 2026

Uh oh!

Uh oh!

liminfei-amd commented Jul 15, 2026

Uh oh!

ruixiang63 commented Jul 15, 2026

Uh oh!

ruixiang63 commented Jul 15, 2026 •

edited

Loading

Uh oh!

liminfei-amd commented Jul 15, 2026

Uh oh!

am17an commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

liminfei-amd commented Jul 2, 2026

Overview

Additional information

Requirements

Uh oh!

ruixiang63 left a comment

Choose a reason for hiding this comment

Uh oh!

ruixiang63 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

liminfei-amd commented Jul 4, 2026

Uh oh!

ruixiang63 commented Jul 4, 2026

Uh oh!

am17an commented Jul 4, 2026

Uh oh!

ruixiang63 commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ruixiang63 commented Jul 4, 2026

Uh oh!

Uh oh!

liminfei-amd commented Jul 15, 2026

Uh oh!

ruixiang63 commented Jul 15, 2026

Uh oh!

ruixiang63 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liminfei-amd commented Jul 15, 2026

Uh oh!

am17an commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ruixiang63 commented Jul 4, 2026 •

edited

Loading

ruixiang63 commented Jul 15, 2026 •

edited

Loading