Skip to content

llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized - #25871

Merged
ggerganov merged 3 commits into
ggml-org:masterfrom
fairydreaming:force-fa-quant-kv-cache
Jul 31, 2026
Merged

llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized#25871
ggerganov merged 3 commits into
ggml-org:masterfrom
fairydreaming:force-fa-quant-kv-cache

Conversation

@fairydreaming

@fairydreaming fairydreaming commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR fixes issue described in detail here. It does it by:

  • making sure that K cache type and V cache type are the same for DeepSeek V4, so that any code referring to V cache type makes correct decisions,
  • auto-enabling FA if V cache is quantized or failing if user wants to use quantized V cache with FA explicitly disabled.

Fixes #25851

Requirements

@fairydreaming
fairydreaming requested a review from ggerganov as a code owner July 18, 2026 19:16
fairydreaming
fairydreaming commented Jul 18, 2026
View reviewed changes
Comment thread src/llama-context.cpp Outdated
}
}

if (model->arch == LLM_ARCH_DEEPSEEK4 && params.type_k != params.type_v) {

@fairydreaming fairydreaming Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this currently applies also to DeepSeek V3.2 and models where hparams.is_mla() is true since they use a view of K cache as V cache, maybe it's worth extending the condition?

@ggerganov ggerganov Jul 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sounds good

ggerganov
ggerganov approved these changes Jul 27, 2026
View reviewed changes
Comment thread src/llama-context.cpp Outdated
}
}

if (model->arch == LLM_ARCH_DEEPSEEK4 && params.type_k != params.type_v) {

@ggerganov ggerganov Jul 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sounds good

@ggerganov
ggerganov merged commit 69e62fc into ggml-org:master Jul 31, 2026
21 of 26 checks passed
edwinbrowwn pushed a commit to edwinbrowwn/llama.cpp-rdna2 that referenced this pull request Aug 1, 2026
@fairydreaming @sszymczy @edwinbrowwn
llama : enforce the same K and V cache types for DeepSeek V4; enable …
82d41e1 
…FA if V cache is quantized (ggml-org#25871)

* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized

* llama : enforce the same K and V cache types for MLA models

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
kashif pushed a commit to kashif/llama.cpp that referenced this pull request Aug 2, 2026
@fairydreaming @sszymczy @kashif
llama : enforce the same K and V cache types for DeepSeek V4; enable …
0615322 
…FA if V cache is quantized (ggml-org#25871)

* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized

* llama : enforce the same K and V cache types for MLA models

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
huaxel pushed a commit to huaxel/CachyLLama that referenced this pull request Aug 2, 2026
@fairydreaming @sszymczy @huaxel
llama : enforce the same K and V cache types for DeepSeek V4; enable …
82cb82f 
…FA if V cache is quantized (ggml-org#25871)

* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized

* llama : enforce the same K and V cache types for MLA models

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
@JorisBolsens JorisBolsens mentioned this pull request Aug 4, 2026
Open
smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026
@fairydreaming @sszymczy @smalinin
llama : enforce the same K and V cache types for DeepSeek V4; enable …
612202a 
…FA if V cache is quantized (ggml-org#25871)

* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized

* llama : enforce the same K and V cache types for MLA models

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
NighmareGit pushed a commit to NighmareGit/atomic-llama-cpp-turboquant that referenced this pull request Aug 8, 2026
@worthant
♻️ chore(merge): sync upstream llama.cpp through 1c3c967
af08eaf 
Merges 251 upstream commits on top of the fork's 392. Base was 22b208b
(2026-07-15).

What this brings in for DeepSeek V4:

- CUDA kernels for the hyper-connection ops and the lightning indexer
  (dsv4-hc.cu, lightning-indexer.cu, upstream ggml-org#25585 and ggml-org#25545). These
  landed upstream after our base, so the graph no longer needs a CPU
  fallback for those ops.
- MTP and DSpark support (ggml-org#25784), the wo_a reshape fix on load, and the
  same-K/V-cache-type enforcement (ggml-org#25871).
- Exclusion of the i32 ffn_gate_tid2eid routing table from quantization,
  which the fork did not carry.

Conflict resolution kept both architectures everywhere the two sides
touched the same code:

- llama-kv-cache: kept the fork's default-off attention-rotation policy
  and its env overrides, took upstream's GLM_DSA addition to the DSA
  indexer arch list.
- llama-context: moved the TurboQuant flash-attention auto-enable above
  upstream's generic quantized-V check, which would otherwise reject
  turbo cache types under -fa off, and dropped the fork's older V-cache
  check in favour of upstream's.
- mmq.cuh: kept the fork's int64 offsets in all three of upstream's new
  NVFP4 branches.
- fattn.cu: dropped the WMMA block, since upstream removed that kernel
  and its helpers entirely; kept the RDNA4 turbo path.
- ggml-cuda.cu: kept the host-staged cross-device copy and routed its
  peer copy through upstream's new virtual-to-physical device mapping.
- chat.cpp: rebuilt on upstream's file with the fork's Inkling and
  Laguna parsers and the leading-whitespace tolerance reapplied;
  thinking_end_tag became thinking_end_tags upstream.
- laguna.cpp/laguna.py and mtmd-image.cpp: took upstream, which already
  carries the fork's own upstreamed review fixes plus later refinements.
- Removed the inherited upstream workflows again, per 0c9a069.

GGML_OP_COUNT is 103: upstream's 101 plus the fork's TURBO_WHT and
FLASH_ATTN_EXT_BANDED.

Also drops a duplicate LLM_ARCH_LAGUNA case in test-llama-archs that the
merge would otherwise have left in moe_mandatory.
@Whamp Whamp mentioned this pull request Aug 10, 2026
Open
satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 11, 2026
@fairydreaming @sszymczy
llama : enforce the same K and V cache types for DeepSeek V4; enable …
956cea5 
…FA if V cache is quantized (ggml-org#25871)

* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized

* llama : enforce the same K and V cache types for MLA models

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 12, 2026
@fairydreaming @sszymczy
llama : enforce the same K and V cache types for DeepSeek V4; enable …
e4fdb3d 
…FA if V cache is quantized (ggml-org#25871)

* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized

* llama : enforce the same K and V cache types for MLA models

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Eval bug: CUDA GGML_ASSERT(tensor->nb[0] == ggml_element_size(tensor)) fitting Deepseek V4 Flash, but manual tensor split works

3 participants

@fairydreaming @ggerganov @sszymczy