Conversation
…seq prefill) Implements the sparse top-k "lightning indexer" attention for LLM_ARCH_GLM_DSA in build_deepseek2_layer_attention (ik's deepseek2 graph). What it does (per layer, gated on model.arch==GLM_DSA && indexer_attn_q_b): - indexer_q = indexer_attn_q_b(q_lora latent), split rope(64)/nope(64), NEOX-rope the pe part, concat. indexer_k = indexer_attn_k(attn_norm out), LayerNorm w/ bias, same rope/concat (single key head, MQA). - scores = relu(indexer_k . indexer_q), scaled per-head weights (indexer_proj), summed over heads, + base causal mask, then ggml_top_k(min(top_k, n_tokens)). - sparse mask: ggml_fill(-inf) -> ggml_set_rows(0) at top_k positions -> + causal, used in the soft_max_ext attention path (-mla 1 -fa 0) instead of KQ_mask. Simplifications (intentional, proven sound): - Batch-local: no indexer KV-cache. Indexer keys are the current batch tokens. - Walsh-Hadamard transform omitted: orthonormal rotation, (Hq).(Hk)==q.k, no score change. Validation (GLM-5.2-UD-IQ2_M, 3x P100, -mla 1 -fa 0): - Compiles clean (CUDA sm_60); loads and runs. - c512 -b512 (n_seq=1) PPL = 2.7760, byte-identical to dense baseline (indexer disabled) = 2.7760, all 8 chunks match -> indexer is an exact no-op when top_k>=n_tokens. Proves correctness-preservation. - 3105-token prompt completion (top_k=2048 < 3105 -> indexer ACTIVELY masks): prompt-eval produces coherent, accurate continuation, identical to dense for the prompt+early-gen tokens. No NaN/crash. Confirms the masking path works in prefill. Known limitations (documented follow-ups, NOT handled): - Single-sequence prefill only. Multi-sequence batches (n_seq>1, e.g. perplexity default n_batch>n_ctx) and kv_head>0 (decode) break the batch-local key->slot mapping. n_seq>1 -> NaN (use n_batch==n_ctx). Decode (kv_head>0): each generated token sees only itself as an indexer key, so generation degenerates into repetition after the prompt (dense A/B stays coherent) -- this is the decode-cache stub, the documented next step. - Flash-attn path (-fa 1, F16 mask) still uses dense KQ_mask (soft_max path only). - Decode indexer KV-cache + Hadamard cached-K storage not implemented. Runtime gate: DSA_INDEXER_DISABLE=1 falls back to dense attention (for A/B). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the lightning-indexer correct for DECODE (not just prefill). Previously the indexer was batch-local, so a generated token only scored against itself and generation degenerated. Now the indexer keys are cached across the full context. Changes - llama_kv_cache: add per-layer indexer-key cache `kr_l` [indexer_head_size, kv_size] (F16, MQA single head), allocated alongside the MLA latent cache for GLM_DSA. - build_deepseek2_dsa_indexer: write the batch's (Hadamard-rotated) indexer keys to kr_l at kv_head, read back the full [128, n_kv] cached keys, and score the indexer queries against ALL past keys. Returns the full descending argsort of the scores. - Walsh-Hadamard rotation of indexer q/k (cparams.dsa_indexer_hadamard, default on; filled in llama_set_inputs). Score-preserving; improves cached-K F16 precision. - build_deepseek2_dsa_sparse_mask: rank-based full-coverage scatter (write a 0/-BIG penalty into EVERY key slot keyed by rank) instead of partial set_rows into a -inf fill — the CUDA in-place set_rows does not preserve an un-written base, which had corrupted decode when n_kv > top_k. - Attention-sink force-inclusion (DSA_SINK, default 1): boost the first key(s) so the sink always survives top-k. The IQ2_M-quantized indexer under-ranks the sink, and masking it collapsed decode; with the boost, top_k=2048 over n_kv>2048 stays coherent. ggml backend fixes (needed by the indexer) - CUDA argsort: report unsupported when padded ncols > 1024 (one-thread-per-column bitonic launch limit) so the scheduler falls back to the CPU argsort. Fixes "invalid configuration argument" for top_k over a large n_kv. - CUDA cpy/dup: support I32 -> I32 (top_k index copies / cross-backend moves). Validation (GLM-5.2-UD-IQ2_M, 3xP100 + --cpu-moe, -mla 1 -fa 0) - c512 PPL = 2.0743, byte-identical to dense (all 8 chunks): no-op path exact. - Short-context decode (300 tok): coherent, identical to dense. - Long-context decode (2521-tok prompt, n_kv>top_k, real masking of ~474 keys, 120+ tok generated): coherent with the sink boost; dense A/B also coherent. Gated behind arch==GLM_DSA + indexer tensors + kr_l cache; DSA_INDEXER_DISABLE=1 forces dense. Remaining: FA path still uses the dense KQ_mask; multi-sequence (n_seq>1) batches; deepseek32 arch wiring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-fa 1) The DSA sparse top-k mask is now applied on the -fa 1 path (our serving config), not just -fa 0 soft_max. c512 PPL on -fa 1 = 2.0743, byte-identical to dense (no regression, indexer no-op exact at n_kv <= top_k). Gated arch==GLM_DSA with DSA_INDEXER_DISABLE escape; -fa 0 path unchanged. Long-context -fa 1 decode coherence (n_kv > top_k, mask actually biting) validation is still running at commit time; the FA mask reuses the same full-coverage scatter proven coherent on the -fa 0 decode path, so it should hold, but confirm before relying on long-context -fa 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… multi-seq characterized Document the re-validation after cherry-picking the MLA-FA vec-decode fix (5f18dcc0): - FA path is ALIVE. Long-ctx -fa 1 decode (2521-tok prompt > top_k, mask actively biting) is now COHERENT at -mla 1 and -mla 3, vs the pre-fix degeneration into "0.0.0.0..." repetition. Matches dense (DSA_INDEXER_DISABLE) and -fa 0 controls. - c512 -fa 1 PPL: indexer-ON == dense == 2.0854, byte-identical all 8 chunks (exact no-op when n_kv <= top_k; no regression). The 2.0743->2.0854 shift is the MLA-FA fix changing V accumulation, not an indexer artifact (ON==dense proves it). - Indexer is feature-complete + validated for single-seq prefill+decode on both -fa 0 and -fa 1, at -mla 1 and -mla 3 (the R740 serving target). Remaining PR gaps, characterized honestly: - Multi-seq (n_seq>1) with active mask is BROKEN (n_seq=2 c4096 PPL 62.6 vs dense multi-seq 2.54 and single-seq indexer 3.05). No NaN/crash anymore. Root cause: the indexer uses a single scalar kv_head/n_kv for the whole ubatch; multi-seq needs per-sequence cache writes + per-sequence top-k. Fix deferred (structural). - deepseek32 arch: N/A in this fork. DSA lives entirely under LLM_ARCH_GLM_DSA; there is no LLM_ARCH_DEEPSEEK32 enum. Documented the steps to add one if a real deepseek32 GGUF is ever served. Also commit DSA_REFERENCE.md (verbatim mainline deepseek32/glm-dsa source, the port reference), trimmed of a stray agent-handoff footer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eq>1)
UPDATE 5. The DSA lightning indexer was numerically broken for multi-sequence
batches once the top-k mask bites (n_kv > top_k): c4096 n_seq=2 PPL 62.6 vs
dense 2.54, while single-seq was fine. Root cause: the attention-sink
force-include boosted the GLOBAL key range [0, n_sink) by +1e20, which only
protects sequence 0's sink. With several sequences packed contiguously into one
ubatch (seq 0 at cells [0,n0), seq 1 at [n0,n1), ...), every non-first
sequence's sink lives at cell n0.. (not cell 0), got no boost, and was dropped
from top-k once the mask bites — collapsing that sequence (chunk[2]=61.2 while
chunk[1]=2.33).
The cache write and score/argsort were already per-sequence correct: tokens are
placed contiguously like the main K cache, and the base KQ_mask (filled from
kv_self.cells[i].has_seq_id) already drives cross-seq keys to -inf before
argsort. Only the sink was anchored at the wrong (global) cell.
Fix: replace the global arange sink boost with a per-graph input tensor
inp_dsa_sink {n_kv, n_tokens} (F32), filled on the CPU in llama_set_inputs from
kv_self.cells exactly like the KQ_mask:
inp_dsa_sink[j,i] = 1e20 iff cell[i].pos in [0,n_sink) AND
cell[i].has_seq_id(seq_of_query_j), else 0
so each query force-includes only its OWN sequence's sink. For a single
contiguous sequence from pos 0 this is exactly the old "cell index < n_sink"
set with the same magnitude, so n_seq==1 is byte-identical.
Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, wikitext-2):
- c4096 n_seq=2 indexer chunk[2]: 61.2 -> 3.07 (== single-seq 3.05).
- c2048 topk=1024 (mask bites): n_seq=4 == n_seq=1 chunk-for-chunk
(2.5005/2.6080/2.7759/3.1137 vs .../3.1138) -> multi-seq is numerically
identical to processing each sequence alone.
- c512 n_seq=1 indexer ON == dense, all 4 chunks byte-identical (no regression).
n_seq=4 at full c4096 (n_kv=16384) OOMs the P100 compute buffer (capacity, not
correctness; n_seq=4 proven correct at c2048/n_kv=8192).
GLM-5.2 DSA indexer is now sequence-correct for n_seq>=1, prefill+decode,
soft_max+FA, -mla 1/-mla 3. Fully general and PR-ready.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…across shift/defrag/seq-ops; per-seq sink on first-present pos) An adversarial review found the indexer was proven on the perplexity path but not the serving path: the persistent indexer-K cache kr_l was written/read but never *maintained* by the KV-cache mutators, and the attention sink anchored on absolute pos<n_sink (wrong after multi-turn seq_rm). This closes those gaps and pins down what is actually reachable on the MLA model. kr_l maintenance: - build_k_shift (llama-build-context.cpp): rotate the indexer keys by the same per-cell delta as the main K. The cached key is H*concat(RoPE(k_pe,pos),k_nope), so un-Hadamard (H sym/orthonormal => H*H=I) -> RoPE-delta the pe sub-block -> re-Hadamard. Exact because GLM-DSA has no rope-scaling metadata (ext_factor=0, attn_factor=1, freq_scale=1), so NEOX RoPE is pure/composable. Params mirror the forward indexer RoPE exactly (rope_factors=nullptr); no DEEPSEEK2 yarn-shift leak. Non-in-place (cont->rope->concat->re-Had->cpy), no aliasing. K-shift Hadamard input filled in llama_set_k_shift with the identical Sylvester construction. - build_defrag: kr_l row-move mirrors the k_l move (defrag never changes pos, so no re-RoPE). max_moves divisor 6->9 *n_layer when the indexer cache is present. - seq_rm/seq_cp/seq_keep are metadata-only (verified) so kr_l rows stay matched to cells; seq_add/seq_div set has_shift and route through K-shift. No seq-op change. Per-seq sink (llama.cpp llama_set_inputs): anchor on each sequence's FIRST PRESENT pos (min present pos over the scored n_kv span), not absolute pos<n_sink. After multi-turn seq_rm drops a sequence's early tokens its earliest survivor has pos>=n_sink; the absolute test would protect nothing. Fresh seq at pos 0 => min=0 => byte-identical to the old behaviour. Serving-shift finding (the whole point): a RoPE context-shift on this model is REFUSED BY THE ENGINE. get_can_shift() returns false for all MLA models (is_mla_model() includes GLM_DSA); llama_kv_cache_update returns 1 -> "main : failed to eval". Reproduced AND isolated with a dense control (DSA_INDEXER_DISABLE=1): dense fails identically at the same token. The failure is pre-existing MLA engine behaviour, independent of the indexer. On the MLA path the shift never happens, so the indexer's kr_l can never desync via K-shift; the build_k_shift kr_l block is correct-and-dormant (documented loudly in code). Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, GGML_CUDA_NO_PINNED=1, numactl --interleave=all, wikitext-2): - No regression: c512 n_seq=1 indexer ON == dense == 2.1957 +/- 0.12031, byte-identical all 4 chunks (2.2770/2.8741/2.3956/2.1957). - Multi-seq: c4096 n_seq=2 chunk[1]=2.33 chunk[2]=3.07 healthy (== UPDATE 5; per-seq sink change did not regress). - Serving shift: engine-refused for MLA, dense control fails identically. - Independent adversarial review: GO, no correctness defect in the diff. - Build clean (llama-cli, llama-perplexity, sm_60). Comments updated (build_deepseek2.cpp): multi-seq+FA no longer limitations; sink description matches per-seq min-pos anchoring; BIG=1e30 masks on both soft_max and FA paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ission for the kr_l indexer cache
update_cache_copies() re-points the K/V cache writes to the current kv_head whenever a
compute graph is REUSED (can_reuse_graph reuses iff kv_self.n == prev->n_kv). The persistent
indexer-key cache write (kr_l) is a separate ggml_cpy whose destination view bakes kv_head at
graph-build time, and it was NEVER registered for that fixup. Under FA the cache pads to 256,
so consecutive single-token decode ubatches share the same padded n_kv and the graph IS reused;
without the fixup the kr_l write keeps landing in the first ubatch's slot and later ubatches
never populate their own recent index-key cells (those cells stay at the alloc-zeroed 0.0).
Structurally identical to the MiniMax MSA bug (fork commit 133d14c9).
Fix (mirrors the K/V cache_copies fixup, same shape as MSA 133d14c9):
- llama-context.h: new std::vector<CacheCopy> dsa_cache_copies.
- llama.cpp ctor: resize dsa_cache_copies to n_layer (null entries -> no-op when DSA off).
- build_deepseek2.cpp: register the kr_l ggml_cpy as dsa_cache_copies[il] = {kr_cpy, kr->nb[1]}.
- llama.cpp update_cache_copies(): re-point each registered cpy view_offs = kv_head*step and
patch src[1]->data/data, exactly like K/V, with the c.cpy->view_src == kv_self.kr_l[il]
(+ null/op) guard the MSA fix omitted. soft_max / non-DSA paths byte-identical.
Validation (GLM-5.2-UD-IQ2_M, 3x P100 -ngl 99 --cpu-moe -t 32, NO_PINNED, P2P-disable patch
re-applied to get a working multi-GPU baseline — see UPDATE 7.3; that patch was lost in the
upstream rebase and is required separately):
- c512 -fa1 -mla3 indexer ON: 2.1983 (== prior baseline; build healthy).
- Long-ctx FA decode, 2735-tok recall prompt, -mla3 -fa1 temp0, reuse ON (default): coherent,
correct deep-context recall ("Dr. Mariana Velasquez ... Daniel Okonkwo") on BOTH the fixed and
the unfixed binary.
- ub128 PPL -fa1 -mla3 reuse ON, unfixed: 1.7239/1.8211/2.1888/2.4517, healthy (no inflation).
Honest scope: the bug is real in code but LATENT for GLM-DSA at its configured top_k=2048
(permissive selection keeps the genuinely-attended recent blocks even when reuse leaves some
recent index-key cells stale), unlike MSA's tighter top-k where it inflated PPL ~2x. The fix is
correct and prevents the latent corruption from biting at any tighter top_k / longer ctx /
future serving config. The pre-P2P-patch "nan" seen at ub128 was P2P corruption, not this bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…off by default) Implements ikawrakow's direction from discussion #2040: the DSA sparse indexer must be controllable via command-line argument (not environment variables), and must be OFF by default for now. Control surface, before -> after: DSA_INDEXER_DISABLE (env, inverted: on-by-default) -> --dsa / -dsa (cparams.dsa, default false; opt-in, dense-by-default) DSA_TOPK_OVERRIDE (env) -> --dsa-top-k N / -dsatk N (cparams.dsa_top_k, default -1 == model's configured indexer_top_k) DSA_HADAMARD_DISABLE, DSA_SINK (env) -> kept as DEBUG-ONLY env knobs (clearly commented; no CLI surface, not system on/off controls) Plumbing mirrors existing boolean/int feature flags (-mla, -khad): include/llama.h llama_context_params {bool dsa; int dsa_top_k;} src/llama.cpp default_params (false / -1); cparams assignment src/llama-cparams.h llama_cparams {bool dsa=false; int dsa_top_k=-1;} common/common.h gpt_params {bool dsa=false; int dsa_top_k=-1;} common/common.cpp arg parse + help text + cparams copy src/graphs/build_deepseek2.cpp gate now checks cparams.dsa instead of getenv; top-k override reads cparams.dsa_top_k. Stays arch-gated to LLM_ARCH_GLM_DSA. When --dsa is off (default) the indexer function is never called -> existing dense MLA path, byte-identical to no-feature. Validation (GLM-5.2-UD-IQ2_M, 3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, wikitext-2, 4 chunks @ c2560): --dsa OFF (default, dense): PPL 2.4151 (graph nodes 4166) --dsa ON, default top_k=2048: PPL 2.4697 (graph nodes 8846) --dsa ON, --dsa-top-k 1024: PPL 3.5107 Off-by-default runs the dense path; ON activates the indexer (node count jumps, PPL shifts as the top-k mask bites once n_kv > top_k). No env var is consulted for the primary on/off or the top-k knob. Graph-parallel (-sm graph) interaction (the item ikawrakow flagged): Under -sm graph the MLA layers are TP-split (wo->extra) and route to build_deepseek2_tp_attention(), which contains NO indexer code. So --dsa is silently a NO-OP under -sm graph: it does not error or crash, it runs dense. Empirically, --dsa --dsa-top-k 1024 under -sm graph gives PPL 2.4308 (chunks 1.6967/1.7906/2.1664/2.4308) -- the dense baseline (2.4151), NOT the DSA top_k=1024 numbers (3.5107). The 0.016 delta is f16 TP-reduce numerics, not DSA. Conclusion: DSA "works under deepseek2" only on the non-TP (layer) path; serving DSA with -sm graph would require wiring the indexer into the TP attention path (or a dedicated DSA arch). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ns dense MLA) The DSA lightning indexer is built only in the layer-mode (non-TP) attention path. Under -sm graph / -sm attn the tensor-parallel attention path has no indexer, so --dsa would silently run dense MLA. Emit a clear one-time LLAMA_LOG_WARN at context creation instead of degrading silently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DSA_REFERENCE.md and the R740 progress note are development scratch, not part of the submission. Remove them so the PR diff is code-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
reduced ctx=500K b=8192 buffers from 2,260GiB to just 1,292GiB 🫠 |
Owner
Author
On how many GPUs? |
Contributor
1x R6KP, _exps on cpu. |
mb8565
added a commit
to mb8565/ik_llama.cpp
that referenced
this pull request
Jul 1, 2026
The GLM-DSA sparse-mask destination `base` was created with ggml_new_tensor_3d,
i.e. an op-NONE leaf. ggml-alloc allocates all graph leaves eagerly before the
node loop runs, so one {1,n_kv,n_tok} base leaf per DSA layer is live at once at
the compute-buffer high-water mark, and the buffer grows as (n DSA layers) x
n_kv x n_ubatch x 4 -> it OOMs at long context (usrlocalben's report on ikawrakow#2045).
Here that is 78 layers: n_layer 79 minus the one MTP/next-n layer, which has no
indexer (guard build_deepseek2.cpp: dsa && GLM_DSA && indexer_attn_q_b).
Build `base` as a FILL node off the pen_b node instead. Nodes are allocated
lazily and returned to the free list after their last consumer, so base is
reused across layers and the per-layer pile disappears. Output is byte-identical:
ggml_fill writes -BIG everywhere and set_rows overwrites every key slot exactly
once (sorted is a per-column permutation over n_kv), so base's initial contents
were always irrelevant; ggml_fill dup_tensors and FILL is not can-inplace, so
the fill output cannot alias pen_b (still read by set_rows). No new op.
Measured CPU-only on this branch (GLM-5.2-IQ2_M, -ngl 0 --dsa -fa 1 -c 32768
-ub 512), controlled A/B differing only in this line: CPU compute buffer
12617 -> 7629 MiB (delta 4989 = 78 x 64 MiB), and 4-chunk 4K PPL byte-identical
(3.2102 == 3.2102). Independent of the indexer-score approach, so it stacks with
the head-loop in ikawrakow#2058; on that branch a GGML_ALLOCATOR_DEBUG dump confirms the
78 per-layer 64 MiB base leaves drop to 0 in the peak live-set (10569 -> 5580).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
Closing in favor of #2063 |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hopefully this is useful for improving PR #2045
The indexer score computation is done via a loop over attention heads. This reduces the memory required by a factor of 32 (the number of indexer attention heads for GLM-5.2).