GLM-DSA: fix -fa 0 garbage perplexity on the batch>8 indexer path (private seed, do not corrupt KQ_mask) - #2069
Merged
Conversation
The batch >8 indexer path in build_deepseek2_dsa_indexer accumulates the per-head scores with ggml_add_inplace into an accumulator that is seeded from a view of KQ_mask. On -fa 0, KQ_mask is the raw F32 input tensor, so the in-place writes land in the shared KQ_mask buffer and corrupt the causal mask that build_deepseek2_dsa_sparse_mask and the later softmax layers read back, which gives garbage perplexity. -fa 1 is unaffected (its F16 mask is cast to a private F32 buffer), and the small-batch path added in ikawrakow#2067 is unaffected (it uses a non-inplace add). Take a private copy of the seed in the batch >8 -fa 0 path (raw F32 mask) before the accumulation, matching what those two paths already do. 4K -fa 0 --dsa PPL goes from thousands to 2.7134 (dense 2.6972, -fa 1 --dsa 2.7111). -fa 1 and non-DSA builds are byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
ikawrakow
approved these changes
Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the GLM-DSA work (#2045, #2066, #2067). With
-fa 0(no flash attention) and--dsa, the DSA path still produces garbage perplexity on the batch >8 (prefill) indexer path.-fa 1 --dsais fine. #2067 already fixed the small-batch (<=8, TG/decode) path as a side effect of switching it to a single non-inplaceggml_add. This PR handles the remaining batch >8 path.Cause:
build_deepseek2_dsa_indexerseeds the indexer score accumulator with a view ofKQ_mask. On-fa 0,KQ_maskis the raw F32 input tensor. The small-batch path (#2067) accumulates with a non-inplaceggml_add, so it does not touchKQ_mask. The batch >8 path still accumulates the heads withggml_add_inplace, which writes into the sharedKQ_maskbuffer and corrupts the causal mask thatbuild_deepseek2_dsa_sparse_maskand the later softmax layers read back. On-fa 1the mask is F16, so the indexer's cast to F32 already gives a private buffer.Fix: in the batch >8 branch, take a private copy of the F32 seed before the in-place accumulation, matching what the small-batch path and
-fa 1already do:The small-batch / TG path is left untouched, so it keeps the #2067 performance work, and the copy is gated to
-fa 0(raw F32 mask) so-fa 1prefill pays nothing. Cost is onen_kvxn_tokensF32 copy per full layer on the batch >8-fa 0path only.Results (unsloth IQ2_M GGUF, top_k 2048, CPU unless noted; on current main dbe2ecb / #2067):
-fa 0 --dsa4K PPL (batch 512, the batch >8 path): thousands (broken) -> 2.7134 with the fix (dense 2.6972,-fa 1 --dsa2.7111)-fa 0 --dsa4K PPL on 3x P100 (CUDA, --cpu-moe): 2.6980, sane and in the dense /-fa 1band (not garbage), so the fix works on CUDA (pre-GLM-DSA: minor optimization #2066 tree, same copy fix)-fa 1and-fa 0-with-fix -> the attention machinery is correct, the DSA delta is pure key selection (pre-GLM-DSA: minor optimization #2066 tree, same copy fix)-fa 0clean at 9K (CPU), coherent at 24K (GPU, 21272-token prompt);-fa 1unaffected throughout (pre-GLM-DSA: minor optimization #2066 tree, same copy fix)-fa 0 --dsaat very long context is memory-heavy (per-layer sparse mask); 24K is the max on 48GB VRAMPerformance: decode is unaffected. The #2067 small-batch fast path is batch-gated, not FA-gated, so
-fa 0decode also takes it and never reaches the patched batch >8 loop. The fix adds onen_kvxn_tokensF32 copy of the causal-mask seed per full layer on the-fa 0prefill path, small next to the per-head matmul loop it guards, and it leaves the TG compute-buffer path that #2067 optimized untouched. If you would rather fold the copy into the accumulation, making head 0's add non-inplace and keeping the rest in-place is an equivalent, slightly cheaper alternative to theggml_cont.