GLM-DSA: fix CPU-only crashes in the sparse-attention path (#2045 was GPU-validated only)

[mgkwill]

PR ikawrakow#2045 adds GLM-DSA sparse attention but was validated on CUDA (3×P100 + --cpu-moe). On a CPU-only build (-ngl 0 --dsa), warmup crashes in four spots — each a place where the CUDA backend tolerates something the CPU backend does not. These fixes make GLM-5.2 --dsa run coherently on CPU. With --dsa off they are no-ops (they only touch the DSA CPU path).

This branch is based directly on glm-dsa-upstream (head 03391cb5) so it's a clean +4-fix delta.

1. set_rows into an F32 destination segfaults

ggml/src/ggml.c, ggml_compute_forward_set_rows_f32. The DSA sparse mask scatters into an F32 base via set_rows, but type_traits[F32].from_float == NULL (set_rows was only ever used for F16/quantized KV rows), so the NULL from_float is invoked → crash at ip=0. Fix: memcpy when the destination is F32. CUDA has a real F32 set_rows path, so this never bit there.

2. ggml_add(F32 score, F16 mask) aborts on CPU

src/graphs/build_deepseek2.cpp, in build_deepseek2_dsa_indexer and build_deepseek2_dsa_sparse_mask. Under -fa 1 the dense KQ_mask is F16; CPU ggml_add only supports F32+F16 when src0 is F16, not F32(score)+F16(mask), and aborts. Fix: cast the causal-mask view to F32 before the add. CUDA's add accepts the mixed types.

3. dsa_fa_mask dim-1 concat must be F32 on CPU

src/graphs/build_deepseek2.cpp, build_deepseek2_dsa_fa_mask. CPU ggml_concat only supports F16 along dim 0 (concat_any); the dim-1 row concat needs concat_f32. Fix: do the row concat in F32 (sparse + padding rows), then cast the padded result to F16. CUDA supports the F16 dim-1 concat directly.

4. Lightning-indexer k_norm epsilon is 0 → ggml_norm aborts

src/llama-hparams.cpp, glm-dsa hparam load. The indexer k_norm is a non-RMS LLM_NORM (LayerNorm) that uses hparams.f_norm_eps in ggml_norm(), but the GLM-DSA GGUF only carries the RMS eps, so f_norm_eps stays 0 and CPU ggml_norm aborts (GGML_ASSERT(eps > 0)). CUDA's norm kernel doesn't assert (eps=0 is numerically tolerable), which is why the CPU attention path was never exercised. Fix: mirror the RMS eps into f_norm_eps.

Validation

GLM-5.2 (UD-Q4_K_M GGUF) on a single-socket Xeon w7-2475X, CPU-only (-ngl 0 --dsa):

• Runs coherently at 49K+ context; correct 30K needle-in-haystack retrieval.
• Prefill ~32 tok/s, flat with context length (the O(L) sparse-attention signature) vs the dense build's O(L²) decline (28 tok/s @6K → ~16.5 @30k) — i.e. ~2× at 30K, the intended DSA benefit, now available on CPU.

🤖 Generated with Claude Code

[mb8565]

Marcus, thanks for running this down independently. We had been working the same CPU-only crashes from our side and landed on a near-identical set of four fixes, so two independent reproductions converging on the same four root causes is about as solid as a diagnosis gets.

Your concat fix is cleaner than ours and we are adopting it. We had generalized ggml_concat to handle dim 1/2/3 in ggml.c; doing the row concat in F32 with the existing concat_f32 and casting to F16 keeps it out of the shared op entirely, which is lower risk and more likely to pass review upstream.

Your prefill numbers are the half we did not have. We had only measured the decode side, on a 3xP100 + --cpu-moe hybrid, where the experts dominate each token and the indexer overhead swamps the sparse-attention saving, so DSA is a throughput loss there through 16K. Your CPU-only prefill result is the case where it clearly pays off. Between the two we have an honest picture of where DSA helps and where it does not.

On our side these fixes also sit on top of ikawrakow's per-head-loop change (ikawrakow#2058), which bounds the indexer compute buffer, and we are sorting out with him how the CPU-only fixes land upstream. Would be glad to have you in that thread (ikawrakow#2045) if you want to weigh in directly; your reproduction and the concat approach are worth having on the record there.