llama : make all KQ masks f16 if FA is used, remove zero attention bias, remove raw_k repeats in DeepSeek V4

[fairydreaming]

Overview

This PR removes the following skeletons from DeepSeek V4 closet:

• it makes KQ masks f16 if FA is used (only the lightning indexer mask is kept f32 since it's added to scores),
• it removes KQ mask type casts entirely,
• it removes dsv4_repeat_streams() that repeats raw_k cache for each stream, as both raw_k and compressed caches are always non-unified it's no longer needed,
• it removes zero kq_b bias tensors that were causing OOM errors like this:

0.06.078.756 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 437894.04 MiB on device 0: cudaMalloc failed: out of memory
0.06.078.764 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 459165182720
0.06.078.766 E graph_reserve: failed to allocate compute buffers

Needs #25351, #25344, #25367, #25434

Additional information

I still don't understand why constructs like:

const bool use_fattn = cparams.flash_attn && (!cparams.kv_unified || n_stream == 1);

were needed at all and why the bias tensors were only created if not

(!cparams.kv_unified || !cparams.flash_attn || kq_mask->ne[3] == 1)

so that means they were created if:

cparams.kv_unified && cparams.flash_attn && kq_mask->ne[3] != 1

My head hurts when I think about what was the purpose. @am17an let me know if that breaks anything.

Tested the code with all combinations of FA and kv_unified, seems to work just fine. Performance is slightly improved.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[fairydreaming]

Would be nice if someone prepared f16 GGML_SET_ROWS for Metal, otherwise there will be performance regression on Macs after this PR. Unfortunately I have no access to the hardware and no experience with Metal at all.

[ggerganov]

Would be nice if someone prepared f16 GGML_SET_ROWS for Metal, otherwise there will be performance regression on Macs after this PR. Unfortunately I have no access to the hardware and no experience with Metal at all.

Taking a look

[ggerganov]

@fairydreaming Feel free to cherry-pick this commit wherever it makes sense: d4a193e

[fairydreaming]

@fairydreaming Feel free to cherry-pick this commit wherever it makes sense: d4a193e

@ggerganov Thanks, backend ops tests for this are in #25344 (hopefully Vulkan tests will pass now and it will be ready to merge), so when #25344 is merged I'm going to create PR from your branch. Not adding it here, that would be against contributing guidelines. 😱

[arch-btw]

@fairydreaming It took me a while to find but this commit broke DeepSeek V4 flash on Vulkan for me on AMD. Unfortunately, it's still broken in the current master.

There isn't much of a log even with -lv 7. It just core dumps when I send the first message in the UI.
The previous commit still works: 082b326
I've also tried with -fa on and without it.

I'm using this quant by alterez:
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf

[arch-btw]

Maybe this?

0.32.068.957 W resolve_fused_ops: layer 2 is assigned to device Vulkan0 but Lightning Indexer is assigned to device CPU (usually due to missing support)
0.32.068.959 W resolve_fused_ops: Lightning Indexer not supported, set to disabled

[fairydreaming]

@arch-btw Thanks for reporting, indeed I see a crash on Vulkan. Added a bug for this: #25579