Extended SYCL oneDNN SDPA to non-FP16 KV caches (Q4_0–Q8_0 and FP32) - #25874
Conversation
github-actions
Bot
added
documentation
|
Hi @johnkarlhill, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
0d01c9d to
b0b76ae
Compare
|
Any reason not to use mkl_fa_dequant_chunk() from your #25025? I'm testing something similar with TILE and it seems to be worth doing. This bounds transient memory allocation pressures that may not be apparent until higher context utilization. |
Short answer: we can't use chunked dequant here because oneDNN's SDPA graph (sdp_primitive_kernel_t) takes the full K, V, Q, and mask as monolithic inputs. It's a single fused kernel call, one graph execution. There is no API to feed K or V in pieces and rejoin with online softmax rescaling the way MKL does in #25025. The MKL kernel can chunk because it owns the outer loop: it dequants one 8192-token chunk of K/V, runs GEMM, does the softmax rescale, and moves to the next chunk. The oneDNN graph compiles all of that (MatMul, Scale, Mask, SoftMax, MatMul) into one systolic kernel. That said, the memory pressure is real. At typical prefill contexts (32K tokens, single head), the dequant buffer is about 64 MB for K+V combined. At 256K it grows to roughly 512 MB transient. This is pool-allocated and released at function return, so it does not creep, but it is a spike. Thinking out loud, once PR25025 is merged, I could see a flag that would shift processing from oneDNN SDPA to to MKL at a certain context, thus minimizing the spike. That way high-context users get bounded memory and still stay accelerated, just via MKL GEMM rather than the fused SDPA kernel. Not sure if the maintainers would let that go through... |
|
One of the problems here (and elsewhere) is that we're providing a user with levers to buy VRAM by spending compute or quality, and then turning around and charging them VRAM behind their backs. The mechanism you're describing about shifting processing does work though. |
I'm not quite sure what the expectation is. The llama.cpp pipeline is FP16. To push anything non-FP16 through, you have to convert it first. The quantized cache has to be decompressed before it can be processed, and that decompression has to happen somewhere. With #25025 I was able to decompress in smaller chunks because I control the outer loop. With this PR the kernel is fused, so I'm not in control of the chunks. And to be clear, this is a very, very temporary spike... the memory is given back immediately at function return. |
arthw
left a comment
arthw
left a comment
There was a problem hiding this comment.
Please provide a detail example to show the benefit of this PR.
I can't get the perf increase on Qwen3.5-4B-Q4_0.gguf B60.
|
@johnkarlhill can you follow the PR template? and do you have some benchmarks or figures that support your changes? |
Did you use quant the cache (for example, cache-type-k = q8_0 / cache-type-v = q8_0)? This PR only extends PR25222's kernel to certain quanted kv caches. |
Updated! |
|
@johnkarlhill Thank you! |
6e9257c to
2736dec
Compare
Done and Done! Thank you! |
arthw
added
the
merge ready
Extends the oneDNN SDPA path (PR ggml-org#25222) to handle non-F16 KV caches by dequantizing or converting K/V to dense FP16 on-device before feeding them into the SDPA graph. The fused systolic kernel then runs identically to the native FP16 path. Supported KV types: - Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl - F32: cont_to_f16_sycl<float> - BF16 and IQ types are excluded (no conversion kernel available) Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only). F16 KV runs at any length (existing behavior). Also includes the stream sync fix (stream->wait_and_throw() unconditional, PR ggml-org#25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are always dequantized to separate buffers). Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
2736dec to
d688c52
Compare
|
Rebased with latest code releases. |
…gml-org#25874) * sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches Extends the oneDNN SDPA path (PR ggml-org#25222) to handle non-F16 KV caches by dequantizing or converting K/V to dense FP16 on-device before feeding them into the SDPA graph. The fused systolic kernel then runs identically to the native FP16 path. Supported KV types: - Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl - F32: cont_to_f16_sycl<float> - BF16 and IQ types are excluded (no conversion kernel available) Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only). F16 KV runs at any length (existing behavior). Also includes the stream sync fix (stream->wait_and_throw() unconditional, PR ggml-org#25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are always dequantized to separate buffers). Co-Authored-By: Claude <noreply@anthropic.com> * docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…gml-org#25874) * sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches Extends the oneDNN SDPA path (PR ggml-org#25222) to handle non-F16 KV caches by dequantizing or converting K/V to dense FP16 on-device before feeding them into the SDPA graph. The fused systolic kernel then runs identically to the native FP16 path. Supported KV types: - Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl - F32: cont_to_f16_sycl<float> - BF16 and IQ types are excluded (no conversion kernel available) Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only). F16 KV runs at any length (existing behavior). Also includes the stream sync fix (stream->wait_and_throw() unconditional, PR ggml-org#25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are always dequantized to separate buffers). Co-Authored-By: Claude <noreply@anthropic.com> * docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…gml-org#25874) * sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches Extends the oneDNN SDPA path (PR ggml-org#25222) to handle non-F16 KV caches by dequantizing or converting K/V to dense FP16 on-device before feeding them into the SDPA graph. The fused systolic kernel then runs identically to the native FP16 path. Supported KV types: - Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl - F32: cont_to_f16_sycl<float> - BF16 and IQ types are excluded (no conversion kernel available) Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only). F16 KV runs at any length (existing behavior). Also includes the stream sync fix (stream->wait_and_throw() unconditional, PR ggml-org#25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are always dequantized to separate buffers). Co-Authored-By: Claude <noreply@anthropic.com> * docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
74 upstream commits, 14 in our paths. Two textual conflicts, both expected, plus ONE silent-drift resolution that did NOT conflict -- which is the whole reason upstream-check.sh warns about auto-merges. ☠☠ fattn.cpp -- THE ONE THAT AUTO-MERGED CLEAN (upstream 66fa168 / ggml-org#25874) ggml-org#25874 extends oneDNN SDPA to non-F16 KV (Q4_0..Q8_0) and inserts its gate ABOVE the MKL gate -- which our 0801 sync had already deferred to MMA for exactly this reason. Its envelope (Q->ne[1] >= 32, K->ne[1] >= 1024, quantized KV now accepted) matches our deploy prefill exactly: q8_0 KV, ubatch 2048, any real depth. Verbatim it would have silently swapped the measured MMA kernel (2.87x served prefill at 43k) for an unmeasured one and staged the whole q8_0 KV cache to dense F16 first. ⇒ added the SAME conjunct, same wording, as the MKL gate below it: !ggml_sycl_fattn_mma_supported(dst). oneDNN keeps its full envelope everywhere MMA declines. A/B reachable both ways without a rebuild (GGML_SYCL_FATTN_MMA=0 / GGML_SYCL_FA_ONEDNN=0 -- the latter is checked inside _supported(), so it still governs the new early return). This is a precedence choice, not a revert, and it is UNMEASURED on this box. B59 is the finding that prices it. The three conflicts git DID flag in this file were cosmetic (debug kname lines + upstream re-calling get_best_fattn_kernel where we hoist it). Kept ours, added upstream's ONEDNN kname lines. concat.cpp -- upstream 6c8dcaa (ggml-org#25852) is a duplicate of our own fix Kept OURS: strict superset (same launch geometry + the GGML_SYCL_CONCAT_WG door + the o[dim] hoist + the i64 loop var). Ours deliberately omits their WARP_SIZE floor, which would make WG=1 unreachable and destroy the one-binary positive control for the 8.36x launch fix. Noted in the source so the next sync does not resolve toward upstream. Assessed, no action needed: 272700b (ggml-org#26105) iGPU classification -- NULL here, B70 is discrete and still reports TYPE_GPU. Relevant later for PVC/Max-1100 and B51. dbadb68 (ggml-org#22789) dynamic split-graph inputs -- mechanical, auto-merged beside our GGML_SCHED_HANDOFF_CENSUS. 596a579 (ggml-org#25784) DeepseekV4 MTP + DSpark -- ⚠ changes SHARED MTP plumbing (n_embd_out vs n_embd_inp in llama_context::decode, set_embeddings_layer_inp bounds, embeddings_layer_inp sized n_layer+1). We run MTP every step. NOT yet gated -- see the build/gate that follows. ☠ NOT YET BUILT, NOT YET GATED. Every absolute in f306 predates this merge.
…gml-org#25874) * sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches Extends the oneDNN SDPA path (PR ggml-org#25222) to handle non-F16 KV caches by dequantizing or converting K/V to dense FP16 on-device before feeding them into the SDPA graph. The fused systolic kernel then runs identically to the native FP16 path. Supported KV types: - Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl - F32: cont_to_f16_sycl<float> - BF16 and IQ types are excluded (no conversion kernel available) Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only). F16 KV runs at any length (existing behavior). Also includes the stream sync fix (stream->wait_and_throw() unconditional, PR ggml-org#25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are always dequantized to separate buffers). Co-Authored-By: Claude <noreply@anthropic.com> * docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
5 participants
Overview
Extends the SYCL oneDNN SDPA path (#25222, now merged) to handle non-FP16 KV caches. The existing implementation gives great XMX SDPA speedup during prefill, but only for native FP16 K/V. If you run a quantized KV cache (q8_0, q4_0, etc.) to save VRAM, you were stuck on the TILE path and paying the performance cost.
Simple approach: before passing K/V into the oneDNN SDPA graph, dequantize or convert them to dense FP16 on-device. From there, the SDPA executes the same fused systolic kernel as the standard FP16 pipeline.
Benchmarks
All runs: B70 or B50, 32K prefill, quantized KV cache (q8_0 or q4_0), MTP on and off. ONEDNN=1 vs ONEDNN=0 (forces TILE fallback). Token gen is identical across all runs since the ONEDNN path does not fire during decode (Q=1).
B70 (Arc Pro B70, 32 GB)
B50 (Arc Pro B50, 16 GB)
test-backend-ops passes 3961/3961. Multi-turn coherence tested on Gemma 26B and Qwen 27B with assorted prompt sequences: clean output, draft acceptance around 90%.
Supported KV types and gate
F16 KV runs native SDPA at any length, same as #25222.
Non-FP16 types gate at K >= 1024 and Q >= 32 (prefill only):
Inherits #25222's BMG-only architecture gate for the underlying oneDNN kernel.
Includes an unconditional stream->wait_and_throw() after SDPA compute, without this the pool allocator reuses device buffers while the GPU is still executing, corrupting the KV cache on subsequent turns (#25741 by @malsbat)(no longer needed after rebase).Requirements