vulkan : dequant q8_0 KV once in coopmat1 - #25494
Merged
0cc4m merged 9 commits intoAug 19, 2026
Merged
Conversation
Assisted-by: Claude (Opus 4.8)
|
There’s a complimentary issue here that may be worth looking at: #25207 (comment) |
github-actions
Bot
added
Vulkan
Contributor
|
Doesn't this defeat the purpose of using the quantized cache? Seems like you'll end up with worse memory usage than just using f16 in the first place. I was curious so I measured the perf on my system: In coopmat2 mode, there's only about a 10% penalty for using q8_0 directly. So I think there's room for improvement in the coopmat1 path without needing to use additional memory. |
Contributor
Author
|
Thanks Jeff, It only dequantizes one f16 layer into scratch at a time, so it keeps the full q8_0 benefit (~6 GB + 256 MB scratch vs ~12 GB for f16 @128k). The one f16 layer in scratch (256 MB @128k) is extra memory, but ~4% on top of the quantized cache. That ~4% (in this PR's current state) is the tradeoff for up to ~2× prefill throughput. The benefit grows with context depth, and grows further on lower-bandwidth unified-memory hardware. As I understand it, AMD/RADV can't use coopmat2, so coopmat1 seems like the only prefill path. The current implementation reuses the existing A few directions I had in mind as follow-ups (open to your view on any):
Are the current tradeoffs ok for this PR or would you suggest the scratch memory usage is addressed first? |
Contributor
|
Thanks for the explanation, maybe this is more viable than I thought. Can you explain why the transpose is needed? I would have expected the quantized and dequantized KV cache to have matching layout, and AFAIK the normal layout for the KV cache is fine. |
Contributor
Author
|
Removing the redundant per-workgroup dequant means materializing the KV as an f16 scratch which is nearly double q8_0's bytes. Read strided (without transposing), that costs more than the dequant it saved: pp512 @128k regressed to 30.8, below the 45.8 q8_0 baseline. But that write has to happen anyway, so we use it to lay the scratch out for an efficient FA read, which brings it to 93 t/s, ~2× the baseline. The tensor strides show why. For the f16 K tensor ne = [128, 256, 4] (HS, KV, n_head_kv), the byte strides are nb = [2, 1024, 256]. Stepping one KV position for a fixed head jumps nb[1] = HS × n_head_kv × 2 = 128 × 4 × 2 = 1024B, where per-head-contiguous would be HS × 2 = 256B. The transpose lays the scratch per-head-contiguous, so the FA reads sequentially. The same reorder may help f16 too, it's strided the same way. Although for f16 it'd be a pure extra copy, with no dequant to ride on. |
Contributor
|
In either case the innermost dimension is contiguous and whole cache lines. Why does it matter whether the rows are permuted? |
Contributor
Author
|
Per-row it's identical, the difference is across rows. One head's rows are 1 KB apart natively vs back-to-back in the scratch, so each workgroup's KV walk is 256-on/768-off instead of a sequential stream. The gaps aren't wasted, the other heads' workgroups want those bytes, but exploiting that takes cache and bandwidth to spare. If I'm reading your f16 number right, the 5090 has both, the stride cost there is negligible. The 8060S has neither, so the stride is paid in full. Dequanting once means the FA loop reads f16, roughly double q8_0's bytes. Higher-bandwidth cards absorb that even with the rows unpermuted. On lower-bandwidth hardware like Strix Halo the doubled strided reads cost more than the decode they save at depth. The transpose is what lets dequant-once work everywhere. It hands the cost back, on a write we're already making. To isolate the transpose I benched dequant-once with a matching-layout (strided) scratch against this PR. Same pass, same scratch, only the layout differs:
Full coverage at 16k/32k, r=5, same session (base = 683f0c7):
Without the transpose, dequant-once vs base goes +3% @16k, −9% @32k, −33% @128k. My read of that progression: the doubled strided traffic costs nothing until the cache stops absorbing it, and the transpose gain (1.34× -> 1.53× -> 3.02×) is that cost handed back. Ryzen AI Max+ 395 (Strix Halo), 64 GB unified LPDDR5X-8000 shared with the CPU. Qwen3 30B A3B Q6_K, Native strided f16 still edges out the coalesced path in your numbers, so I'd expect strided ≈ coalesced on higher bandwidth cards. Only if it's of interest: the no-transpose variant is a host-only two-hunk change. I can post the diff here or push a branch to my fork. |
Contributor
|
It shouldn't matter if it's skipping 768B if it's still fetching whole cache lines. Isn't the cache line size 128B? |
Contributor
Author
|
Yes, 128B lines, fully used in both layouts, so the cost sits below the line level. The transpose wasn't theory-first, the version without it regressed, and that's what sent me measuring. This box is UMA so I could test where: the CPU shares the DRAM and memory controller. Sequential sustains 73 GB/s. The same data in 256B chunks back-to-back: 70, so chunking is free. The same 256B chunks at 1KB stride: 22. Same chunks, same fully-used lines, only the gaps differ. A 3.2× tax, consistent with the 3.02× the transpose recovers on the GPU at 128k — and it scales with the gap: 512B per 1KB lands at 30. I don't know exactly what's charging for it down there, my guess would be row-buffer locality. |
|
My observation is that with 2x Radeon 9070XT this pr results in faster prompt processing of Qwen 3.6 27b too (first measurement usually fluctates but subsequent are not: it seems that this pr makes PP lose less ptps as prompt grows): pr: |
fewtarius
added a commit
to fewtarius/CachyLLama
that referenced
this pull request
Jul 19, 2026
Backport of ggml-org#25494 (Nathanw1014), adapted for CachyLLama's APU/iGPU target hardware (32-128 GB UMA boxes, not discrete GPUs). Upstream mechanics are unchanged: a fused dequant+transpose shader materialises the FA scratch as a per-head-contiguous f16 K+V in prealloc_x, then the f16 FA path runs coalesced. Coopmat1 stops re-dequantising the whole KV cache inside every Q workgroup on every prefill step. What CachyLLama adds: 1. New common::host_available_ram() in common/host-ram.{h,cpp}. Extracted from the two duplicate inline implementations in kv-ssd-cache.cpp and kv_page_manager.cpp so three callers share one code path. 2. Three env-var controls on the device struct: GGML_VK_NO_FA_SCRATCH_TRANSPOSE=1 disable entirely GGML_VK_FA_SCRATCH_SAFETY_MB=N tune the runtime safety margin (default 1024 MiB) GGML_VK_FA_SCRATCH_FORCE=1 bypass the host-RAM check 3. Per-prefill host-RAM gate. Before allocating the scratch, check MemAvailable. If required_scratch + safety_margin > available, fall back to the slow coopmat1 path and log once. Without this, 32 GB boxes at 128k context with a 35B model would OOM on the ~512 MiB scratch. 4. CMake wiring. ggml-vulkan is built into a shared library; linking it against the shared llama-common creates a cyclic dependency (llama-common already depends on ggml-vulkan). Compile the single host-ram.cpp TU directly into ggml-vulkan instead. 5. Documentation in README.md and AGENTS.md, including the CachyLLama-only patch tracking table. CUDA/HIP fix deferred. Nathan's CUDA/HIP fix (fa-tile-dequant-on-load) has no upstream PR yet and requires CUDA/HIP toolchain to build-test, which is not available in this environment. Tracked separately for follow-up once upstream PR opens. Verified: - cmake --build builds clean (Vulkan backend, common library, all tests) - ctest passes for test-sampling, test-chat, test-tokenizer-0, test-grammar-parser, test-grammar-integration, test-chat-peg-parser, test-chat-auto-parser, test-chat-template, test-quantize-fns, test-ssd-cache-caps (10/10 tests, 0 failures) - test-backend-ops -o FLASH_ATTN_EXT runs without crashes on Vulkan + CPU - strings confirms dequant_q8_0_transpose shader, all three env vars, and the warning strings are linked into libggml-vulkan.so Hardware verification (gfx1151 numbers) requires a Strix Halo box. Independent reproduction is in the upstream thread (lev_werkstatt on 128 GB / MiniMax-M2.7 230B-A10B: pp512 +23-60% at 16k-128k). Closes: tracks ggml-org#25494
fewtarius
added a commit
to fewtarius/CachyLLama
that referenced
this pull request
Jul 19, 2026
Backport of ggml-org#25494 (Nathanw1014), adapted for CachyLLama's APU/iGPU target hardware (32-128 GB UMA boxes, not discrete GPUs). Upstream mechanics are unchanged: a fused dequant+transpose shader materialises the FA scratch as a per-head-contiguous f16 K+V in prealloc_x, then the f16 FA path runs coalesced. Coopmat1 stops re-dequantising the whole KV cache inside every Q workgroup on every prefill step. Host-RAM gate (CachyLLama-specific) Before allocating the scratch the FA dispatch checks whether the result fits in available host memory and falls back to the pre-fix coopmat1 path if not. The gate runs every prefill step and reads a fresh MemAvailable, so memory pressure from other workloads is detected immediately. Three incremental layers of safety: 1. Query reliability: common::host_available_ram_query() returns false on platforms where the answer is unreliable (Windows, kernels older than 3.14, sysinfo() failure). The gate treats unknown as "cannot allocate" and uses the slow path, instead of trusting a fabricated 8 GiB number. The legacy common::host_available_ram() keeps the 8 GiB fallback for SSD cache auto-sizing callers that want a planning number regardless. 2. Linux accuracy: switched from sysinfo.freeram (genuinely free RAM only) to /proc/meminfo MemAvailable (free + reclaimable cache, the kernel's own estimate of "available without swapping"). Matches macOS's more accurate vm_statistics64 free+inactive semantic. 3. UMA awareness: on integrated GPUs (Strix Halo, Apple Silicon, AMD APU, Intel integrated) the GPU memory pool IS host RAM. MemAvailable counts reclaimable page cache, but reclaiming it hurts SSD read-ahead and other system services. The gate reserves a 30% headroom when device->uma is true, preventing scratch allocations from causing cache thrash. Three env-var controls on the device struct: GGML_VK_NO_FA_SCRATCH_TRANSPOSE=1 disable the fast path entirely GGML_VK_FA_SCRATCH_SAFETY_MB=N tune the runtime safety margin (default 1024 MiB) GGML_VK_FA_SCRATCH_FORCE=1 bypass the host-RAM check Build wiring: ggml-vulkan is a shared library; linking it against the shared llama-common creates a cyclic dep. Compile the single host-ram.cpp TU directly into ggml-vulkan via target_sources(). Verified on Strix Halo (gfx1151) Qwen3-Coder-30B-A3B q8_0 KV at 15k context (Q4_K_XL quant, Vulkan): Generation: 194 -> 61 ms/tok (3.2x faster) Decode scales with KV cache size; the same mechanism now applies on memory-constrained boxes where the scratch would have OOMed. CUDA/HIP fix deferred. Nathan's CUDA/HIP fix (fa-tile-dequant-on-load) has no upstream PR yet and requires CUDA/HIP toolchain to build-test, which is not available in this environment. Closes: tracks ggml-org#25494
fewtarius
added a commit
to fewtarius/llama-ai
that referenced
this pull request
Jul 19, 2026
…ant fix) Brings in bd4f2875b vulkan : dequant q8_0 KV once in coopmat1, the backport of ggml-org/llama.cpp#25494 with CachyLLama host-RAM gating for 32 GB boxes. Verified on Strix Halo (gfx1151) Qwen3-Coder-30B-A3B q8_0 KV at 15k context: generation 194 -> 61 ms/tok (3.2x faster), scales with KV cache size.
fewtarius
added a commit
to fewtarius/llama-ai
that referenced
this pull request
Jul 19, 2026
…ant + host-RAM hardening) Brings in the merged commit for vulkan : dequant q8_0 KV once in coopmat1 (Nathanw1014 #25494 backport) with CachyLLama's three-layer host-RAM gate: 1. host_available_ram_query() returns false on platforms that can't answer reliably (Windows, kernels older than 3.14, sysinfo() failure). The FA scratch gate treats unknown as "cannot allocate" rather than trusting a fabricated 8 GiB number. Legacy SSD cache callers keep the 8 GiB fallback. 2. Linux: switched from sysinfo.freeram to /proc/meminfo MemAvailable. Matches macOS's free + reclaimable-cache semantic; the previous code under-reported available RAM on Linux and left perf on the table. 3. UMA: when the Vulkan device is integrated (Strix Halo, Apple Silicon, AMD APU, Intel integrated), the gate reserves a 30% headroom. On unified memory the GPU pool IS host RAM, and reclaiming page cache for a scratch allocation kills SSD throughput on other workloads. The fast path itself is unchanged: fused dequant+transpose shader writes a per-head-contiguous f16 K+V to prealloc_x, then the f16 FA path runs coalesced. Verified on Strix Halo (gfx1151) Qwen3-Coder-30B-A3B q8_0 KV at 15k context (Q4_K_XL quant, Vulkan): generation 194 -> 61 ms/tok (3.2x faster), scales with KV cache size. Closes: tracks ggml-org/llama.cpp#25494
Assisted-by: Claude (Opus 4.8)
Assisted-by: Claude (Opus 4.8)
Assisted-by: Claude (Opus 4.8)
Contributor
Author
|
Testing on a 3070 - CM2 was engaging the scratch path for no benefit (slight regression, 3–6% at 16k–32k), I implemented a gate for when CM2 is engaged. While reviewing I realized the tests could never reach the dequant path. Added FA test coverage, which turned up an actual bug with native-layout quant K/V, which is fixed and now has a regression test. Everything's validated both paths on my 3070 and Strix Halo (cm2-native + forced-KHR on the 3070; KHR_coopmat + scalar on the Strix) |
fewtarius
added a commit
to fewtarius/CachyLLama
that referenced
this pull request
Jul 23, 2026
…-org#25494 The bd4f287 / 37d2a1a backport of Nathanw1014's coopmat1 FA scratch shader dropped two guards that Nathan added in upstream patch 3/5 and 4/5 of ggml-org#25494: k->nb[1] >= k->nb[2] && v->nb[1] >= v->nb[2] !ctx->device->coopmat2 Without !coopmat2 the path engages on every Vulkan device with q8_0 K/V and a prefill of N >= 64, including devices (Strix Halo, RDNA3+) whose coopmat2 path already decodes quant K/V natively during the matrix load. The f16 scratch then drifts downstream matmul inputs and we land in mul_mat_q with OOB reads -> HSA_STATUS_ERROR_MEMORY_FAULT on HIP / vk::DeviceLostError on Vulkan. The nb[1] >= nb[2] gate restricts the path to KV cache layouts whose source memory order matches what the dequant_q8_0_transpose shader assumes; llama.cpp's get_k / get_v produce views that fall outside the gate, so we end up conservative (path not engaged on Strix Halo, which is the desired behavior regardless of coopmat2 status). CachyLLama-only additions (common::host_available_ram_query gate, GGML_VK_NO_FA_SCRATCH_TRANSPOSE / GGML_VK_FA_SCRATCH_FORCE / GGML_VK_FA_SCRATCH_SAFETY_MB env-var hooks) are preserved exactly as they were — only the two upstream guards are restored. References: fewtarius/llama-ai#8 (current Strix Halo regression) ggml-org#25494 (upstream origin of the backport)
smpurkis
pushed a commit
to smpurkis/halo-llama.cpp
that referenced
this pull request
Jul 24, 2026
Backport of ggml-org#25494 (Nathanw1014), adapted for CachyLLama's APU/iGPU target hardware (32-128 GB UMA boxes, not discrete GPUs). Upstream mechanics are unchanged: a fused dequant+transpose shader materialises the FA scratch as a per-head-contiguous f16 K+V in prealloc_x, then the f16 FA path runs coalesced. Coopmat1 stops re-dequantising the whole KV cache inside every Q workgroup on every prefill step. What CachyLLama adds: 1. New common::host_available_ram() in common/host-ram.{h,cpp}. Extracted from the two duplicate inline implementations in kv-ssd-cache.cpp and kv_page_manager.cpp so three callers share one code path. 2. Three env-var controls on the device struct: GGML_VK_NO_FA_SCRATCH_TRANSPOSE=1 disable entirely GGML_VK_FA_SCRATCH_SAFETY_MB=N tune the runtime safety margin (default 1024 MiB) GGML_VK_FA_SCRATCH_FORCE=1 bypass the host-RAM check 3. Per-prefill host-RAM gate. Before allocating the scratch, check MemAvailable. If required_scratch + safety_margin > available, fall back to the slow coopmat1 path and log once. Without this, 32 GB boxes at 128k context with a 35B model would OOM on the ~512 MiB scratch. 4. CMake wiring. ggml-vulkan is built into a shared library; linking it against the shared llama-common creates a cyclic dependency (llama-common already depends on ggml-vulkan). Compile the single host-ram.cpp TU directly into ggml-vulkan instead. 5. Documentation in README.md and AGENTS.md, including the CachyLLama-only patch tracking table. CUDA/HIP fix deferred. Nathan's CUDA/HIP fix (fa-tile-dequant-on-load) has no upstream PR yet and requires CUDA/HIP toolchain to build-test, which is not available in this environment. Tracked separately for follow-up once upstream PR opens. Verified: - cmake --build builds clean (Vulkan backend, common library, all tests) - ctest passes for test-sampling, test-chat, test-tokenizer-0, test-grammar-parser, test-grammar-integration, test-chat-peg-parser, test-chat-auto-parser, test-chat-template, test-quantize-fns, test-ssd-cache-caps (10/10 tests, 0 failures) - test-backend-ops -o FLASH_ATTN_EXT runs without crashes on Vulkan + CPU - strings confirms dequant_q8_0_transpose shader, all three env vars, and the warning strings are linked into libggml-vulkan.so Hardware verification (gfx1151 numbers) requires a Strix Halo box. Independent reproduction is in the upstream thread (lev_werkstatt on 128 GB / MiniMax-M2.7 230B-A10B: pp512 +23-60% at 16k-128k). Closes: tracks ggml-org#25494
|
more numbers on gfx1151 with a mix of gemma4/qwen3.6 models, dense and MoE, depth 0/16384/32768. Let me know if you need more. Models from unsloth: cmd : llama-bench -ngl 999 -fa 1 -dio 1 -m ~/models/Qwen3.6-27B-UD-Q4_K_XL.gguf -m ~/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -m ~/models/gemma-4-31B-it-UD-Q4_K_XL.gguf -m ~/models/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -n 0 -p 512 -d 0,16384,32768 -ctk q8_0 -ctv q8_0 KV q8_0 before
build: 9b2a088 (10190) KV q8_0 after
build: 9b2a088 (10190) and KV f16 since it was also done in previous comments KV f16 before
build: 9b2a088 (10190) KV f16 after
build: 9b2a088 (10190) TLDR looks very good at depth with KV Q8, the rest seems unchanged or within error range. |
smpurkis
pushed a commit
to smpurkis/halo-llama.cpp
that referenced
this pull request
Jul 30, 2026
Backport of ggml-org#25494 (Nathanw1014), adapted for CachyLLama's APU/iGPU target hardware (32-128 GB UMA boxes, not discrete GPUs). Upstream mechanics are unchanged: a fused dequant+transpose shader materialises the FA scratch as a per-head-contiguous f16 K+V in prealloc_x, then the f16 FA path runs coalesced. Coopmat1 stops re-dequantising the whole KV cache inside every Q workgroup on every prefill step. What CachyLLama adds: 1. New common::host_available_ram() in common/host-ram.{h,cpp}. Extracted from the two duplicate inline implementations in kv-ssd-cache.cpp and kv_page_manager.cpp so three callers share one code path. 2. Three env-var controls on the device struct: GGML_VK_NO_FA_SCRATCH_TRANSPOSE=1 disable entirely GGML_VK_FA_SCRATCH_SAFETY_MB=N tune the runtime safety margin (default 1024 MiB) GGML_VK_FA_SCRATCH_FORCE=1 bypass the host-RAM check 3. Per-prefill host-RAM gate. Before allocating the scratch, check MemAvailable. If required_scratch + safety_margin > available, fall back to the slow coopmat1 path and log once. Without this, 32 GB boxes at 128k context with a 35B model would OOM on the ~512 MiB scratch. 4. CMake wiring. ggml-vulkan is built into a shared library; linking it against the shared llama-common creates a cyclic dependency (llama-common already depends on ggml-vulkan). Compile the single host-ram.cpp TU directly into ggml-vulkan instead. 5. Documentation in README.md and AGENTS.md, including the CachyLLama-only patch tracking table. CUDA/HIP fix deferred. Nathan's CUDA/HIP fix (fa-tile-dequant-on-load) has no upstream PR yet and requires CUDA/HIP toolchain to build-test, which is not available in this environment. Tracked separately for follow-up once upstream PR opens. Verified: - cmake --build builds clean (Vulkan backend, common library, all tests) - ctest passes for test-sampling, test-chat, test-tokenizer-0, test-grammar-parser, test-grammar-integration, test-chat-peg-parser, test-chat-auto-parser, test-chat-template, test-quantize-fns, test-ssd-cache-caps (10/10 tests, 0 failures) - test-backend-ops -o FLASH_ATTN_EXT runs without crashes on Vulkan + CPU - strings confirms dequant_q8_0_transpose shader, all three env vars, and the warning strings are linked into libggml-vulkan.so Hardware verification (gfx1151 numbers) requires a Strix Halo box. Independent reproduction is in the upstream thread (lev_werkstatt on 128 GB / MiniMax-M2.7 230B-A10B: pp512 +23-60% at 16k-128k). Closes: tracks ggml-org#25494
smpurkis
pushed a commit
to smpurkis/halo-llama.cpp
that referenced
this pull request
Jul 30, 2026
…-org#25494 The bd4f287 / 37d2a1a backport of Nathanw1014's coopmat1 FA scratch shader dropped two guards that Nathan added in upstream patch 3/5 and 4/5 of ggml-org#25494: k->nb[1] >= k->nb[2] && v->nb[1] >= v->nb[2] !ctx->device->coopmat2 Without !coopmat2 the path engages on every Vulkan device with q8_0 K/V and a prefill of N >= 64, including devices (Strix Halo, RDNA3+) whose coopmat2 path already decodes quant K/V natively during the matrix load. The f16 scratch then drifts downstream matmul inputs and we land in mul_mat_q with OOB reads -> HSA_STATUS_ERROR_MEMORY_FAULT on HIP / vk::DeviceLostError on Vulkan. The nb[1] >= nb[2] gate restricts the path to KV cache layouts whose source memory order matches what the dequant_q8_0_transpose shader assumes; llama.cpp's get_k / get_v produce views that fall outside the gate, so we end up conservative (path not engaged on Strix Halo, which is the desired behavior regardless of coopmat2 status). CachyLLama-only additions (common::host_available_ram_query gate, GGML_VK_NO_FA_SCRATCH_TRANSPOSE / GGML_VK_FA_SCRATCH_FORCE / GGML_VK_FA_SCRATCH_SAFETY_MB env-var hooks) are preserved exactly as they were — only the two upstream guards are restored. References: fewtarius/llama-ai#8 (current Strix Halo regression) ggml-org#25494 (upstream origin of the backport)
fastlorenzo
added a commit
to fastlorenzo/containers
that referenced
this pull request
Aug 9, 2026
llama-swap carrying ggml-org/llama.cpp#25494 (unmerged): dequant q8_0 KV once in the Vulkan coopmat1 flash-attention path instead of once per workgroup — the exact configuration ai-box's generative routes run (kv: q8_0 + -fa 1 on gfx1151). Author-reported on a 30B-A3B MoE with q8_0 KV: pp512 @32k 200->282 t/s, @65k 99->166 t/s, decode flat, greedy output byte-identical. Two build stages reproduce upstream llama.cpp's own .devops/vulkan.Dockerfile (same base, packages, cmake flags, plus the embedded web UI), applying patches/*.patch to the checked-out release tag first, then graft the patched llama-server AND the ggml backend .so files onto the stock llama-swap image ai-box already runs — the Vulkan FA code is in libggml-vulkan.so, not the binary, so a binary-only copy would benchmark as a null result. Grafting onto the same base keeps the A/B one variable. VERSION (llama.cpp tag) and BASE_IMAGE (the llama-swap build from that same tag) must move together; renovate is disabled for this app so a bot cannot bump one alone. amd64 only — one gfx1151 APU. Delete this app when #25494 merges upstream. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeffbolznv
reviewed
Aug 9, 2026
| const bool v_quant = v->type != GGML_TYPE_F16 && v->type != GGML_TYPE_BF16 && v->type != GGML_TYPE_F32; | ||
| const bool use_dequant_kv = k_quant && v_quant && neq1 >= 64 && | ||
| k->nb[0] == ggml_type_size(k->type) && v->nb[0] == ggml_type_size(v->type) && | ||
| k->nb[1] >= k->nb[2] && v->nb[1] >= v->nb[2] && |
Contributor
There was a problem hiding this comment.
I think the shader is assuming a specific permutation, but this check is looser than that.
| { const std::vector<uint32_t> pc = { (uint32_t)HSV, (uint32_t)nev2, (uint32_t)KV, 0, v_nel }; | ||
| ggml_vk_dispatch_pipeline(ctx, subctx, tr_v, { v_buf, v_dst }, pc, { v_nel, 1, 1 }); } | ||
| ggml_vk_sync_buffers(ctx, subctx); | ||
| ctx->prealloc_x_need_sync = true; |
Contributor
There was a problem hiding this comment.
This needs to be set at the end of the function, since the contents of prealloc_x are used in the FA dispatch. Otherwise it can be clobbered in the next ggml_vk_sync_buffers. (Codex found this one)
Contributor
|
I missed that this also applies to non-coopmat GPUs, your title claims coopmat1-only. AMD Radeon Pro VII:
Intel A770 on Linux:
The AMD result is okay, Intel is not. We might have to disable this on Intel, but it depends on how it behaves on Windows and also on Intel Battlemage. @rillomas can you take a look? |
fastlorenzo
added a commit
to fastlorenzo/containers
that referenced
this pull request
Aug 11, 2026
…ch (#2883) llama-swap carrying ggml-org/llama.cpp#25494 (unmerged): dequant q8_0 KV once in the Vulkan coopmat1 flash-attention path instead of once per workgroup — the exact configuration ai-box's generative routes run (kv: q8_0 + -fa 1 on gfx1151). Author-reported on a 30B-A3B MoE with q8_0 KV: pp512 @32k 200->282 t/s, @65k 99->166 t/s, decode flat, greedy output byte-identical. Two build stages reproduce upstream llama.cpp's own .devops/vulkan.Dockerfile (same base, packages, cmake flags, plus the embedded web UI), applying patches/*.patch to the checked-out release tag first, then graft the patched llama-server AND the ggml backend .so files onto the stock llama-swap image ai-box already runs — the Vulkan FA code is in libggml-vulkan.so, not the binary, so a binary-only copy would benchmark as a null result. Grafting onto the same base keeps the A/B one variable. VERSION (llama.cpp tag) and BASE_IMAGE (the llama-swap build from that same tag) must move together; renovate is disabled for this app so a bot cannot bump one alone. amd64 only — one gfx1151 APU. Delete this app when #25494 merges upstream. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Arc Pro B70, Ubuntu 26.04, 7.0.0-29-generic, Mesa 26.1.6
Arc Pro B70, Windows 24H2, Driver 32.0.101.8864
Arc Pro B50, Windows 11 25H2, Driver 32.0.101.8864, 64GB RAM
Arc A770, Windows 11 25H2, Driver 32.0.101.8864, 32GB RAM
UX7-368H, Windows 11 25H2, Driver 32.0.101.8864, 64GB RAM
Core 5 330, Windows 11 25H2, Driver 32.0.101.8974, 16GB RAM
U9-288V, Windows 11 26H2, Driver 32.0.101.8974, 32GB RAM
U7-265H, Windows 11 24H2, Driver 32.0.101.8974, 32GB RAM
U7-265H, Windows 11 24H2, Driver 32.0.101.8864, 32GB RAM
llama-bench logsArc Pro B70, Ubuntu 26.04, 7.0.0-29-generic, Mesa 26.1.6BeforeAfterArc Pro B70, Windows 24H2, Driver 32.0.101.8864BeforeAfterArc Pro B50, Windows 25H2, Driver 32.0.101.8864BeforeAfterArc A770, Windows 11 25H2, Driver 32.0.101.8864, 32GB RAMBeforeAfterUX7-368H, Windows 11 25H2, Driver 32.0.101.8864, 64GB RAMBeforeAfterCore 5 330, Windows 11 25H2, Driver 32.0.101.8974, 16GB RAMBeforeAfterU9-288V, Windows 11 26H2, 32.0.101.8974, 32GB RAMBeforeAfterU7-265H, Windows 11 24H2, Driver 32.0.101.8974, 32GB RAMBeforeAfterU7-265H, Windows 11 24H2, Driver 32.0.101.8864, 32GB RAMBeforeAfter |
fewtarius
added a commit
to fewtarius/CachyLLama
that referenced
this pull request
Aug 13, 2026
…014) Bring q4_0, q4_1, q5_0, q5_1 onto the same DEQUANT_TRANSPOSE path that q8_0 already uses (CachyLLama via PR ggml-org#25494). All five are bit-exact equal to the native FA f16 path once dequant-once is engaged, and on Strix Halo (RADV gfx1151) q4 lands the same 2.64x prefill win that q8 gets. iq4_nl stays native-only (upstream 8161641). Cherry-picked from nathanw1014/llama.cpp strix-halo-vulkan (569987e); the q8_0 base was already in CachyLLama so we only port the four new types. Generation in vulkan-shaders-gen.cpp widened the type list, and the four create_pipeline() calls share the same {256*16, 1, 1} workgroup.
Contributor
|
Thank you. Maybe we need to disable it for non-coopmat Intel and integrated Intel? I'd rather not block this PR for longer while we track this down. |
Contributor
Based on the data we have now, I guess enabling only for Xe2 dGPU looks OK? Others have mixed results or regressions. |
0cc4m
requested changes
Aug 15, 2026
| (uint64_t)ggml_nelements(v) * sizeof(ggml_fp16_t) <= ctx->device->properties.limits.maxStorageBufferRange && | ||
| ctx->device->pipeline_dequant_transpose[k->type] != nullptr && | ||
| ctx->device->pipeline_dequant_transpose[v->type] != nullptr && | ||
| // coopmat2 already decodes quant K/V during the matrix load, so the f16 scratch is pure overhead there |
Contributor
There was a problem hiding this comment.
I don't think coopmat2 does anything different here. This comment is just speculation. It should just say that the coopmat2 path does not benefit from this feature.
| ctx->device->pipeline_dequant_transpose[k->type] != nullptr && | ||
| ctx->device->pipeline_dequant_transpose[v->type] != nullptr && | ||
| // coopmat2 already decodes quant K/V during the matrix load, so the f16 scratch is pure overhead there | ||
| !ctx->device->coopmat2; |
Contributor
There was a problem hiding this comment.
Please limit the feature to non-Intel and Intel+coopmat support here.
Contributor
Author
|
I have added the suggested gating, and cleaned up the comment. thanks. |
This comment was marked as low quality.
This comment was marked as low quality.
Contributor
|
Then you didn't test correctly, most likely. Make sure to run with q8_0 kv and with some kv depth, or the effect of Flash Attention is too small. |
This comment was marked as low quality.
This comment was marked as low quality.
|
@sswtodo you are offtopic : this is about q8_0 KV quant, not Q8 model quant. |
This comment was marked as low quality.
This comment was marked as low quality.
rillomas
reviewed
Aug 18, 2026
Nathanw1014
force-pushed
the
vulkan-coopmat1-fa-dequant-transpose
branch
from
August 19, 2026 06:53
5944326 to
b890946
Compare
0cc4m
approved these changes
Aug 19, 2026
jeffbolznv
approved these changes
Aug 19, 2026
thecodacus
pushed a commit
to thecodacus/llama.cpp
that referenced
this pull request
Sep 7, 2026
* vulkan : dequant q8_0 KV once in coopmat1 Assisted-by: Claude (Opus 4.8) * vulkan : fall back instead of aborting when FA scratch exceeds maxStorageBufferRange * vulkan : require KV-cache layout in FA dequant path Assisted-by: Claude (Opus 4.8) * vulkan : skip FA dequant path on coopmat2 Assisted-by: Claude (Opus 4.8) * tests : add contiguously-allocated quant K/V FA tests Assisted-by: Claude (Opus 4.8) * vulkan : trim comments * vulkan : tighten permutation checks for FA path * vulkan : set prealloc_x_need_sync after the FA dispatch * vulkan : exclude Intel Xe1 from FA dequant path
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Removing redundant coopmat1 FA dequantization of KV at prefill. Coopmat1 currently Dequants 32 times (once per workgroup), Reorganises the F16 KV to per-head-contiguous in scratch to enable faster memory-bound read by FA.
Closes #25491
Additional information
Scratch size grows with KV cache size, the trade off for inflight memory-usage vs token throughput: (~268 MB @128k).
Patched vs unpatched greedy output (temp 0, seed 1, ~15.5k-token prompt) is byte-identical, no change in results, just speed.
Unpatched:
Patched:
Requirements
Assisted with benchmarking, analysis, and review; design + implementation directed by me