llama : rotate activations for better quantization#21038
Conversation
|
This PR does seem to improve PPL as compared to tip of master using I ran everything CPU-only backend so as to also compare https://github.com/Aaryan-Kapoor/llama.cpp/tree/turboquant-tq3_0 from @Aaryan-Kapoor to see how that specific For reference:
mainline llama.cpp PPL wiki.test.rawData Resultsmainline llama.cpp PPL wiki.test.raw
👈 DetailsExperimentMeasure perplexity against wiki.test.raw varying kv-cache quantization. Test QuantTest Rig
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=OFF -DGGML_RPC=OFF -DGGML_BLAS=OFF -DGGML_VULKAN=OFF
cmake --build build --config Release -j $(nproc)Command |
|
I've run a pretty big permutation matrix for the Qwen3.5-9B model against the master branch and this PR branch. The following images are the ratios vs F16/F16: Master branch PPL and KLD and this PR PPL and KLD Here's the table of data comparing master vs this PR
And attached here is the full raw run data for every invocation: |
Interesting results, which indicate that Q4_0, even before this PR, is superior to TQ_3. However as a word of caution, this is likely due to a very experimental and early implementation of this specific fork and not indicative of the actual performance of TurboQuant, as it is supposed to be effectively lossless compared to fp16 kv cache. That is not the case here. |
|
I think we can actually rotate the V tensor using smaller matrices (64 x 64) which should result in better quality of the V cache. We cannot do that for the Q and K because it would not preserve the dot product. Pushed a change to do that 832e326 and just from a quick PPL sanity check it looks slightly better. Note, using 64 instead of 32 because the Metal matrix multiplication kernels require Edit:
On second thought, I think it does preserve it. Something to try too. Here is the patch: More rotations for Q and Kdiff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
index 8dfc92b71..84bcf26be 100644
--- a/src/llama-graph.cpp
+++ b/src/llama-graph.cpp
@@ -2096,8 +2096,9 @@ static std::unique_ptr<llm_graph_input_attn_kv> build_attn_inp_kv_impl(
const bool can_rot =
!hparams.is_n_embd_k_gqa_variable() &&
!hparams.is_n_embd_v_gqa_variable() &&
- ggml_is_power_of_2(hparams.n_embd_head_k()) &&
+ //ggml_is_power_of_2(hparams.n_embd_head_k()) &&
//ggml_is_power_of_2(hparams.n_embd_head_v()) &&
+ hparams.n_embd_head_k() % 64 == 0 &&
hparams.n_embd_head_v() % 64 == 0 &&
hparams.n_embd_head_k() >= 64 &&
hparams.n_embd_head_v() >= 64 &&
@@ -2105,7 +2106,7 @@ static std::unique_ptr<llm_graph_input_attn_kv> build_attn_inp_kv_impl(
ggml_is_quantized(mctx_cur->type_v());
if (can_rot) {
- const auto nk = hparams.n_embd_head_k();
+ const auto nk = 64;
const auto nv = 64;
inp->self_rotk = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, nk, nk);
@@ -2453,8 +2454,9 @@ llm_graph_input_attn_kv_iswa * llm_graph_context::build_attn_inp_kv_iswa() const
const bool can_rot =
!hparams.is_n_embd_k_gqa_variable() &&
!hparams.is_n_embd_v_gqa_variable() &&
- ggml_is_power_of_2(hparams.n_embd_head_k()) &&
+ //ggml_is_power_of_2(hparams.n_embd_head_k()) &&
//ggml_is_power_of_2(hparams.n_embd_head_v()) &&
+ hparams.n_embd_head_k() % 64 == 0 &&
hparams.n_embd_head_v() % 64 == 0 &&
hparams.n_embd_head_k() >= 64 &&
hparams.n_embd_head_v() >= 64 &&
@@ -2462,7 +2464,7 @@ llm_graph_input_attn_kv_iswa * llm_graph_context::build_attn_inp_kv_iswa() const
ggml_is_quantized(mctx_cur->get_base()->type_v());
if (can_rot) {
- const auto nk = hparams.n_embd_head_k();
+ const auto nk = 64;
const auto nv = 64;
inp->self_rotk = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, nk, nk); |
It can't approach turboquant's performance because it's missing the QJL part of turboquant. (I think?) |
|
I re-ran the suite against the updated new commit, looks like the KLD has generally improved a little bit across the board! Heatmap results: 
Details data table
Full run archives: |
What about higher context, like for example 100K? I believe that's where kv cache quantization actually harms the model from multiplications errors that accumulated. |
|
@AesSedai Thanks for the results. It seems important to track the KLD rather than PPL (maybe more significant for Qwen3.5). I did some additional tests with randomized Hadamard matrices (as suggested by @sashkboos in #6444 (comment)). Will follow-up in next PRs. |
|
|
||
| auto * data = (float *) tensor->data; | ||
|
|
||
| data[0*n + 0] = 1.0 / sqrtf(n); |
There was a problem hiding this comment.
This always gets me, at least this one makes esthetic sense. :)
|
There is a slowdown which is expected, however probably we should have a flag to opt out?
|
PolarQuant is just a rotation by a random matrix to spread the energy of any outliers across all dimensions prior to quantization. A normalized Hadamard matrix may or may not produce similar results. QJL is extremely important however, PolarQuant (and your Hadamard) gives biased attention scores (despite the good seeming MSE) which results in rank flips. QJL uses the 'unreasonable effectiveness of random projections' to give unbiased results. The theory (more accurately, the lemma) is that randomly projecting a vector to lower dimensions will preserve pairwise distances (and therefore dot products) in expectation. TurboQuant quantizes the error residual all the way to 1-bit (the sign) which somehow works well enough. The best technical yet intuitive explanation I saw on the subject so far is by some AI Researcher from Amazon: https://darshanfofadiya.com/research-papers/turboquant/ |
I don't think we should, the only reason to quantize kv-cache is to save memory, if that comes at the cost of speed, so be it, an option to reduce quality does not make sense (unless you think this is not a general quality improvement). |
|
My opinion is that any new breaking change should have an option to turn it off, in case of any unforeseen issues. This change changes outputs of the LLM (even if they are materially better), so just as an example someone having tests downstream would see tests break.
I think when using |
This comment was marked as off-topic.
This comment was marked as off-topic.
You could say we are in the business of making breaking changes and that If anything, I guess adding an env-var for some grace period would be sufficient.
Well, until you pointed out the below I would have said that no such test were likely to exist. :)
|
I disagree. You could use quantized KV Cache to save memory in order to put more layers on the GPU. In that case, the purpose of it would be to increase speed and this change would be counterproductive to that goal. |
You'd most likely gain much more than you lose, so calling it counterproductive is perhaps a bit far fetched. |
Actually I'm not sure it would be worth it to be on by default for q8_0 since the benefits are much lesser compared to q4 and the performance loss is significant Details
|
Granted, it's probably pointless for q8_0, so perhaps add a check. |
|
It seems that evaluating AIME25 could be another sensitivity test for confirming the improvement of rotating the activations. I did a few runs today with Here is the table with the hyperlinks preserved for each score:
Used the new It's interesting that with Example of reasoning trace with KV type `Q4_0` without rotations
Regarding For reference, the expected score of this eval per the 
|
Yeah, based on the score, there is a clear benefit from rotating activations for q8_0, and I think it makes sense to have them on by default as the purpose of Q8_0 is to save memory for KV Cache for the same quality and with attn-rot it is noticeably closer to that goal. Furthermore, I'm not sure if AIME25 tests at long context as well. Quality differences might become even more noticeable at a longer context. Performance is a concern, however. In the end, I think the best course of action would be to make attn-rots an opt-out/opt-in option with a simple flag. |
- Register GGML_TYPE_TURBO3_0 and GGML_TYPE_TURBO4_0 in kv_cache_types so --cache-type-k turbo3 / --cache-type-v turbo3 are recognized - Fix double V un-rotation: upstream PR ggml-org#21038 (attn_rot_v) already handles Hadamard rotation for quantized KV cache types including turbo. Make TurboQuant WHT fallback only when upstream rotation is not active (else if instead of sequential if blocks)
Includes: - fix: handle non-capturing groups (?:...) in JSON schema pattern converter (ggml-org#21124) - memory: respect unified KV cache in hybrid memory for eval tasks (ggml-org#21224) - fix: CUDA FA kernel selection, head dimension 512 support - rotate activations for better quantization (ggml-org#21038) - Various parser, jinja, webui, and CI fixes Conflicts resolved: - llama-kv-cache.cpp: keep TurboQuant InnerQ stubs + upstream Hadamard helpers - llama-graph.cpp: keep TurboQuant V-padding + upstream self_v_rot - fattn-tile.cu: add upstream D=512 before TurboQuant HIP guard - fattn.cu: combine D=512 (upstream) + D=640 (TurboQuant) exclusions
Notable upstream changes: - ggml-org#21038: rotate activations for better KV cache quantization - ggml-org#21074: generic NVFP4 MMQ kernel - ggml-org#21271: fix FA kernel selection logic - ggml-org#21238: fix mmvq mmid kernel selection - ggml-org#20998: FA support for head dim 512 - ggml-org#20920: docker cuda12 bump to 12.9.1 Conflicts resolved: - fattn.cu: took upstream (adds head_dim 512 exclusion) - mmq.cu/mmq.cuh: kept both TQ3 types + upstream additions - llama-graph.cpp: kept both turbo WHT + upstream v_rot - docker.yml: took upstream - tests/CMakeLists.txt: took upstream
|
It appears attention rotations are currently disabled for the Gemma 4 architecture. |
…V cache quants Adds graph-level Hadamard rotation for Q/K/V when using standard quant types (q4_0, q8_0, etc.) in the KV cache. Dramatically improves quantized KV quality. Our turbo V un-rotation takes priority when turbo types are used. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tion Turbo types have their own FWHT rotation in set-rows/fattn. The generic rotation from PR ggml-org#21038 would double-rotate, corrupting results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* llama : rotate activations for better quantization * cont : rotate V more + refactor * cont : rotate caches separately + support non-power-of-2 head sizes * cont : simplify * cont : add reference for V rotation * cont : refactor * cont : support context shift * cont : consolidate * cont : dedup + allow different types for the rotation matrix * cont : add env variable to disable rotation * cont : simplify attn rot kv cache logic + rename env * cont : pre-compute the Hadamard matrices
The online Hadamard rotation (PR ggml-org#21038) exists to suppress quantization error. On an F16 lid cache it is a no-op and costs one extra mul_mat per layer. Remove the force-enable for GLM_DSA and DEEPSEEK2 and let the lid cache follow the standard rule (quantized type + aligned head dim). Guard the Hadamard mul_mats in the builder and the set_input_k_rot call on self_k_rot_lid being non-null, since it can legitimately be null now.
|
Just wanted to leave this image here since this post is referenced quite a bit: The following results come from the official SLANG website, but they show that GPTQ and AIME to be sensitive to KV quant. It's NVFP4 but still very relevant, I think. @ggerganov maybe you can incorporate that into your future testing suite like you did with the python-based AIME testing. 
|
* llama : rotate activations for better quantization * cont : rotate V more + refactor * cont : rotate caches separately + support non-power-of-2 head sizes * cont : simplify * cont : add reference for V rotation * cont : refactor * cont : support context shift * cont : consolidate * cont : dedup + allow different types for the rotation matrix * cont : add env variable to disable rotation * cont : simplify attn rot kv cache logic + rename env * cont : pre-compute the Hadamard matrices
* llama : rotate activations for better quantization * cont : rotate V more + refactor * cont : rotate caches separately + support non-power-of-2 head sizes * cont : simplify * cont : add reference for V rotation * cont : refactor * cont : support context shift * cont : consolidate * cont : dedup + allow different types for the rotation matrix * cont : add env variable to disable rotation * cont : simplify attn rot kv cache logic + rename env * cont : pre-compute the Hadamard matrices
Wires the Hadamard scaffold (landed earlier on this branch) into ml/nn/attention.go. When OLLAMA_KV_ROTATE=1 and the head dim is a multiple of 64, Q/K/V are rotated by a fixed normalized 64×64 Sylvester matrix before cache.Put; the attention output is rotated again to undo V's rotation (Sylvester H is symmetric so H·H = I). Math: orthogonal H satisfies (QH)(KH)^T = QK^T so attention scores are preserved. Storing rotated K/V in the cache is the whole point — it smooths the per-coordinate distribution before quantization, which is what closes the q4_0/q8_0 quality gap to fp16 (upstream PR ggml-org/llama.cpp#21038, commit 744c0c73). Helpers added in ml/nn/hadamard.go: - hadamard64 — package-level cached float slice (computed once) - blockRotate(ctx, t, h) — applies H_64 block-diagonally along Dim(0) - hadamardTensor(ctx) — materializes hadamard64 as a backend tensor - noteIncompatibleHeadDim — one-shot slog.Warn per head_dim when the flag is set but the dim isn't compatible Default off: when OLLAMA_KV_ROTATE is unset (the default), behavior is byte-identical to before — every gate short-circuits on the envconfig check before any rotation work happens. Design doc updated to reflect what landed and to note two known limitations: (1) the gate reads the env each call, so mid-process toggle would mix rotated/unrotated entries; (2) the gate checks Q's head dim only, so head-dim asymmetry in V is not yet handled. llamarunner cherry-pick of upstream commit 744c0c73 deferred to a separate follow-up.
Wires the Hadamard scaffold (landed earlier on this branch) into ml/nn/attention.go. When OLLAMA_KV_ROTATE=1 and the head dim is a multiple of 64, Q/K/V are rotated by a fixed normalized 64×64 Sylvester matrix before cache.Put; the attention output is rotated again to undo V's rotation (Sylvester H is symmetric so H·H = I). Math: orthogonal H satisfies (QH)(KH)^T = QK^T so attention scores are preserved. Storing rotated K/V in the cache is the whole point — it smooths the per-coordinate distribution before quantization, which is what closes the q4_0/q8_0 quality gap to fp16 (upstream PR ggml-org/llama.cpp#21038, commit 744c0c73). Helpers added in ml/nn/hadamard.go: - hadamard64 — package-level cached float slice (computed once) - blockRotate(ctx, t, h) — applies H_64 block-diagonally along Dim(0) - hadamardTensor(ctx) — materializes hadamard64 as a backend tensor - noteIncompatibleHeadDim — one-shot slog.Warn per head_dim when the flag is set but the dim isn't compatible Default off: when OLLAMA_KV_ROTATE is unset (the default), behavior is byte-identical to before — every gate short-circuits on the envconfig check before any rotation work happens. Design doc updated to reflect what landed and to note two known limitations: (1) the gate reads the env each call, so mid-process toggle would mix rotated/unrotated entries; (2) the gate checks Q's head dim only, so head-dim asymmetry in V is not yet handled. llamarunner cherry-pick of upstream commit 744c0c73 deferred to a separate follow-up.
* llama : rotate activations for better quantization * cont : rotate V more + refactor * cont : rotate caches separately + support non-power-of-2 head sizes * cont : simplify * cont : add reference for V rotation * cont : refactor * cont : support context shift * cont : consolidate * cont : dedup + allow different types for the rotation matrix * cont : add env variable to disable rotation * cont : simplify attn rot kv cache logic + rename env * cont : pre-compute the Hadamard matrices
Investigation of master llama.cpp ggml-org#21038 attention rotation in our fork, ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE / LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork (rotation OFF on both sides); the contribution is per-side control plus a matrix documenting which models want which knob. Headline finding while running the matrix: corpus PPL on wikitext-2 is unreliable for KV-quantization evaluation on gemma-class instruct models. Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it GGUFs; KLD against the fp16-KV reference points the opposite direction. Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512 and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best on row). The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B, large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against fp16-KV logits as the oracle for KV-quant evaluation on this model class. Independently observed prior: vektorprime in ggml-org#21394, AesSedai's tables in ggml-org#21038, ggerganov's "track KLD rather than PPL" comment in same PR, localbench. This paper adds the controlled per-side measurement, the cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise floor on Metal, and the resulting engineering policy.
20 participants
Overview
In anticipation of the incoming flood of vibe generated PRs implementing TurboQuant, I'm raising the baseline a bit using a very simple interpretation of the idea of using Hadamard transform to reduce outliers in the attention and improve the quantization quality:
This works because:
The implementation is very simple and backend agnostic - use Hadamard matrix of size
n x n, normalized by1/sqrt(n)so that it is orthonormal and can be used both for the forward and backward rotation. Technically any rotation matrix (and it's inverse) should work - I just think this is what is commonly used due to it's simplicity. The implementation does not introduce new types and is compatible with all existing quantizations. It adds 4 matrix multiplication operators in the attention, though I think some of them can be fused in the attention weights (similar to QuaRot).I don't know what is the impact of the remaining techniques explained in TurboQuant (PolarQuant, QJL, etc.). They could be important and can potentially improve further on top of this. In any case, having a better baseline at almost 0 cost won't hurt. Only based on the PPL data below, I think this should never be worse that before, though it needs a bit more evaluation.
Note: MLA is not supported
Additional information
Here are some PPL results before and after using base models. Ideally, there should be KLD data too, but leaving it for people to play with it and see if it looks good.
Model: Qwen3 0.6B BF16
https://huggingface.co/Qwen/Qwen3-0.6B-Base
Baseline F16 cache: PPL = 13.6711 +/- 0.21422
Model: Qwen3 8B BF16
https://huggingface.co/Qwen/Qwen3-8B-Base
Baseline F16 cache: PPL = 7.3203 +/- 0.09901
Model: Gemma3 4B Q8_0
https://huggingface.co/google/gemma-3-4b-pt
Baseline F16 cache: PPL = 7.6905 +/- 0.10483
Model: Qwen3.5 4B F16
https://huggingface.co/Qwen/Qwen3.5-4B-Base
Baseline F16 cache: PPL = 8.3266 +/- 0.11623
TODOs
LLAMA_ATTN_ROT_DISABLE)Next PRs
Requirements