CUDA backend: 3-bit uniform KV cache (turbo3, 4.6x compression, 96% f16 speed)

[nalditopr]

Summary

CUDA backend for TurboQuant 3-bit KV cache compression on NVIDIA GPUs. Based on Lucien2468/Ollama-TurboQuant-Integration design.

turbo3: Simple 3-bit uniform quantization — round(x/d) clamped to [-4,3], packed 8 elements per 3 bytes. dp4a hardware dot product. No rotation, no codebooks. Just works.

--cache-type-k turbo3 --cache-type-v turbo3

Flash attention auto-enabled — no -fa on needed.

Benchmarks (Qwen3.5-35B-A3B MoE, RTX 5090 32GB)

Cache	Prefill (tok/s)	Decode (tok/s)	vs f16	Compression
f16	6,956	198	100%	1.0x
turbo3	6,774	189	96%	4.6x

1M Context on 32GB VRAM

	f16	turbo3
KV cache	20.5 GiB	4.5 GiB
Total GPU	44.3 GiB (spills to RAM)	32.4 GiB (fits!)
Decode	unusable (RAM thrashing)	106 tok/s

Implementation

Block format: [d:fp16][qs:12B] = 14 bytes per 32 elements (3.5 bpw)

CUDA files

• turbo-quant.cu/cuh — dequant kernel, declarations
• vecdotq.cuh — vec_dot_turbo3_0_q8_1 with dp4a (from Lucien reference)
• cpy-utils.cuh — GPU quantize for set_rows
• dequantize.cuh — float2 dequant interface
• cpy.cu — turbo3→F32 CPY for FA graph dequant
• mmvq.cu — MMVQ dispatch
• convert.cu — to_fp16/to_fp32 dispatch
• common.cuh — type traits
• fattn.cu — returns NONE (uses graph-level dequant + MMA)
• ggml-cuda.cu — supports_op for MUL_MAT, SET_ROWS, GET_ROWS, CPY

Graph/context

• llama-graph.cpp — FA: cast turbo3 K/V to F32 before MMA
• llama-context.cpp — auto-enable flash_attn for turbo3

CPU

• ggml-turbo-quant.c — quantize/dequant matching CUDA format
• ggml-common.h — block_turbo3_0 struct

Test plan

• Output quality on Qwen3.5-35B-A3B (identical to f16)
• 1M context fits in 32GB VRAM
• Auto-FA works without explicit flag
• Benchmarks: 96% f16 decode, 97% prefill
• Perplexity evaluation
• Dense model testing (where KV savings are larger)

🤖 Generated with Claude Code

[TheTom]

Please sync to TOT and resolve conflicts

[TheTom]

Thank you @nalditopr for the CUDA implementation. The 96% f16 decode speed with dp4a is impressive engineering, and the 1M context fit on 32GB is a great demo.

However, this approach uses uniform 3-bit quantization without WHT rotation or Lloyd-Max centroids. @signalnine's recent CUDA optimization work (19 kernel versions tested) includes an ablation that directly addresses this tradeoff. His TQ4_0 prototype (WHT + uniform quant + dp4a) hit 237 t/s but scored 0.04 PPL worse than plain q4_0. The WHT decorrelation only provides quality benefit when paired with non-linear centroids. Without them, the compression is equivalent to existing q4_0/q8_0 at similar bit rates.

Our fork already ships turbo3 CUDA with WHT + Lloyd-Max (best quality at 3-bit) and the community has validated it across multiple GPUs. The decode speed gap vs dp4a approaches is a known fundamental tradeoff documented here: https://github.com/signalnine/llama-cpp-turboquant/blob/feature/tq4-weight-cuda/docs/tq4-weight-cuda-optimization-log.md

Closing as the quality/speed tradeoff does not favor this approach over our existing implementation. The dp4a kernel patterns may be useful for future work on affine codebooks or hybrid approaches.