CUDA backend: 3-bit uniform KV cache (turbo3, 4.6x compression, 96% f16 speed)#15

Closed

nalditopr wants to merge 95 commits into

TheTom:feature/turboquant-kv-cachefrom

nalditopr:feature/tq3-reference-approach

Commits on Mar 26, 2026

Commits on Mar 27, 2026

experiment: batched byte extraction + explicit bit field pre-extract

committed
experiment: profiling modes for turbo3 decode bottleneck isolation

TheTom
and
claude
committed
fix: turbo4 SET_ROWS corruption, tail-block truncation, constant coupling (Issue #29)

seanrasch
and
claude
committed
experiment: 4-entry magnitude LUT + branchless sign (XOR trick)
TheTom
committed
experiment: force non-vec FA path for turbo3 (nl=2 vs nl=8)

TheTom
and
claude
committed
fix: stack overflow in turbo4 CPU init — 64KB array on worker thread stack

seanrasch
and
claude
committed
experiment: zero-LUT select chain — 2-level ternary, no constant memory
TheTom
committed
feat: auto-detect hardware, use 4-mag LUT on pre-M5 (+38-45% decode)

committed
experiment: 2-pair half2 LUT — only 2 constant addresses per lookup
TheTom
committed
experiment: deferred norm multiply (batch float4 * norm at end)
TheTom
committed
revert to proven 4-mag + per-element norm (deferred norm was slower)

TheTom
and
claude
committed
experiment: named-register centroid×norm — 4 constant reads upfront, zero divergence, ternary select
TheTom
committed
revert to 4-mag LUT (proven best), document all findings
TheTom
committed
experiment: inline block processing — bypass template dequant in FA inner loop
TheTom
committed
experiment: inline block WORSE on M2 (-10-15%), reverted to 4-mag

TheTom
and
claude
committed
experiment: FULLY BRANCHLESS FMA decode — zero ternary, zero memory, zero branch
TheTom
committed
final: 12 approaches tested, 4-mag LUT is the hardware limit

TheTom
and
claude
committed
experiment: SIMD SHUFFLE magnitude select — cross-lane LUT replacement
TheTom
committed
experiment: simd_shuffle 14.7 at 8K — close to 4-mag (15.1) but not better

TheTom
and
claude
committed
experiment: fused block dot — per-centroid Q accumulation, 4 constant reads total
TheTom
committed
experiment: fused block dot 8.1 at 8K — worst result, 64 comparisons kill it

TheTom
and
claude
committed
experiment: 4-mag helps M5 at 16K (+2.4%) but hurts at 32K (-7.3%)

TheTom
and
claude
committed
experiment: M5 LUT cost grows to 34% at 32K context

TheTom
and
claude
committed
feat: sparse V dequant — +12% decode at 32K, zero quality loss

TheTom
and
claude
committed
feat: sparse V dequant — +22% decode at 32K on M5, auto-enabled

TheTom
and
claude
committed
Merge remote-tracking branch 'upstream/feature/turboquant-kv-cache' into feature/turboquant-kv-cache
seanrasch
committed
Merge pull request #4 from seanrasch/feature/turboquant-kv-cache
TheTom
authored
Revert "Merge pull request #4 from seanrasch/feature/turboquant-kv-cache"
TheTom
committed

Commits on Mar 28, 2026

Add CUDA backend for TurboQuant KV cache compression (turbo3)

localadmin
and
claude
committed
Add turbo4 flash attention VEC support + sparse V dequant

localadmin
and
claude
committed
Add MMA dequant path for turbo3/turbo4 prefill + turbo4 FA VEC

localadmin
and
claude
committed
WIP: Pivot to reference TQ3_0 approach (per-block WHT32, MMVQ)

localadmin
and
claude
committed
Reference TQ3 approach: graph-level dequant for FA, MMVQ vec_dot

localadmin
and
claude
committed
Add CPY turbo3→F32 support, V dequant in non-FA path

localadmin
and
claude
committed
Working reference TQ3 approach: MMVQ decode 96 tok/s, correct output

localadmin
and
claude
committed
turbo3 reaches 98% of f16 decode, 107% prefill speed

localadmin
and
claude
committed
Add WHT linearity optimization for V decode (inspired by RotorQuant PR)

localadmin
and
claude
committed
Optimize dequant kernels + fix WHT32 block size assert

localadmin
and
claude
committed
Auto-enable flash_attn for turbo3 KV cache

localadmin
and
claude
committed
Port turbo4 to PolarQuant design from ollama fork

localadmin
and
claude
committed
Rewrite turbo3 to Lucien's 3-bit uniform quantization (no rotation)

localadmin
and
claude
committed

Commits on Mar 29, 2026

Implement full TurboQuant paper algorithm (WHT128 + Lloyd-Max + norm)

localadmin
and
claude
committed
Revert "Implement full TurboQuant paper algorithm (WHT128 + Lloyd-Max + norm)"
localadmin
committed