CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark - #26705

Merged

am17an merged 10 commits into

ggml-org:masterfrom

praneshgo:pgonegandla/mmvq-q4k-q5k-branchless

Sep 7, 2026

Commits on Aug 31, 2026

Update Q4_K and Q5_K to use branchless computation, which stops the scale unpack being re-executed for every column in mmvq, improving perf at batch sizes > 1
praneshgo
committed
10d708a
Gating the change off from DGX Spark due to no gain
praneshgo
committed
b686848
Adding prefetch gated to Spark, making branchless change in Q4_K and Q5_K general and modifying switch points based on latest perf data
praneshgo
committed
6754cee
Guard the mmvq L2 prefetch against MUSA as well as HIP
praneshgo
committed
9b9cf68
Define the mmvq L2 prefetch only under the Spark guard
praneshgo
committed
63daa18

Commits on Sep 2, 2026

Update switch point for Q4_K to accommodate more models
praneshgo
committed
99e1393

Commits on Sep 4, 2026

Remove stale comments
praneshgo
committed
2097c8c

Commits on Sep 7, 2026