Skip to content

ggml-cuda: fix a mountain of VGPR spills on Q2_K - #29910

Closed
thelittlefireman wants to merge 4 commits into
ggml-org:masterfrom
thelittlefireman:todo_q2k_amd_mmq_spill_fix
Closed

ggml-cuda: fix a mountain of VGPR spills on Q2_K#29910
thelittlefireman wants to merge 4 commits into
ggml-org:masterfrom
thelittlefireman:todo_q2k_amd_mmq_spill_fix

Conversation

@thelittlefireman

@thelittlefireman thelittlefireman commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

Q2_K mmq spill massively at least on AMD. Fix it by changing to more gentle unroll and removing a not needed temporary loop.

Additional information

AMD gfx906 / MI50 GCN5 - Q2_K DP4A
Meta-Llama-3-8B-Instruct-Q2_K.gguf, pp2048, -b 2048, -r 10.

-ub Master (t/s) Optimized (t/s) Delta
16 310.24 318.84 +2.77%
32 362.25 426.41 +17.71%
64 169.92 406.99 +139.52%
128 181.72 454.36 +150.03%
256 197.86 531.68 +168.72%
512 204.92 571.23 +178.76%

AMD gfx1152 RDNA 3.5 - Q2_K MMA
Meta-Llama-3-8B-Instruct-Q2_K.gguf, pp2048, -b 2048, -r 10

-ub Master (t/s) Optimized (t/s) Delta
16 69.09 92.57 +33.98%
32 98.26 117.49 +19.57%
64 128.82 128.20 -0.48%
128 135.69 135.13 -0.41%
256 191.79 188.99 -1.46%
512 204.13 205.85 +0.84%

On both i try different unroll 1,2,4 and mesure spill. unroll 1 was the best for RDNA3.5/MMA and on GCN5 unroll 2 was the best on D4PA.

Architecture Target MMQ path Master spills Optimized spills Spill reduction
GCN gfx906 DP4A 164 0 -100%
RDNA1 gfx1010 DP4A 720 67 -90.7%
RDNA2 gfx1030 DP4A 836 0 -100%
RDNA3 gfx1100 MMA 1387 0 -100%
RDNA4 gfx1200 MMA 2041 0 -100%
CDNA gfx90a MMA 126 0 -100%
test-backend-ops 
for OP in MUL_MAT MUL_MAT_ID; do
    echo "===== $OP Q2_K ====="
    HIP_VISIBLE_DEVICES=0 \
    ./build-todo_q2k_amd_mmq_spill_fix/bin/test-backend-ops \
        -b ROCm0 \
        -o "$OP" \
        -p 'type_a=q2_K'
done
===== MUL_MAT Q2_K =====
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 32752 MiB):
  Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Testing 2 devices

Backend 1/2: ROCm0
  Device description: AMD Radeon Graphics
  Device memory: 32752 MB (32732 MB free)

  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=1,k=4096,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=8,k=4096,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
ggml_backend_cuda_graph_compute: CUDA graph warmup reset
ggml_backend_cuda_graph_compute: CUDA graph warmup reset
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  MUL_MAT(type_a=q2_K,type_b=f32,m=1,n=64,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): OK
  13/13 tests passed
  Backend ROCm0: OK
Backend 2/2: CPU
  Skipping
2/2 backends passed
OK
===== MUL_MAT_ID Q2_K =====
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 32752 MiB):
  Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Testing 2 devices

Backend 1/2: ROCm0
  Device description: AMD Radeon Graphics
  Device memory: 32752 MB (32732 MB free)

  MUL_MAT_ID(type_a=q2_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768,amax=1.000000,m_v=0): OK
  MUL_MAT_ID(type_a=q2_K,type_b=f32,n_mats=4,n_used=2,b=0,m=512,n=1,k=256,amax=1.000000,m_v=0): OK
  MUL_MAT_ID(type_a=q2_K,type_b=f32,n_mats=4,n_used=2,b=0,m=512,n=32,k=256,amax=1.000000,m_v=0): OK
  3/3 tests passed
  Backend ROCm0: OK
Backend 2/2: CPU
  Skipping
2/2 backends passed
OK

I run a separate github actions to mesure spill on sm61 and sm70 and it seems to remove all the spill too.

Architecture CUDA arch Baseline spill stores Baseline spill loads Optimized spill stores Optimized spill loads Q2_K kernels spilling
Pascal sm61 28 B 20 B 0 B 0 B 2 → 0
Volta sm70 536 B 524 B 0 B 0 B 2 → 0

Could someone with Nvidia Pascal/Volta could confirms that ? Otherwise i will add a #if GGML_USE_HIP around the both unroll 2.

Requirements

@thelittlefireman
thelittlefireman requested a review from a team as a code owner October 3, 2026 15:13
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Oct 3, 2026
This was referenced Oct 5, 2026
Draft
Draft
@JohannesGaessler

JohannesGaessler commented Oct 5, 2026

Copy link
Copy Markdown
Contributor
Performance

Table generated with scripts/compare-llama-bench.py

GPU Model Microbatch size Test t/s b11350 t/s d6658bc Speedup
Strix Halo llama 8B Q2_K_S 8 pp1024 144.00 142.77 0.99
Strix Halo llama 8B Q2_K_S 16 pp1024 237.71 348.56 1.47
Strix Halo llama 8B Q2_K_S 32 pp1024 367.81 470.54 1.28
Strix Halo llama 8B Q2_K_S 64 pp1024 506.02 508.90 1.01
Strix Halo llama 8B Q2_K_S 128 pp1024 559.55 537.04 0.96
Strix Halo llama 8B Q2_K_S 256 pp1024 619.15 617.81 1.00
Strix Halo llama 8B Q2_K_S 512 pp1024 816.15 813.39 1.00
MI60 / MI50 llama 8B Q2_K_S 8 pp1024 203.33 203.17 1.00
MI60 / MI50 llama 8B Q2_K_S 16 pp1024 344.40 346.09 1.00
MI60 / MI50 llama 8B Q2_K_S 32 pp1024 378.49 429.46 1.13
MI60 / MI50 llama 8B Q2_K_S 64 pp1024 399.11 377.56 0.95
MI60 / MI50 llama 8B Q2_K_S 128 pp1024 491.73 466.76 0.95
MI60 / MI50 llama 8B Q2_K_S 256 pp1024 563.96 535.74 0.95
MI60 / MI50 llama 8B Q2_K_S 512 pp1024 605.43 572.10 0.94
MI100 llama 8B Q2_K_S 8 pp1024 440.46 329.84 0.75
MI100 llama 8B Q2_K_S 16 pp1024 837.35 630.81 0.75
MI100 llama 8B Q2_K_S 32 pp1024 1091.52 820.41 0.75
MI100 llama 8B Q2_K_S 64 pp1024 1355.95 888.82 0.66
MI100 llama 8B Q2_K_S 128 pp1024 1600.57 990.63 0.62
MI100 llama 8B Q2_K_S 256 pp1024 2389.29 2379.99 1.00
MI100 llama 8B Q2_K_S 512 pp1024 3278.83 3251.06 0.99
P40 llama 8B Q2_K_S 8 pp1024 156.58 156.65 1.00
P40 llama 8B Q2_K_S 16 pp1024 317.85 336.55 1.06
P40 llama 8B Q2_K_S 32 pp1024 500.29 386.82 0.77
P40 llama 8B Q2_K_S 64 pp1024 637.16 435.45 0.68
P40 llama 8B Q2_K_S 128 pp1024 742.62 501.90 0.68
P40 llama 8B Q2_K_S 256 pp1024 828.80 562.77 0.68
P40 llama 8B Q2_K_S 512 pp1024 870.03 589.98 0.68
RX 6800 llama 8B Q2_K_S 8 pp1024 131.68 131.71 1.00
RX 6800 llama 8B Q2_K_S 16 pp1024 256.92 214.06 0.83
RX 6800 llama 8B Q2_K_S 32 pp1024 293.81 344.72 1.17
RX 6800 llama 8B Q2_K_S 64 pp1024 312.17 424.58 1.36
RX 6800 llama 8B Q2_K_S 128 pp1024 370.55 536.21 1.45
RX 6800 llama 8B Q2_K_S 256 pp1024 452.96 629.28 1.39
RX 6800 llama 8B Q2_K_S 512 pp1024 497.63 669.43 1.35
RX 9060 XT llama 8B Q2_K_S 8 pp1024 178.48 175.38 0.98
RX 9060 XT llama 8B Q2_K_S 16 pp1024 453.78 494.37 1.09
RX 9060 XT llama 8B Q2_K_S 32 pp1024 519.57 659.51 1.27
RX 9060 XT llama 8B Q2_K_S 64 pp1024 564.91 786.89 1.39
RX 9060 XT llama 8B Q2_K_S 128 pp1024 694.04 904.18 1.30
RX 9060 XT llama 8B Q2_K_S 256 pp1024 750.38 958.83 1.28
RX 9060 XT llama 8B Q2_K_S 512 pp1024 1064.69 964.16 0.91
V100-PCIE-32GB llama 8B Q2_K_S 8 pp1024 460.68 460.56 1.00
V100-PCIE-32GB llama 8B Q2_K_S 16 pp1024 709.09 709.50 1.00
V100-PCIE-32GB llama 8B Q2_K_S 32 pp1024 959.59 784.23 0.82
V100-PCIE-32GB llama 8B Q2_K_S 64 pp1024 724.11 723.95 1.00
V100-PCIE-32GB llama 8B Q2_K_S 128 pp1024 1280.40 1280.16 1.00
V100-PCIE-32GB llama 8B Q2_K_S 256 pp1024 2221.97 2224.60 1.00
V100-PCIE-32GB llama 8B Q2_K_S 512 pp1024 3174.16 3170.82 1.00

The results I get are mixed and quite different and don't match what you're reporting. I am using the q2_K_S model which has all tensors as q2_K rather than the "q2_K" model which contains some q3_K tensors. However, for the q2_K model I'm observing the same pattern:

GPU Model Microbatch size Test t/s b11350 t/s d6658bc Speedup
MI60 / MI50 llama 8B Q2_K_M 8 pp1024 196.56 197.04 1.00
MI60 / MI50 llama 8B Q2_K_M 16 pp1024 308.10 309.77 1.01
MI60 / MI50 llama 8B Q2_K_M 32 pp1024 401.94 434.42 1.08
MI60 / MI50 llama 8B Q2_K_M 64 pp1024 455.41 441.19 0.97
MI60 / MI50 llama 8B Q2_K_M 128 pp1024 507.34 492.95 0.97
MI60 / MI50 llama 8B Q2_K_M 256 pp1024 598.31 580.17 0.97
MI60 / MI50 llama 8B Q2_K_M 512 pp1024 651.54 627.83 0.96

@IMbackK

IMbackK commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

what also dosent make sense here is that the hip quality check should have picked up on these spills.

Edit: it did the spills are real i just missed the kernel in the list.

@IMbackK IMbackK self-assigned this Oct 6, 2026
@thelittlefireman

thelittlefireman commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author
Performance

Table generated with scripts/compare-llama-bench.py

GPU Model Microbatch size Test t/s b11350 t/s d6658bc Speedup
Strix Halo llama 8B Q2_K_S 8 pp1024 144.00 142.77 0.99
Strix Halo llama 8B Q2_K_S 16 pp1024 237.71 348.56 1.47
Strix Halo llama 8B Q2_K_S 32 pp1024 367.81 470.54 1.28
Strix Halo llama 8B Q2_K_S 64 pp1024 506.02 508.90 1.01
Strix Halo llama 8B Q2_K_S 128 pp1024 559.55 537.04 0.96
Strix Halo llama 8B Q2_K_S 256 pp1024 619.15 617.81 1.00
Strix Halo llama 8B Q2_K_S 512 pp1024 816.15 813.39 1.00
MI60 / MI50 llama 8B Q2_K_S 8 pp1024 203.33 203.17 1.00
MI60 / MI50 llama 8B Q2_K_S 16 pp1024 344.40 346.09 1.00
MI60 / MI50 llama 8B Q2_K_S 32 pp1024 378.49 429.46 1.13
MI60 / MI50 llama 8B Q2_K_S 64 pp1024 399.11 377.56 0.95
MI60 / MI50 llama 8B Q2_K_S 128 pp1024 491.73 466.76 0.95
MI60 / MI50 llama 8B Q2_K_S 256 pp1024 563.96 535.74 0.95
MI60 / MI50 llama 8B Q2_K_S 512 pp1024 605.43 572.10 0.94
MI100 llama 8B Q2_K_S 8 pp1024 440.46 329.84 0.75
MI100 llama 8B Q2_K_S 16 pp1024 837.35 630.81 0.75
MI100 llama 8B Q2_K_S 32 pp1024 1091.52 820.41 0.75
MI100 llama 8B Q2_K_S 64 pp1024 1355.95 888.82 0.66
MI100 llama 8B Q2_K_S 128 pp1024 1600.57 990.63 0.62
MI100 llama 8B Q2_K_S 256 pp1024 2389.29 2379.99 1.00
MI100 llama 8B Q2_K_S 512 pp1024 3278.83 3251.06 0.99
P40 llama 8B Q2_K_S 8 pp1024 156.58 156.65 1.00
P40 llama 8B Q2_K_S 16 pp1024 317.85 336.55 1.06
P40 llama 8B Q2_K_S 32 pp1024 500.29 386.82 0.77
P40 llama 8B Q2_K_S 64 pp1024 637.16 435.45 0.68
P40 llama 8B Q2_K_S 128 pp1024 742.62 501.90 0.68
P40 llama 8B Q2_K_S 256 pp1024 828.80 562.77 0.68
P40 llama 8B Q2_K_S 512 pp1024 870.03 589.98 0.68
RX 6800 llama 8B Q2_K_S 8 pp1024 131.68 131.71 1.00
RX 6800 llama 8B Q2_K_S 16 pp1024 256.92 214.06 0.83
RX 6800 llama 8B Q2_K_S 32 pp1024 293.81 344.72 1.17
RX 6800 llama 8B Q2_K_S 64 pp1024 312.17 424.58 1.36
RX 6800 llama 8B Q2_K_S 128 pp1024 370.55 536.21 1.45
RX 6800 llama 8B Q2_K_S 256 pp1024 452.96 629.28 1.39
RX 6800 llama 8B Q2_K_S 512 pp1024 497.63 669.43 1.35
RX 9060 XT llama 8B Q2_K_S 8 pp1024 178.48 175.38 0.98
RX 9060 XT llama 8B Q2_K_S 16 pp1024 453.78 494.37 1.09
RX 9060 XT llama 8B Q2_K_S 32 pp1024 519.57 659.51 1.27
RX 9060 XT llama 8B Q2_K_S 64 pp1024 564.91 786.89 1.39
RX 9060 XT llama 8B Q2_K_S 128 pp1024 694.04 904.18 1.30
RX 9060 XT llama 8B Q2_K_S 256 pp1024 750.38 958.83 1.28
RX 9060 XT llama 8B Q2_K_S 512 pp1024 1064.69 964.16 0.91
V100-PCIE-32GB llama 8B Q2_K_S 8 pp1024 460.68 460.56 1.00
V100-PCIE-32GB llama 8B Q2_K_S 16 pp1024 709.09 709.50 1.00
V100-PCIE-32GB llama 8B Q2_K_S 32 pp1024 959.59 784.23 0.82
V100-PCIE-32GB llama 8B Q2_K_S 64 pp1024 724.11 723.95 1.00
V100-PCIE-32GB llama 8B Q2_K_S 128 pp1024 1280.40 1280.16 1.00
V100-PCIE-32GB llama 8B Q2_K_S 256 pp1024 2221.97 2224.60 1.00
V100-PCIE-32GB llama 8B Q2_K_S 512 pp1024 3174.16 3170.82 1.00

The results I get are mixed and quite different and don't match what you're reporting. I am using the q2_K_S model which has all tensors as q2_K rather than the "q2_K" model which contains some q3_K tensors. However, for the q2_K model I'm observing the same pattern:

GPU Model Microbatch size Test t/s b11350 t/s d6658bc Speedup
MI60 / MI50 llama 8B Q2_K_M 8 pp1024 196.56 197.04 1.00
MI60 / MI50 llama 8B Q2_K_M 16 pp1024 308.10 309.77 1.01
MI60 / MI50 llama 8B Q2_K_M 32 pp1024 401.94 434.42 1.08
MI60 / MI50 llama 8B Q2_K_M 64 pp1024 455.41 441.19 0.97
MI60 / MI50 llama 8B Q2_K_M 128 pp1024 507.34 492.95 0.97
MI60 / MI50 llama 8B Q2_K_M 256 pp1024 598.31 580.17 0.97
MI60 / MI50 llama 8B Q2_K_M 512 pp1024 651.54 627.83 0.96

Thanks for the tests. That's strange. I Will try to reproduce and fix.
Edit:

  • I think i have a clue, I've use the wrong master branch to compare

@thelittlefireman

thelittlefireman commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@JohannesGaessler I can't reproduce. Can you give me your rocm version that you use to compare ? Thanks a lot

GPU Model Microbatch size Test t/s 254b177 (b11350) t/s d6658bc (b11353) Delta Speedup
MI50 gfx906 llama 8B Q2_K_M 16 pp1024 314.89 323.55 +2.75% 1.03x
MI50 gfx906 llama 8B Q2_K_M 32 pp1024 367.38 432.06 +17.61% 1.18x
MI50 gfx906 llama 8B Q2_K_M 64 pp1024 170.46 411.40 +141.35% 2.41x
MI50 gfx906 llama 8B Q2_K_M 128 pp1024 182.40 460.97 +152.72% 2.53x
MI50 gfx906 llama 8B Q2_K_M 256 pp1024 198.51 538.38 +171.21% 2.71x
MI50 gfx906 llama 8B Q2_K_M 512 pp1024 205.43 580.12 +182.39% 2.82x

@IMbackK

IMbackK commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

I see very strange and inconsistent (but consistent run to run) results too. rocm 7.2.4 AMD clang version 23.0.0

GPU Model Microbatch size Test t/s master t/s todo_q2k_amd_mmq_spill_fix Speedup
MI100 llama 8B Q2_K_M 16 pp1024 830.52 660.67 0.80
MI100 llama 8B Q2_K_M 32 pp1024 1153.46 880.36 0.76
MI100 llama 8B Q2_K_M 64 pp1024 1304.88 972.00 0.74
MI100 llama 8B Q2_K_M 128 pp1024 1497.83 1064.97 0.71
MI100 llama 8B Q2_K_M 256 pp1024 2227.06 2229.35 1.00
MI100 llama 8B Q2_K_M 512 pp1024 3004.68 3007.67 1.00
MI100 llama 8B Q2_K_M 512 tg128 98.66 99.57 1.01
GPU Model Microbatch size Test t/s master t/s todo_q2k_amd_mmq_spill_fix Speedup
RX 7900 XTX llama 8B Q2_K_M 16 pp1024 924.83 816.17 0.88
RX 7900 XTX llama 8B Q2_K_M 32 pp1024 274.86 1041.55 3.79
RX 7900 XTX llama 8B Q2_K_M 64 pp1024 1227.55 1263.34 1.03
RX 7900 XTX llama 8B Q2_K_M 128 pp1024 1619.34 1544.44 0.95
RX 7900 XTX llama 8B Q2_K_M 256 pp1024 3011.66 3032.86 1.01
RX 7900 XTX llama 8B Q2_K_M 512 pp1024 3077.62 3066.52 1.00
RX 7900 XTX llama 8B Q2_K_M 512 tg128 146.43 146.25 1.00
GPU Model Microbatch size Test t/s master t/s todo_q2k_amd_mmq_spill_fix Speedup
AI PRO R9700 llama 8B Q2_K_M 16 pp1024 688.02 797.03 1.16
AI PRO R9700 llama 8B Q2_K_M 32 pp1024 550.74 1314.41 2.39
AI PRO R9700 llama 8B Q2_K_M 64 pp1024 341.45 1577.03 4.62
AI PRO R9700 llama 8B Q2_K_M 128 pp1024 403.73 1921.55 4.76
AI PRO R9700 llama 8B Q2_K_M 256 pp1024 425.51 2136.65 5.02
AI PRO R9700 llama 8B Q2_K_M 512 pp1024 330.50 2173.36 6.58
AI PRO R9700 llama 8B Q2_K_M 512 tg128 117.31 118.41 1.01

@thelittlefireman

thelittlefireman commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

I see very strange and inconsistent (but consistent run to run) results too. rocm 7.2.4 AMD clang version 23.0.0

GPU Model Microbatch size Test t/s master t/s todo_q2k_amd_mmq_spill_fix Speedup
MI100 llama 8B Q2_K_M 16 pp1024 830.52 660.67 0.80
MI100 llama 8B Q2_K_M 32 pp1024 1153.46 880.36 0.76
MI100 llama 8B Q2_K_M 64 pp1024 1304.88 972.00 0.74
MI100 llama 8B Q2_K_M 128 pp1024 1497.83 1064.97 0.71
MI100 llama 8B Q2_K_M 256 pp1024 2227.06 2229.35 1.00
MI100 llama 8B Q2_K_M 512 pp1024 3004.68 3007.67 1.00
MI100 llama 8B Q2_K_M 512 tg128 98.66 99.57 1.01
GPU Model Microbatch size Test t/s master t/s todo_q2k_amd_mmq_spill_fix Speedup
RX 7900 XTX llama 8B Q2_K_M 16 pp1024 924.83 816.17 0.88
RX 7900 XTX llama 8B Q2_K_M 32 pp1024 274.86 1041.55 3.79
RX 7900 XTX llama 8B Q2_K_M 64 pp1024 1227.55 1263.34 1.03
RX 7900 XTX llama 8B Q2_K_M 128 pp1024 1619.34 1544.44 0.95
RX 7900 XTX llama 8B Q2_K_M 256 pp1024 3011.66 3032.86 1.01
RX 7900 XTX llama 8B Q2_K_M 512 pp1024 3077.62 3066.52 1.00
RX 7900 XTX llama 8B Q2_K_M 512 tg128 146.43 146.25 1.00
GPU Model Microbatch size Test t/s master t/s todo_q2k_amd_mmq_spill_fix Speedup
AI PRO R9700 llama 8B Q2_K_M 16 pp1024 688.02 797.03 1.16
AI PRO R9700 llama 8B Q2_K_M 32 pp1024 550.74 1314.41 2.39
AI PRO R9700 llama 8B Q2_K_M 64 pp1024 341.45 1577.03 4.62
AI PRO R9700 llama 8B Q2_K_M 128 pp1024 403.73 1921.55 4.76
AI PRO R9700 llama 8B Q2_K_M 256 pp1024 425.51 2136.65 5.02
AI PRO R9700 llama 8B Q2_K_M 512 pp1024 330.50 2173.36 6.58
AI PRO R9700 llama 8B Q2_K_M 512 tg128 117.31 118.41 1.01

I'm on rocm 7.14, could you try with it to see if it's a toolchain issue ?

@JohannesGaessler

JohannesGaessler commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

On my MI50 machine I am using:

┌─[johannesg@johannes-x10sra] - [~] - [10450]
└─[$] hipcc --version                                                                                                                                                [20:31:27]
HIP version: 7.2.53211-9999
AMD clang version 22.0.0git (/srcdest/rocm-llvm f58b06dce1f9c15707c5f808fd002e18c2accf7e)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm/lib/llvm/bin

@thelittlefireman

thelittlefireman commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@JohannesGaessler @IMbackK

I try to compile on your arch with 7.14 and 7.12. to see if there is difference. There is some VGPR/Spill count changes between two but i can't explain the stranges bench/numbers that you get.

I need help to understand, sorry.

  • Baseline: 254b177307b14d055bbe4c050c9c441a1e06d8a4 (b11350)
  • PR: d6658bc9c4318f0d1b640da0a855c8e568fc8371
  • Metrics: VGPR count / VGPR spills reported by the compiler.
  • N/A: Metric could not be reliably extracted.
Compiler Architecture Specialization Baseline (VGPR / Spill) PR (VGPR / Spill)
ROCm 7.2.1 gfx906 J=64, false 128 / 11 70 / 0
ROCm 7.2.1 gfx906 J=64, true 128 / 13 71 / 0
ROCm 7.14 gfx906 J=64, false 128 / 146 68 / 0
ROCm 7.14 gfx906 J=64, true 128 / 148 71 / 0
ROCm 7.2.1 gfx908 J=16, false N/A 53 / 0
ROCm 7.2.1 gfx908 J=16, true 87 / 0 58 / 0
ROCm 7.2.1 gfx908 J=32, false 107 / 0 64 / 0
ROCm 7.2.1 gfx908 J=32, true 113 / 0 77 / 0
ROCm 7.2.1 gfx908 J=48, false 123 / 0 84 / 0
ROCm 7.2.1 gfx908 J=64, false 85 / 0 100 / 0
ROCm 7.2.1 gfx908 J=64, true 94 / 0 110 / 0
ROCm 7.14 gfx908 J=16, false 72 / 0 53 / 0
ROCm 7.14 gfx908 J=16, true 84 / 0 64 / 0
ROCm 7.14 gfx908 J=32, false 92 / 0 64 / 0
ROCm 7.14 gfx908 J=32, true 127 / 0 75 / 0
ROCm 7.14 gfx908 J=48, false 128 / 5 81 / 0
ROCm 7.14 gfx908 J=64, false 81 / 0 95 / 0
ROCm 7.14 gfx908 J=64, true 128 / 85 105 / 0
ROCm 7.2.1 gfx90a J=16, false 142 / 0 56 / 0
ROCm 7.2.1 gfx90a J=16, true 144 / 0 63 / 0
ROCm 7.2.1 gfx90a J=32, false 200 / 0 70 / 0
ROCm 7.2.1 gfx90a J=32, true 192 / 0 78 / 0
ROCm 7.2.1 gfx90a J=48, false 256 / 0 90 / 0
ROCm 7.2.1 gfx90a J=64, false 256 / 33 104 / 0
ROCm 7.2.1 gfx90a J=64, true 256 / 31 112 / 0
ROCm 7.14 gfx90a J=16, false 102 / 0 54 / 0
ROCm 7.14 gfx90a J=16, true 134 / 0 62 / 0
ROCm 7.14 gfx90a J=32, false 206 / 0 68 / 0
ROCm 7.14 gfx90a J=32, true 176 / 0 76 / 0
ROCm 7.14 gfx90a J=48, false 252 / 0 88 / 0
ROCm 7.14 gfx90a J=64, false 256 / 25 94 / 0
ROCm 7.14 gfx90a J=64, true 256 / 31 110 / 0

@JohannesGaessler

JohannesGaessler commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Sorry, but I don't understand it either.

@thelittlefireman
thelittlefireman force-pushed the todo_q2k_amd_mmq_spill_fix branch from 3ab8634 to f2179ef Compare October 12, 2026 21:36
@taronaeo taronaeo removed documentation Improvements or additions to documentation model Model specific build Compilation issues testing Everything test related Vulkan Issues specific to the Vulkan backend devops improvements to build systems and github actions server ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language Apple Metal https://en.wikipedia.org/wiki/Metal_(API) Ascend NPU issues specific to Ascend NPUs OpenCL Issues specific to the OpenCL backend IBM zDNN issues specific to IBM zDNN Accelerator Hexagon mtmd Related to multimodal functionality (video/image/audio) jinja parser Issues related to the jinja parser OpenVINO WebGPU server/ui conversion vendor labels Oct 13, 2026
@pwilkin

pwilkin commented Oct 13, 2026

Copy link
Copy Markdown
Member

I think I can explain.

Depending on the clang version used, the ROCm compiler generates different bytecode :)

See this thread: gufo-org/gufo#255

So, the spill depends what version you compile the code with.

@thelittlefireman

thelittlefireman commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

I agree. There seems to be a regression between rocm 7.2.1 and rocm 7.14 on shcheduling or vgpr selection. On Q2_K vgpr spill are muliply by 10 on master.

@JohannesGaessler

JohannesGaessler commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Quite frankly, I have concerns about maintainability if we try to accommodate multiple different ROCm versions at the same time. I think the only viable paths for is to either

  1. always optimize for the newest ROCm version found on a specific Linux distribution, or
  2. elevate one of the maintainers to the status of ROCm pope and let them decide what the correct version to build against is.

@thelittlefireman

thelittlefireman commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

@JohannesGaessler @pwilkin
Moving rocm upgrade analysis to a discussion: #30622

For this PR I will rollback the 3 pragma unroll X to pragma unroll and redo bench to see if removing the temporay loop is still a good thing for Q2_K or not. if not ==> close.

@thelittlefireman

thelittlefireman commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Conclusion useless ==> Close

Prefill -ub Baseline (t/s) PR (only loop removed) (t/s) Delta
1024 8 188.81 188.68 -0.07 %
1024 16 307.88 299.69 -2.66 %
1024 32 402.05 396.30 -1.43 %
1024 64 453.67 442.04 -2.56 %
1024 128 506.99 494.34 -2.50 %
1024 256 593.82 577.53 -2.74 %
1024 512 647.48 628.61 -2.91 %
2048 8 186.59 186.59 0.00 %
2048 16 304.72 296.83 -2.59 %
2048 32 398.29 391.25 -1.77 %
2048 64 448.20 437.10 -2.48 %
2048 128 499.70 487.23 -2.50 %
2048 256 585.92 569.92 -2.73 %
2048 512 635.42 617.48 -2.82 %

@leeliu103 leeliu103 mentioned this pull request Oct 20, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov Awaiting requested review from ggerganov
@IMbackK IMbackK Awaiting requested review from IMbackK
@JohannesGaessler JohannesGaessler Awaiting requested review from JohannesGaessler

Assignees

@IMbackK IMbackK

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@thelittlefireman @JohannesGaessler @IMbackK @pwilkin @taronaeo