HIP: RDNA 4 MUL_MAT optimizations by itterative · Pull Request #25940 · ggml-org/llama.cpp

itterative · 2026-07-20T21:47:44Z

Overview

RDNA4 fixes for Q6_K and Q2_K and updates for mmq conditions.

Additional information

Checked with test-backend-ops test -o MUL_MAT and test-backend-ops grad -o MUL_MAT -p "type_a=(q2_K|q4_0|q4_1|q4_K|q5_0|q5_1|q5_K|q6_K)". Compiled using ROCM 7.15 (TheRock 20260717)

For both Q6_K and Q2_K, it seems like rocm llvm doesn't handle current code appropriately (Q6_K doesn't do float conversion before int multiplication and Q2_K loop unrolls and spill into scratch).

For the quants sweep, tried to sweep both n (ne11) and n_experts.

test-backend-ops perf -o MUL_MAT

type	n = 1	n = 1 (PR)	delta	n=512	n=512 (PR)	delta
Q4_0	3.59	3.66	+1.9%	78.40	82.54	+5.3%
Q4_1	4.13	4.18	+1.2%	76.80	78.32	+2.0%
Q5_0	2.99	3.02	+1.0%	66.13	63.66	-3.7%
Q5_1	3.26	3.33	+2.1%	59.14	63.43	+7.3%
Q2_K	2.80	2.83	+1.1%	2.51	70.78	+28.2x
Q4_K	2.57	2.74	+6.6%	77.69	78.01	+0.4%
Q5_K	2.37	2.51	+5.9%	75.00	75.82	+1.1%
Q6_K	1.87	2.06	+10.2%	36.62	69.66	+1.90x

llama-bench -m Qwen3.5-9B-Q4_K_M.gguf -p 512,2048,4096 -n 128 -r 3

test	master	PR	delta
pp512	3205.84 ± 264.03	3642.47 ± 365.29	+13.6%
pp2048	3282.09 ± 2.08	3801.26 ± 1.25	+15.8%
pp4096	3221.14 ± 1.70	3716.26 ± 2.82	+15.4%
tg128	80.59 ± 0.16	78.99 ± 0.15	-2.0%

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, used Claude & Kimi K3 to investigate root cause and perform sweep across different quants

fishchev · 2026-07-21T08:42:33Z

llama-bench table seems to have reversed column values? master is faster than PR on pp, but shown as positive delta.

diadras · 2026-07-22T15:21:30Z

Yesterday I pulled this PR, compiled new binaries and ran some tests. They showed a good speedup since yesterday. I have been using llama-server as I usually do with 0 issues.

In short: Thanks for the fix! It had an immediate ~37% PP speedup on my setup!

The longer version:

I did some testing to make sure the PR worked with the models and quants that I use daily and here are the results:

(GLM 5.2 generated but humanly checked for validity)

Hardware

GPU: AMD Radeon AI PRO R9700 (gfx1201, RDNA4), 32 GB VRAM
ROCm 7.2.4
OC/UV: SCLK +200 MHz offset (max 2550 MHz), MCLK 1375 MHz (stock 1258), VDDGFX -75 mV undervolt, power cap 330 W

Builds (identical cmake flags, both built with GCC 13.3.0)

	Pre-PR (baseline)	Post-PR
llama.cpp version	10069	10079
Git SHA	`178a6c449`	`55b972a35`
PR present	No	Yes (#25940)
PR files changed	—	`mmq.cu` (+25 lines), `mmq-vec-dot.cuh` (+2 lines)

cmake flags (both builds):
  -DCMAKE_HIP_ARCHITECTURES=gfx1201
  -DGGML_HIP=ON
  -DGPU_TARGETS=gfx1201
  -DGGML_HIP_ROCWMMA_FATTN=ON
  -DGGML_HIP_GRAPHS=ON
  -DGGML_CUDA_FORCE_MMQ=ON
  -DGGML_CUDA_FA_ALL_QUANTS=ON
  -DGGML_HIP_NO_VMM=ON
  -DGGML_HIP_MMQ_MFMA=ON
  -DGGML_NATIVE=ON
  -DCMAKE_BUILD_TYPE=Release

KV cache quants tested: f16 (default), q8_0/q8_0, q5_1/q5_1, q4_0/q4_0, q8_0/q4_0 (mixed key/value). Results below are f16; KV cache quant had <1% effect on the PR's delta in all configs.

Methodology: Warmup pass (pp4096 -n 1 -r 1, discarded) + 8 repetitions. Same model order, same GPU, single sequential script. Both binaries benchmarked with identical methodology.

Results — prompt processing (t/s)

Model	KV	Test	Pre-PR	Post-PR	Delta
Qwen3.6-27B Q6_K	f16	pp512	683.96	935.22	+36.7%
		pp2048	672.07	920.77	+37.0%
		pp4096	664.91	907.28	+36.5%
Ornith 35B Q6_K	f16	pp512	2153.15	2757.33	+28.1%
		pp2048	2105.03	2710.32	+28.8%
		pp4096	2056.35	2650.53	+28.9%
Qwen3.6-35B-A3B Q4_K_M	f16	pp512	2673.58	2785.85	+4.2%
		pp2048	2610.85	2719.63	+4.2%
		pp4096	2547.33	2650.67	+4.1%
Ornith 35B IQ4_XS	f16	pp512	3539.68	3547.02	-0.1%
		pp2048	3489.55	3490.65	+0.0%
		pp4096	3415.74	3421.04	+0.2%

Results — token generation (tg128, t/s)

Model	Pre-PR	Post-PR	Delta
Qwen3.6-27B Q6_K	22.08	22.07	±0.0%
Ornith 35B Q6_K	78.90	78.84	-0.1%
Qwen3.6-35B-A3B Q4_K_M	83.04	82.63	-0.5%
Ornith 35B IQ4_XS	80.32	80.40	+0.1%

Summary: Q6_K models see +28-37% prompt processing gains across all batch sizes. Q4_K_M sees +4%. IQ4_XS unchanged. Token generation flat (±0.5%). Gains are KV-cache-independent (tested q8_0, q5_1, q4_0, mixed — all within ±1% of f16 delta). No regressions

jhandwe · 2026-07-23T22:35:04Z

I briefly tested this with llama-bench on a IQ3 quant of Qwen3.6 (my previous install was commit fde69a3607ff48b20bbb73e9a0059d5a142a5cf1 from May). It reported a small increases in text genereation speed, and moderate to large speedup in prompt processing (the largest increase was roughly double on pp1024 with ubatch 512). In particular the pp t/s numbers look much more uniform now, where before speed sometimes fell off a cliff.

Great fix, I hope this can get merged soon!In particular

Geramy · 2026-07-24T19:50:06Z

Im just curious why did you decide to specify unroll prior to the for loop? Did this directly effect performance? Does this have implications for other rocm architectures?

itterative · 2026-07-24T20:17:54Z

@Geramy From what I've found, rocm's llvm would try to unroll the loop and end up not being able to reuse the registers in the loop. Disabling unrolling allows it to reuse the registers, so it doesn't spill into the scratch memory. I've tested this on rdna 3 as well and I remember there wasn't a regression, but I can try it again to be sure. I can't test this on cdna as I don't have any.

Geramy · 2026-07-24T20:20:31Z

@Geramy From what I've found, rocm's llvm would try to unroll the loop and end up not being able to reuse the registers in the loop. Disabling unrolling allows it to reuse the registers, so it doesn't spill into the scratch memory. I've tested this on rdna 3 as well and I remember there wasn't a regression, but I can try it again to be sure. I can't test this on cdna as I don't have any.

Oh that’s okay, I mostly was curious what the deeper intention was and that’s pretty cool you have been able to isolate this.

JohannesGaessler · 2026-07-25T07:18:30Z

Please use real models rather than test-backend-ops to determine performance changes. The sweep should be across all relevant combinations of hardware, quantization format, and batch size that are being affected. For example, I recently tested performance like this:

export mn=llama_3-8b
for q in q4_0 q4_1 q5_0 q5_1 q8_0 q2_k q3_k_s q4_k_s q5_k_s q6_k iq1_s iq2_xxs iq2_xs iq2_s iq3_xxs iq3_xs iq3_s iq3_m iq4_nl iq4_xs; do echo $q; ./bench --model models/opt/${mn}-${q}.gguf -r 1 -fa 1 -n 0 -p 2048 -ub "16-2048*2" -o sql|sqlite3 llama-bench.sqlite; sleep 10; done

You can then use scripts/compare-llama-bench.py to generate a table for the resulting speedup. Rather than one large PR it would probably make sense for you to make limited PRs per quantization format where you also optimize the values in mmq-config-rdna4.cuh.

itterative · 2026-07-25T07:40:30Z

@JohannesGaessler Could you point me to the model on huggingface? Or did you mean I should quant this locally? Afaik, some quant providers will mix different types of quants in different layers to get a better perplexity or kld, so comparison will be inconsistent.

In terms of splitting the PR, should I make 3 separate PRs?

Then for the hardware, I don't have any cdna gpus, I only have rdna 3 and 4. I know the Q6 and Q2 changes will affect cdna build, but I can't test these.

JohannesGaessler · 2026-07-25T11:04:19Z

Just use any small model and produce the quants yourself, something like 8b is already enough. I use LLaMA for consistency across time but it doesn't really matter.

I can provide test converage for CDNA. But importantly: I only recently refactored the MMQ code and the values in mmq-config-*.cuh are heavily unoptimized for AMD. Before a PR that makes changes to the base code that affects all quantization types and GPUs I want you to make a PR that tunes those values. Changes to the base code shared across quantization formats and GPUs only make sense if it improves the performance given optimal tuning.

itterative · 2026-07-25T15:32:02Z

@JohannesGaessler rebased on 298219f and ran the benchmarks. I'll only put some of the ones that have a difference bigger than 5% since the table is fairly big. I'm uploading the tables as I got them from scripts/compare-llama-bench.py as well.

I couldn't get some of the IQ quants, since I don't have an imatrix. If you want me to also get those, I'll look into it.

Regarding mmq-config-rdna3.cuh and mmq-config-rdna4.cuh, it makes sense if you want to create those first. I can change this to a draft if you want.

RDNA 3 (7900 XTX)

Model	Microbatch size	Test	t/s master	t/s rdna4-matmult-optimizations	Speedup
llama 8B Q2_K_M	16	pp2048	579.65	694.55	1.20
llama 8B Q2_K_M	32	pp2048	654.56	948.65	1.45
llama 8B Q2_K_M	64	pp2048	412.39	1139.57	2.76
llama 8B Q2_K_M	128	pp2048	553.85	1431.39	2.58
llama 8B Q4_K_M	128	pp2048	2026.88	2140.90	1.06
llama 8B Q5_K_M	128	pp2048	2031.74	2134.47	1.05
llama 8B Q6_K	32	pp2048	1003.93	1120.78	1.12
llama 8B Q6_K	64	pp2048	1283.67	1540.13	1.20
llama 8B Q6_K	128	pp2048	1494.03	1886.43	1.26

RDNA 4 (9070 XT)

Model	Microbatch size	Test	t/s master	t/s rdna4-matmult-optimizations	Speedup
llama 8B Q2_K_M	16	pp2048	751.26	677.50	0.90
llama 8B Q2_K_M	32	pp2048	860.31	1061.93	1.23
llama 8B Q2_K_M	64	pp2048	523.13	1299.47	2.48
llama 8B Q2_K_M	128	pp2048	645.21	1988.34	3.08
llama 8B Q2_K_M	256	pp2048	746.67	3334.40	4.47
llama 8B Q2_K_M	512	pp2048	295.43	4280.25	14.49
llama 8B Q2_K_M	1024	pp2048	324.32	4529.75	13.97
llama 8B Q2_K_M	2048	pp2048	325.48	4391.80	13.49
llama 8B Q4_1	1024	pp2048	4799.89	5044.70	1.05
llama 8B Q4_K_M	128	pp2048	2800.90	2956.68	1.06
llama 8B Q4_K_M	256	pp2048	3993.42	4272.65	1.07
llama 8B Q4_K_M	512	pp2048	4245.26	4682.12	1.10
llama 8B Q4_K_M	1024	pp2048	4281.41	4974.96	1.16
llama 8B Q4_K_M	2048	pp2048	4226.45	4833.58	1.14
llama 8B Q4_K_S	1024	pp2048	4752.23	4985.90	1.05
llama 8B Q5_0	512	pp2048	3630.63	3965.32	1.09
llama 8B Q5_0	1024	pp2048	3643.11	4627.01	1.27
llama 8B Q5_0	2048	pp2048	3580.73	4676.64	1.31
llama 8B Q5_1	512	pp2048	3494.29	3932.81	1.13
llama 8B Q5_1	1024	pp2048	3553.67	4587.21	1.29
llama 8B Q5_1	2048	pp2048	3517.92	4638.30	1.32
llama 8B Q5_K_M	128	pp2048	2778.34	2939.71	1.06
llama 8B Q5_K_M	256	pp2048	3933.70	4212.65	1.07
llama 8B Q5_K_M	512	pp2048	4171.98	4592.60	1.10
llama 8B Q5_K_M	1024	pp2048	4204.37	4860.01	1.16
llama 8B Q5_K_M	2048	pp2048	4153.48	4783.32	1.15
llama 8B Q5_K_S	1024	pp2048	4652.75	4863.85	1.05
llama 8B Q6_K	32	pp2048	1092.62	1188.90	1.09
llama 8B Q6_K	64	pp2048	1401.81	1674.61	1.19
llama 8B Q6_K	128	pp2048	1878.61	2511.53	1.34
llama 8B Q6_K	256	pp2048	2488.08	3441.91	1.38
llama 8B Q6_K	512	pp2048	2520.76	4331.92	1.72
llama 8B Q6_K	1024	pp2048	2631.99	4822.12	1.83
llama 8B Q6_K	2048	pp2048	2606.88	4719.27	1.81

rdna3-comparison.txt
rdna4-comparison.txt

Geramy · 2026-07-25T15:37:16Z

If either of you want I could start doing some optimizations for 3.5 the reason I hopped onto this PR is because I was looking for llama work to do and so I would like to create a MMQ dispatch config for RDNA3 and 3.5 specifically since they don’t share the same architecture as RDNA2 or 4 and they don’t have their own configuration. @JohannesGaessler @itterative

JohannesGaessler · 2026-07-25T19:43:20Z

Sorry, my previous post had a typo. It should have read: "Before a PR that makes changes to the base code that affects all quantization types and GPUs I want you to make a PR that tunes those values." I don't have the capacity right now to tune those values myself.

I'll only put some of the ones that have a difference bigger than 5% since the table is fairly big.

Use HTML tags like <details> and <summary>.

itterative · 2026-07-27T11:12:44Z

I'll give it a try. For now, I'll change the pr to a draft.

Geramy · 2026-07-27T21:43:15Z

@JohannesGaessler I have made a tune PR for 3.5 and 4, also added 3 but I do not have 3 hardware, if @itterative you want to run a test I would be grateful and if you want to suggest changes to the tuning do so.

#26199

itterative · 2026-07-28T04:35:23Z

I've done a script last night to tune the mmq table on both rdna 3 and 4. It finished this morming; will compare results later on today.

Geramy · 2026-07-28T05:31:31Z

I've done a script last night to tune the mmq table on both rdna 3 and 4. It finished this morming; will compare results later on today.

Awesome, I look forward to it it’s I’m GMT -08 so I’ll catch it on my next cycle haha.

itterative · 2026-07-28T13:25:51Z

@Geramy Some have a speedup, some have some regression. Now, since you made a pr, not sure where I should put mine. I'll probably wait until yours gets merged.

RDNA 4 mmq (not including fixes from this pr)

Model	Microbatch size	Test	t/s Geramy/mmq-rdna3-configs	t/s local/rdna-mmq-config	Speedup
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	872.32	879.21	1.01
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	1302.93	1167.75	0.90
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	2040.49	2136.98	1.05
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2772.64	2820.07	1.02
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	4065.20	4070.83	1.00
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	4299.12	4305.26	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	4202.56	4210.62	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	4123.34	4134.50	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	880.21	884.08	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1327.80	1206.22	0.91
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	2074.40	2127.98	1.03
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2795.15	2804.98	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	4099.13	4098.43	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	4344.59	4347.82	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	4248.83	4247.51	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	4169.77	4170.21	1.00
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1101.39	1101.47	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1805.22	1804.10	1.00
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	2519.63	2523.60	1.00
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	3242.99	3258.28	1.00
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	4758.73	4757.27	1.00
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	5073.66	5078.09	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	4986.03	4989.87	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	4938.29	4937.12	1.00
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1155.24	1155.60	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1788.37	1784.66	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	2524.05	2526.58	1.00
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	3219.63	3234.67	1.00
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	4737.46	4739.94	1.00
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	5031.15	5034.10	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	4947.07	4951.27	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	4922.70	4925.08	1.00
llama 8B Q2_K_M	16	pp2048	811.90	668.26	0.82
llama 8B Q2_K_M	32	pp2048	877.13	881.49	1.00
llama 8B Q2_K_M	64	pp2048	466.22	445.91	0.96
llama 8B Q2_K_M	128	pp2048	581.76	559.79	0.96
llama 8B Q2_K_M	256	pp2048	657.16	636.35	0.97
llama 8B Q2_K_M	512	pp2048	295.66	298.06	1.01
llama 8B Q2_K_M	1024	pp2048	324.23	326.82	1.01
llama 8B Q2_K_M	2048	pp2048	325.72	331.43	1.02
llama 8B Q3_K_L	16	pp2048	789.07	932.31	1.18
llama 8B Q3_K_L	32	pp2048	1532.36	1529.30	1.00
llama 8B Q3_K_L	64	pp2048	2117.24	2116.26	1.00
llama 8B Q3_K_L	128	pp2048	2772.98	2764.17	1.00
llama 8B Q3_K_L	256	pp2048	3922.50	3923.53	1.00
llama 8B Q3_K_L	512	pp2048	4156.27	4155.73	1.00
llama 8B Q3_K_L	1024	pp2048	4112.44	4112.83	1.00
llama 8B Q3_K_L	2048	pp2048	4120.14	4122.03	1.00
llama 8B Q3_K_M	16	pp2048	792.43	937.36	1.18
llama 8B Q3_K_M	32	pp2048	1523.56	1526.49	1.00
llama 8B Q3_K_M	64	pp2048	2124.95	2133.27	1.00
llama 8B Q3_K_M	128	pp2048	2776.22	2786.91	1.00
llama 8B Q3_K_M	256	pp2048	3947.67	3948.83	1.00
llama 8B Q3_K_M	512	pp2048	4190.14	4192.57	1.00
llama 8B Q3_K_M	1024	pp2048	4142.60	4142.13	1.00
llama 8B Q3_K_M	2048	pp2048	4160.61	4161.85	1.00
llama 8B Q3_K_S	16	pp2048	672.58	871.80	1.30
llama 8B Q3_K_S	32	pp2048	1475.67	1482.78	1.00
llama 8B Q3_K_S	64	pp2048	2016.64	2029.17	1.01
llama 8B Q3_K_S	128	pp2048	2639.23	2652.54	1.01
llama 8B Q3_K_S	256	pp2048	3670.58	3672.66	1.00
llama 8B Q3_K_S	512	pp2048	3894.42	3895.18	1.00
llama 8B Q3_K_S	1024	pp2048	3860.12	3862.32	1.00
llama 8B Q3_K_S	2048	pp2048	3872.67	3872.04	1.00
llama 8B Q4_0	16	pp2048	1124.72	1123.97	1.00
llama 8B Q4_0	32	pp2048	1781.32	1782.49	1.00
llama 8B Q4_0	64	pp2048	2384.56	2382.23	1.00
llama 8B Q4_0	128	pp2048	3277.27	3286.93	1.00
llama 8B Q4_0	256	pp2048	4776.71	4752.73	0.99
llama 8B Q4_0	512	pp2048	5086.61	5068.15	1.00
llama 8B Q4_0	1024	pp2048	5035.74	5015.49	1.00
llama 8B Q4_0	2048	pp2048	5049.74	5028.38	1.00
llama 8B Q4_1	16	pp2048	1034.66	1035.35	1.00
llama 8B Q4_1	32	pp2048	1712.87	1706.95	1.00
llama 8B Q4_1	64	pp2048	2423.73	2420.64	1.00
llama 8B Q4_1	128	pp2048	3123.88	3122.03	1.00
llama 8B Q4_1	256	pp2048	4557.36	4545.69	1.00
llama 8B Q4_1	512	pp2048	4858.99	4843.21	1.00
llama 8B Q4_1	1024	pp2048	4810.19	4800.99	1.00
llama 8B Q4_1	2048	pp2048	4830.14	4822.83	1.00
llama 8B Q4_K_M	16	pp2048	952.97	953.21	1.00
llama 8B Q4_K_M	32	pp2048	1523.80	1525.45	1.00
llama 8B Q4_K_M	64	pp2048	2177.53	2177.21	1.00
llama 8B Q4_K_M	128	pp2048	2803.26	2807.49	1.00
llama 8B Q4_K_M	256	pp2048	4029.50	4028.69	1.00
llama 8B Q4_K_M	512	pp2048	4266.28	4266.00	1.00
llama 8B Q4_K_M	1024	pp2048	4290.70	4293.33	1.00
llama 8B Q4_K_M	2048	pp2048	4318.61	4318.35	1.00
llama 8B Q4_K_S	16	pp2048	993.83	993.00	1.00
llama 8B Q4_K_S	32	pp2048	1610.20	1607.15	1.00
llama 8B Q4_K_S	64	pp2048	2346.13	2348.53	1.00
llama 8B Q4_K_S	128	pp2048	3058.91	3073.33	1.00
llama 8B Q4_K_S	256	pp2048	4493.92	4497.29	1.00
llama 8B Q4_K_S	512	pp2048	4821.27	4817.58	1.00
llama 8B Q4_K_S	1024	pp2048	4768.54	4771.59	1.00
llama 8B Q4_K_S	2048	pp2048	4801.48	4794.37	1.00
llama 8B Q5_0	16	pp2048	1003.62	1003.51	1.00
llama 8B Q5_0	32	pp2048	1594.79	1592.46	1.00
llama 8B Q5_0	64	pp2048	2253.97	2251.56	1.00
llama 8B Q5_0	128	pp2048	2189.48	2994.15	1.37
llama 8B Q5_0	256	pp2048	3400.98	4342.22	1.28
llama 8B Q5_0	512	pp2048	3641.43	4638.80	1.27
llama 8B Q5_0	1024	pp2048	3654.84	4616.30	1.26
llama 8B Q5_0	2048	pp2048	3644.26	4627.31	1.27
llama 8B Q5_1	16	pp2048	770.52	770.47	1.00
llama 8B Q5_1	32	pp2048	1343.35	1344.59	1.00
llama 8B Q5_1	64	pp2048	2103.30	2112.15	1.00
llama 8B Q5_1	128	pp2048	2058.54	2827.63	1.37
llama 8B Q5_1	256	pp2048	3271.90	4173.67	1.28
llama 8B Q5_1	512	pp2048	3520.81	4535.94	1.29
llama 8B Q5_1	1024	pp2048	3558.88	4534.45	1.27
llama 8B Q5_1	2048	pp2048	3575.76	4548.74	1.27
llama 8B Q5_K_M	16	pp2048	960.26	959.66	1.00
llama 8B Q5_K_M	32	pp2048	1543.78	1543.33	1.00
llama 8B Q5_K_M	64	pp2048	2144.21	2148.06	1.00
llama 8B Q5_K_M	128	pp2048	2773.10	2789.74	1.01
llama 8B Q5_K_M	256	pp2048	3965.98	3964.53	1.00
llama 8B Q5_K_M	512	pp2048	4193.49	4192.48	1.00
llama 8B Q5_K_M	1024	pp2048	4217.92	4215.86	1.00
llama 8B Q5_K_M	2048	pp2048	4238.63	4236.61	1.00
llama 8B Q5_K_S	16	pp2048	995.87	997.15	1.00
llama 8B Q5_K_S	32	pp2048	1621.24	1625.05	1.00
llama 8B Q5_K_S	64	pp2048	2296.68	2298.34	1.00
llama 8B Q5_K_S	128	pp2048	3026.77	3024.47	1.00
llama 8B Q5_K_S	256	pp2048	4415.39	4411.76	1.00
llama 8B Q5_K_S	512	pp2048	4698.41	4699.27	1.00
llama 8B Q5_K_S	1024	pp2048	4666.31	4666.12	1.00
llama 8B Q5_K_S	2048	pp2048	4684.39	4678.11	1.00
llama 8B Q6_K	16	pp2048	825.91	827.94	1.00
llama 8B Q6_K	32	pp2048	1231.32	1233.03	1.00
llama 8B Q6_K	64	pp2048	1576.60	1571.37	1.00
llama 8B Q6_K	128	pp2048	1878.05	1880.21	1.00
llama 8B Q6_K	256	pp2048	2505.49	2505.76	1.00
llama 8B Q6_K	512	pp2048	2535.83	2535.51	1.00
llama 8B Q6_K	1024	pp2048	2646.28	2646.65	1.00
llama 8B Q6_K	2048	pp2048	2585.68	2647.38	1.02

Geramy · 2026-07-28T15:14:06Z

@itterative i could give you or access to my repo if you like and you can merge our tunes, best tune stays. :)

itterative · 2026-07-29T15:55:34Z

Pushed the tunes for RDNA 3 & 4 from my side in #26284

IMbackK · 2026-09-07T07:24:06Z

is this still relevant vs master? if not please close it

itterative · 2026-09-07T07:28:23Z

@IMbackK the fixes in ggml/src/ggml-cuda/mmq-vec-dot.cuh should still be relevant for master imo. I'll make some time this week to rebase and go over the changes and check whether the mmq changes should still be included and will let you know.

itterative · 2026-09-09T18:14:39Z

@IMbackK I've ran the llama 8b on rdna 4 and it does seems like it's still worth adding this in. I haven't tried to sweep ggml_cuda_should_use_mmq again for rdna 4, I can try doing that in the weekend in if you want.

I'll run rdna 3 an report if there's any changes on that.

RDNA 4 master vs PR

Model	Microbatch size	Test	t/s master	t/s rdna4-matmult-optimizations	Speedup
llama 8B IQ1_M - 1.75 bpw	16	pp2048	284.62	284.43	1.00
llama 8B IQ1_M - 1.75 bpw	16	tg128	134.79	134.53	1.00
llama 8B IQ1_M - 1.75 bpw	32	pp2048	530.16	548.17	1.03
llama 8B IQ1_M - 1.75 bpw	32	tg128	134.75	134.63	1.00
llama 8B IQ1_M - 1.75 bpw	64	pp2048	910.20	1040.67	1.14
llama 8B IQ1_M - 1.75 bpw	64	tg128	134.77	134.57	1.00
llama 8B IQ1_M - 1.75 bpw	128	pp2048	1426.44	1779.53	1.25
llama 8B IQ1_M - 1.75 bpw	128	tg128	134.67	134.67	1.00
llama 8B IQ1_M - 1.75 bpw	256	pp2048	2237.02	3044.89	1.36
llama 8B IQ1_M - 1.75 bpw	256	tg128	134.64	134.53	1.00
llama 8B IQ1_M - 1.75 bpw	512	pp2048	2424.05	4113.09	1.70
llama 8B IQ1_M - 1.75 bpw	512	tg128	134.68	134.44	1.00
llama 8B IQ1_M - 1.75 bpw	1024	pp2048	2621.01	4457.01	1.70
llama 8B IQ1_M - 1.75 bpw	1024	tg128	134.68	134.41	1.00
llama 8B IQ1_M - 1.75 bpw	2048	pp2048	2634.35	4488.72	1.70
llama 8B IQ1_M - 1.75 bpw	2048	tg128	134.55	134.51	1.00
llama 8B IQ1_S - 1.5625 bpw	16	pp2048	1035.81	1053.78	1.02
llama 8B IQ1_S - 1.5625 bpw	16	tg128	127.05	127.29	1.00
llama 8B IQ1_S - 1.5625 bpw	32	pp2048	1457.39	1624.06	1.11
llama 8B IQ1_S - 1.5625 bpw	32	tg128	126.93	127.13	1.00
llama 8B IQ1_S - 1.5625 bpw	64	pp2048	1654.36	2131.99	1.29
llama 8B IQ1_S - 1.5625 bpw	64	tg128	126.85	127.10	1.00
llama 8B IQ1_S - 1.5625 bpw	128	pp2048	2108.99	2898.70	1.37
llama 8B IQ1_S - 1.5625 bpw	128	tg128	126.83	127.10	1.00
llama 8B IQ1_S - 1.5625 bpw	256	pp2048	2718.63	3949.08	1.45
llama 8B IQ1_S - 1.5625 bpw	256	tg128	126.75	127.02	1.00
llama 8B IQ1_S - 1.5625 bpw	512	pp2048	2470.90	4220.78	1.71
llama 8B IQ1_S - 1.5625 bpw	512	tg128	126.74	127.00	1.00
llama 8B IQ1_S - 1.5625 bpw	1024	pp2048	2539.31	4185.60	1.65
llama 8B IQ1_S - 1.5625 bpw	1024	tg128	126.75	127.05	1.00
llama 8B IQ1_S - 1.5625 bpw	2048	pp2048	2539.32	4196.44	1.65
llama 8B IQ1_S - 1.5625 bpw	2048	tg128	126.87	127.13	1.00
llama 8B IQ2_M - 2.7 bpw	16	pp2048	668.46	668.62	1.00
llama 8B IQ2_M - 2.7 bpw	16	tg128	86.47	86.36	1.00
llama 8B IQ2_M - 2.7 bpw	32	pp2048	1174.96	1174.48	1.00
llama 8B IQ2_M - 2.7 bpw	32	tg128	86.41	86.29	1.00
llama 8B IQ2_M - 2.7 bpw	64	pp2048	1632.47	1631.33	1.00
llama 8B IQ2_M - 2.7 bpw	64	tg128	86.35	86.23	1.00
llama 8B IQ2_M - 2.7 bpw	128	pp2048	2289.82	2290.93	1.00
llama 8B IQ2_M - 2.7 bpw	128	tg128	86.36	86.22	1.00
llama 8B IQ2_M - 2.7 bpw	256	pp2048	3017.22	3015.65	1.00
llama 8B IQ2_M - 2.7 bpw	256	tg128	86.36	86.19	1.00
llama 8B IQ2_M - 2.7 bpw	512	pp2048	3032.93	3031.38	1.00
llama 8B IQ2_M - 2.7 bpw	512	tg128	86.35	86.22	1.00
llama 8B IQ2_M - 2.7 bpw	1024	pp2048	3167.74	3162.78	1.00
llama 8B IQ2_M - 2.7 bpw	1024	tg128	86.36	86.31	1.00
llama 8B IQ2_M - 2.7 bpw	2048	pp2048	3119.66	3119.21	1.00
llama 8B IQ2_M - 2.7 bpw	2048	tg128	86.37	86.29	1.00
llama 8B IQ2_S - 2.5 bpw	16	pp2048	662.73	661.76	1.00
llama 8B IQ2_S - 2.5 bpw	16	tg128	86.66	86.69	1.00
llama 8B IQ2_S - 2.5 bpw	32	pp2048	1165.63	1165.84	1.00
llama 8B IQ2_S - 2.5 bpw	32	tg128	86.69	86.65	1.00
llama 8B IQ2_S - 2.5 bpw	64	pp2048	1652.15	1653.82	1.00
llama 8B IQ2_S - 2.5 bpw	64	tg128	86.61	86.64	1.00
llama 8B IQ2_S - 2.5 bpw	128	pp2048	2344.73	2344.22	1.00
llama 8B IQ2_S - 2.5 bpw	128	tg128	86.67	86.65	1.00
llama 8B IQ2_S - 2.5 bpw	256	pp2048	3066.08	3064.57	1.00
llama 8B IQ2_S - 2.5 bpw	256	tg128	86.68	86.63	1.00
llama 8B IQ2_S - 2.5 bpw	512	pp2048	3076.55	3073.14	1.00
llama 8B IQ2_S - 2.5 bpw	512	tg128	86.57	86.62	1.00
llama 8B IQ2_S - 2.5 bpw	1024	pp2048	3204.64	3200.43	1.00
llama 8B IQ2_S - 2.5 bpw	1024	tg128	86.66	86.67	1.00
llama 8B IQ2_S - 2.5 bpw	2048	pp2048	3153.19	3151.60	1.00
llama 8B IQ2_S - 2.5 bpw	2048	tg128	86.70	86.63	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	pp2048	644.61	650.48	1.01
llama 8B IQ2_XS - 2.3125 bpw	16	tg128	88.41	88.46	1.00
llama 8B IQ2_XS - 2.3125 bpw	32	pp2048	1086.05	1171.36	1.08
llama 8B IQ2_XS - 2.3125 bpw	32	tg128	88.36	88.43	1.00
llama 8B IQ2_XS - 2.3125 bpw	64	pp2048	1316.47	1589.33	1.21
llama 8B IQ2_XS - 2.3125 bpw	64	tg128	88.31	88.36	1.00
llama 8B IQ2_XS - 2.3125 bpw	128	pp2048	1750.56	2272.25	1.30
llama 8B IQ2_XS - 2.3125 bpw	128	tg128	88.31	88.37	1.00
llama 8B IQ2_XS - 2.3125 bpw	256	pp2048	2240.67	3001.41	1.34
llama 8B IQ2_XS - 2.3125 bpw	256	tg128	88.35	88.34	1.00
llama 8B IQ2_XS - 2.3125 bpw	512	pp2048	2028.29	3049.58	1.50
llama 8B IQ2_XS - 2.3125 bpw	512	tg128	88.35	88.35	1.00
llama 8B IQ2_XS - 2.3125 bpw	1024	pp2048	2147.01	3206.87	1.49
llama 8B IQ2_XS - 2.3125 bpw	1024	tg128	88.31	88.36	1.00
llama 8B IQ2_XS - 2.3125 bpw	2048	pp2048	2146.82	3216.59	1.50
llama 8B IQ2_XS - 2.3125 bpw	2048	tg128	88.33	88.39	1.00
llama 8B IQ2_XXS - 2.0625 bpw	16	pp2048	827.80	841.80	1.02
llama 8B IQ2_XXS - 2.0625 bpw	16	tg128	91.01	91.05	1.00
llama 8B IQ2_XXS - 2.0625 bpw	32	pp2048	1244.77	1359.16	1.09
llama 8B IQ2_XXS - 2.0625 bpw	32	tg128	90.90	90.92	1.00
llama 8B IQ2_XXS - 2.0625 bpw	64	pp2048	1560.02	1971.36	1.26
llama 8B IQ2_XXS - 2.0625 bpw	64	tg128	90.90	90.89	1.00
llama 8B IQ2_XXS - 2.0625 bpw	128	pp2048	1995.60	2698.03	1.35
llama 8B IQ2_XXS - 2.0625 bpw	128	tg128	90.88	90.85	1.00
llama 8B IQ2_XXS - 2.0625 bpw	256	pp2048	2595.55	3692.43	1.42
llama 8B IQ2_XXS - 2.0625 bpw	256	tg128	90.88	90.90	1.00
llama 8B IQ2_XXS - 2.0625 bpw	512	pp2048	2378.18	3954.22	1.66
llama 8B IQ2_XXS - 2.0625 bpw	512	tg128	90.86	90.83	1.00
llama 8B IQ2_XXS - 2.0625 bpw	1024	pp2048	2449.17	3940.97	1.61
llama 8B IQ2_XXS - 2.0625 bpw	1024	tg128	90.86	90.82	1.00
llama 8B IQ2_XXS - 2.0625 bpw	2048	pp2048	2450.49	3956.80	1.61
llama 8B IQ2_XXS - 2.0625 bpw	2048	tg128	90.86	90.83	1.00
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	794.46	794.65	1.00
llama 8B IQ3_S - 3.4375 bpw	16	tg128	81.97	82.04	1.00
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	1123.46	1123.20	1.00
llama 8B IQ3_S - 3.4375 bpw	32	tg128	81.93	81.97	1.00
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	1938.46	1938.44	1.00
llama 8B IQ3_S - 3.4375 bpw	64	tg128	81.90	81.90	1.00
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2689.50	2689.45	1.00
llama 8B IQ3_S - 3.4375 bpw	128	tg128	81.92	81.87	1.00
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	3642.16	3642.56	1.00
llama 8B IQ3_S - 3.4375 bpw	256	tg128	81.90	81.90	1.00
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	3837.79	3836.42	1.00
llama 8B IQ3_S - 3.4375 bpw	512	tg128	81.92	81.90	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	3796.33	3788.30	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	tg128	81.97	81.89	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	3726.87	3727.29	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	tg128	81.93	81.90	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	810.38	810.43	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	tg128	84.91	84.90	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1163.02	1162.93	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	tg128	84.93	84.87	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	1962.44	1962.32	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	tg128	84.89	84.84	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2705.57	2707.76	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	tg128	84.89	84.82	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	3669.84	3669.48	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	tg128	84.86	84.81	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	3877.25	3875.78	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	tg128	84.85	84.83	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	3832.48	3828.39	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	tg128	84.87	84.80	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	3767.06	3767.74	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	tg128	84.86	84.82	1.00
llama 8B IQ3_XS - 3.3 bpw	16	pp2048	799.57	798.91	1.00
llama 8B IQ3_XS - 3.3 bpw	16	tg128	82.37	82.29	1.00
llama 8B IQ3_XS - 3.3 bpw	32	pp2048	1115.83	1116.13	1.00
llama 8B IQ3_XS - 3.3 bpw	32	tg128	82.24	82.25	1.00
llama 8B IQ3_XS - 3.3 bpw	64	pp2048	1951.21	1954.77	1.00
llama 8B IQ3_XS - 3.3 bpw	64	tg128	82.21	82.22	1.00
llama 8B IQ3_XS - 3.3 bpw	128	pp2048	2685.35	2691.60	1.00
llama 8B IQ3_XS - 3.3 bpw	128	tg128	82.21	82.25	1.00
llama 8B IQ3_XS - 3.3 bpw	256	pp2048	3667.97	3666.20	1.00
llama 8B IQ3_XS - 3.3 bpw	256	tg128	82.20	82.21	1.00
llama 8B IQ3_XS - 3.3 bpw	512	pp2048	3876.86	3875.19	1.00
llama 8B IQ3_XS - 3.3 bpw	512	tg128	82.19	82.21	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	pp2048	3837.58	3824.74	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	tg128	82.17	82.23	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	pp2048	3769.23	3766.30	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	tg128	82.22	82.24	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	pp2048	772.67	776.86	1.01
llama 8B IQ3_XXS - 3.0625 bpw	16	tg128	84.13	84.56	1.01
llama 8B IQ3_XXS - 3.0625 bpw	32	pp2048	1140.71	1143.63	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	tg128	84.10	84.43	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	pp2048	1884.50	1884.20	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	tg128	84.08	84.36	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	pp2048	2605.70	2613.35	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	tg128	84.06	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	pp2048	3580.00	3584.86	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	tg128	84.09	84.29	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	pp2048	3802.70	3810.54	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	tg128	84.10	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	pp2048	3762.34	3759.52	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	tg128	84.12	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	pp2048	3694.85	3694.86	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	tg128	84.13	84.32	1.00
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1083.93	1084.63	1.00
llama 8B IQ4_NL - 4.5 bpw	16	tg128	106.07	106.10	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1685.04	1687.15	1.00
llama 8B IQ4_NL - 4.5 bpw	32	tg128	106.04	106.10	1.00
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	2339.63	2345.13	1.00
llama 8B IQ4_NL - 4.5 bpw	64	tg128	105.94	105.98	1.00
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	3208.36	3218.72	1.00
llama 8B IQ4_NL - 4.5 bpw	128	tg128	105.94	105.95	1.00
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	4342.66	4340.10	1.00
llama 8B IQ4_NL - 4.5 bpw	256	tg128	105.92	105.94	1.00
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	4633.79	4630.26	1.00
llama 8B IQ4_NL - 4.5 bpw	512	tg128	105.93	105.95	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	4592.77	4586.06	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	tg128	105.93	105.94	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	4556.42	4561.29	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	tg128	105.94	105.95	1.00
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1108.81	1107.21	1.00
llama 8B IQ4_XS - 4.25 bpw	16	tg128	110.10	109.99	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1706.88	1704.38	1.00
llama 8B IQ4_XS - 4.25 bpw	32	tg128	110.06	109.98	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	2326.47	2316.95	1.00
llama 8B IQ4_XS - 4.25 bpw	64	tg128	109.97	109.89	1.00
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	3239.51	3224.98	1.00
llama 8B IQ4_XS - 4.25 bpw	128	tg128	110.01	109.90	1.00
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	4388.56	4384.44	1.00
llama 8B IQ4_XS - 4.25 bpw	256	tg128	110.00	109.92	1.00
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	4661.13	4657.92	1.00
llama 8B IQ4_XS - 4.25 bpw	512	tg128	109.99	109.86	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	4617.16	4609.32	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	tg128	110.01	109.92	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	4600.23	4598.96	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	tg128	110.00	109.92	1.00
llama 8B Q2_K_M	16	pp2048	640.36	722.81	1.13
llama 8B Q2_K_M	16	tg128	102.54	102.70	1.00
llama 8B Q2_K_M	32	pp2048	506.34	1130.19	2.23
llama 8B Q2_K_M	32	tg128	102.45	102.59	1.00
llama 8B Q2_K_M	64	pp2048	311.93	1348.57	4.32
llama 8B Q2_K_M	64	tg128	102.17	102.56	1.00
llama 8B Q2_K_M	128	pp2048	373.60	1956.71	5.24
llama 8B Q2_K_M	128	tg128	102.04	102.57	1.01
llama 8B Q2_K_M	256	pp2048	408.98	3086.65	7.55
llama 8B Q2_K_M	256	tg128	102.00	102.53	1.01
llama 8B Q2_K_M	512	pp2048	289.68	3747.77	12.94
llama 8B Q2_K_M	512	tg128	101.88	102.59	1.01
llama 8B Q2_K_M	1024	pp2048	316.59	4084.12	12.90
llama 8B Q2_K_M	1024	tg128	101.87	102.61	1.01
llama 8B Q2_K_M	2048	pp2048	317.43	4099.88	12.92
llama 8B Q2_K_M	2048	tg128	101.84	102.66	1.01
llama 8B Q2_K_S	16	pp2048	609.58	756.03	1.24
llama 8B Q2_K_S	16	tg128	112.66	112.68	1.00
llama 8B Q2_K_S	32	pp2048	372.18	1024.35	2.75
llama 8B Q2_K_S	32	tg128	108.39	112.58	1.04
llama 8B Q2_K_S	64	pp2048	223.60	1171.47	5.24
llama 8B Q2_K_S	64	tg128	112.38	112.52	1.00
llama 8B Q2_K_S	128	pp2048	249.89	1733.23	6.94
llama 8B Q2_K_S	128	tg128	112.15	112.37	1.00
llama 8B Q2_K_S	256	pp2048	281.69	2961.79	10.51
llama 8B Q2_K_S	256	tg128	112.01	112.40	1.00
llama 8B Q2_K_S	512	pp2048	199.49	4036.12	20.23
llama 8B Q2_K_S	512	tg128	111.93	112.46	1.00
llama 8B Q2_K_S	1024	pp2048	218.15	4427.63	20.30
llama 8B Q2_K_S	1024	tg128	111.90	112.46	1.01
llama 8B Q2_K_S	2048	pp2048	219.29	4498.87	20.52
llama 8B Q2_K_S	2048	tg128	111.95	112.48	1.00
llama 8B Q3_K_L	16	pp2048	764.11	763.68	1.00
llama 8B Q3_K_L	16	tg128	91.96	91.96	1.00
llama 8B Q3_K_L	32	pp2048	1407.37	1411.49	1.00
llama 8B Q3_K_L	32	tg128	91.92	91.87	1.00
llama 8B Q3_K_L	64	pp2048	1970.92	1969.58	1.00
llama 8B Q3_K_L	64	tg128	91.89	91.83	1.00
llama 8B Q3_K_L	128	pp2048	2674.76	2679.17	1.00
llama 8B Q3_K_L	128	tg128	91.88	91.85	1.00
llama 8B Q3_K_L	256	pp2048	3535.24	3531.43	1.00
llama 8B Q3_K_L	256	tg128	91.88	91.83	1.00
llama 8B Q3_K_L	512	pp2048	3731.98	3724.95	1.00
llama 8B Q3_K_L	512	tg128	91.85	91.84	1.00
llama 8B Q3_K_L	1024	pp2048	3723.80	3721.21	1.00
llama 8B Q3_K_L	1024	tg128	91.86	91.86	1.00
llama 8B Q3_K_L	2048	pp2048	3738.81	3774.80	1.01
llama 8B Q3_K_L	2048	tg128	91.87	91.84	1.00
llama 8B Q3_K_M	16	pp2048	769.75	769.35	1.00
llama 8B Q3_K_M	16	tg128	95.89	95.72	1.00
llama 8B Q3_K_M	32	pp2048	1418.86	1418.19	1.00
llama 8B Q3_K_M	32	tg128	95.84	95.67	1.00
llama 8B Q3_K_M	64	pp2048	1982.02	1982.02	1.00
llama 8B Q3_K_M	64	tg128	95.80	95.63	1.00
llama 8B Q3_K_M	128	pp2048	2682.60	2681.16	1.00
llama 8B Q3_K_M	128	tg128	95.80	95.64	1.00
llama 8B Q3_K_M	256	pp2048	3560.96	3559.31	1.00
llama 8B Q3_K_M	256	tg128	95.78	95.62	1.00
llama 8B Q3_K_M	512	pp2048	3763.88	3762.89	1.00
llama 8B Q3_K_M	512	tg128	95.80	95.63	1.00
llama 8B Q3_K_M	1024	pp2048	3752.93	3751.01	1.00
llama 8B Q3_K_M	1024	tg128	95.81	95.62	1.00
llama 8B Q3_K_M	2048	pp2048	3772.71	3788.61	1.00
llama 8B Q3_K_M	2048	tg128	95.82	95.64	1.00
llama 8B Q3_K_S	16	pp2048	660.28	659.93	1.00
llama 8B Q3_K_S	16	tg128	90.91	90.89	1.00
llama 8B Q3_K_S	32	pp2048	1360.39	1357.48	1.00
llama 8B Q3_K_S	32	tg128	90.86	90.84	1.00
llama 8B Q3_K_S	64	pp2048	1891.89	1890.61	1.00
llama 8B Q3_K_S	64	tg128	90.81	90.84	1.00
llama 8B Q3_K_S	128	pp2048	2574.14	2570.85	1.00
llama 8B Q3_K_S	128	tg128	90.83	90.80	1.00
llama 8B Q3_K_S	256	pp2048	3338.86	3335.69	1.00
llama 8B Q3_K_S	256	tg128	90.84	90.75	1.00
llama 8B Q3_K_S	512	pp2048	3498.08	3491.77	1.00
llama 8B Q3_K_S	512	tg128	90.79	90.78	1.00
llama 8B Q3_K_S	1024	pp2048	3514.54	3509.43	1.00
llama 8B Q3_K_S	1024	tg128	90.80	90.81	1.00
llama 8B Q3_K_S	2048	pp2048	3526.19	3521.60	1.00
llama 8B Q3_K_S	2048	tg128	90.81	90.79	1.00
llama 8B Q4_0	16	pp2048	1049.61	1048.69	1.00
llama 8B Q4_0	16	tg128	108.02	108.06	1.00
llama 8B Q4_0	32	pp2048	1670.65	1666.41	1.00
llama 8B Q4_0	32	tg128	107.98	108.02	1.00
llama 8B Q4_0	64	pp2048	2227.74	2223.54	1.00
llama 8B Q4_0	64	tg128	107.90	107.98	1.00
llama 8B Q4_0	128	pp2048	3087.10	3093.96	1.00
llama 8B Q4_0	128	tg128	107.90	107.97	1.00
llama 8B Q4_0	256	pp2048	4215.32	4208.30	1.00
llama 8B Q4_0	256	tg128	107.90	107.98	1.00
llama 8B Q4_0	512	pp2048	4493.66	4490.73	1.00
llama 8B Q4_0	512	tg128	107.89	107.96	1.00
llama 8B Q4_0	1024	pp2048	4504.99	4529.48	1.01
llama 8B Q4_0	1024	tg128	107.89	107.92	1.00
llama 8B Q4_0	2048	pp2048	4528.52	4604.01	1.02
llama 8B Q4_0	2048	tg128	107.90	107.94	1.00
llama 8B Q4_1	16	pp2048	998.97	998.95	1.00
llama 8B Q4_1	16	tg128	100.61	100.68	1.00
llama 8B Q4_1	32	pp2048	1615.43	1616.52	1.00
llama 8B Q4_1	32	tg128	100.55	100.69	1.00
llama 8B Q4_1	64	pp2048	2245.72	2246.65	1.00
llama 8B Q4_1	64	tg128	100.51	100.61	1.00
llama 8B Q4_1	128	pp2048	2990.73	3002.87	1.00
llama 8B Q4_1	128	tg128	100.51	100.60	1.00
llama 8B Q4_1	256	pp2048	4032.33	4027.95	1.00
llama 8B Q4_1	256	tg128	100.51	100.61	1.00
llama 8B Q4_1	512	pp2048	4281.92	4280.09	1.00
llama 8B Q4_1	512	tg128	100.51	100.61	1.00
llama 8B Q4_1	1024	pp2048	4280.04	4463.86	1.04
llama 8B Q4_1	1024	tg128	100.50	100.56	1.00
llama 8B Q4_1	2048	pp2048	4294.31	4537.49	1.06
llama 8B Q4_1	2048	tg128	100.51	100.57	1.00
llama 8B Q4_K_M	16	pp2048	919.56	922.50	1.00
llama 8B Q4_K_M	16	tg128	100.49	100.47	1.00
llama 8B Q4_K_M	32	pp2048	1457.08	1480.18	1.02
llama 8B Q4_K_M	32	tg128	100.48	100.45	1.00
llama 8B Q4_K_M	64	pp2048	2027.48	2087.43	1.03
llama 8B Q4_K_M	64	tg128	100.43	100.37	1.00
llama 8B Q4_K_M	128	pp2048	2698.64	2827.19	1.05
llama 8B Q4_K_M	128	tg128	100.41	100.36	1.00
llama 8B Q4_K_M	256	pp2048	3606.75	3809.27	1.06
llama 8B Q4_K_M	256	tg128	100.40	100.37	1.00
llama 8B Q4_K_M	512	pp2048	3822.33	4190.86	1.10
llama 8B Q4_K_M	512	tg128	100.39	100.36	1.00
llama 8B Q4_K_M	1024	pp2048	3869.13	4429.03	1.14
llama 8B Q4_K_M	1024	tg128	100.40	100.25	1.00
llama 8B Q4_K_M	2048	pp2048	3890.95	4379.71	1.13
llama 8B Q4_K_M	2048	tg128	100.42	100.37	1.00
llama 8B Q4_K_S	16	pp2048	960.72	960.60	1.00
llama 8B Q4_K_S	16	tg128	104.95	104.97	1.00
llama 8B Q4_K_S	32	pp2048	1541.70	1542.75	1.00
llama 8B Q4_K_S	32	tg128	104.89	104.91	1.00
llama 8B Q4_K_S	64	pp2048	2174.63	2171.09	1.00
llama 8B Q4_K_S	64	tg128	104.80	104.81	1.00
llama 8B Q4_K_S	128	pp2048	2940.94	2939.38	1.00
llama 8B Q4_K_S	128	tg128	104.77	104.80	1.00
llama 8B Q4_K_S	256	pp2048	4000.04	4004.29	1.00
llama 8B Q4_K_S	256	tg128	104.75	104.78	1.00
llama 8B Q4_K_S	512	pp2048	4280.77	4278.43	1.00
llama 8B Q4_K_S	512	tg128	104.74	104.79	1.00
llama 8B Q4_K_S	1024	pp2048	4273.55	4435.61	1.04
llama 8B Q4_K_S	1024	tg128	104.75	104.74	1.00
llama 8B Q4_K_S	2048	pp2048	4299.93	4512.33	1.05
llama 8B Q4_K_S	2048	tg128	104.80	104.75	1.00
llama 8B Q5_0	16	pp2048	897.54	896.77	1.00
llama 8B Q5_0	16	tg128	93.29	93.21	1.00
llama 8B Q5_0	32	pp2048	1472.38	1470.94	1.00
llama 8B Q5_0	32	tg128	93.20	93.16	1.00
llama 8B Q5_0	64	pp2048	2107.78	2106.78	1.00
llama 8B Q5_0	64	tg128	93.20	93.13	1.00
llama 8B Q5_0	128	pp2048	2385.42	2386.11	1.00
llama 8B Q5_0	128	tg128	93.21	93.13	1.00
llama 8B Q5_0	256	pp2048	3384.76	3384.37	1.00
llama 8B Q5_0	256	tg128	93.20	93.15	1.00
llama 8B Q5_0	512	pp2048	3617.66	3524.09	0.97
llama 8B Q5_0	512	tg128	93.21	93.09	1.00
llama 8B Q5_0	1024	pp2048	3647.90	4110.97	1.13
llama 8B Q5_0	1024	tg128	93.20	93.12	1.00
llama 8B Q5_0	2048	pp2048	3662.42	4329.05	1.18
llama 8B Q5_0	2048	tg128	93.20	93.13	1.00
llama 8B Q5_1	16	pp2048	751.26	751.26	1.00
llama 8B Q5_1	16	tg128	88.09	87.97	1.00
llama 8B Q5_1	32	pp2048	1285.99	1287.33	1.00
llama 8B Q5_1	32	tg128	88.07	87.97	1.00
llama 8B Q5_1	64	pp2048	1972.95	1978.00	1.00
llama 8B Q5_1	64	tg128	88.01	87.93	1.00
llama 8B Q5_1	128	pp2048	2010.06	2012.98	1.00
llama 8B Q5_1	128	tg128	88.02	87.92	1.00
llama 8B Q5_1	256	pp2048	2893.93	2894.61	1.00
llama 8B Q5_1	256	tg128	88.01	87.92	1.00
llama 8B Q5_1	512	pp2048	2971.14	3527.63	1.19
llama 8B Q5_1	512	tg128	88.01	87.90	1.00
llama 8B Q5_1	1024	pp2048	3186.43	4097.73	1.29
llama 8B Q5_1	1024	tg128	88.02	87.90	1.00
llama 8B Q5_1	2048	pp2048	3203.86	4309.63	1.35
llama 8B Q5_1	2048	tg128	88.03	87.89	1.00
llama 8B Q5_K_M	16	pp2048	918.09	920.77	1.00
llama 8B Q5_K_M	16	tg128	90.31	90.32	1.00
llama 8B Q5_K_M	32	pp2048	1442.54	1466.54	1.02
llama 8B Q5_K_M	32	tg128	90.27	90.29	1.00
llama 8B Q5_K_M	64	pp2048	1999.69	2053.05	1.03
llama 8B Q5_K_M	64	tg128	90.24	90.23	1.00
llama 8B Q5_K_M	128	pp2048	2677.81	2803.83	1.05
llama 8B Q5_K_M	128	tg128	90.22	90.23	1.00
llama 8B Q5_K_M	256	pp2048	3542.01	3735.00	1.05
llama 8B Q5_K_M	256	tg128	90.22	90.23	1.00
llama 8B Q5_K_M	512	pp2048	3738.75	4095.40	1.10
llama 8B Q5_K_M	512	tg128	90.23	90.22	1.00
llama 8B Q5_K_M	1024	pp2048	3784.24	4320.70	1.14
llama 8B Q5_K_M	1024	tg128	90.23	90.22	1.00
llama 8B Q5_K_M	2048	pp2048	3805.01	4457.82	1.17
llama 8B Q5_K_M	2048	tg128	90.24	90.22	1.00
llama 8B Q5_K_S	16	pp2048	954.47	954.30	1.00
llama 8B Q5_K_S	16	tg128	92.49	92.52	1.00
llama 8B Q5_K_S	32	pp2048	1522.77	1524.70	1.00
llama 8B Q5_K_S	32	tg128	92.48	92.47	1.00
llama 8B Q5_K_S	64	pp2048	2131.81	2132.01	1.00
llama 8B Q5_K_S	64	tg128	92.43	92.43	1.00
llama 8B Q5_K_S	128	pp2048	2903.53	2903.33	1.00
llama 8B Q5_K_S	128	tg128	92.43	92.42	1.00
llama 8B Q5_K_S	256	pp2048	3912.17	3913.36	1.00
llama 8B Q5_K_S	256	tg128	92.44	92.42	1.00
llama 8B Q5_K_S	512	pp2048	4153.53	4155.00	1.00
llama 8B Q5_K_S	512	tg128	92.44	92.40	1.00
llama 8B Q5_K_S	1024	pp2048	4165.65	4312.18	1.04
llama 8B Q5_K_S	1024	tg128	92.44	92.39	1.00
llama 8B Q5_K_S	2048	pp2048	4184.19	4452.33	1.06
llama 8B Q5_K_S	2048	tg128	92.42	92.40	1.00
llama 8B Q6_K	16	pp2048	770.61	783.28	1.02
llama 8B Q6_K	16	tg128	79.47	79.48	1.00
llama 8B Q6_K	32	pp2048	1121.19	1217.77	1.09
llama 8B Q6_K	32	tg128	79.43	79.48	1.00
llama 8B Q6_K	64	pp2048	1480.09	1717.14	1.16
llama 8B Q6_K	64	tg128	79.39	79.39	1.00
llama 8B Q6_K	128	pp2048	1821.22	2308.90	1.27
llama 8B Q6_K	128	tg128	79.40	79.43	1.00
llama 8B Q6_K	256	pp2048	2280.54	2958.83	1.30
llama 8B Q6_K	256	tg128	79.39	79.43	1.00
llama 8B Q6_K	512	pp2048	2319.67	3891.06	1.68
llama 8B Q6_K	512	tg128	79.41	79.43	1.00
llama 8B Q6_K	1024	pp2048	2420.69	4329.69	1.79
llama 8B Q6_K	1024	tg128	79.42	79.43	1.00
llama 8B Q6_K	2048	pp2048	2424.51	4422.00	1.82
llama 8B Q6_K	2048	tg128	79.42	79.44	1.00
llama 8B Q8_0	16	pp2048	885.24	886.20	1.00
llama 8B Q8_0	16	tg128	66.48	66.48	1.00
llama 8B Q8_0	32	pp2048	1460.09	1461.55	1.00
llama 8B Q8_0	32	tg128	66.44	66.42	1.00
llama 8B Q8_0	64	pp2048	2170.49	2167.93	1.00
llama 8B Q8_0	64	tg128	66.42	66.38	1.00
llama 8B Q8_0	128	pp2048	3062.88	3069.78	1.00
llama 8B Q8_0	128	tg128	66.42	66.33	1.00
llama 8B Q8_0	256	pp2048	4297.78	4300.17	1.00
llama 8B Q8_0	256	tg128	66.43	66.34	1.00
llama 8B Q8_0	512	pp2048	4612.20	4610.29	1.00
llama 8B Q8_0	512	tg128	66.42	66.34	1.00
llama 8B Q8_0	1024	pp2048	4631.58	4627.88	1.00
llama 8B Q8_0	1024	tg128	66.42	66.35	1.00
llama 8B Q8_0	2048	pp2048	4671.95	4667.18	1.00
llama 8B Q8_0	2048	tg128	66.41	66.35	1.00

RDNA 4 master vs 140878b (original mmq conditions)

Model	Microbatch size	Test	t/s master	t/s `140878b`	Speedup
llama 8B IQ1_M - 1.75 bpw	16	pp2048	284.62	286.50	1.01
llama 8B IQ1_M - 1.75 bpw	16	tg128	134.79	135.07	1.00
llama 8B IQ1_M - 1.75 bpw	32	pp2048	530.16	549.48	1.04
llama 8B IQ1_M - 1.75 bpw	32	tg128	134.75	135.03	1.00
llama 8B IQ1_M - 1.75 bpw	64	pp2048	910.20	1040.34	1.14
llama 8B IQ1_M - 1.75 bpw	64	tg128	134.77	134.93	1.00
llama 8B IQ1_M - 1.75 bpw	128	pp2048	1426.44	1763.30	1.24
llama 8B IQ1_M - 1.75 bpw	128	tg128	134.67	135.17	1.00
llama 8B IQ1_M - 1.75 bpw	256	pp2048	2237.02	2939.85	1.31
llama 8B IQ1_M - 1.75 bpw	256	tg128	134.64	135.06	1.00
llama 8B IQ1_M - 1.75 bpw	512	pp2048	2424.05	3886.48	1.60
llama 8B IQ1_M - 1.75 bpw	512	tg128	134.68	134.91	1.00
llama 8B IQ1_M - 1.75 bpw	1024	pp2048	2621.01	4214.16	1.61
llama 8B IQ1_M - 1.75 bpw	1024	tg128	134.68	134.95	1.00
llama 8B IQ1_M - 1.75 bpw	2048	pp2048	2634.35	4241.17	1.61
llama 8B IQ1_M - 1.75 bpw	2048	tg128	134.55	134.80	1.00
llama 8B IQ1_S - 1.5625 bpw	16	pp2048	1035.81	1054.08	1.02
llama 8B IQ1_S - 1.5625 bpw	16	tg128	127.05	127.20	1.00
llama 8B IQ1_S - 1.5625 bpw	32	pp2048	1457.39	1625.09	1.12
llama 8B IQ1_S - 1.5625 bpw	32	tg128	126.93	127.14	1.00
llama 8B IQ1_S - 1.5625 bpw	64	pp2048	1654.36	2129.62	1.29
llama 8B IQ1_S - 1.5625 bpw	64	tg128	126.85	127.09	1.00
llama 8B IQ1_S - 1.5625 bpw	128	pp2048	2108.99	2866.24	1.36
llama 8B IQ1_S - 1.5625 bpw	128	tg128	126.83	127.05	1.00
llama 8B IQ1_S - 1.5625 bpw	256	pp2048	2718.63	3785.40	1.39
llama 8B IQ1_S - 1.5625 bpw	256	tg128	126.75	127.05	1.00
llama 8B IQ1_S - 1.5625 bpw	512	pp2048	2470.90	3981.39	1.61
llama 8B IQ1_S - 1.5625 bpw	512	tg128	126.74	127.07	1.00
llama 8B IQ1_S - 1.5625 bpw	1024	pp2048	2539.31	3985.17	1.57
llama 8B IQ1_S - 1.5625 bpw	1024	tg128	126.75	127.01	1.00
llama 8B IQ1_S - 1.5625 bpw	2048	pp2048	2539.32	3986.76	1.57
llama 8B IQ1_S - 1.5625 bpw	2048	tg128	126.87	127.07	1.00
llama 8B IQ2_M - 2.7 bpw	16	pp2048	668.46	668.61	1.00
llama 8B IQ2_M - 2.7 bpw	16	tg128	86.47	86.46	1.00
llama 8B IQ2_M - 2.7 bpw	32	pp2048	1174.96	1173.77	1.00
llama 8B IQ2_M - 2.7 bpw	32	tg128	86.41	86.32	1.00
llama 8B IQ2_M - 2.7 bpw	64	pp2048	1632.47	1632.58	1.00
llama 8B IQ2_M - 2.7 bpw	64	tg128	86.35	86.27	1.00
llama 8B IQ2_M - 2.7 bpw	128	pp2048	2289.82	2290.60	1.00
llama 8B IQ2_M - 2.7 bpw	128	tg128	86.36	86.26	1.00
llama 8B IQ2_M - 2.7 bpw	256	pp2048	3017.22	3016.19	1.00
llama 8B IQ2_M - 2.7 bpw	256	tg128	86.36	86.21	1.00
llama 8B IQ2_M - 2.7 bpw	512	pp2048	3032.93	3033.20	1.00
llama 8B IQ2_M - 2.7 bpw	512	tg128	86.35	86.30	1.00
llama 8B IQ2_M - 2.7 bpw	1024	pp2048	3167.74	3167.87	1.00
llama 8B IQ2_M - 2.7 bpw	1024	tg128	86.36	86.29	1.00
llama 8B IQ2_M - 2.7 bpw	2048	pp2048	3119.66	3118.69	1.00
llama 8B IQ2_M - 2.7 bpw	2048	tg128	86.37	86.30	1.00
llama 8B IQ2_S - 2.5 bpw	16	pp2048	662.73	661.42	1.00
llama 8B IQ2_S - 2.5 bpw	16	tg128	86.66	86.70	1.00
llama 8B IQ2_S - 2.5 bpw	32	pp2048	1165.63	1167.12	1.00
llama 8B IQ2_S - 2.5 bpw	32	tg128	86.69	86.66	1.00
llama 8B IQ2_S - 2.5 bpw	64	pp2048	1652.15	1652.10	1.00
llama 8B IQ2_S - 2.5 bpw	64	tg128	86.61	86.66	1.00
llama 8B IQ2_S - 2.5 bpw	128	pp2048	2344.73	2345.91	1.00
llama 8B IQ2_S - 2.5 bpw	128	tg128	86.67	86.67	1.00
llama 8B IQ2_S - 2.5 bpw	256	pp2048	3066.08	3064.74	1.00
llama 8B IQ2_S - 2.5 bpw	256	tg128	86.68	86.63	1.00
llama 8B IQ2_S - 2.5 bpw	512	pp2048	3076.55	3075.83	1.00
llama 8B IQ2_S - 2.5 bpw	512	tg128	86.57	86.65	1.00
llama 8B IQ2_S - 2.5 bpw	1024	pp2048	3204.64	3204.85	1.00
llama 8B IQ2_S - 2.5 bpw	1024	tg128	86.66	86.67	1.00
llama 8B IQ2_S - 2.5 bpw	2048	pp2048	3153.19	3152.90	1.00
llama 8B IQ2_S - 2.5 bpw	2048	tg128	86.70	86.65	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	pp2048	644.61	649.73	1.01
llama 8B IQ2_XS - 2.3125 bpw	16	tg128	88.41	88.38	1.00
llama 8B IQ2_XS - 2.3125 bpw	32	pp2048	1086.05	1174.96	1.08
llama 8B IQ2_XS - 2.3125 bpw	32	tg128	88.36	88.36	1.00
llama 8B IQ2_XS - 2.3125 bpw	64	pp2048	1316.47	1593.04	1.21
llama 8B IQ2_XS - 2.3125 bpw	64	tg128	88.31	88.27	1.00
llama 8B IQ2_XS - 2.3125 bpw	128	pp2048	1750.56	2252.59	1.29
llama 8B IQ2_XS - 2.3125 bpw	128	tg128	88.31	88.33	1.00
llama 8B IQ2_XS - 2.3125 bpw	256	pp2048	2240.67	2906.74	1.30
llama 8B IQ2_XS - 2.3125 bpw	256	tg128	88.35	88.29	1.00
llama 8B IQ2_XS - 2.3125 bpw	512	pp2048	2028.29	2925.90	1.44
llama 8B IQ2_XS - 2.3125 bpw	512	tg128	88.35	88.28	1.00
llama 8B IQ2_XS - 2.3125 bpw	1024	pp2048	2147.01	3088.74	1.44
llama 8B IQ2_XS - 2.3125 bpw	1024	tg128	88.31	88.34	1.00
llama 8B IQ2_XS - 2.3125 bpw	2048	pp2048	2146.82	3089.87	1.44
llama 8B IQ2_XS - 2.3125 bpw	2048	tg128	88.33	88.37	1.00
llama 8B IQ2_XXS - 2.0625 bpw	16	pp2048	827.80	839.01	1.01
llama 8B IQ2_XXS - 2.0625 bpw	16	tg128	91.01	90.98	1.00
llama 8B IQ2_XXS - 2.0625 bpw	32	pp2048	1244.77	1364.66	1.10
llama 8B IQ2_XXS - 2.0625 bpw	32	tg128	90.90	90.88	1.00
llama 8B IQ2_XXS - 2.0625 bpw	64	pp2048	1560.02	1971.32	1.26
llama 8B IQ2_XXS - 2.0625 bpw	64	tg128	90.90	90.88	1.00
llama 8B IQ2_XXS - 2.0625 bpw	128	pp2048	1995.60	2666.16	1.34
llama 8B IQ2_XXS - 2.0625 bpw	128	tg128	90.88	90.80	1.00
llama 8B IQ2_XXS - 2.0625 bpw	256	pp2048	2595.55	3547.47	1.37
llama 8B IQ2_XXS - 2.0625 bpw	256	tg128	90.88	90.83	1.00
llama 8B IQ2_XXS - 2.0625 bpw	512	pp2048	2378.18	3749.62	1.58
llama 8B IQ2_XXS - 2.0625 bpw	512	tg128	90.86	90.81	1.00
llama 8B IQ2_XXS - 2.0625 bpw	1024	pp2048	2449.17	3764.07	1.54
llama 8B IQ2_XXS - 2.0625 bpw	1024	tg128	90.86	90.83	1.00
llama 8B IQ2_XXS - 2.0625 bpw	2048	pp2048	2450.49	3767.49	1.54
llama 8B IQ2_XXS - 2.0625 bpw	2048	tg128	90.86	90.84	1.00
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	794.46	794.54	1.00
llama 8B IQ3_S - 3.4375 bpw	16	tg128	81.97	82.04	1.00
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	1123.46	1123.15	1.00
llama 8B IQ3_S - 3.4375 bpw	32	tg128	81.93	81.98	1.00
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	1938.46	1940.05	1.00
llama 8B IQ3_S - 3.4375 bpw	64	tg128	81.90	81.96	1.00
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2689.50	2696.59	1.00
llama 8B IQ3_S - 3.4375 bpw	128	tg128	81.92	81.98	1.00
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	3642.16	3642.83	1.00
llama 8B IQ3_S - 3.4375 bpw	256	tg128	81.90	81.94	1.00
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	3837.79	3838.68	1.00
llama 8B IQ3_S - 3.4375 bpw	512	tg128	81.92	81.93	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	3796.33	3794.86	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	tg128	81.97	81.96	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	3726.87	3726.66	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	tg128	81.93	81.95	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	810.38	810.21	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	tg128	84.91	84.91	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1163.02	1163.19	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	tg128	84.93	84.92	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	1962.44	1961.33	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	tg128	84.89	84.85	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2705.57	2708.16	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	tg128	84.89	84.83	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	3669.84	3668.83	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	tg128	84.86	84.83	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	3877.25	3873.93	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	tg128	84.85	84.87	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	3832.48	3830.57	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	tg128	84.87	84.86	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	3767.06	3766.96	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	tg128	84.86	84.86	1.00
llama 8B IQ3_XS - 3.3 bpw	16	pp2048	799.57	799.58	1.00
llama 8B IQ3_XS - 3.3 bpw	16	tg128	82.37	82.39	1.00
llama 8B IQ3_XS - 3.3 bpw	32	pp2048	1115.83	1116.66	1.00
llama 8B IQ3_XS - 3.3 bpw	32	tg128	82.24	82.27	1.00
llama 8B IQ3_XS - 3.3 bpw	64	pp2048	1951.21	1956.75	1.00
llama 8B IQ3_XS - 3.3 bpw	64	tg128	82.21	82.21	1.00
llama 8B IQ3_XS - 3.3 bpw	128	pp2048	2685.35	2690.81	1.00
llama 8B IQ3_XS - 3.3 bpw	128	tg128	82.21	82.19	1.00
llama 8B IQ3_XS - 3.3 bpw	256	pp2048	3667.97	3668.86	1.00
llama 8B IQ3_XS - 3.3 bpw	256	tg128	82.20	82.22	1.00
llama 8B IQ3_XS - 3.3 bpw	512	pp2048	3876.86	3876.70	1.00
llama 8B IQ3_XS - 3.3 bpw	512	tg128	82.19	82.23	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	pp2048	3837.58	3834.97	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	tg128	82.17	82.19	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	pp2048	3769.23	3770.02	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	tg128	82.22	82.23	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	pp2048	772.67	776.35	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	tg128	84.13	84.64	1.01
llama 8B IQ3_XXS - 3.0625 bpw	32	pp2048	1140.71	1144.07	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	tg128	84.10	84.42	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	pp2048	1884.50	1884.10	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	tg128	84.08	84.34	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	pp2048	2605.70	2609.47	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	tg128	84.06	84.33	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	pp2048	3580.00	3585.98	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	tg128	84.09	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	pp2048	3802.70	3807.87	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	tg128	84.10	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	pp2048	3762.34	3763.88	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	tg128	84.12	84.30	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	pp2048	3694.85	3696.67	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	tg128	84.13	84.29	1.00
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1083.93	1084.76	1.00
llama 8B IQ4_NL - 4.5 bpw	16	tg128	106.07	106.07	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1685.04	1689.27	1.00
llama 8B IQ4_NL - 4.5 bpw	32	tg128	106.04	106.04	1.00
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	2339.63	2345.11	1.00
llama 8B IQ4_NL - 4.5 bpw	64	tg128	105.94	105.95	1.00
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	3208.36	3223.28	1.00
llama 8B IQ4_NL - 4.5 bpw	128	tg128	105.94	105.95	1.00
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	4342.66	4341.95	1.00
llama 8B IQ4_NL - 4.5 bpw	256	tg128	105.92	105.96	1.00
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	4633.79	4634.43	1.00
llama 8B IQ4_NL - 4.5 bpw	512	tg128	105.93	105.94	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	4592.77	4591.94	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	tg128	105.93	105.95	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	4556.42	4554.35	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	tg128	105.94	105.92	1.00
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1108.81	1106.95	1.00
llama 8B IQ4_XS - 4.25 bpw	16	tg128	110.10	110.03	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1706.88	1703.31	1.00
llama 8B IQ4_XS - 4.25 bpw	32	tg128	110.06	110.02	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	2326.47	2323.08	1.00
llama 8B IQ4_XS - 4.25 bpw	64	tg128	109.97	109.93	1.00
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	3239.51	3241.77	1.00
llama 8B IQ4_XS - 4.25 bpw	128	tg128	110.01	109.91	1.00
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	4388.56	4385.06	1.00
llama 8B IQ4_XS - 4.25 bpw	256	tg128	110.00	109.93	1.00
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	4661.13	4656.47	1.00
llama 8B IQ4_XS - 4.25 bpw	512	tg128	109.99	109.94	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	4617.16	4612.31	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	tg128	110.01	109.95	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	4600.23	4598.75	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	tg128	110.00	109.98	1.00
llama 8B Q2_K_M	16	pp2048	640.36	722.98	1.13
llama 8B Q2_K_M	16	tg128	102.54	102.71	1.00
llama 8B Q2_K_M	32	pp2048	506.34	1130.45	2.23
llama 8B Q2_K_M	32	tg128	102.45	102.59	1.00
llama 8B Q2_K_M	64	pp2048	311.93	1348.64	4.32
llama 8B Q2_K_M	64	tg128	102.17	102.60	1.00
llama 8B Q2_K_M	128	pp2048	373.60	1671.51	4.47
llama 8B Q2_K_M	128	tg128	102.04	102.61	1.01
llama 8B Q2_K_M	256	pp2048	408.98	1847.26	4.52
llama 8B Q2_K_M	256	tg128	102.00	102.61	1.01
llama 8B Q2_K_M	512	pp2048	289.68	1912.43	6.60
llama 8B Q2_K_M	512	tg128	101.88	102.64	1.01
llama 8B Q2_K_M	1024	pp2048	316.59	2056.08	6.49
llama 8B Q2_K_M	1024	tg128	101.87	102.67	1.01
llama 8B Q2_K_M	2048	pp2048	317.43	2079.32	6.55
llama 8B Q2_K_M	2048	tg128	101.84	102.72	1.01
llama 8B Q2_K_S	16	pp2048	609.58	757.99	1.24
llama 8B Q2_K_S	16	tg128	112.66	112.87	1.00
llama 8B Q2_K_S	32	pp2048	372.18	1026.27	2.76
llama 8B Q2_K_S	32	tg128	108.39	112.74	1.04
llama 8B Q2_K_S	64	pp2048	223.60	1173.57	5.25
llama 8B Q2_K_S	64	tg128	112.38	112.65	1.00
llama 8B Q2_K_S	128	pp2048	249.89	1397.08	5.59
llama 8B Q2_K_S	128	tg128	112.15	112.67	1.00
llama 8B Q2_K_S	256	pp2048	281.69	1500.62	5.33
llama 8B Q2_K_S	256	tg128	112.01	112.64	1.01
llama 8B Q2_K_S	512	pp2048	199.49	1566.96	7.85
llama 8B Q2_K_S	512	tg128	111.93	112.72	1.01
llama 8B Q2_K_S	1024	pp2048	218.15	1712.72	7.85
llama 8B Q2_K_S	1024	tg128	111.90	112.79	1.01
llama 8B Q2_K_S	2048	pp2048	219.29	1740.48	7.94
llama 8B Q2_K_S	2048	tg128	111.95	112.81	1.01
llama 8B Q3_K_L	16	pp2048	764.11	759.86	0.99
llama 8B Q3_K_L	16	tg128	91.96	91.92	1.00
llama 8B Q3_K_L	32	pp2048	1407.37	1411.69	1.00
llama 8B Q3_K_L	32	tg128	91.92	91.87	1.00
llama 8B Q3_K_L	64	pp2048	1970.92	1971.65	1.00
llama 8B Q3_K_L	64	tg128	91.89	91.85	1.00
llama 8B Q3_K_L	128	pp2048	2674.76	2669.54	1.00
llama 8B Q3_K_L	128	tg128	91.88	91.86	1.00
llama 8B Q3_K_L	256	pp2048	3535.24	3533.36	1.00
llama 8B Q3_K_L	256	tg128	91.88	91.84	1.00
llama 8B Q3_K_L	512	pp2048	3731.98	3726.24	1.00
llama 8B Q3_K_L	512	tg128	91.85	91.87	1.00
llama 8B Q3_K_L	1024	pp2048	3723.80	3723.19	1.00
llama 8B Q3_K_L	1024	tg128	91.86	91.88	1.00
llama 8B Q3_K_L	2048	pp2048	3738.81	3738.06	1.00
llama 8B Q3_K_L	2048	tg128	91.87	91.90	1.00
llama 8B Q3_K_M	16	pp2048	769.75	769.34	1.00
llama 8B Q3_K_M	16	tg128	95.89	95.73	1.00
llama 8B Q3_K_M	32	pp2048	1418.86	1418.08	1.00
llama 8B Q3_K_M	32	tg128	95.84	95.75	1.00
llama 8B Q3_K_M	64	pp2048	1982.02	1980.24	1.00
llama 8B Q3_K_M	64	tg128	95.80	95.70	1.00
llama 8B Q3_K_M	128	pp2048	2682.60	2685.30	1.00
llama 8B Q3_K_M	128	tg128	95.80	95.69	1.00
llama 8B Q3_K_M	256	pp2048	3560.96	3560.74	1.00
llama 8B Q3_K_M	256	tg128	95.78	95.69	1.00
llama 8B Q3_K_M	512	pp2048	3763.88	3765.81	1.00
llama 8B Q3_K_M	512	tg128	95.80	95.70	1.00
llama 8B Q3_K_M	1024	pp2048	3752.93	3754.47	1.00
llama 8B Q3_K_M	1024	tg128	95.81	95.72	1.00
llama 8B Q3_K_M	2048	pp2048	3772.71	3772.45	1.00
llama 8B Q3_K_M	2048	tg128	95.82	95.74	1.00
llama 8B Q3_K_S	16	pp2048	660.28	660.50	1.00
llama 8B Q3_K_S	16	tg128	90.91	91.02	1.00
llama 8B Q3_K_S	32	pp2048	1360.39	1361.26	1.00
llama 8B Q3_K_S	32	tg128	90.86	90.89	1.00
llama 8B Q3_K_S	64	pp2048	1891.89	1890.37	1.00
llama 8B Q3_K_S	64	tg128	90.81	90.85	1.00
llama 8B Q3_K_S	128	pp2048	2574.14	2570.71	1.00
llama 8B Q3_K_S	128	tg128	90.83	90.81	1.00
llama 8B Q3_K_S	256	pp2048	3338.86	3337.36	1.00
llama 8B Q3_K_S	256	tg128	90.84	90.81	1.00
llama 8B Q3_K_S	512	pp2048	3498.08	3494.63	1.00
llama 8B Q3_K_S	512	tg128	90.79	90.82	1.00
llama 8B Q3_K_S	1024	pp2048	3514.54	3513.50	1.00
llama 8B Q3_K_S	1024	tg128	90.80	90.84	1.00
llama 8B Q3_K_S	2048	pp2048	3526.19	3524.73	1.00
llama 8B Q3_K_S	2048	tg128	90.81	90.86	1.00
llama 8B Q4_0	16	pp2048	1049.61	1048.39	1.00
llama 8B Q4_0	16	tg128	108.02	107.94	1.00
llama 8B Q4_0	32	pp2048	1670.65	1668.70	1.00
llama 8B Q4_0	32	tg128	107.98	107.93	1.00
llama 8B Q4_0	64	pp2048	2227.74	2227.34	1.00
llama 8B Q4_0	64	tg128	107.90	107.84	1.00
llama 8B Q4_0	128	pp2048	3087.10	3086.69	1.00
llama 8B Q4_0	128	tg128	107.90	107.86	1.00
llama 8B Q4_0	256	pp2048	4215.32	4205.89	1.00
llama 8B Q4_0	256	tg128	107.90	107.84	1.00
llama 8B Q4_0	512	pp2048	4493.66	4489.14	1.00
llama 8B Q4_0	512	tg128	107.89	107.85	1.00
llama 8B Q4_0	1024	pp2048	4504.99	4495.60	1.00
llama 8B Q4_0	1024	tg128	107.89	107.84	1.00
llama 8B Q4_0	2048	pp2048	4528.52	4519.81	1.00
llama 8B Q4_0	2048	tg128	107.90	107.85	1.00
llama 8B Q4_1	16	pp2048	998.97	998.29	1.00
llama 8B Q4_1	16	tg128	100.61	100.70	1.00
llama 8B Q4_1	32	pp2048	1615.43	1608.11	1.00
llama 8B Q4_1	32	tg128	100.55	100.69	1.00
llama 8B Q4_1	64	pp2048	2245.72	2243.91	1.00
llama 8B Q4_1	64	tg128	100.51	100.61	1.00
llama 8B Q4_1	128	pp2048	2990.73	2994.12	1.00
llama 8B Q4_1	128	tg128	100.51	100.63	1.00
llama 8B Q4_1	256	pp2048	4032.33	4029.57	1.00
llama 8B Q4_1	256	tg128	100.51	100.63	1.00
llama 8B Q4_1	512	pp2048	4281.92	4279.60	1.00
llama 8B Q4_1	512	tg128	100.51	100.62	1.00
llama 8B Q4_1	1024	pp2048	4280.04	4276.83	1.00
llama 8B Q4_1	1024	tg128	100.50	100.64	1.00
llama 8B Q4_1	2048	pp2048	4294.31	4292.66	1.00
llama 8B Q4_1	2048	tg128	100.51	100.64	1.00
llama 8B Q4_K_M	16	pp2048	919.56	922.07	1.00
llama 8B Q4_K_M	16	tg128	100.49	100.48	1.00
llama 8B Q4_K_M	32	pp2048	1457.08	1480.80	1.02
llama 8B Q4_K_M	32	tg128	100.48	100.44	1.00
llama 8B Q4_K_M	64	pp2048	2027.48	2090.70	1.03
llama 8B Q4_K_M	64	tg128	100.43	100.38	1.00
llama 8B Q4_K_M	128	pp2048	2698.64	2828.78	1.05
llama 8B Q4_K_M	128	tg128	100.41	100.40	1.00
llama 8B Q4_K_M	256	pp2048	3606.75	3813.24	1.06
llama 8B Q4_K_M	256	tg128	100.40	100.37	1.00
llama 8B Q4_K_M	512	pp2048	3822.33	4038.63	1.06
llama 8B Q4_K_M	512	tg128	100.39	100.40	1.00
llama 8B Q4_K_M	1024	pp2048	3869.13	4082.60	1.06
llama 8B Q4_K_M	1024	tg128	100.40	100.38	1.00
llama 8B Q4_K_M	2048	pp2048	3890.95	4109.02	1.06
llama 8B Q4_K_M	2048	tg128	100.42	100.41	1.00
llama 8B Q4_K_S	16	pp2048	960.72	960.44	1.00
llama 8B Q4_K_S	16	tg128	104.95	104.89	1.00
llama 8B Q4_K_S	32	pp2048	1541.70	1541.07	1.00
llama 8B Q4_K_S	32	tg128	104.89	104.84	1.00
llama 8B Q4_K_S	64	pp2048	2174.63	2169.51	1.00
llama 8B Q4_K_S	64	tg128	104.80	104.79	1.00
llama 8B Q4_K_S	128	pp2048	2940.94	2936.94	1.00
llama 8B Q4_K_S	128	tg128	104.77	104.72	1.00
llama 8B Q4_K_S	256	pp2048	4000.04	4001.21	1.00
llama 8B Q4_K_S	256	tg128	104.75	104.73	1.00
llama 8B Q4_K_S	512	pp2048	4280.77	4278.38	1.00
llama 8B Q4_K_S	512	tg128	104.74	104.73	1.00
llama 8B Q4_K_S	1024	pp2048	4273.55	4274.01	1.00
llama 8B Q4_K_S	1024	tg128	104.75	104.72	1.00
llama 8B Q4_K_S	2048	pp2048	4299.93	4183.02	0.97
llama 8B Q4_K_S	2048	tg128	104.80	104.79	1.00
llama 8B Q5_0	16	pp2048	897.54	896.43	1.00
llama 8B Q5_0	16	tg128	93.29	93.26	1.00
llama 8B Q5_0	32	pp2048	1472.38	1470.92	1.00
llama 8B Q5_0	32	tg128	93.20	93.24	1.00
llama 8B Q5_0	64	pp2048	2107.78	2105.57	1.00
llama 8B Q5_0	64	tg128	93.20	93.16	1.00
llama 8B Q5_0	128	pp2048	2385.42	2385.19	1.00
llama 8B Q5_0	128	tg128	93.21	93.21	1.00
llama 8B Q5_0	256	pp2048	3384.76	3384.29	1.00
llama 8B Q5_0	256	tg128	93.20	93.19	1.00
llama 8B Q5_0	512	pp2048	3617.66	3614.47	1.00
llama 8B Q5_0	512	tg128	93.21	93.20	1.00
llama 8B Q5_0	1024	pp2048	3647.90	3645.99	1.00
llama 8B Q5_0	1024	tg128	93.20	93.20	1.00
llama 8B Q5_0	2048	pp2048	3662.42	3663.07	1.00
llama 8B Q5_0	2048	tg128	93.20	93.22	1.00
llama 8B Q5_1	16	pp2048	751.26	751.43	1.00
llama 8B Q5_1	16	tg128	88.09	88.11	1.00
llama 8B Q5_1	32	pp2048	1285.99	1286.73	1.00
llama 8B Q5_1	32	tg128	88.07	88.10	1.00
llama 8B Q5_1	64	pp2048	1972.95	1977.89	1.00
llama 8B Q5_1	64	tg128	88.01	88.05	1.00
llama 8B Q5_1	128	pp2048	2010.06	2011.91	1.00
llama 8B Q5_1	128	tg128	88.02	88.04	1.00
llama 8B Q5_1	256	pp2048	2893.93	2892.75	1.00
llama 8B Q5_1	256	tg128	88.01	88.05	1.00
llama 8B Q5_1	512	pp2048	2971.14	2970.97	1.00
llama 8B Q5_1	512	tg128	88.01	88.04	1.00
llama 8B Q5_1	1024	pp2048	3186.43	3186.23	1.00
llama 8B Q5_1	1024	tg128	88.02	88.06	1.00
llama 8B Q5_1	2048	pp2048	3203.86	3204.61	1.00
llama 8B Q5_1	2048	tg128	88.03	88.05	1.00
llama 8B Q5_K_M	16	pp2048	918.09	920.64	1.00
llama 8B Q5_K_M	16	tg128	90.31	90.34	1.00
llama 8B Q5_K_M	32	pp2048	1442.54	1468.80	1.02
llama 8B Q5_K_M	32	tg128	90.27	90.33	1.00
llama 8B Q5_K_M	64	pp2048	1999.69	2053.92	1.03
llama 8B Q5_K_M	64	tg128	90.24	90.26	1.00
llama 8B Q5_K_M	128	pp2048	2677.81	2805.59	1.05
llama 8B Q5_K_M	128	tg128	90.22	90.25	1.00
llama 8B Q5_K_M	256	pp2048	3542.01	3737.93	1.06
llama 8B Q5_K_M	256	tg128	90.22	90.26	1.00
llama 8B Q5_K_M	512	pp2048	3738.75	3939.60	1.05
llama 8B Q5_K_M	512	tg128	90.23	90.25	1.00
llama 8B Q5_K_M	1024	pp2048	3784.24	3994.84	1.06
llama 8B Q5_K_M	1024	tg128	90.23	90.26	1.00
llama 8B Q5_K_M	2048	pp2048	3805.01	4014.84	1.06
llama 8B Q5_K_M	2048	tg128	90.24	90.27	1.00
llama 8B Q5_K_S	16	pp2048	954.47	954.74	1.00
llama 8B Q5_K_S	16	tg128	92.49	92.52	1.00
llama 8B Q5_K_S	32	pp2048	1522.77	1523.20	1.00
llama 8B Q5_K_S	32	tg128	92.48	92.49	1.00
llama 8B Q5_K_S	64	pp2048	2131.81	2131.89	1.00
llama 8B Q5_K_S	64	tg128	92.43	92.41	1.00
llama 8B Q5_K_S	128	pp2048	2903.53	2911.32	1.00
llama 8B Q5_K_S	128	tg128	92.43	92.43	1.00
llama 8B Q5_K_S	256	pp2048	3912.17	3912.68	1.00
llama 8B Q5_K_S	256	tg128	92.44	92.43	1.00
llama 8B Q5_K_S	512	pp2048	4153.53	4156.89	1.00
llama 8B Q5_K_S	512	tg128	92.44	92.42	1.00
llama 8B Q5_K_S	1024	pp2048	4165.65	4165.03	1.00
llama 8B Q5_K_S	1024	tg128	92.44	92.43	1.00
llama 8B Q5_K_S	2048	pp2048	4184.19	4186.34	1.00
llama 8B Q5_K_S	2048	tg128	92.42	92.45	1.00
llama 8B Q6_K	16	pp2048	770.61	783.66	1.02
llama 8B Q6_K	16	tg128	79.47	79.49	1.00
llama 8B Q6_K	32	pp2048	1121.19	1214.14	1.08
llama 8B Q6_K	32	tg128	79.43	79.46	1.00
llama 8B Q6_K	64	pp2048	1480.09	1715.03	1.16
llama 8B Q6_K	64	tg128	79.39	79.44	1.00
llama 8B Q6_K	128	pp2048	1821.22	2308.30	1.27
llama 8B Q6_K	128	tg128	79.40	79.42	1.00
llama 8B Q6_K	256	pp2048	2280.54	2959.04	1.30
llama 8B Q6_K	256	tg128	79.39	79.43	1.00
llama 8B Q6_K	512	pp2048	2319.67	2997.42	1.29
llama 8B Q6_K	512	tg128	79.41	79.42	1.00
llama 8B Q6_K	1024	pp2048	2420.69	3162.09	1.31
llama 8B Q6_K	1024	tg128	79.42	79.43	1.00
llama 8B Q6_K	2048	pp2048	2424.51	3170.71	1.31
llama 8B Q6_K	2048	tg128	79.42	79.44	1.00
llama 8B Q8_0	16	pp2048	885.24	884.22	1.00
llama 8B Q8_0	16	tg128	66.48	66.38	1.00
llama 8B Q8_0	32	pp2048	1460.09	1459.36	1.00
llama 8B Q8_0	32	tg128	66.44	66.38	1.00
llama 8B Q8_0	64	pp2048	2170.49	2169.32	1.00
llama 8B Q8_0	64	tg128	66.42	66.36	1.00
llama 8B Q8_0	128	pp2048	3062.88	3064.51	1.00
llama 8B Q8_0	128	tg128	66.42	66.34	1.00
llama 8B Q8_0	256	pp2048	4297.78	4290.85	1.00
llama 8B Q8_0	256	tg128	66.43	66.34	1.00
llama 8B Q8_0	512	pp2048	4612.20	4601.03	1.00
llama 8B Q8_0	512	tg128	66.42	66.34	1.00
llama 8B Q8_0	1024	pp2048	4631.58	4616.89	1.00
llama 8B Q8_0	1024	tg128	66.42	66.32	1.00
llama 8B Q8_0	2048	pp2048	4671.95	4661.04	1.00
llama 8B Q8_0	2048	tg128	66.41	66.33	1.00

RDNA 4 140878b (original mmq conditions) vs PR

Model	Microbatch size	Test	t/s `140878b`	t/s rdna4-matmult-optimizations	Speedup
llama 8B IQ1_M - 1.75 bpw	16	pp2048	286.50	284.43	0.99
llama 8B IQ1_M - 1.75 bpw	16	tg128	135.07	134.53	1.00
llama 8B IQ1_M - 1.75 bpw	32	pp2048	549.48	548.17	1.00
llama 8B IQ1_M - 1.75 bpw	32	tg128	135.03	134.63	1.00
llama 8B IQ1_M - 1.75 bpw	64	pp2048	1040.34	1040.67	1.00
llama 8B IQ1_M - 1.75 bpw	64	tg128	134.93	134.57	1.00
llama 8B IQ1_M - 1.75 bpw	128	pp2048	1763.30	1779.53	1.01
llama 8B IQ1_M - 1.75 bpw	128	tg128	135.17	134.67	1.00
llama 8B IQ1_M - 1.75 bpw	256	pp2048	2939.85	3044.89	1.04
llama 8B IQ1_M - 1.75 bpw	256	tg128	135.06	134.53	1.00
llama 8B IQ1_M - 1.75 bpw	512	pp2048	3886.48	4113.09	1.06
llama 8B IQ1_M - 1.75 bpw	512	tg128	134.91	134.44	1.00
llama 8B IQ1_M - 1.75 bpw	1024	pp2048	4214.16	4457.01	1.06
llama 8B IQ1_M - 1.75 bpw	1024	tg128	134.95	134.41	1.00
llama 8B IQ1_M - 1.75 bpw	2048	pp2048	4241.17	4488.72	1.06
llama 8B IQ1_M - 1.75 bpw	2048	tg128	134.80	134.51	1.00
llama 8B IQ1_S - 1.5625 bpw	16	pp2048	1054.08	1053.78	1.00
llama 8B IQ1_S - 1.5625 bpw	16	tg128	127.20	127.29	1.00
llama 8B IQ1_S - 1.5625 bpw	32	pp2048	1625.09	1624.06	1.00
llama 8B IQ1_S - 1.5625 bpw	32	tg128	127.14	127.13	1.00
llama 8B IQ1_S - 1.5625 bpw	64	pp2048	2129.62	2131.99	1.00
llama 8B IQ1_S - 1.5625 bpw	64	tg128	127.09	127.10	1.00
llama 8B IQ1_S - 1.5625 bpw	128	pp2048	2866.24	2898.70	1.01
llama 8B IQ1_S - 1.5625 bpw	128	tg128	127.05	127.10	1.00
llama 8B IQ1_S - 1.5625 bpw	256	pp2048	3785.40	3949.08	1.04
llama 8B IQ1_S - 1.5625 bpw	256	tg128	127.05	127.02	1.00
llama 8B IQ1_S - 1.5625 bpw	512	pp2048	3981.39	4220.78	1.06
llama 8B IQ1_S - 1.5625 bpw	512	tg128	127.07	127.00	1.00
llama 8B IQ1_S - 1.5625 bpw	1024	pp2048	3985.17	4185.60	1.05
llama 8B IQ1_S - 1.5625 bpw	1024	tg128	127.01	127.05	1.00
llama 8B IQ1_S - 1.5625 bpw	2048	pp2048	3986.76	4196.44	1.05
llama 8B IQ1_S - 1.5625 bpw	2048	tg128	127.07	127.13	1.00
llama 8B IQ2_M - 2.7 bpw	16	pp2048	668.61	668.62	1.00
llama 8B IQ2_M - 2.7 bpw	16	tg128	86.46	86.36	1.00
llama 8B IQ2_M - 2.7 bpw	32	pp2048	1173.77	1174.48	1.00
llama 8B IQ2_M - 2.7 bpw	32	tg128	86.32	86.29	1.00
llama 8B IQ2_M - 2.7 bpw	64	pp2048	1632.58	1631.33	1.00
llama 8B IQ2_M - 2.7 bpw	64	tg128	86.27	86.23	1.00
llama 8B IQ2_M - 2.7 bpw	128	pp2048	2290.60	2290.93	1.00
llama 8B IQ2_M - 2.7 bpw	128	tg128	86.26	86.22	1.00
llama 8B IQ2_M - 2.7 bpw	256	pp2048	3016.19	3015.65	1.00
llama 8B IQ2_M - 2.7 bpw	256	tg128	86.21	86.19	1.00
llama 8B IQ2_M - 2.7 bpw	512	pp2048	3033.20	3031.38	1.00
llama 8B IQ2_M - 2.7 bpw	512	tg128	86.30	86.22	1.00
llama 8B IQ2_M - 2.7 bpw	1024	pp2048	3167.87	3162.78	1.00
llama 8B IQ2_M - 2.7 bpw	1024	tg128	86.29	86.31	1.00
llama 8B IQ2_M - 2.7 bpw	2048	pp2048	3118.69	3119.21	1.00
llama 8B IQ2_M - 2.7 bpw	2048	tg128	86.30	86.29	1.00
llama 8B IQ2_S - 2.5 bpw	16	pp2048	661.42	661.76	1.00
llama 8B IQ2_S - 2.5 bpw	16	tg128	86.70	86.69	1.00
llama 8B IQ2_S - 2.5 bpw	32	pp2048	1167.12	1165.84	1.00
llama 8B IQ2_S - 2.5 bpw	32	tg128	86.66	86.65	1.00
llama 8B IQ2_S - 2.5 bpw	64	pp2048	1652.10	1653.82	1.00
llama 8B IQ2_S - 2.5 bpw	64	tg128	86.66	86.64	1.00
llama 8B IQ2_S - 2.5 bpw	128	pp2048	2345.91	2344.22	1.00
llama 8B IQ2_S - 2.5 bpw	128	tg128	86.67	86.65	1.00
llama 8B IQ2_S - 2.5 bpw	256	pp2048	3064.74	3064.57	1.00
llama 8B IQ2_S - 2.5 bpw	256	tg128	86.63	86.63	1.00
llama 8B IQ2_S - 2.5 bpw	512	pp2048	3075.83	3073.14	1.00
llama 8B IQ2_S - 2.5 bpw	512	tg128	86.65	86.62	1.00
llama 8B IQ2_S - 2.5 bpw	1024	pp2048	3204.85	3200.43	1.00
llama 8B IQ2_S - 2.5 bpw	1024	tg128	86.67	86.67	1.00
llama 8B IQ2_S - 2.5 bpw	2048	pp2048	3152.90	3151.60	1.00
llama 8B IQ2_S - 2.5 bpw	2048	tg128	86.65	86.63	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	pp2048	649.73	650.48	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	tg128	88.38	88.46	1.00
llama 8B IQ2_XS - 2.3125 bpw	32	pp2048	1174.96	1171.36	1.00
llama 8B IQ2_XS - 2.3125 bpw	32	tg128	88.36	88.43	1.00
llama 8B IQ2_XS - 2.3125 bpw	64	pp2048	1593.04	1589.33	1.00
llama 8B IQ2_XS - 2.3125 bpw	64	tg128	88.27	88.36	1.00
llama 8B IQ2_XS - 2.3125 bpw	128	pp2048	2252.59	2272.25	1.01
llama 8B IQ2_XS - 2.3125 bpw	128	tg128	88.33	88.37	1.00
llama 8B IQ2_XS - 2.3125 bpw	256	pp2048	2906.74	3001.41	1.03
llama 8B IQ2_XS - 2.3125 bpw	256	tg128	88.29	88.34	1.00
llama 8B IQ2_XS - 2.3125 bpw	512	pp2048	2925.90	3049.58	1.04
llama 8B IQ2_XS - 2.3125 bpw	512	tg128	88.28	88.35	1.00
llama 8B IQ2_XS - 2.3125 bpw	1024	pp2048	3088.74	3206.87	1.04
llama 8B IQ2_XS - 2.3125 bpw	1024	tg128	88.34	88.36	1.00
llama 8B IQ2_XS - 2.3125 bpw	2048	pp2048	3089.87	3216.59	1.04
llama 8B IQ2_XS - 2.3125 bpw	2048	tg128	88.37	88.39	1.00
llama 8B IQ2_XXS - 2.0625 bpw	16	pp2048	839.01	841.80	1.00
llama 8B IQ2_XXS - 2.0625 bpw	16	tg128	90.98	91.05	1.00
llama 8B IQ2_XXS - 2.0625 bpw	32	pp2048	1364.66	1359.16	1.00
llama 8B IQ2_XXS - 2.0625 bpw	32	tg128	90.88	90.92	1.00
llama 8B IQ2_XXS - 2.0625 bpw	64	pp2048	1971.32	1971.36	1.00
llama 8B IQ2_XXS - 2.0625 bpw	64	tg128	90.88	90.89	1.00
llama 8B IQ2_XXS - 2.0625 bpw	128	pp2048	2666.16	2698.03	1.01
llama 8B IQ2_XXS - 2.0625 bpw	128	tg128	90.80	90.85	1.00
llama 8B IQ2_XXS - 2.0625 bpw	256	pp2048	3547.47	3692.43	1.04
llama 8B IQ2_XXS - 2.0625 bpw	256	tg128	90.83	90.90	1.00
llama 8B IQ2_XXS - 2.0625 bpw	512	pp2048	3749.62	3954.22	1.05
llama 8B IQ2_XXS - 2.0625 bpw	512	tg128	90.81	90.83	1.00
llama 8B IQ2_XXS - 2.0625 bpw	1024	pp2048	3764.07	3940.97	1.05
llama 8B IQ2_XXS - 2.0625 bpw	1024	tg128	90.83	90.82	1.00
llama 8B IQ2_XXS - 2.0625 bpw	2048	pp2048	3767.49	3956.80	1.05
llama 8B IQ2_XXS - 2.0625 bpw	2048	tg128	90.84	90.83	1.00
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	794.54	794.65	1.00
llama 8B IQ3_S - 3.4375 bpw	16	tg128	82.04	82.04	1.00
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	1123.15	1123.20	1.00
llama 8B IQ3_S - 3.4375 bpw	32	tg128	81.98	81.97	1.00
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	1940.05	1938.44	1.00
llama 8B IQ3_S - 3.4375 bpw	64	tg128	81.96	81.90	1.00
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2696.59	2689.45	1.00
llama 8B IQ3_S - 3.4375 bpw	128	tg128	81.98	81.87	1.00
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	3642.83	3642.56	1.00
llama 8B IQ3_S - 3.4375 bpw	256	tg128	81.94	81.90	1.00
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	3838.68	3836.42	1.00
llama 8B IQ3_S - 3.4375 bpw	512	tg128	81.93	81.90	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	3794.86	3788.30	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	tg128	81.96	81.89	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	3726.66	3727.29	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	tg128	81.95	81.90	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	810.21	810.43	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	tg128	84.91	84.90	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1163.19	1162.93	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	tg128	84.92	84.87	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	1961.33	1962.32	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	tg128	84.85	84.84	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2708.16	2707.76	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	tg128	84.83	84.82	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	3668.83	3669.48	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	tg128	84.83	84.81	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	3873.93	3875.78	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	tg128	84.87	84.83	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	3830.57	3828.39	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	tg128	84.86	84.80	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	3766.96	3767.74	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	tg128	84.86	84.82	1.00
llama 8B IQ3_XS - 3.3 bpw	16	pp2048	799.58	798.91	1.00
llama 8B IQ3_XS - 3.3 bpw	16	tg128	82.39	82.29	1.00
llama 8B IQ3_XS - 3.3 bpw	32	pp2048	1116.66	1116.13	1.00
llama 8B IQ3_XS - 3.3 bpw	32	tg128	82.27	82.25	1.00
llama 8B IQ3_XS - 3.3 bpw	64	pp2048	1956.75	1954.77	1.00
llama 8B IQ3_XS - 3.3 bpw	64	tg128	82.21	82.22	1.00
llama 8B IQ3_XS - 3.3 bpw	128	pp2048	2690.81	2691.60	1.00
llama 8B IQ3_XS - 3.3 bpw	128	tg128	82.19	82.25	1.00
llama 8B IQ3_XS - 3.3 bpw	256	pp2048	3668.86	3666.20	1.00
llama 8B IQ3_XS - 3.3 bpw	256	tg128	82.22	82.21	1.00
llama 8B IQ3_XS - 3.3 bpw	512	pp2048	3876.70	3875.19	1.00
llama 8B IQ3_XS - 3.3 bpw	512	tg128	82.23	82.21	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	pp2048	3834.97	3824.74	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	tg128	82.19	82.23	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	pp2048	3770.02	3766.30	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	tg128	82.23	82.24	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	pp2048	776.35	776.86	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	tg128	84.64	84.56	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	pp2048	1144.07	1143.63	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	tg128	84.42	84.43	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	pp2048	1884.10	1884.20	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	tg128	84.34	84.36	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	pp2048	2609.47	2613.35	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	tg128	84.33	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	pp2048	3585.98	3584.86	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	tg128	84.32	84.29	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	pp2048	3807.87	3810.54	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	tg128	84.32	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	pp2048	3763.88	3759.52	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	tg128	84.30	84.32	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	pp2048	3696.67	3694.86	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	tg128	84.29	84.32	1.00
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1084.76	1084.63	1.00
llama 8B IQ4_NL - 4.5 bpw	16	tg128	106.07	106.10	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1689.27	1687.15	1.00
llama 8B IQ4_NL - 4.5 bpw	32	tg128	106.04	106.10	1.00
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	2345.11	2345.13	1.00
llama 8B IQ4_NL - 4.5 bpw	64	tg128	105.95	105.98	1.00
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	3223.28	3218.72	1.00
llama 8B IQ4_NL - 4.5 bpw	128	tg128	105.95	105.95	1.00
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	4341.95	4340.10	1.00
llama 8B IQ4_NL - 4.5 bpw	256	tg128	105.96	105.94	1.00
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	4634.43	4630.26	1.00
llama 8B IQ4_NL - 4.5 bpw	512	tg128	105.94	105.95	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	4591.94	4586.06	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	tg128	105.95	105.94	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	4554.35	4561.29	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	tg128	105.92	105.95	1.00
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1106.95	1107.21	1.00
llama 8B IQ4_XS - 4.25 bpw	16	tg128	110.03	109.99	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1703.31	1704.38	1.00
llama 8B IQ4_XS - 4.25 bpw	32	tg128	110.02	109.98	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	2323.08	2316.95	1.00
llama 8B IQ4_XS - 4.25 bpw	64	tg128	109.93	109.89	1.00
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	3241.77	3224.98	0.99
llama 8B IQ4_XS - 4.25 bpw	128	tg128	109.91	109.90	1.00
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	4385.06	4384.44	1.00
llama 8B IQ4_XS - 4.25 bpw	256	tg128	109.93	109.92	1.00
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	4656.47	4657.92	1.00
llama 8B IQ4_XS - 4.25 bpw	512	tg128	109.94	109.86	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	4612.31	4609.32	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	tg128	109.95	109.92	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	4598.75	4598.96	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	tg128	109.98	109.92	1.00
llama 8B Q2_K_M	16	pp2048	722.98	722.81	1.00
llama 8B Q2_K_M	16	tg128	102.71	102.70	1.00
llama 8B Q2_K_M	32	pp2048	1130.45	1130.19	1.00
llama 8B Q2_K_M	32	tg128	102.59	102.59	1.00
llama 8B Q2_K_M	64	pp2048	1348.64	1348.57	1.00
llama 8B Q2_K_M	64	tg128	102.60	102.56	1.00
llama 8B Q2_K_M	128	pp2048	1671.51	1956.71	1.17
llama 8B Q2_K_M	128	tg128	102.61	102.57	1.00
llama 8B Q2_K_M	256	pp2048	1847.26	3086.65	1.67
llama 8B Q2_K_M	256	tg128	102.61	102.53	1.00
llama 8B Q2_K_M	512	pp2048	1912.43	3747.77	1.96
llama 8B Q2_K_M	512	tg128	102.64	102.59	1.00
llama 8B Q2_K_M	1024	pp2048	2056.08	4084.12	1.99
llama 8B Q2_K_M	1024	tg128	102.67	102.61	1.00
llama 8B Q2_K_M	2048	pp2048	2079.32	4099.88	1.97
llama 8B Q2_K_M	2048	tg128	102.72	102.66	1.00
llama 8B Q2_K_S	16	pp2048	757.99	756.03	1.00
llama 8B Q2_K_S	16	tg128	112.87	112.68	1.00
llama 8B Q2_K_S	32	pp2048	1026.27	1024.35	1.00
llama 8B Q2_K_S	32	tg128	112.74	112.58	1.00
llama 8B Q2_K_S	64	pp2048	1173.57	1171.47	1.00
llama 8B Q2_K_S	64	tg128	112.65	112.52	1.00
llama 8B Q2_K_S	128	pp2048	1397.08	1733.23	1.24
llama 8B Q2_K_S	128	tg128	112.67	112.37	1.00
llama 8B Q2_K_S	256	pp2048	1500.62	2961.79	1.97
llama 8B Q2_K_S	256	tg128	112.64	112.40	1.00
llama 8B Q2_K_S	512	pp2048	1566.96	4036.12	2.58
llama 8B Q2_K_S	512	tg128	112.72	112.46	1.00
llama 8B Q2_K_S	1024	pp2048	1712.72	4427.63	2.59
llama 8B Q2_K_S	1024	tg128	112.79	112.46	1.00
llama 8B Q2_K_S	2048	pp2048	1740.48	4498.87	2.58
llama 8B Q2_K_S	2048	tg128	112.81	112.48	1.00
llama 8B Q3_K_L	16	pp2048	759.86	763.68	1.01
llama 8B Q3_K_L	16	tg128	91.92	91.96	1.00
llama 8B Q3_K_L	32	pp2048	1411.69	1411.49	1.00
llama 8B Q3_K_L	32	tg128	91.87	91.87	1.00
llama 8B Q3_K_L	64	pp2048	1971.65	1969.58	1.00
llama 8B Q3_K_L	64	tg128	91.85	91.83	1.00
llama 8B Q3_K_L	128	pp2048	2669.54	2679.17	1.00
llama 8B Q3_K_L	128	tg128	91.86	91.85	1.00
llama 8B Q3_K_L	256	pp2048	3533.36	3531.43	1.00
llama 8B Q3_K_L	256	tg128	91.84	91.83	1.00
llama 8B Q3_K_L	512	pp2048	3726.24	3724.95	1.00
llama 8B Q3_K_L	512	tg128	91.87	91.84	1.00
llama 8B Q3_K_L	1024	pp2048	3723.19	3721.21	1.00
llama 8B Q3_K_L	1024	tg128	91.88	91.86	1.00
llama 8B Q3_K_L	2048	pp2048	3738.06	3774.80	1.01
llama 8B Q3_K_L	2048	tg128	91.90	91.84	1.00
llama 8B Q3_K_M	16	pp2048	769.34	769.35	1.00
llama 8B Q3_K_M	16	tg128	95.73	95.72	1.00
llama 8B Q3_K_M	32	pp2048	1418.08	1418.19	1.00
llama 8B Q3_K_M	32	tg128	95.75	95.67	1.00
llama 8B Q3_K_M	64	pp2048	1980.24	1982.02	1.00
llama 8B Q3_K_M	64	tg128	95.70	95.63	1.00
llama 8B Q3_K_M	128	pp2048	2685.30	2681.16	1.00
llama 8B Q3_K_M	128	tg128	95.69	95.64	1.00
llama 8B Q3_K_M	256	pp2048	3560.74	3559.31	1.00
llama 8B Q3_K_M	256	tg128	95.69	95.62	1.00
llama 8B Q3_K_M	512	pp2048	3765.81	3762.89	1.00
llama 8B Q3_K_M	512	tg128	95.70	95.63	1.00
llama 8B Q3_K_M	1024	pp2048	3754.47	3751.01	1.00
llama 8B Q3_K_M	1024	tg128	95.72	95.62	1.00
llama 8B Q3_K_M	2048	pp2048	3772.45	3788.61	1.00
llama 8B Q3_K_M	2048	tg128	95.74	95.64	1.00
llama 8B Q3_K_S	16	pp2048	660.50	659.93	1.00
llama 8B Q3_K_S	16	tg128	91.02	90.89	1.00
llama 8B Q3_K_S	32	pp2048	1361.26	1357.48	1.00
llama 8B Q3_K_S	32	tg128	90.89	90.84	1.00
llama 8B Q3_K_S	64	pp2048	1890.37	1890.61	1.00
llama 8B Q3_K_S	64	tg128	90.85	90.84	1.00
llama 8B Q3_K_S	128	pp2048	2570.71	2570.85	1.00
llama 8B Q3_K_S	128	tg128	90.81	90.80	1.00
llama 8B Q3_K_S	256	pp2048	3337.36	3335.69	1.00
llama 8B Q3_K_S	256	tg128	90.81	90.75	1.00
llama 8B Q3_K_S	512	pp2048	3494.63	3491.77	1.00
llama 8B Q3_K_S	512	tg128	90.82	90.78	1.00
llama 8B Q3_K_S	1024	pp2048	3513.50	3509.43	1.00
llama 8B Q3_K_S	1024	tg128	90.84	90.81	1.00
llama 8B Q3_K_S	2048	pp2048	3524.73	3521.60	1.00
llama 8B Q3_K_S	2048	tg128	90.86	90.79	1.00
llama 8B Q4_0	16	pp2048	1048.39	1048.69	1.00
llama 8B Q4_0	16	tg128	107.94	108.06	1.00
llama 8B Q4_0	32	pp2048	1668.70	1666.41	1.00
llama 8B Q4_0	32	tg128	107.93	108.02	1.00
llama 8B Q4_0	64	pp2048	2227.34	2223.54	1.00
llama 8B Q4_0	64	tg128	107.84	107.98	1.00
llama 8B Q4_0	128	pp2048	3086.69	3093.96	1.00
llama 8B Q4_0	128	tg128	107.86	107.97	1.00
llama 8B Q4_0	256	pp2048	4205.89	4208.30	1.00
llama 8B Q4_0	256	tg128	107.84	107.98	1.00
llama 8B Q4_0	512	pp2048	4489.14	4490.73	1.00
llama 8B Q4_0	512	tg128	107.85	107.96	1.00
llama 8B Q4_0	1024	pp2048	4495.60	4529.48	1.01
llama 8B Q4_0	1024	tg128	107.84	107.92	1.00
llama 8B Q4_0	2048	pp2048	4519.81	4604.01	1.02
llama 8B Q4_0	2048	tg128	107.85	107.94	1.00
llama 8B Q4_1	16	pp2048	998.29	998.95	1.00
llama 8B Q4_1	16	tg128	100.70	100.68	1.00
llama 8B Q4_1	32	pp2048	1608.11	1616.52	1.01
llama 8B Q4_1	32	tg128	100.69	100.69	1.00
llama 8B Q4_1	64	pp2048	2243.91	2246.65	1.00
llama 8B Q4_1	64	tg128	100.61	100.61	1.00
llama 8B Q4_1	128	pp2048	2994.12	3002.87	1.00
llama 8B Q4_1	128	tg128	100.63	100.60	1.00
llama 8B Q4_1	256	pp2048	4029.57	4027.95	1.00
llama 8B Q4_1	256	tg128	100.63	100.61	1.00
llama 8B Q4_1	512	pp2048	4279.60	4280.09	1.00
llama 8B Q4_1	512	tg128	100.62	100.61	1.00
llama 8B Q4_1	1024	pp2048	4276.83	4463.86	1.04
llama 8B Q4_1	1024	tg128	100.64	100.56	1.00
llama 8B Q4_1	2048	pp2048	4292.66	4537.49	1.06
llama 8B Q4_1	2048	tg128	100.64	100.57	1.00
llama 8B Q4_K_M	16	pp2048	922.07	922.50	1.00
llama 8B Q4_K_M	16	tg128	100.48	100.47	1.00
llama 8B Q4_K_M	32	pp2048	1480.80	1480.18	1.00
llama 8B Q4_K_M	32	tg128	100.44	100.45	1.00
llama 8B Q4_K_M	64	pp2048	2090.70	2087.43	1.00
llama 8B Q4_K_M	64	tg128	100.38	100.37	1.00
llama 8B Q4_K_M	128	pp2048	2828.78	2827.19	1.00
llama 8B Q4_K_M	128	tg128	100.40	100.36	1.00
llama 8B Q4_K_M	256	pp2048	3813.24	3809.27	1.00
llama 8B Q4_K_M	256	tg128	100.37	100.37	1.00
llama 8B Q4_K_M	512	pp2048	4038.63	4190.86	1.04
llama 8B Q4_K_M	512	tg128	100.40	100.36	1.00
llama 8B Q4_K_M	1024	pp2048	4082.60	4429.03	1.08
llama 8B Q4_K_M	1024	tg128	100.38	100.25	1.00
llama 8B Q4_K_M	2048	pp2048	4109.02	4379.71	1.07
llama 8B Q4_K_M	2048	tg128	100.41	100.37	1.00
llama 8B Q4_K_S	16	pp2048	960.44	960.60	1.00
llama 8B Q4_K_S	16	tg128	104.89	104.97	1.00
llama 8B Q4_K_S	32	pp2048	1541.07	1542.75	1.00
llama 8B Q4_K_S	32	tg128	104.84	104.91	1.00
llama 8B Q4_K_S	64	pp2048	2169.51	2171.09	1.00
llama 8B Q4_K_S	64	tg128	104.79	104.81	1.00
llama 8B Q4_K_S	128	pp2048	2936.94	2939.38	1.00
llama 8B Q4_K_S	128	tg128	104.72	104.80	1.00
llama 8B Q4_K_S	256	pp2048	4001.21	4004.29	1.00
llama 8B Q4_K_S	256	tg128	104.73	104.78	1.00
llama 8B Q4_K_S	512	pp2048	4278.38	4278.43	1.00
llama 8B Q4_K_S	512	tg128	104.73	104.79	1.00
llama 8B Q4_K_S	1024	pp2048	4274.01	4435.61	1.04
llama 8B Q4_K_S	1024	tg128	104.72	104.74	1.00
llama 8B Q4_K_S	2048	pp2048	4183.02	4512.33	1.08
llama 8B Q4_K_S	2048	tg128	104.79	104.75	1.00
llama 8B Q5_0	16	pp2048	896.43	896.77	1.00
llama 8B Q5_0	16	tg128	93.26	93.21	1.00
llama 8B Q5_0	32	pp2048	1470.92	1470.94	1.00
llama 8B Q5_0	32	tg128	93.24	93.16	1.00
llama 8B Q5_0	64	pp2048	2105.57	2106.78	1.00
llama 8B Q5_0	64	tg128	93.16	93.13	1.00
llama 8B Q5_0	128	pp2048	2385.19	2386.11	1.00
llama 8B Q5_0	128	tg128	93.21	93.13	1.00
llama 8B Q5_0	256	pp2048	3384.29	3384.37	1.00
llama 8B Q5_0	256	tg128	93.19	93.15	1.00
llama 8B Q5_0	512	pp2048	3614.47	3524.09	0.97
llama 8B Q5_0	512	tg128	93.20	93.09	1.00
llama 8B Q5_0	1024	pp2048	3645.99	4110.97	1.13
llama 8B Q5_0	1024	tg128	93.20	93.12	1.00
llama 8B Q5_0	2048	pp2048	3663.07	4329.05	1.18
llama 8B Q5_0	2048	tg128	93.22	93.13	1.00
llama 8B Q5_1	16	pp2048	751.43	751.26	1.00
llama 8B Q5_1	16	tg128	88.11	87.97	1.00
llama 8B Q5_1	32	pp2048	1286.73	1287.33	1.00
llama 8B Q5_1	32	tg128	88.10	87.97	1.00
llama 8B Q5_1	64	pp2048	1977.89	1978.00	1.00
llama 8B Q5_1	64	tg128	88.05	87.93	1.00
llama 8B Q5_1	128	pp2048	2011.91	2012.98	1.00
llama 8B Q5_1	128	tg128	88.04	87.92	1.00
llama 8B Q5_1	256	pp2048	2892.75	2894.61	1.00
llama 8B Q5_1	256	tg128	88.05	87.92	1.00
llama 8B Q5_1	512	pp2048	2970.97	3527.63	1.19
llama 8B Q5_1	512	tg128	88.04	87.90	1.00
llama 8B Q5_1	1024	pp2048	3186.23	4097.73	1.29
llama 8B Q5_1	1024	tg128	88.06	87.90	1.00
llama 8B Q5_1	2048	pp2048	3204.61	4309.63	1.34
llama 8B Q5_1	2048	tg128	88.05	87.89	1.00
llama 8B Q5_K_M	16	pp2048	920.64	920.77	1.00
llama 8B Q5_K_M	16	tg128	90.34	90.32	1.00
llama 8B Q5_K_M	32	pp2048	1468.80	1466.54	1.00
llama 8B Q5_K_M	32	tg128	90.33	90.29	1.00
llama 8B Q5_K_M	64	pp2048	2053.92	2053.05	1.00
llama 8B Q5_K_M	64	tg128	90.26	90.23	1.00
llama 8B Q5_K_M	128	pp2048	2805.59	2803.83	1.00
llama 8B Q5_K_M	128	tg128	90.25	90.23	1.00
llama 8B Q5_K_M	256	pp2048	3737.93	3735.00	1.00
llama 8B Q5_K_M	256	tg128	90.26	90.23	1.00
llama 8B Q5_K_M	512	pp2048	3939.60	4095.40	1.04
llama 8B Q5_K_M	512	tg128	90.25	90.22	1.00
llama 8B Q5_K_M	1024	pp2048	3994.84	4320.70	1.08
llama 8B Q5_K_M	1024	tg128	90.26	90.22	1.00
llama 8B Q5_K_M	2048	pp2048	4014.84	4457.82	1.11
llama 8B Q5_K_M	2048	tg128	90.27	90.22	1.00
llama 8B Q5_K_S	16	pp2048	954.74	954.30	1.00
llama 8B Q5_K_S	16	tg128	92.52	92.52	1.00
llama 8B Q5_K_S	32	pp2048	1523.20	1524.70	1.00
llama 8B Q5_K_S	32	tg128	92.49	92.47	1.00
llama 8B Q5_K_S	64	pp2048	2131.89	2132.01	1.00
llama 8B Q5_K_S	64	tg128	92.41	92.43	1.00
llama 8B Q5_K_S	128	pp2048	2911.32	2903.33	1.00
llama 8B Q5_K_S	128	tg128	92.43	92.42	1.00
llama 8B Q5_K_S	256	pp2048	3912.68	3913.36	1.00
llama 8B Q5_K_S	256	tg128	92.43	92.42	1.00
llama 8B Q5_K_S	512	pp2048	4156.89	4155.00	1.00
llama 8B Q5_K_S	512	tg128	92.42	92.40	1.00
llama 8B Q5_K_S	1024	pp2048	4165.03	4312.18	1.04
llama 8B Q5_K_S	1024	tg128	92.43	92.39	1.00
llama 8B Q5_K_S	2048	pp2048	4186.34	4452.33	1.06
llama 8B Q5_K_S	2048	tg128	92.45	92.40	1.00
llama 8B Q6_K	16	pp2048	783.66	783.28	1.00
llama 8B Q6_K	16	tg128	79.49	79.48	1.00
llama 8B Q6_K	32	pp2048	1214.14	1217.77	1.00
llama 8B Q6_K	32	tg128	79.46	79.48	1.00
llama 8B Q6_K	64	pp2048	1715.03	1717.14	1.00
llama 8B Q6_K	64	tg128	79.44	79.39	1.00
llama 8B Q6_K	128	pp2048	2308.30	2308.90	1.00
llama 8B Q6_K	128	tg128	79.42	79.43	1.00
llama 8B Q6_K	256	pp2048	2959.04	2958.83	1.00
llama 8B Q6_K	256	tg128	79.43	79.43	1.00
llama 8B Q6_K	512	pp2048	2997.42	3891.06	1.30
llama 8B Q6_K	512	tg128	79.42	79.43	1.00
llama 8B Q6_K	1024	pp2048	3162.09	4329.69	1.37
llama 8B Q6_K	1024	tg128	79.43	79.43	1.00
llama 8B Q6_K	2048	pp2048	3170.71	4422.00	1.39
llama 8B Q6_K	2048	tg128	79.44	79.44	1.00
llama 8B Q8_0	16	pp2048	884.22	886.20	1.00
llama 8B Q8_0	16	tg128	66.38	66.48	1.00
llama 8B Q8_0	32	pp2048	1459.36	1461.55	1.00
llama 8B Q8_0	32	tg128	66.38	66.42	1.00
llama 8B Q8_0	64	pp2048	2169.32	2167.93	1.00
llama 8B Q8_0	64	tg128	66.36	66.38	1.00
llama 8B Q8_0	128	pp2048	3064.51	3069.78	1.00
llama 8B Q8_0	128	tg128	66.34	66.33	1.00
llama 8B Q8_0	256	pp2048	4290.85	4300.17	1.00
llama 8B Q8_0	256	tg128	66.34	66.34	1.00
llama 8B Q8_0	512	pp2048	4601.03	4610.29	1.00
llama 8B Q8_0	512	tg128	66.34	66.34	1.00
llama 8B Q8_0	1024	pp2048	4616.89	4627.88	1.00
llama 8B Q8_0	1024	tg128	66.32	66.35	1.00
llama 8B Q8_0	2048	pp2048	4661.04	4667.18	1.00
llama 8B Q8_0	2048	tg128	66.33	66.35	1.00

itterative · 2026-09-09T19:39:24Z

I ran the same benchmark on rdna 3 as well. These are all using -r 1 option in llama-bench, and they might not be most stable. I can rerun these with more repeats, but it will take a fair bit of time to get these. Also, the changes also affect cdna, and I can't test this one myself.

RDNA 3 master vs PR

Model	Microbatch size	Test	t/s master	t/s rdna4-matmult-optimizations	Speedup
llama 8B IQ1_M - 1.75 bpw	16	pp2048	365.90	365.24	1.00
llama 8B IQ1_M - 1.75 bpw	16	tg128	175.81	176.32	1.00
llama 8B IQ1_M - 1.75 bpw	32	pp2048	580.58	673.72	1.16
llama 8B IQ1_M - 1.75 bpw	32	tg128	175.51	176.05	1.00
llama 8B IQ1_M - 1.75 bpw	64	pp2048	1150.60	1159.41	1.01
llama 8B IQ1_M - 1.75 bpw	64	tg128	175.53	176.10	1.00
llama 8B IQ1_M - 1.75 bpw	128	pp2048	1781.85	1763.11	0.99
llama 8B IQ1_M - 1.75 bpw	128	tg128	175.51	175.88	1.00
llama 8B IQ1_M - 1.75 bpw	256	pp2048	2982.22	2977.26	1.00
llama 8B IQ1_M - 1.75 bpw	256	tg128	175.53	175.82	1.00
llama 8B IQ1_M - 1.75 bpw	512	pp2048	3016.78	3022.06	1.00
llama 8B IQ1_M - 1.75 bpw	512	tg128	175.46	175.71	1.00
llama 8B IQ1_M - 1.75 bpw	1024	pp2048	3062.48	3065.01	1.00
llama 8B IQ1_M - 1.75 bpw	1024	tg128	175.40	175.71	1.00
llama 8B IQ1_M - 1.75 bpw	2048	pp2048	3231.59	3227.47	1.00
llama 8B IQ1_M - 1.75 bpw	2048	tg128	175.34	175.62	1.00
llama 8B IQ1_S - 1.5625 bpw	16	pp2048	1051.19	1035.33	0.98
llama 8B IQ1_S - 1.5625 bpw	16	tg128	183.59	183.60	1.00
llama 8B IQ1_S - 1.5625 bpw	32	pp2048	1206.22	1556.48	1.29
llama 8B IQ1_S - 1.5625 bpw	32	tg128	183.05	183.29	1.00
llama 8B IQ1_S - 1.5625 bpw	64	pp2048	1993.32	2022.42	1.01
llama 8B IQ1_S - 1.5625 bpw	64	tg128	183.01	183.11	1.00
llama 8B IQ1_S - 1.5625 bpw	128	pp2048	2367.01	2341.94	0.99
llama 8B IQ1_S - 1.5625 bpw	128	tg128	182.92	183.11	1.00
llama 8B IQ1_S - 1.5625 bpw	256	pp2048	3272.11	3272.38	1.00
llama 8B IQ1_S - 1.5625 bpw	256	tg128	182.98	183.12	1.00
llama 8B IQ1_S - 1.5625 bpw	512	pp2048	3457.66	3459.00	1.00
llama 8B IQ1_S - 1.5625 bpw	512	tg128	182.84	183.01	1.00
llama 8B IQ1_S - 1.5625 bpw	1024	pp2048	3628.81	3628.65	1.00
llama 8B IQ1_S - 1.5625 bpw	1024	tg128	182.65	182.86	1.00
llama 8B IQ1_S - 1.5625 bpw	2048	pp2048	3631.88	3629.01	1.00
llama 8B IQ1_S - 1.5625 bpw	2048	tg128	182.76	182.79	1.00
llama 8B IQ2_M - 2.7 bpw	16	pp2048	703.43	702.51	1.00
llama 8B IQ2_M - 2.7 bpw	16	tg128	113.23	113.21	1.00
llama 8B IQ2_M - 2.7 bpw	32	pp2048	1137.60	1141.11	1.00
llama 8B IQ2_M - 2.7 bpw	32	tg128	113.21	113.19	1.00
llama 8B IQ2_M - 2.7 bpw	64	pp2048	1542.66	1557.65	1.01
llama 8B IQ2_M - 2.7 bpw	64	tg128	113.16	113.12	1.00
llama 8B IQ2_M - 2.7 bpw	128	pp2048	1976.09	1985.34	1.00
llama 8B IQ2_M - 2.7 bpw	128	tg128	113.12	113.12	1.00
llama 8B IQ2_M - 2.7 bpw	256	pp2048	2906.23	2910.38	1.00
llama 8B IQ2_M - 2.7 bpw	256	tg128	113.32	113.32	1.00
llama 8B IQ2_M - 2.7 bpw	512	pp2048	2955.34	2962.43	1.00
llama 8B IQ2_M - 2.7 bpw	512	tg128	113.32	113.39	1.00
llama 8B IQ2_M - 2.7 bpw	1024	pp2048	3002.86	2999.65	1.00
llama 8B IQ2_M - 2.7 bpw	1024	tg128	113.30	113.26	1.00
llama 8B IQ2_M - 2.7 bpw	2048	pp2048	3117.51	3123.34	1.00
llama 8B IQ2_M - 2.7 bpw	2048	tg128	113.14	113.16	1.00
llama 8B IQ2_S - 2.5 bpw	16	pp2048	704.13	700.68	1.00
llama 8B IQ2_S - 2.5 bpw	16	tg128	113.51	113.40	1.00
llama 8B IQ2_S - 2.5 bpw	32	pp2048	1145.56	1145.66	1.00
llama 8B IQ2_S - 2.5 bpw	32	tg128	113.39	113.26	1.00
llama 8B IQ2_S - 2.5 bpw	64	pp2048	1575.11	1576.02	1.00
llama 8B IQ2_S - 2.5 bpw	64	tg128	113.01	113.18	1.00
llama 8B IQ2_S - 2.5 bpw	128	pp2048	1981.70	1977.70	1.00
llama 8B IQ2_S - 2.5 bpw	128	tg128	113.25	113.14	1.00
llama 8B IQ2_S - 2.5 bpw	256	pp2048	2947.41	2949.65	1.00
llama 8B IQ2_S - 2.5 bpw	256	tg128	113.42	113.26	1.00
llama 8B IQ2_S - 2.5 bpw	512	pp2048	2963.02	2962.72	1.00
llama 8B IQ2_S - 2.5 bpw	512	tg128	113.46	113.26	1.00
llama 8B IQ2_S - 2.5 bpw	1024	pp2048	2999.72	3004.40	1.00
llama 8B IQ2_S - 2.5 bpw	1024	tg128	113.33	113.20	1.00
llama 8B IQ2_S - 2.5 bpw	2048	pp2048	3127.61	3124.93	1.00
llama 8B IQ2_S - 2.5 bpw	2048	tg128	113.29	113.16	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	pp2048	701.30	699.12	1.00
llama 8B IQ2_XS - 2.3125 bpw	16	tg128	114.89	114.99	1.00
llama 8B IQ2_XS - 2.3125 bpw	32	pp2048	953.24	1172.99	1.23
llama 8B IQ2_XS - 2.3125 bpw	32	tg128	114.78	114.81	1.00
llama 8B IQ2_XS - 2.3125 bpw	64	pp2048	1505.52	1514.15	1.01
llama 8B IQ2_XS - 2.3125 bpw	64	tg128	114.55	114.48	1.00
llama 8B IQ2_XS - 2.3125 bpw	128	pp2048	1930.99	1912.48	0.99
llama 8B IQ2_XS - 2.3125 bpw	128	tg128	114.66	114.74	1.00
llama 8B IQ2_XS - 2.3125 bpw	256	pp2048	2930.33	2926.39	1.00
llama 8B IQ2_XS - 2.3125 bpw	256	tg128	114.70	114.84	1.00
llama 8B IQ2_XS - 2.3125 bpw	512	pp2048	3019.02	3012.87	1.00
llama 8B IQ2_XS - 2.3125 bpw	512	tg128	114.77	114.86	1.00
llama 8B IQ2_XS - 2.3125 bpw	1024	pp2048	3002.14	3002.26	1.00
llama 8B IQ2_XS - 2.3125 bpw	1024	tg128	114.70	114.85	1.00
llama 8B IQ2_XS - 2.3125 bpw	2048	pp2048	3210.37	3203.95	1.00
llama 8B IQ2_XS - 2.3125 bpw	2048	tg128	114.70	114.77	1.00
llama 8B IQ2_XXS - 2.0625 bpw	16	pp2048	866.85	860.00	0.99
llama 8B IQ2_XXS - 2.0625 bpw	16	tg128	117.55	117.55	1.00
llama 8B IQ2_XXS - 2.0625 bpw	32	pp2048	1015.29	1268.97	1.25
llama 8B IQ2_XXS - 2.0625 bpw	32	tg128	117.34	117.34	1.00
llama 8B IQ2_XXS - 2.0625 bpw	64	pp2048	1833.65	1856.20	1.01
llama 8B IQ2_XXS - 2.0625 bpw	64	tg128	117.29	117.27	1.00
llama 8B IQ2_XXS - 2.0625 bpw	128	pp2048	2206.03	2192.79	0.99
llama 8B IQ2_XXS - 2.0625 bpw	128	tg128	117.22	117.22	1.00
llama 8B IQ2_XXS - 2.0625 bpw	256	pp2048	3044.08	3045.12	1.00
llama 8B IQ2_XXS - 2.0625 bpw	256	tg128	117.26	117.19	1.00
llama 8B IQ2_XXS - 2.0625 bpw	512	pp2048	3138.28	3153.83	1.00
llama 8B IQ2_XXS - 2.0625 bpw	512	tg128	117.14	117.18	1.00
llama 8B IQ2_XXS - 2.0625 bpw	1024	pp2048	3422.59	3421.21	1.00
llama 8B IQ2_XXS - 2.0625 bpw	1024	tg128	117.16	117.14	1.00
llama 8B IQ2_XXS - 2.0625 bpw	2048	pp2048	3418.07	3412.44	1.00
llama 8B IQ2_XXS - 2.0625 bpw	2048	tg128	117.14	117.10	1.00
llama 8B IQ3_S - 3.4375 bpw	16	pp2048	758.15	756.19	1.00
llama 8B IQ3_S - 3.4375 bpw	16	tg128	110.10	110.02	1.00
llama 8B IQ3_S - 3.4375 bpw	32	pp2048	986.57	989.12	1.00
llama 8B IQ3_S - 3.4375 bpw	32	tg128	110.17	109.87	1.00
llama 8B IQ3_S - 3.4375 bpw	64	pp2048	1858.45	1856.89	1.00
llama 8B IQ3_S - 3.4375 bpw	64	tg128	110.08	109.85	1.00
llama 8B IQ3_S - 3.4375 bpw	128	pp2048	2225.07	2236.03	1.00
llama 8B IQ3_S - 3.4375 bpw	128	tg128	110.03	109.74	1.00
llama 8B IQ3_S - 3.4375 bpw	256	pp2048	3033.23	3033.37	1.00
llama 8B IQ3_S - 3.4375 bpw	256	tg128	110.00	109.75	1.00
llama 8B IQ3_S - 3.4375 bpw	512	pp2048	3110.84	3103.51	1.00
llama 8B IQ3_S - 3.4375 bpw	512	tg128	109.97	109.76	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	pp2048	3318.62	3312.56	1.00
llama 8B IQ3_S - 3.4375 bpw	1024	tg128	109.96	109.70	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	pp2048	3264.78	3266.21	1.00
llama 8B IQ3_S - 3.4375 bpw	2048	tg128	109.92	109.64	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	pp2048	783.80	781.99	1.00
llama 8B IQ3_S mix - 3.66 bpw	16	tg128	111.02	111.02	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	pp2048	1062.79	1064.17	1.00
llama 8B IQ3_S mix - 3.66 bpw	32	tg128	110.84	110.82	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	pp2048	1878.44	1876.06	1.00
llama 8B IQ3_S mix - 3.66 bpw	64	tg128	110.80	110.79	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	pp2048	2252.70	2250.15	1.00
llama 8B IQ3_S mix - 3.66 bpw	128	tg128	110.75	110.75	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	pp2048	3055.02	3056.19	1.00
llama 8B IQ3_S mix - 3.66 bpw	256	tg128	110.72	110.74	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	pp2048	3131.68	3133.71	1.00
llama 8B IQ3_S mix - 3.66 bpw	512	tg128	110.68	110.74	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	pp2048	3339.98	3339.41	1.00
llama 8B IQ3_S mix - 3.66 bpw	1024	tg128	110.65	110.68	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	pp2048	3287.57	3288.38	1.00
llama 8B IQ3_S mix - 3.66 bpw	2048	tg128	110.65	110.66	1.00
llama 8B IQ3_XS - 3.3 bpw	16	pp2048	732.53	731.41	1.00
llama 8B IQ3_XS - 3.3 bpw	16	tg128	110.70	110.67	1.00
llama 8B IQ3_XS - 3.3 bpw	32	pp2048	922.93	921.95	1.00
llama 8B IQ3_XS - 3.3 bpw	32	tg128	110.53	110.60	1.00
llama 8B IQ3_XS - 3.3 bpw	64	pp2048	1854.13	1854.01	1.00
llama 8B IQ3_XS - 3.3 bpw	64	tg128	110.54	110.51	1.00
llama 8B IQ3_XS - 3.3 bpw	128	pp2048	2187.01	2184.47	1.00
llama 8B IQ3_XS - 3.3 bpw	128	tg128	110.51	110.50	1.00
llama 8B IQ3_XS - 3.3 bpw	256	pp2048	2966.68	2969.81	1.00
llama 8B IQ3_XS - 3.3 bpw	256	tg128	110.49	110.50	1.00
llama 8B IQ3_XS - 3.3 bpw	512	pp2048	3060.97	3061.93	1.00
llama 8B IQ3_XS - 3.3 bpw	512	tg128	110.49	110.46	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	pp2048	3289.51	3288.86	1.00
llama 8B IQ3_XS - 3.3 bpw	1024	tg128	110.51	110.46	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	pp2048	3241.93	3239.81	1.00
llama 8B IQ3_XS - 3.3 bpw	2048	tg128	110.45	110.48	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	pp2048	715.83	716.35	1.00
llama 8B IQ3_XXS - 3.0625 bpw	16	tg128	111.50	111.39	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	pp2048	953.58	953.34	1.00
llama 8B IQ3_XXS - 3.0625 bpw	32	tg128	111.46	111.39	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	pp2048	1754.52	1747.85	1.00
llama 8B IQ3_XXS - 3.0625 bpw	64	tg128	111.36	111.15	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	pp2048	2121.22	2121.10	1.00
llama 8B IQ3_XXS - 3.0625 bpw	128	tg128	111.36	111.37	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	pp2048	2943.75	2942.02	1.00
llama 8B IQ3_XXS - 3.0625 bpw	256	tg128	111.52	111.48	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	pp2048	3055.08	3055.82	1.00
llama 8B IQ3_XXS - 3.0625 bpw	512	tg128	111.56	111.53	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	pp2048	3208.54	3207.16	1.00
llama 8B IQ3_XXS - 3.0625 bpw	1024	tg128	111.50	111.53	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	pp2048	3203.72	3210.19	1.00
llama 8B IQ3_XXS - 3.0625 bpw	2048	tg128	111.50	111.44	1.00
llama 8B IQ4_NL - 4.5 bpw	16	pp2048	1067.31	1067.40	1.00
llama 8B IQ4_NL - 4.5 bpw	16	tg128	129.86	129.86	1.00
llama 8B IQ4_NL - 4.5 bpw	32	pp2048	1209.07	1209.71	1.00
llama 8B IQ4_NL - 4.5 bpw	32	tg128	129.82	129.80	1.00
llama 8B IQ4_NL - 4.5 bpw	64	pp2048	2167.32	2170.46	1.00
llama 8B IQ4_NL - 4.5 bpw	64	tg128	129.82	129.77	1.00
llama 8B IQ4_NL - 4.5 bpw	128	pp2048	2541.49	2551.72	1.00
llama 8B IQ4_NL - 4.5 bpw	128	tg128	129.80	129.68	1.00
llama 8B IQ4_NL - 4.5 bpw	256	pp2048	3480.75	3479.13	1.00
llama 8B IQ4_NL - 4.5 bpw	256	tg128	129.81	129.75	1.00
llama 8B IQ4_NL - 4.5 bpw	512	pp2048	3692.54	3689.48	1.00
llama 8B IQ4_NL - 4.5 bpw	512	tg128	129.82	127.98	0.99
llama 8B IQ4_NL - 4.5 bpw	1024	pp2048	3760.33	3757.69	1.00
llama 8B IQ4_NL - 4.5 bpw	1024	tg128	129.78	129.69	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	pp2048	3707.48	3706.41	1.00
llama 8B IQ4_NL - 4.5 bpw	2048	tg128	129.83	129.75	1.00
llama 8B IQ4_XS - 4.25 bpw	16	pp2048	1090.13	1092.84	1.00
llama 8B IQ4_XS - 4.25 bpw	16	tg128	137.76	137.96	1.00
llama 8B IQ4_XS - 4.25 bpw	32	pp2048	1235.82	1236.36	1.00
llama 8B IQ4_XS - 4.25 bpw	32	tg128	137.52	137.72	1.00
llama 8B IQ4_XS - 4.25 bpw	64	pp2048	2162.64	2164.50	1.00
llama 8B IQ4_XS - 4.25 bpw	64	tg128	137.39	137.64	1.00
llama 8B IQ4_XS - 4.25 bpw	128	pp2048	2572.34	2577.08	1.00
llama 8B IQ4_XS - 4.25 bpw	128	tg128	137.35	137.74	1.00
llama 8B IQ4_XS - 4.25 bpw	256	pp2048	3509.82	3507.38	1.00
llama 8B IQ4_XS - 4.25 bpw	256	tg128	137.30	137.72	1.00
llama 8B IQ4_XS - 4.25 bpw	512	pp2048	3721.54	3718.32	1.00
llama 8B IQ4_XS - 4.25 bpw	512	tg128	137.26	137.70	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	pp2048	3796.83	3795.58	1.00
llama 8B IQ4_XS - 4.25 bpw	1024	tg128	137.28	137.66	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	pp2048	3770.04	3764.69	1.00
llama 8B IQ4_XS - 4.25 bpw	2048	tg128	137.29	137.69	1.00
llama 8B Q2_K_M	16	pp2048	858.53	762.87	0.89
llama 8B Q2_K_M	16	tg128	141.44	141.57	1.00
llama 8B Q2_K_M	32	pp2048	272.41	982.93	3.61
llama 8B Q2_K_M	32	tg128	140.68	141.43	1.01
llama 8B Q2_K_M	64	pp2048	1169.56	1228.38	1.05
llama 8B Q2_K_M	64	tg128	140.86	141.47	1.00
llama 8B Q2_K_M	128	pp2048	1582.25	1523.71	0.96
llama 8B Q2_K_M	128	tg128	140.82	141.28	1.00
llama 8B Q2_K_M	256	pp2048	2937.56	2937.27	1.00
llama 8B Q2_K_M	256	tg128	140.85	141.38	1.00
llama 8B Q2_K_M	512	pp2048	2861.93	2855.79	1.00
llama 8B Q2_K_M	512	tg128	140.77	141.26	1.00
llama 8B Q2_K_M	1024	pp2048	3004.51	3005.50	1.00
llama 8B Q2_K_M	1024	tg128	140.76	141.24	1.00
llama 8B Q2_K_M	2048	pp2048	3225.40	3235.00	1.00
llama 8B Q2_K_M	2048	tg128	140.77	141.27	1.00
llama 8B Q2_K_S	16	pp2048	873.55	720.53	0.82
llama 8B Q2_K_S	16	tg128	153.87	153.84	1.00
llama 8B Q2_K_S	32	pp2048	183.69	878.44	4.78
llama 8B Q2_K_S	32	tg128	153.37	153.91	1.00
llama 8B Q2_K_S	64	pp2048	946.85	1053.57	1.11
llama 8B Q2_K_S	64	tg128	153.54	153.83	1.00
llama 8B Q2_K_S	128	pp2048	1431.04	1302.23	0.91
llama 8B Q2_K_S	128	tg128	153.42	153.60	1.00
llama 8B Q2_K_S	256	pp2048	2902.15	2896.55	1.00
llama 8B Q2_K_S	256	tg128	153.50	153.68	1.00
llama 8B Q2_K_S	512	pp2048	3007.54	3006.69	1.00
llama 8B Q2_K_S	512	tg128	153.56	153.75	1.00
llama 8B Q2_K_S	1024	pp2048	3003.29	3018.73	1.01
llama 8B Q2_K_S	1024	tg128	153.50	153.70	1.00
llama 8B Q2_K_S	2048	pp2048	3216.89	3214.43	1.00
llama 8B Q2_K_S	2048	tg128	153.35	153.65	1.00
llama 8B Q3_K_L	16	pp2048	915.53	912.42	1.00
llama 8B Q3_K_L	16	tg128	118.39	118.38	1.00
llama 8B Q3_K_L	32	pp2048	1414.69	1415.03	1.00
llama 8B Q3_K_L	32	tg128	118.11	118.14	1.00
llama 8B Q3_K_L	64	pp2048	1887.74	1886.78	1.00
llama 8B Q3_K_L	64	tg128	117.92	118.03	1.00
llama 8B Q3_K_L	128	pp2048	2262.10	2265.64	1.00
llama 8B Q3_K_L	128	tg128	117.85	117.87	1.00
llama 8B Q3_K_L	256	pp2048	3019.71	3021.12	1.00
llama 8B Q3_K_L	256	tg128	117.76	117.88	1.00
llama 8B Q3_K_L	512	pp2048	3057.83	3060.35	1.00
llama 8B Q3_K_L	512	tg128	117.71	117.75	1.00
llama 8B Q3_K_L	1024	pp2048	3314.56	3311.29	1.00
llama 8B Q3_K_L	1024	tg128	117.68	117.73	1.00
llama 8B Q3_K_L	2048	pp2048	3323.16	3320.79	1.00
llama 8B Q3_K_L	2048	tg128	117.56	117.65	1.00
llama 8B Q3_K_M	16	pp2048	912.38	911.70	1.00
llama 8B Q3_K_M	16	tg128	124.85	124.68	1.00
llama 8B Q3_K_M	32	pp2048	1434.61	1432.75	1.00
llama 8B Q3_K_M	32	tg128	124.57	124.42	1.00
llama 8B Q3_K_M	64	pp2048	1897.20	1893.57	1.00
llama 8B Q3_K_M	64	tg128	124.39	124.23	1.00
llama 8B Q3_K_M	128	pp2048	2281.00	2281.73	1.00
llama 8B Q3_K_M	128	tg128	124.30	124.13	1.00
llama 8B Q3_K_M	256	pp2048	3049.59	3051.89	1.00
llama 8B Q3_K_M	256	tg128	124.21	124.04	1.00
llama 8B Q3_K_M	512	pp2048	3082.26	3083.56	1.00
llama 8B Q3_K_M	512	tg128	124.18	124.02	1.00
llama 8B Q3_K_M	1024	pp2048	3347.10	3346.50	1.00
llama 8B Q3_K_M	1024	tg128	124.14	123.92	1.00
llama 8B Q3_K_M	2048	pp2048	3350.84	3351.59	1.00
llama 8B Q3_K_M	2048	tg128	124.08	123.89	1.00
llama 8B Q3_K_S	16	pp2048	859.35	857.63	1.00
llama 8B Q3_K_S	16	tg128	122.57	122.69	1.00
llama 8B Q3_K_S	32	pp2048	1342.68	1351.26	1.01
llama 8B Q3_K_S	32	tg128	122.27	122.41	1.00
llama 8B Q3_K_S	64	pp2048	1811.87	1818.85	1.00
llama 8B Q3_K_S	64	tg128	121.95	122.34	1.00
llama 8B Q3_K_S	128	pp2048	2193.16	2196.17	1.00
llama 8B Q3_K_S	128	tg128	122.15	122.29	1.00
llama 8B Q3_K_S	256	pp2048	2917.88	2917.08	1.00
llama 8B Q3_K_S	256	tg128	122.19	122.28	1.00
llama 8B Q3_K_S	512	pp2048	2948.50	2950.30	1.00
llama 8B Q3_K_S	512	tg128	122.14	122.27	1.00
llama 8B Q3_K_S	1024	pp2048	3189.24	3186.57	1.00
llama 8B Q3_K_S	1024	tg128	122.11	122.21	1.00
llama 8B Q3_K_S	2048	pp2048	3196.16	3192.99	1.00
llama 8B Q3_K_S	2048	tg128	122.04	122.20	1.00
llama 8B Q4_0	16	pp2048	1046.42	1044.46	1.00
llama 8B Q4_0	16	tg128	133.19	132.75	1.00
llama 8B Q4_0	32	pp2048	1056.70	1058.68	1.00
llama 8B Q4_0	32	tg128	133.06	132.65	1.00
llama 8B Q4_0	64	pp2048	2085.04	2088.77	1.00
llama 8B Q4_0	64	tg128	132.98	132.61	1.00
llama 8B Q4_0	128	pp2048	2499.84	2505.62	1.00
llama 8B Q4_0	128	tg128	133.01	132.60	1.00
llama 8B Q4_0	256	pp2048	3437.19	3440.93	1.00
llama 8B Q4_0	256	tg128	133.03	132.63	1.00
llama 8B Q4_0	512	pp2048	3624.54	3636.39	1.00
llama 8B Q4_0	512	tg128	133.04	131.92	0.99
llama 8B Q4_0	1024	pp2048	3754.51	3753.59	1.00
llama 8B Q4_0	1024	tg128	133.04	132.31	0.99
llama 8B Q4_0	2048	pp2048	3775.39	3773.64	1.00
llama 8B Q4_0	2048	tg128	133.13	132.66	1.00
llama 8B Q4_1	16	pp2048	1007.45	1005.38	1.00
llama 8B Q4_1	16	tg128	128.17	126.88	0.99
llama 8B Q4_1	32	pp2048	1579.94	1577.80	1.00
llama 8B Q4_1	32	tg128	128.03	126.79	0.99
llama 8B Q4_1	64	pp2048	2063.21	2057.21	1.00
llama 8B Q4_1	64	tg128	128.00	126.77	0.99
llama 8B Q4_1	128	pp2048	2412.29	2415.47	1.00
llama 8B Q4_1	128	tg128	127.99	126.75	0.99
llama 8B Q4_1	256	pp2048	3292.62	3295.64	1.00
llama 8B Q4_1	256	tg128	127.94	126.73	0.99
llama 8B Q4_1	512	pp2048	3478.20	3477.36	1.00
llama 8B Q4_1	512	tg128	127.96	126.73	0.99
llama 8B Q4_1	1024	pp2048	3665.22	3662.79	1.00
llama 8B Q4_1	1024	tg128	127.93	126.66	0.99
llama 8B Q4_1	2048	pp2048	3687.01	3680.94	1.00
llama 8B Q4_1	2048	tg128	127.95	126.72	0.99
llama 8B Q4_K_M	16	pp2048	934.78	940.06	1.01
llama 8B Q4_K_M	16	tg128	124.40	124.46	1.00
llama 8B Q4_K_M	32	pp2048	1482.14	1509.33	1.02
llama 8B Q4_K_M	32	tg128	124.26	124.42	1.00
llama 8B Q4_K_M	64	pp2048	1902.55	1962.46	1.03
llama 8B Q4_K_M	64	tg128	124.21	124.29	1.00
llama 8B Q4_K_M	128	pp2048	2254.81	2355.45	1.04
llama 8B Q4_K_M	128	tg128	124.23	124.23	1.00
llama 8B Q4_K_M	256	pp2048	3205.06	3207.77	1.00
llama 8B Q4_K_M	256	tg128	124.29	124.41	1.00
llama 8B Q4_K_M	512	pp2048	3497.99	3501.30	1.00
llama 8B Q4_K_M	512	tg128	124.27	124.38	1.00
llama 8B Q4_K_M	1024	pp2048	3619.90	3619.80	1.00
llama 8B Q4_K_M	1024	tg128	124.26	124.37	1.00
llama 8B Q4_K_M	2048	pp2048	3591.86	3597.93	1.00
llama 8B Q4_K_M	2048	tg128	124.24	124.35	1.00
llama 8B Q4_K_S	16	pp2048	987.78	987.84	1.00
llama 8B Q4_K_S	16	tg128	128.70	128.63	1.00
llama 8B Q4_K_S	32	pp2048	1595.14	1592.29	1.00
llama 8B Q4_K_S	32	tg128	128.67	128.53	1.00
llama 8B Q4_K_S	64	pp2048	2051.06	2047.01	1.00
llama 8B Q4_K_S	64	tg128	128.46	128.46	1.00
llama 8B Q4_K_S	128	pp2048	2428.18	2432.10	1.00
llama 8B Q4_K_S	128	tg128	128.56	128.45	1.00
llama 8B Q4_K_S	256	pp2048	3310.37	3308.07	1.00
llama 8B Q4_K_S	256	tg128	128.57	128.49	1.00
llama 8B Q4_K_S	512	pp2048	3483.95	3485.88	1.00
llama 8B Q4_K_S	512	tg128	128.58	128.51	1.00
llama 8B Q4_K_S	1024	pp2048	3662.49	3665.33	1.00
llama 8B Q4_K_S	1024	tg128	128.60	128.52	1.00
llama 8B Q4_K_S	2048	pp2048	3679.90	3677.67	1.00
llama 8B Q4_K_S	2048	tg128	128.59	128.49	1.00
llama 8B Q5_0	16	pp2048	915.60	916.15	1.00
llama 8B Q5_0	16	tg128	118.02	118.04	1.00
llama 8B Q5_0	32	pp2048	878.27	877.00	1.00
llama 8B Q5_0	32	tg128	117.96	117.99	1.00
llama 8B Q5_0	64	pp2048	1969.59	1970.99	1.00
llama 8B Q5_0	64	tg128	118.01	118.00	1.00
llama 8B Q5_0	128	pp2048	2414.69	2411.81	1.00
llama 8B Q5_0	128	tg128	118.07	117.96	1.00
llama 8B Q5_0	256	pp2048	3291.39	3293.02	1.00
llama 8B Q5_0	256	tg128	118.11	117.99	1.00
llama 8B Q5_0	512	pp2048	3455.53	3452.80	1.00
llama 8B Q5_0	512	tg128	118.08	118.00	1.00
llama 8B Q5_0	1024	pp2048	3587.66	3583.87	1.00
llama 8B Q5_0	1024	tg128	118.06	117.97	1.00
llama 8B Q5_0	2048	pp2048	3586.15	3585.66	1.00
llama 8B Q5_0	2048	tg128	118.08	117.97	1.00
llama 8B Q5_1	16	pp2048	729.50	730.83	1.00
llama 8B Q5_1	16	tg128	111.96	112.36	1.00
llama 8B Q5_1	32	pp2048	1232.30	1230.38	1.00
llama 8B Q5_1	32	tg128	111.88	112.31	1.00
llama 8B Q5_1	64	pp2048	1792.57	1791.56	1.00
llama 8B Q5_1	64	tg128	111.94	112.37	1.00
llama 8B Q5_1	128	pp2048	2255.66	2247.44	1.00
llama 8B Q5_1	128	tg128	111.93	112.31	1.00
llama 8B Q5_1	256	pp2048	3119.07	3115.69	1.00
llama 8B Q5_1	256	tg128	111.94	112.30	1.00
llama 8B Q5_1	512	pp2048	3349.72	3350.32	1.00
llama 8B Q5_1	512	tg128	111.96	112.32	1.00
llama 8B Q5_1	1024	pp2048	3540.99	3536.71	1.00
llama 8B Q5_1	1024	tg128	111.93	112.30	1.00
llama 8B Q5_1	2048	pp2048	3540.08	3538.44	1.00
llama 8B Q5_1	2048	tg128	111.94	112.31	1.00
llama 8B Q5_K_M	16	pp2048	971.14	976.33	1.01
llama 8B Q5_K_M	16	tg128	111.46	111.31	1.00
llama 8B Q5_K_M	32	pp2048	1457.04	1485.48	1.02
llama 8B Q5_K_M	32	tg128	111.34	111.18	1.00
llama 8B Q5_K_M	64	pp2048	1895.79	1952.51	1.03
llama 8B Q5_K_M	64	tg128	111.28	111.13	1.00
llama 8B Q5_K_M	128	pp2048	2223.42	2316.66	1.04
llama 8B Q5_K_M	128	tg128	111.27	111.13	1.00
llama 8B Q5_K_M	256	pp2048	3152.85	3157.59	1.00
llama 8B Q5_K_M	256	tg128	111.32	111.18	1.00
llama 8B Q5_K_M	512	pp2048	3434.14	3434.33	1.00
llama 8B Q5_K_M	512	tg128	111.29	111.13	1.00
llama 8B Q5_K_M	1024	pp2048	3541.49	3539.29	1.00
llama 8B Q5_K_M	1024	tg128	111.28	111.11	1.00
llama 8B Q5_K_M	2048	pp2048	3522.40	3520.00	1.00
llama 8B Q5_K_M	2048	tg128	111.26	111.09	1.00
llama 8B Q5_K_S	16	pp2048	1034.10	1033.90	1.00
llama 8B Q5_K_S	16	tg128	112.50	112.42	1.00
llama 8B Q5_K_S	32	pp2048	1563.94	1560.96	1.00
llama 8B Q5_K_S	32	tg128	112.45	112.42	1.00
llama 8B Q5_K_S	64	pp2048	2043.00	2038.73	1.00
llama 8B Q5_K_S	64	tg128	112.39	112.34	1.00
llama 8B Q5_K_S	128	pp2048	2386.21	2384.37	1.00
llama 8B Q5_K_S	128	tg128	112.35	112.33	1.00
llama 8B Q5_K_S	256	pp2048	3242.45	3241.29	1.00
llama 8B Q5_K_S	256	tg128	112.38	112.33	1.00
llama 8B Q5_K_S	512	pp2048	3412.14	3413.70	1.00
llama 8B Q5_K_S	512	tg128	112.38	112.28	1.00
llama 8B Q5_K_S	1024	pp2048	3573.12	3570.84	1.00
llama 8B Q5_K_S	1024	tg128	112.35	112.31	1.00
llama 8B Q5_K_S	2048	pp2048	3587.13	3581.38	1.00
llama 8B Q5_K_S	2048	tg128	112.36	112.30	1.00
llama 8B Q6_K	16	pp2048	770.08	796.25	1.03
llama 8B Q6_K	16	tg128	101.20	101.43	1.00
llama 8B Q6_K	32	pp2048	1113.29	1226.93	1.10
llama 8B Q6_K	32	tg128	101.08	101.34	1.00
llama 8B Q6_K	64	pp2048	1429.38	1640.93	1.15
llama 8B Q6_K	64	tg128	101.06	101.26	1.00
llama 8B Q6_K	128	pp2048	1691.58	2054.39	1.21
llama 8B Q6_K	128	tg128	101.02	101.30	1.00
llama 8B Q6_K	256	pp2048	2773.50	2778.46	1.00
llama 8B Q6_K	256	tg128	101.04	101.30	1.00
llama 8B Q6_K	512	pp2048	2914.06	2919.53	1.00
llama 8B Q6_K	512	tg128	100.99	101.26	1.00
llama 8B Q6_K	1024	pp2048	2959.76	2954.77	1.00
llama 8B Q6_K	1024	tg128	100.98	101.22	1.00
llama 8B Q6_K	2048	pp2048	3158.80	3159.06	1.00
llama 8B Q6_K	2048	tg128	101.04	101.27	1.00
llama 8B Q8_0	16	pp2048	968.03	966.35	1.00
llama 8B Q8_0	16	tg128	89.08	89.18	1.00
llama 8B Q8_0	32	pp2048	1185.35	1172.52	0.99
llama 8B Q8_0	32	tg128	88.90	89.06	1.00
llama 8B Q8_0	64	pp2048	2174.33	2153.87	0.99
llama 8B Q8_0	64	tg128	88.83	88.94	1.00
llama 8B Q8_0	128	pp2048	2602.78	2580.97	0.99
llama 8B Q8_0	128	tg128	88.76	88.87	1.00
llama 8B Q8_0	256	pp2048	3493.79	3464.74	0.99
llama 8B Q8_0	256	tg128	88.69	88.87	1.00
llama 8B Q8_0	512	pp2048	3743.00	3717.35	0.99
llama 8B Q8_0	512	tg128	88.62	88.86	1.00
llama 8B Q8_0	1024	pp2048	3810.15	3790.09	0.99
llama 8B Q8_0	1024	tg128	88.55	88.87	1.00
llama 8B Q8_0	2048	pp2048	3822.26	3810.05	1.00
llama 8B Q8_0	2048	tg128	88.54	88.88	1.00

leonmorlando22 · 2026-09-09T22:55:22Z

gfx1200 coverage data for the RDNA4 matmul work

AI disclosure: assembled with AI assistance (Muse Spark 1.3 driving local runs,
analysis drafted by the agent). Every number is measured on the hardware below;
commits, flags, and commands included for reproduction.

Environment

AMD Radeon RX 9060 XT 16 GB (gfx1200 - underrepresented next to gfx1201 in
this PR's data), ROCm/HIP 7.2.53211, kernel 7.2.3-cachyos-bore, Ryzen 5 5600X
llama.cpp master 6d9c82ea2, HIP-only tree, default codegen
(no -mllvm overrides), -DAMDGPU_TARGETS=gfx1200

Coverage for the n=512 regressions

test-backend-ops perf -o MUL_MAT, m=4096, k=14336 (same shapes as this PR):

quant	n=1 us	n=512 rate
q2_K	78.86	9.89 TFLOPS
q6_K	160.09	15.57 TFLOPS
q4_K	74.22	30.52 TFLOPS
q4_0 (ref)	59.93	33.61 TFLOPS
iq4_xs (ref)	55.20	37.61 TFLOPS

q2_K and q6_K at n=512 are off a cliff vs the field on gfx1200 - consistent with
the un-fixed side of this PR's scope on this build. Happy to re-run these two
cells against any branch.

Model-level anchors (same card, factory files)

llama-bench -ngl 999 -fa 1 -p 512 -n 128 -r 3, ROCm0:

model	pp512	tg128
Llama-3.1-8B Q3_K_S (factory)	1981.8+/-341.6*	58.03
Llama-3.1-8B Q5_K_M (factory)	1754.4+/-322.8*	50.41
Qwen3.8-27B IQ3_S	589.7	17.54

* pp512 +/- is cold-first-rep clock warmup, stable across decode reps.
Q3 moves 64% of Q5's bytes but decodes 1.15x faster - the model-level echo of the
kernel table. Full 18-quant sweep (both backends) held back — #20934 is closed, so I'm filing it as a separate issue and will link it here.

Notes for reviewers

-p type_a= needs exact GGML names (q2_K...q6_K capital-K; iq3_xs/iq3_m
do not exist as types) - tripped over this, documenting so others don't.
K-quants sweep extra m/k shape combos in the harness; table uses m=4096, k=14336 throughout (e.g. q4_K n=1 is 74.22 us there vs 122.98 us at
m=14336,k=4096 - shape matters, compare carefully).
Box available for A/B-ing candidate patches on gfx1200.

itterative requested a review from a team as a code owner July 20, 2026 21:47

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Jul 20, 2026

IMbackK self-assigned this Jul 24, 2026

itterative force-pushed the rdna4-matmult-optimizations branch from c38ecc9 to ae6e2bd Compare July 25, 2026 15:20

itterative marked this pull request as draft July 27, 2026 11:12

Geramy mentioned this pull request Jul 27, 2026

HIP: MMQ Dispatch config modification - separation of RDNA3, 3.5 from 4 and tune 4. #26199

Merged

itterative mentioned this pull request Jul 29, 2026

HIP: tune rdna 3 mmq configs #26284

Merged

IMbackK removed their assignment Aug 25, 2026

github-actions Bot mentioned this pull request Aug 25, 2026

FreeToken ROCm Watch — notification channel Castoff995/FreeToken-rocm-test#1

Draft

This comment was marked as spam.

Sign in to view

itterative added 3 commits September 9, 2026 18:20

fix: q6_k mma

0a71b4f

fix: q2_k mma

140878b

perf: sweep quants for best performance

2ba97e5

itterative force-pushed the rdna4-matmult-optimizations branch from ae6e2bd to 2ba97e5 Compare September 9, 2026 19:39

itterative marked this pull request as ready for review September 9, 2026 19:39

Nyovelt mentioned this pull request Sep 10, 2026

HIP : route Q6_K through MMQ up to ne11 = 1024 on RDNA 3.5 halo-box/strix-llama.cpp#38

Open

Uh oh!

Conversation

itterative commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

test-backend-ops perf -o MUL_MAT

llama-bench -m Qwen3.5-9B-Q4_K_M.gguf -p 512,2048,4096 -n 128 -r 3

Requirements

Uh oh!

fishchev commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

diadras commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jhandwe commented Jul 23, 2026

Uh oh!

Geramy commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itterative commented Jul 24, 2026

Uh oh!

Geramy commented Jul 24, 2026

Uh oh!

JohannesGaessler commented Jul 25, 2026

Uh oh!

itterative commented Jul 25, 2026

Uh oh!

JohannesGaessler commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itterative commented Jul 25, 2026

RDNA 3 (7900 XTX)

RDNA 4 (9070 XT)

Uh oh!

Geramy commented Jul 25, 2026

Uh oh!

JohannesGaessler commented Jul 25, 2026

Uh oh!

itterative commented Jul 27, 2026

Uh oh!

Geramy commented Jul 27, 2026

Uh oh!

itterative commented Jul 28, 2026

Uh oh!

Geramy commented Jul 28, 2026

Uh oh!

itterative commented Jul 28, 2026

Uh oh!

Geramy commented Jul 28, 2026

Uh oh!

itterative commented Jul 29, 2026

Uh oh!

This comment was marked as spam.

IMbackK commented Sep 7, 2026

Uh oh!

itterative commented Sep 7, 2026

Uh oh!

itterative commented Sep 9, 2026

Uh oh!

itterative commented Sep 9, 2026

Uh oh!

leonmorlando22 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

gfx1200 coverage data for the RDNA4 matmul work

Environment

Coverage for the n=512 regressions

Model-level anchors (same card, factory files)

Notes for reviewers

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

itterative commented Jul 20, 2026 •

edited

Loading

fishchev commented Jul 21, 2026 •

edited

Loading

diadras commented Jul 22, 2026 •

edited

Loading

Geramy commented Jul 24, 2026 •

edited

Loading

JohannesGaessler commented Jul 25, 2026 •

edited

Loading

leonmorlando22 commented Sep 9, 2026 •

edited

Loading