CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover by praneshgo · Pull Request #26079 · ggml-org/llama.cpp

praneshgo · 2026-07-24T11:34:11Z

Overview

On CUDA, quantized decode uses mul_mat_vec_q while ne11 <= MMVQ_MAX_BATCH_SIZE (a build-time constant, 8), and MMQ (int8 tensor cores) above that. The switch point is fixed at compile time.

mul_mat_vec_q re-decodes the weights for every output column; MMQ decodes them once into shared memory and reuses them. So mvq's decode work grows with ne11 and MMQ's doesn't, which means the best switch point depends on how expensive the quant type is to decode as well as on the batch size.

This PR adds per-quant switch points on various HW families and batches above this switch point use MMQ instead of mvq.

This largely seems to help dense K-quant decode. The gains shown in the Performance section below are for K-quants (Q2_K/Q3_K / Q4_K / Q5_K / Q6_K) on dense models. Legacy Q4_0 / Q8_0 and IQ2_XS do not seem to benefit — for those, mvq stays ahead across the whole batch range. Lowering the crossover there would only slow things down.

Performance

RTX 5090 (sm_120)

GPU: NVIDIA RTX 5090 (Blackwell sm_120, 32 GB). Decode t/s from
llama-bench -p 1..8 -n 0 -embd 1 -r 50 on --pure single-type ggufs, clocks pinned.
ne11 equals the -p value. mvq = mul_mat_vec_q, MMQ = the tensor-core path.

Q2_K (this branch keeps mvq for ne11 <= 5, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	486	317	168	93
1	MMQ	487	319	169	92
2	mvq	837	548	293	164
2	MMQ	578	375	224	112
3	mvq	1095	660	363	200
3	MMQ	861	561	336	167
4	mvq	1294	753	426	238
4	MMQ	1139	740	446	220
5	mvq	1435	847	487	272
5	MMQ	1385	917	551	274
6	mvq	1554	858	513	282
6	MMQ	1649	1089	658	332
7	mvq	1678	938	565	290
7	MMQ	1930	1271	765	387
8	mvq	1756	970	587	294
8	MMQ	2191	1438	869	440
gain @ ne11=8		+25%	+48%	+48%	+49%

Q3_K (this branch keeps mvq for ne11 <= 5, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	370	225	123	67
1	MMQ	372	224	123	69
2	mvq	688	412	228	127
2	MMQ	554	354	209	107
3	mvq	968	567	319	175
3	MMQ	820	523	313	160
4	mvq	1207	694	390	216
4	MMQ	1087	694	414	212
5	mvq	1402	764	454	248
5	MMQ	1343	858	513	265
6	mvq	1502	856	509	272
6	MMQ	1575	1025	612	315
7	mvq	1712	937	556	291
7	MMQ	1852	1189	715	366
8	mvq	1696	935	558	291
8	MMQ	2089	1345	811	417
gain @ ne11=8		+23%	+44%	+45%	+43%

Q4_K (this branch keeps mvq for ne11 <= 5, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	443	251	139	75
1	MMQ	438	251	139	74
2	mvq	763	457	257	138
2	MMQ	554	354	212	105
3	mvq	1101	635	353	193
3	MMQ	823	524	318	159
4	mvq	1317	725	404	228
4	MMQ	1086	697	421	209
5	mvq	1415	779	460	258
5	MMQ	1334	855	522	259
6	mvq	1524	841	502	269
6	MMQ	1586	1028	626	308
7	mvq	1607	889	536	273
7	MMQ	1837	1192	725	357
8	mvq	1723	932	557	292
8	MMQ	2097	1358	822	414
gain @ ne11=8		+22%	+46%	+48%	+42%

Q5_K (this branch keeps mvq for ne11 <= 5, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	399	221	124	65
1	MMQ	398	222	124	65
2	mvq	712	402	229	123
2	MMQ	521	321	191	97
3	mvq	1028	577	326	175
3	MMQ	776	478	286	145
4	mvq	1263	689	389	214
4	MMQ	1027	634	379	192
5	mvq	1394	765	452	246
5	MMQ	1262	781	471	238
6	mvq	1461	811	489	255
6	MMQ	1500	937	561	285
7	mvq	1633	867	521	272
7	MMQ	1742	1083	649	330
8	mvq	1733	896	539	280
8	MMQ	1978	1235	737	376
gain @ ne11=8		+14%	+38%	+37%	+34%

Q6_K (this branch keeps mvq for ne11 <= 7, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	359	189	108	56
1	MMQ	358	189	109	56
2	mvq	655	360	207	107
2	MMQ	469	280	169	85
3	mvq	947	521	300	158
3	MMQ	701	417	253	127
4	mvq	1213	655	380	203
4	MMQ	928	551	336	169
5	mvq	1359	719	432	228
5	MMQ	1134	681	416	209
6	mvq	1548	799	474	256
6	MMQ	1361	814	497	249
7	mvq	1685	866	518	272
7	MMQ	1570	945	579	290
8	mvq	1688	928	566	283
8	MMQ	1791	1075	659	331
gain @ ne11=8		+6%	+16%	+16%	+17%

DGX Spark GB10 (sm_121)

GPU: NVIDIA GB10 (Blackwell sm_121, 48 SM, 128 GB LPDDR5 unified - no carveout), driver 610.43.02, Ubuntu 24.04, CUDA 13.3. SM clock pinned at 2400 MHz.

Decode t/s from llama-bench -p 1..8 -n 0 -embd 1 -r 50 on the same --pure single-type ggufs, A-B-B-A ordering, 4 runs per arm. The mvq arm is upstream at the branch base (sm_121 falls through to the default of 8 there); the MMQ arm is the same commit with a one-line local patch forcing MMQ from ne11=2. ne11=1 therefore runs identical code in both arms - that control came in at +0.00% mean, 0.3% worst over 32 blocks.

Only Q2_K crosses within ne11 <= 8 on this hardware, at 6; that is the single entry the sm_121 branch carries. Every other quant type keeps mvq ahead through ne11=8 and falls through to the default.

Q2_K (crosses at ne11 = 6; this branch switches there)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	177	84	51	24
1	MMQ	177	84	51	24
2	mvq	301	140	87	40
2	MMQ	239	117	72	33
3	mvq	457	236	142	66
3	MMQ	354	175	108	49
4	mvq	548	280	168	80
4	MMQ	471	232	142	65
5	mvq	697	331	198	94
5	MMQ	582	288	177	81
6	mvq	746	347	208	99
6	MMQ	696	344	212	97
7	mvq	791	365	218	104
7	MMQ	805	401	246	112
8	mvq	805	370	221	105
8	MMQ	898	456	280	128
gain @ ne11=8		+12%	+23%	+27%	+21%

Q3_K (never crosses within ne11 <= 8; stays on the default of 8)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	122	56	34	16
1	MMQ	121	56	34	16
2	mvq	256	122	75	35
2	MMQ	193	87	55	25
3	mvq	363	168	104	47
3	MMQ	287	130	83	37
4	mvq	472	231	140	65
4	MMQ	382	174	110	49
5	mvq	595	276	168	76
5	MMQ	474	216	136	61
6	mvq	682	318	192	87
6	MMQ	567	258	162	74
7	mvq	771	357	214	100
7	MMQ	659	300	189	85
8	mvq	830	384	228	108
8	MMQ	743	342	215	97
gain @ ne11=8		-10%	-11%	-6%	-10%

Q4_K (never crosses within ne11 <= 8; stays on the default of 8)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	112	50	31	14
1	MMQ	112	50	31	14
2	mvq	213	97	61	27
2	MMQ	186	85	53	25
3	mvq	315	141	88	40
3	MMQ	278	127	79	37
4	mvq	425	194	120	55
4	MMQ	370	169	105	49
5	mvq	517	236	144	66
5	MMQ	457	210	130	60
6	mvq	607	278	170	78
6	MMQ	547	251	155	72
7	mvq	672	311	186	88
7	MMQ	635	292	180	84
8	mvq	721	333	199	94
8	MMQ	718	332	205	95
gain @ ne11=8		-0%	-0%	+3%	+1%

Q5_K (never crosses within ne11 <= 8; stays on the default of 8)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	91	40	25	11
1	MMQ	91	40	25	11
2	mvq	177	79	50	22
2	MMQ	151	69	43	20
3	mvq	269	121	75	34
3	MMQ	225	104	64	30
4	mvq	361	162	100	46
4	MMQ	300	138	86	40
5	mvq	431	195	120	55
5	MMQ	371	171	106	49
6	mvq	517	233	142	66
6	MMQ	443	205	127	59
7	mvq	605	274	166	78
7	MMQ	515	238	148	68
8	mvq	642	295	179	85
8	MMQ	580	271	168	78
gain @ ne11=8		-10%	-8%	-6%	-8%

Q6_K (never crosses within ne11 <= 8; stays on the default of 8)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	78	34	21	10
1	MMQ	79	34	21	10
2	mvq	151	66	41	19
2	MMQ	129	57	35	16
3	mvq	224	99	62	28
3	MMQ	193	85	53	24
4	mvq	297	131	82	37
4	MMQ	257	113	70	33
5	mvq	355	156	97	45
5	MMQ	318	141	87	40
6	mvq	426	188	117	54
6	MMQ	381	168	104	49
7	mvq	491	216	135	62
7	MMQ	443	196	121	56
8	mvq	563	249	154	71
8	MMQ	504	223	138	64
gain @ ne11=8		-10%	-11%	-10%	-10%

RTX 4090 (sm_89)

GPU: NVIDIA RTX 4090 (Ada Lovelace sm_89, 128 SM, 24 GB), driver 610.44, Windows 11, CUDA 12.9. SM clock pinned at 2400 MHz.

Decode t/s from llama-bench -p 1..8 -n 0 -embd 1 -r 50 on --pure single-type ggufs, A-B-B-A ordering. ne11 equals the -p value. mvq = mul_mat_vec_q, MMQ = the tensor-core path. The mvq arm is a stock build (sm_89 previously fell through to the default of 8), the MMQ arm forces MMQ from ne11=2, so ne11=1 runs identical code in both and its delta is the measurement noise floor: mean +0.03%, worst 0.8% over 31 blocks.

Q2_K (this branch keeps mvq for ne11 <= 4, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	410	247	139	74
1	MMQ	411	247	139	73
2	mvq	696	440	249	133
2	MMQ	526	336	201	100
3	mvq	904	570	321	176
3	MMQ	771	493	295	148
4	mvq	1066	658	376	206
4	MMQ	1003	648	385	195
5	mvq	1181	743	428	236
5	MMQ	1215	791	469	240
6	mvq	1298	784	454	245
6	MMQ	1425	930	554	284
7	mvq	1366	802	462	251
7	MMQ	1622	1068	635	327
8	mvq	1481	858	493	270
8	MMQ	1817	1204	713	371
gain @ ne11=8		+23%	+40%	+45%	+37%

Q3_K (this branch keeps mvq for ne11 <= 6, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	327	193	111	58
1	MMQ	327	192	111	58
2	mvq	592	351	201	108
2	MMQ	446	256	156	73
3	mvq	740	442	257	141
3	MMQ	655	377	231	109
4	mvq	917	539	313	174
4	MMQ	856	495	303	143
5	mvq	1098	667	378	211
5	MMQ	1041	604	370	176
6	mvq	1251	744	422	233
6	MMQ	1221	716	438	209
7	mvq	1268	731	429	232
7	MMQ	1399	827	503	242
8	mvq	1342	779	455	247
8	MMQ	1570	935	568	274
gain @ ne11=8		+17%	+20%	+25%	+11%

Q4_K (this branch keeps mvq for ne11 <= 7, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	317	171	100	50
1	MMQ	317	171	100	50
2	mvq	568	313	183	95
2	MMQ	441	251	148	70
3	mvq	819	462	273	139
3	MMQ	646	370	219	104
4	mvq	1019	593	348	180
4	MMQ	841	486	288	137
5	mvq	1156	687	398	210
5	MMQ	1027	597	352	169
6	mvq	1266	736	428	229
6	MMQ	1206	706	417	201
7	mvq	1351	772	448	240
7	MMQ	1385	815	480	232
8	mvq	1420	791	455	243
8	MMQ	1542	920	543	263
gain @ ne11=8		+9%	+16%	+19%	+8%

Q5_K (this branch keeps mvq for ne11 <= 7, MMQ above)

ne11	kernel	Llama-3.2-3B	Llama-3.1-8B	Gemma-4-12B	Qwen3.6-27B
1	mvq	280	146	85	43
1	MMQ	279	146	86	43
2	mvq	500	273	161	81
2	MMQ	408	229	138	65
3	mvq	728	395	235	120
3	MMQ	598	336	204	97
4	mvq	909	518	305	156
4	MMQ	783	444	268	128
5	mvq	1078	620	365	188
5	MMQ	950	545	328	158
6	mvq	1212	708	404	216
6	MMQ	1122	644	387	188
7	mvq	1296	718	414	223
7	MMQ	1282	747	443	217
8	mvq	1368	758	436	234
8	MMQ	1440	844	497	246
gain @ ne11=8		+5%	+11%	+14%	+5%

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: AI was used to partially assist while making the code edits and to understand the contexts of the code base.

praneshgo · 2026-07-24T12:34:15Z

@gaugarg-nv @ORippler can you please review this PR? Thanks.

ORippler

Once the batch goes past 1 the vector kernel becomes compute-bound and MMQ is often faster, but the switch point is fixed at compile time.

If this were the case, we would expect perf gain for all quants & models, not necessarily just a selected few. My feeling is we should look to add a dedicated small-BS kernel instead of tweaking thresholds. @JohannesGaessler mentioned he optimized MMQ only for the large-BS regime. If we want to go for this, imo we need a more complete sweep on HW & quantization types. Other comments:

How did you generate the perf numbers? Please paste the CLI. Also, ensure -embd 1 is set so you generate logits for all elements of the seq
The env variable should be added here https://github.com/ggml-org/llama.cpp/blob/3a19db0185bf7d7517992fd813640a0ff2ca0a23/docs/build.md#runtime-cuda-environmental-variables

praneshgo · 2026-07-27T14:21:26Z

If this were the case, we would expect perf gain for all quants & models, not necessarily just a selected few.

Fair point. I have now updated the PR description to reflect a more accurate description of the PR.

How did you generate the perf numbers? Please paste the CLI. Also, ensure -embd 1 is set so you generate logits for all elements of the seq

The exact commands used are as below. The description now shows updated perf numbers obtained with the commands containing -embd 1. Also using --pure in quantization step now, so as to avoid mixtures of multiple quant types in the results.

CLI

Quantized into necessary quant types using commands similar to this

llama-quantize --pure --allow-requantize base-Q8_0.gguf pure_Q4_K.gguf Q4_K

Clocks pinned so near-crossover deltas aren't smeared by boost/thermal drift:

nvidia-smi -lgc 2400,2400
nvidia-smi -lmc 14001

A/B, identical except the env var (8 = stock, 1 = force MMQ from ne11>=2):

GGML_CUDA_MMVQ_MAX=8 llama-bench -m pure_Q4_K.gguf -ngl 999 -fa 1 -p 1,2,3,4,5,6,7,8 -n 0 -embd 1 -r 50
GGML_CUDA_MMVQ_MAX=1 llama-bench -m pure_Q4_K.gguf -ngl 999 -fa 1 -p 1,2,3,4,5,6,7,8 -n 0 -embd 1 -r 50

nvidia-smi -rgc && nvidia-smi -rmc afterwards.

The env variable should be added here

Removed the runtime env variable in the latest commit and switched to a per-HW switch points that switch based on the quant type. There are existing examples in the modified files, doing the same for AMD HW.

Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover (default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core path, which is faster once quantized decode becomes compute-bound at B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss). The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls back to the default. The override is applied consistently in both the mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged.

…e env var.

gaugarg-nv · 2026-08-04T13:08:58Z

Change looks good to me. We will have to revisit these thresholds if/when MMVQ kernels are optimized.

ORippler · 2026-08-14T17:26:45Z

How would these thresholds change based on #26705, if any?

praneshgo · 2026-08-14T19:57:50Z

How would these thresholds change based on #26705, if any?

On incorporating changes from #26705 and rerunning the experiment (on RTX 5090), I see that the switch points move by 2 for Q4_K and Q5_K quants and remains same for all the remaining quants.

#26705 does not however apply to DGX Spark in its current state, so I do not expect any change in its switch points.

quant	3B	8B	12B	27B	switch point on RTX 5090
Q4_K	5 -> 7	4 -> 6	3 -> 4	4 -> 6	5 -> 7
Q5_K	5 -> 7	4 -> 6	4 -> 5	4 -> 6	5 -> 7
Q2_K	5 -> 5	4 -> 4	3 -> 3	4 -> 5	5 -> 5
Q3_K	5 -> 5	4 -> 4	3 -> 3	4 -> 4	5 -> 5
Q6_K	7 -> 7	6 -> 5	5 -> 6	6 -> 6	7 -> 7
Q4_0	8 -> 8	8 -> 8	8 -> 8	8 -> 8	8 -> 8
Q8_0	8 -> 8	8 -> 8	8 -> 8	8 -> 8	8 -> 8
IQ4_XS	8 -> 8	8 -> 8	8 -> 8	8 -> 8	8 -> 8

I can make the corresponding switch point changes when #26705 PR lands.

ORippler

Please clarify the intention. Currently, MUL_MAT_ID has an independent get_mmvq_mmid_max_batch() helper, which would need to reflect the changes here as well if the intent is for the new crossover points to apply to MoE/MUL_MAT_ID too.

ORippler · 2026-08-18T16:00:55Z

I can make the corresponding switch point changes when #26705 PR lands.

Might make sense to convert this to a stacked PR then, if there is a clear dependence. Not sure if stacked PRs can span cross-forks yet though

praneshgo · 2026-08-18T16:25:36Z

Please clarify the intention. Currently, MUL_MAT_ID has an independent get_mmvq_mmid_max_batch() helper, which would need to reflect the changes here as well if the intent is for the new crossover points to apply to MoE/MUL_MAT_ID too.

I ran the sweep with MoE as well, but it was not giving the benefits as we see in dense models, so the change does not apply to MoE. Specifically, on Qwen3-30B-A3B Q4_K_M on an RTX 5090, forcing MUL_MAT_ID onto MMQ costs 38-62% across batches 2-8 — mvq wins the entire supported range and there is no crossover to find.
Since MoE is not in the scope of this PR, I did not make changes to MUL_MAT_ID. @gaugarg-nv's comment previously (link) and my reply to it also mentions the same.

Might make sense to convert this to a stacked PR then, if there is a clear dependence.

I can explore stacking these PRs if we prefer that, however the intention of both these changes is independent from each other, as well as the changes needed to make due to the switch point dependency are small anyway, so I can also submit the PRs independently.

… decode crossover (ggml-org#26079) * CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover (default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core path, which is faster once quantized decode becomes compute-bound at B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss). The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls back to the default. The override is applied consistently in both the mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged. * Added Blackwell specific switch point, to reduce dependence on runtime env var. * Add per-HW switch point values for DGX Spark and removing runtime env var * Adding switch points for Ada, tested on RTX 4090 * Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE * Reverting an unnecessary conditional * Update ggml/src/ggml-cuda/mmvq.cu --------- Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com> Co-authored-by: Oliver Simons <osimons@nvidia.com>

ggml_cuda_should_use_mmvq() gained per-architecture switch points in ggml-org#26079, but the NVIDIA branches test cc for exact equality (Ada 890, Blackwell 1200, DGX Spark 1210). Ampere (sm_80 cc 800, sm_86 cc 860, sm_87) matches none of them and falls through to ne11 <= MMVQ_MAX_BATCH_SIZE (8). ggml_cuda_mul_mat() tests MMVQ before MMQ and returns on the first match, so on Ampere every batch of 8 or fewer takes MMVQ even though ggml_cuda_should_use_mmq() early-returns true for turing_mma_available(cc). This is most visible in speculative decoding, where the verify batch is ne11 = 1 + n_draft (3-8 for typical draft depths) - squarely inside the range routed to the slower kernel on K-quants. Measured on RTX 3090 (sm_86), Qwen3.8-27B UD-Q4_K_XL, llama-server with --spec-type draft-mtp, greedy, ignore_eos, 8 real prompts, 512 output tokens: n_max verify batch MMVQ (current) MMQ delta 2 3 60.13 t/s 62.87 +4.6% 3 4 58.38 68.31 +17.0% 4 5 56.68 73.96 +30.5% 5 6 51.66 73.82 +42.9% Draft acceptance is unchanged (0.721 -> 0.722 at n_max=2), confirming this is a dispatch effect rather than a change in drafting behaviour. Threshold chosen by llama-bench sweep (-n 0 -p 1..8 -r 12), MMVQ vs MMQ per batch size: MMVQ wins at B=2 (71.8 vs 63.2), ties at B=1 and B=3, and MMQ wins from B=4 up (+14.8% at B=4 rising to +67% at B=8). The crossover is therefore at 3, making 2 the correct threshold - CDNA1 independently uses 2 for Q4_K. Restricted to Q4_K/Q5_K/Q6_K, the types present in the measured model. Q2_K and Q3_K were not measured and keep the default.

GenerelSchwerz · 2026-08-27T19:15:29Z

I'm unsure if this is relevant but for a greedy run of 128 tokens, this PR causes a difference in mtp generation, fixed-seed, on Blackwell cards. I tested this on a 5070ti. The changes are due to floating point differences. Perplexity is essentially the same, and because this was merged I am unsure whether or not this is an issue in the first place.

I noticed that prior to this commit, earlier MTP generation still deviated from non-mtp output (again greedy, fixed-seed testing) but only further into the generation. I can provide reproduction if necessary.

… decode crossover (ggml-org#26079) * CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover (default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core path, which is faster once quantized decode becomes compute-bound at B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss). The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls back to the default. The override is applied consistently in both the mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged. * Added Blackwell specific switch point, to reduce dependence on runtime env var. * Add per-HW switch point values for DGX Spark and removing runtime env var * Adding switch points for Ada, tested on RTX 4090 * Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE * Reverting an unnecessary conditional * Update ggml/src/ggml-cuda/mmvq.cu --------- Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com> Co-authored-by: Oliver Simons <osimons@nvidia.com>

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Jul 24, 2026

gaugarg-nv requested changes Jul 27, 2026

View reviewed changes

Comment thread ggml/src/ggml-cuda/mmvq.cu Outdated

Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated

ORippler reviewed Jul 27, 2026

View reviewed changes

Comment thread ggml/src/ggml-cuda/mmvq.cu Outdated

praneshgo requested a review from gaugarg-nv July 27, 2026 16:19

praneshgo added 2 commits July 29, 2026 21:31

Added Blackwell specific switch point, to reduce dependence on runtim…

f2a3efa

…e env var.

praneshgo force-pushed the pgonegandla/mmvq-mmq-crossover branch from 54f4199 to 1cb7ee6 Compare July 29, 2026 21:35

praneshgo changed the title ~~CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover~~ CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover Jul 29, 2026

praneshgo requested a review from ORippler July 29, 2026 21:49

gaugarg-nv removed model Model specific build Compilation issues labels Jul 30, 2026

praneshgo requested a review from a team as a code owner August 3, 2026 16:41

gaugarg-nv approved these changes Aug 10, 2026

View reviewed changes

ORippler requested changes Aug 18, 2026

View reviewed changes

praneshgo requested a review from ORippler August 18, 2026 16:25

ORippler requested changes Aug 19, 2026

View reviewed changes

Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated

Reverting an unnecessary conditional

614d524

praneshgo requested a review from ORippler August 19, 2026 10:54

ORippler approved these changes Aug 19, 2026

View reviewed changes

Comment thread ggml/src/ggml-cuda/mmvq.cu Outdated

Update ggml/src/ggml-cuda/mmvq.cu

41a70d7

gaugarg-nv approved these changes Aug 20, 2026

View reviewed changes

ORippler merged commit 2b56210 into ggml-org:master Aug 20, 2026
22 of 23 checks passed

claude Bot mentioned this pull request Aug 21, 2026

fix(container): update image ghcr.io/ggml-org/llama.cpp (server-cuda-b10524 ➔ server-cuda-b10548) drag0n141/home-ops#7410

Merged

1 task

This was referenced Aug 24, 2026

vulkan : add GGML_VK_MMV_MAX_COLS override for mul_mat_vec dispatch (#25356) #27652

Open

Vulkan: batched decode throughput cliff at n_tokens=9 on many-expert MoE (fixed 8-token thresholds in MMV dispatch) #25356

Open

praneshgo mentioned this pull request Aug 27, 2026

CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark #26705

Merged

zhihz mentioned this pull request Aug 31, 2026

Feature Request: Add an sm_86 entry to the MMVQ cutoff table (Q4_0 crosses at 7 on A10, worth 9.1%) #28090

Open

4 tasks

cuhawk mentioned this pull request Aug 31, 2026

CUDA: mul_mat is not batch-invariant for IQ quants at n=5 and n=8 on SM120 #28111

Open

jasstrong mentioned this pull request Sep 3, 2026

cuda: shared-quantize cache, residual and elementwise-chain fusions (generic half of #338) TheTom/llama-cpp-turboquant#343

Merged

Navigation Menu

Uh oh!

CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover - #26079

CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover#26079
ORippler merged 7 commits into
ggml-org:masterfrom
praneshgo:pgonegandla/mmvq-mmq-crossover

praneshgo commented Jul 24, 2026 •

edited by ORippler

Loading

Uh oh!

praneshgo commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

ORippler left a comment

Uh oh!

Uh oh!

praneshgo commented Jul 27, 2026 •

edited

Loading

Uh oh!

gaugarg-nv commented Aug 4, 2026

Uh oh!

ORippler commented Aug 14, 2026

Uh oh!

praneshgo commented Aug 14, 2026 •

edited

Loading

Uh oh!

ORippler left a comment

Uh oh!

ORippler commented Aug 18, 2026 •

edited

Loading

Uh oh!

praneshgo commented Aug 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GenerelSchwerz commented Aug 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

praneshgo commented Jul 24, 2026 • edited by ORippler Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

RTX 5090 (sm_120)

Q2_K (this branch keeps mvq for ne11 <= 5, MMQ above)

Q3_K (this branch keeps mvq for ne11 <= 5, MMQ above)

Q4_K (this branch keeps mvq for ne11 <= 5, MMQ above)

Q5_K (this branch keeps mvq for ne11 <= 5, MMQ above)

Q6_K (this branch keeps mvq for ne11 <= 7, MMQ above)

DGX Spark GB10 (sm_121)

Q2_K (crosses at ne11 = 6; this branch switches there)

Q3_K (never crosses within ne11 <= 8; stays on the default of 8)

Q4_K (never crosses within ne11 <= 8; stays on the default of 8)

Q5_K (never crosses within ne11 <= 8; stays on the default of 8)

Q6_K (never crosses within ne11 <= 8; stays on the default of 8)

RTX 4090 (sm_89)

Q2_K (this branch keeps mvq for ne11 <= 4, MMQ above)

Q3_K (this branch keeps mvq for ne11 <= 6, MMQ above)

Q4_K (this branch keeps mvq for ne11 <= 7, MMQ above)

Q5_K (this branch keeps mvq for ne11 <= 7, MMQ above)

Requirements

Uh oh!

praneshgo commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

praneshgo commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaugarg-nv commented Aug 4, 2026

Uh oh!

ORippler commented Aug 14, 2026

Uh oh!

praneshgo commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

ORippler commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

praneshgo commented Aug 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GenerelSchwerz commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

praneshgo commented Jul 24, 2026 •

edited by ORippler

Loading

praneshgo commented Jul 27, 2026 •

edited

Loading

praneshgo commented Aug 14, 2026 •

edited

Loading

ORippler commented Aug 18, 2026 •

edited

Loading

GenerelSchwerz commented Aug 27, 2026 •

edited

Loading