cuda: unblock mmq for MoE on sm_60

[dfriehs]

Overview

Tesla P100's get a nice (~20%) pp speedup by using mmq even without native dp4a. For -sm tensor configurations with multiple cards, this increases to around a +100% pp speedup (for 2 cards, possibly more with more cards). Select K-quants are excluded as they lead to a drop in performance with mmq: Q2_K, Q4_K, Q5_K, Q6_K.

This PR is similar to, but inverse of, it's inspiration: #24588, where gfx900 had mmq always turned on even without dp4a, and kept mmq for MoE. sm_60 will now use mmq, but only for MoE.

This might be relevant for even older NVidia hardware as well, but I only have access to a 4GB Quadro that might not be a good reference point (and no models that come to mind). For now it's gated to only the P100.

Additional information

Tests are on two P100 cards (powerlimited to 150W, connected via PCIe 3.0 x8). See comments below for tests for each quantization type (exps quantized to type tested, all other tensors Q4_0).

I tested single card performance for quantization types that fit on one card, and the speed-up closely matches -sm layer improvements.

original test, prefer results in comments below

`GGML_CUDA_P2P=1 llama-batched-bench -dev CUDA0,CUDA1 -sm tensor -no-kvu -ngl all -fa on -fit off -ub 2048 -ctk q8_0 -ctv q8_0 -npp 0,128,512,1024,2048,4096,32768,65536 -ntg 128 -npl 1 -m Qwen3.6-35B-A3B-Q4.gguf`

master:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.887	67.82	1.887	67.82
128	128	1	256	0.727	176.15	1.861	68.80	2.587	98.95
512	128	1	640	1.769	289.43	1.877	68.21	3.646	175.56
1024	128	1	1152	2.684	381.54	1.875	68.27	4.559	252.70
2048	128	1	2176	4.282	478.30	1.878	68.14	6.160	353.23
4096	128	1	4224	8.592	476.71	1.898	67.46	10.490	402.68
32768	128	1	32896	79.078	414.38	2.228	57.46	81.305	404.60
65536	128	1	65664	176.232	371.87	2.605	49.14	178.837	367.17

this pr:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.908	67.08	1.908	67.08
128	128	1	256	0.359	356.78	1.869	68.48	2.228	114.91
512	128	1	640	0.645	794.20	1.887	67.84	2.531	252.82
1024	128	1	1152	0.983	1041.43	1.889	67.76	2.872	401.09
2048	128	1	2176	1.660	1234.10	1.889	67.77	3.548	613.25
4096	128	1	4224	3.352	1222.08	1.906	67.16	5.258	803.42
32768	128	1	32896	34.815	941.19	2.238	57.21	37.053	887.81
65536	128	1	65664	88.220	742.87	2.614	48.97	90.834	722.90

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: No.

[am17an]

I have not tested this since I don't have a Pascal device but it looks to okay to me

[mb8565]

I tested this PR and recorded the prefill results on 2x Tesla P100, stock master 0278d8362, this PR's change behind an env flag so the same binary does both arms. Qwen3.6-35B-A3B-UD-Q4_K_XL, GGML_CUDA_P2P=1, llama-bench -p 2048 -r 3.

config	MMQ off	MMQ on	delta
stock default	365.52	219.18	-40%
-ub 2048 alone	521.24	301.55	-42%
-fa on + q8_0 KV	367.94	211.67	-42%
your combo minus -sm tensor	526.83	293.44	-44%
-sm tensor + -fa on	247.40	295.99	+20%
your combo (with -sm tensor)	370.93	539.36	+45%

Best config vs best config, on a 5-repeat re-run: +2.3%.

I used llama-bench rather than llama-batched-bench -fit off -no-kvu, and UD-Q4_K_XL rather than plain Q4. I reproduced the sign of your result, not its magnitude.

The -sm dependence seemed worth flagging.

[dfriehs]

Yeah, reproducing with Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf I'm getting horrible numbers in comparison as well. I can only guess that the difference is because the P100 can't handle some datatypes for mmq that well; my benchmarks above used a custom quant with IQ4_NL for all experts, while the UD quant seems to use Q6_K, Q5_K and Q4_K.

I guess I'll convert this to a draft until I can get a closer look at which quants make sense for the P100 to support for mmq.

[dfriehs]

I ran some tests with exps quantized to most types, and there are some that get a lot worse with mmq, with others more than doubling. For most types, both -sm layer/tensor improve, although layer benefits less. I don't think the small improvement for -sm tensor + mmq for select k-quants is enough to special case them. Also exciting is that with mmq enabled, -sm tensor has both better tg and pp than -sm layer for at least Qwen 3.6 35B-A3B.

quant	t/s layer	t/s tensor	t/s layer mmq	t/s tensor mmq
Q1_0	628.56	514.98	757.83	1261.90
Q2_0	573.81	477.82	737.18	1238.31
Q4_0	583.74	477.29	722.34	1219.73
Q4_1	607.81	478.50	713.18	1205.11
Q5_0	560.34	464.43	713.48	1198.58
Q5_1	580.79	463.78	709.72	1196.90
IQ1_S	582.21	466.24	717.19	1213.75
IQ1_M	551.95	459.62	550.09	456.44
IQ2_XXS	551.02	425.54	725.02	1214.13
IQ2_XS	566.18	454.69	713.14	1198.41
IQ2_S	565.08	450.11	710.47	1191.67
IQ3_XXS	563.41	449.61	717.54	1203.85
IQ3_S	565.07	448.96	712.77	1195.57
IQ4_XS	597.03	471.74	725.17	1211.36
IQ4_NL	605.17	446.09	727.28	1217.19
Q2_K	608.29	479.73	225.35	422.11
Q3_K	588.45	464.43	684.71	1153.93
Q4_K	563.45	465.14	324.63	573.27
Q5_K	591.53	469.16	272.03	499.24
Q6_K	586.72	480.66	307.83	565.17

scripts used for testing

_quants=(
  Q1_0 Q2_0 Q4_0 Q4_1 Q5_0 Q5_1
  IQ1_S IQ1_M IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_XS IQ4_NL
  Q2_K Q3_K Q4_K Q5_K Q6_K
)

for q in "${_quants[@]}"
do
  llama-quantize --tensor-type "exps=$q" --tensor-type "blk|embd|out=q4_0" --imatrix Qwen3.6-35B-A3B-imatrix.gguf Qwen3.6-35B-A3B-BF16.gguf "qwen35moe-$q.gguf" "$q"
done

export CUDA_VISIBLE_DEVICES=0,1

_quants=(
  Q1_0 Q2_0 Q4_0 Q4_1 Q5_0 Q5_1
  IQ1_S IQ1_M IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_XS IQ4_NL
  Q2_K Q3_K Q4_K Q5_K Q6_K
)

for q in "${_quants[@]}"
do
  llama-bench -m "qwen35moe-$q.gguf" -lm dio -ngl 999 -ub 2048 -p 2048 -n 32 -sm layer,tensor
done

I'm planning to switch on quant type for the P100 in ggml_cuda_should_use_mmq for now (will write the code and push later today), but it might be interesting to check out why exactly some of the quants get so much worse in the future. Thank you for testing, @mb8565.

[dfriehs]

I've finished testing all quants that support mmq right now, here's the last few ones:

quant	t/s layer	t/s tensor	t/s layer mmq	t/s tensor mmq
Q8_0 *	463.80	274.58	517.94	382.85
Q8_0 **	644.95	291.30	750.07	1523.54
MXFP4	612.46	476.98	731.80	1228.07
NVFP4	543.62	399.80	726.24	1217.29

*: -ngl 32
**: 4 Tesla P100 instead of 2

Apart from IQ1_M, which has no mmq (is this an oversight?), the only quants that do not benefit from mmq are Q2_K, Q4_K, Q5_K and Q6_K. Those quants will not use mmq on the P100 right now, but there's a TODO in there to check what the issue might be.

[dfriehs]

@am17an would it be alright to ask for a re-review? If you need me to run more tests I can do so as well.

I'm planning to look into tuning sm_60 mmq configs in a future PR unless you want it part of this one. K-quant changes will need to be a separate PR either way unless the slowdowns are just disastrous tuning.

[am17an]

I don't have hardware to test this, so I would rely on other people's reports on this patch.

[Kenzu]

I did a small test with Ornith-1.0-35B-UD-IQ4_NL.gguf, My pp stats in llama-server went from ~300 (layer, tensor was ~200) to ~900 (tensor). Token/s is close to the same 50-60. Very happy with this. But it did nothing on Q5_XL.
This is on a HP Z440 Xeon 2695v3 + 2x P100 16GB.