cuda: extract Q1_0 elements via __byte_perm

[dfriehs]

Overview

Unpack Q1_0 elements via __byte_perm, leading to a nice increase in t/s (+5-10%) and a modest one for pp (+1-2.5%).

Tagging @khosravipasha, in case you are interested in this one as well.

Additional information

test-backend-ops test passes before and with 8373d2bbc0f96ba6940a81d3f3c385823a69b5ae, and KL divergence is 0 between the two. n=4 shows a slowdown in test-backend-ops perf, but llama-batched-bench shows increased t/s even for B=4. Maybe there is some tuning to update?

I'm not able to test HIP/ROCm or MUSA. If either don't support __byte_perm or slow down I will add a fallback path.

test-backend-ops perf

before 8373d2bbc0f96ba6940a81d3f3c385823a69b5ae:

Backend 1/2: CUDA0
  Device description: NVIDIA GeForce RTX 3090
  Device memory: 24120 MB (23374 MB free)

  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   443040 runs -  22.60 us/run - 117.44 MFLOP/run -  5.20 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   386382 runs -  25.89 us/run - 234.88 MFLOP/run -  9.07 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   330292 runs -  30.30 us/run - 352.32 MFLOP/run - 11.63 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   273705 runs -  36.54 us/run - 469.76 MFLOP/run - 12.85 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   225378 runs -  44.39 us/run - 587.20 MFLOP/run - 13.23 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   139849 runs -  71.55 us/run - 939.52 MFLOP/run - 13.13 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):  16034 runs - 623.73 us/run -  60.13 GFLOP/run - 96.40 TFLOPS
  Backend CUDA0: OK

with 8373d2bbc0f96ba6940a81d3f3c385823a69b5ae:

Backend 1/2: CUDA0
  Device description: NVIDIA GeForce RTX 3090
  Device memory: 24120 MB (23250 MB free)

  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   511200 runs -  19.58 us/run - 117.44 MFLOP/run -  6.00 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   482232 runs -  20.74 us/run - 234.88 MFLOP/run - 11.33 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   349036 runs -  28.65 us/run - 352.32 MFLOP/run - 12.30 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   253896 runs -  39.40 us/run - 469.76 MFLOP/run - 11.92 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   239400 runs -  41.78 us/run - 587.20 MFLOP/run - 14.06 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   148302 runs -  67.43 us/run - 939.52 MFLOP/run - 13.93 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):  16404 runs - 609.66 us/run -  60.13 GFLOP/run - 98.63 TFLOPS
  Backend CUDA0: OK

llama-batched-bench

merged output of
llama-batched-bench -no-kvu -ngl all -fit off -ub 1024 -npp 0,256,4096 -ntg 256 -npl 1,2,4,8 -m Bonsai-8B-Q1_0.gguf
and
llama-batched-bench -no-kvu -ngl all -fit off -ub 1024 -npp 16384,32768 -ntg 256 -npl 1 -m Bonsai-8B-Q1_0.gguf

before 8373d2bbc0f96ba6940a81d3f3c385823a69b5ae:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	1.141	224.31	1.141	224.31
0	256	2	512	0.000	0.00	1.401	365.40	1.401	365.40
0	256	4	1024	0.000	0.00	2.015	508.09	2.015	508.09
0	256	8	2048	0.000	0.00	2.786	735.10	2.786	735.10
256	256	1	512	0.051	5032.44	1.136	225.40	1.187	431.47
256	256	2	1024	0.094	5455.92	1.435	356.72	1.529	669.65
256	256	4	2048	0.181	5663.62	2.089	490.15	2.270	902.22
256	256	8	4096	0.345	5933.45	2.886	709.70	3.231	1267.77
4096	256	1	4352	0.740	5532.66	1.317	194.31	2.058	2114.87
4096	256	2	8704	1.464	5595.90	1.779	287.73	3.243	2683.61
4096	256	4	17408	2.917	5616.59	2.777	368.71	5.694	3057.07
4096	256	8	34816	5.826	5624.60	4.214	486.05	10.039	3467.93
16384	256	1	16640	3.708	4419.06	1.832	139.74	5.540	3003.85
32768	256	1	33024	9.600	3413.21	2.535	100.98	12.135	2721.28

with 8373d2bbc0f96ba6940a81d3f3c385823a69b5ae:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	1.017	251.66	1.017	251.66
0	256	2	512	0.000	0.00	1.118	457.92	1.118	457.92
0	256	4	1024	0.000	0.00	1.809	566.01	1.809	566.01
0	256	8	2048	0.000	0.00	2.564	798.83	2.564	798.83
256	256	1	512	0.050	5167.54	1.008	254.04	1.057	484.28
256	256	2	1024	0.092	5577.10	1.153	444.08	1.245	822.66
256	256	4	2048	0.176	5829.71	1.884	543.52	2.060	994.34
256	256	8	4096	0.338	6066.06	2.664	768.90	3.001	1364.81
4096	256	1	4352	0.722	5674.43	1.188	215.49	1.910	2278.72
4096	256	2	8704	1.427	5740.23	1.488	344.10	2.915	2985.89
4096	256	4	17408	2.842	5765.03	2.563	399.55	5.405	3220.80
4096	256	8	34816	5.676	5772.59	3.974	515.39	9.650	3607.81
16384	256	1	16640	3.647	4492.57	1.704	150.23	5.351	3109.74
32768	256	1	33024	9.476	3458.00	2.407	106.37	11.883	2779.16

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: No.

[dfriehs]

Same benchmarks for Bonsai-27B-Q1_0.gguf:

llama-batched-bench (master, bf2c86d)

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	3.475	73.67	3.475	73.67
0	256	2	512	0.000	0.00	4.499	113.80	4.499	113.80
0	256	4	1024	0.000	0.00	6.781	151.01	6.781	151.01
0	256	8	2048	0.000	0.00	11.190	183.01	11.190	183.01
256	256	1	512	0.178	1437.26	3.460	73.99	3.638	140.73
256	256	2	1024	0.343	1491.77	4.538	112.84	4.881	209.80
256	256	4	2048	0.671	1526.89	6.855	149.38	7.526	272.14
256	256	8	4096	1.329	1540.83	11.310	181.09	12.639	324.08
4096	256	1	4352	2.701	1516.59	3.556	71.99	6.257	695.58
4096	256	2	8704	5.379	1523.05	4.718	108.52	10.097	862.05
4096	256	4	17408	10.745	1524.78	7.206	142.10	17.951	969.74
4096	256	8	34816	21.591	1517.66	12.002	170.64	33.593	1036.40
16384	256	1	16640	11.322	1447.06	3.770	67.91	15.092	1102.56
32768	256	1	33024	24.676	1327.92	4.098	62.48	28.774	1147.71

llama-batched-bench (this PR, 19e2181)

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	3.040	84.20	3.040	84.20
0	256	2	512	0.000	0.00	3.558	143.91	3.558	143.91
0	256	4	1024	0.000	0.00	6.402	159.96	6.402	159.96
0	256	8	2048	0.000	0.00	10.254	199.73	10.254	199.73
256	256	1	512	0.174	1475.15	3.039	84.24	3.212	159.38
256	256	2	1024	0.337	1520.76	3.600	142.23	3.937	260.13
256	256	4	2048	0.655	1563.09	6.463	158.44	7.118	287.72
256	256	8	4096	1.297	1578.49	10.356	197.76	11.653	351.48
4096	256	1	4352	2.641	1551.18	3.134	81.68	5.775	753.63
4096	256	2	8704	5.277	1552.54	3.776	135.60	9.052	961.51
4096	256	4	17408	10.550	1552.97	6.811	150.34	17.361	1002.69
4096	256	8	34816	21.158	1548.69	11.074	184.94	32.232	1080.16
16384	256	1	16640	11.160	1468.16	3.373	75.90	14.533	1145.01
32768	256	1	33024	24.362	1345.07	3.693	69.31	28.055	1177.12

[Green-Sky]

Wait what, qwen3.6 27b 1bit model??

[khosravipasha]

Just launched today :)
https://huggingface.co/collections/prism-ml/bonsai-27b

[Green-Sky]

Device 0: NVIDIA GeForce RTX 2070, compute capability 7.5, VMM: yes, VRAM: 7777 MiB

$ llama-batched-bench -m models/Bonsai-27B-Q1_0.gguf -ctk q8_0 -ctv q8_0 -no-kvu -ngl all -fit off -c 4096 -ub 1024 -npp 0,256,4096 -ntg 256 -npl 1,2,4

llama_batched_bench: n_kv_max = 4096, n_batch = 2048, n_ubatch = 1024, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -2, n_threads = 12, n_threads_batch = 12

without pr (12127de)

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	11.227	22.80	11.227	22.80
0	256	2	512	0.000	0.00	14.127	36.24	14.127	36.24
0	256	4	1024	0.000	0.00	24.711	41.44	24.711	41.44
256	256	1	512	0.676	378.72	11.180	22.90	11.856	43.18
256	256	2	1024	1.312	390.17	14.202	36.05	15.515	66.00
256	256	4	2048	2.516	407.04	24.940	41.06	27.456	74.59

with pr

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	10.271	24.92	10.271	24.92
0	256	2	512	0.000	0.00	13.049	39.24	13.049	39.24
0	256	4	1024	0.000	0.00	23.467	43.64	23.467	43.64
256	256	1	512	0.665	384.90	10.277	24.91	10.943	46.79
256	256	2	1024	1.283	399.04	13.133	38.99	14.416	71.03
256	256	4	2048	2.451	417.71	23.606	43.38	26.057	78.60

Small but noticeable uplift for both pp and tg.

[JohannesGaessler]

This seems logically correct to me though I haven't actually tested it for correctness or performance. @khosravipasha I'll leave whether or not this should be merged at your discretion.

[dfriehs]

Is there anything you are concerned about regarding correctness? I can offer a small script that should check all 16 bit values exhaustively for equivalent unpacking.

Needs torch, numpy and ninja installed:
https://gist.github.com/dfriehs/047ef379247ea163e0124e8faed937fe

[khosravipasha]

Overall looks good to us. I am running our KL logits tests and speed test to be sure. Will share numbers in a bit. Don't see anything concerning.

[khosravipasha]

The logits seems to match perfect and see around 5 percent faster token generation on L40S depending on model size so safe to merge.