build question newbie

[gilbrotheraway]

hello, i just found this repo and I'm getting incredible performance on my rock5b SBC

i saw some build flags flowing around like
DGGML_NATIVE=1
OpenMP
flax-vectors-something
tinyblas

I'm wondering what they do and if I'm missing any other one to squeeze even more performance

here are some quick numbers i got

k_llama.cpp$
user@rock-5b:/srv/dev-disk-by-uuid-0444eaaf-0405-4373-ad45-74f5ca64d1df/fast/github/ik_llama.cpp$ ./build/bin/llama-bench -m models/bitnet1582b4t-iq2_bn.gguf  -m models/bitnet1582b4t-iq2_bn_r4.gguf -m models/deepcogito_cogito-v1-preview-llama-3B-IQ4_NL.gguf -m models/deepcogito_cogito-v1-preview-llama-3B-Q4_0.gguf -m models/deepcogito_cogito-v1-preview-llama-3B-Q4_K_M.gguf -m models/deepcogito_cogito-v1-preview-llama-3B-Q4_K_S.gguf -p 64,128,256,512,1024 -n 64,128,256,512,1024 -t 4 -rtr 1
| model                          |       size |     params | backend
   | threads | rtr |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --: | ------------: | ---------------: |
============ Repacked 211 tensors
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |          pp64 |    318.86 ± 6.89 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |         pp128 |    238.43 ± 0.36 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |         pp256 |    158.87 ± 0.16 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |         pp512 |     98.19 ± 0.11 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |        pp1024 |     70.59 ± 0.04 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |          tg64 |    161.93 ± 0.04 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |         tg128 |    150.32 ± 0.47 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |         tg256 |    131.80 ± 0.06 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |         tg512 |    106.54 ± 0.03 |
| bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B | CPU        |       4 |   1 |        tg1024 |     74.70 ± 0.08 |
============ Repacked 1 tensors
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |          pp64 |    318.16 ± 0.97 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |         pp128 |    236.25 ± 1.11 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |         pp256 |    157.40 ± 0.17 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |         pp512 |     97.44 ± 0.10 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |        pp1024 |     70.36 ± 0.04 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |          tg64 |    162.03 ± 0.04 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |         tg128 |    150.46 ± 0.04 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |         tg256 |    131.58 ± 1.27 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |         tg512 |    106.38 ± 0.22 |
| bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet | 934.16 MiB |     2.74 B
| CPU        |       4 |   1 |        tg1024 |     74.93 ± 0.03 |
============ Repacked 197 tensors
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |          pp64 |    312.00 ± 0.70 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |         pp128 |    228.23 ± 0.85 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |         pp256 |    150.19 ± 0.27 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |         pp512 |     90.48 ± 0.15 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |        pp1024 |     64.53 ± 0.04 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |          tg64 |    170.81 ± 0.05 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |         tg128 |    155.30 ± 0.03 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |         tg256 |    130.97 ± 0.09 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |         tg512 |     96.60 ± 0.17 |
| llama ?B IQ4_NL - 4.5 bpw      |   1.98 GiB |     3.61 B | CPU
   |       4 |   1 |        tg1024 |     59.32 ± 0.03 |
============ Repacked 194 tensors
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |          pp64 |    142.40 ± 0.18 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         pp128 |    122.02 ± 0.12 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         pp256 |     95.33 ± 0.11 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         pp512 |     67.30 ± 0.08 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |        pp1024 |     51.75 ± 0.03 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |          tg64 |    101.11 ± 0.05 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         tg128 |     95.60 ± 0.01 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         tg256 |     84.97 ± 0.02 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         tg512 |     69.57 ± 0.06 |
| llama ?B Q4_0                  |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |        tg1024 |     48.06 ± 0.03 |
============ Repacked 197 tensors
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |          pp64 |    309.64 ± 0.78 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |         pp128 |    227.22 ± 1.13 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |         pp256 |    149.46 ± 0.34 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |         pp512 |     90.10 ± 0.12 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |        pp1024 |     64.23 ± 0.05 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |          tg64 |    164.21 ± 0.07 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |         tg128 |    149.79 ± 0.07 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |         tg256 |    125.76 ± 0.06 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |         tg512 |     94.72 ± 0.08 |
| llama ?B Q4_K - Medium         |   2.08 GiB |     3.61 B | CPU
   |       4 |   1 |        tg1024 |     58.99 ± 0.07 |
============ Repacked 197 tensors
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |          pp64 |    310.07 ± 1.15 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         pp128 |    226.93 ± 0.88 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         pp256 |    149.10 ± 0.58 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         pp512 |     90.04 ± 0.12 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |        pp1024 |     64.23 ± 0.05 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |          tg64 |    164.18 ± 0.04 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         tg128 |    150.28 ± 0.07 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         tg256 |    125.84 ± 0.04 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |         tg512 |     94.57 ± 0.12 |
| llama ?B Q4_K - Small          |   1.99 GiB |     3.61 B | CPU
   |       4 |   1 |        tg1024 |     58.67 ± 0.05 |
build: c9eec172 (3644)

8B

build/bin/llama-bench -m models/deepcogito_cogito-v1-preview-llama-8B-IQ4_NL.gguf -p 64,128,256,512 -n
64,128,256,512 -t 4 -rtr 1
| model                          |       size |     params | backend
   | threads | rtr |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --: | ------------: | ---------------: |
============ Repacked 225 tensors
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |          pp64 |    183.79 ± 3.47 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |         pp128 |    139.43 ± 0.79 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |         pp256 |     94.39 ± 0.20 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |         pp512 |     57.99 ± 0.04 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |          tg64 |    110.81 ± 0.03 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |         tg128 |    100.95 ± 0.03 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |         tg256 |     85.88 ± 0.10 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU
   |       4 |   1 |         tg512 |     65.49 ± 0.03 |

this is like 2000% improvement

Thank you very much 🙏

[gilbrotheraway]

llama3.2/gemma3 are way worse on tg but same pp performance

now when trying to chat with cogito I'm getting only this, any tips what's going on?

.Form comntSTSTSTSTnt g gSTntntSTSTSTST g g g gSTSTSTSTSTST gnt g g gntnt gST null gST g nullSTnt g g gntST gSTST gST gST g null null gntSTST g gSTnt g gSTSTntntSTSTST g gSTSTSTST g null gSTntSTST g gSTSTnt g gntntSTST null g g gSTSTST nullST g gSTSTntSTntntSTSTST gntST null g null g nullnt
llama_print_timings:        load time =    2283.50 ms
llama_print_timings:      sample time =      16.71 ms /   128 runs   (    0.13 ms per token,  7661.00 tokens per second)
llama_print_timings: prompt eval time =      38.83 ms /     5 tokens (    7.77 ms per token,   128.76 tokens per second)
llama_print_timings:        eval time =    1211.22 ms /   127 runs   (    9.54 ms per token,   104.85 tokens per second)
llama_print_timings:       total time =    1293.44 ms /   132 tokens
Log end

[ikawrakow]

What is cogito?

[saood06]

What is cogito?

I'm assuming he's referring to this: https://huggingface.co/collections/deepcogito/cogito-v1-preview-67eb105721081abe4ce2ee53

[mcm007]

The t/s looks too high for a SBC, maybe the .gguf model is corrupt?

Results on "11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz" with bartowski cogito-v1-preview-llama-8B IQ4_NL which produces good output:

./llama-bench -m /models1/deepcogito_cogito-v1-preview-llama-8B-IQ4_NL.gguf -p 64,128,256,512 -n 64,128,256,512 -t 4 -rtr 1
| model                          |       size |     params | backend    | threads | rtr |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --: | ------------: | ---------------: |
============ Repacked 225 tensors
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |          pp64 |     31.93 ± 1.24 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         pp128 |     23.98 ± 9.64 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         pp256 |     21.24 ± 5.84 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         pp512 |     21.92 ± 2.58 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |          tg64 |      7.97 ± 1.48 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         tg128 |      8.34 ± 0.62 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         tg256 |      8.86 ± 0.02 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         tg512 |      7.67 ± 0.79 |

build: 55fb9c81 (3643)

[saood06]

Also here are the tables from the first post

Click me for Table 1

model	size	params	backend	threads	rtr	test	t/s
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp64	318.86 ± 6.89
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp128	238.43 ± 0.36
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp256	158.87 ± 0.16
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp512	98.19 ± 0.11
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp1024	70.59 ± 0.04
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg64	161.93 ± 0.04
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg128	150.32 ± 0.47
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg256	131.80 ± 0.06
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg512	106.54 ± 0.03
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg1024	74.70 ± 0.08
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp64	318.16 ± 0.97
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp128	236.25 ± 1.11
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp256	157.40 ± 0.17
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp512	97.44 ± 0.10
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp1024	70.36 ± 0.04
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg64	162.03 ± 0.04
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg128	150.46 ± 0.04
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg256	131.58 ± 1.27
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg512	106.38 ± 0.22
bitnet-25 2B IQ2_BN_R4 - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg1024	74.93 ± 0.03
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	pp64	312.00 ± 0.70
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	pp128	228.23 ± 0.85
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	pp256	150.19 ± 0.27
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	pp512	90.48 ± 0.15
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	pp1024	64.53 ± 0.04
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	tg64	170.81 ± 0.05
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	tg128	155.30 ± 0.03
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	tg256	130.97 ± 0.09
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	tg512	96.60 ± 0.17
llama ?B IQ4_NL - 4.5 bpw	1.98 GiB	3.61 B	CPU	4	1	tg1024	59.32 ± 0.03
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	pp64	142.40 ± 0.18
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	pp128	122.02 ± 0.12
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	pp256	95.33 ± 0.11
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	pp512	67.30 ± 0.08
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	pp1024	51.75 ± 0.03
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	tg64	101.11 ± 0.05
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	tg128	95.60 ± 0.01
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	tg256	84.97 ± 0.02
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	tg512	69.57 ± 0.06
llama ?B Q4_0	1.99 GiB	3.61 B	CPU	4	1	tg1024	48.06 ± 0.03
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	pp64	309.64 ± 0.78
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	pp128	227.22 ± 1.13
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	pp256	149.46 ± 0.34
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	pp512	90.10 ± 0.12
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	pp1024	64.23 ± 0.05
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	tg64	164.21 ± 0.07
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	tg128	149.79 ± 0.07
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	tg256	125.76 ± 0.06
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	tg512	94.72 ± 0.08
llama ?B Q4_K - Medium	2.08 GiB	3.61 B	CPU	4	1	tg1024	58.99 ± 0.07
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	pp64	310.07 ± 1.15
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	pp128	226.93 ± 0.88
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	pp256	149.10 ± 0.58
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	pp512	90.04 ± 0.12
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	pp1024	64.23 ± 0.05
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	tg64	164.18 ± 0.04
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	tg128	150.28 ± 0.07
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	tg256	125.84 ± 0.04
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	tg512	94.57 ± 0.12
llama ?B Q4_K - Small	1.99 GiB	3.61 B	CPU	4	1	tg1024	58.67 ± 0.05

Click me for Table 2

model	size	params	backend	threads	rtr	test	t/s
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	pp64	183.79 ± 3.47
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	pp128	139.43 ± 0.79
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	pp256	94.39 ± 0.20
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	pp512	57.99 ± 0.04
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	tg64	110.81 ± 0.03
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	tg128	100.95 ± 0.03
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	tg256	85.88 ± 0.10
llama 8B IQ4_NL - 4.5 bpw	4.35 GiB	8.03 B	CPU	4	1	tg512	65.49 ± 0.03

[gilbrotheraway]

The t/s looks too high for a SBC, maybe the .gguf model is corrupt?

Results on "11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz" with bartowski cogito-v1-preview-llama-8B IQ4_NL which produces good output:

./llama-bench -m /models1/deepcogito_cogito-v1-preview-llama-8B-IQ4_NL.gguf -p 64,128,256,512 -n 64,128,256,512 -t 4 -rtr 1
| model                          |       size |     params | backend    | threads | rtr |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --: | ------------: | ---------------: |
============ Repacked 225 tensors
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |          pp64 |     31.93 ± 1.24 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         pp128 |     23.98 ± 9.64 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         pp256 |     21.24 ± 5.84 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         pp512 |     21.92 ± 2.58 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |          tg64 |      7.97 ± 1.48 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         tg128 |      8.34 ± 0.62 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         tg256 |      8.86 ± 0.02 |
| llama 8B IQ4_NL - 4.5 bpw      |   4.35 GiB |     8.03 B | CPU        |       4 |   1 |         tg512 |      7.67 ± 0.79 |

build: 55fb9c81 (3643)

I'm using the same one from bartowski, if you need me to test any models just say I'm on 300mbit connection

ive tested about a dozen models they all show crazy performance no idea why
I'm new to these things do you set any tokenizer/template/whatever?

all i did was
build (cmake w/ no extra flags)
download model/config files via wget
run/bench

[gilbrotheraway]

yeah, they all output gibberish

main llama.cpp works no problem

[ikawrakow]

I'm not familiar with this space, so had to look up what "rock 5b" is. According to this it has one Cortex-A76 and one Cortex-A55 CPU. For this the performance numbers look too high. Which means that most likely the iqk matrix multiplications that I have added do not get invoked, and it falls back to stock ggml implementation (ggml is the inference library behind llama.cpp). Most likely something goes wrong there, which leads to crazy performance and gibberish output. I did try to maintain this use case (the fallback to stock ggml) in a working condition for a while, but I think it is broken now.

I assume you are running Linux on this board? Can you do cat /proc/cpuinfo?

[saood06]

I'm not familiar with this space, so had to look up what "rock 5b" is. According to this it has one Cortex-A76 and one Cortex-A55 CPU. For this the performance numbers look too high.

It has eight cores in total, "Quad Cortex®-A76 @ 2.2~2.4GHz and a Quad Cortex®-A55 @ 1.8GHz" from what I think is the official product page. But even with that the performance still seems too high.

[gilbrotheraway]

I assume you are running Linux on this board? Can you do cat /proc/cpuinfo?

yup, that's the same soc orange pi 5+ uses, which I've seen somebody running here in this repo

processor       : 6
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x4
CPU part        : 0xd0b
CPU revision    : 0
processor       : 7
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x4
CPU part        : 0xd0b
CPU revision    : 0

i get same performance on q4km and iq4nl are you sure it has to do with iqk mm?

[gilbrotheraway]

But even with that the performance still seems too high.

yeah i tried bitnet.cpp first = 0.5t/s
then came here expecting like 10-15t/s @8B
oh boy

but this little board is pretty capable I'm running 30 containers load average is ~0.5
NPU does 16x simultaneous 1080p@30fps transcodes at 18w
fun board

maybe it's some dependency missing? but main runs normally...

[ikawrakow]

i get same performance on q4km and iq4nl are you sure it has to do with iqk mm?

You are getting 110 t/s for tg64 with LLaMA-8B. This is GPU territory (I get 130 t/s on my RTX-4080, 55 t/s on the M2-Max 30-core GPU). So, most likely the matrix multiplications don't get done at all.

The CPU flags look completely unfamiliar, so I cannot deduce from there if the NEON extensions get automatically enabled (required for this repo to work correctly on ARM CPUs).

[gilbrotheraway]

i get same performance on q4km and iq4nl are you sure it has to do with iqk mm?

You are getting 110 t/s for tg64 with LLaMA-8B. This is GPU territory (I get 130 t/s on my RTX-4080, 55 t/s on the M2-Max 30-core GPU). So, most likely the matrix multiplications don't get done at all.

The CPU flags look completely unfamiliar, so I cannot deduce from there if the NEON extensions get automatically enabled (required for this repo to work correctly on ARM CPUs).

they do

[ikawrakow]

So, I'm finding that the asimddp CPU feature that you have should enable __ARM_FEATURE_DOTPROD. With that things should work correctly.

What is the compiler being used?

[gilbrotheraway]

So, I'm finding that the asimddp CPU feature that you have should enable __ARM_FEATURE_DOTPROD. With that things should work correctly.

What is the compiler being used?

gcc-12.2
i noticed bitnet.cpp uses clang-19 would that help?

DOTPROD shows enabled on main

could the llamafile feature interfere?