Test various AI clustering setups on 10 node Pi 5 cluster

[geerlingguy]

Want to document any oddities using the CPU cores on a small Pi CM5 cluster I'm building... also any strangeness running this stuff on Arm and Pi OS (Debian).

[geerlingguy]

I can't get Vulkan going, even after installing mesa-vulkan-drivers, and it's likely related to ggml-org/llama.cpp#9801

I've also found Pi OS Lite doesn't include a couple dependencies llama.cpp enjoys having out of the box, so I've added a new setting in config.yml for easy package install for missing packages:

llama_extra_packages: ['ccache', 'libcurl4-openssl-dev']

With that, llama.cpp compiles, it just takes a while on the Pi :)

[geerlingguy]

This will also only be a 9 node cluster, because one of the nodes seems to enjoy dying on me. Not sure why.

[geerlingguy]

Llama-3.2-3B-Instruct-Q4_K_M.gguf - Single Node

Testing a single node (build: 792b44f2 (6319)):

model	size	params	backend	ngl	fa	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	RPC	99	1	pp512	15.08 ± 0.00
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	RPC	99	1	tg128	6.07 ± 0.01
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	RPC	99	1	pp512+tg128	11.11 ± 0.00

[geerlingguy]

Llama-3.3-70B-Instruct-Q4_K_M.gguf - Cluster

model	size	params	backend	ngl	fa	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	RPC	99	1	pp512	0.65 ± 0.00

...and it crashed with:

/opt/llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp:592: Remote RPC server crashed or returned malformed response

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x00007fffa478829c in __GI___wait4 (pid=15187, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#0  0x00007fffa478829c in __GI___wait4 (pid=15187, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x00007fffa4ba3e54 in ggml_print_backtrace () from /opt/llama.cpp/build/bin/libggml-base.so
#2  0x00007fffa4ba3f9c in ggml_abort () from /opt/llama.cpp/build/bin/libggml-base.so
#3  0x00007fffa4556ea8 in ggml_backend_rpc_buffer_set_tensor(ggml_backend_buffer*, ggml_tensor*, void const*, unsigned long, unsigned long) () from /opt/llama.cpp/build/bin/libggml-rpc.so
#4  0x00007fffa4bb6d58 in ggml_backend_tensor_copy () from /opt/llama.cpp/build/bin/libggml-base.so
#5  0x00007fffa4bbace8 in ggml_backend_sched_graph_compute_async () from /opt/llama.cpp/build/bin/libggml-base.so
#6  0x00007fffa4cf0184 in llama_context::graph_compute(ggml_cgraph*, bool) () from /opt/llama.cpp/build/bin/libllama.so
#7  0x00007fffa4cf1830 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /opt/llama.cpp/build/bin/libllama.so
#8  0x00007fffa4cf6cfc in llama_context::decode(llama_batch const&) () from /opt/llama.cpp/build/bin/libllama.so
#9  0x00007fffa4cf7af0 in llama_decode () from /opt/llama.cpp/build/bin/libllama.so
#10 0x000055559b6b063c in main ()
[Inferior 1 (process 15020) detached]

I'm going to run it again with:

llama_bench_opts: "-n 16 -p 32 -fa 1 -r 1"

[geerlingguy]

model	size	params	backend	ngl	fa	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	RPC	99	1	pp32	0.66 ± 0.00
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	RPC	99	1	tg16	0.28 ± 0.00

Not a speed daemon, but it ran, at least.

It used 57.4W of power during the run, for the entire cluster. (Idle power is around 48W).

[geerlingguy]

Ran into #10 so filed that issue.

[geerlingguy]

Distributed Llama, running the Llama 3.1 70B model available for it:

$ ./dllama inference --model models/llama3_3_70b_instruct_q40/dllama_model_llama3_3_70b_instruct_q40.m         --tokenizer models/llama3_3_70b_instruct_q40/dllama_tokenizer_llama3_3_70b_instruct_q40.t         --buffer-float-type q80 --prompt "Hello world" --steps 256 --max-seq-len 4096         --nthreads 4 --workers 10.0.2.237:9999 10.0.2.206:9999 10.0.2.241:9999 10.0.2.245:9999 10.0.2.236:9999 10.0.2.232:9999 10.0.2.221:9999
📄 AddBos: 0
📄 BosId: 128000 (<|begin_of_text|>)
📄 EosId: 128001 (<|end_of_text|>) 128009 (<|eot_id|>) 
📄 RegularVocabSize: 128000
📄 SpecialVocabSize: 256
💡 Arch: Llama
💡 HiddenAct: Silu
💡 Dim: 8192
💡 HeadDim: 128
💡 QDim: 8192
💡 KvDim: 1024
💡 HiddenDim: 28672
💡 VocabSize: 128256
💡 nLayers: 80
💡 nHeads: 64
💡 nKvHeads: 8
💡 OrigSeqLen: 131072
💡 SeqLen: 4096
💡 NormEpsilon: 0.000010
💡 RopeType: Llama3.1
💡 RopeTheta: 500000
💡 RopeScaling: f=8.0, l=1.0, h=4.0, o=8192
📀 RequiredMemory: 9038 MB
⭕ Socket[0]: connecting to 10.0.2.237:9999 worker
⭕ Socket[0]: connected
⭕ Socket[1]: connecting to 10.0.2.206:9999 worker
⭕ Socket[1]: connected
⭕ Socket[2]: connecting to 10.0.2.241:9999 worker
⭕ Socket[2]: connected
⭕ Socket[3]: connecting to 10.0.2.245:9999 worker
⭕ Socket[3]: connected
⭕ Socket[4]: connecting to 10.0.2.236:9999 worker
⭕ Socket[4]: connected
⭕ Socket[5]: connecting to 10.0.2.232:9999 worker
⭕ Socket[5]: connected
⭕ Socket[6]: connecting to 10.0.2.221:9999 worker
⭕ Socket[6]: connected
⭕ Network is initialized
🧠 CPU: neon dotprod fp16
💿 Loading weights...
💿 Weights loaded
🚁 Network is in non-blocking mode
Hello world
🔷️ Eval  526 ms Sync  662 ms | Sent  9744 kB Recv  9958 kB | (1 tokens)
🔶 Pred  481 ms Sync  685 ms | Sent  9744 kB Recv  9958 kB |  also
🔶 Pred  481 ms Sync  685 ms | Sent  9744 kB Recv  9958 kB |  enable
🔶 Pred  486 ms Sync  685 ms | Sent  9744 kB Recv  9958 kB |  website
...
Evaluation
   nBatches: 32
    nTokens: 1
   tokens/s: 0.84 (1189.46 ms/tok)
Prediction
    nTokens: 255
   tokens/s: 0.85 (1173.78 ms/tok)
⭕ Network is closed

The cluster hit around 70W.

[geerlingguy]

Trying out Exo, I was able to get the 9 nodes seeing each other running exo, but I then ran into this unanswered issue: exo-explore/exo#655

[geerlingguy]

Now with Exo, I'm getting:

No such file or directory: 'clang'

See: exo-explore/exo#672

I installed clang on all the nodes, then tried again, and wound up with:

Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): <AioRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Unexpected <class 'subprocess.CalledProcessError'>: Command '['clang', '-c', '-x', 'c', '-march=native', '--target=aarch64-none-unknown-elf', '-O2', '-fPIC', '-ffreestanding', '-fno-math-errno', '-nostdlib', '-', '-o', '-']' returned non-zero exit status 1."
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Unexpected <class \'subprocess.CalledProcessError\'>: Command \'[\'clang\', \'-c\', \'-x\', \'c\', \'-march=native\', \'--target=aarch64-none-unknown-elf\', \'-O2\', \'-fPIC\', \'-ffreestanding\', \'-fno-math-errno\', \'-nostdlib\', \'-\', \'-o\', \'-\']\' returned non-zero exit status 1.", grpc_status:2, created_time:"2025-08-29T17:30:18.177163161-05:00"}"

Going to give up on Exo working. It was so close :P