Benchmark Framework Desktop Mainboard and 4-node cluster

[geerlingguy]

Testing the Framework Desktop - AMD Ryzen AI Max+ 395 with Radeon 8090S. (Four pre-production units were shipped to me for local cluster testing).

Single Node configuration (128 GB RAM):

Cluster configuration (4x Mainboard - 512 GB RAM)

Initial tests (above) were run using 2.5 Gbps Ethernet interconnect.

I later changed to 5 Gbps using a NICGIGA switch, and racked it in a black T1 mini rack shipped by DeskPi, along with four of their currently-in-prototype Framework Desktop mini rack trays. Mini rack build showcase here.

I also tested Thunderbolt node-to-node interconnects, but could only get 10 Gbps over TB4 using thunderbolt0/thunderbolt1 interfaces).

For more benchmarks (focusing on CPU, GPU, disk, net, etc.), see:

• I clustered four Framework Mainboards to test huge LLMs
• sbc-reviews: Framework Desktop.

All my automation for testing is in the Beowulf AI Cluster repo.

[geerlingguy]

IMPORTANT NOTE

Please scroll down past this first comment, I did not just test with Ollama and the CPU lol

---

First set of results, using ollama defaults, which seemed to only use CPU cores and not GPU:

Llama - CPU

Running benchmark 3 times using model: llama3.2:3b (152W)

Run	Eval Rate (Tokens/Second)
1	46.21 tokens/s
2	45.23 tokens/s
3	44.64 tokens/s
Average Eval Rate	45.36 tokens/second

Running benchmark 3 times using model: llama3.1:8b (142W)

Run	Eval Rate (Tokens/Second)
1	21.02 tokens/s
2	20.83 tokens/s
3	20.84 tokens/s
Average Eval Rate	20.89 tokens/second

Running benchmark 3 times using model: llama2:13b (142W)

Run	Eval Rate (Tokens/Second)
1	11.75 tokens/s
2	11.81 tokens/s
3	14.41 tokens/s
Average Eval Rate	12.65 tokens/second

DeepSeek - CPU

Running benchmark 3 times using model: deepseek-r1:1.5b (142W)

Run	Eval Rate (Tokens/Second)
1	85.56 tokens/s
2	85.05 tokens/s
3	85.55 tokens/s
Average Eval Rate	85.38 tokens/second

Running benchmark 3 times using model: deepseek-r1:8b (141W)

Run	Eval Rate (Tokens/Second)
1	19.57 tokens/s
2	19.63 tokens/s
3	19.27 tokens/s
Average Eval Rate	19.49 tokens/second

Running benchmark 3 times using model: deepseek-r1:14b (140W)

Run	Eval Rate (Tokens/Second)
1	11.36 tokens/s
2	11.38 tokens/s
3	11.39 tokens/s
Average Eval Rate	11.37 tokens/second

Running benchmark 3 times using model: deepseek-r1:70b (141W)

Run	Eval Rate (Tokens/Second)
1	2.44 tokens/s
2	2.43 tokens/s
3	2.43 tokens/s
Average Eval Rate	2.43 tokens/second

[geerlingguy]

Second set of results, Ollama was a bit annoying, as I couldn't get ROCm working with Fedora 42, and Ollama + Vulkan is also a little hard. I tried forcing iGPU usage (see ollama/ollama#5186 (comment)), but had trouble with ROCm. [Update: later I was able to get ROCm working with Fedora Rawhide]

I also read through lhl's extensive documentation on getting ROCm going on Linux with Strix Halo, where he noted getting better performance with the Vulkan backend than ROCm.

So I switched to llama.cpp, and followed my old guide for running llama.cpp with Vulkan on a Pi 5. The required dependencies were slightly different on Fedora 42 versus the Debian-based Pi OS I used previously. Here are the exact commands I ran to build llama.cpp from source:

cd Downloads
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

sudo dnf install cmake vulkan vulkan-tools glslc vulkan-loader-devel libcurl-devel
cmake -B build -DGGML_VULKAN=1
cmake --build build --config Release

Then download a model from HuggingFace and test it:

# Download llama3.2:3b
cd models && wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

# Run it.
cd ../
./build/bin/llama-cli -m "models/Llama-3.2-3B-Instruct-Q4_K_M.gguf" -no-cnv -p "Why is the blue sky blue?" -e -ngl 100 -t 4

# You should see in the output, ggml_vulkan detected the iGPU:
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv)...

I couldn't find a way to utilize the Strix Halo NPU yet, though. It seems like nobody's cracked that nut.

Here are the results with the iGPU and llama.cpp with the Vulkan backend:

Llama 3.2 3b - GPU (Vulkan)

cd models
wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf
cd ../

./build/bin/llama-bench -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -n 128 -p 512,4096 -pg 4096,128 -ngl 99 -r 2
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 8ad7b3e6 (6019)

model	size	params	backend	ngl	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	pp512	1581.18 ± 10.80
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	pp4096	1059.94 ± 1.85
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	tg128	88.14 ± 1.28
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	pp4096+tg128	652.67 ± 1.39

Consumed around 133W during these tests.

Llama 3.1 70b - GPU (Vulkan)

cd models
wget https://huggingface.co/bartowski/Meta-Llama-3.1-70B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf
cd ../

./build/bin/llama-bench -m models/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf -n 128 -ngl 99 -fa 1 -r 2
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 8ad7b3e6 (6019)

model	size	params	backend	ngl	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan	99	pp512	34.21 ± 0.01
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan	99	pp4096	32.54 ± 0.01
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan	99	tg128	4.97 ± 0.03
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan	99	pp4096+tg128	27.26 ± 0.00

Consumed around 133W during these tests.

Qwen2.5 14b - GPU (Vulkan)

cd models
wget https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-GGUF/resolve/main/Qwen2.5-14B-Instruct-Q4_K_M.gguf
cd ../

./build/bin/llama-bench -m models/Qwen2.5-14B-Instruct-Q4_K_M.gguf -n 128 -p 512,4096 -pg 4096,128 -ngl 99 -r 2
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 8ad7b3e6 (6019)

model	size	params	backend	ngl	test	t/s
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	Vulkan	99	pp512	321.48 ± 0.61
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	Vulkan	99	pp4096	266.84 ± 0.17
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	Vulkan	99	tg128	22.97 ± 0.16
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	Vulkan	99	pp4096+tg128	184.66 ± 0.36

Consumed around 140W during these tests.

[geerlingguy]

Comparisons

See, for comparison:

• Nvidia A400, A4000, AMD Pro W7700 on System76 Thelio Astra
• AMD RX 6700 XT on Raspberry Pi 5

[geerlingguy]

Cluster Results - llama.cpp RPC

My first attempt at building a cluster for AI models is to use llama.cpp's built in RPC functionality.

I may also try:

• https://github.com/b4rtaz/distributed-llama (mentions explicit Vulkan support, with flags...)
• https://github.com/exo-explore/exo

But for RPC, I've built up an automated deployment playbook with Ansible called beowulf-ai-cluster that sets everything up, and connects everything together.

To manually set things up, once you have the main setup playbook done, you can run the following on each node (e.g. 10.0.2.209:

/opt/llama.cpp$ build/bin/rpc-server --host 10.0.2.209 --port 50052

Then run llama-cli connecting to multiple RPC nodes:

/opt/llama.cpp$ build/bin/llama-cli -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -p "Hello, my name is" --rpc 10.0.2.209:50052, -ngl 99

Llama 3.1 70B Instruct Q4_K_M

Running on a single node:

model	size	params	backend	ngl	fa	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan,RPC	99	1	pp512	36.44 ± 0.03
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan,RPC	99	1	pp4096	34.79 ± 0.01
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan,RPC	99	1	tg128	5.01 ± 0.00
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan,RPC	99	1	pp4096+tg128	36.72 ± 0.01

Result on 4 nodes using Vulkan backend, 2.5 Gbps inter-node networking:

model	size	params	backend	ngl	fa	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan,RPC	99	1	pp512	43.87 ± 0.03
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan,RPC	99	1	tg128	4.42 ± 0.00
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	Vulkan,RPC	99	1	pp512+tg128	15.64 ± 0.01

Llama 3.1 405B Q4_K_M

Result on 4 nodes using Vulkan backend, 2.5 Gbps inter-node networking:

$ build/bin/llama-bench -v -m models/Llama-3.1-405B-Q4_K_M.gguf -n 128 -p 512 -pg 512,128 -ngl 99 -fa 1 -r 2 --rpc 10.0.2.233:50052,10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	fa	test	t/s
llama ?B Q4_K - Medium	226.37 GiB	405.85 B	Vulkan,RPC	99	1	pp512	3.99 ± 0.21
llama ?B Q4_K - Medium	226.37 GiB	405.85 B	Vulkan,RPC	99	1	tg128	0.70 ± 0.00
llama ?B Q4_K - Medium	226.37 GiB	405.85 B	Vulkan,RPC	99	1	pp512+tg128	2.17 ± 0.00

Result on 4 nodes using Vulkan backend, 5 Gbps inter-node networking:

$ build/bin/llama-bench -v -m models/Llama-3.1-405B-Q4_K_M.gguf -n 128 -p 512 -pg 512,128 -ngl 125 -fa 1 -r 2 --rpc 10.0.2.233:50052,10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052

model	size	params	backend	ngl	fa	test	t/s
llama ?B Q4_K - Medium	226.37 GiB	405.85 B	Vulkan,RPC	99	1	pp512	4.62 ± 0.20
llama ?B Q4_K - Medium	226.37 GiB	405.85 B	Vulkan,RPC	99	1	tg128	0.71 ± 0.00
llama ?B Q4_K - Medium	226.37 GiB	405.85 B	Vulkan,RPC	99	1	pp512+tg128	2.23 ± 0.00

Also, attempting to get it to load and run a bit faster, I found that using --mmap 0 gets a little boost, and I believe --threads 32 gave me all the hyperthreading support on the CPU as well. Still trying to figure out how to get models to load up and run faster on this cluster with llama.cpp's RPC support, but it seems like things don't really run in parallel, but in sequence. So, good for loading larger models, not good for performance.

build/bin/llama-bench -v -m models/Llama-3.1-405B-Q4_K_M.gguf --no-warmup --mmap 0 --threads 32 -n 128 -p 512 -ngl 150 -fa 1 -r 2 --rpc 10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052

Also note that I had it not use --rpc for the main node, as it automatically picks up Vulkan0 on it, I don't know how to disable that and force llama-bench to use the RPC nodes only (or if that's even an option). If you include the node that's running the bench/cli command among the rpc servers, it will quickly run out of RAM and start swapping!

DeepSeek R1 671B Q4_K_M

First I merged all the gguf files together for convenience:

./build/bin/llama-gguf-split --merge models/DeepSeek-R1/DeepSeek-R1-Q4_K_M-00001-of-00011.gguf models/DeepSeek-R1-Q4_K_M.gguf

Then I had to make sure the environment variable export GGML_VK_PREFER_HOST_MEMORY=1 was set on all RPC nodes (and the main node). Still need to make sure RPC only using the other three nodes so llama can use the main node directly with Vulkan and no RPC...

Then I ran DeepSeek R1 with:

build/bin/llama-cli -m models/DeepSeek-R1-Q4_K_M.gguf -p "Hello, my name is" -ngl 99 --no-mmap --rpc 10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052

I needed to use --no-mmap so it could load the whole model even though it can't map it all to system RAM (since it's larger than 128 GB).

However, that didn't work... see geerlingguy/beowulf-ai-cluster#2

I think I can only use llama-bench as llama-cli (maybe?) kept trying to use the local Vulkan0 device instead of RPC, and that would run into memory issues for some reason...?

So instead, I ran llama-bench:

build/bin/llama-bench -v -m models/DeepSeek-R1-Q4_K_M.gguf -n 128 -p 512 -pg 512,128 -ngl 99 -fa 1 -r 2 --rpc 10.0.2.233:50052,10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052

Running on a single node: DNF, out of memory

Running across 4 nodes: DNF, out of memory

DeepSeek V3 671B Q4_K_M

I combined the split files again, to make it easier to work with it on node 1:

sudo ./build/bin/llama-gguf-split --merge /mnt/nas01.mmoffice.net/AI\ Models/DeepSeek-V3/DeepSeek-V3-Q4_K_M-00001-of-00009.gguf models/DeepSeek-V3-Q4_K_M.gguf

Result on 4 nodes using Vulkan backend:

build/bin/llama-bench -v -m models/DeepSeek-V3-Q4_K_M.gguf -n 128 -p 512 -pg 512,128 -ngl 99 -fa 1 -r 2 --rpc 10.0.2.233:50052,10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052

... DNF, system died

[geerlingguy]

Cluster results - distributed-llama

I installed distributed-llama from source using DLLAMA_VULKAN=1 when building, to enable Vulkan support (which worked a treat). Then I downloaded their specialized Llama 3.1 405B Instruct Q40 model:

cd /opt/distributed-llama && sudo python launch.py llama3_1_405b_instruct_q40

That takes... a long time. Once it was finished:

./dllama inference --model models/llama3_1_405b_instruct_q40/dllama_model_llama3_1_405b_instruct_q40.m --tokenizer models/llama3_1_405b_instruct_q40/dllama_tokenizer_llama3_1_405b_instruct_q40.t --buffer-float-type q80 --prompt "Hello world" --steps 256 --max-seq-len 4096 --nthreads 1 --workers 10.0.2.209:9999 10.0.2.242:9999 10.0.2.223:9999 --gpu-index 0

I had quite an ordeal getting Llama 3.1 405B to run on the full cluster—I could do it on the CPU (32 threads per worker), but had trouble on the iGPU with Vulkan.

See issue Test distributed-llama for clustered AI for the entire saga.

Run this on each of the worker nodes:

./dllama worker --port 9999 --nthreads 1 --gpu-index 0

In the end, I found out I was trying to pass too much context, so I reduced that to 4096 and then ran it again, achieving... 0.45 tokens/s!

./dllama inference --model models/llama3_1_405b_instruct_q40/dllama_model_llama3_1_405b_instruct_q40.m --tokenizer models/llama3_1_405b_instruct_q40/dllama_tokenizer_llama3_1_405b_instruct_q40.t --buffer-float-type q80 --prompt "Hello world" --steps 256 --max-seq-len 4096 --nthreads 1 --workers 10.0.2.209:9999 10.0.2.242:9999 10.0.2.223:9999 --gpu-index 0
📄 AddBos: 0
📄 BosId: 128000 (<|begin_of_text|>)
📄 EosId: 128001 (<|end_of_text|>) 128009 (<|eot_id|>) 
📄 RegularVocabSize: 128000
📄 SpecialVocabSize: 256
💡 Arch: Llama
💡 HiddenAct: Silu
💡 Dim: 16384
💡 HeadDim: 128
💡 QDim: 16384
💡 KvDim: 2048
💡 HiddenDim: 53248
💡 VocabSize: 128256
💡 nLayers: 126
💡 nHeads: 128
💡 nKvHeads: 16
💡 OrigSeqLen: 131072
💡 SeqLen: 4096
💡 NormEpsilon: 0.000010
💡 RopeType: Llama3.1
💡 RopeTheta: 500000
💡 RopeScaling: f=8.0, l=1.0, h=4.0, o=8192
📀 RequiredMemory: 66433363 kB
⭕ Socket[0]: connecting to 10.0.2.209:9999 worker
⭕ Socket[0]: connected
⭕ Socket[1]: connecting to 10.0.2.242:9999 worker
⭕ Socket[1]: connected
⭕ Socket[2]: connecting to 10.0.2.223:9999 worker
⭕ Socket[2]: connected
⭕ Network is initialized
🌋 Device: Radeon 8060S Graphics (RADV GFX1151)
🌋 DeviceApiVersion: 1.4.311
🌋 MaxComputeSharedMemory: 64 kB
🌋 Heap[1]: 72341 MB
💿 Loading weights...
💿 Weights loaded
🚁 Network is in non-blocking mode
Hello world
🔷️ Eval48935 ms Sync   21 ms | Sent 13044 kB Recv 13227 kB | (1 tokens)
🔶 Pred 2036 ms Sync   73 ms | Sent 13044 kB Recv 13227 kB | 哪
🔶 Pred 1956 ms Sync   89 ms | Sent 13044 kB Recv 13227 kB |  on
🔶 Pred 2126 ms Sync   59 ms | Sent 13044 kB Recv 13227 kB | assistant
🔶 Pred 2167 ms Sync   53 ms | Sent 13044 kB Recv 13227 kB | ations
🔶 Pred 2171 ms Sync   47 ms | Sent 13044 kB Recv 13227 kB | ~
🔶 Pred 2199 ms Sync   37 ms | Sent 13044 kB Recv 13227 kB |  noticing
...
🔶 Pred 2258 ms Sync   40 ms | Sent 13044 kB Recv 13227 kB |  enqueue
🔶 Pred 2241 ms Sync   41 ms | Sent 13044 kB Recv 13227 kB |  configurations

Evaluation
   nBatches: 32
    nTokens: 1
   tokens/s: 0.02 (48957.16 ms/tok)
Prediction
    nTokens: 255
   tokens/s: 0.45 (2216.47 ms/tok)
⭕ Network is closed

But towards the end it just kept spitting out enqueue and configurations over and over.

[geerlingguy]

Cluster results - llama.cpp with ROCm instead of Vulkan

I've upgraded to Fedora Rawhide so I could get the latest version of ROCm, supporting the Strix Halo iGPU (gfx1151), and recompiled llama.cpp with ROCm support. See Test ROCm with llama.cpp on Fedora Rawhide.

$ build/bin/llama-bench -m "models/Llama-3.2-3B-Instruct-Q4_K_M.gguf" -n 128 -p 512 -pg 512,128 -ngl 99 -fa 1 -r 2
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
build: c81de6e1 (6092)

model	size	params	backend	ngl	fa	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	ROCm,RPC	99	1	pp512	573.52 ± 12.56
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	ROCm,RPC	99	1	tg128	74.98 ± 0.00
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	ROCm,RPC	99	1	pp512+tg128	244.87 ± 0.20

Consumed around 153W on the single node with this test.

Llama 3.1 70B Instruct Q4_K_M

Running on a single node:

build/bin/llama-bench -v -m models/Llama-3.3-70B-Instruct-Q4_K_M.gguf --threads 32 -n 128 -p 512 -pg 512,128 -ngl 125 -fa 1 -r 2
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32

model	size	params	backend	ngl	threads	fa	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	ROCm,RPC	125	32	1	pp512	23.13 ± 0.02
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	ROCm,RPC	125	32	1	tg128	4.47 ± 0.00
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	ROCm,RPC	125	32	1	pp512+tg128	12.40 ± 0.01

Consumed around 139W on the single node.

Result on 4 nodes using Vulkan backend, 5 Gbps inter-node networking:

build/bin/llama-bench -v -m models/Llama-3.3-70B-Instruct-Q4_K_M.gguf --threads 32 -n 128 -p 512 -pg 512,128 -ngl 125 -fa 1 -r 2 --rpc 10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32

model	size	params	backend	ngl	threads	fa	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	ROCm,RPC	125	32	1	pp512	21.79 ± 0.02
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	ROCm,RPC	125	32	1	tg128	4.07 ± 0.00
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	ROCm,RPC	125	32	1	pp512+tg128	11.40 ± 0.01

Consumed around 181W on the cluster.

Llama 3.1 405B Q4_K_M

Result on 4 nodes using ROCm backend, 5 Gbps inter-node networking:

build/bin/llama-bench -v -m models/Llama-3.1-405B-Q4_K_M.gguf --no-warmup --mmap 0 --threads 32 -n 128 -p 512 -pg 512,128 -ngl 125 -fa 1 -r 2 --rpc 10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
...
load_tensors: layer 124 assigned to device ROCm0, is_swa = 0
load_tensors: layer 125 assigned to device ROCm0, is_swa = 0
load_tensors: layer 126 assigned to device CPU, is_swa = 0
load_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type ROCm_Host, using CPU instead
alloc_tensor_range: failed to allocate RPC[10.0.2.209:50052] buffer of size 67576004864
llama_model_load: error loading model: unable to allocate RPC[10.0.2.209:50052] buffer
llama_model_load_from_file_impl: failed to load model
main: error: failed to load model 'models/Llama-3.1-405B-Q4_K_M.gguf'

See: geerlingguy/beowulf-ai-cluster#7 (comment) (for full log).

It seems like large models are a bit finicky on a cluster with llama.cpp. Definitely 'experimental' stage right now.

All those who proclaim it's easy to run giant 300+ GB models on local clusters... see what happens if you get them off the one tested happy path with the hand-tweaked model on a specific set of one vendor's hardware :D

[geerlingguy]

gpt-oss models - Vulkan

Today OpenAI introduced two new open models, so I thought I'd take them for a spin. I quickly found flash attention caused a segfault when working with the gpt-oss models, so I disabled that.

build/bin/llama-bench -m models/gpt-oss-20b-F16.gguf --threads 32 -n 128 -p 512 -pg 512,128 -ngl 125 -r 2
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: fd1234cb (6096)

model	size	params	backend	ngl	threads	test	t/s
gpt-oss ?B F16	12.83 GiB	20.91 B	Vulkan,RPC	125	32	pp512	564.23 ± 0.46
gpt-oss ?B F16	12.83 GiB	20.91 B	Vulkan,RPC	125	32	tg128	45.01 ± 0.05
gpt-oss ?B F16	12.83 GiB	20.91 B	Vulkan,RPC	125	32	pp512+tg128	167.77 ± 0.09

Power consumption around 97W

And the 120b model:

build/bin/llama-bench -m models/gpt-oss-120b-F16.gguf --threads 32 -n 128 -p 512 -pg 512,128 -ngl 125 -r 2
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: fd1234cb (6096)

model	size	params	backend	ngl	threads	test	t/s
gpt-oss ?B F16	60.87 GiB	116.83 B	Vulkan,RPC	125	32	pp512	216.93 ± 0.74
gpt-oss ?B F16	60.87 GiB	116.83 B	Vulkan,RPC	125	32	tg128	33.12 ± 0.00
gpt-oss ?B F16	60.87 GiB	116.83 B	Vulkan,RPC	125	32	pp512+tg128	101.13 ± 0.13

Power consumption around 98W

Running 120b over the network, I got less performance as RPC was doing its round-robin thing:

build/bin/llama-bench -m models/gpt-oss-120b-F16.gguf --threads 32 -n 128 -p 512 -pg 512,128 -ngl 125 -r 2 --rpc 10.0.2.233:50052,10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: fd1234cb (6096)

model	size	params	backend	ngl	threads	test	t/s
gpt-oss ?B F16	60.87 GiB	116.83 B	Vulkan,RPC	125	32	pp512	199.38 ± 0.24
gpt-oss ?B F16	60.87 GiB	116.83 B	Vulkan,RPC	125	32	tg128	24.09 ± 0.06
gpt-oss ?B F16	60.87 GiB	116.83 B	Vulkan,RPC	125	32	pp512+tg128	79.67 ± 0.02

Power consumption around 138W

[geerlingguy]

@lhl mentioned in my Beowulf AI Cluster project that a smaller quantiziation of DeepSeek R1 would run on the cluster, so I tried DeepSeek-R1-Q4_K_M.gguf. First, I put everything together with gguf-split:

/opt/llama.cpp$ sudo ./build/bin/llama-gguf-split --merge /mnt/nas01.mmoffice.net/AI\ Models/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf /opt/DeepSeek-DeepSeek-R1-Q4_K_M.gguf

Then I ran the model on the cluster:

build/bin/llama-bench -v -m /opt/DeepSeek-DeepSeek-R1-Q4_K_M.gguf --no-warmup --mmap 0 --threads 32 -n 128 -p 512 -pg 512,128 -ngl 125 -fa 1 -r 2 --rpc 10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
build: fd1234cb (6096)

model	size	params	backend	ngl	threads	fa	mmap	test	t/s
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	Vulkan,RPC	125	32	1	0	pp512	26.86 ± 0.35
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	Vulkan,RPC	125	32	1	0	tg128	8.25 ± 0.04
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	Vulkan,RPC	125	32	1	0	pp512+tg128	17.22 ± 0.04

[lhl]

FYI, I ran a test, you can barely fit Q4_K_M, although I'd recommend Q4_K_XL or Q3_K_XL https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF with the ROCm backend (use --mmap 0 if you don't want to wait forever for loads).

The ROCm backend does a better job of balancing the layers for memory. BTW, llama-bench behavior has changed to be the same as llama-cli/server (you can start an rpc server on where you're running it, don't include it in the rpc servers, and it'll do the right thing); https://github.com/lhl/strix-halo-testing/tree/main/rpc-test#deepseek-r1-q4_k_m

[eclecticc]

gpt-oss models - Vulkan

Today OpenAI introduced two new open models, so I thought I'd take them for a spin. I quickly found flash attention caused a segfault when working with the gpt-oss models, so I disabled that.

Just a note that the MXFP4 versions of these models should run faster and may not impact the quality of output (the models were apparently also trained in FP4?).

[geerlingguy]

The ROCm backend does a better job of balancing the layers for memory. BTW, llama-bench behavior has changed to be the same as llama-cli/server (you can start an rpc server on where you're running it, don't include it in the rpc servers, and it'll do the right thing); https://github.com/lhl/strix-halo-testing/tree/main/rpc-test#deepseek-r1-q4_k_m

Good to know! You got 6-7 tokens/s, that's not bad at all!

[kth8]

Odd that Ollama didn't recognize the iGPU when Strix Halo support was added 5 months ago in version 0.6.2.

[geerlingguy]

@kth8 - note that I got Ollama to work with ROCm after upgrading to Fedora Rawhide. For some reason Fedora 42 wasn't allowing me to set up rocm at all. Not sure if it was a packaging issue or what.

But ROCm seemed to give slightly worse performance than Vulkan under llama.cpp. I think it's boneheaded Ollama decided to not implement Vulkan support, even though it's been working great in a community fork for months.