Performance of llama.cpp on NVIDIA DGX Spark #16578
Replies: 13 comments 45 replies
-
|
Thanks for the benchmark! I would like to request additional benchmark for a very popular model GLM-4.5-Air-FP8: and quants for it:
|
Beta Was this translation helpful? Give feedback.
-
|
Saw the benchmark results. Thank you so much for the work! Appreciate very much. |
Beta Was this translation helpful? Give feedback.
-
|
Hi. It would be great to see a Qwen Next 80B benchmark for these two models: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Not support yet with open pr currently |
Beta Was this translation helpful? Give feedback.
-
Yeah I really want to see the performance of a specific model comparing full 16 bit precision, Q8, Q4, FP4 and FP8. None the less, thank you for the wonderful data! |
Beta Was this translation helpful? Give feedback.
-
|
Getting similar performance with my Farmework Desktop. Thanks for helping my FOMO. |
Beta Was this translation helpful? Give feedback.
-
|
Same chip so should be the same perf. Asus says shipping date of November 3 here. |
Beta Was this translation helpful? Give feedback.
-
I'm sure the connectx-7 200GB networking has something to do with the pricing difference :) |
Beta Was this translation helpful? Give feedback.
-
Interesting, the Asus GB10 seems to run with 240W power adapter, much higher than the DGX Spark. I wonder if you will get more performance given the higher power intake. |
Beta Was this translation helpful? Give feedback.
-
I havent seen the specs. But its possible ASUS just used a power adapter with a high enough rating for the device? For example, I can plug a 90watt compatible power adapter into my 45watt laptop. It will pull what it needs to. |
Beta Was this translation helpful? Give feedback.
-
|
@bartlettroscoe i benched gpt-oss 120b on Framework Desktop a couple months ago: geerlingguy/ai-benchmarks#21 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
Can you run the classic llama 2 7B Q4_0 so it can be compared on the chart? |
Beta Was this translation helpful? Give feedback.
-
|
Super interesting, thanks for sharing, Georgi!
Could you please help me understand: Does "-d" mean KV cache length before the "-p" prefill happens? What does "-ub" define, eg batch size? |
Beta Was this translation helpful? Give feedback.
-
Yes.
Yes. |
Beta Was this translation helpful? Give feedback.
-
|
Could you add llama2-7b result to #15013? |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, thank you! So whats the sense of a dgx spark? I mean sure it has 128gb memory, but i can offload bigger models between 96gb vram and the rest to normal Ram (CPU)... Its too expensive for what it offers. If the DGX Spark would be around 2k, like the Ryzen Max 395+ Mini-PC's it would be fine and okay. PS: And a Mac Mini/Studio is a much better option at 4k usd/eur, compared to a DGX Sparc. |
Beta Was this translation helpful? Give feedback.
-
|
Ryzen AI Max 395+ Cannot do FP8 or FP4. The DGX Spark will have a HUGE speedup when using FP8 (Theoretically 2x for FP8 and 4x for FP4), which is what a lot of benchmarking posts/videos arent addressing. I also do not consider the RTX 6000 Pro (Blackwell) a fair comparison. That's probbably an $8,000+ GPU, plus the additional cost of the system ($1000 if being fairly modest...). Someone might need that much VRAM to finetune or run/test a model, but not have $9,000 to spend. Dont get me wrong it is a niche. Even a Mac Mini M4 Max has 546 GB/s memory, versus the 273 GB/s on the DGX spark, but the FP8 and FP4 capabilities should also be factored in here and have significant value. This is why I would love to see the benchmarks for running the Q4/Q8 vs FP4FP8 |
Beta Was this translation helpful? Give feedback.
-
The benchmarks clearly show that the gpt-oss-120B at MXFP4 is comparable between the Ryzen AI Max 395+ and DGX Spark. That's FP4 model, unless I am missing something. Also Q4 and Q8 (GUUF) are INT and not FP, and thus would not benefit from the sparse "FP" speed up claimed by Nvidia.. |
Beta Was this translation helpful? Give feedback.
-
|
Guys, please dont take fp4 or fp8 as a win. Let me explain: Comparing embedding Models is actually great, because you can simply query the resulting vector database and see the quantisation impacts. From my tests, no matter which Model, be it Qwen3-Embedding or BGE-M3 or anything else, the impact of Quantisation is Huge! FP32 is Amazing So you Guys want to tell me that FP4 is a win? I didnt tested fp4 tho, not even fp8, so i cant say for sure. Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
It depends on the model. In many cases, in my experience FP4 does a fantastic job. Also NVFP4 has the potential to be amazing. So is it situational? Sure, it can be. But I don't think it's something that can be ignored. Also, FP8 is also great, I have found little reason to not use it. |
Beta Was this translation helpful? Give feedback.
-
|
I'd agree that everyone should eval for their particular downstream tasks rather than just trusting perplexity or KLD. When running quants on my 405B model I ran JA MT-Bench evals and was surprised to find a bigger difference with FP8-Dynamic than IQ3_M. @icsy7867 I know you're just theory-crafting instead of running tests, but see my PRO 6000 TensorRT/NVFP4 benchmark below, but there is zero throughput benefit from NVFP4. Maybe related to NVIDIA/TransformerEngine#2255 - I never use TensorRT and it's impossible to build so I just used the latest docker for my tests (tensorrt-llm/release:1.2.0rc0) but I've put my full scripts/details online so it's easy for anyone to rent any GPU they want to check any configuration/variation for themselves. |
Beta Was this translation helpful? Give feedback.
-
|
@ggerganov Are there llama.cpp benchmarks for the AGX Thor? It seems it's similar offering but Nvidia markets it as twice as fast. There are no official detailed spec sheet for the DGX Spark to make a comparison to the Thor (2560 cuda cores and 92 tensor cores), but Nvidia claims 2PLOPS (sparse FP4) for the Thor and 1PFLOPS (sparse FP4) for the Spark. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not familiar with AGX Thor. But if you have one, you can easily run the same benchmarks on it. |
Beta Was this translation helpful? Give feedback.
-
|
Quick tldr: Thor is sm_110 (formerly sm_101) with the datacenter-style tensor cores - including tensor memory. And no raytracing cores. While Spark is sm_121 with the full consumer Blackwell feature set. Thor and Spark have relatively similar memory bandwidth. The Thor CPU is much slower. Vector throughput on Thor is 1/3rd of the one on DGX Spark but you get twice the matrix throughput. Thor has 4 cursed Synopsys 25GbE NICs (set to 10GbE by default, see https://docs.nvidia.com/jetson/archives/r38.2/DeveloperGuide/SD/Kernel/Enable25GbEthernetOnQSFP.html as it doesn't have auto-negociation of the link rate) exposed via a QSFP connector providing 4x25GbE while Spark systems have regular ConnectX-7. Thor uses a downstream L4T stack instead of regular NVIDIA drivers unlike Spark. But at least the CUDA SDK is the same unlike prior Tegras. Oh and you get less other IO too. Side note: might be better to also consider GB10 systems from OEMs. Those are available for cheaper than AGX Thor devkits too. |
Beta Was this translation helpful? Give feedback.
-
I don't have one unfortunately, hoping whoever does will run those benchmarks.
This is a very weird and interesting tradeoff. |
Beta Was this translation helpful? Give feedback.
-
@woachk does "tensor memory" here refer to TMEM? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. |
Beta Was this translation helpful? Give feedback.
-
|
For those curious about Thor performance gpt-oss-20b-gguf# ./bin/llama-bench -m /workspace/models/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 | 2008.85 ± 4.18 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 | 60.85 ± 0.17 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 1862.13 ± 4.80 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 55.03 ± 0.06 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 1740.90 ± 3.24 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 53.58 ± 0.18 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 1446.75 ± 3.01 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 52.49 ± 1.94 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 1193.93 ± 0.72 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 48.33 ± 0.04 |
build: f9fb33f2 (6771)Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF# ./bin/llama-bench -m /workspace/models/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF/qwen3-coder-30b-a3b-instruct-q8_0.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 | 1654.25 ± 1.80 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 | 44.26 ± 0.11 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 1410.87 ± 2.22 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 39.46 ± 0.04 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 1228.69 ± 1.78 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 36.88 ± 0.13 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 985.39 ± 7.04 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 33.55 ± 0.01 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 686.45 ± 0.93 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 26.92 ± 0.05 |
build: f9fb33f2 (6771)gpt-oss-120b# ./bin/llama-bench -m /workspace/models/gpt-oss-120b-GGUF/gpt-oss-120b-mxfp4-00001-of-00003.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 | 967.20 ± 6.04 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 | 42.00 ± 0.09 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 932.85 ± 2.33 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 38.81 ± 0.04 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 892.28 ± 2.88 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 39.22 ± 1.05 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 827.57 ± 1.28 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 37.77 ± 0.01 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 677.70 ± 1.06 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 34.02 ± 0.02 |
build: f9fb33f2 (6771) |
Beta Was this translation helpful? Give feedback.
-
|
That commit only applies the change to |
Beta Was this translation helpful? Give feedback.
-
|
I did a quick one off build where I removed the conditional around the scheduling block to force spin and I do see a consistent improvement. Just looking at power draw there is a probably at least another 10-20% performance untapped on thor beyond moving it to the spin scheduler. Currently looks like we are mostly cpu bound. 
Llama-bench Test Results (Qwen3moe 30B)schedule Default Spin Improvement (%) Average improvement: 1.86% Llama-batched-bench Test ResultsPP=4096: PP=8192: Spin schedule
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
Test: llama-bench
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 | 1700.05 ± 2.02 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 | 45.67 ± 0.11 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 1446.22 ± 3.54 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 40.64 ± 0.05 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 1257.35 ± 0.75 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 38.09 ± 0.09 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 992.37 ± 1.89 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 33.62 ± 0.01 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 687.30 ± 0.48 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 27.05 ± 0.03 |
Test: llama-batched-bench
| PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
| 4096 | 32 | 1 | 4128 | 2.537 | 1614.38 | 0.789 | 40.54 | 3.327 | 1240.92 |
| 4096 | 32 | 2 | 8256 | 4.949 | 1655.30 | 1.301 | 49.18 | 6.250 | 1320.87 |
| 4096 | 32 | 4 | 16512 | 9.887 | 1657.09 | 1.663 | 76.98 | 11.550 | 1429.62 |
| 4096 | 32 | 8 | 33024 | 19.739 | 1660.11 | 2.289 | 111.86 | 22.027 | 1499.25 |
| 4096 | 32 | 16 | 66048 | 39.464 | 1660.65 | 3.279 | 156.14 | 42.743 | 1545.23 |
| 4096 | 32 | 32 | 132096 | 78.936 | 1660.49 | 5.033 | 203.46 | 83.968 | 1573.16 |
| 8192 | 32 | 1 | 8224 | 5.314 | 1541.47 | 0.839 | 38.14 | 6.153 | 1336.50 |
| 8192 | 32 | 2 | 16448 | 10.614 | 1543.68 | 1.396 | 45.86 | 12.009 | 1369.61 |
| 8192 | 32 | 4 | 32896 | 21.220 | 1544.24 | 1.888 | 67.79 | 23.108 | 1423.59 |
| 8192 | 32 | 8 | 65792 | 42.394 | 1545.87 | 2.792 | 91.68 | 45.187 | 1456.01 |
| 8192 | 32 | 16 | 131584 | 84.800 | 1545.66 | 4.206 | 121.73 | 89.006 | 1478.37 |
| 8192 | 32 | 32 | 263168 | 169.577 | 1545.87 | 6.867 | 149.11 | 176.444 | 1491.51 | |
Beta Was this translation helpful? Give feedback.
-
|
For prompt processing there's a lot more on the table but that means switching to tcgen05 MMA instructions. (Which is a separate instruction set than the regular tensor core one) And there's also the matter of using lower precision MMAs in general |
Beta Was this translation helpful? Give feedback.
-
|
I believe that Thor doesn't support tcgen05 because it doesn't have tensor-memory |
Beta Was this translation helpful? Give feedback.
-
|
Thor does have tensor memory - it uses the data centre tensor cores (it's sm_110[a]), Spark does not. See https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma |
Beta Was this translation helpful? Give feedback.
-
|
Would love to see accuracy of the same models on main banchmarks running in DGX as they will vary on different HW & FW in addition to the speed. As its clearly sing here https://artificialanalysis.ai/models/gpt-oss-120b/providers |
Beta Was this translation helpful? Give feedback.
-
|
Please bench the full Qwen3 coder model |
Beta Was this translation helpful? Give feedback.
-
|
There isn't any measurable benefits in terms of quality compared to |
Beta Was this translation helpful? Give feedback.
-
|
I am just impressed that it might run at all. It's there any bench on fine-tuning? |
Beta Was this translation helpful? Give feedback.
-
|
Would love to see this this cluster setup in the comparison table too |
Beta Was this translation helpful? Give feedback.
-
|
AFAICT this is vaporware. |
Beta Was this translation helpful? Give feedback.
-
|
On the subject of Spark and Thor, I have been looking for alternatives to TensorRT for python-free and community driven inference engine. I'm looking to leverage nvfp4 tensor cores , and wonder if there's any project or folks working to support those in llama.cpp? |
Beta Was this translation helpful? Give feedback.
-
|
The whole Blackwell product range, from the RTX 5050 onwards to the B200/300 through iGPUs |
Beta Was this translation helpful? Give feedback.
-
|
That said: NVIDIA/TransformerEngine#2255 |
Beta Was this translation helpful? Give feedback.
-
|
Just as an FYI, I don't have a Spark but I tested NVP4 on an RTX PRO 6000 (Llama 3.1 8B Instruct). NVP4 w/ TensorRT does not perform better than llama.cpp at bs=1, and at higher concurency, doesn't take a lead until c=32. I didn't test quality loss, but from a pure throughput perspective, I don't think the current NVFP4 implementation is particularly good. Certainly not worth all the custom quanting and other hassles...
|
Beta Was this translation helpful? Give feedback.
-
|
@lhl what's the prefill sequence length in the profiles above? |
Beta Was this translation helpful? Give feedback.
-
|
This is using a standard vLLM bench - ShareGPT w/ prefill 1024 and decode 128 I believe. If you have a specific use case it's probably best to just trying the device directly - I think they're available for a buck or two on Vast or Runpod. I think the compute is particularly strong for a client card. For example, the PRO 6000 actually beats an H100 on our Whisper inference sweeps. (Still trains much slower though) Here's my LLM sweep scripts (and raw results) btw: https://github.com/AUGMXNT/speed-benchmarking/tree/main/nvfp4 |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Overview
This document summarizes the performance of
llama.cppfor various models on the new NVIDIA DGX Spark.Benchmarks include:
pp) and generation (tg) at various context depths (d)Models:
gpt-oss-20bgpt-oss-120bQwen3 Coder 30B A3BQwen2.5 Coder 7BGemma 3 4B QATGLM 4.5 AirFeel free to request additional benchmarks for models and use cases.
Benchmarks
Using the following commands:
History
2025 Oct 14 (b6761)7ea15bb Initial numbers2025 Oct 15 (b6767)5acd455 Improved decode via CUDA: Changing the CUDA scheduling strategy to spin #16585gpt-oss-20b
Model: https://huggingface.co/ggml-org/gpt-oss-20b-GGUF
llama-benchDevice 0: NVIDIA GB10, compute capability 12.1, VMM: yes
build: 5acd455 (6767)
llama-batched-benchgpt-oss-120b
Model: https://huggingface.co/ggml-org/gpt-oss-120b-GGUF
llama-benchDevice 0: NVIDIA GB10, compute capability 12.1, VMM: yes
build: 5acd455 (6767)
llama-batched-benchQwen3 Coder 30B A3B
Model: https://huggingface.co/ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF
llama-benchDevice 0: NVIDIA GB10, compute capability 12.1, VMM: yes
build: 5acd455 (6767)
llama-batched-benchQwen2.5 Coder
Model: https://huggingface.co/ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF
llama-benchDevice 0: NVIDIA GB10, compute capability 12.1, VMM: yes
build: 5acd455 (6767)
llama-batched-benchGemma 3 4B QAT
Model: https://huggingface.co/ggml-org/gemma-3-4b-it-qat-GGUF
llama-benchDevice 0: NVIDIA GB10, compute capability 12.1, VMM: yes
build: 5acd455 (6767)
llama-batched-benchGLM 4.5 Air
Model: https://huggingface.co/unsloth/GLM-4.5-Air-GGUF/tree/main
llama-benchDevice 0: NVIDIA GB10, compute capability 12.1, VMM: yes
build: 5acd455 (6767)
llama-batched-benchMore info
Beta Was this translation helpful? Give feedback.
All reactions