llama : add --n-cpu-ffn option

[John-194]

Overview

Putting dense model layers on the CPU via --ngl causes major slowdowns. I added an option similar to the existing --n-cpu-moe which is --n-cpu-ffn. It puts user specified amount of FFN sublayers for dense models.

Additional information

My reddit post shows that people are interested. This PR would significantly simplify it for them.

To keep it simple, I made --n-cpu-ffn blindly take N layers from 0. An optimization is possible by prioritizing the largest FFN layers first for a possible future PR, which would allow for an extra 10% tg speed vs the current sequential method (tested on the new UD-Q4_K_M).

Here is a speed comparison vs ngl (MTP has different speeds for prose and code so I separated both for accuracy):

I achieved a tolerable speed of 15-25 t/s tg using Q4_K_M model with a large context of over 90k. The speed/context results are not from -ot alone, but it is the main parameter that made it useable at a longer context using only 16 GB of VRAM.

HW:
RTX 4070 Ti SUPER (16 GB VRAM), i5-13600KF, 32 GB dual-channel DDR5 @ 5800 MHz + tuned timings, Ubuntu 24.04 LTS

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, AI implemented it with my oversight.

[troed]

As one of the persons who have been using Qwen 3.6 27B Q4_K_M and now Qwen 3.8 27B Q4_K_M on 16GB VRAM thanks to your explanation of ffn offloading I think there's a huge benefit in making an easily settable option like this as you propose in this PR. While I still see people sometimes claim they cannot run an MoE because they don't have the VRAM for it, most people now know it's possible by offloading experts to RAM precisely because of the cpu-moe and n-cpu-moe options in llama.cpp.

Offloading for dense models and still keep respectable performance is still mostly unknown even though it really works and offers a viable path to running dense models on "low" VRAM cards - and with the corresponding ffn offload settings it will hopefully become much more common knowledge.

[kroaton]

Really worthwhile for low VRAM users.

[Green-Sky]

Would help if the pr op could stand on its own, and not requires the link to an external service.

[John-194]

Would help if the pr op could stand on its own, and not requires the link to an external service.

You are right.
I updated the PR with a speed comparison chart, added additional info, and updated the link to a new post that I made that shows my full llama.cpp setup for those interested (I want to keep the PR short, simple, and to the point, but if you think I should include more information here say so).

[randomqhacker]

IMHO this is what llama.cpp is all about, enabling more people to run LLMs locally! I can actually run Qwen 3.x 27B at a usable speed on my low-end card with this FFN-on-CPU technique!

[fairydreaming]

Not sure why this was ignored, so I tested it on my rig (Epyc 9374F + RTX PRO 6000 Max-Q). Note that this is somewhat special case as my RAM bandwidth is quite high (if I remember correctly around 360 GB/s).

Whole model in VRAM

$ ./bin/llama-batched-bench -m ~/ggufs/Qwen3.5-27B-Q8_0.gguf -npl 1 -npp 2048,8192,16384,65536,131072 -ntg 128 -ub 2048

llama_batched_bench: n_kv_max = 262144, n_batch = 2048, n_ubatch = 2048, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  2048 |    128 |    1 |   2176 |    0.639 |  3203.11 |    2.632 |    48.64 |    3.271 |   665.23 |
|  8192 |    128 |    1 |   8320 |    2.659 |  3081.39 |    2.673 |    47.88 |    5.332 |  1560.43 |
| 16384 |    128 |    1 |  16512 |    5.509 |  2973.96 |    2.725 |    46.97 |    8.234 |  2005.23 |
| 65536 |    128 |    1 |  65664 |   28.347 |  2311.88 |    3.039 |    42.12 |   31.386 |  2092.12 |
|131072 |    128 |    1 | 131200 |   75.049 |  1746.49 |    3.446 |    37.15 |   78.494 |  1671.46 |

VRAM usage: 45258MiB

32 layers on CPU (-ngl 33)

$ ./bin/llama-batched-bench -m ~/ggufs/Qwen3.5-27B-Q8_0.gguf -npl 1 -npp 2048,8192,16384,65536,131072 -ntg 128 -ub 2048 -ngl $((65-32))

llama_batched_bench: n_kv_max = 262144, n_batch = 2048, n_ubatch = 2048, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = 33, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  2048 |    128 |    1 |   2176 |    1.192 |  1718.43 |    9.470 |    13.52 |   10.662 |   204.10 |
|  8192 |    128 |    1 |   8320 |    4.854 |  1687.77 |    9.861 |    12.98 |   14.715 |   565.41 |
| 16384 |    128 |    1 |  16512 |    9.861 |  1661.48 |   11.187 |    11.44 |   21.048 |   784.49 |
| 65536 |    128 |    1 |  65664 |   46.259 |  1416.72 |   14.115 |     9.07 |   60.374 |  1087.63 |
|131072 |    128 |    1 | 131200 |  113.751 |  1152.27 |   25.980 |     4.93 |  139.732 |   938.94 |

VRAM usage: 25042MiB

We can lower VRAM usage by 20GB, but our PP is much lower and TG is crap for longer context length.

FFN tensors on CPU in 32 layers (-ncffn 32)

$ ./bin/llama-batched-bench -m ~/ggufs/Qwen3.5-27B-Q8_0.gguf -npl 1 -npp 2048,8192,16384,65536,131072 -ntg 128 -ub 2048 -ncffn 32
0.00.617.302 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance

llama_batched_bench: n_kv_max = 262144, n_batch = 2048, n_ubatch = 2048, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  2048 |    128 |    1 |   2176 |    0.970 |  2112.34 |    5.958 |    21.48 |    6.928 |   314.10 |
|  8192 |    128 |    1 |   8320 |    3.968 |  2064.51 |    5.997 |    21.34 |    9.965 |   834.93 |
| 16384 |    128 |    1 |  16512 |    8.073 |  2029.46 |    6.047 |    21.17 |   14.120 |  1169.38 |
| 65536 |    128 |    1 |  65664 |   38.118 |  1719.28 |    6.312 |    20.28 |   44.430 |  1477.91 |
|131072 |    128 |    1 | 131200 |   94.324 |  1389.59 |    6.684 |    19.15 |  101.009 |  1298.90 |

VRAM usage: 36602MiB

So we traded 11.6GB of VRAM for a huge increase in TG, PP is also higher. Very nice!

Matching -ncffn 32 VRAM usage with -ngl

But then I thought - hey, I can simply increase my -ngl value instead to match the VRAM usage of -ncffn 32 case and compare the performance of both.

$ ./bin/llama-batched-bench -m ~/ggufs/Qwen3.5-27B-Q8_0.gguf -npl 1 -npp 2048,8192,16384,65536,131072 -ntg 128 -ub 2048 -ngl $((65-15))

llama_batched_bench: n_kv_max = 262144, n_batch = 2048, n_ubatch = 2048, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = 50, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  2048 |    128 |    1 |   2176 |    0.901 |  2272.93 |    5.931 |    21.58 |    6.832 |   318.49 |
|  8192 |    128 |    1 |   8320 |    3.707 |  2209.68 |    6.089 |    21.02 |    9.797 |   849.27 |
| 16384 |    128 |    1 |  16512 |    7.560 |  2167.18 |    6.611 |    19.36 |   14.171 |  1165.17 |
| 65536 |    128 |    1 |  65664 |   36.759 |  1782.86 |    7.823 |    16.36 |   44.582 |  1472.88 |
|131072 |    128 |    1 | 131200 |   94.706 |  1383.98 |   11.914 |    10.74 |  106.621 |  1230.53 |

VRAM usage: 36722MiB

Hmm, PP is slightly faster than with -ncffn, TG matches the performance of -ncffn case for small context length but for longer context -ncffn was clearly faster. So this PR makes sense.

Edit: re-ran tests with 32 offloaded layers to make the difference more pronounced.
Edit2: Added section with -ngl matching -ncffn VRAM usage
Edit3: Added 131072 PP test case

[ddh0]

Testing Qwen3.8-27B-IQ3_XXS.gguf on RTX 4060 Ti 16GB + Ryzen 7 7700X + DDR5 @ 4800MT/s. Both of the following runs used approximately the same amount of VRAM.

Using -ngl 52

env -i ./build/bin/llama-batched-bench -m ~/gguf/Qwen3.8-27B-IQ3_XXS.gguf -npl 1 -npp 65536 -ntg 128 -ub 1024 -ngl 52 -ctk q4_0 -ctv q4_0

llama_batched_bench: n_kv_max = 262144, n_batch = 2048, n_ubatch = 1024, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = 52, n_threads = 8, n_threads_batch = 8

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
| 65536 |    128 |    1 |  65664 |  118.385 |   553.58 |   20.138 |     6.36 |  138.523 |   474.03 |

2.21.880.683 I llama_perf_context_print:        load time =    3163.60 ms
2.21.880.684 I llama_perf_context_print: prompt eval time =  119137.26 ms / 65552 tokens (    1.82 ms per token,   550.22 tokens per second)
2.21.880.684 I llama_perf_context_print:        eval time =   20136.45 ms /   128 runs   (  157.32 ms per token,     6.36 tokens per second)
2.21.880.685 I llama_perf_context_print:       total time =  141688.10 ms / 65680 tokens
2.21.880.685 I llama_perf_context_print:    graphs reused =        127

Using -ncffn 32

env -i ./build/bin/llama-batched-bench -m ~/gguf/Qwen3.8-27B-IQ3_XXS.gguf -npl 1 -npp 65536 -ntg 128 -ub 1024 -ncffn 32 -ctk q4_0 -ctv q4_0

llama_batched_bench: n_kv_max = 262144, n_batch = 2048, n_ubatch = 1024, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 8, n_threads_batch = 8

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
| 65536 |    128 |    1 |  65664 |  117.142 |   559.46 |   16.722 |     7.65 |  133.864 |   490.53 |

2.17.279.136 I llama_perf_context_print:        load time =    3218.24 ms
2.17.279.137 I llama_perf_context_print: prompt eval time =  118376.59 ms / 65552 tokens (    1.81 ms per token,   553.76 tokens per second)
2.17.279.137 I llama_perf_context_print:        eval time =   16720.97 ms /   128 runs   (  130.63 ms per token,     7.66 tokens per second)
2.17.279.138 I llama_perf_context_print:       total time =  137083.80 ms / 65680 tokens
2.17.279.138 I llama_perf_context_print:    graphs reused =        127

Summary

• PP speed: 550 --> 553 t/s
• TG speed: 6.36 --> 7.66 t/s

[pwilkin]

I think this is both useful and low-risk so I don't see a problem with merging. @CISC @ngxson WDYT?

[CISC]

Either adapt this to take the regex and rename or replace it in llama-bench.

[John-194]

Done. Should --n-cpu-ffn also become part of llama-bench, mirroring --n-cpu-moe?

[CISC]

It would be useful, doesn't have to be in this PR though.

[gopinath87607]

can we able to use the dflash2 with this ?

[CISC]

Wonder what happened to CI? Will try to close/open...

[CISC]

Wonder what happened to CI? Will try to close/open...

Yay, that worked.

[miversen33]

What are the differences between this and --override-tensor?

[Green-Sky]

What are the differences between this and --override-tensor?

convenience

On that note, it would be nice if the process would print ot parameters of what happend bc of fit, ngl, n-cpu-ffn etc.