ggml-cpu: improve --n-cpu-moe TG performance - #20596
Conversation
| if (ggml_can_fuse_subgraph(cgraph, node_n, 3, fuse_ops, outputs, 1)) { | ||
| struct ggml_tensor * node1 = cgraph->nodes[node_n + 1]; | ||
| struct ggml_tensor * glu = cgraph->nodes[node_n + 2]; | ||
| // Fused path for `--n-cpu-moe` when n_tokens = 1 |
There was a problem hiding this comment.
Why do we do it only for n_tokens == 1?
There was a problem hiding this comment.
Single token is a special case because you don't need to create the row mapping. Also in this I let each thread quantize the activation to remove a barrier. The barrier cost was around 30% of the total run time in the cpu part, with this change it's about 7%
There was a problem hiding this comment.
Hm, that's unexpected for the barrier to be so expensive. I'll need to double-check - do you have a patch that I can apply to test the barrier path?
Btw, you should space the wdata with CACHE_LINE_SIZE_F32 to avoid false sharing.
There was a problem hiding this comment.
We have test-barrier that measures the overhead with tiny graphs.
@am17an can you share what numbers you get from that.
Ideally with and without OMP (GGML_OPENMP=OFF)
There was a problem hiding this comment.
@ggerganov on master if you do
CUDA_VISIBLE_DEVICES=4 perf record ./build/bin/llama-bench -m /opt/models/Qwen3.5-35B-A3B-Q4_K_S.gguf -fa 1 -n 32 -r 20 -p 0 -ncmoe 99 -t 8
with OpenMP (the default build)
graph-compute with
n_threads: 4
n_nodes: 2000
n_rounds: 100
graph-compute took 516226 usec
5162.26 usec per-iter
2581.13 nsec per-node
graph-compute with
n_threads: 4
n_nodes: 4
n_rounds: 10000
graph-compute with
gf0 n_nodes: 4
gf1 n_nodes: 8
n_threads: 4
n_rounds: 1000
With OpenMP=OFF
graph-compute with
n_threads: 4
n_nodes: 2000
n_rounds: 100
graph-compute took 479589 usec
4795.89 usec per-iter
2397.95 nsec per-node
graph-compute with
n_threads: 4
n_nodes: 4
n_rounds: 10000
graph-compute with
gf0 n_nodes: 4
gf1 n_nodes: 8
n_threads: 4
n_rounds: 1000
There was a problem hiding this comment.
Also since this barrier cost is non-trivial, it has a meaningful effect for this case, when there are too many threads contending. Maybe we should limit the n_tasks based on some heuristic. Can someone else also confirm these results (I'm not sure if NUMA is playing a role here). For reference all results on a AMD EPYC 7742 64-Core Processor
CUDA_VISIBLE_DEVICES=4 ./build/bin/llama-bench -m /opt/models/qwen3_next_80b_a3b_instruct-iq4_xs.gguf -fa 1 -n 32 -r 10 -p 0 -ncmoe 999 -t 8,16,32,48,64
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32111 MiB):
Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32111 MiB (31604 MiB free)
| model | size | params | backend | ngl | threads | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | --------------: | -------------------: |
| qwen3next 80B.A3B IQ4_XS - 4.25 bpw | 39.67 GiB | 79.67 B | CUDA | 99 | 8 | 1 | tg32 | 30.85 ± 0.48 |
| qwen3next 80B.A3B IQ4_XS - 4.25 bpw | 39.67 GiB | 79.67 B | CUDA | 99 | 16 | 1 | tg32 | 29.81 ± 0.45 |
| qwen3next 80B.A3B IQ4_XS - 4.25 bpw | 39.67 GiB | 79.67 B | CUDA | 99 | 32 | 1 | tg32 | 28.76 ± 0.58 |
| qwen3next 80B.A3B IQ4_XS - 4.25 bpw | 39.67 GiB | 79.67 B | CUDA | 99 | 48 | 1 | tg32 | 27.48 ± 0.30 |
| qwen3next 80B.A3B IQ4_XS - 4.25 bpw | 39.67 GiB | 79.67 B | CUDA | 99 | 64 | 1 | tg32 | 26.19 ± 0.35 |
There was a problem hiding this comment.
I'll take a look later today. Btw, I was rather thinking to compare the fused version:
- no barrier + all threads reading src1 (current PR)
- barrier + 1 thread reading src1
There was a problem hiding this comment.
Here is what I'm getting with OpenMP=OFF
AMD EPYC 7543
5177.86 usec per-iter
2588.93 nsec per-node
AMD Ryzen 395+
1559.2 usec per-iter
779.601 nsec per-node
Mac M4 Pro
1338.91 usec per-iter
669.456 nsec per-node
Galaxy S24+
1873.33 usec per-iter
936.669 nsec per-node
Your numbers are a bit high but inline with the AMD EPYC.
With OpenMP=OFF
4795.89 usec per-iter 2397.95 nsec per-node
Also from your profile results it looks like it's not really the barrier itself that is expensive.
Most of the cycles are spent in the barrier_wait.
That just means that some of the cores are completing their chunks faster than the others cores.
Maybe our chunking is suboptimal in this case?
We recently updated MUL_MAT and FA to improve dynamic chunking.
Perhaps, it's a similar issue here.
There was a problem hiding this comment.
@max-krasnyansky that profile is from the master branch. MUL MAT ID also has the same chunked approach if I'm not wrong
github-actions
Bot
added
the
ggml
|
I don't think I am seeing any benefit on M2 Ultra:
0c0cf6f does not make much difference compared to the PR branch. This is without OpenMP. |
|
The results seem to have a lot of variability. Especially at t=2 threads. In any case it should not have a slowdown, because it's doing less flops, barriers and memory accesses than baseline. |
|
I repeated this machine on another recent AMD machine (AMD EPYC 9B14 96-Core Processor), using
|
|
@ggerganov in your command don't you have to specify |
|
Hm, I must have copied the wrong command - without In any case, here is another run with ./scripts/compare-commits.sh 983df142a99b764f5cbe9acb6b9ec862f861d353 pr/20596 llama-bench -m ~/models/qwen3.5-35B-a3b-q4_0-pure.gguf -m ~/models/qwen3-30b-a3b/ggml-model-q8_0.gguf -fa 1 -p 0 -n 32 -r 10 -mmp 1 -t 1,2,4,8,16 -ncmoe 99
Here is your command on the M2 Studio, DGX Spark and a 16-core AMD Ryzen 9:
./scripts/compare-commits.sh master pr/20596 llama-bench -hf unsloth/Qwen3.5-35B-A3B-GGUF -n 32 -p 0 -r 10 -t 2,4,8,16 -fa 1 -ncmoe 999
GGML_CUDA=ON ./scripts/compare-commits.sh master pr/20596 llama-bench -hf unsloth/Qwen3.5-35B-A3B-GGUF -n 32 -p 0 -r 10 -t 1,2,4,8 -fa 1 -ncmoe 999
CUDA_VISIBLE_DEVICES=0 GGML_CUDA=ON ./scripts/compare-commits.sh master pr/20596 llama-bench -hf unsloth/Qwen3.5-35B-A3B-GGUF -n 32 -p 0 -r 10 -t 2,4,8,16,32 -fa 1 -ncmoe 999
There is indeed a lot of variability from run to run, so it's difficult to make a conclusion. Maybe what we need to bench is running with Do the results look more stable on your machines? |
|
Yes I get pretty stable results. I tried PR |
|
@max-krasnyansky maybe you can also test on your machines? I'm not sure how to get rid of the noise in the measurement. |
|
@ggerganov can you try again, for |
…hed ⏭️ + TheTom#76 CPU Fusion ⏭️ 3 parallele Tiefen-Evals für Tier-3 Items: AtomicBot-ai#74 Vulkan Descriptor Indexing (Bindless) ❌ VERWORFEN: - Redundant mit TheTom#85 Push Descriptors (✅ implementiert 2026-07-14) - Push Descriptors eliminieren dieselben CPU-Aufrufe - TheTom#85-Benchmark auf Mars RADV: ±0.1-0.3% (Rauschen) - Bindless würde über Push-Descriptors hinaus <0.5% bringen - Workload-Mismatch: Bindless für draw-heavy Rendering, nicht Compute - Mars/Venus bandwidth-bound, nicht descriptor-bound - Aufwand revidiert: 2-4 → 3-5 Wochen (Shader-Rewrite aller .comp-Files) AtomicBot-ai#75 Non-blocking Pipeline Scheduling ⏭️ SPÄTER: - PR ggml-org#19922 closed (2026-03-03, unmerged, 4+ Mo stale) - Fork hat bereits Upstream-Pipeline-Parallelismus - Konflikt mit TheTom#79 TP (✅+23-32% tg, split-mode-exklusiv) - NVIDIA ungetestet, PP-Regression auf 4x MI50 gemeldet - 2-GPU-Setup → geringer Bubble-Hebel - Aufwand revidiert: 3-4 → 4-6 Wochen TheTom#76 CPU Backend Operator Fusion ⏭️ SPÄTER: - RMS_NORM+MUL Fusion bereits im Fork (PR ggml-org#22423 upstream-merged) - MoE Gated FFN riskant: PR ggml-org#20596 zeigt Regressionen auf Consumer-CPUs (M2: 0.98-1.00x, qwen3moe 30B: 0.75-0.96x bei t=2-4) - Nur auf 96-Core-EPYC konsistente Gains (1.04-1.08x) - Styx/Uranus haben Consumer-CPUs → wahrscheinlich Regression - Re-Eval wenn PR ggml-org#20596 gemerged mit Regression-Freiheit
3 participants
Add a special path for gated activation for n_tokens=1, useful when doing
-n-cpu-moeUsing a 5090 with
-ncmoe 999