CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token by ynankani · Pull Request #27621 · ggml-org/llama.cpp

ynankani · 2026-08-23T18:22:38Z

Overview

This PR extends the BS=1 MoE fusions in the CUDA backend to specdec 2-8 tokens, like MTP and Dflash. The main idea is that mul_mat_vec_q_moe kernel now folds the gate projection, biases, the per-expert NVFP4 scales, the gate scale, and the GLU into the up projection, and the topk_moe router fusion(softmax->topk->get_rows earlier only for nrows==1) is extended to higher batches like 4, this can be further increased to 8 so that fusion can be applied to the full block of 8 rows. This gives 2–22% E2E decode speed-up depending on model, quant, and draft width, with the largest gains at draft widths 2 and 4.

Additional information

Below scores are collected on RTX-PRO-BLACKWELL 6000
Seeing more perf benefit with NVFP4, as scale fusion is also done in it

case	width	parent t/s	final t/s	delta	accept parent	accept final
Qwen3.6-35B NVFP4 MTP	1	227.85	250.06	9.80%	0.8085	0.8613
Qwen3.6-35B NVFP4 MTP	2	251	275.86	9.90%	0.7574	0.7212
Qwen3.6-35B NVFP4 MTP	4	258.12	292.63	13.40%	0.5438	0.6007
Qwen3.6-35B NVFP4 dFlash	1	221.99	237.83	7.10%	0.8214	0.8478
Qwen3.6-35B NVFP4 dFlash	2	256.64	279.5	8.90%	0.7212	0.7095
Qwen3.6-35B NVFP4 dFlash	4	270.84	328.9	21.40%	0.503	0.622
nano3.5 NVFP4 MTP	1	157.97	160.78	1.80%	0.748	0.748
nano3.5 NVFP4 MTP	2	138.77	142.54	2.70%	0.5895	0.5895
nano3.5 NVFP4 MTP	3	117.4	122.52	4.40%	0.465	0.465

case	width	parent t/s	final t/s	delta	accept parent	accept final
Qwen3.6-35B Q4_K_M MTP	1	247.17	256.88	3.90%	0.8273	0.854
Qwen3.6-35B Q4_K_M MTP	2	274.67	303.17	10.40%	0.7662	0.775
Qwen3.6-35B Q4_K_M MTP	4	292.52	307.79	5.20%	0.6571	0.6525
Qwen3.6-35B Q4_K_M dFlash	1	248.06	253.92	2.40%	0.8406	0.854
Qwen3.6-35B Q4_K_M dFlash	2	282.23	301	6.70%	0.733	0.7095
Qwen3.6-35B Q4_K_M dFlash	4	273.73	290.14	6.00%	0.4971	0.509

model	Microbatch	parent PPL	final PPL
Qwen3.6-35B Q4_K_M	1	6.2051 ± 0.337	6.2051 ± 0.337
Qwen3.6-35B Q4_K_M	8	6.2172 ± 0.338	6.2019 ± 0.337
Qwen3.6-35B NVFP4	1	6.3644 ± 0.349	6.3644 ± 0.349
Qwen3.6-35B NVFP4	8	6.3826 ± 0.351	6.3766 ± 0.351

I will be collecting more perf data on different hardware (WIP)

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, paired with Claude on this

am17an · 2026-08-24T03:56:25Z

The only reason to not do it earlier was because of the compile times. Can you check whether compile times blow up because of this? If so, we can move mmvq into separate compilation units.

EDIT: Okay disregard above, this is using the MoE kernel created by @gaugarg-nv, so this should not affect compile times

am17an · 2026-08-24T04:01:37Z

+                switch (active_glu) {
+                    case GGML_GLU_OP_SWIGLU:
+                        result *= ggml_cuda_op_silu_single(gate_value);
+                        break;
+                    case GGML_GLU_OP_GEGLU:
+                        result *= ggml_cuda_op_gelu_single(gate_value);
+                        break;
+                    case GGML_GLU_OP_SWIGLU_OAI:
+                        result = ggml_cuda_op_swiglu_oai_single(gate_value, result);
+                        break;
+                    default:
+                        result = result * gate_value;


let's also add sqrtsoftplus used for deepseek4

I might be missing something here, I don't see sqrtsoftplus in the glu op enum

llama.cpp/ggml/include/ggml.h

Line 623 in c060ca9

enum ggml_glu_op {

. Also sqrt_softplus is already fused in topk-moe right?

llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu

Line 2825 in c060ca9

} else if (unary_op == GGML_UNARY_OP_SOFTPLUS && node_idx + 1 < n_nodes &&

Ah okay, then that can be done in a follow-up PR. I was thinking I already added it in #25896

ORippler · 2026-08-24T15:01:29Z

Can we add a test-case for the new multi-token test_topk_moe case?

ynankani · 2026-08-24T19:21:52Z

Can we add a test-case for the new multi-token test_topk_moe case?

Eval test are already there (for 1,4,8,9,22 rows)
https://github.com/ggml-org/llama.cpp/pull/27621/changes#diff-2749fdb8974ec96afa18444a9d546409318b0a862709139b677eee468c479578L10042.

ynankani · 2026-08-25T04:27:26Z

Performance sweep DGX Spark

model	method	width	parent t/s	head t/s	delta
Qwen3.6-35B NVFP4	MTP	1	88.1	90.57	2.80%
Qwen3.6-35B NVFP4	MTP	2	101.87	104.79	2.90%
Qwen3.6-35B NVFP4	MTP	4	103.47	103.6	0.10%
Qwen3.6-35B NVFP4	dFlash	1	81.99	82.39	0.50%
Qwen3.6-35B NVFP4	dFlash	2	99.8	104.54	4.70%
Qwen3.6-35B NVFP4	dFlash	4	112.74	117.99	4.70%
Qwen3.6-35B Q4_K_M	MTP	1	79.12	83.11	5.10%
Qwen3.6-35B Q4_K_M	MTP	2	90.33	91.31	1.10%
Qwen3.6-35B Q4_K_M	MTP	4	91.52	92.33	0.90%
Qwen3.6-35B Q4_K_M	dFlash	1	75.06	75.85	1.00%
Qwen3.6-35B Q4_K_M	dFlash	2	87.41	90.52	3.60%
Qwen3.6-35B Q4_K_M	dFlash	4	96.49	98.05	1.60%
nano3.5 NVFP4	MTP	1	90.61	92.4	2.00%
nano3.5 NVFP4	MTP	2	103.31	105.28	1.90%
nano3.5 NVFP4	MTP	4	105.03	107.36	2.20%

ORippler

Thanks! Some more comments from my side, but looking mostly good

ORippler · 2026-08-25T09:42:09Z

+    // With multiple rows, keep all rows in one block so no row overwrites logits before another row reads them.
+    if (is_topk_moe && ggml_nrows(cgraph->nodes[node_idx]) <= TOPK_MOE_ROWS_PER_BLOCK) {


Please check for memory-aliasing aliasing with * bias (applications outside of llama.cpp may not have bias as run-time constants/weights)

bias is model weight so it can never collide with scratch output right

For external applications you are right, updated to exempt only logits

am17an · 2026-08-31T09:53:35Z

@ynankani please rebase. @ORippler good to merge?

…outer fusion were resticted to 1 token Signed-off-by: ynankani <ynankani@nvidia.com>

Signed-off-by: ynankani <ynankani@nvidia.com>

ORippler

Thanks!

…outer fusion were restricted to 1 token (ggml-org#27621) * CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were resticted to 1 token Signed-off-by: ynankani <ynankani@nvidia.com> * Address review comments Signed-off-by: ynankani <ynankani@nvidia.com> * Add SWIGLU_CLAMP case to multi-token moe fusion Signed-off-by: ynankani <ynankani@nvidia.com> --------- Signed-off-by: ynankani <ynankani@nvidia.com>

… to own fork - cherry-pick upstream ggml-org/llama.cpp#27621 (minus SWIGLU_CLAMP) onto bmoe/expert-ready-hook: fused mul_mat_vec_q_moe handles N>1 batches, topk_moe router 4->8 rows/block, relaxed MMID/alias guards - test-backend-ops: unconditional m_batch {2,4,8} fusion cases + topk boundary cases; 900/900 MUL_MAT_VEC_FUSION + 416/416 TOPK_MOE on CUDA0 - submodule: Helldez/llama.cpp -> OllyJohnston/llama.cpp (public fork), pin 493fe5566 -> 0fe77575e (full fork line incl. port) - docs/seam.md + CHANGELOG 0.24.0 + engine version bump

KV restore batching (ggml-org#27991), kv-cells seq-scan early stop (ggml-org#28011), MOE fusion to specdec + multi-token (ggml-org#27621), mm_ids_helper templated fast path (ggml-org#27978), qwen4exp recurrent state rollback (ggml-org#28123), n_layer_nextn load order (ggml-org#28159), FA K/V XOR-swizzle smem tiles (ggml-org#25635), --lazy-mode -lzm (ggml-org#27837/ggml-org#27969). TQ3/TurboQuant stack and vitriol-* integration auto-merged clean; no conflicts. Experiment E1 of mining-experiment-master-plan-2026-09-01.

…outer fusion were restricted to 1 token (ggml-org#27621) * CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were resticted to 1 token Signed-off-by: ynankani <ynankani@nvidia.com> * Address review comments Signed-off-by: ynankani <ynankani@nvidia.com> * Add SWIGLU_CLAMP case to multi-token moe fusion Signed-off-by: ynankani <ynankani@nvidia.com> --------- Signed-off-by: ynankani <ynankani@nvidia.com>

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 23, 2026

ynankani marked this pull request as ready for review August 24, 2026 03:37

ynankani requested review from a team and ggerganov as code owners August 24, 2026 03:37

am17an reviewed Aug 24, 2026

View reviewed changes

am17an approved these changes Aug 24, 2026

View reviewed changes

github-actions Bot mentioned this pull request Aug 24, 2026

FreeToken ROCm Watch — notification channel Castoff995/FreeToken-rocm-test#1

Draft

ORippler requested changes Aug 25, 2026

View reviewed changes

ynankani force-pushed the ynankani/specdec_moe_fuse branch from 22cf151 to 78a8219 Compare August 25, 2026 17:24

am17an mentioned this pull request Aug 29, 2026

ggml: add SWIGLU_CLAMP #27930

Merged

ynankani added 3 commits August 31, 2026 10:20

CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-r…

7ff3d43

…outer fusion were resticted to 1 token Signed-off-by: ynankani <ynankani@nvidia.com>

Address review comments

0fc74e6

Signed-off-by: ynankani <ynankani@nvidia.com>

Add SWIGLU_CLAMP case to multi-token moe fusion

2373d10

Signed-off-by: ynankani <ynankani@nvidia.com>

ynankani force-pushed the ynankani/specdec_moe_fuse branch from 78a8219 to 2373d10 Compare August 31, 2026 11:04

ORippler approved these changes Aug 31, 2026

View reviewed changes

am17an approved these changes Aug 31, 2026

View reviewed changes

am17an merged commit 41ef91f into ggml-org:master Aug 31, 2026
26 of 30 checks passed

Patt92 mentioned this pull request Aug 31, 2026

Eval bug: CUDA/HIP: MoE models produce garbage output on RDNA3.5 (gfx1151) since #27621 #28113

Closed

		// With multiple rows, keep all rows in one block so no row overwrites logits before another row reads them.
		if (is_topk_moe && ggml_nrows(cgraph->nodes[node_idx]) <= TOPK_MOE_ROWS_PER_BLOCK) {

Uh oh!

Conversation

ynankani commented Aug 23, 2026

Overview

Additional information

Requirements

Uh oh!

am17an commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

am17an Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ynankani Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

am17an Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ORippler commented Aug 24, 2026

Uh oh!

ynankani commented Aug 24, 2026

Uh oh!

ynankani commented Aug 25, 2026

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ORippler Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

ynankani Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

ynankani Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

am17an commented Aug 31, 2026

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

am17an commented Aug 24, 2026 •

edited

Loading

ynankani Aug 24, 2026 •

edited

Loading