ggml: add GATED_DELTA_NET op

[am17an]

Add CPU/CUDA impl for GATED_DELTA_NET used in qwen3next and a lot of upcoming recent attention models. This is a basic vector impl and not the chunking impl, although this should work for n_tokens > 1 as a reference implementation. I tested this vs build_delta_net_autoregressive and the results were good. I plan to add the chunked implementation for CPU and CUDA.

master:

model	size	params	backend	threads	fa	test	t/s
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CPU	16	1	tg32	4.77 ± 0.03
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CPU	16	1	tg32 @ d1024	4.55 ± 0.13

sched_reserve: graph nodes = 14990 (with bs=512), 6242 (with bs=1)

ggml_op_gated_delta_net added to the qwen3next graph (not added in the PR)

model	size	params	backend	threads	fa	test	t/s
qwen35moe ?B Q4_K - Small	18.55 GiB	34.66 B	CPU	16	1	tg32	11.08 ± 0.20
qwen35moe ?B Q4_K - Small	18.55 GiB	34.66 B	CPU	16	1	tg32 @ d1024	11.21 ± 0.07

sched_reserve: graph nodes = 14990 (with bs=512), 5342 (with bs=1)

[ggerganov]

I think it is too early to implement the dedicated delta net ops. There are still many things to optimize in the existing implementation (you can keep track of my progress in #19375). After that we have to consolidate the KDA version of the delta net (#18792). Btw the l2 norm should not be part of this OP - fixed in my branch. Also not sure how to handle the 2 variants of this operator (autoregressive and chunked).

So I think we can experiment with a dedicated op in a branch, but merging this in master will likely take time.

[am17an]

@ggerganov I defer to your judgement, my thinking was that qwen3.5 is already a major model series, so even if the op is just for that model it makes sense.

for KDA, AFAIK it the gate is a matrix, so it will just be another dot product instead of a scale. For chunk vs autoregressive, we have the vec FA path for CPU which now serves a reference kernel. I was thinking it would be the same here, the autoregressive kernel remains the simple kernel while chunking is the optimisation, both solve the same recurrence.

[ggerganov]

Ok, let's prototype a branch that also has this op together with the CUDA implementation rebased on #19375. I will then add the Metal version of the kernel and from there we can consider a quicker merge if things are looking good. Also, want to see if having this op will allow the CUDA graphs to be more easily enabled.

[pwilkin]

So this is basically what the Transformers implementations have as the "recurrent" implementation, right? No chunking, just iterating over tokens.

[am17an]

@pwilkin yes, just calculating the recurrence token by token

[ggerganov]

Btw, should also consider small batch sizes larger than 1 to be handled by this operator too. I'm not sure where the break-even point would be, but I imagine that processing a few tokens auto-regressively (i.e. more than 1 and less than ~16) would be more efficient compared to the chunking path. Also don't forget that dim 3 will handle separate sequences - though from a quick look, this implementation already accounts for that.

[am17an]

Btw, should also consider small batch sizes larger than 1 to be handled by this operator too. I'm not sure where the break-even point would be, but I imagine that processing a few tokens auto-regressively (i.e. more than 1 and less than ~16) would be more efficient compared to the chunking path.

Yes for small amount of tokens we can just run a loop even in CUDA. I have not looked into the chunked impl yet, but I will invest some time in finding the breakeven point

Also don't forget that dim 3 will handle separate sequences - though from a quick look, this implementation already accounts for that.

I think this should be fine, the work is split among dim1 * dim3 (heads * sequences)

[ymcki]

Great performance gain for inference. Looking forward to seeing your implementation done for the major backends.

If you have plan to do the chunking version as well, it will be great if it is based on the block implementation done at fla.

https://github.com/fla-org/flash-linear-attention/blob/main/fla/ops/kda/chunk_intra_token_parallel.py
https://github.com/fla-org/flash-linear-attention/blob/main/fla/ops/kda/chunk_intra.py

[pwilkin]

Looks clean to me. Are you planning on doing the chunking version here as well, or separate op / PR?

[ggerganov]

Converted to draft since I am not sure if my comment was clear: #19504 (comment). First we will be prototyping a new branch and after that we will consider adding the new op.

[pwilkin]

Should we use this PR or will you create a dedicated branch?

[am17an]

@ggerganov I removed the norm, and also added the autoregressive cuda op in 01eda69, it passes test-backend-ops. I have not done the rebase on #19375

[ggerganov]

Just a heads up, I will be rebasing the #19375 branch from time to time. Hope it's not a big issue. Just always put your commits on top. I'm hoping to merge in a day or two.

[am17an]

I did a quick perf test this PR + #19375 + replacing the autoregressive for qwen3next with gated_delta_net. On a 5090

Details

master

model	size	params	backend	ngl	fa	test	t/s
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32	83.92 ± 0.39
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d1024	84.45 ± 0.36
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d2048	84.20 ± 0.61
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d4096	83.82 ± 0.56
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d8192	83.43 ± 1.73
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d16384	83.56 ± 0.47

PR:

model	size	params	backend	ngl	fa	test	t/s
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32	105.95 ± 0.36
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d1024	105.05 ± 0.91
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d2048	105.33 ± 0.42
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d4096	105.10 ± 0.50
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d8192	98.13 ± 1.79
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d16384	97.22 ± 0.49

[ggerganov]

For reference, what do you get with CUDA graphs forced enabled:

diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
index f3d8317e1..605cb3ed4 100644
--- a/ggml/src/ggml-cuda/ggml-cuda.cu
+++ b/ggml/src/ggml-cuda/ggml-cuda.cu
@@ -2894,7 +2894,7 @@ static bool ggml_cuda_graph_check_compability(ggml_cgraph * cgraph) {
 #endif
         }
 
-        if (node->op == GGML_OP_ADD &&
+        if (false && node->op == GGML_OP_ADD &&
             node->src[1] && node->src[1]->ne[1] > 1 &&
             (node->src[0] ? node->src[0]->name != gemma3n_per_layer_proj_src0_name : true) &&
             (node->src[1] ? node->src[1]->name != gemma3n_per_layer_proj_src1_name : true) &&

[am17an]

With force enabled CUDA graphs

Details

model	size	params	backend	ngl	fa	test	t/s
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32	111.89 ± 2.48
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d1024	135.26 ± 6.93
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d2048	135.89 ± 4.95
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d4096	134.77 ± 4.67
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d8192	123.30 ± 6.07
qwen3next 80B.A3B Q2_K - Medium	27.12 GiB	79.67 B	CUDA	99	1	tg32 @ d16384	121.22 ± 5.28

[am17an]

I see actually a huge difference in PP on CPU when just using the autoregressive kernel instead of the current one i.e. just use the fused op regardless of n_tokens. But I think I will optimize this later

[jacekpoplawski]

Great speedup on tg (Qwen Next and Qwen 3.5)!

  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes

[jeffbolznv]

Hi @ProgenyAlpha, just wanted to check whether you still plan to submit a PR for the vulkan backend support.

[CISC]

Huh, not sure exactly what's happening, but MUSA build is now throwing an ICE:
https://github.com/ggml-org/llama.cpp/actions/runs/22774203231/job/66063122971#step:6:416

Edit: This killed our Docker release as well:
https://github.com/ggml-org/llama.cpp/actions/runs/22814140371/job/66176342700#step:9:900

[am17an]

@CISC not sure who maintains the MUSA backend, but it seems like a compiler bug

[ggerganov]

@yeahdongcn PTAL at the MUSA issue above.

@am17an In the meantime we can change supports_op to return false for MUSA

[yeahdongcn]

@yeahdongcn PTAL at the MUSA issue above.

@am17an In the meantime we can change supports_op to return false for MUSA

No problem. I'll try a local build first and see if I should open an internal ticket. Thanks!

[ProgenyAlpha]

Hi @ProgenyAlpha, just wanted to check whether you still plan to submit a PR for the vulkan backend support.

I wasn't sure where the thread was going so I wanted to let you guys cook and see how things unfolded before I jump back in. I'll rebase and work on that this week if I have time. Thanks for pinging me!