speculative : draft performance improvement (+10% t/s with deeper draft depth possible with this) + token rollback bugfix

[PatrickWalther]

Overview

Implemented draft improvements that make higher draft depth faster. For my setup, draft depth 3 was the best. With the changes, draft depth 5 delivers ~10% more t/s than draft depth 3.

My setup: Qwen3.8-27B Q8_0, 2x RTX 5090, llama.cpp draft-mtp speculative decoding, -sm tensor -fa on -c 32768 -np 1, temperature 0.

change	how it works	t/s change / improvement
Draft all tokens in one decode (LLAMA_SPEC_CHAIN=1)	Before: one GPU call per draft token, plus a CPU round trip after each call to pick the token. Now: one GPU call produces all draft tokens and picks them on the GPU.	~+2-6 t/s
Return 2 numbers per draft token instead of the full score list	The GPU sends back only the picked token and its probability, not scores for all 151k vocab entries. The draft also scores only the 32k most common tokens (LLAMA_SPEC_CHAIN_SUB); the main model still checks against the full vocabulary, so output does not change.	~+5-11 t/s
Keep a full copy of the output layer on each GPU (LLAMA_META_MIRROR_OUTPUT=1)	Before, the layer was split across the two GPUs and every check step needed GPU-to-GPU traffic to combine the halves. Costs one layer of extra VRAM per GPU, removes that traffic.	~+5-8 t/s
Keep a prepared work plan per batch size (LLAMA_SCHED_POOL=N)	The same few batch sizes repeat every round. Before, each one overwrote the previous one's plan, so almost every step re-planned from scratch.	~+5 t/s
Remember how work was split across the GPUs	Splitting a step's work between the two GPUs was recomputed every time. A repeated step now reuses the stored split.	No negative impact and makes the other improvement possible
Remove extra synchronization from the GPU-to-GPU add	The two GPUs already wait for each other inside the kernel; the extra per-chunk sync events only added overhead.	~+3 t/s
Bug fix: wrong state after undoing tokens	When the server rolled tokens back, short batches had overwritten the saved states it rolled back to.	Bugfix

Additional information

"Remember how work was split across the GPUs" and "Remove extra synchronization" are always-on, other changes are behind a flag.

Benchmarks:

Test script starts each binary with (-sm tensor -fa on -c 32768 -np 1 --spec-type draft-mtp, depth per config), GGML_CUDA_PDL=1,
temperature 0.

It runs four fixed workloads - code_gen (fresh code),
code_edit (modify given code), code_echo (verbatim rewrite), prose -
and takes generation t/s from the server's own timings.

Config	Depth	Round results	Mean
Upstream b10444, official binary	3	143.6, 149.6	146.6
Upstream b10441, local toolchain (control)	3	143.2, 141.2	142.2
PR build, chain config	5	161.8, 158.9	160.4

With no environment variables set, behavior is unchanged

Chain config = LLAMA_SPEC_CHAIN=1 LLAMA_META_MIRROR_OUTPUT=1 LLAMA_SCHED_POOL=8, --spec-draft-n-max 5.

Category	Control n3	PR chain n5	Change
code_gen	138.2	148.4	+7.4%
code_edit	137.7	150.6	+9.4%
code_echo	169.9	217.4	+28.0%
prose	123.0	125.3	+1.9%
mixed avg	142.2	160.4	+12.8%

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Used AI coding agent to search for code that had the highest chance to yield performance gains, had it implement the code in this PR under my review and test it against benchmarks.

[ggml-gh-bot]

Hi @PatrickWalther, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[pich]

This was the largest runtime improvement in my Qwen3.8-27B experiment.

I applied commit 2c9cb8f on top of my #26001 + #26048 + #26705 runtime. With the same model, command, prompt and ten-run deterministic workload:

before #27173: 45.866 tok/s
with #27173: 55.402 tok/s
incremental gain: +9.536 tok/s, +20.79%
gain versus clean b10454: +9.980 tok/s, +21.97%

Each arm produced one stable response hash. Weighted draft acceptance changed from 42.46% to 37.59%, but throughput still increased sharply because chained drafting removed enough per-token GPU launches and synchronization overhead to make n_max=8 useful.

In a separate greedy gate, target-only decoding reached 21.189 tok/s and embedded MTP reached 59.456 tok/s, or 2.81x. I am not mixing that separate result into the 21.97% runtime A/B.

Hardware: RTX PRO 4000 Blackwell SFF, sm120a, 24,467 MiB, CUDA 12.9.86, driver 610.57.04. Context 262,144, Q4_0 target KV, F16 draft KV, batch 512, ubatch 256, four recurrent checkpoints.

Full write-up:
https://piszczek.pl/blog/qwen38-27b-256k-50-tps-24gb-gpu

Released model:
https://huggingface.co/cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF

[PatrickWalther]

ggml/src/ggml-backend-meta.cpp:605:77: warning: cast from 'const ggml_tensor *' to 'ggml_tensor *' drops const qualifier [-Wcast-qual]

Should be fixed with meta : fix -Wcast-qual warnings

[am17an]

create 1 PR for each change if you intend to get any of this merged

[Flo5k5]

Validation on V100 + multi-turn function calling — fix works, but ~3.4× throughput regression

Tested this PR against the exact scenario of #27296 (intermittent stream cancellation + corrupted tool-call args in multi-turn function calling with --spec-type draft-mtp).

Setup: 2× Tesla V100-PCIE-32GB, CUDA 11.5, master f280b269 (Aug 24) + this PR, Qwen3.8-27B Q8_0, -c 524288 -np 2 -ctk/ctv q8_0 -fa on --jinja --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.3.

Production incident (unpatched): Aug 25, on a multi-turn FC agent session (~155k ctx, tool loop running 21 min), the server canceled the stream mid-thinking — observed twice 5s apart (client SDK retry killed as well). This matches the #27296 / #26425 family.

With this PR: behaviorally clean on identical harness scenarios (valid tool-call JSON, no cancellations) — no functional regression found.

However, generation throughput drops ~3.4×: identical prompt/sampling, thinking-heavy generation:

• unpatched same-day refs: 50.0 / 51.0 tok/s (draft acceptance ~64%)
• this PR: 15.2 / 14.7 / 14.3 tok/s (acceptance ~51%, draft tokens emitted nearly double: ~1043 drafts for 800 generated tokens vs ~770)

The deferred/chained catch-up path appears to make each speculative pass much more expensive on this setup. Would it be possible to make the chaining opt-in (or otherwise reduce its cost)? My reading is the delta-net rollback fix itself (preserving deep snapshot slots) is separable from the chaining refactor — the former is what fixes #27296 for us, the latter is what costs the throughput.

Happy to re-bench any revision on this V100 rig and share logs.

[Flo5k5]

Follow-up: minimal variant of this fix without the chaining — same functional outcome, no throughput regression

Following up on my perf comment above: we isolated the delta-net snapshot fix from the chaining refactor, and added a small complementary reset. This 21-line variant (on top of master f280b269) covers the cross-request contamination mechanism described in #27296 / #26425:

1. src/models/delta-net-base.cpp — the snapshot-slot preservation from this PR (deep rollback states no longer overwritten by short clamped batches). Extracted as-is.
2. common/speculative.cpp — when begin() detects the new prompt does not extend the slot's draft state (pos_max < N-1), reset the per-sequence MTP carryover (pending_h / verify_h / verify_h_rows). Without this, the stale h-row of a previous request gets paired with the first token of the new conversation — which is the "prompt leaks across prompts" / degraded-drafts path.

Measured on the rig from my previous comment (V100 ×2, Qwen3.8-27B, draft-mtp n-max 4):

build	multi-turn FC	throughput
master f280b269	stream canceled mid-tool-call (~daily in production)	50 tok/s
this PR (full)	clean	14.7 tok/s (÷3.4)
minimal variant below	clean (5/5 valid tool-call rounds)	47.9 / 46.5 tok/s

So on this hardware the entire regression comes from the deferred/chained catch-up path, not from the delta-net fix itself. Would a split (rollback fix + optional chaining behind a flag) be acceptable? Happy to turn this into a proper PR with a test if that helps.

diff --git a/common/speculative.cpp b/common/speculative.cpp
index 4eef2212..c0bd599b 100644
--- a/common/speculative.cpp
+++ b/common/speculative.cpp
@@ -1428,6 +1428,16 @@ struct common_speculative_impl_draft_mtp : public common_speculative_impl {
                     "(need_embd / logits=1 on every prompt position?). "
                     "Drafts may degrade.\n",
                     (int) pos_max, N - 1);
+
+            // [local fix-min #27296] the new prompt does not extend this slot draft
+            // state: the per-sequence MTP carryover belongs to a previous request.
+            // Keep it from poisoning the first catch-up row pairing (stale h-row
+            // paired with the first token of an unrelated conversation).
+            if (pending_h.size() > (size_t) seq_id) {
+                pending_h[seq_id].assign(n_embd, 0.0f);
+                verify_h[seq_id].clear();
+                verify_h_rows[seq_id] = 0;
+            }
         }
     }
 
diff --git a/src/models/delta-net-base.cpp b/src/models/delta-net-base.cpp
index ad661264..99591bc3 100644
--- a/src/models/delta-net-base.cpp
+++ b/src/models/delta-net-base.cpp
@@ -499,10 +499,18 @@ ggml_tensor * llm_build_delta_net_base::build_conv_state(
         // this logic assumes that the last (n_rs_seq + 1) tokens of a sequence in a batch are inside
         //   the same ubatch, which `split_equal()` guarantees via its n_keep_tail argument
 
-        const int64_t K = (int64_t) cparams.n_rs_seq + 1;
+        const int64_t n_tok = conv_input->ne[0] - conv_states->ne[0];
+
+        // slot s holds the conv state as of s tokens back. A batch with n_tok tokens defines
+        // slots 0..n_tok. Deeper slots keep their previous content: an earlier, larger batch
+        // may have written them, and a rollback that returns to that batch's position still
+        // reads them (test-recurrent-state-rollback). The old full-depth loop overwrote those
+        // slots with clamped duplicates of slot n_tok. The gated-delta-net snapshot write has
+        // the same bound.
+        const int64_t K = std::min<int64_t>((int64_t) cparams.n_rs_seq, n_tok) + 1;
 
         for (int64_t t = 1; t <= K; ++t) {
-            const int64_t s_idx  = std::max<int64_t>(0, conv_input->ne[0] - conv_states->ne[0] - K + t);
+            const int64_t s_idx  = n_tok - K + t;
             const int64_t s_slot = K - t;
 
             ggml_tensor * conv_state_last =

[remeh]

@Flo5k5 FYI I've tried to apply the patch of your last comment (in speculative.cpp and delta-net-base.cpp) but it looks like I still have poisoning across prompts with only this change.

[47Hunter47]

Reference baseline from a single RTX 3090: Qwen3.6-27B IQ4_XS, draft-mtp n-max 4, FA on, q4_0/q8_0 KV, 130K context, mem clock locked at 5001: ~45 t/s generation, 38-71% acceptance depending on context.

Happy to test this PR on our end and report the delta. The regression report in the thread so far is V100-only, so an Ampere data point might help triage.