speculative-simple : support MTP speculative decoding by pdhinaka · Pull Request #25679 · ggml-org/llama.cpp

pdhinaka · 2026-07-14T19:01:48Z

Overview

Add Gemma4 MTP support to llama-speculative-simple.

I have read and agree with the contributing guidelines
AI usage disclosure:

Yes, AI was used to draft initial code changes which were later reviewed and updated by a human contributor.

pdhinaka · 2026-07-14T19:03:49Z

Example Run:

Gemma 4 E2B:

adb shell "cd /data/local/tmp/llama.cpp; ulimit -c unlimited; \
  LD_LIBRARY_PATH=./lib ADSP_LIBRARY_PATH=./lib ./bin/llama-speculative-simple --no-mmap \
    -m  /data/local/tmp/gguf/gemma-4-E2B_q4_0-it.gguf \
    -md /data/local/tmp/gguf/gemma-4-E2B-it-assistant-Q4_0.gguf \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    -fa on -ngl 99 --ctx-size 8192 --device HTP0 --spec-draft-device none \
    -t 6 --seed 42 -n 300 -p 'Explain how a transformer neural network works.'"
0.00.881.003 W common_get_device_memory_data_impl: device HTP0 did not report memory; --fit will not use it
0.00.960.819 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.01.497.152 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.500.179 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.511.639 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.10.247.418 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.10.250.135 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.10.260.118 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.10.655.599 W llama_kv_cache: layer   3: sharing with layer 34. k = 0x7c61343000, v = 0x7c61b43000
0.10.655.622 W llama_kv_cache: layer   0: sharing with layer 33. k = 0x7d3dc75000, v = 0x7d3dcf5000
0.10.655.624 W llama_kv_cache: layer   1: sharing with layer 33. k = 0x7d3dc75000, v = 0x7d3dcf5000
0.10.655.626 W llama_kv_cache: layer   2: sharing with layer 33. k = 0x7d3dc75000, v = 0x7d3dcf5000


<bos>Explain how a transformer neural network works.

A Transformer neural network is a type of neural network architecture that revolutionized sequence-to-sequence (seq2seq) tasks. Unlike recurrent neural networks (RNNs) that process data sequentially, Transformers use a mechanism called self-attention to allow them to process all parts of the input at once, making them highly parallelizable and capable of capturing long-range dependencies effectively.

A typical Transformer architecture consists of an **Encoder** and a **Decoder**, both stacks of identical layers.

### 1. The Encoder

The Encoder is a module that processes the input sequence (e.g., an English sentence) and transforms it into a rich, contextual representation.

*   **Input Processing:** The input is split into individual tokens (words or sub-words), and each token is embedded into a vector space.
*   **Self-Attention Mechanism:** This is the core of the Encoder. For every token in the input, the self-attention mechanism calculates how much "attention" it should pay to all other tokens in the sequence. This allows the network to build a contextual representation where each output vector is informed by the entire input sequence simultaneously.
*   **Feed-Forward Network:** After the attention mechanism, the output passes through a standard feed-forward network, which further transforms the representation.
*   **Stack of Layers:** The Encoder is typically composed of multiple identical blocks stacked sequentially. This deep stack allows for hierarchical feature extraction.

### 2. The Decoder

The



0.29.624.572 I encoded    9 tokens in    0.130 seconds, speed:   69.127 t/s
0.29.624.573 I decoded  302 tokens in   18.754 seconds, speed:   16.103 t/s
0.29.624.573 I
0.29.624.573 I n_draft   = 3
0.29.624.574 I n_predict = 302
0.29.624.574 I n_drafted = 441
0.29.624.574 I n_accept  = 155
0.29.624.575 I accept    = 35.147%
0.29.624.575 I
0.29.624.575 I draft:

0.29.624.575 I
0.29.624.576 I target:

0.29.624.577 I common_perf_print:    sampling time =     297.19 ms
0.29.624.578 I common_perf_print:    samplers time =     223.03 ms /   302 tokens
0.29.624.580 I common_perf_print:        load time =    8248.58 ms
0.29.624.582 I common_perf_print: prompt eval time =   14985.00 ms /   598 tokens (   25.06 ms per token,    39.91 tokens per second)
0.29.624.583 I common_perf_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
0.29.624.584 I common_perf_print:       total time =   20088.10 ms /   599 tokens
0.29.624.585 I common_perf_print: unaccounted time =    4805.91 ms /  23.9 %      (total - sampling - prompt eval - eval) / (total)
0.29.624.585 I common_perf_print:    graphs reused =        146

ggerganov · 2026-07-15T08:36:58Z

+
+        const bool spec_mtp = std::find(params.speculative.types.begin(),
+                                        params.speculative.types.end(),
+                                        COMMON_SPECULATIVE_TYPE_DRAFT_MTP) != params.speculative.types.end();
+        if (spec_mtp) {
+            cparams.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
+        }
+        cparams.n_rs_seq  = 0;
+        cparams.ctx_other = ctx_tgt;   // ctx_tgt already loaded above (line 44)
+


With the recent refactor in #25056, it should be possible to simplify this logic here for loading the draft model. Want to give it a try? We can also do it in a follow-up PR.

Thanks, I can see that logic now lives in common. I can switch this block to use the methods from that refactor. I can update this PR with those changes and run my on-device tests again.

I have just pushed a new commit that updates the draft model loading logic to use the common helper functions.

ggerganov · 2026-07-28T09:03:44Z

    // check if the context supports partial sequence removal
    const bool use_ckpt_tgt = (common_context_can_seq_rm(ctx_tgt)       == COMMON_CONTEXT_SEQ_RM_TYPE_FULL);
-    const bool use_ckpt_dft = (common_context_can_seq_rm(ctx_dft.get()) == COMMON_CONTEXT_SEQ_RM_TYPE_FULL);
+    const bool use_ckpt_dft = (common_context_can_seq_rm(ctx_dft) == COMMON_CONTEXT_SEQ_RM_TYPE_FULL);


Fix the vertical alignment here

ggerganov · 2026-07-28T09:03:49Z

    // eval the prompt
    llama_decode(ctx_tgt,       llama_batch_get_one(inp.data(), inp.size() - 1));
-    llama_decode(ctx_dft.get(), llama_batch_get_one(inp.data(), inp.size() - 1));
+    llama_decode(ctx_dft, llama_batch_get_one(inp.data(), inp.size() - 1));


ggerganov · 2026-07-28T09:03:56Z


            llama_memory_seq_rm(llama_get_memory(ctx_tgt),       seq_id, n_past, -1);
-            llama_memory_seq_rm(llama_get_memory(ctx_dft.get()), seq_id, n_past, -1);
+            llama_memory_seq_rm(llama_get_memory(ctx_dft), seq_id, n_past, -1);


Need to take an extra look - the is_mem_shared should not be necessary

ggerganov · 2026-08-11T13:05:32Z

Superseded by #26904

pdhinaka requested a review from ggerganov as a code owner July 14, 2026 19:01

github-actions Bot added the examples label Jul 14, 2026

ggerganov self-assigned this Jul 14, 2026

ggerganov reviewed Jul 15, 2026

View reviewed changes

theIvanR mentioned this pull request Jul 15, 2026

Eval bug: MTP decoding crash on pre-Ampere GPUs (with working patch!) #25713

Open

pdhinaka force-pushed the spec-simple-mtp-fix branch from 5c28281 to a016595 Compare July 17, 2026 19:22

ggerganov previously approved these changes Jul 28, 2026

View reviewed changes

pdhinaka added 3 commits July 31, 2026 14:00

speculative-simple : support MTP speculative decoding

ccf844f

speculative-simple : now uses common helpers for draft model loading

746ec87

speculative-simple : fix MTP prompt prefill corrupting shared KV cache

2016af9

pdhinaka force-pushed the spec-simple-mtp-fix branch from a016595 to 2016af9 Compare July 31, 2026 21:01

ggerganov closed this Aug 11, 2026

Navigation Menu

Uh oh!

speculative-simple : support MTP speculative decoding - #25679

speculative-simple : support MTP speculative decoding#25679
pdhinaka wants to merge 3 commits into
ggml-org:masterfrom
qualcomm:spec-simple-mtp-fix

pdhinaka commented Jul 14, 2026

Uh oh!

pdhinaka commented Jul 14, 2026 •

edited

Loading

Uh oh!

ggerganov Jul 15, 2026

Uh oh!

pdhinaka Jul 16, 2026 •

edited

Loading

Uh oh!

pdhinaka Jul 17, 2026

Uh oh!

ggerganov Jul 28, 2026

Uh oh!

ggerganov Jul 28, 2026

Uh oh!

ggerganov Jul 28, 2026

Uh oh!

ggerganov commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

pdhinaka commented Jul 14, 2026

Overview

Uh oh!

pdhinaka commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example Run:

Gemma 4 E2B:

Uh oh!

ggerganov Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

pdhinaka Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pdhinaka Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pdhinaka commented Jul 14, 2026 •

edited

Loading

pdhinaka Jul 16, 2026 •

edited

Loading