DeepSeek 4 MTP implementation

[SamuelOliveirads]

Implementation of MTP for DSV4: For some reason, they removed the MTP layer from the original model again which meant that several published GGUFs would need to be reconverted. To avoid this, I used the GGUF that preserves only the MTP layer, similar to how Gemma 4 works.

Here are the benchmark results:

Task	Baseline t/s (mean ± std)	MTP n=2 t/s (mean ± std)	Change	Acceptance
code	10.22 ± 0.89	12.39 ± 0.39	+21.2%	1,860/2,340 — 79.49%
extract	10.64 ± 0.50	11.74 ± 0.77	+10.3%	2,920/3,960 — 73.74%
story	9.62 ± 0.41	11.06 ± 0.20	+15.0%	11,290/17,360 — 65.03%

Arguments used:

$env:GGML_CUDA_NO_PINNED = '1'; & '.\build\bin\llama-spec-bench.exe' -m '<target.gguf>' -ngl 99 --cpu-moe -t 24 --main-gpu 1 --ctx-size 32768 -b 2048 -ub 2048 -fa on --seed 42 --temp 0 --predict 2000 --jinja --repeat 10 --retry 10

-md '<mtp.gguf>' --spec-type 'mtp:n_max=2,p_min=0.0' --spec-ckpt-mode per-step -ngld 99

Some notes:

1. If you want to create a GGUF, use the convert_hf_to_gguf.py --mtp flag to create a model with only the MTP layer.
2. As a result of PR DeepSeek V4 spec checkpoints #2205, use only n_max <= 7 to take advantage of the more optimized workflow for spec-ckpt-mode per-step, otherwise you’ll get a fallback warning for the other, slower methods.
3. For some reason, when I compared the results with and without MTP, I noticed a variation in the logit probabilities, a fluctuation significant enough to skew the outputs. Mainline suffers from the same problem, and I believe it’s due to how CSA/HCA/LID states are managed. I didn’t pinpoint the exact issue and the PR [Feat] DeepSeek V4 Rebased  vllm-project/vllm#40860 with 16k lines of changes didn’t help me much either. If anyone knows why this happened, I can apply fixes later.

[SamuelOliveirads]

Oh, and this is the result from mainline ggml-org/llama.cpp#25784

Prompt	Baseline	MTP n=2	Uplift
code	6.3 t/s	9.1 t/s	+44.4%
extract	9.8 t/s	12.1 t/s	+23.5%
story	9.4 t/s	11.4 t/s	+21.3%

[Nexesenex]

Do you use Q4_0 or Q8_0 for the mtp? (sorry if I didn't catch the info!)

[SamuelOliveirads]

Do you use Q4_0 or Q8_0 for the mtp? (sorry if I didn't catch the info!)

I used DeepSeek-V4-Flash-MXFP4 and DeepSeek-V4-Flash-MTP-Q4_0

[joelfarthing]

3. For some reason, when I compared the results with and without MTP, I noticed a variation in the logit probabilities, a fluctuation significant enough to skew the outputs. Mainline suffers from the same problem, and I believe it’s due to how CSA/HCA/LID states are managed.

I ran into this exact thing with openPangu's MTP a few days ago and spent a while chasing it. It turned out not to be the state management, it was batch-shape numerics in the target forward.

MTP's change of batch shape is enough to make CUDA pick different kernels and reduction orders, so the same row's logits come out slightly different. The delta is tiny, a fraction of a logprob, but it's enough to flip a close greedy tie and send the text down a different branch.

What convinced me: on one MTP-loaded server, compare request-level n_max=0 against n_max=1 with cache_prompt off, then force ub=1. At ub=1 the two go bit-identical (same token IDs and selected logprobs) while drafting stays active with the same accept counts. That pins it to the physical microbatch shape, not acceptance, rollback, or sampler state. Plain no-spec output also changes between ub=2048 and ub=1 on its own, so the effect isn't MTP-specific, which matches your point that mainline shows it too. I built the pre-MTP commit as well and saw the same drift.

One thing that tripped me up early: loading the NextN tensors changes the allocation layout and gives a small token-0 difference even with zero drafts. Comparing n_max=0 vs n_max=1 on the same loaded server removes that.

I wound up concluding that making MTP output bit-identical to no-spec means serializing the target verify back to one token at a time, which is exactly what kills the batched-verify speedup. So it's lossless or fast, not both, unless you treat the near-tie divergence as expected finite-precision variation and just confirm it isn't changing tokens on real prompts.

This is all openPangu, and DS4's CSA/HCA/LID state may be different, so the balance may not be identical. But the ub=1 test should tell you fast whether it's the same batch-shape effect or something in the state handling.

[ikawrakow]

My results are somehow better it seems. I'm getting 28.8 t/s for quicksort (+37%), 28.1 t/s for extract (+34%) and 27.7 t/s for story (+32%), even though acceptance rates are slightly lower than yours (72% for story, 73% for extract, 64% for story).

OK, the above was from just a single run each with llama-server. Here is what I get with llama-spec-bench

task	stage	runs	metric n	tok/s mean/std	rate mean/std	a.len mean/std
code	mtp	10	10	30.55/0.17	80.00%/0.00%	2.60/0.00
extract	mtp	10	10	27.91/0.07	72.34%/0.00%	2.45/0.00
story	mtp	10	10	27.16/0.04	61.74%/0.00%	2.23/0.00

Base performance is about 21 t/s on my system, so between 30% (story) and 45% (code).

[ikawrakow]

Concerning the change in logits: up to a point yes, differences are expected due to the non-associativity of floating point operations and the fact that batch size > 1 changes how floating points operations are performed. You need to provide more details on the difference so we can understand if it is within expected limits, or if there is a real bug.

[ikawrakow]

I have been wondering about this distinction also in previous MTP implementations. Why is it needed? Isn't it so that when mtp_op_type is not MTP_OP_WARMUP or MTP_OP_UPDATE_ACCEPTED it is MTP_OP_GEN_DRAFT, in which case n_tokens = 1, so just

hidden_state = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_hidden, n_tokens);

is 100% equivalent.

[ikawrakow]

and not cb(hidden_state, "inp_mtp_states", -1) ?

[ikawrakow]

I'm not directly comparing with other MTP incarnations, but this seems very familiar. One of these days we should extract this into a function and use that in all MTP graphs.

[SamuelOliveirads]

All three reviews touch on a pattern that appears in all MTP graphs, in that case I believe it would be better to create a subsequent PR that rewrites them. I also agree that they need to be updated, since they were logic I implemented back in the MVP.

[ikawrakow]

Let's have a few more people testing this before merging.

[ikawrakow]

Oh, please fix the merge conflict.

[SamuelOliveirads]

Concerning the change in logits: up to a point yes, differences are expected due to the non-associativity of floating point operations and the fact that batch size > 1 changes how floating points operations are performed. You need to provide more details on the difference so we can understand if it is within expected limits, or if there is a real bug.

When I tested the implementation I used the quicksort prompt with args such as:

GGML_CUDA_NO_PINNED=1
-ngl 99 --cpu-moe -ngld 99 -t 24 --main-gpu 1
--ctx-size 32768 -b 2048 -ub 2048
-fa on
--seed 42 --temp 0 --top-k 1 --top-p 1
DSV4 MTP n_max=2, p_min=0.0

I replayed the exact target verification batch after restoring the pre-verification checkpoint, this replay was bit-identical with checkpoint_replay_max_abs = 0 across six sampled rows. Now, with batched target validation versus sequential target validation the logit prefix produced a difference of around 0.30–1.24. The first recorded top-1 mismatch was row 8 at position 21: token 2910 versus 412. A later accepted verification path selected 645 instead of 15255.

[Ph0rk0z]

I tried to run sweep bench on new ds4-flash around Q2 and with this commit it segfaults. I'm not using MTP and let it allocate pinned memory. Didn't test server yet but assume its the same thing.

No error listed besides segmentation fault. I thought it was the quant, but if I back this out, files work.

[philpax]

I've extracted the MTP head from the original DSv4F, and converted it with ik's converter: https://huggingface.co/philpax/DeepSeek-V4-Flash-MTP-Only-GGUF

I can confirm that it is ~compatible with a 0731 quant when run with a recent ik containing this PR, and that it results in a marginal speedup on my hybrid configuration.