DeepseekV4 MTP + DSpark by am17an · Pull Request #25784 · ggml-org/llama.cpp

am17an · 2026-07-16T17:17:19Z

Overview

Note

Deepseek did not ship MTP with the latest deepseek models (0731). Only use DSpark! Here is one https://huggingface.co/am17an/DeepseekV4-Flash-20260731-DSpark/

Roughly ~50% speedup with --spec-draft-n-max 2 with MTP. Currently WIP, will cleanup over the next few days. Note that we the new checkpoint (20260731) only the DSpark head is included. This PR also adds the DSpark for the model.

Additional information

Performance on MTP bench on a DGX spark (with MTP):

Before

  code_python        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=16.4
  code_cpp           pred= 192 draft=   0 acc=   0 rate=n/a tok/s=16.5
  explain_concept    pred= 192 draft=   0 acc=   0 rate=n/a tok/s=16.5
  summarize          pred= 116 draft=   0 acc=   0 rate=n/a tok/s=16.5
  qa_factual         pred= 142 draft=   0 acc=   0 rate=n/a tok/s=16.5
  translation        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=16.5
  creative_short     pred= 176 draft=   0 acc=   0 rate=n/a tok/s=16.1
  stepwise_math      pred= 192 draft=   0 acc=   0 rate=n/a tok/s=16.3
  long_code_review   pred= 192 draft=   0 acc=   0 rate=n/a tok/s=16.5

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1586,
  "total_draft": 0,
  "total_draft_accepted": 0,
  "aggregate_accept_rate": null,
  "wall_s_total": 102.15
}

After

  code_python        pred= 192 draft= 170 acc= 105 rate=0.618 tok/s=25.2
  code_cpp           pred= 192 draft= 167 acc= 107 rate=0.641 tok/s=26.6
  explain_concept    pred= 192 draft= 172 acc= 105 rate=0.611 tok/s=26.2
  summarize          pred= 107 draft=  84 acc=  64 rate=0.762 tok/s=28.2
  qa_factual         pred= 167 draft= 132 acc= 101 rate=0.765 tok/s=29.7
  translation        pred= 192 draft= 163 acc= 109 rate=0.669 tok/s=27.8
  creative_short     pred= 192 draft= 192 acc=  94 rate=0.490 tok/s=23.2
  stepwise_math      pred= 192 draft= 152 acc= 114 rate=0.750 tok/s=29.4
  long_code_review   pred= 192 draft= 163 acc= 109 rate=0.669 tok/s=26.5

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1618,
  "total_draft": 1395,
  "total_draft_accepted": 908,
  "aggregate_accept_rate": 0.6509,
  "wall_s_total": 66.9
}

With DSpark (--spec-type draft-dspark --spec-draft-n-max 5)

  code_python        pred= 192 draft= 261 acc= 137 rate=0.525 tok/s=29.9
  code_cpp           pred= 192 draft= 315 acc= 128 rate=0.406 tok/s=26.9
  explain_concept    pred= 192 draft= 370 acc= 116 rate=0.314 tok/s=22.8
  summarize          pred= 111 draft= 150 acc=  81 rate=0.540 tok/s=33.4
  qa_factual         pred= 157 draft= 210 acc= 115 rate=0.548 tok/s=34.1
  translation        pred= 192 draft= 221 acc= 145 rate=0.656 tok/s=38.2
  creative_short     pred=  77 draft= 185 acc=  40 rate=0.216 tok/s=19.8
  stepwise_math      pred= 192 draft= 234 acc= 144 rate=0.615 tok/s=39.3
  long_code_review   pred= 192 draft= 290 acc= 132 rate=0.455 tok/s=30.0

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1497,
  "total_draft": 2236,
  "total_draft_accepted": 1038,
  "aggregate_accept_rate": 0.4642,
  "wall_s_total": 55.95
}

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, paired with Claude and Codex

kabachuha · 2026-07-16T20:47:01Z

Is there a way to download Deepseek MTP as a module, not as a part of the whole GGUF? Some quants, like Unsloth, have them absent from the ggufs

am17an · 2026-07-17T07:10:48Z

@kabachuha @ddh0 uploaded some MTP only GGUFs https://huggingface.co/ddh0/DeepSeek-V4-Flash-GGUF

jack10768 · 2026-07-17T07:15:02Z

@kabachuha @ddh0 uploaded some MTP only GGUFs https://huggingface.co/ddh0/DeepSeek-V4-Flash-GGUF

Lol, I was just about to convert shard 46 and try your PR. Any idea if it will work for rocm or will I be the one to tell you. On gfx1201 - 4 x radeon 9700, the rest offloaded to ddr4 system ram.

kabachuha · 2026-07-17T10:20:48Z

@am17an Thank you ❤️

For noobs, what commandline flag can I use to enable it in llama.cpp when using a separate file in addition to the main quant?

Edit.: Okay, I think I found out. It's --model-draft

jack10768 · 2026-07-17T16:42:58Z

Thanks @am17an your PR works for me.

Ubuntu 24.06
Kernel - 6.17 hwe, amd-dkms patched with linux 7.0 power scheduler
Rocm 7.2.0

Hardware: Gigabye MC62-g40, 5955wx, 512gb ddr4 ECC ram 3200mhz, 4 x radeon 9700

Used mtp head from https://huggingface.co/ddh0/DeepSeek-V4-Flash-GGUF (Q8_0)

could not fit MTP gguf with fit, could only do so with n-cpu-moe regex:
--fit off
--override-tensor blk.(0|3|6|9|12|15|18|21|24|27|30|33|36|39|42).ffn_(down|gate|up)_exps.weight=CPU

ran with CTX 131072

Prompts - each run got 2 prompts:

simple: give me a brief tutorial on how transformers work in inference with 1 numerical example.
review an agreement (19k tokens)

Ok the rest is drafted by AI because my markdown just sucks:

DeepSeek V4 Flash PR25784 Benchmark (4× Radeon AI PRO R9700)

Hardware

Component	Value
GPUs	4× AMD Radeon AI PRO R9700 (32 GB)
CPU	AMD Threadripper PRO 5955WX
RAM	512 GB DDR4 ECC
Backend	ROCm
Context	131072
Batch / uBatch	2048 / 2048
Flash Attention	Enabled
Pipeline Parallelism	Enabled

Main Model Placement

--fit off
--override-tensor \
'blk\.(0|3|6|9|12|15|18|21|24|27|30|33|36|39|42)\.ffn_(down|gate|up)_exps\.weight=CPU'

This spills 15 MoE expert blocks (~45 GiB) to CPU while keeping all fused DeepSeek V4 kernels enabled.

Results

Configuration	Prompt (t/s)	Decode (t/s)	Acceptance	Mean Accepted	Notes
Baseline (no MTP)	126.17	12.20	—	—	Same placement as MTP
GPU MTP2	124.88	13.20	56.5%	2.13	`spec-draft-n-max=2`
GPU MTP2 + `--spec-draft-p-min 0.5`	124.94	13.45	69.8%	2.26	Best MTP2 result
GPU MTP1	124.81	15.10	76.3%	1.86	Best overall throughput

Relative Decode Speed

Configuration	Speedup vs Baseline
GPU MTP2	+8.2%
GPU MTP2 + `p-min 0.5`	+10.2%
GPU MTP1	+23.8% (different benchmark run; same placement should be used for final confirmation)

Observations

--spec-draft-p-min 0.5 substantially improved MTP2 acceptance (56.5% → 69.8%).
Prompt processing throughput was essentially unchanged (~1% difference).
GPU-resident MTP consistently outperformed CPU-resident MTP.
Spilling one additional MoE expert layer (layer 42) was sufficient to fit the GPU draft model while preserving all fused DeepSeek V4 kernels.
A placement-matched baseline is important when evaluating speculative decoding gains.

Just really wanted to share because there are not many with 4x 9700 out there.

drrros · 2026-07-17T20:21:17Z

could not fit MTP gguf with fit, could only do so with n-cpu-moe regex:

Also had strange issue with fit - model loaded with fit on (used separate draft @ Q8) but it only fitted to 2 GPUs of 3, 2 fist was loaded according to fit-target, but third one was loaded with 5Gbs (out of 24), despite fit-target = 2048,4096,1024

fuzz549 · 2026-07-22T04:40:53Z

With one rtx pro 6000 and one rtx 5060ti, this branch appears to not work unless I put at least some weights on the host RAM (with -ncmoe 1, or -ot "blk.42.ffn_up_exps.weight=CPU"), even though I should have enough VRAM left (6gb on the 6000, and 4gb on the 5060ti with ). Also, it crashes immediately unless I set --fit off.

With -ncmoe 1, it appears to work correctly (acceptance rate is high, mean length >2 for mtp=3).

Command, compiled from 753d015:

./llama-server -m /models/unsloth/DeepSeek-V4-Flash-GGUF/UD-IQ3_XXS/DeepSeek-V4-Flash-UD-IQ3_XXS-00001-of-00004.gguf --model-draft /models/DeepSeek-V4-Flash-MTP-Q4_0.gguf --spec-type draft-mtp -lv 5 -c 16384 -np 1 -ts 38/6  --no-mmap --host 0.0.0.0 --port 9292 --spec-draft-device CUDA0 -fit off -ncmoe 0

Output near crash:

1.21.462.739 I llama_dsv4_comp_state:      CUDA0 DSV4 lid state buffer size =     1.12 MiB
1.21.462.752 I llama_dsv4_comp_state:      CUDA1 DSV4 lid state buffer size =     0.19 MiB
1.21.462.753 I llama_dsv4_comp_state: lid ratio = 4, state = 8 x 256, streams = 1, rs_seq = 3, layers = 21, size =    1.31 MiB
1.21.462.940 D llama_context: enumerating backends
1.21.466.284 D llama_context: backend_ptrs.size() = 3
1.21.476.292 I llama_context: pipeline parallelism enabled
1.21.476.298 I sched_reserve: reserving ...
1.21.476.299 D sched_reserve: max_nodes = 42496
1.21.502.336 D sched_reserve: reserving full memory module
1.21.502.358 D sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
1.21.502.359 D graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
/build/source/ggml/src/ggml-backend.cpp:1347: GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libggml-base.so.0(+0x1b23d) [0x7cbfff94623d]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libggml-base.so.0(ggml_print_backtrace+0x216) [0x7cbfff9466f6]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libggml-base.so.0(ggml_abort+0x144) [0x7cbfff9468b4]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libggml-base.so.0(ggml_backend_sched_split_graph+0x25e6) [0x7cbfff965116]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama.so.0(_ZN13llama_context13graph_reserveEjjjPK22llama_memory_context_ibPm+0x68d) [0x7cbffeef3a2d]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama.so.0(_ZN13llama_context17resolve_fused_opsEPK22llama_memory_context_ij+0x6dd) [0x7cbffeef448d]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama.so.0(_ZN13llama_context13sched_reserveEv+0x67c) [0x7cbffeef504c]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama.so.0(_ZN13llama_contextC1ERK11llama_model20llama_context_params+0xcf1) [0x7cbffeef7791]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama.so.0(llama_init_from_model+0x14a) [0x7cbffeef8b7a]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama-common.so.0(_ZN18common_init_resultC2ER13common_paramsb+0x931) [0x7cbfff61ff31]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama-common.so.0(_Z23common_init_from_paramsR13common_paramsb+0x46) [0x7cbfff622d26]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0x57b) [0x7cbfffbcf24b]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x37c3) [0x7cbfffaf9083]
/nix/store/xx6h8c2993d5k8jnr7gsspsjqfxpp1aw-llama-cpp-cuda-0.0.0/lib/libllama-server-impl.so(_Z12llama_serveriPPc+0xeab) [0x7cbfffafb17b]
/nix/store/ias8xacs1h3jy7xgwi2awvim61k2ji6c-glibc-2.42-67/lib/libc.so.6(+0x2b285) [0x7cbfe362b285]
/nix/store/ias8xacs1h3jy7xgwi2awvim61k2ji6c-glibc-2.42-67/lib/libc.so.6(__libc_start_main+0x88) [0x7cbfe362b338]
builds/result/bin/llama-server(+0x1085) [0x63b9467fb085]

am17an · 2026-07-22T05:52:51Z

@fuzz549 can you try this PR #25832?

fuzz549 · 2026-07-31T03:31:37Z

I assume you have rebased that merged PR into this branch and recompiled. The error is still happening. (Both the immediate crash with --fit on, and the later crash with --fit off).

ggerganov

Should the llama_kv_cache_dsv4::state_write and llama_kv_cache_dsv4::state_read handle the n_rs stuff in some way? My guess is the test-recurrent-state-rollback would fail atm?

anikifoss · 2026-07-31T21:02:04Z

@kabachuha @ddh0 uploaded some MTP only GGUFs https://huggingface.co/ddh0/DeepSeek-V4-Flash-GGUF

These MTP ggufs are working great with DeepSeek-V4-Flash-0731!

mattepiu · 2026-08-03T12:31:56Z

Seems a regression, Single RTX 6000 Blackwell here, (512k context) :

Without DSPARK                 89 tokens 2.9s 30.84 t/s
With DSPARK (n-cpu-moe 16)    146 tokens 8.1s 18.10 t/s

About 94Gb VRAM usage both.
model: UD_Q3_K_XL
Drafters tried:
am17an/DeepseekV4-Flash-20260731-DSpark,
alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF
draft-n-max tried: 2, 3, 5

createthis · 2026-08-03T14:36:59Z

Single RTX 6000 pro 96gb here with 768gb of system ram. Built commit 563dec.

Here it is with dspark disabled:

Details

# NUMA NPS=4
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
echo 3 | sudo tee /proc/sys/vm/drop_caches

./build/bin/llama-server \
    --model /data2/bullerwins/DeepSeek-V4-Flash-0731-GGUF/DeepSeek-V4-Flash-0731-MXFP4_MOE-Q8_0.gguf \
    --alias DeepSeek-V4-Flash-0731:MXFP4_MOE-Q8_0 \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26)\.ffn_.*=CUDA0,exps=CPU" \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 1.0 \
    --top-p 1.0 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --port 11434

0.19.035.971 I slot print_timing: id  3 | task 0 | prompt eval time =     154.13 ms /     5 tokens (   30.83 ms per token,    32.44 tokens per second)
0.19.035.975 I slot print_timing: id  3 | task 0 |        eval time =    3277.54 ms /   139 tokens (   23.58 ms per token,    42.41 tokens per second)
0.19.035.975 I slot print_timing: id  3 | task 0 |       total time =    3431.68 ms /   144 tokens

And here it is with dspark enabled:

Details

# NUMA NPS=4
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
echo 3 | sudo tee /proc/sys/vm/drop_caches

./build/bin/llama-server \
    --model /data2/bullerwins/DeepSeek-V4-Flash-0731-GGUF/DeepSeek-V4-Flash-0731-MXFP4_MOE-Q8_0.gguf \
    --alias DeepSeek-V4-Flash-0731:MXFP4_MOE-Q8_0 \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26)\.ffn_.*=CUDA0,exps=CPU" \
    -ub 4096 -b 4096 \
    --spec-type draft-dspark \
    --spec-draft-n-max 5 \
    --fit off \
    --seed 3407 \
    --temp 1.0 \
    --top-p 1.0 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --port 11434

0.20.068.375 I slot print_timing: id  3 | task 0 | prompt eval time =     159.72 ms /     5 tokens (   31.94 ms per token,    31.30 tokens per second)
0.20.068.379 I slot print_timing: id  3 | task 0 |        eval time =    3657.26 ms /   139 tokens (   26.31 ms per token,    38.01 tokens per second)
0.20.068.380 I slot print_timing: id  3 | task 0 |       total time =    3816.98 ms /   144 tokens

Same simple prompt, just test.

Slower with dspark. Is this because I'm running a hybrid ram + vram system?

am17an · 2026-08-03T14:39:25Z

@createthis Probably yes, but you can try to fully load the drafter on the GPU and see if it improves things

createthis · 2026-08-03T18:20:19Z

@am17an I wasn't specifying --model-draft so I don't think it was working anyway. I also added --n-gpu-layers-draft 99.

With dspark:

Details

./build/bin/llama-server \
    --model /data2/unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --model-draft /data2/unsloth/DeepSeek-V4-Flash-0731-GGUF/dspark/dspark-DeepSeek-V4-Flash-0731-BF16.gguf \
    --alias DeepSeek-V4-Flash-0731:UD-Q8_K_XL \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    --n-gpu-layers-draft 99 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21)\.ffn_.*=CUDA0,exps=CPU" \
    -ub 4096 -b 4096 \
    --spec-type draft-dspark \
    --spec-draft-n-max 5 \
    --fit off \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --port 11434

0.42.061.896 I slot print_timing: id  3 | task 0 | prompt eval time =     454.35 ms /     5 tokens (   90.87 ms per token,    11.00 tokens per second)
0.42.061.900 I slot print_timing: id  3 | task 0 |        eval time =    7755.19 ms /   158 tokens (   49.08 ms per token,    20.37 tokens per second)
0.42.061.901 I slot print_timing: id  3 | task 0 |       total time =    8209.54 ms /   163 tokens

Without dspark:

Details

./build/bin/llama-server \
    --model /data2/unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --alias DeepSeek-V4-Flash-0731:UD-Q8_K_XL \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24)\.ffn_.*=CUDA0,exps=CPU" \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --port 11434

0.28.401.326 I slot print_timing: id  3 | task 0 | prompt eval time =     205.57 ms /     5 tokens (   41.11 ms per token,    24.32 tokens per second)
0.28.401.330 I slot print_timing: id  3 | task 0 |        eval time =    8298.51 ms /   203 tokens (   40.88 ms per token,    24.46 tokens per second)
0.28.401.330 I slot print_timing: id  3 | task 0 |       total time =    8504.09 ms /   208 tokens

However, with UD-IQ1_S where it mostly fits on the GPU I can see something happening.

With dspark and a longer 5k prompt:

Details

./build/bin/llama-server \
    --model /data2/unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-IQ1_S/DeepSeek-V4-Flash-0731-UD-IQ1_S-00001-of-00003.gguf \
    --model-draft /data2/unsloth/DeepSeek-V4-Flash-0731-GGUF/dspark/dspark-DeepSeek-V4-Flash-0731-BF16.gguf \
    --alias DeepSeek-V4-Flash-0731:UD-IQ1_S \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    --n-gpu-layers-draft 99 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42)\.ffn_.*=CUDA0,exps=CPU" \
    -ub 4096 -b 4096 \
    --spec-type draft-dspark \
    --spec-draft-n-max 5 \
    --fit off \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --port 11434

1.59.390.131 I slot print_timing: id  3 | task 0 | prompt eval time =    3787.09 ms /  5301 tokens (    0.71 ms per token,  1399.76 tokens per second)
1.59.390.138 I slot print_timing: id  3 | task 0 |        eval time =   83854.39 ms /  6871 tokens (   12.20 ms per token,    81.94 tokens per second)
1.59.390.139 I slot print_timing: id  3 | task 0 |       total time =   87641.48 ms / 12172 tokens

Without dspark:

Details

./build/bin/llama-server \
    --model /data2/unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-IQ1_S/DeepSeek-V4-Flash-0731-UD-IQ1_S-00001-of-00003.gguf \
    --alias DeepSeek-V4-Flash-0731:UD-IQ1_S \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42)\.ffn_.*=CUDA0,exps=CPU" \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --port 11434

2.07.918.874 I slot print_timing: id  3 | task 0 | prompt eval time =    2832.24 ms /  5301 tokens (    0.53 ms per token,  1871.66 tokens per second)
2.07.918.877 I slot print_timing: id  3 | task 0 |        eval time =  111437.50 ms /  6963 tokens (   16.00 ms per token,    62.48 tokens per second)
2.07.918.877 I slot print_timing: id  3 | task 0 |       total time =  114269.74 ms / 12264 tokens

This is the expected result?

am17an · 2026-08-03T18:25:29Z

@createthis I've posted my results in the PR description which is when the model and the drafter are both on the GPU. I have not measured various tensor offloads

d-shehu · 2026-08-03T18:46:52Z

I'm puzzled why this model runs ~ 16% slower with dspark. It doesn't seem to be pure memory bandwidth. In contrast, running Qwen 3.5 397 with MTP and offloading MOE to CPU, I got a 25-50% boost (from 8 t/s to 12 t/s). And that's a larger model with more layers in RAM across 2 machines.

Also as of b10229 offloading layers with RPC is still brittle and prone to crashing. I've tested with single server (Vulkan) and I see crashes likewise. ROCm always crashes unlike other models.

Perhaps this arch is more solid with Cuda?

transwarp829 · 2026-08-03T21:21:07Z

Sorry for disturbing but I just want to share my results on this. My setup is RTX 4090 + EPYC 7B13 with 4 channel DDR4, all MoE weights offloaded to CPU.

TL;DR: on my setup, --spec-draft-n-max 1 works best on general tasks. For tasks that are likely to produce higher acceptance rate, can try --spec-draft-n-max 2. --spec-draft-n-max > 2 will result in performance regression.
For the drafter, it is MoE so can apply -cmoed to save VRAM. Different -ncmoed parameter seems to have no observable influence on performance.

Here is the command I use to run with or without (just remove the last few lines) DSpark:

Details

./llama-server `
-m "C:\Users\transwarp829\.lmstudio\models\unsloth\DeepSeek-V4-Flash-0731-GGUF\DeepSeek-V4-Flash-0731-UD-IQ3_S-00001-of-00004.gguf" `
-a "DeepSeek-V4-Flash-0731-UD-IQ3_S" `
-t 48 `
-c 393216 `
-b 4096 `
-ub 4096 `
-fa 1 `
-ctk bf16 `
-ctv bf16 `
-lm mlock `
-cmoe  `
-ngl 999 `
--temp 1.0 `
--top_p 1.0 `
-np 1 `
-kvu `
--fit off `
-lv 4 `
-cmoed `
--spec-draft-n-max 2 `
-ngld all `
-md "C:\Users\transwarp829\.lmstudio\models\am17an\DeepseekV4-Flash-20260731-DSpark.gguf" `
--spec-type draft-dspark `

Baseline speed without DSpark:

7.10.663.972 I slot print_timing: id  0 | task 0 | prompt eval time =   20573.13 ms /  5636 tokens (    3.65 ms per token,   273.95 tokens per second)
7.10.663.983 I slot print_timing: id  0 | task 0 |        eval time =  305686.97 ms /  4010 tokens (   76.23 ms per token,    13.12 tokens per second)

Tests on `-ncmoed` parameters:

(to be able to fully load the drafter in VRAM, here parameters are changed to -c 8192 -b 1024 -ub 1024, with --spec-draft-n-max 5)

Details

Drafter fully on GPU:

7.02.328.811 I slot print_timing: id  0 | task 0 | prompt eval time =   35447.91 ms /  5636 tokens (    6.29 ms per token,   158.99 tokens per second)
7.02.328.823 I slot print_timing: id  0 | task 0 |        eval time =  254165.70 ms /  2556 tokens (   99.44 ms per token,    10.06 tokens per second)

-ncmoed 1:

6.31.124.127 I slot print_timing: id  0 | task 0 | prompt eval time =   34698.09 ms /  5636 tokens (    6.16 ms per token,   162.43 tokens per second)
6.31.124.139 I slot print_timing: id  0 | task 0 |        eval time =  256935.85 ms /  2553 tokens (  100.64 ms per token,     9.94 tokens per second)

-ncmoed 2:

7.03.327.018 I slot print_timing: id  0 | task 0 | prompt eval time =   35626.56 ms /  5636 tokens (    6.32 ms per token,   158.20 tokens per second)
7.03.327.030 I slot print_timing: id  0 | task 0 |        eval time =  250877.81 ms /  2556 tokens (   98.15 ms per token,    10.19 tokens per second)

-ncmoed 3:

5.50.186.623 I slot print_timing: id  0 | task 0 | prompt eval time =   34868.11 ms /  5636 tokens (    6.19 ms per token,   161.64 tokens per second)
5.50.186.632 I slot print_timing: id  0 | task 0 |        eval time =  255685.37 ms /  2552 tokens (  100.19 ms per token,     9.98 tokens per second)

It seems that the performance is nearly identical with different `-ncmoed` parameter on my setup. So just offload them to CPU to save VRAM.

Tests on `--spec-draft-n-max`:

Details

Task	`--spec-draft-n-max` setting	Token generation speed	Draft acceptance rate	Speedup compared to baseline
translation	`--spec-draft-n-max 1`	14.53 t/s	0.80699	+10.7%
translation	`--spec-draft-n-max 2`	15.03 t/s	0.69690	+14.6%
translation	`--spec-draft-n-max 3`	12.49 t/s	0.51393	-4.8%
translation	`--spec-draft-n-max 4`	11.36 t/s	0.49870	-13.4%
translation	`--spec-draft-n-max 5`	8.79 t/s	0.34143	-33.0%
code review	`--spec-draft-n-max 1`	13.57t/s	0.70157	+3.4%
code review	`--spec-draft-n-max 2`	12.75t/s	0.56278	-2.8%
code review	`--spec-draft-n-max 2` (another round)	13.37t/s	0.61761	+1.9%
code review	`--spec-draft-n-max 5`	9.07t/s	0.35388	-30.9%
creative writing	`--spec-draft-n-max 1`	13.20t/s	0.62031	+0.6%
creative writing	`--spec-draft-n-max 1` (another round)	12.86t/s	0.60234	-2.0%
creative writing	`--spec-draft-n-max 2`	11.79t/s	0.45682	-10.1%

acceptance per position (translation) = (0.767, 0.488, 0.264, 0.137, 0.051)
acceptance per position (code review) = (0.692, 0.458, 0.295, 0.197, 0.128)
acceptance per position (creative writing) = (0.594, 0.319), --spec-draft-n-max >2 not tested

There could be large variations between different round of tests. Generally, --spec-draft-n-max 1 works best for me. For tasks that is likely to produce higher acceptance rate, can try --spec-draft-n-max 2. --spec-draft-n-max > 2 will result in performance regression.

kelheor · 2026-08-04T16:38:30Z

@transwarp829

Can you specify your RAM? I tried to reproduce your config but 128Gb definitely not enough for that.

transwarp829 · 2026-08-04T18:16:55Z

@transwarp829

Can you specify your RAM? I tried to reproduce your config but 128Gb definitely not enough for that.

My RAM is 256GB. With other daily programs in background, typical total usage is ~145GB, with ~109GB (seems increasing with longer context) usage from llama-server.exe reported by Windows Task Manager, but I am not sure if it is accurate.

clort81 · 2026-08-07T22:02:03Z

Roughly ~50% speedup with --spec-draft-n-max 2 with MTP.

Can't get this working.

What speed-up is achieved on full weight (162GB) model with 2x3090 + 128GB SDRAM?

https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF/resolve/main/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf

Merges 251 upstream commits on top of the fork's 392. Base was 22b208b (2026-07-15). What this brings in for DeepSeek V4: - CUDA kernels for the hyper-connection ops and the lightning indexer (dsv4-hc.cu, lightning-indexer.cu, upstream ggml-org#25585 and ggml-org#25545). These landed upstream after our base, so the graph no longer needs a CPU fallback for those ops. - MTP and DSpark support (ggml-org#25784), the wo_a reshape fix on load, and the same-K/V-cache-type enforcement (ggml-org#25871). - Exclusion of the i32 ffn_gate_tid2eid routing table from quantization, which the fork did not carry. Conflict resolution kept both architectures everywhere the two sides touched the same code: - llama-kv-cache: kept the fork's default-off attention-rotation policy and its env overrides, took upstream's GLM_DSA addition to the DSA indexer arch list. - llama-context: moved the TurboQuant flash-attention auto-enable above upstream's generic quantized-V check, which would otherwise reject turbo cache types under -fa off, and dropped the fork's older V-cache check in favour of upstream's. - mmq.cuh: kept the fork's int64 offsets in all three of upstream's new NVFP4 branches. - fattn.cu: dropped the WMMA block, since upstream removed that kernel and its helpers entirely; kept the RDNA4 turbo path. - ggml-cuda.cu: kept the host-staged cross-device copy and routed its peer copy through upstream's new virtual-to-physical device mapping. - chat.cpp: rebuilt on upstream's file with the fork's Inkling and Laguna parsers and the leading-whitespace tolerance reapplied; thinking_end_tag became thinking_end_tags upstream. - laguna.cpp/laguna.py and mtmd-image.cpp: took upstream, which already carries the fork's own upstreamed review fixes plus later refinements. - Removed the inherited upstream workflows again, per 0c9a069. GGML_OP_COUNT is 103: upstream's 101 plus the fork's TURBO_WHT and FLASH_ATTN_EXT_BANDED. Also drops a duplicate LLM_ARCH_LAGUNA case in test-llama-archs that the merge would otherwise have left in moe_mandatory.

…ge-0 router MY MERGE ERROR, not a pre-existing bug. Two self-consistent designs got stitched into a hybrid that was neither: OURS (pre-merge) target exports UNCOLLAPSED taps, hc_mult*n_embd wide; the DSpark encoder graph collapses them with ggml_mean before fc. UPSTREAM (ggml-org#25784) target collapses AT THE TAP via dsv4_hc_mean, so taps are n_embd wide and fc consumes them directly. The merge took upstream's dsv4_hc_mean in deepseek4.cpp but left our hc_mult-wide consumers in common/speculative.cpp, because that file auto-merged with no conflict and I never diffed it against the design change I had accepted next door. Same failure mode that silently dropped 171 lines of router-fleet config earlier today: a clean auto-merge means the TEXT did not overlap, not that the SEMANTICS compose. Symptom: the gather read 16384 floats per layer from a tap that fills only 4096, so 12288 floats of uninitialised memory per layer per token went into DSpark stage 0 -- and the worker's validation guard caught it: E process_ubatch: expert dispatch failed: worker :8803 rejected expert(s) 1,3,5 on layer 43 with code 2: pipe: expert dispatch has a non-finite weight E llama_decode: failed to decode, ret = -3 Adopting upstream's collapse-at-the-tap end to end. Width chain now: tap 4096 -> buffer stride 4096 -> gather stride 4096 -> n_embd_enc 3*4096=12288 -> fc input 12288 (fc.weight is (12288, 4096) in the GGUF) speculative.cpp:964 n_embd_enc drops hc_mult speculative.cpp:1091 gather stride drops hc_mult llama-context.cpp layer_inp_size for DEEPSEEK4 is n_embd, matching the tap; output_reserve, extract_layer_inputs and output_reorder verified to agree on that width deepseek4.cpp encoder graph feeds fc directly; the reshape/permute/mean was a SECOND collapse over already-collapsed data Worth noting the EAGLE3 gather at speculative.cpp:620 already used the collapsed n_embd_tgt and was correct throughout. The DSpark impl beside it carried the hc-wide assumption. The two are now structurally identical on this point, which is the smell test that the merge is finally coherent rather than merely quiet. Also moves 4x less data across the extraction path. Builds clean: build-hip llama + llama-server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lz74kRRC7s6a3hSMLbf1Gt

74 upstream commits, 14 in our paths. Two textual conflicts, both expected, plus ONE silent-drift resolution that did NOT conflict -- which is the whole reason upstream-check.sh warns about auto-merges. ☠☠ fattn.cpp -- THE ONE THAT AUTO-MERGED CLEAN (upstream 66fa168 / ggml-org#25874) ggml-org#25874 extends oneDNN SDPA to non-F16 KV (Q4_0..Q8_0) and inserts its gate ABOVE the MKL gate -- which our 0801 sync had already deferred to MMA for exactly this reason. Its envelope (Q->ne[1] >= 32, K->ne[1] >= 1024, quantized KV now accepted) matches our deploy prefill exactly: q8_0 KV, ubatch 2048, any real depth. Verbatim it would have silently swapped the measured MMA kernel (2.87x served prefill at 43k) for an unmeasured one and staged the whole q8_0 KV cache to dense F16 first. ⇒ added the SAME conjunct, same wording, as the MKL gate below it: !ggml_sycl_fattn_mma_supported(dst). oneDNN keeps its full envelope everywhere MMA declines. A/B reachable both ways without a rebuild (GGML_SYCL_FATTN_MMA=0 / GGML_SYCL_FA_ONEDNN=0 -- the latter is checked inside _supported(), so it still governs the new early return). This is a precedence choice, not a revert, and it is UNMEASURED on this box. B59 is the finding that prices it. The three conflicts git DID flag in this file were cosmetic (debug kname lines + upstream re-calling get_best_fattn_kernel where we hoist it). Kept ours, added upstream's ONEDNN kname lines. concat.cpp -- upstream 6c8dcaa (ggml-org#25852) is a duplicate of our own fix Kept OURS: strict superset (same launch geometry + the GGML_SYCL_CONCAT_WG door + the o[dim] hoist + the i64 loop var). Ours deliberately omits their WARP_SIZE floor, which would make WG=1 unreachable and destroy the one-binary positive control for the 8.36x launch fix. Noted in the source so the next sync does not resolve toward upstream. Assessed, no action needed: 272700b (ggml-org#26105) iGPU classification -- NULL here, B70 is discrete and still reports TYPE_GPU. Relevant later for PVC/Max-1100 and B51. dbadb68 (ggml-org#22789) dynamic split-graph inputs -- mechanical, auto-merged beside our GGML_SCHED_HANDOFF_CENSUS. 596a579 (ggml-org#25784) DeepseekV4 MTP + DSpark -- ⚠ changes SHARED MTP plumbing (n_embd_out vs n_embd_inp in llama_context::decode, set_embeddings_layer_inp bounds, embeddings_layer_inp sized n_layer+1). We run MTP every step. NOT yet gated -- see the build/gate that follows. ☠ NOT YET BUILT, NOT YET GATED. Every absolute in f306 predates this merge.

github-actions Bot added model Model specific server conversion labels Jul 16, 2026

ggerganov self-assigned this Jul 16, 2026

am17an force-pushed the dsv4-mtp branch from 079dbc9 to 1c5915a Compare July 17, 2026 08:28

am17an marked this pull request as ready for review July 21, 2026 03:52

am17an requested review from a team, CISC and ggerganov as code owners July 21, 2026 03:52

am17an force-pushed the dsv4-mtp branch from 1c5915a to 38174d8 Compare July 21, 2026 04:06

am17an requested a review from a team as a code owner July 29, 2026 07:18

github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Jul 29, 2026

am17an changed the title ~~[WIP] DeepseekV4 MTP~~ DeepseekV4 MTP Jul 29, 2026

am17an force-pushed the dsv4-mtp branch from 50e655f to d171a39 Compare July 29, 2026 16:03

SamuelOliveirads mentioned this pull request Jul 30, 2026

DeepSeek 4 MTP implementation ikawrakow/ik_llama.cpp#2216

Merged

ggerganov reviewed Jul 31, 2026

View reviewed changes

Comment thread src/llama-kv-cache.h Outdated

ggerganov reviewed Jul 31, 2026

View reviewed changes

Comment thread src/llama-kv-cache-dsv4.h

ggerganov reviewed Jul 31, 2026

View reviewed changes

am17an mentioned this pull request Jul 31, 2026

Deepseek4 #26369

Open

13 tasks

am17an force-pushed the dsv4-mtp branch from d171a39 to 7c1c77e Compare August 1, 2026 05:27

github-actions Bot mentioned this pull request Aug 3, 2026

Reddit News Daily 2026-08-03 gitlawr/reddit-daily-news#325

Open

TheTom pushed a commit to TheTom/llama-cpp-turboquant that referenced this pull request Aug 3, 2026

DeepseekV4 MTP + DSpark (ggml-org#25784)

07e01ec

noonghunna mentioned this pull request Aug 3, 2026

fattn: stale D=512 guard forces VEC for batched attention — compiled MMA templates unreachable (~2.4× prefill on DSv4, f16 path) spiritbuun/buun-llama-cpp#98

Closed

relic664 mentioned this pull request Aug 3, 2026

Eval bug: using -devd CUDA0 on draft-dspark causes model to crash #26475

Open

TheTom mentioned this pull request Aug 3, 2026

Sync fork with upstream master (adds deepseek4 / DeepSeek-V4 support) TheTom/llama-cpp-turboquant#257

Closed

belarusian pushed a commit to belarusian/llama.cpp that referenced this pull request Aug 4, 2026

DeepseekV4 MTP + DSpark (ggml-org#25784)

f36f498

ruixiang63 mentioned this pull request Aug 4, 2026

spec: support DSpark for Gemma4 #25549

Open

smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026

DeepseekV4 MTP + DSpark (ggml-org#25784)

5177811

noonghunna mentioned this pull request Aug 6, 2026

bump(llama.cpp): server-cuda-b9967 → b10236 (DSpark; unblocks DeepSeek-Flash slugs) noonghunna/club-3090#904

Merged

This comment has been minimized.

Sign in to view

Skelectric mentioned this pull request Aug 7, 2026

Feature Request: Add DSpark speculative decoding support ikawrakow/ik_llama.cpp#2267

Closed

4 tasks

satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 11, 2026

DeepseekV4 MTP + DSpark (ggml-org#25784)

210c3db

satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 12, 2026

DeepseekV4 MTP + DSpark (ggml-org#25784)

558d2f7

cap10bad mentioned this pull request Aug 13, 2026

Eval bug: DeepSeek-V4-Flash degenerates into repetition and leaks special tokens in long agentic chats (Metal, b10289) #26694

Open

This was referenced Aug 14, 2026

WE WANT DSPARK lmstudio-ai/lmstudio-bug-tracker#2281

Open

[Feature] WE WANT DSPARK unslothai/unsloth#8848

Closed

thermodynamic-retard mentioned this pull request Aug 14, 2026

Library (in-process) hosts cannot use speculative decoding / DSpark — no core C-API for draft/spec params #27089

Open

Uh oh!

Conversation

am17an commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

kabachuha commented Jul 16, 2026

Uh oh!

am17an commented Jul 17, 2026

Uh oh!

jack10768 commented Jul 17, 2026

Uh oh!

kabachuha commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jack10768 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DeepSeek V4 Flash PR25784 Benchmark (4× Radeon AI PRO R9700)

Hardware

Main Model Placement

Results

Relative Decode Speed

Observations

Uh oh!

drrros commented Jul 17, 2026

Uh oh!

fuzz549 commented Jul 22, 2026

Uh oh!

am17an commented Jul 22, 2026

Uh oh!

fuzz549 commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

anikifoss commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattepiu commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

createthis commented Aug 3, 2026

Uh oh!

am17an commented Aug 3, 2026

Uh oh!

createthis commented Aug 3, 2026

Uh oh!

am17an commented Aug 3, 2026

Uh oh!

d-shehu commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

transwarp829 commented Aug 3, 2026

Baseline speed without DSpark:

Tests on -ncmoed parameters:

Tests on --spec-draft-n-max:

Uh oh!

kelheor commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

transwarp829 commented Aug 4, 2026

Uh oh!

This comment has been minimized.

clort81 commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What speed-up is achieved on full weight (162GB) model with 2x3090 + 128GB SDRAM?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

am17an commented Jul 16, 2026 •

edited

Loading

kabachuha commented Jul 17, 2026 •

edited

Loading

jack10768 commented Jul 17, 2026 •

edited

Loading

fuzz549 commented Jul 31, 2026 •

edited

Loading

anikifoss commented Jul 31, 2026 •

edited

Loading

mattepiu commented Aug 3, 2026 •

edited

Loading

d-shehu commented Aug 3, 2026 •

edited

Loading

Tests on `-ncmoed` parameters:

Tests on `--spec-draft-n-max`:

kelheor commented Aug 4, 2026 •

edited

Loading

clort81 commented Aug 7, 2026 •

edited

Loading