llama : load MTP tensors only if they are really used by fairydreaming · Pull Request #26296 · ggml-org/llama.cpp

fairydreaming · 2026-07-29T22:01:58Z

Overview

This PR skips loading of MTP tensors in ~~GLM_DSA arch~~ models that support MTP if no --spec-type draft-mtp was used.

Fixes #26290

Additional information

I added new load_mtp boolean flag to llama model params and model loader. Let me know if there's a better way.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: NO

Sciguy429 · 2026-07-30T14:04:30Z

Just throwing in a suggestion here to make this generic behavior across all models and not just GLM.

This is an issue across any model with integrated MTP support. The tensors are loaded indiscriminately, regardless of speculative settings. Which can be quite annoying for users with hardware which dose not benefit from MTP anyways. The 1-3GB of extra RAM use can actually start to matter if you are running hardware at its very limit.

(The only current solution to this problem is to strip the MTP tensors from a GGUF entirely.)

…rt MTP

fairydreaming · 2026-07-30T20:03:47Z

@Sciguy429 Added remaining models that support MTP. Change not tested yet as I don't have GGUFs at hand.

fairydreaming · 2026-07-31T09:50:02Z

Tested this with Qwen3.6-27B-Q4_K_M.gguf.

MTP enabled

 ./bin/llama-server -m ~/ggufs/Qwen3.6-27B-Q4_K_M.gguf -c $((1*65536)) -b 2048 -ub 2048 -np 1 --no-repack --host 192.168.18.10 --spec-type draft-mtp
0.00.189.677 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.324.440 W srv  llama_server: -----------------
0.00.324.444 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.324.444 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.324.444 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.324.445 W srv  llama_server: -----------------
0.00.325.784 I srv    load_model: loading model '/home/phm/ggufs/Qwen3.6-27B-Q4_K_M.gguf'
0.03.644.559 I common_speculative_init_result: creating MTP draft context against the target model '/home/phm/ggufs/Qwen3.6-27B-Q4_K_M.gguf'
0.03.883.647 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 65536, kv_unified = 'false'
0.04.159.636 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.04.159.678 I srv  llama_server: model loaded
0.04.159.683 I srv  llama_server: listening on http://192.168.18.10:8080
0.22.307.220 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.22.307.414 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.25.688.234 I slot print_timing: id  0 | task 0 | n_decoded =    254, tg =  83.91 t/s, tg_3s =  83.91 t/s
0.28.693.356 I slot print_timing: id  0 | task 0 | n_decoded =    549, tg =  91.01 t/s, tg_3s =  98.17 t/s
0.31.711.038 I slot print_timing: id  0 | task 0 | n_decoded =    851, tg =  94.03 t/s, tg_3s = 100.08 t/s
0.34.741.799 I slot print_timing: id  0 | task 0 | n_decoded =   1188, tg =  98.34 t/s, tg_3s = 111.19 t/s
0.37.744.661 I slot print_timing: id  0 | task 0 | n_decoded =   1463, tg =  96.99 t/s, tg_3s =  91.58 t/s
0.40.753.877 I slot print_timing: id  0 | task 0 | n_decoded =   1741, tg =  96.23 t/s, tg_3s =  92.38 t/s
0.43.769.580 I slot print_timing: id  0 | task 0 | n_decoded =   1975, tg =  93.56 t/s, tg_3s =  77.59 t/s
0.46.797.744 I slot print_timing: id  0 | task 0 | n_decoded =   2196, tg =  90.98 t/s, tg_3s =  72.98 t/s
0.49.803.917 I slot print_timing: id  0 | task 0 | n_decoded =   2451, tg =  90.30 t/s, tg_3s =  84.83 t/s
0.52.821.620 I slot print_timing: id  0 | task 0 | n_decoded =   2706, tg =  89.72 t/s, tg_3s =  84.50 t/s
0.55.856.094 I slot print_timing: id  0 | task 0 | n_decoded =   2981, tg =  89.80 t/s, tg_3s =  90.63 t/s
0.58.862.186 I slot print_timing: id  0 | task 0 | n_decoded =   3304, tg =  91.27 t/s, tg_3s = 107.45 t/s
1.01.867.053 I slot print_timing: id  0 | task 0 | n_decoded =   3638, tg =  92.79 t/s, tg_3s = 111.15 t/s
1.04.874.965 I slot print_timing: id  0 | task 0 | n_decoded =   3952, tg =  93.62 t/s, tg_3s = 104.39 t/s
1.07.891.154 I slot print_timing: id  0 | task 0 | n_decoded =   4231, tg =  93.54 t/s, tg_3s =  92.50 t/s
1.10.165.924 I slot print_timing: id  0 | task 0 | prompt eval time =     353.62 ms /    18 tokens (   19.65 ms per token,    50.90 tokens per second)
1.10.165.928 I slot print_timing: id  0 | task 0 |        eval time =   47504.77 ms /  4408 tokens (   10.78 ms per token,    92.79 tokens per second)
1.10.165.929 I slot print_timing: id  0 | task 0 |       total time =   47858.40 ms /  4426 tokens
1.10.165.933 I slot print_timing: id  0 | task 0 |    graphs reused =       1345
1.10.165.936 I slot print_timing: id  0 | task 0 | draft acceptance = 0.74572 ( 3047 accepted /  4086 generated), mean len =  3.24
1.10.165.983 I slot      release: id  0 | task 0 | stop processing: n_tokens = 4427, truncated = 0
^C1.16.794.038 I srv    operator(): operator(): cleaning up before exit...

VRAM usage: 22594MiB

MTP disabled

$ ./bin/llama-server -m ~/ggufs/Qwen3.6-27B-Q4_K_M.gguf -c $((1*65536)) -b 2048 -ub 2048 -np 1 --no-repack --host 192.168.18.10
0.00.166.710 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.269.367 W srv  llama_server: -----------------
0.00.269.371 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.269.371 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.269.371 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.269.371 W srv  llama_server: -----------------
0.00.270.696 I srv    load_model: loading model '/home/phm/ggufs/Qwen3.6-27B-Q4_K_M.gguf'
0.00.953.358 W model has unused tensor blk.64.attn_norm.weight (size = 20480 bytes) -- ignoring
0.00.953.365 W model has unused tensor blk.64.post_attention_norm.weight (size = 20480 bytes) -- ignoring
0.00.953.369 W model has unused tensor blk.64.attn_q.weight (size = 35389440 bytes) -- ignoring
0.00.953.371 W model has unused tensor blk.64.attn_k.weight (size = 2949120 bytes) -- ignoring
0.00.953.373 W model has unused tensor blk.64.attn_v.weight (size = 4300800 bytes) -- ignoring
0.00.953.379 W model has unused tensor blk.64.attn_output.weight (size = 17694720 bytes) -- ignoring
0.00.953.381 W model has unused tensor blk.64.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.00.953.383 W model has unused tensor blk.64.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.00.953.385 W model has unused tensor blk.64.ffn_gate.weight (size = 50135040 bytes) -- ignoring
0.00.953.387 W model has unused tensor blk.64.ffn_down.weight (size = 73113600 bytes) -- ignoring
0.00.953.389 W model has unused tensor blk.64.ffn_up.weight (size = 50135040 bytes) -- ignoring
0.00.953.392 W model has unused tensor blk.64.nextn.eh_proj.weight (size = 55705600 bytes) -- ignoring
0.00.953.395 W model has unused tensor blk.64.nextn.enorm.weight (size = 20480 bytes) -- ignoring
0.00.953.397 W model has unused tensor blk.64.nextn.hnorm.weight (size = 20480 bytes) -- ignoring
0.00.953.402 W model has unused tensor blk.64.nextn.shared_head_norm.weight (size = 20480 bytes) -- ignoring
0.03.234.820 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 65536, kv_unified = 'false'
0.03.247.641 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.03.247.682 I srv  llama_server: model loaded
0.03.247.685 I srv  llama_server: listening on http://192.168.18.10:8080
0.15.249.152 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.15.249.350 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.18.532.619 I slot print_timing: id  0 | task 0 | n_decoded =    197, tg =  65.43 t/s, tg_3s =  65.43 t/s
0.21.534.254 I slot print_timing: id  0 | task 0 | n_decoded =    391, tg =  65.03 t/s, tg_3s =  64.63 t/s
0.24.546.389 I slot print_timing: id  0 | task 0 | n_decoded =    585, tg =  64.82 t/s, tg_3s =  64.41 t/s
0.27.554.963 I slot print_timing: id  0 | task 0 | n_decoded =    778, tg =  64.65 t/s, tg_3s =  64.15 t/s
0.30.568.345 I slot print_timing: id  0 | task 0 | n_decoded =    971, tg =  64.53 t/s, tg_3s =  64.05 t/s
0.33.580.207 I slot print_timing: id  0 | task 0 | n_decoded =   1163, tg =  64.40 t/s, tg_3s =  63.75 t/s
0.36.587.425 I slot print_timing: id  0 | task 0 | n_decoded =   1354, tg =  64.28 t/s, tg_3s =  63.51 t/s
0.39.590.150 I slot print_timing: id  0 | task 0 | n_decoded =   1544, tg =  64.15 t/s, tg_3s =  63.28 t/s
0.42.600.767 I slot print_timing: id  0 | task 0 | n_decoded =   1734, tg =  64.03 t/s, tg_3s =  63.11 t/s
0.45.609.852 I slot print_timing: id  0 | task 0 | n_decoded =   1923, tg =  63.91 t/s, tg_3s =  62.81 t/s
0.48.611.248 I slot print_timing: id  0 | task 0 | n_decoded =   2111, tg =  63.80 t/s, tg_3s =  62.64 t/s
0.51.621.490 I slot print_timing: id  0 | task 0 | n_decoded =   2299, tg =  63.68 t/s, tg_3s =  62.45 t/s
0.54.636.011 I slot print_timing: id  0 | task 0 | n_decoded =   2487, tg =  63.58 t/s, tg_3s =  62.36 t/s
0.57.646.461 I slot print_timing: id  0 | task 0 | n_decoded =   2674, tg =  63.48 t/s, tg_3s =  62.12 t/s
1.00.649.564 I slot print_timing: id  0 | task 0 | n_decoded =   2860, tg =  63.38 t/s, tg_3s =  61.94 t/s
1.03.660.245 I slot print_timing: id  0 | task 0 | n_decoded =   3046, tg =  63.28 t/s, tg_3s =  61.78 t/s
1.06.665.578 I slot print_timing: id  0 | task 0 | n_decoded =   3231, tg =  63.17 t/s, tg_3s =  61.56 t/s
1.09.677.542 I slot print_timing: id  0 | task 0 | n_decoded =   3416, tg =  63.08 t/s, tg_3s =  61.42 t/s
1.12.679.147 I slot print_timing: id  0 | task 0 | n_decoded =   3600, tg =  62.98 t/s, tg_3s =  61.30 t/s
1.15.689.150 I slot print_timing: id  0 | task 0 | n_decoded =   3784, tg =  62.89 t/s, tg_3s =  61.13 t/s
1.18.696.981 I slot print_timing: id  0 | task 0 | n_decoded =   3967, tg =  62.79 t/s, tg_3s =  60.84 t/s
1.21.697.904 I slot print_timing: id  0 | task 0 | n_decoded =   4149, tg =  62.70 t/s, tg_3s =  60.65 t/s
1.24.700.285 I slot print_timing: id  0 | task 0 | n_decoded =   4331, tg =  62.61 t/s, tg_3s =  60.62 t/s
1.26.272.604 I slot print_timing: id  0 | task 0 | prompt eval time =     272.36 ms /    17 tokens (   16.02 ms per token,    62.42 tokens per second)
1.26.272.608 I slot print_timing: id  0 | task 0 |        eval time =   70750.85 ms /  4426 tokens (   15.99 ms per token,    62.56 tokens per second)
1.26.272.609 I slot print_timing: id  0 | task 0 |       total time =   71023.21 ms /  4443 tokens
1.26.272.614 I slot print_timing: id  0 | task 0 |    graphs reused =       4408
1.26.272.646 I slot      release: id  0 | task 0 | stop processing: n_tokens = 4442, truncated = 0
^C1.29.579.433 I srv    operator(): operator(): cleaning up before exit...

VRAM usage: 21032MiB

IMO looks good.

ggerganov

Would need to update the #9289 when merged.

ggerganov · 2026-07-31T09:52:33Z

    llama_model_loader ml(/*metadata*/ nullptr, /*set_tensor_data*/ nullptr, /*set_tensor_data_ud*/ nullptr,
-        fname_inp, splits, /*file*/ nullptr, /*load_mode*/ load_mode, /*check_tensors*/ true, /*no_alloc*/ false, kv_overrides, nullptr);
+        fname_inp, splits, /*file*/ nullptr, /*load_mode*/ load_mode, /*check_tensors*/ true, /*no_alloc*/ false, /*load_mtp*/ true, kv_overrides, nullptr);


for next PRs: we should refactor these params in a helper struct

am17an

When was this behavior introduced? I remember the layers were not loaded in case mtp was not used.

fairydreaming · 2026-07-31T11:36:06Z

When was this behavior introduced? I remember the layers were not loaded in case mtp was not used.

@am17an Tried to test how it worked in your old mtp-clean branch, but llama-server fails to build without UI and UI fails to build because of HF download failing. So much for the reproducibility of old builds...

Edit: removed HF download from CMakeLists.txt and the build finished.

MTP not enabled: VRAM usage 22526MiB
MTP enabled: VRAM usage 27160MiB

OK that's weird, why so much more than in current master?

pwilkin · 2026-07-31T12:02:40Z

@fairydreaming wrong context calculation, it was fixed a while later.

Sciguy429 · 2026-07-31T12:14:35Z

@fairydreaming sorry, had to redownload a quant to do some testing because most of mine are stripped at this point.

I tested Step3.7 Flash and got similar results to you, notable drop in memory allocation between mainline and this PR. Everything seems to be working.

fairydreaming · 2026-07-31T12:47:42Z

OK I did some more tests with Qwen3.6-27B-Q4_K_M.gguf to make sure this PR works as expected.

An old build from 1st June:

MTP: 22530MiB
No MTP: 21292MiB

Current master:

MTP: 22534MiB
No MTP: 21292MiB

This PR:

MTP: 22534MiB
No MTP: 21016MiB

So VRAM without MTP usage is 276 MiB less than in master/old build from 1st June. One layer of Qwen3.6-27B-Q4_K_M is approximately 16.8 GiB / 64 = 262MiB so close enough.

@am17an Still having doubts?

am17an · 2026-07-31T12:50:08Z

This was always there then, let's merge it and fix the issue

* llama : load MTP tensors only if they are really used * llama : skip loading MTP (if not used) in remaining models that support MTP --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

llama : load MTP tensors only if they are really used

fb6a9cc

github-actions Bot added the model Model specific label Jul 29, 2026

This was referenced Jul 29, 2026

Eval bug: NextN/MTP tensors now load by default for existing GGUFs, no load-time opt-out (regression from #25980) #26290

Closed

model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2) #25980

Merged

llama : skip loading MTP (if not used) in remaining models that suppo…

4499d9f

…rt MTP

fairydreaming marked this pull request as ready for review July 31, 2026 09:50

fairydreaming requested review from a team, CISC and ggerganov as code owners July 31, 2026 09:50

fairydreaming requested a review from am17an July 31, 2026 09:51

ggerganov approved these changes Jul 31, 2026

View reviewed changes

am17an approved these changes Jul 31, 2026

View reviewed changes

fairydreaming merged commit 82dbc4f into ggml-org:master Jul 31, 2026
20 of 26 checks passed

fairydreaming mentioned this pull request Jul 31, 2026

changelog : libllama API #9289

Open

nyo16 mentioned this pull request Aug 1, 2026

Bump llama.cpp to ddd4ec142 (b10217), release v0.8.41 nyo16/llama_cpp_ex#81

Merged

CISC mentioned this pull request Aug 1, 2026

model : only load mimo2 mtp tensors if used #26412

Merged

Navigation Menu

Uh oh!

llama : load MTP tensors only if they are really used - #26296

llama : load MTP tensors only if they are really used#26296
fairydreaming merged 2 commits into
ggml-org:masterfrom
fairydreaming:mtp-no-load-default

fairydreaming commented Jul 29, 2026 •

edited

Loading

Uh oh!

Sciguy429 commented Jul 30, 2026

Uh oh!

fairydreaming commented Jul 30, 2026

Uh oh!

fairydreaming commented Jul 31, 2026

Uh oh!

ggerganov left a comment

Uh oh!

ggerganov Jul 31, 2026

Uh oh!

am17an left a comment

Uh oh!

fairydreaming commented Jul 31, 2026 •

edited

Loading

Uh oh!

pwilkin commented Jul 31, 2026

Uh oh!

Sciguy429 commented Jul 31, 2026

Uh oh!

fairydreaming commented Jul 31, 2026

Uh oh!

am17an commented Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

fairydreaming commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

Sciguy429 commented Jul 30, 2026

Uh oh!

fairydreaming commented Jul 30, 2026

Uh oh!

fairydreaming commented Jul 31, 2026

MTP enabled

MTP disabled

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

am17an left a comment

Choose a reason for hiding this comment

Uh oh!

fairydreaming commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Jul 31, 2026

Uh oh!

Sciguy429 commented Jul 31, 2026

Uh oh!

fairydreaming commented Jul 31, 2026

An old build from 1st June:

Current master:

This PR:

Uh oh!

am17an commented Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fairydreaming commented Jul 29, 2026 •

edited

Loading

fairydreaming commented Jul 31, 2026 •

edited

Loading