model: add MTP support for Nemotron model - #26725
Merged
Merged
Conversation
am17an
approved these changes
Aug 7, 2026
am17an
left a comment
am17an
left a comment
Contributor
There was a problem hiding this comment.
cpp side changes look ok.
ggerganov
marked this pull request as ready for review
August 7, 2026 18:46
CISC
reviewed
Aug 7, 2026
gaugarg-nv
reviewed
Aug 8, 2026
gaugarg-nv
reviewed
Aug 8, 2026
| Qcur, Kcur, Vcur, nullptr, nullptr, nullptr, kq_scale, il); | ||
| cb(cur, "mtp_attn_out", il); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
The inp_out_ids gather can happen at this line, before the MoE FFN.
We need only attention to run over all n_tokens to fill KV.
This should help improve prefill perf a bit.
Member
Author
There was a problem hiding this comment.
Makes sense. I moved the inp_out_ids gather before the MoE FFN so attention still runs over all tokens while the MoE only processes the kept rows.
Guarded it the same way mimo2 does (crop_before_ffn = inp_out_ids && (!embeddings_nextn || embeddings_nextn_masked)).
CISC
approved these changes
Aug 10, 2026
mlogix
added a commit
to SynoriAI/llama.cpp
that referenced
this pull request
Aug 11, 2026
…, multi-output backend sampling, pocket-tts Merges 45 upstream commits (0865990..ebb546b, b10362-16-gebb546b7e) into synori/llama-update-mtp-fit. Zero conflicts; all vendored patches carried over untouched. Primary motivation — new Meta architecture: * 62bf73d model: Muse Glimmer Support (ggml-org#26841) LLM_ARCH_MUSE_GLIMMER + src/models/muse-glimmer.cpp + the mtmd vision tower in tools/mtmd/models/muse-glimmer.cpp and conversion/muse_glimmer.py. Other notable changes that touch our public API surface: * dd1ea52 llama : support multi-output backend sampling (ggml-org#25532) llama_context_params gains n_outputs_max_per_seq; llama_sampler_i.backend_init takes it as a third argument; new backend_reset / copy_state vtable slots and llama_sampler_copy(). * 153d324 llama : default load-mode auto, avoids mmap on iGPUs (ggml-org#26081) llama_load_mode gains LLAMA_LOAD_MODE_AUTO = -1 (enum is now signed). * 6e62ba5 mtmd: support pocket-tts (ggml-org#26871) mtmd_gen_inp/mtmd_gen_out gain seed/temp/feats/is_eos; new mtmd_gen_inp_default(); mtmd_helper_gen_audio_step_gen() takes out_stop. * 157b81f model : Granite-Switch Architecture (ggml-org#25107) * 7a20b41 model: MTP support for Nemotron (ggml-org#26725) and cc078b4 Dflash support for nemotron-3.5 (ggml-org#26905) * e23e944 vendor : cpp-httplib 0.53.0, 4c6766f vendor : subprocess.h sync Vendored patches preserved: * 919fde3 feat(rpc): thread-local last_error accessor — intact, upstream touched ggml-rpc.cpp by one unrelated line. * 3679b23 Fixes (RPC) — intact. * 2aa76c7 fix(metal): drop stray kernel_pad_f32 — still applies; upstream has since refactored pad into a templated kernel_pad_impl<T>, so the duplicate definition that referenced the nonexistent kargs_pad.s0..s3 is gone on both sides and nothing had to be re-applied.
crusaderky
pushed a commit
to crusaderky/llama.cpp
that referenced
this pull request
Aug 12, 2026
* model: add MTP support for Nemotron Nano model * model: add mtp_flags for nemotron model * address review comments
huaxel
pushed a commit
to huaxel/CachyLLama
that referenced
this pull request
Aug 12, 2026
* model: add MTP support for Nemotron Nano model * model: add mtp_flags for nemotron model * address review comments
Ooooze
pushed a commit
to AtomicBot-ai/atomic-llama-cpp-turboquant-nightly
that referenced
this pull request
Aug 14, 2026
* model: add MTP support for Nemotron Nano model * model: add mtp_flags for nemotron model * address review comments (cherry picked from commit 7a20b41)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR adds MTP support for the Nemotron Nano model. Performance is not yet optimal and depends on PR #26623 being merged first.
I tested this PR on the top of PR #26623, and the performance looks good.
Additional information
Requirements