common: support --models-dir loading MTP assistant models - #24431
Merged
Conversation
ngxson
requested changes
Jun 10, 2026
Contributor
Author
|
A gentle ping for visibility. With this PR, one can use auto-detection of MTP assistant models without requiring to duplicate |
Contributor
Author
|
Any updates? I've changed MTP model name check to be prefix only. |
EZForever
marked this pull request as draft
July 11, 2026 09:30
EZForever
marked this pull request as ready for review
August 4, 2026 03:46
ngxson
reviewed
Aug 14, 2026
| common_file_info model_file; | ||
| common_file_info first_shard_file; | ||
| common_file_info mmproj_file; | ||
| common_file_info mtp_file; |
Collaborator
There was a problem hiding this comment.
Suggested change
| common_file_info mtp_file; | |
| common_file_info draft_file; |
| if (string_ends_with(file.name, ".gguf")) { | ||
| if (file.name.find("mmproj") != std::string::npos) { | ||
| mmproj_file = file; | ||
| } else if (file.name.rfind("mtp-", 0) != std::string::npos) { |
Collaborator
There was a problem hiding this comment.
maybe also handle dflash/dspark
ngxson
approved these changes
Aug 14, 2026
Collaborator
|
@EZForever can merge once you confirm that it works |
Contributor
Author
|
Tested with |
CowboyTim
pushed a commit
to aardbeiplantje/llama.cpp
that referenced
this pull request
Aug 16, 2026
…4431) * common: support --models-dir loading MTP assistant models * common: preset: check for MTP models with strict prefix * common: preset: Take advantage of PR ggml-org#27005 * handle other draft types * drop eagle3 * clean up --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
brittlewis12
pushed a commit
to brittlewis12/llama.cpp
that referenced
this pull request
Aug 17, 2026
…4431) * common: support --models-dir loading MTP assistant models * common: preset: check for MTP models with strict prefix * common: preset: Take advantage of PR ggml-org#27005 * handle other draft types * drop eagle3 * clean up --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
fewtarius
added a commit
to fewtarius/CachyLLama
that referenced
this pull request
Aug 24, 2026
Brings 218 upstream commits into CachyLLama without losing any of our features. Key carried-over changes from upstream: - llama.cpp v0.2.0 / ggml v0.21.0 version bumps - Vulkan FA MMQ fp32 scaling (ggml-org#27413), PAD_REFLECT_1D (ggml-org#26586), tiled transpose (ggml-org#26585), null checks in queue command pools cleanup (ggml-org#27353) - ggml: rope_set_offset on multiple backends, recurrent state rollback - Vulkan coopmat1 SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for Intel Xe - server: LLAMA_SERVER_SLOTS_N_DIFF (ggml-org#27600), /metrics during llama_decode (ggml-org#27041), index.html no-cache (ggml-org#27006), make-release workflow - model: MiniMax-M1/Text01 (ggml-org#27018), Kimi-K3 (ggml-org#26185), BailingMoE3 (ggml-org#26608), GraniteSWA (ggml-org#25505), GLM-4.5-Air MTP, DSV4 tensor split (-sm tensor) - ui: Chat Conversation Tabbed navigation, settings refactor - common: --models-dir loading MTP assistant models (ggml-org#24431), --load-mode replacing --mmap (ggml-org#26934), json.h abstraction (ggml-org#27511) - vendor: cpp-httplib 0.53.1, BoringSSL 0.20260813.0, vendor/hash CachyLLama features preserved through conflict resolution: - Persistent SSD-backed KV cache (3-tier hot/warm/cold + system prompt cache) - Per-user isolation (user_id, per-user concurrency cap, slot affinity) - MoE expert residency + co-activation tracking - CachyLLama Vulkan Lightning Indexer (108/108 on Strix Halo) + DSV4 hyper-connection fused ops + DSV4 sparse FA + coopmat shaders - FA quant-KV dequant-once + f16 contiguize (with host-RAM safety gate) - DFlash framework + Laguna-S-2.1 model support - DFlash d2t reduced-vocab draft support (upstream merge) - Context checkpoint ring buffer + SWA skip + memory budget scaling - Stable-prefix LCP gate + prompt_stable_prefix_tokens param - conv_hash conversation-boundary detection - All CachyLLama Vulkan shaders (concat_transpose, lightning_indexer, mmid_row_lists, flash_attn_top_k, dequant_f16_transpose) - common::host_available_ram() utility - llama-moe-residency + llama-moe-coact modules Manual conflict resolution touches: src/models/dflash.cpp (DFlash d2t + aux_norm), src/llama-kv-cache-dsv4.cpp (state snapshot fix), src/llama- memory-recurrent.cpp (rs_idx bounds check), src/llama-model-saver.cpp (DSV4 compress_ratios + swiglu_clamp sizing), ggml/src/ggml-vulkan/ {ggml-vulkan.cpp,vulkan-shaders-gen.cpp,vulkan-shaders/dequant_q8_0. comp,vulkan-shaders/flash_attn.comp,vulkan-shaders/copy_transpose_02. comp} (CachyLLama shader registration + FA scratch gate), ggml/src/ ggml-cuda/mmvq.cu (RDNA3_5 + GB10 enum), gguf-py/gguf/constants.py (DFlash ENC_AUX_NORM + D2T tensors), tests/{CMakeLists.txt,test-backend- ops.cpp,test-llama-archs.cpp,test-recurrent-state-rollback.cpp} (test additions), tools/{CMakeLists.txt,server/*} (server_batch embd support + spec_is_replay + user_id routing + MCP servers + CORS), and docs/{AGENTS.md,README.md} (kept CachyLLama branding). Verified: full build succeeds, test-backend-ops Vulkan LIGHTNING_INDEXER + FLASH_ATTN pass on Strix Halo. Based on a re-merge from the 20260824 (pristine pre-merge) branch after a previous agent's merge attempt produced an unbuildable state from -X ours that wiped shader float-typing and broke the dequant_q8_0 + flash_attn shaders with redefinition errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Make
--models-dircorrectly identify MTP assistant models (mtp-*.ggufas generated by conversion scripts) and load them correctly as such, similar to how mmproj models gets discovered and loaded automatically.Additional information
The current logic is not aware of
mtp-*.ggufbeing a companion model like mmproj, and will sometimes try to load it as the main model, causing model load errors.Tested with latest master and gemma-4-E4B-it-qat UD-Q4_K_XL:
before.log
after.log
NOTE: The model didn't actually load successfully in the after log due to an assert hit in later stages. It is not caused by this PR, but a separate bug, as loading the model without using
--models-diralso causes this. New issue report opened as #24482.Requirements