Skip to content

common: support --models-dir loading MTP assistant models #24431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ngxson merged 7 commits into from
Aug 15, 2026
Merged

common: support --models-dir loading MTP assistant models #24431

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
80cf6e2
common: support --models-dir loading MTP assistant models
EZForever Jun 10, 2026
8296a4b
common: preset: check for MTP models with strict prefix
EZForever Jun 11, 2026
c74aeaa
common: preset: Take advantage of PR #27005
EZForever Aug 14, 2026
9a52383
Merge branch 'master' into common-models-dir-mtp
ngxson Aug 14, 2026
ef16f56
handle other draft types
ngxson Aug 14, 2026
41571d3
drop eagle3
ngxson Aug 14, 2026
16c45f3
clean up
ngxson Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions common/preset.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,7 @@ struct local_model {
std::string name;
std::string path;
std::string path_mmproj;
std::string path_mtp;
};

common_presets common_preset_context::load_from_models_dir(const std::string & models_dir) const {
Expand All @@ -395,10 +396,13 @@ common_presets common_preset_context::load_from_models_dir(const std::string & m
common_file_info model_file;
common_file_info first_shard_file;
common_file_info mmproj_file;
common_file_info mtp_file;

@ngxson ngxson Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
common_file_info mtp_file;
common_file_info draft_file;

for (const auto & file : files) {
if (string_ends_with(file.name, ".gguf")) {
if (file.name.find("mmproj") != std::string::npos) {
mmproj_file = file;
} else if (file.name.find("mtp-") != std::string::npos) {
Comment thread
EZForever marked this conversation as resolved.
Outdated
mtp_file = file;
} else if (file.name.find("-00001-of-") != std::string::npos) {
first_shard_file = file;
} else {
Expand All @@ -410,7 +414,8 @@ common_presets common_preset_context::load_from_models_dir(const std::string & m
local_model model{
/* name */ name,
/* path */ first_shard_file.path.empty() ? model_file.path : first_shard_file.path,
/* path_mmproj */ mmproj_file.path // can be empty
/* path_mmproj */ mmproj_file.path, // can be empty
/* path_mtp */ mtp_file.path // can be empty
};
if (!model.path.empty()) {
models.push_back(model);
Expand All @@ -428,7 +433,8 @@ common_presets common_preset_context::load_from_models_dir(const std::string & m
local_model model{
/* name */ name,
/* path */ file.path,
/* path_mmproj */ ""
/* path_mmproj */ "",
/* path_mtp */ ""
};
models.push_back(model);
}
Expand All @@ -443,6 +449,10 @@ common_presets common_preset_context::load_from_models_dir(const std::string & m
if (!model.path_mmproj.empty()) {
preset.set_option(*this, "LLAMA_ARG_MMPROJ", model.path_mmproj);
}
if (!model.path_mtp.empty()) {
preset.set_option(*this, "LLAMA_ARG_SPEC_TYPE", "draft-mtp");
preset.set_option(*this, "LLAMA_ARG_SPEC_DRAFT_MODEL", model.path_mtp);
}
out[preset.name] = preset;
}

Expand Down