Skip to content

llama : meta split state for combined gate+up ffn_up (phi3) #24936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
krystophny wants to merge 2 commits into from
Closed

llama : meta split state for combined gate+up ffn_up (phi3) #24936

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6ad03e3
llama : meta split state for combined gate+up ffn_up (phi3)
krystophny Jun 23, 2026
a0caeb1
llama : guard combined ffn_up split, use per-layer n_ff
krystophny Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions src/llama-model.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -550,6 +550,16 @@ struct ggml_backend_meta_split_state llama_meta_device_get_split_state(const str
GGML_ASSERT(tensor->ne[axis] == n_embd + 2*n_embd_gqa);
return {{n_embd, 1}, {n_embd_gqa, 2}};
}
// phi3 and modern-bert pack gate|up into a dense ffn_up of width 2*n_ff;
// ordinary separate ffn_up/ffn_gate stay at n_ff, so this stays a guard
// that falls through to the default even split rather than an assert.
if (std::regex_match(tensor_name, pattern_ffn_up_gate_weight) ||
std::regex_match(tensor_name, pattern_ffn_up_gate_bias)) {
const int64_t n_ff = hparams.n_ff(il);
if (n_ff != 0 && tensor->ne[axis] == 2*n_ff) {
return {{n_ff, 2}};
}
}
if (std::regex_match(tensor_name, pattern_ffn_gate_up_weight)) {
const int64_t n_ff_exp = hparams.n_ff_exp;
GGML_ASSERT(tensor->ne[axis] == 2*n_ff_exp);
Expand Down