Skip to content

fix(mtmd): handle Gemma 4 audio projector embedding size #24091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ngxson merged 2 commits into from
Jun 4, 2026
+2 −2
Merged

fix(mtmd): handle Gemma 4 audio projector embedding size #24091

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b4e8bba
mtmd: handle Gemma 4 audio projector embedding size
abetlen Jun 3, 2026
fd6cad5
rm projection_dim from clip_n_mmproj_embd
ngxson Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tools/mtmd/clip.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4347,6 +4347,8 @@ int clip_n_mmproj_embd(const struct clip_ctx * ctx) {
return ctx->model.mm_input_proj_w->ne[0];
case PROJECTOR_TYPE_GEMMA4V:
case PROJECTOR_TYPE_GEMMA4UV:
case PROJECTOR_TYPE_GEMMA4A:
Copy link
Copy Markdown
Member

@CISC CISC Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not incorrect, but this used to be further down (where PROJECTOR_TYPE_GEMMA4UA now is), using hparams.projection_dim instead, don't know if there's a specific reason for that? cc/ @ngxson

Copy link
Copy Markdown
Contributor

@ngxson ngxson Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm ok seems like all of them should be in the same code block here

the ctx->model.hparams.projection_dim code branch should be removed, it was an oversight from the initial PR for gemma 4 audio; well, that's the nasty thing about reviewing a large AI-generated PR

case PROJECTOR_TYPE_GEMMA4UA:
return ctx->model.mm_input_proj_w->ne[1];
case PROJECTOR_TYPE_IDEFICS3:
return ctx->model.mm_fc_w->ne[1];
Expand Down Expand Up @@ -4381,8 +4383,6 @@ int clip_n_mmproj_embd(const struct clip_ctx * ctx) {
return ctx->model.mm_fc_w->ne[1];
case PROJECTOR_TYPE_LFM2A:
return ctx->model.position_embeddings->ne[0];
case PROJECTOR_TYPE_GEMMA4UA:
return ctx->model.hparams.projection_dim;
case PROJECTOR_TYPE_GRANITE_SPEECH:
return ctx->model.qf_proj_linear_w->ne[1];
case PROJECTOR_TYPE_GLM4V:
Expand Down
Loading