Skip to content

Make it possible to load siglip models from local files #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
maxlund wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Make it possible to load siglip models from local files #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d1ce130
Make it possible to load siglip models from local files
Apr 5, 2025
d15dce1
add torch in requirements.txt
Apr 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions mlx_embeddings/utils.py
View file
Open in desktop
Copy link
Owner

@Blaizzy Blaizzy Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is cool!

Copy link
Author

@maxlund maxlund Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

Any idea why running on MLX doesn't seem to give us any noticable performance gains compared to torch on MPS?

Original file line number Diff line number Diff line change
Expand Up @@ -167,17 +167,20 @@ def load_model(

# siglip models have a different image size
if "siglip" in config["model_type"]:
# Extract the image size
image_size = re.search(
r"patch\d+-(\d+)(?:-|$)", kwargs["path_to_repo"]
).group(1)
# Extract the patch size
patch_size = re.search(r"patch(\d+)", kwargs["path_to_repo"]).group(1)
patch_size = (
re.search(r"\d+", patch_size).group()
if re.search(r"\d+", patch_size)
else patch_size
)
if not isinstance(image_size := model_config.get("image_size"), int):
# Extract the image size from hf repo name if not supplied
image_size = re.search(
r"patch\d+-(\d+)(?:-|$)", kwargs["path_to_repo"]
).group(1)

if not isinstance(patch_size := model_config.get("patch_size"), int):
# Extract the patch size from hf repo if not supplied
patch_size = re.search(r"patch(\d+)", kwargs["path_to_repo"]).group(1)
patch_size = (
re.search(r"\d+", patch_size).group()
if re.search(r"\d+", patch_size)
else patch_size
)
if model_args.vision_config.image_size != int(image_size):
model_args.vision_config.image_size = int(image_size)
if model_args.vision_config.patch_size != int(patch_size):
Expand Down
1 change: 1 addition & 0 deletions requirements.txt
View file
Open in desktop
Copy link
Author

@maxlund maxlund Apr 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if you want to pin this to some version, just thought I'd add it since I can't run the repo without torch installed

Copy link
Owner

@Blaizzy Blaizzy Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to remove torch here

Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ mlx>=0.16.3
mlx-vlm>=0.1.21
transformers[sentencepiece]>=4.44.0
huggingface-hub>=0.25.1
torch