Skip to content

[Model] Siglip2 Model Support #27566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vllm-bot merged 5 commits into from
Oct 27, 2025
Merged

[Model] Siglip2 Model Support #27566

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4dfc5d4
[Model] Siglip Model Support
piood Oct 27, 2025
5a15ab9
fix
piood Oct 27, 2025
1970105
fix
piood Oct 27, 2025
463c27f
fix
piood Oct 27, 2025
39ee47a
fix
piood Oct 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/models/supported_models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -774,7 +774,7 @@ The following table lists those that are tested in vLLM.
| `CLIPModel` | CLIP | T / I | `openai/clip-vit-base-patch32`, `openai/clip-vit-large-patch14`, etc. | | |
| `LlavaNextForConditionalGeneration`<sup>C</sup> | LLaVA-NeXT-based | T / I | `royokong/e5-v` | | ✅︎ |
| `Phi3VForCausalLM`<sup>C</sup> | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | | ✅︎ |
| `SiglipModel` | SigLIP | T / I | `google/siglip-base-patch16-224` | | |
| `SiglipModel` | SigLIP, SigLIP2 | T / I | `google/siglip-base-patch16-224`, `google/siglip2-base-patch16-224` | | |
| `*ForConditionalGeneration`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | \* | N/A | \* | \* |

<sup>C</sup> Automatically converted into an embedding model via `--convert embed`. ([details](./pooling_models.md#model-conversion))
Expand Down
2 changes: 1 addition & 1 deletion tests/models/multimodal/pooling/test_siglip.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
}
)

MODELS = ["google/siglip-base-patch16-224"]
MODELS = ["google/siglip-base-patch16-224", "google/siglip2-base-patch16-224"]


def _run_test(
Expand Down
8 changes: 5 additions & 3 deletions vllm/model_executor/models/siglip.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,9 +174,11 @@ class SiglipMultiModalProcessor(BaseMultiModalProcessor[SiglipProcessingInfo]):
@cached_property
def image_token_id(self) -> int:
tokenizer = self.info.get_tokenizer()
dummy_token_id = 0

assert dummy_token_id not in tokenizer.all_special_ids
dummy_token_id = next(
token_id
for token_id in range(tokenizer.vocab_size)
if token_id not in tokenizer.all_special_ids
)

return dummy_token_id

Expand Down
12 changes: 11 additions & 1 deletion vllm/transformers_utils/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,10 @@
)
from transformers import GenerationConfig, PretrainedConfig
from transformers.models.auto.image_processing_auto import get_image_processor_config
from transformers.models.auto.modeling_auto import MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
from transformers.models.auto.modeling_auto import (
MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,
MODEL_MAPPING_NAMES,
)
from transformers.models.auto.tokenization_auto import get_tokenizer_config
from transformers.utils import CONFIG_NAME as HF_CONFIG_NAME

Expand Down Expand Up @@ -616,6 +619,13 @@ def get_config(
model_type = MODEL_FOR_CAUSAL_LM_MAPPING_NAMES[config.model_type]
config.update({"architectures": [model_type]})

# Architecture mapping for models without explicit architectures field
if not config.architectures:
if config.model_type not in MODEL_MAPPING_NAMES:
raise ValueError(f"Cannot find architecture name for {config.model_type}")
model_type = MODEL_MAPPING_NAMES[config.model_type]
config.update({"architectures": [model_type]})
Comment on lines +623 to +627
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The added logic to automatically determine the architecture for models without an explicit architectures field is good for robustness. However, raising a ValueError if the model_type is not found in MODEL_MAPPING_NAMES might be too strict. A more graceful fallback could involve attempting to load the model with trust_remote_code=True or issuing a warning and proceeding with a default architecture. This could prevent the system from failing completely when encountering a new or less common model type. Also, consider adding a log message to indicate when this automatic architecture detection is being used, which can help with debugging and understanding the system's behavior.

Suggested change
if not config.architectures:
if config.model_type not in MODEL_MAPPING_NAMES:
raise ValueError(f"Model type {config.model_type} not supported")
model_type = MODEL_MAPPING_NAMES[config.model_type]
config.update({"architectures": [model_type]})
if not config.architectures:
if config.model_type not in MODEL_MAPPING_NAMES:
logger.warning(f"Model type {config.model_type} not found in MODEL_MAPPING_NAMES. Attempting to proceed without explicit architecture.")
# Optionally, try loading with trust_remote_code or assign a default architecture here
config.update({"architectures": ["AutoModel"]}) # Example: setting a default architecture
else:
model_type = MODEL_MAPPING_NAMES[config.model_type]
config.update({"architectures": [model_type]})


# ModelOpt 0.31.0 and after saves the quantization config in the model
# config file.
quantization_config = config_dict.get("quantization_config", None)
Expand Down