model support: Sarashina2VisionForCausalLM

[CatherineSue]

Motivation

This PR focues on adding support for Sarashina2VisionForCausalLM architecture. Such as sbintuitions/sarashina2-vision-8b

The model type is built upon Qwen2VL and LlamaForCausalLM.

Caveats

1. sbintuitions/sarashina2-vision-8b has outdated imports from a previous version of transformers. See: https://huggingface.co/sbintuitions/sarashina2-vision-8b/blob/main/processing_sarashina2_vision.py#L41
2. The chat template of the model doesn't follow the normal Jinja multimodal standard. It directly use message['content'] without checking the type. We added a customized chat template.

Modifications

• Add processor for sarashina2_vision.
• Add model file for sarashina2_vision.
• Add get_input_embeddings to LlamaModel as it is required in Qwen2VLProcessor
• Add customized chat template in examples/chat_template. It is able to check message['content']['type']

Accuracy Tests

Simple Example

Server start up command:

python3 -m sglang.launch_server --model /models/sbintuitions/sarashina2-vision-8b --port=8080 --tp-size=8 --chat-template=examples/chat_template/vision_template_sarashina.jinja --trust-remote-code --disable-fast-image-processor

Image: "https://images.pexels.com/photos/210186/pexels-photo-210186.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=4096&w=4096"

Text-only: "What is 1+1? What is the capital of France?"

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @CatherineSue, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces support for the Sarashina2VisionForCausalLM model, a multimodal architecture that combines a Llama-based language model with a Qwen2VL vision component. The changes enable the system to process and generate responses based on both text and image inputs for this specific model, addressing its unique processing requirements and chat template format.

Highlights

• New Model Support: Added comprehensive support for the Sarashina2VisionForCausalLM multimodal model, which integrates a Llama text backbone with a Qwen2VL vision encoder.
• Processor Integration: Implemented a dedicated processor for Sarashina2Vision to handle multimodal data, including image processing and tokenization.
• Llama Model Enhancement: Extended the LlamaModel with a get_input_embeddings method, required for compatibility with the new vision model's processing.
• Dynamic Image Sizing: Updated the get_processor utility to automatically inject default image size parameters for Sarashina2Vision models, similar to Qwen2-VL.
• Custom Chat Template: A customized chat template was added to correctly handle the model's specific multimodal message content structure, addressing a caveat in its default behavior.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for the Sarashina2VisionForCausalLM model. The changes are well-structured, including the model definition, a dedicated processor, and necessary modifications to existing utilities.

My review focuses on the new model implementation and processor. I've identified a bug in the model's __init__ method related to config handling, an unused variable, and a maintainability concern in the processor's monkey-patching logic. The suggestions aim to fix the bug, improve code clarity, and make the code more robust to upstream changes.

Overall, this is a solid contribution. Addressing the feedback will further improve the quality and maintainability of the new model support.

[gemini-code-assist]

This block for initializing the Llama text model can be simplified to remove redundant code and fix a no-op assignment. The line config.vocab_size = config.vocab_size is a bug and has no effect. The logic can be consolidated for better readability and robustness.

        if hasattr(text_config, "model_type") and text_config.model_type == "llama":
            llama_config = LlamaConfig(**text_config.__dict__)
        else:
            llama_config = config

        # Set vocab_size from main config if available
        if hasattr(config, "vocab_size"):
            llama_config.vocab_size = config.vocab_size

        self.llm = LlamaForCausalLM(
            llama_config,
            quant_config=quant_config,
            prefix=add_prefix("llm", prefix),
        )

[gemini-code-assist]

The loaded_params variable is initialized but never used. It should be removed to improve code clarity.

[gemini-code-assist]

This hardcoded set of allowed_params is brittle and may break if the signature of the upstream Sarashina2VisionImageProcessor._preprocess method changes. While this monkey-patching is a necessary workaround, consider adding a comment with a direct link to the source of this signature on Hugging Face to make future maintenance easier.