refactor vision model loading

[irexyc]

Motivation

Some vlm models have finetuned vision weights and there is no need to download original vision model.

[AllentDan]

May check if deepseek vl also needs refactoring to avoid downloading weight.

[lvhan028]

The following models still require downloading the vision model

1. internlm-xcomposer2-4khd-7b
2. internlm-xcomposer2-vl
3. internlm-xcomposer2-7b

[lvhan028]

@AllentDan I tried to install minigemini by

pip install git+https://github.com/dvlab-research/MiniGemini.git --no-deps

But the pipeline still raise error

ModuleNotFoundError: No module named 'minigemini'

[AllentDan]

git+https://github.com/dvlab-research/MiniGemini.git

Seems they changed the repo name. https://github.com/dvlab-research/MGM

[lvhan028]

git+https://github.com/dvlab-research/MiniGemini.git

Seems they changed the repo name. https://github.com/dvlab-research/MGM

@irexyc may update the hint

[irexyc]

May check if deepseek vl also needs refactoring to avoid downloading weight.

deepseek vl won't download vision model
https://github.com/deepseek-ai/DeepSeek-VL/blob/main/deepseek_vl/models/clip_encoder.py#L71-L75
https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat/blob/main/config.json
https://huggingface.co/deepseek-ai/deepseek-vl-1.3b-chat/blob/main/config.json

[AllentDan]

Tested pipeline and api_server OK.

• llava-v1.6-34b
• Mini-Gemini-7B
• deepseek-vl-1.3b-chat
• Yi-VL-6B