Add qwen2.5 vl

[maximizemaxwell]

What does this PR do?

Add support for Qwen-2.5-VL

Part of Issue

#2814

[lucasjinreal]

Nice work, does the result verified? Also, would consider add quantization support maybe

[maximizemaxwell]

It's still on work though..

[lucasjinreal]

Oh. no

[akshayballal95]

Nice Work. This is definitely needed. But I see that the model uses conv3d which is not there yet in Candle. How are you planning to handle that?

[maximizemaxwell]

@akshayballal95 I’ve been struggling this for a while. While flattening the temporal dimension and using Conv2D (reshaping [B, T, C, H, W] to [B*T, C, H, W]) may be possible, it changes the computation and is incompatible with pretrained weights. Implementing native Conv3D support in Candle seems to be the proper solution.

[akshayballal95]

Well, we have been asking for it

#795 (comment)

[EricLBuehler]

@maximizemaxwell you can look at: https://github.com/EricLBuehler/mistral.rs/blob/4608202c128da44b84157573dbc8ff1a1146f64c/mistralrs-core/src/layers.rs#L1965-L2036

This is written under the assumption that the temporal patch size==kernel_sizes[0]==2, which is true for Qwen 2.5 VL.

[maximizemaxwell]

Gosh I didn’t realize there was already an issue — I just created a new one with the same content, thinking it didn’t exist.
For now, I think it’s best to focus on implementing conv3d and postpone support for Qwen-VL.
I’ll think through a concrete implementation plan for conv3d tomorrow and open a PR under the issue you shared.

[lucasjinreal]

We need someone to review these features as already implemented in the community. Please merge it into candle so that other people could bring more broad model support to candle!