Chat template: return vectorized output in processors

[zucchini-nlp]

What does this PR do?

Part of #33948. This PR adds support for return_tensors="pt" when calling chat templates for processors. That way users can obtain inputs in tensor format and pass it directly to the model, instead of having to call processor with a formatted prompt + visuals.

For images we use the existing functionality load_images and for videos I added a few functions. We usually use av in all video related model docs since decord had problems with CUDA in the past. Apart from that we can use opencv or torchvision for video loading. I did a small benchmark run to load and sample uniformly 32 frames from around ~100 videos and av was the slowest of all them, while decord was the fastest. Therefore I decided to add helper with all possible backends and let users switch whenever they want to. By default we use opencv as it is a more common CV framework than any others provided here.

In the future we might start using torchvision when we add VideoProcessor class and support VideoProcessorFast (see #33504).

These are the results of small benchmarking with ~100 videos:

# Time taken for decord: 475.2979 sec
# Time taken for opencv: 614.6062 sec
# Time taken for av: 1067.0860 sec
# Time taken for torchvision: 1924.0433 sec

Review from @Rocketknight1 for templates and @qubvel for general CV related modifications.

[Rocketknight1]

This looks good to me! Just to clarify, the idea is that if you pass a chat to apply_chat_template and some of the content fields contain images or videos, and tokenize=True, then images and videos are loaded and processed, so that the output is ready to pass to the model?

[zucchini-nlp]

the idea is that if you pass a chat to apply_chat_template and some of the content fields contain images or videos, and tokenize=True, then images and videos are loaded and processed, so that the output is ready to pass to the model?

Yep, exactly!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qubvel]

Thanks for working on this!

The main question that is not clear to me is: what is the backend selection strategy? Should we pass it explicitly? I see load_video is used without passing backend and num_frames arguments

[zucchini-nlp]

welcome back @qubvel ! Okey, I'll add more typehints and better docs. The backend should be selectable by the user, but we default to the one that works in all cases and has no weird cuda related failures. Prob we should document this somewhere, but I didn't yet find a good place for it

[qubvel]

Thanks, some nits!

[qubvel]

Some additional kwargs might be required here, e.g. timeout, but probably fine for now

[zucchini-nlp]

Huh, I don't know why it requested review from some many people, feel free to unsubscribe, sorry

[ArthurZucker]

For me super good in terms of API!
I am mostly wondering if this does not pose security threats as we are opening links vs before the user had to open the link explicitly in his code.

[zucchini-nlp]

Hmm, good point about the security. We actually already have a few processors that open links for you, e.g. Idefics and Pixtral. Haven't seen anyone flag it as a security breach so maybe it's not a big deal?

[stevhliu]

Thanks!

[ArthurZucker]

Let's remove it or put it in the benchmark file, but probably an overkill!

[qubvel]

Also this file seems unrelated

[qubvel]

And this one as well

[hmellor]

This block breaks lazy importing of cv2 which vllm strictly enforces. It happens when vLLM imports from transformers.image_utils import ImageInput. vLLM cannot upgrade to v4.49.0 because of it vllm-project/vllm#13905.

Would it be possible to delay this import? This would be preferable to lazily importing ImageInput everywhere it's used in vLLM.

cc @ArthurZucker