mtmd: support "frame merge" for qwen-vl-based models

[ngxson]

Overview

Important

This PR is part of #18389 , it doesn't yet provide end-to-end pipeline for video input. Please only report bugs if you really understand the changes in this PR.

Continue #20224 (moved here because I cannot push to the old PR)

How it works:

• Qwen-VL-based model have a conv3d for input image, that allows merging 2 input images (must be same size) into the same output embeddings. This speeds up the decoding speed and reduce memory usage
• This implementation automatically merge 2 input images if they are (1) placed one right next to the other and (2) have the exact same size, no additional public API is added

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: no

[ngxson]

@ggml-org/maintainers appreciate if someone can give approval(s), thanks!

[Farmadupe]

I'm fairly sure it can't be assumed that two subsequent images in any prompt should be merged. They may be totally unrelated. AFAIK "conv3d" in the qwen vit is an MLP that does arbitrary transformation of image patches.

For example, if a user uploads two screenshots of same size (one image is a handbag, and one is an ebay listing of handbags), with prompt "OCR the listing for this particular handbag", the LM decoder wouldn't be able to do the job.

In general, v1/chat/completions API cannot unambiguously describe temporal data. Adding video support to llama-server will almost definitely require a nonstandard extension.

[ngxson]

server adds a newline between 2 images, so there is no chances they are being merged.

this is only useful for a custom prompt formatting (aka the upcoming mtmd_helper_video interface) that will take advantage of this. and even if user reports problems about this, we can simply add a new API mtmd_bitmap_set_merge(false) to explicitly disable this logic

[ngxson]

@ggml-org/maintainers another approval please 🙏

[Farmadupe]

Noting a couple of technical points post-merger:

• I'm fairly sure that this is a temporal merging layer and is only intended to run on frames that are known to be temporally related.
• Adjacency of image chunks in the token stream does not prove or disprove that frames are temporally related.
• The presernce of newlines between content array elements should be a function of the jinja template. I don't think it's possible to assert that that will always be the case unless special measures are taken.