webui: support video files as input

[foldl]

Overview

Support adding video files as input. This can fix #20741.

Everything is done almost the same as audio files.

Note

To make video input work, at least 3 modifications are needed:

• mtmd.
• server.
• webui.

This PR only updated WebUI.

Detailed Modifications

1. Add a menu item for uploading video files;
2. Show an icon in the chat input box (ChatAttachmentsListItemThumbnailFile) like ChatAttachmentsPreviewThumbnailStrip;
3. A new preview window for video files;
4. Video files are sent to the server through input_video (just like input_audio for audio files);
5. Two types of video files are defined (mp4 and ogg);
6. On Model Information window, video modality is shown as "Vision (Video)", and the vision modality is shown as "Vision (Image)";
7. Add a new bool field video to Modalities.

Test & Sceenshots

I have tested this with chatllm.cpp.

Additional information

Some findings or thoughts that are out of the scope of this PR.

• How to properly show the modalities of image-only, and image-video?
• Video files often contain audio. At present, when sending to servers, media types are inferred from file extension but not the menu item which is clicked by users.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO. I coded this all by myself (copied and modified some existing codes).

[ngxson]

IMO this can be an acceptable stop-gap solution. But just one concern is that we will eventually have native video support in mtmd, so we should make sure changes from this PR can be easily reverted when it happens.

[foldl]

This is for mtmd (see the issue #20741). Why this needs to be reverted when video support in mtmd is ready?

[allozaur]

Please rebase this on latest commit on master and solve conflicts.

[ServeurpersoCom]

LGTM, video has been added symmetrically to image and audio.
If refactor is required, all three can be maintained in the same way.

[chigkim]

Is that possible to submit a video file via OpenAI Chat Completion or Responses API? If so, would you mind showing an example snippet of json payload?
Thanks!

[ServeurpersoCom]

Is that possible to submit a video file via OpenAI Chat Completion or Responses API? If so, would you mind showing an example snippet of json payload? Thanks!

The PR adds an input_video content part that mirrors input_audio exactly. Based on the diff (chat.service.ts and api.d.ts), the WebUI sends something like this on /v1/chat/completions:

{
  "model": "your-model",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Describe this video." },
        {
          "type": "input_video",
          "input_video": {
            "data": "<base64 encoded video bytes>",
            "format": "mp4"
          }
        }
      ]
    }
  ]
}

format is one of "mp4", "ogg" or "auto". The server side needs to understand input_video for this to work end to end.

[OPerepadia]

Is it required to set a specific flag to enable video input?

I tried with Gemma 4 E4B but there is no option to upload a video file.

In the model info, Vision (Video) is not shown either

But E4B has video capability

unsloth/gemma-4-E4B-it-GGUF · Hugging Face

Extended Multimodalities – Processes Text, Image with variable aspect ratio and resolution support (all models), Video, and Audio (featured natively on the E2B and E4B models).

Starting the llama-server with this command

./build/bin/llama-server \
    -hf unsloth/gemma-4-E4B-it-GGUF:Q8_K_XL \
    -c 32768 \
    --n-gpu-layers auto \
    --mmproj-auto --mmproj-offload

[foldl]

@OPerepadia this is implemented ahead of mtmd. Once video support is ready in mtmd (and a little update in server), video support will works.

See also #20741.

[chigkim]

Thanks @ServeurpersoCom for the format.

However, if I send the following message, I get this error below.

Data has base64.b64encode video. I encoded the same way I encode image.

def encode_video(path):
    with open(path, "rb") as video_file:
        content = video_file.read()
    return base64.b64encode(content).decode("utf-8")

data = encode_video(path)

Here's a message.

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Describe this video." },
    {
      "type": "input_video",
      "input_video": {
        "data": data,
        "format": "mp4"
      }
    }
  ]
}

srv operator(): got exception: {"error":{"code":400,"message":"unsupported content[].type","type":"invalid_request_error"}}

If I send without the block for video below, it works, but the model says it can't find video as expected.

    {
      "type": "input_video",
      "input_video": {
        "data": data,
        "format": "mp4"
      }
    }

I tried with both qwen-3.6 and gemma-4.
Thanks for your help!

[ServeurpersoCom]

However, if I send the following message, I get this error below.

That's exactly the request llama-ui sends, your JSON is correct. The piece that's missing is the backend: the server's content parser doesn't handle input_video yet, so it rejects it. The client half is in, the server half isn't, which is why even the llama-ui itself can't do video end to end right now.

[foldl]

@chigkim To make video input work, at least 3 modifications are needed:

• mtmd.
• server.
• webui.

This PR only updated WebUI. If you want to see how the whole thing works, you can try it with chatllm.cpp (SmolVL, Gemma-4-E2B).

[chigkim]

Ah ok, for some reason I thought the entire workflow was ready. :)