[Bugfix] Fix base64 JPEG video frames returning empty metadata

[he-yufeng]

Problem

Sending base64-encoded JPEG frames as video input returns a 400 error:

VideoMetadata.__init__() missing 1 required positional argument: 'total_num_frames'

VideoMediaIO.load_base64 handles the video/jpeg media type by decoding individual JPEG frames and stacking them, but returns an empty metadata dict {}. Downstream code passes this to transformers.video_utils.VideoMetadata(**metadata) which requires total_num_frames as a positional argument.

Root Cause

In vllm/multimodal/media/video.py:83-85:

return np.stack(
    [np.asarray(load_frame(frame_data)) for frame_data in data.split(",")]
), {}  # <-- empty metadata

All other video loading paths (via load_bytes → video_loader.load_bytes) return properly populated metadata through create_hf_metadata(), but the base64 JPEG path was missed.

Fix

Populate the metadata dict with the same fields other loaders return:

• total_num_frames: frame count from the stacked array
• fps: from request kwargs (default 1)
• duration: computed from total_num_frames / fps
• video_backend: "jpeg_sequence" to distinguish from other backends
• frames_indices: [0, 1, ..., N-1] since all frames are used directly
• do_sample_frames: False since frames are pre-extracted by the client

Test plan

• Verify base64 JPEG frame video input no longer returns 400
• Ensure metadata fields match what VideoMetadata expects

Fixes #37274

[gemini-code-assist]

Code Review

The pull request effectively addresses the bug where base64-encoded JPEG video frames were returning empty metadata, leading to a VideoMetadata initialization error. The fix correctly populates the metadata dictionary with essential fields such as total_num_frames, fps, duration, video_backend, frames_indices, and do_sample_frames, ensuring compatibility with downstream video processing components. This is a critical improvement for the robustness of video input handling.

[gemini-code-assist]

The previous implementation returned an empty dictionary for metadata, which caused a runtime error when transformers.video_utils.VideoMetadata was initialized. This change correctly populates the metadata with all necessary fields, resolving the total_num_frames missing argument issue and ensuring proper video metadata propagation. This is a critical fix for the functionality of base64 JPEG video frame processing.

[mergify]

Hi @universeplayer, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[Isotr0py]

Can you add a regression test at tests/multimodal/media/test_video.py?

[mergify]

Hi @universeplayer, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10