[Sunt-ing]

Purpose

Qwen2.5-VL currently loses video metadata before calling the HuggingFace processor. Since the HF video processor refactor, second_per_grid_ts is computed from video_metadata.sampled_fps; when vLLM drops that metadata, pre-decoded videos fall back to a default sampled fps instead of the request/video fps. That value drives the temporal M-RoPE scale used by the model, so videos tagged at 1fps and 30fps get the same temporal spacing in vLLM.

This fixes the Qwen2.5-VL path by keeping video metadata in the data parser, splitting (video, metadata) tuples before the HF processor call, and forwarding video_metadata so HF can compute sampled_fps from the real frame metadata. The dummy video builder also attaches synthetic metadata, matching the Qwen3-VL pattern and keeping memory profiling inputs valid once metadata is required.

Scope: the Qwen2.5-VL processor path. Qwen2-VL does not capture second_per_grid_ts in the same way, so that would be a separate change. vLLM's video loader already attaches metadata for URL/file inputs; the changed behavior is that bare pre-decoded video arrays without metadata now fail clearly instead of silently using a fallback fps.

Duplicate-work check: I did not find an open PR covering Qwen2.5-VL video metadata forwarding. PR #23058 is Qwen2.5-Omni only, PR #43198 only lets video frame sampling use a request fps override, PR #45555 only adds loader-side metadata that this path still dropped, and PR #47407 only bounds M-RoPE positions.

Test Plan

Run a real Qwen/Qwen2.5-VL-3B-Instruct LLM.generate path on an RTX 4090, instrument the live model's iter_mm_grid_thw, and compare the temporal t_factor for the same video tagged as fps=1.0 and fps=30.0 before, after, and after reverting the fix. Add one CPU regression case to the existing Qwen2-VL processor test file to assert second_per_grid_ts tracks fps, run affected Qwen2.5-VL video tests, and run ruff on all changed files.

Test Result

Upstream main commit e433634c781dce7ee55563424b8136ad3495c1be before the fix, same video at two fps values:

input fps	t_factor
1.0	0.1669921875
30.0	0.1669921875

After the fix:

input fps	t_factor
1.0	4.0
30.0	0.1337890625

Reverting the patch restored the before behavior:

input fps	t_factor
1.0	0.1669921875
30.0	0.1669921875

Focused regression and lint validation:

python -m pytest -q --tb=short tests/models/multimodal/processing/test_qwen2_vl.py::test_qwen2_5_vl_video_fps_to_second_per_grid_ts
# 1 passed, 18 warnings in 11.04s

python -m ruff check vllm/model_executor/models/qwen2_5_vl.py tests/models/multimodal/processing/test_qwen2_vl.py tests/models/multimodal/generation/test_common.py tests/models/multimodal/generation/test_qwen2_5_vl.py tests/models/multimodal/generation/test_vit_cudagraph.py tests/renderers/test_process_multi_modal_uuids.py examples/generate/multimodal/vision_language_offline.py
# All checks passed!

python -m ruff format --check vllm/model_executor/models/qwen2_5_vl.py tests/models/multimodal/processing/test_qwen2_vl.py tests/models/multimodal/generation/test_common.py tests/models/multimodal/generation/test_qwen2_5_vl.py tests/models/multimodal/generation/test_vit_cudagraph.py tests/renderers/test_process_multi_modal_uuids.py examples/generate/multimodal/vision_language_offline.py
# 7 files already formatted

Broader validation on the same patch:

python -m pytest -q --tb=short tests/models/multimodal/processing/test_qwen2_vl.py
# 17 passed, 38 warnings in 25.62s

python -m pytest -q --tb=short tests/renderers/test_process_multi_modal_uuids.py
# 9 passed, 16 warnings in 21.96s

python -m pytest -q --tb=short tests/models/multimodal/generation/test_vit_cudagraph.py -k "test_vit_cudagraph_video and qwen2_5_vl"
# 1 passed, 21 deselected, 17 warnings in 115.39s

VLLM_WORKER_MULTIPROC_METHOD=spawn python -m pytest -q --tb=short tests/models/multimodal/generation/test_qwen2_5_vl.py
# 12 passed, 34 warnings in 1084.89s

End-to-end reproduction

Environment: RTX 4090, Qwen/Qwen2.5-VL-3B-Instruct, VLLM_USE_FLASHINFER_SAMPLER=0, VLLM_ENABLE_V1_MULTIPROCESSING=0, enforce_eager=True. The before, after, and revert runs below used the same upstream-main checkout at e433634c781dce7ee55563424b8136ad3495c1be; for the after run I applied this PR patch to the worktree, so git_head remains that upstream-main commit.

qwen25vl_fps_e2e.py

import os
import subprocess

import numpy as np

os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0")
os.environ.setdefault("VLLM_ENABLE_V1_MULTIPROCESSING", "0")

import vllm
import vllm.model_executor.models.qwen2_5_vl as q25


def git_head():
    try:
        return subprocess.check_output(
            ["git", "rev-parse", "HEAD"], text=True
        ).strip()
    except Exception:
        return "unknown"


print(f"git_head={git_head()}")
print(f"vllm_import={vllm.__file__}")

captured = []
orig_iter = q25.Qwen2_5_VLForConditionalGeneration.iter_mm_grid_thw


def patched_iter(self, mm_features):
    for item in orig_iter(self, mm_features):
        captured.append(float(item[4]))
        yield item


q25.Qwen2_5_VLForConditionalGeneration.iter_mm_grid_thw = patched_iter

from vllm import LLM, SamplingParams

MODEL = os.environ.get("MODEL", "Qwen/Qwen2.5-VL-3B-Instruct")

num_frames = 64
video = np.zeros((num_frames, 56, 56, 3), dtype=np.uint8)
for frame_idx in range(num_frames):
    video[frame_idx, :, :, 0] = (frame_idx * 5) % 256

prompt = (
    "<|im_start|>user\n"
    "<|vision_start|><|video_pad|><|vision_end|>"
    "Describe the video.<|im_end|>\n"
    "<|im_start|>assistant\n"
)
llm = LLM(
    model=MODEL,
    max_model_len=8192,
    limit_mm_per_prompt={"video": 1, "image": 0},
    gpu_memory_utilization=0.55,
    enforce_eager=True,
    trust_remote_code=True,
)


def metadata(fps):
    return {
        "fps": fps,
        "duration": num_frames / fps,
        "total_num_frames": num_frames,
        "frames_indices": list(range(num_frames)),
        "video_backend": "opencv",
        "do_sample_frames": False,
    }


results = {}
for fps in (1.0, 30.0):
    captured.clear()
    llm.generate(
        {
            "prompt": prompt,
            "multi_modal_data": {"video": [(video, metadata(fps))]},
        },
        SamplingParams(max_tokens=4, temperature=0.0),
    )
    results[fps] = list(captured)
    print(f"input_fps={fps} captured_t_factors={captured}")

fps_ignored = results[1.0] == results[30.0] and bool(results[1.0])
t1 = results.get(1.0, [None])[0]
t30 = results.get(30.0, [None])[0]
print(f"t_factor(fps=1)={t1}  t_factor(fps=30)={t30}")
print(f"FPS_IGNORED_E2E={fps_ignored}")
print("expected_if_honored: fps=1 -> t_factor~4.0 ; fps=30 -> t_factor~0.13")

Run:

VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_ENABLE_V1_MULTIPROCESSING=0 MODEL=Qwen/Qwen2.5-VL-3B-Instruct python qwen25vl_fps_e2e.py

Output before the fix on upstream main commit:

git_head=e433634c781dce7ee55563424b8136ad3495c1be
vllm_import=/root/autodl-tmp/code/vllm-3-refresh-codex-20260706/vllm/__init__.py
input_fps=1.0 captured_t_factors=[0.1669921875]
input_fps=30.0 captured_t_factors=[0.1669921875]
t_factor(fps=1)=0.1669921875  t_factor(fps=30)=0.1669921875
FPS_IGNORED_E2E=True
expected_if_honored: fps=1 -> t_factor~4.0 ; fps=30 -> t_factor~0.13

Output after applying this PR patch to the same checkout:

git_head=e433634c781dce7ee55563424b8136ad3495c1be
vllm_import=/root/autodl-tmp/code/vllm-3-refresh-codex-20260706/vllm/__init__.py
input_fps=1.0 captured_t_factors=[4.0]
input_fps=30.0 captured_t_factors=[0.1337890625]
t_factor(fps=1)=4.0  t_factor(fps=30)=0.1337890625
FPS_IGNORED_E2E=False
expected_if_honored: fps=1 -> t_factor~4.0 ; fps=30 -> t_factor~0.13

Output after reverting the patch:

git_head=e433634c781dce7ee55563424b8136ad3495c1be
vllm_import=/root/autodl-tmp/code/vllm-3-refresh-codex-20260706/vllm/__init__.py
input_fps=1.0 captured_t_factors=[0.1669921875]
input_fps=30.0 captured_t_factors=[0.1669921875]
t_factor(fps=1)=0.1669921875  t_factor(fps=30)=0.1669921875
FPS_IGNORED_E2E=True
expected_if_honored: fps=1 -> t_factor~4.0 ; fps=30 -> t_factor~0.13

AI assistance was used to prepare this PR.

cc @DarkLight1337

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--47736.org.readthedocs.build/en/47736/

[Isotr0py]

Actually, this is more likely a regression from huggingface's video processor refactoring, because the default fps before is 2 instead of 24, which made sure t_factor consistent.

https://github.com/huggingface/transformers/blob/33d1d715b0260efc1c2df1c16d864186b5bb9437/src/transformers/models/qwen2_5_vl/processing_qwen2_5_vl.py#L133

Anyway, thanks for fixing this!

[Sunt-ing]

Thanks @Isotr0py. I removed the test_vit_backend_functionality.py change and updated the description to avoid implying the old HF path defaulted to 24fps; the root cause is the HF metadata path now depending on video_metadata.sampled_fps, which vLLM was dropping.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Sunt-ing.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Sunt-ing]

Hi @Isotr0py, sorry for not advancing this PR for a while. I just fixed all the conflicts. Please review it again, thanks!

[DarkLight1337]

/ci run

[github-actions]

❌ This PR is 13 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

[DarkLight1337]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #90158 for commit a83b686577ad.

⚠️ This PR is 20 commits behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.

[DarkLight1337]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #90349 for commit 636f07544006.

⚠️ This PR is 2 commits behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.