[Multimodal] Add lossy keyframe-only video loader (pyav_keyframes)

[WindChimeRan]

Purpose

Add an opt-in, lossy video loader pyav_keyframes that samples only bitstream keyframes: one header-only demux pass to enumerate keyframe PTS, then seek-decode exactly those frames. No P/B-frame is ever decoded.

VLLM_VIDEO_LOADER_BACKEND=pyav_keyframes \
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --media-io-kwargs '{"video": {"num_frames": 16}}'  # decode cost: ≤16 I-frame decodes per clip, any clip length

Targets prefill-heavy workloads where CPU video decoding, not the GPU, dominates the pipeline. All results below are from single-token multiple-choice video QA (classification-style, max_tokens=1), and motion-/temporal-order-sensitive tasks measurably degrade (see Accuracy). Clips with fewer keyframes than the budget get balanced duplication, with true source indices in frames_indices.

Documented in docs/features/multimodal_inputs.md.

Builds on the PyAV codec (#39986, #42586). #44598 (draft) pursues the complementary lossless goal — decoding the same exact frames faster — while this PR changes which frames are sampled; happy to rebase onto its _demux_keyframes() if it lands first.

Exp setting

• E2E: offline LLM.chat, single-token MC video QA, Qwen2.5-VL-7B-Instruct, 1×A100-80GB, bf16, max_tokens=1, num_frames=16, max_pixels=256·28²; N=1990 = NExTQA MC test (1000) + MVBench 55×18 subtasks (990); baseline = lossless cv2 loader, identical settings.

Results

Qwen2.5-VL-7B, offline single-token MC video-QA (max_tokens=1), N=1990 (NExTQA 1000 + MVBench 990), 1×A100-80GB, identical resolution/engine settings — vs the faithful qwen2_vl (fps=2) baseline:

Loader	NExTQA	MVBench	NExTQA req/s	MVBench req/s
qwen2_vl (faithful, #45555)	82.7%	67.7%	1.06	1.73
pyav_keyframes (this PR)	79.6%	53.2%	5.58	5.14

~3–5× throughput for −3.1 pt NExTQA / −14.4 pt MVBench. NExTQA (scene/state QA) holds within ~3 pt; the MVBench cost is real and concentrates in motion / temporal-order subtasks (action_antonym −47.3, moving_attribute−38.2, object_existence −36.4). The decode win is data-independent: a synthetic H.264 microbench extracts 16 frames in near-constant tens of ms whether the clip is 30 s or 10 min (vs the lossless path growing with clip length).

Full reproducible experiments: https://github.com/WindChimeRan/offline_video_vllm

Test Plan

2 CPU tests (pytest tests/multimodal/test_video.py -k pyav_keyframes):

• samples only keyframes — multi-GOP clip → frames land on GOP boundaries, pixel markers agree with frames_indices.
• oversamples instead of decoding non-keyframes — 2-keyframe clip, 8 slots → balanced 4+4 duplication, honestly labeled.

Test Result

pytest tests/multimodal/test_video.py -k pyav_keyframes -v   # 2 passed

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[mergify]

Documentation preview: https://vllm--45203.org.readthedocs.build/en/45203/

[Isotr0py]

I'm hesitant to merge this PR, because each model has different frames sampling method for best video accuracy:

vllm/vllm/multimodal/video.py

Lines 699 to 714 in 0cd9b7a

	@VIDEO_LOADER_REGISTRY.register(
	"glm46v",
	video_processor="Glm46VVideoProcessor",
	)
	class GLM46VVideoBackend(VideoBackend):
	"""GLM-4.6V dynamic FPS video backend.
	Faithfully replicates the frame sampling logic from transformers'
	``Glm46VVideoProcessor.sample_frames``:
	- Dynamic FPS thresholds based on effective video duration:
	``{≤30s: 3fps, ≤300s: 1fps, >300s: 0.5fps}``
	- ``temporal_patch_size`` multiplier (default 2) applied to extract count
	- Duration capped at 2400s, frame count capped at 640
	- Even frame count enforced (append last frame if odd)
	"""

I think the ground truth in your experiment should be model-specific video loader with correct frames sampling strategy, instead of the default opencv fallback. Otherwise the video benchmark accuracy will still be distorted.

[WindChimeRan]

@Isotr0py

Why the opencv baseline is the right ground truth for this experiment. Qwen2.5-VL has no processor-mapped loader (processor2backend covers GLM-4V/4.6V/GLM-GA/Molmo2 only), so the default opencv loader is what vLLM executes for it today.

This loader cannot displace a model-specific one. the env var is a no-op for models with a mapped loader; pyav_keyframes can only ever replace the generic fallback. I'll add a doc line making this explicit.

Motivating workload. This came out of an eBay-scale video classification pipeline: large volume, small label set, not motion/temporal-order sensitive, ~1 output token — where CPU-side video decode is the measured bottleneck, not the GPU. Keyframe extraction is nearly free because the container already indexes I-frames; no P/B-frame is ever decoded.

Intended use: gated by a cheap A/B on the target data. The loss is data-dependent (−52.7 pt on action_antonym vs ±2 pt on 10/18 MVBench subtasks), so the recommended adoption is label-free: run lossy vs uniform on a sample of your own data and compare exact-match agreement on predicted labels (plus KL of the output distribution) to price the accuracy–throughput tradeoff before enabling. I'll add this guidance to the docs as well.

[Isotr0py]

Why the opencv baseline is the right ground truth for this experiment. Qwen2.5-VL has no processor-mapped loader (processor2backend covers GLM-4V/4.6V/GLM-GA/Molmo2 only), so the default opencv loader is what vLLM executes for it today.

Actually, Qwen2.5-VL has no processor-mapped loader in vLLM is just because I don't have bandwidth to implement it recently. 😅

You can refer to processor implementation in Transformers which provides a correct frames sampling strategy:
https://github.com/huggingface/transformers/blob/fa15bb439932dadbb79256233c63b5e8dabb5a1f/src/transformers/models/qwen2_vl/video_processing_qwen2_vl.py#L122-L190

[WindChimeRan]

@Isotr0py Thanks for pointing me to the HF reference.

I'll build the Qwen2.5-VL processor-mapped loader as a separate PR, then rerun accuracy against it as the ground truth

[WindChimeRan]

close because #46609 supersede this. and seek_mode="approximate" is exactly what we want.