[Multimodal] Add lossy keyframe-only video loader (pyav_keyframes)

docs/features/multimodal_inputs.md

@@ -841,6 +841,33 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
 
 Works with common video formats like MP4 when using OpenCV backends.
 
+#### Keyframe-Only Video Sampling (Lossy)
+
+Video frames are sampled by a pluggable loader selected with the `VLLM_VIDEO_LOADER_BACKEND` environment variable (default: `opencv`, uniform sampling). Video codecs store a complete image only at periodic *keyframes* (I-frames), each starting a group of pictures (GOP) of typically 2–10 seconds; the frames in between (P/B-frames) are deltas that can only be reconstructed by decoding forward from the preceding keyframe. The `pyav_keyframes` loader trades temporal accuracy for decode speed: it samples only bitstream keyframes, so decode cost is bounded by the frame budget regardless of clip length — P/B-frames are never decoded.
+
+```bash
+VLLM_VIDEO_LOADER_BACKEND=pyav_keyframes \
+vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
+  --media-io-kwargs '{"video": {"num_frames": 16}}'
+```
+
+`num_frames` is the regular frame budget from `--media-io-kwargs` (default 32, also settable per request via the `media_io_kwargs` API field); the env variable applies to offline `LLM` use as well. In our measurements decode time is near-constant (tens of milliseconds for 16 frames, whether the clip is 30 seconds or 10 minutes), roughly 4–40× faster than the lossless PyAV decode path, with the gap growing with clip length.
+
+This targets prefill-heavy workloads that generate few tokens, where CPU-side video decoding rather than the GPU dominates the pipeline. Keyframe positions are already indexed in the container, so they can be enumerated without decoding anything. The measured results below come from single-token multiple-choice video QA; treat other workloads as unvalidated.
+
+**Behavior:**
+
+- Returned frames sit on GOP boundaries rather than a uniform stride, so temporal coverage follows the source encoding (typically one frame per 2–10 seconds of video).
+- If a clip has fewer keyframes than `num_frames`, keyframes are duplicated (balanced) to fill the request; `frames_indices` always reports true source indices.
+- It is never selected automatically for any model; the `opencv` codec and `frame_recovery` are rejected.
+- Models with their own model-specific video loader (e.g. GLM-4V/4.6V, Molmo2) ignore `VLLM_VIDEO_LOADER_BACKEND`: the processor-mapped backend takes precedence, so `pyav_keyframes` only ever replaces the default `opencv` loader. It can still be forced per request via `media_io_kwargs` `{"video": {"video_backend": "pyav_keyframes"}}`, which is unvalidated for those models.
+
+!!! warning
+    This is a lossy trade-off. In a Qwen2.5-VL-7B evaluation (NExTQA + MVBench, 16 frames), NExTQA accuracy was unchanged (79.6% → 79.5%) while MVBench dropped 11.3 points overall, concentrated in motion- and temporal-order subtasks (`action_antonym`: −52.7 points). Avoid it for motion counting, temporal ordering, or action recognition; the accuracy impact varies with model, task, and source GOP structure.
+
+!!! tip
+    Because the cost is data-dependent, gate adoption with a label-free A/B on a sample of your own data: run `pyav_keyframes` vs. the default loader at identical settings and compare exact-match agreement on predicted labels (optionally KL divergence of the output distributions). Enable it only where the agreement justifies the speedup.
+
 #### Pre-extracted Frame Sequences with `media_io_kwargs`
 
 When you extract video frames on the client side and send them as `video/jpeg` (base64-concatenated JPEG frames), you can preserve the original video metadata by using `media_io_kwargs` in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction.

tests/multimodal/test_video.py

@@ -420,6 +420,12 @@ def test_pyav_dynamic_backend_loads_frames(
         assert metadata["video_backend"] == "pyav_dynamic"
 
 
+def _green_markers(frames: npt.NDArray) -> list[int]:
+    """Recover per-frame index markers encoded on the green channel."""
+    h, w = frames.shape[1], frames.shape[2]
+    return [int(f[h // 2, w // 2, 1]) for f in frames]
+
+
 def test_pyav_backend_returns_target_frames_not_keyframes():
     """Regression test: PyAV must decode forward past the seek keyframe.
 
@@ -446,7 +452,7 @@ def test_pyav_backend_returns_target_frames_not_keyframes():
     requested = list(metadata["frames_indices"])
     assert len(requested) == num_sampled
 
-    actual = [int(f[height // 2, width // 2, 1]) for f in frames]
+    actual = _green_markers(frames)
 
     assert len(set(actual)) == num_sampled, (
         f"PyAV returned only {len(set(actual))} distinct frames for "
@@ -463,6 +469,59 @@ def test_pyav_backend_returns_target_frames_not_keyframes():
         )
 
 
+# ============================================================================
+# PyAV Keyframe-Only Loader Tests
+# ============================================================================
+
+
+def test_pyav_keyframes_samples_only_keyframes():
+    """Core contract — the inverse of the uniform pyav codec test above:
+    returned frames must sit on GOP boundaries, labeled with their true
+    source index."""
+    # 120 frames with a keyframe every 10 → keyframes at 0, 10, ..., 110
+    video_bytes = create_long_gop_video(num_frames=120, gop_size=10)
+
+    loader = VIDEO_LOADER_REGISTRY.load("pyav_keyframes")
+    frames, metadata = loader.load_bytes(video_bytes, num_frames=6)
+
+    assert frames.shape == (6, 64, 64, 3)
+    assert metadata["video_backend"] == "pyav_keyframes"
+    assert metadata["do_sample_frames"] is False
+
+    indices = metadata["frames_indices"]
+    assert indices == sorted(indices)
+    assert len(set(indices)) == 6, "12 keyframes available; no dups expected"
+    assert all(idx % 10 == 0 for idx in indices), (
+        f"Keyframes were encoded every 10 frames; got indices={indices}"
+    )
+
+    # Pixels must agree with the metadata labels.
+    for marker, idx in zip(_green_markers(frames), indices):
+        assert abs(marker - idx) <= 10, (
+            f"Pixel marker {marker} disagrees with metadata index {idx}"
+        )
+
+
+def test_pyav_keyframes_oversamples_instead_of_decoding_non_keyframes():
+    """Hard contract: with fewer keyframes than num_frames, duplicate
+    keyframes (balanced) — never fall back to P/B-frame decode."""
+    # 60 frames with a keyframe every 30 → only 2 keyframes (0 and 30)
+    video_bytes = create_long_gop_video(num_frames=60, gop_size=30)
+
+    loader = VIDEO_LOADER_REGISTRY.load("pyav_keyframes")
+    frames, metadata = loader.load_bytes(video_bytes, num_frames=8)
+
+    assert frames.shape[0] == 8
+    indices = metadata["frames_indices"]
+    assert set(indices) == {0, 30}, f"Expected only the 2 keyframes: {indices}"
+    assert indices.count(0) == 4 and indices.count(30) == 4, (
+        f"Duplication should be balanced, got {indices}"
+    )
+
+    for marker, idx in zip(_green_markers(frames), indices):
+        assert abs(marker - idx) <= 10
+
+
 @pytest.mark.parametrize(
     "loader_key, kwargs, expected_num_frames",
     [

tests/multimodal/utils.py

@@ -71,26 +71,31 @@ def create_long_gop_video(
     fps: int = 30,
     width: int = 64,
     height: int = 64,
+    gop_size: int | None = None,
 ) -> bytes:
-    """Encode an H.264 clip with one keyframe and green-channel = frame index.
+    """Encode an H.264 clip with green-channel = frame index markers.
 
     The marker lets a test recover which frame the decoder actually returned,
-    independent of any metadata label.
+    independent of any metadata label. By default the whole clip is a single
+    GOP (one keyframe at frame 0); pass ``gop_size`` to place a keyframe
+    every ``gop_size`` frames instead.
     """
     import io
 
     import av
 
+    keyint = gop_size or num_frames
+
     buf = io.BytesIO()
     with av.open(buf, mode="w", format="mp4") as container:
         stream = container.add_stream("h264", rate=fps)
         stream.width = width
         stream.height = height
         stream.pix_fmt = "yuv420p"
-        stream.codec_context.gop_size = num_frames
+        stream.codec_context.gop_size = keyint
         stream.codec_context.max_b_frames = 0
         stream.codec_context.options = {
-            "x264-params": (f"scenecut=0:keyint={num_frames}:min-keyint={num_frames}")
+            "x264-params": (f"scenecut=0:keyint={keyint}:min-keyint={keyint}")
         }
         for i in range(num_frames):
             img = np.zeros((height, width, 3), dtype=np.uint8)

vllm/multimodal/video.py

@@ -1343,3 +1343,121 @@ def load_bytes(
             valid_frame_indices=valid_frame_indices,
         )
         return frames, metadata
+
+
+@VIDEO_LOADER_REGISTRY.register("pyav_keyframes")
+class PyAVKeyframeVideoBackend(VideoLoader, PyAVVideoBackendMixin):
+    """Keyframe-only sampling video backend (lossy, opt-in).
+
+    Samples exclusively from the bitstream's keyframes: one demux pass
+    enumerates keyframe PTS values (packet headers only, no decode), then
+    each kept keyframe is seek-decoded directly. Decode work is bounded by
+    ``num_frames`` I-frame decodes per clip regardless of clip length — no
+    P/B-frame is ever decoded.
+
+    This is an explicit accuracy/throughput trade-off: returned frames sit
+    on GOP boundaries rather than a uniform temporal stride, so temporal
+    coverage follows the source encoding. Tasks that depend on dense or
+    uniform temporal sampling (motion, ordering, counting) degrade; see
+    ``docs/features/multimodal_inputs.md`` for measured trade-offs. Opt in
+    deliberately via ``VLLM_VIDEO_LOADER_BACKEND=pyav_keyframes``.
+
+    When the clip has fewer keyframes than ``num_frames``, the available
+    keyframes are oversampled (balanced duplication) so downstream
+    processors still receive exactly ``num_frames`` frames.
+    ``frames_indices`` reports the true source index of every returned
+    (possibly duplicated) keyframe, keeping temporal positional encodings
+    consistent with the pixels.
+    """
+
+    @classmethod
+    def load_bytes(
+        cls,
+        data: bytes,
+        num_frames: int = -1,
+        fps: int = -1,
+        max_duration: int = 300,
+        frame_recovery: bool = False,
+        *,
+        backend: Literal["pyav"] = "pyav",
+        **kwargs,
+    ) -> tuple[npt.NDArray, dict[str, Any]]:
+        assert not frame_recovery, (
+            "frame_recovery is only available for `opencv` backend"
+        )
+        assert backend == "pyav", (
+            f"pyav_keyframes decodes keyframes via PyAV only; got {backend!r}"
+        )
+
+        with av.open(BytesIO(data)) as container:
+            source = cls.get_metadata(container)
+            stream = container.streams.video[0]
+            time_base = stream.time_base
+            if source.original_fps > 0:
+                src_fps = source.original_fps
+            elif source.duration > 0 and source.total_frames_num > 0:
+                # Container omitted average_rate but reported frames and
+                # duration; derive the rate so frames_indices and the fps
+                # metadata field stay consistent.
+                src_fps = source.total_frames_num / source.duration
+                source = source._replace(original_fps=src_fps)
+            else:
+                raise ValueError(
+                    "pyav_keyframes: container reports no frame rate and no "
+                    "frame-count/duration pair to derive one; cannot map "
+                    "keyframe PTS to frame indices"
+                )
+
+            kf_pts = [
+                packet.pts
+                for packet in container.demux(stream)
+                if packet.is_keyframe and packet.pts is not None
+            ]
+            n_kf = len(kf_pts)
+            if n_kf == 0:
+                raise ValueError("pyav_keyframes: no keyframes found in bitstream")
+
+            num_frames_to_sample = num_frames if num_frames > 0 else n_kf
+            if fps > 0 and source.duration > 0:
+                num_frames_to_sample = min(
+                    num_frames_to_sample, math.floor(source.duration * fps)
+                )
+            num_frames_to_sample = max(1, num_frames_to_sample)
+
+            # Round-not-truncate so e.g. n_kf=2 / num_frames=16 duplicates
+            # each keyframe 8x (balanced) instead of 15x+1x (the
+            # linspace+int-truncate skew).
+            picks = (
+                np.round(np.linspace(0, n_kf - 1, num_frames_to_sample))
+                .astype(int)
+                .tolist()
+            )
+            target_pts = [kf_pts[i] for i in picks]
+
+            # SLICE threading only affects decode, not the demux pass above.
+            stream.thread_type = "SLICE"
+            unique_pts = list(dict.fromkeys(target_pts))
+            pts_to_frame: dict[int, npt.NDArray] = {}
+            for tpts in unique_pts:
+                # seek() lands on the keyframe at-or-before tpts; tpts came
+                # from the keyframe PTS list, so the next decoded frame is
+                # exactly that keyframe.
+                container.seek(tpts, stream=stream)
+                frame = next(container.decode(stream), None)
+                if frame is None:
+                    raise ValueError(
+                        f"pyav_keyframes: keyframe pts={tpts} demuxed but "
+                        "no frame decoded; bitstream may be corrupt"
+                    )
+                pts_to_frame[tpts] = frame.to_ndarray(format="rgb24")
+
+            frames_list = [pts_to_frame[tpts] for tpts in target_pts]
+            valid_indices = [
+                int(round(float(tpts * time_base) * src_fps)) for tpts in target_pts
+            ]
+
+        return np.stack(frames_list), cls.create_hf_metadata(
+            source=source,
+            video_backend="pyav_keyframes",
+            valid_frame_indices=valid_indices,
+        )