[Multimodal] Optimize sparse frames decoding for PyAV video backend

[Isotr0py]

Purpose

Test Plan

• Test on a video with 900 frames (30s, fps=30):

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[WindChimeRan]

I've opened #45203, which adds an opt-in lossy keyframe-only sampling policy (pyav_keyframes).

I think it's complementary to this PR. This PR makes decoding the same exact frames faster, while mine is keyframes-only for prefilling-heavy video classification scenarios, and pays for it in accuracy on motion-sensitive tasks.

[NicolasHug]

As requested by @Isotr0py I benchmarked this against TorchCodec and against the previous pyav backend in #46609 (see results there). Here are my 2 cents:

• This new pyav backend is much faster than the previous pyav backend on sparse sampling, but slower on dense sampling (with high number of threads, which is the current vllm default).
• TorchCodec is faster than both pyav backends on both sampling scenarios: TC is much faster on dense sampling , and a tiny bit faster on sparse sampling.

The reason TC is faster is that it's smart about when it should seek vs just decode frames forwards. I recently implemented a heuristic for that (see meta-pytorch/torchcodec#1488). You could implement the same heuristic for pyav, although of course it's up to you to decide whether you want to own that complexity within vllm, or let TorchCodec handle it.

My note also explains why your new pyav implementation is slower than the previous one on dense sampling: sometimes, it's just better to NOT seek! And this is especially true with large number of threads in FRAME threading (what this PR is doing).

---

What this PR is doing via _demux_keyframes() is what TorchCodec's does by default: it scans the file to find the keyframes, which allows for exact and fast seeking behavior. It's worth noting that this scan strategy can be detrimental:

• for long videos, it's sometimes just better not to scan, because the scan isn't free! See e.g. https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/approximate_mode.html#performance-frame-decoding-and-clip-sampling
• typically when the video is streaming from the web, you definitely don't want to scan for keyframes because that would download / materialize the entire stream in memory. Streaming videos is not supported by vllm at the moment I think, but it could be in the future?

TorchCodec solves all this already, and allows to not scan via seek_mode="approximate". If you want to allow that with the pyav backend, you'd need yet another implementation, potentially many.

---

On keyframe sampling: TorchCodec exposes a private VideoDecoder(...)._get_key_frame_indices() method. I'm happy to expose it publicly if you need it.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork