Skip to content

[Multimodal] Bound PyAV FFmpeg decode threads via num_ffmpeg_threads - #54118

Closed
liulanze wants to merge 1 commit into
vllm-project:mainfrom
liulanze:fix/pyav-bounded-decode-threads
Closed

liulanze wants to merge 1 commit into
vllm-project:mainfrom
liulanze:fix/pyav-bounded-decode-threads

Conversation

@liulanze

@liulanze liulanze commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

FIX #53973

The PyAV video backend sets stream.thread_type = "SLICE" but never sets thread_count, so FFmpeg auto-sizes the slice-thread pool to min(cpu_count + 1, 16) workers per open codec. Under serving concurrency this multiplies into far more decode threads than cores, and decode threads hang permanently in pthread_cond_wait inside avpriv_slicethread_execute. Each hung decode pins a media-pool slot forever and throughput collapses until restart.

This PR adds num_ffmpeg_threads for the pyav backend via --media-io-kwargs, mirroring the TorchCodec option, and defaults it to a bounded 4 so the stock serving configuration cannot oversubscribe.

Measured single-stream cost (1080p H.264, 60 frames, 12-CPU host):

content 0 (auto=13) 4 2 1
1 slice/frame (x264 default) 3352 ms 3376 ms 3453 ms 3321 ms
12 slices/frame (sliced-threads encode) 676 ms 1080 ms 1936 ms 3276 ms

Test Plan

python -m pytest tests/multimodal/test_video.py -k "pyav or backend_kwargs_are_separated or rejects_options or lazy_imported or imports_only_selected or edit_list or validates_frame_recovery" -v
  • test_pyav_backend_bounds_decode_threads (new, parametrized 2/0): asserts the codec context's thread_count — exact bound for N>0, resolved auto (>=1) for 0. Fails without the fix on any host (auto resolves to cpu_count + 1, never 2).
  • test_video_backend_kwargs_are_separated_from_sampling_kwargs: two new pyav rows — explicit override routing plus a pin on the bounded default.
  • test_video_backend_rejects_options_for_another_decoder: retargeted to seek_mode on pyav; also pins that opencv still rejects num_ffmpeg_threads.
  • test_pyav_backend_loads_frames: passes num_ffmpeg_threads=2 end-to-end through load_bytes.

Root-cause repro (before/after): 8-thread pool decoding 64 videos concurrently through the real loader.load_bytes(..., backend="pyav") path, sampling /proc/<pid>/task (the issue's accounting), on av 14.2.0 = libavcodec 61.19.100 (the exact version the issue reports).

Test Result

Unit tests all pass on Linux (python:3.12 container, VLLM_USE_PRECOMPILED=1 editable install at the base commit): 25 passed, 59 deselected in 98.02s. ruff check and ruff format --check clean.

Root-cause repro (12 CPUs, av 14.2.0, 8 concurrent decodes, 1280x720x60 H.264):

baseline threads peak delta model
before (thread_count unset) 23 127 +104 8 × (12+1) auto workers = 104 ✔
after (default 4) 23 55 +32 8 × 4 workers = 32 ✔

This PR was developed in an AI-assisted pair-programming workflow (Claude Code); the submitter reviewed every changed line and ran all tests.

…threads

The PyAV video backend set thread_type = "SLICE" without a thread_count,
so FFmpeg auto-sized min(cpu_count + 1, 16) slice workers per open codec.
Under serving concurrency this multiplies across media-pool slots and
processes into far more decode threads than cores, and decode threads hang
permanently in pthread_cond_wait inside avpriv_slicethread_execute
(PyAV-Org/PyAV#615 has the identical stack).

Expose num_ffmpeg_threads for the pyav backend through media_io_kwargs,
mirroring the torchcodec option (0 = FFmpeg auto sizing), and default it
to a bounded 4 so the stock serving configuration cannot oversubscribe.

FIX vllm-project#53973

Signed-off-by: Lanze Liu <lanzetech@gmail.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--54118.org.readthedocs.build/en/54118/

@mergify mergify Bot added documentation Improvements or additions to documentation multi-modality Related to multi-modality (#4194) labels Aug 27, 2026
@DarkLight1337
DarkLight1337 requested a review from Isotr0py August 28, 2026 04:26
@liulanze liulanze changed the title Bound PyAV FFmpeg decode threads via num_ffmpeg_threads [Multimodal] Bound PyAV FFmpeg decode threads via num_ffmpeg_threads Aug 28, 2026
@mergify

mergify Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liulanze.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Aug 29, 2026
@DarkLight1337

DarkLight1337 commented Aug 30, 2026 •

Copy link
Copy Markdown
Member

Closing as we have removed pyav backend (#54231)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation multi-modality Related to multi-modality (#4194) needs-rebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: PyAV video backend deadlocks under serving concurrency — unbounded SLICE threading in decode_frames

2 participants