[Multimodal] Use GPU NVDEC for EPD encoder-only instance video media IO - #53675
Conversation
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Isotr0py
changed the title
|
Documentation preview: https://vllm--53675.org.readthedocs.build/en/53675/ |
mergify
Bot
added
documentation
|
This pull request has merge conflicts that must be resolved before it can be |
# Conflicts: # examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py Signed-off-by: Isotr0py <Isotr0py@outlook.com>
video_embeds support is already upstream; restore qwen3_vl.py and test_audio.py to the upstream versions so this branch only carries the torchcodec NVDEC device feature. Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
| device_type, | ||
| ) | ||
|
|
||
| def _resolve_mm_video_decode_device(self) -> None: |
There was a problem hiding this comment.
Am I missing something? Where does the check for encode-only instance occur?
There was a problem hiding this comment.
Oh, I missed this one, fixed in https://github.com/vllm-project/vllm/pull/53675/changes/84b1d07d55e22bdfa0980b3905a8ab147e03c9e8..62adfadc317bc604e5f4d87e64a6cfad8353e86b
(I think we should clean the processor/mediaio device resolver function in followup PR, it's a bit messy 😅)
There was a problem hiding this comment.
Wait, after second thought, I think we should tie video decode device with mm processor device instead, otherwise it will cause duplicated d2h/h2d transfer. 🤔
|
This pull request has merge conflicts that must be resolved before it can be |
# Conflicts: # tests/multimodal/test_parse.py # vllm/multimodal/parse.py Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
| runtime_kwargs.pop("pool_size", None) | ||
| # The decode device determines which device the decoded frames | ||
| # (and thus the processor pipeline) live on; startup-only. | ||
| runtime_kwargs.pop("device", None) |
There was a problem hiding this comment.
When an encoder defaults to {"backend": "torchcodec", "device": "cuda"}, a request overriding backend to "opencv" retains device="cuda". This fails with ValueError: device is not supported by the 'opencv' backend.
Could we clear the automatically injected codec-specific options when the request switches backends?
There was a problem hiding this comment.
Done in 8effb2c. We will further clean up current kwargs resolving in following PR, current dictionary is not flexible and clear enough.
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
|
/ci run |
|
✅ Triggered Buildkite CI #88285 for commit |
3 participants
Purpose
Test Plan
disagg_epd_proxy.py; ECExampleConnector with node-local shared storageQwen/Qwen3.5-35B-A3B(bf16),--enforce-eager, PD--gpu-memory-utilization 0.8,--enable-mm-embedsvllm bench serve --dataset-name hf --dataset-path yale-nlp/MMVU, burst mode (--request-rate inf),--max-concurrency 64, seed 0Test Result
fps=2 tier (30 frames/video)
fps=4 tier (60 frames/video)
All 1600/1600 requests succeeded in each run; zero NVDEC fallbacks (no
cpu_fallbackwarnings in encoder logs).Per-stage breakdown (proxy logs, per-request averages, run 1)
Attribution: NVDEC decode vs GPU preprocessing
Ablation on an isolated encode-only instance (same clean node, back-to-back runs; MMVU, 256 prompts, concurrency 64):
Total A→D: 4.9x encoder-side. In the full 1E3PD pipeline this dilutes to +94-115% because the PD replicas become the bottleneck.
The encode stage (fetch + decode + HF preprocess + ViT + EC publish) dominates the baseline. With tc_cuda it shrinks ~14x, and the PD side becomes the system bottleneck (decode_ttfb/total rise as PD replicas saturate) — i.e., the encoder is no longer the limiter, which is the intended operating point for EPD scaling.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.