[Don't merge][Experiment] Move MediaIO decode to MM preprocessing by Isotr0py · Pull Request #57752 · vllm-project/vllm

Isotr0py · 2026-09-20T05:00:06Z

Purpose

Move media decoding (image/video/audio) out of the chat parsing layer and
into the mm processor, behind the processor cache. Downloads stay where they
are (async, concurrent, in the renderer); what crosses the boundary is now a
LazyMedia handle (encoded bytes + decode callable) instead of a decoded
object.

LazyMedia (vllm/multimodal/media/base.py) decodes on first access,
thread-safe, with map() for stacked transforms and release_bytes() for
early memory release. Hashing uses the encoded bytes (images keep their
exact eager-path hashes via an eager header-open that preserves EXIF
ImageID / io_config semantics), so a processor-cache hit skips decoding
entirely and the raw bytes are freed right after hashing.
apply() is split into two phases: phase 1 (hash, cache lookup, submit
miss-item decode futures) runs on the single-worker _mm_executor; the
decode await happens on the event loop; phase 2 (HF processing, cache
writes) runs back on the worker and re-derives the cache state. The worker
never blocks on a decode join, so multi-second video decodes overlap across
requests while the P0/P1 cache ordering protocol is preserved. Processors
overriding apply() / _cached_apply_hf_processor (EncDec, ASR, several VL
models) automatically keep the previous blocking path via
supports_two_phase_apply.
Memory: cache-hit items cost zero pixel memory; original bytes are released
after hashing (hit) or after HF processing (miss).

Design doc: docs/design/lazy-media-decode.md.

Intentional behavior changes:

Corrupt-media decode errors now surface as HTTP 422
(VLLMUnprocessableEntityError) from the processor instead of 400 at fetch
time.
Audio/video cache keys are derived from the encoded bytes instead of the
decoded waveform/frames (in-process cache; keys change, no persistence
concerns). Image keys are unchanged.
Byte-consuming model processors (e.g. Dots3 NOTE) keep access to
original_bytes through HF processing on the cached path; miss-item bytes
are released only afterwards (try/finally).

Related / duplicate-work check

Test Plan

Unit / component (from the worktree):

.venv/bin/python -m pytest tests/multimodal/ -q
.venv/bin/python -m pytest tests/renderers/ -q
.venv/bin/python -m pytest tests/entrypoints/unit_tests/test_chat_utils.py \
    tests/renderers/test_process_multi_modal_uuids.py -q

Microbenchmark:

.venv/bin/python benchmarks/benchmark_lazy_decode.py

E2E serving benchmark — hardware: one GB200 node (single GPU per run),
Qwen/Qwen2.5-VL-7B-Instruct from a local HF cache:

export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 TOKENIZERS_PARALLELISM=false
export CUDA_VISIBLE_DEVICES=0

# Server (one per side under test). --mm-processor-cache-gb 24 keeps all 175
# MMVU videos cached between the back-to-back video runs; --allowed-local-
# media-path is bench-only (pools and the MMVU snapshot are local files).
PYTHONPATH=$CODE .venv/bin/python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-VL-7B-Instruct --served-model-name qwen-vl --port 8201 \
  --max-model-len 32768 --gpu-memory-utilization 0.9 --enforce-eager \
  --mm-processor-cache-gb 24 --allowed-local-media-path / \
  --no-enable-log-requests

# Warmup (also primes the mm processor cache for the image-pool runs):
PYTHONPATH=$CODE .venv/bin/python -m vllm.entrypoints.cli.main bench serve \
  --backend openai-chat --base-url http://127.0.0.1:8201 \
  --model Qwen/Qwen2.5-VL-7B-Instruct --served-model-name qwen-vl \
  --endpoint /v1/chat/completions \
  --dataset-name custom_image --dataset-path pool_hits.jsonl \
  --num-prompts 16 --max-concurrency 8 --request-rate inf \
  --output-len 32 --ignore-eos --seed 42

# 720p image pool, mm cache hot (pool_hits.jsonl: 8 JPEGs cycled over 400
# varied prompts):
PYTHONPATH=$CODE .venv/bin/python -m vllm.entrypoints.cli.main bench serve \
  --backend openai-chat --base-url http://127.0.0.1:8201 \
  --model Qwen/Qwen2.5-VL-7B-Instruct --served-model-name qwen-vl \
  --endpoint /v1/chat/completions \
  --dataset-name custom_image --dataset-path pool_hits.jsonl \
  --num-prompts 400 --max-concurrency 32 --request-rate inf \
  --output-len 32 --ignore-eos --seed 42 --save-result \
  --result-dir results --result-filename hits-$TAG.json

# Unique noise images, mm cache never hits (miss-path regression check):
PYTHONPATH=$CODE .venv/bin/python -m vllm.entrypoints.cli.main bench serve \
  --backend openai-chat --base-url http://127.0.0.1:8201 \
  --model Qwen/Qwen2.5-VL-7B-Instruct --served-model-name qwen-vl \
  --endpoint /v1/chat/completions \
  --dataset-name random-mm \
  --random-mm-base-items-per-request 1 \
  --random-mm-bucket-config "{(720,1280,1):1.0}" \
  --random-mm-num-mm-items-range-ratio 0.0 --random-input-len 64 \
  --num-prompts 400 --max-concurrency 32 --request-rate inf \
  --output-len 32 --ignore-eos --seed 42 --save-result \
  --result-dir results --result-filename miss-$TAG.json

# 4K image pool, mm cache hot (prime first with --num-prompts 16
# --max-concurrency 8 against pool_4k.jsonl):
PYTHONPATH=$CODE .venv/bin/python -m vllm.entrypoints.cli.main bench serve \
  --backend openai-chat --base-url http://127.0.0.1:8201 \
  --model Qwen/Qwen2.5-VL-7B-Instruct --served-model-name qwen-vl \
  --endpoint /v1/chat/completions \
  --dataset-name custom_image --dataset-path pool_4k.jsonl \
  --num-prompts 400 --max-concurrency 32 --request-rate inf \
  --output-len 32 --ignore-eos --seed 42 --save-result \
  --result-dir results --result-filename hits4k-$TAG.json

# Video (MMVU): run 1 is cold (miss); run 2 re-sends the same 200 videos in
# the same order against the same server, so the server-side mm processor
# cache is hot (hit):
for run in miss hit; do
  PYTHONPATH=$CODE .venv/bin/python -m vllm.entrypoints.cli.main bench serve \
    --backend openai-chat --base-url http://127.0.0.1:8201 \
    --model Qwen/Qwen2.5-VL-7B-Instruct --served-model-name qwen-vl \
    --endpoint /v1/chat/completions \
    --dataset-name hf --dataset-path yale-nlp/MMVU \
    --num-prompts 200 --max-concurrency 32 --request-rate inf \
    --output-len 32 --ignore-eos --seed 42 --save-result \
    --result-dir results --result-filename video-$run-$TAG.json
done

Model-level eval: not run — the change does not alter model numerics
(identical decoded pixels/frames reach the HF processor; hash-parity tests
cover cache-key stability). A tests/models/multimodal sweep on GPU is
recommended before merge.

Test Result

Unit (on top of upstream/main 4cc15f2):

tests/multimodal/: 637 passed, 1 skipped; 5 failed — all reproduce on
clean upstream/main (GPU/NVDEC/model-download fixtures unavailable here).
tests/renderers/: 514 passed; 2 failed — reproduce on clean
upstream/main (missing optional cohere_melody; gated model download).
tests/entrypoints/unit_tests/test_chat_utils.py +
test_process_multi_modal_uuids.py: 115 passed.

E2E (all runs 400/400 image and 175/175 video requests completed):

Scenario	Baseline req/s	This PR req/s	Baseline mean TTFT	This PR mean TTFT	Delta
720p image pool (cache hit)	28.29	28.66	257.6 ms	246.1 ms	+1% / -4%
random-mm images (cache miss)	19.72	21.79	446.4 ms	274.9 ms	+10% / -38%
4K image pool (cache hit)	4.41	4.44	1403.1 ms	1111.2 ms	parity / -21%
MMVU video (cache miss)	2.05	1.97	11.22 s	11.47 s	-4% / +2% (noise)
MMVU video (cache hit)	9.04	10.00	932.0 ms	732.0 ms	+11% / -21%

Microbenchmark highlights (benchmarks/benchmark_lazy_decode.py, CPU):

Cache-hit cost per 2048x2048 JPEG request: 33.0 ms -> 4.1 ms (8.1x); PNG
fixed from parity to 3.5-4.2x after gating the Pillow-12 getexif() hidden
decode.
Parallel decode of 8x 2048x2048 JPEGs on the media pool: 222 ms -> 31 ms
(7.2x, 8 workers).
RSS for 100 cached 1024x1024 JPEG hits: +362 MB eager -> +0.3 MB lazy.

AI-assisted contribution

This contribution was developed with AI assistance (Kimi Code). A human
submitter has reviewed the changes and ran the tests listed above.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. — N/A (no public API/config change); design doc included under docs/design/.

Move media decoding out of the fetch/chat-parsing layer into the multi-modal processor, per docs/design/lazy-media-decode.md: - New LazyMedia type (media/base.py): holds encoded bytes + a decode callable, decodes thread-safely on first access, supports map() for stacked transforms, release_bytes(), non-decoding __repr__, and pickle conversion to an eager MediaWithBytes. MediaIO gains default load_bytes_lazy/load_base64_lazy/load_file_lazy so all media subclasses switch without changes (VideoMediaIO overrides load_base64_lazy only for the jpeg_sequence base64 format). - MediaConnector now returns LazyMedia from load_from_url(_async) for http/data/file URLs; download, disk cache, and domain/size checks are unchanged. fetch_image* no longer wraps UnidentifiedImageError since decode errors move to the processor. - The hasher hashes LazyMedia from original_bytes (or header EXIF / io_config when header_image is set) and never triggers pixel decode. - parse.py keeps items lazy: audio stacks resample/channel-normalize onto the decode via map(), video unpacks frames/metadata lazily and defers the video_needs_metadata check to decode time, and ProcessorBatchItems.get_raw() allows cache-miss selection without triggering decode. - The processor adds _decode_lazy_items: cache-miss items are decoded in parallel on the connector's global_thread_pool, in-flight decodes are always awaited, and the first failure is wrapped as VLLMUnprocessableEntityError (422). Cache-hit items are never decoded and their bytes are released right after hashing. Timed via a new "decode_mm_items" span. Intentional behavior changes: decode errors on media fetched by URL now surface as 422 instead of 400, and audio/video cache hashes switch from decoded content to original bytes (in-process cache only). Tests: new LazyMedia unit tests (tests/multimodal/media/test_base.py), no-decode hashing cases (tests/multimodal/test_hasher.py), lazy parse cases (tests/multimodal/test_parse.py), cache-hit-skips-decode / parallel-decode / decode-error-to-422 cases (tests/multimodal/test_processing.py), and updated connector + entrypoint test unwrap sites. Verified: tests/multimodal/media/, test_hasher.py, test_parse.py, test_processing.py all pass; test_cache.py passes except test_sleep_wake_preserves_mm_cache_consistency which fails identically on the base commit (missing compiled vllm_flash_attn in this environment); 4 test_video.py GPU/model-download failures and 12 test_chat_utils.py collection errors are likewise pre-existing. Co-authored-by: Kimi Code <noreply@moonshot.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>

…emantics - image: eager header-open + lazy pixel-decode; io_config/EXIF computed at fetch time so lazy hashes match the eager path exactly - audio: eager encoded-size validation for load_bytes/load_base64/load_file lazy variants - video: jpeg_sequence validation stays eager, frame decode deferred; GPU/NVDEC backend selection happens at decode time - chat_utils: vision_chunk lazy items pre-decoded off the event loop; decode errors propagate as 422 instead of being swallowed - hasher: lazy image branch prefixes raw bytes with the b"image" key to match the eager MediaWithBytes path - processor: cache-miss items keep original_bytes through HF processing (byte-consuming models like dots3_note), released in finally afterwards - dots3_note: accept LazyMedia alongside MediaWithBytes for raw-bytes video - tests: laziness, hash equivalence, parallel decode, 422 error mapping, deferred metadata checks Co-authored-by: Kimi Code <noreply@moonshot.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>

Co-authored-by: Kimi Code <noreply@moonshot.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>

- hasher: guard the EXIF probe for header-opened PNGs (Pillow 12 PngImageFile.getexif() triggers a full load() when info lacks exif); PNG cache hits are now hash-bound instead of decode-bound - benchmarks/benchmark_lazy_decode.py: microbenchmark for the lazy decode frontend gains (hit/miss cost per modality, hash throughput, parallel decode, memory) Co-authored-by: Kimi Code <noreply@moonshot.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>

_validate_mm_uuids enumerated ModalityDataItems, which unwraps via __getitem__ -> .media and triggers a full decode of every lazy item on the single-worker _mm_executor before the cache lookup. For 30-frame videos this serialized the whole mm queue and regressed video TTFT up to 3x on cache hits. Validate on the raw reference (get_raw) instead; the check only needs item-is-None. Validated on GB200 (Qwen2.5-VL-7B, MMVU, 64-prompt sample): video-hit 9.96 req/s (baseline 9.04), video-miss TTFT 14.1s (was 15.7s, baseline 11.2s -- residual miss-path gap tracked as follow-up). Co-authored-by: Kimi Code <noreply@moonshot.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>

_process_multimodal_async now runs phase 1 (hash, cache lookup, submit miss-item decode futures) on the single-worker _mm_executor, awaits the decodes on the event loop, and runs phase 2 (HF processing, merge, cache writes) back on the worker. The worker is no longer blocked on decode joins, restoring cross-request decode overlap that the serial worker previously serialized for multi-second video decodes. Phase 2 re-derives the cache state so interleaved requests handle miss->hit and hit->evicted flips correctly; all cache reads/writes stay serialized on the worker, preserving the P0/P1 ordering protocol. Processors overriding apply/_cached_apply_hf_processor (EncDec, ASR, and several VL models) keep the previous blocking path via supports_two_phase_apply. GB200 smoke (Qwen2.5-VL-7B, MMVU, 64 prompts): video-miss TTFT 14.1s -> 10.9s (baseline 11.2s); video-hit 10.3 req/s (baseline 9.04). Co-authored-by: Kimi Code <noreply@moonshot.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>

mergify · 2026-09-20T05:00:42Z

Documentation preview: https://vllm--57752.org.readthedocs.build/en/57752/

mergify · 2026-09-20T05:11:21Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

DarkLight1337 · 2026-09-21T15:27:19Z

@claude review If you were given the opportunity to redesign the chat_utils media loading integration with MM processor, what are the three main changes you would make?

claude

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

DarkLight1337 · 2026-09-30T07:35:52Z

@claude review

DarkLight1337 · 2026-09-30T07:51:57Z

@claude review

claude

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

claude

Code review found no issues

No high-confidence issues detected in this change.

This review covers commit d81c52f, which is no longer the latest commit on this pull request; later commits are not covered by it.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Preserve no-cache processor overrides and borrowed media refs, include image decoder settings in JPEG-sequence keys, and drain two-phase processing on cancellation. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 · 2026-09-30T10:45:12Z

@claude review

claude

This automated review ran again on the latest commits and found no new bugs. The previously flagged issues from earlier passes now appear genuinely fixed in the code (not just closed as threads):

MediaRef.release() now clears _decoder as well as _data, so the decode closure's own reference to the encoded bytes is actually dropped (vllm/multimodal/media/base.py, image.py)
The async HTTP fetch path now builds refs inside run_in_executor instead of calling the decoder-building code directly on the event loop (vllm/multimodal/media/connector.py)
VideoProcessorItems.metadata now resolves per-item metadata for MediaRef-backed videos instead of always returning None (vllm/multimodal/parse.py)
Decode-error messages now report the original per-modality item index via get_original_index, not the compacted miss-list index (vllm/multimodal/processing/processor.py)

Given the size and complexity of this refactor to the core multimodal decode pipeline, and that the PR is still explicitly tagged "[Don't merge][Experiment]", a human look remains worthwhile before this merges.

Extended reasoning...

This run's diff (d81c52f..HEAD) is a follow-up to a 39-file, ~4.7k-line lazy-media-decode refactor across vllm/multimodal, vllm/renderers, and connector/processor code; the latest commits touch media/base.py, media/image.py, media/video.py, parse.py, processing/inputs.py, processing/processor.py and renderers/base.py. No auth/crypto surface is touched, but it does touch the media fetch/decode/cache-keying path, which is security-relevant for correctness of size limits and domain allow-listing (unchanged in this diff). I independently re-read each of the four previously-flagged locations and confirmed the fixes in the current code (decoder dropped on release, executor-wrapped async ref building, per-item metadata resolution, original-index decode errors) rather than relying on thread-resolution metadata alone. The bug-hunting system reported zero new findings this run and the exit reason was dry_streak, but the change remains large, experimental, and explicitly marked not for merging, so I'm deferring rather than approving.

mergify · 2026-09-30T11:54:07Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

mergify · 2026-10-05T10:56:53Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

noooop · 2026-10-09T08:01:42Z

@@ -0,0 +1,533 @@
+# SPDX-License-Identifier: Apache-2.0


INTEL(R) XEON(R) GOLD 6554S

hasher algorithm: blake3 Pillow 12.3.0, numpy 2.3.5 ======================================================================================== IMAGES: per-request frontend cost (mean +/- std, ms) ======================================================================================== size fmt bytes eager(old) ref hit ref miss hit speedup ---------------------------------------------------------------------------------------- 512x512 JPEG 0.16M 3.81 +/- 0.11 0.22 +/- 0.00 3.50 +/- 0.06 17.4x 512x512 PNG 0.79M 8.14 +/- 0.03 0.79 +/- 0.01 8.46 +/- 0.11 10.3x 1024x1024 JPEG 0.63M 18.16 +/- 0.08 0.66 +/- 0.01 16.00 +/- 0.07 27.5x 1024x1024 PNG 3.15M 32.76 +/- 0.08 3.03 +/- 0.01 34.37 +/- 0.31 10.8x 2048x2048 JPEG 2.52M 71.95 +/- 0.60 2.48 +/- 0.02 64.47 +/- 1.89 29.1x 2048x2048 PNG 12.60M 102.69 +/- 2.46 12.09 +/- 0.87 106.78 +/- 0.16 8.5x 3840x2160 JPEG 4.98M 110.10 +/- 0.61 4.77 +/- 0.00 102.57 +/- 1.08 23.1x 3840x2160 PNG 24.90M 146.42 +/- 0.83 23.35 +/- 0.02 158.16 +/- 1.90 6.3x Blended per-request cost at cache hit rate p: new = p*hit + (1-p)*miss size fmt old (eager) new p=0.5 saved new p=0.9 saved new p=0.99 saved --------------------------------------------------------------------------------------- 512x512 JPEG 3.81 1.86 51% 0.55 86% 0.25 93% 512x512 PNG 8.14 4.62 43% 1.56 81% 0.87 89% 1024x1024 JPEG 18.16 8.33 54% 2.19 88% 0.81 96% 1024x1024 PNG 32.76 18.70 43% 6.16 81% 3.34 90% 2048x2048 JPEG 71.95 33.47 53% 8.68 88% 3.10 96% 2048x2048 PNG 102.69 59.44 42% 21.56 79% 13.03 87% 3840x2160 JPEG 110.10 53.67 51% 14.55 87% 5.75 95% 3840x2160 PNG 146.42 90.76 38% 36.83 75% 24.70 83% ======================================================================================== AUDIO (16 kHz mono WAV): per-request frontend cost (mean +/- std, ms) ======================================================================================== duration bytes eager(old) ref hit ref miss hit speedup -------------------------------------------------------------------------------- 1s 0.03M 3.24 +/- 0.03 0.04 +/- 0.00 3.24 +/- 0.03 72.2x 3s 0.10M 6.49 +/- 0.73 0.10 +/- 0.00 6.28 +/- 0.04 61.9x 10s 0.32M 7.61 +/- 0.89 0.31 +/- 0.00 7.59 +/- 0.03 24.2x Note: eager hashes the decoded float32 PCM (4 B/sample) while a ref hashes the encoded WAV (2 B/sample), so the hit path also halves the bytes fed to the hasher -- and no longer resamples to get them. ======================================================================================== VIDEO (2 s, 30 fps, 640x480 mp4): per-request frontend cost (ms) ======================================================================================== note: the eager path hashes the decoded frames and metadata; a ref hashes the encoded bytes and its decode spec. encoded size: 5.54 MB eager(old) : 32.42 +/- 1.46 ref hit : 5.21 +/- 0.01 (6.2x) ref miss : 29.95 +/- 0.12 ======================================================================================== HASH THROUGHPUT (blake3) vs decode cost ======================================================================================== payload hash ms GB/s --------------------------- 1MB 0.243 4.02 8MB 1.938 4.03 32MB 7.824 3.99 Hash vs full eager decode for the image matrix above: size fmt bytes hash ms decode ms decode/hash ------------------------------------------------------------ 512x512 JPEG 0.16M 0.004 3.81 968x 512x512 PNG 0.79M 0.004 8.13 1852x 1024x1024 JPEG 0.63M 0.004 18.16 4282x 1024x1024 PNG 3.15M 0.004 32.76 7625x 2048x2048 JPEG 2.52M 0.005 71.94 15898x 2048x2048 PNG 12.60M 0.004 102.68 22996x 3840x2160 JPEG 4.98M 0.005 110.10 23367x 3840x2160 PNG 24.90M 0.005 146.42 30597x ======================================================================================== PARALLEL DECODE: 8 x 2048x2048 JPEG, global_thread_pool (8 workers) ======================================================================================== serial decode in-process : 420.45 ms thread pool (1 worker) : 518.72 ms global_thread_pool (8 workers): 76.98 ms wall-time speedup : 5.46x ======================================================================================== MEMORY: encoded bytes held vs decoded RGB pixels ======================================================================================== size fmt encoded decoded RGB ratio ------------------------------------------------- 512x512 JPEG 0.16M 0.79M 5.0x 512x512 PNG 0.79M 0.79M 1.0x 1024x1024 JPEG 0.63M 3.15M 5.0x 1024x1024 PNG 3.15M 3.15M 1.0x 2048x2048 JPEG 2.52M 12.58M 5.0x 2048x2048 PNG 12.60M 12.58M 1.0x 3840x2160 JPEG 4.98M 24.88M 5.0x 3840x2160 PNG 24.90M 24.88M 1.0x Cache hits avoid decoded pixels. Processing releases its forked wrappers after phase 2; caller-owned refs retain encoded payloads and image headers until their owners release or drop them. RSS for 100 retained 1024x1024 JPEG cache-hit items (630 KB encoded each): eager (decoded image kept): + 357.3 MB ref (original released): + 0.2 MB fork (parent retained) : + 0.2 MB One encoded payload is shared; RSS includes allocator reuse.

9800X3D

hasher algorithm: blake3 Pillow 12.3.0, numpy 2.3.5 ======================================================================================== IMAGES: per-request frontend cost (mean +/- std, ms) ======================================================================================== size fmt bytes eager(old) ref hit ref miss hit speedup ---------------------------------------------------------------------------------------- 512x512 JPEG 0.16M 1.35 +/- 0.12 0.08 +/- 0.00 1.16 +/- 0.01 16.7x 512x512 PNG 0.79M 2.65 +/- 0.01 0.32 +/- 0.00 2.79 +/- 0.00 8.3x 1024x1024 JPEG 0.63M 5.91 +/- 0.01 0.27 +/- 0.00 5.32 +/- 0.01 22.3x 1024x1024 PNG 3.15M 10.49 +/- 0.01 1.24 +/- 0.00 11.14 +/- 0.01 8.5x 2048x2048 JPEG 2.52M 23.57 +/- 0.02 1.00 +/- 0.00 21.26 +/- 0.02 23.5x 2048x2048 PNG 12.60M 33.16 +/- 0.01 4.90 +/- 0.00 35.90 +/- 0.02 6.8x 3840x2160 JPEG 4.98M 39.10 +/- 0.04 1.96 +/- 0.00 36.05 +/- 0.06 20.0x 3840x2160 PNG 24.90M 51.20 +/- 0.07 9.67 +/- 0.01 56.20 +/- 0.11 5.3x Blended per-request cost at cache hit rate p: new = p*hit + (1-p)*miss size fmt old (eager) new p=0.5 saved new p=0.9 saved new p=0.99 saved --------------------------------------------------------------------------------------- 512x512 JPEG 1.35 0.62 54% 0.19 86% 0.09 93% 512x512 PNG 2.65 1.56 41% 0.57 79% 0.34 87% 1024x1024 JPEG 5.91 2.79 53% 0.77 87% 0.32 95% 1024x1024 PNG 10.49 6.19 41% 2.23 79% 1.33 87% 2048x2048 JPEG 23.57 11.13 53% 3.03 87% 1.21 95% 2048x2048 PNG 33.16 20.40 38% 8.00 76% 5.21 84% 3840x2160 JPEG 39.10 19.01 51% 5.37 86% 2.30 94% 3840x2160 PNG 51.20 32.93 36% 14.32 72% 10.14 80% ======================================================================================== AUDIO (16 kHz mono WAV): per-request frontend cost (mean +/- std, ms) ======================================================================================== duration bytes eager(old) ref hit ref miss hit speedup -------------------------------------------------------------------------------- 1s 0.03M 1.08 +/- 0.04 0.02 +/- 0.00 1.07 +/- 0.01 62.5x 3s 0.10M 2.16 +/- 0.01 0.04 +/- 0.00 2.17 +/- 0.01 51.1x 10s 0.32M 2.49 +/- 0.01 0.13 +/- 0.00 2.55 +/- 0.01 19.3x Note: eager hashes the decoded float32 PCM (4 B/sample) while a ref hashes the encoded WAV (2 B/sample), so the hit path also halves the bytes fed to the hasher -- and no longer resamples to get them. ======================================================================================== VIDEO (2 s, 30 fps, 640x480 mp4): per-request frontend cost (ms) ======================================================================================== note: the eager path hashes the decoded frames and metadata; a ref hashes the encoded bytes and its decode spec. encoded size: 5.54 MB eager(old) : 16.61 +/- 0.88 ref hit : 2.16 +/- 0.01 (7.7x) ref miss : 17.44 +/- 0.39 ======================================================================================== HASH THROUGHPUT (blake3) vs decode cost ======================================================================================== payload hash ms GB/s --------------------------- 1MB 0.092 10.66 8MB 0.680 11.49 32MB 2.691 11.61 Hash vs full eager decode for the image matrix above: size fmt bytes hash ms decode ms decode/hash ------------------------------------------------------------ 512x512 JPEG 0.16M 0.002 1.35 881x 512x512 PNG 0.79M 0.002 2.65 1730x 1024x1024 JPEG 0.63M 0.002 5.91 3736x 1024x1024 PNG 3.15M 0.002 10.49 6720x 2048x2048 JPEG 2.52M 0.002 23.57 15077x 2048x2048 PNG 12.60M 0.002 33.16 21066x 3840x2160 JPEG 4.98M 0.002 39.10 24495x 3840x2160 PNG 24.90M 0.002 51.19 31145x ======================================================================================== PARALLEL DECODE: 8 x 2048x2048 JPEG, global_thread_pool (8 workers) ======================================================================================== serial decode in-process : 144.75 ms thread pool (1 worker) : 170.53 ms global_thread_pool (8 workers): 35.38 ms wall-time speedup : 4.09x ======================================================================================== MEMORY: encoded bytes held vs decoded RGB pixels ======================================================================================== size fmt encoded decoded RGB ratio ------------------------------------------------- 512x512 JPEG 0.16M 0.79M 5.0x 512x512 PNG 0.79M 0.79M 1.0x 1024x1024 JPEG 0.63M 3.15M 5.0x 1024x1024 PNG 3.15M 3.15M 1.0x 2048x2048 JPEG 2.52M 12.58M 5.0x 2048x2048 PNG 12.60M 12.58M 1.0x 3840x2160 JPEG 4.98M 24.88M 5.0x 3840x2160 PNG 24.90M 24.88M 1.0x Cache hits avoid decoded pixels. Processing releases its forked wrappers after phase 2; caller-owned refs retain encoded payloads and image headers until their owners release or drop them. RSS for 100 retained 1024x1024 JPEG cache-hit items (630 KB encoded each): eager (decoded image kept): + 360.3 MB ref (original released): + 0.0 MB fork (parent retained) : + 0.3 MB One encoded payload is shared; RSS includes allocator reuse.

Isotr0py added 6 commits September 19, 2026 13:06

[MM] Add lazy media decode design doc

68fd9ee

Co-authored-by: Kimi Code <noreply@moonshot.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>

mergify Bot added documentation Improvements or additions to documentation frontend multi-modality Related to multi-modality (#4194) performance Performance-related issues labels Sep 20, 2026

mergify Bot added the needs-rebase label Sep 20, 2026

DarkLight1337 mentioned this pull request Sep 21, 2026

[Frontend] Add video decode cache #47856

Open

DarkLight1337 marked this pull request as ready for review September 21, 2026 14:59

DarkLight1337 requested review from AndreasKaratzas, DarkLight1337, NickLucche, aarnphm, njhill, noooop, robertgshaw2-redhat, shen-shanshan, tjtanaa and ywang96 as code owners September 21, 2026 14:59

claude Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread vllm/multimodal/media/base.py Outdated

Comment thread vllm/multimodal/media/connector.py Outdated

DarkLight1337 marked this pull request as draft September 21, 2026 16:38

Merge branch 'main' into mm/processor-decode-refactor

ca360b3

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

claude Bot reviewed Sep 30, 2026

View reviewed changes

Merge branch 'main' into mm/processor-decode-refactor

d81c52f

claude Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread vllm/multimodal/processing/processor.py

Address comment

5f26f70

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

claude Bot reviewed Sep 30, 2026

View reviewed changes

Clean up code

36b67e8

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

mergify Bot removed the needs-rebase label Sep 30, 2026

DarkLight1337 and others added 3 commits September 30, 2026 08:50

More clean up

849c4f1

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Improve code quality

93518c2

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

claude Bot reviewed Sep 30, 2026

View reviewed changes

mergify Bot added the needs-rebase label Sep 30, 2026

DarkLight1337 added 3 commits September 30, 2026 14:58

Merge branch 'main' into mm/processor-decode-refactor

7effcc7

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix

ade1a91

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Unnecessary assert

248ac30

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

mergify Bot removed the needs-rebase label Sep 30, 2026

DarkLight1337 added 3 commits September 30, 2026 16:20

More cleanup

4c15108

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

More clean up

e05a51c

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Clean up

436e470

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

mergify Bot added the pooling label Oct 4, 2026

mergify Bot added the needs-rebase label Oct 5, 2026

sricursion mentioned this pull request Oct 6, 2026

[Bugfix][Frontend] Return 400 for images rejected by Pillow's bomb check #60308

Open

6 tasks done

jperezdealgaba mentioned this pull request Oct 8, 2026

[Security] Isolate OpenCV video decode in a child process #60612

Open

noooop reviewed Oct 9, 2026

View reviewed changes

luca-888 mentioned this pull request Oct 10, 2026

[Performance]: Resize video frames while decoding to skip full-resolution copies #60985

Open

1 task done

Conversation

Isotr0py commented Sep 20, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Related / duplicate-work check

Test Plan

Test Result

AI-assisted contribution

Uh oh!

mergify Bot commented Sep 20, 2026

Uh oh!

mergify Bot commented Sep 20, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

DarkLight1337 commented Sep 21, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

DarkLight1337 commented Sep 30, 2026

Uh oh!

DarkLight1337 commented Sep 30, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 commented Sep 30, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Sep 30, 2026

Uh oh!

mergify Bot commented Oct 5, 2026

Uh oh!

noooop Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

noooop Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Isotr0py commented Sep 20, 2026 •

edited by github-actions Bot

Loading