Repository navigation
[Feat][MM Cache] Skip media loading & decoding if UUID based cache lookup succeeds #55583
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 5 commits
9b8363f
1a6a430
5853f06
0bade85
46cd0e0
166ffdb
5a234b2
3d28775
623e1d2
5deff0a
d442a08
1348d4e
b8b67c7
a0fd123
2214793
bd588bc
5586325
6bfac47
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1406,3 +1406,19 @@ Just like with offline inference, you can skip sending media if you expect cache | |
| }, | ||
|
|
||
| ``` | ||
|
|
||
| #### Early UUID Cache Lookups | ||
|
|
||
| When using the HF renderer, vLLM can probe the multi-modal processor cache | ||
| before loading an image or video URL. Off by default, opt in via `VLLM_EARLY_UUID_LOOKUPS=1`. | ||
|
|
||
| The first request with an image or video URL and UUID loads and processes the | ||
| media as usual. Repeated requests probe multimodal cache first: | ||
| * ***On a hit*** vLLM skips loading and decoding the URL. | ||
| * ***On a cache miss*** vLLM uses the URL as a fallback and processes the media normally. | ||
|
|
||
| A race condition is possible if the multimodal cache is evicted between the early lookup and media loading. | ||
| Use `skip_early_mm_lookup` to opt-out per request. | ||
|
|
||
| The UUID is authoritative on a cache hit; vLLM does not verify that the URL | ||
| still identifies the cached media (see [Multimodal Media UUID Security](../usage/security.md#multimodal-media-uuid-security)) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -C 12 \
'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|uuid' \
vllm/multimodal vllm/entrypointsRepository: vllm-project/vllm Length of output: 50374 🤖 get_repo_knowledge executed:
Length of output: 621 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- matching source files ---'
rg -l \
'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|MultiModal.*Cache|uuid' \
vllm/multimodal vllm/entrypoints \
-g '*.py' | head -80
printf '%s\n' '--- focused matches ---'
rg -n -C 8 \
'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup' \
vllm/multimodal vllm/entrypoints \
-g '*.py' | head -1200Repository: vllm-project/vllm Length of output: 41781 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- focused source matches ---'
rg -n -C 8 \
'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup' \
vllm/multimodal vllm/entrypoints \
-g '*.py' | head -1200Repository: vllm-project/vllm Length of output: 40008 🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n \
'class BaseMultiModalProcessorCache|def (get|set).*cache|multi_modal_uuids|skip_early_mm_lookup' \
vllm/multimodal vllm/entrypoints \
-g '*.py' -g '!**/compiled/**' -g '!**/third_party/**'Repository: vllm-project/vllm Length of output: 1761 🏁 Script executed: #!/bin/bash
set -euo pipefail
git ls-files 'vllm/multimodal/*.py' 'vllm/multimodal/**/*.py' | head -100Repository: vllm-project/vllm Length of output: 1246 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- UUID lookup path ---'
sed -n '600,690p' vllm/entrypoints/chat_utils.py
sed -n '690,770p' vllm/entrypoints/chat_utils.py
printf '%s\n' '--- cache key and implementations ---'
sed -n '150,220p' vllm/multimodal/cache.py
sed -n '321,540p' vllm/multimodal/cache.py
printf '%s\n' '--- hash construction ---'
rg -n -C 12 'get_mm_item_hash|class ProcessorInputs|mm_hash' \
vllm/multimodal/processing/inputs.py vllm/multimodal/hasher.pyRepository: vllm-project/vllm Length of output: 22136 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- complete hash function ---'
sed -n '30,86p' vllm/multimodal/processing/inputs.py
printf '%s\n' '--- UUID lookup callers and resolution ---'
rg -n -C 12 \
'is_mm_item_cached\(|uuid.*url|parse_(image|video)|resolve_items|MultiModalUUIDItems' \
vllm/entrypoints/chat_utils.py vllm/multimodal/parse.py vllm/multimodal/processing \
-g '*.py' | head -1000
printf '%s\n' '--- security documentation ---'
rg -n -C 20 'Multimodal Media UUID Security|UUID.*authoritative|uuid.*security' \
docs/usage/security.md docs/features/multimodal_inputs.mdRepository: vllm-project/vllm Length of output: 50373 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- documented UUID security contract ---'
rg -n 'Multimodal Media UUID Security|UUID|uuid' docs/usage/security.md | tail -80
sed -n '1390,1445p' docs/features/multimodal_inputs.md
printf '%s\n' '--- exact security section ---'
python3 - <<'PY'
from pathlib import Path
p = Path("docs/usage/security.md")
text = p.read_text()
needle = "Multimodal Media UUID Security"
start = text.find(needle)
print(text[start:start+3500] if start >= 0 else "section not found")
PYRepository: vllm-project/vllm Length of output: 6425 IDOR (CWE-639): Authorization Bypass Through User-Controlled Key (IDOR) Reachability: External · Exploitability: Difficult Do not treat UUID-authoritative hits as an authorization boundary. The cache intentionally uses 🤖 Prompt for AI Agents |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -74,6 +74,7 @@ | |
| VLLM_MEDIA_CACHE: str = "" | ||
| VLLM_MEDIA_CACHE_MAX_SIZE_MB: int = 5120 | ||
| VLLM_MEDIA_CACHE_TTL_HOURS: float = 24 | ||
| VLLM_EARLY_UUID_LOOKUPS: bool = False | ||
| VLLM_MEDIA_FETCH_MAX_RETRIES: int = 3 | ||
| VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB: int = 256 | ||
| VLLM_MEDIA_URL_ALLOW_REDIRECTS: bool = True | ||
|
|
@@ -978,6 +979,12 @@ def _resolve_rust_cli_path() -> str | None: | |
| "VLLM_MEDIA_CACHE_TTL_HOURS": lambda: float( | ||
| os.getenv("VLLM_MEDIA_CACHE_TTL_HOURS", "24") | ||
| ), | ||
| # Whether to probe the multi-modal processor cache before loading image or | ||
| # video URLs with UUIDs. Cache hits skip URL loading and decoding. Disabled | ||
| # by default. | ||
| "VLLM_EARLY_UUID_LOOKUPS": lambda: bool( | ||
| int(os.getenv("VLLM_EARLY_UUID_LOOKUPS", "0")) | ||
| ), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -C 12 \
'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|VLLM_EARLY_UUID_LOOKUPS|uuid' \
vllm/multimodal vllm/entrypointsRepository: vllm-project/vllm Length of output: 50374 🤖 get_repo_knowledge executed:
Length of output: 627 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- candidate files ---'
fd -t f . vllm/multimodal vllm/entrypoints | rg 'cache|processor|multi_modal|multimodal|protocol|api' | head -200
printf '%s\n' '--- exact symbols ---'
rg -n -g '*.py' \
'class BaseMultiModalProcessorCache|BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|VLLM_EARLY_UUID_LOOKUPS|early.*lookup|uuid' \
vllm/multimodal vllm/entrypoints --glob '!**/static/**' --glob '!**/*.min.js' \
| head -400Repository: vllm-project/vllm Length of output: 34214 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- cache-related files ---'
rg --files vllm | rg 'multimodal|multi_modal|cache' | head -200
printf '%s\n' '--- UUID and early-lookup references, excluding generated/static content ---'
rg -n -g '*.py' \
'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|VLLM_EARLY_UUID_LOOKUPS|early_mm|uuid' \
vllm/multimodal vllm/entrypoints \
--glob '!**/static/**' --glob '!**/*.min.js' \
| head -500Repository: vllm-project/vllm Length of output: 36349 🏁 Script executed: #!/bin/bash
set -euo pipefail
for f in $(rg --files vllm | rg 'multimodal|multi_modal' | rg '\.py$' | head -120); do
if rg -q 'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|VLLM_EARLY_UUID_LOOKUPS' "$f"; then
echo "===== $f ====="
rg -n -C 8 'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|VLLM_EARLY_UUID_LOOKUPS' "$f"
fi
doneRepository: vllm-project/vllm Length of output: 9970 🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -g '*.py' \
'BaseMultiModalProcessorCache|multi_modal_uuids|skip_early_mm_lookup|VLLM_EARLY_UUID_LOOKUPS|early_mm_lookup' \
vllm | head -600Repository: vllm-project/vllm Length of output: 7370 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- definitions and callers ---'
rg -n -g '*.py' \
'class BaseMultiModalProcessorCache|def .*lookup|def .*cache|multi_modal_uuids|skip_early_mm_lookup|VLLM_EARLY_UUID_LOOKUPS' \
vllm/multimodal vllm/entrypoints \
--glob '!**/static/**' \
| head -500Repository: vllm-project/vllm Length of output: 5718 🤖 get_repo_knowledge executed:
Length of output: 597 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import urllib.request
urls = [
"https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/multimodal/processing.py",
"https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/multimodal/cache.py",
]
for url in urls:
try:
data = urllib.request.urlopen(url, timeout=10).read().decode()
except Exception as exc:
print(f"{url}: {exc}")
continue
print(f"===== {url} =====")
lines = data.splitlines()
for i, line in enumerate(lines):
if any(x in line for x in (
"BaseMultiModalProcessorCache", "multi_modal_uuids",
"skip_early_mm_lookup", "VLLM_EARLY_UUID_LOOKUPS",
)):
lo, hi = max(0, i-12), min(len(lines), i+30)
print("\n".join(f"{j+1}:{lines[j]}" for j in range(lo, hi)))
PYRepository: vllm-project/vllm Length of output: 596 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '===== vllm/entrypoints/chat_utils.py:590-675 ====='
sed -n '590,675p' vllm/entrypoints/chat_utils.py
printf '%s\n' '===== vllm/multimodal/cache.py:1-220 ====='
sed -n '1,220p' vllm/multimodal/cache.py
printf '%s\n' '===== vllm/multimodal/cache.py:321-540 ====='
sed -n '321,540p' vllm/multimodal/cache.pyRepository: vllm-project/vllm Length of output: 17193 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '===== early lookup call sites ====='
sed -n '610,690p' vllm/entrypoints/chat_utils.py
sed -n '1040,1090p' vllm/entrypoints/chat_utils.py
sed -n '1250,1295p' vllm/entrypoints/chat_utils.py
sed -n '1715,1770p' vllm/entrypoints/chat_utils.py
printf '%s\n' '===== cache construction and key operations ====='
sed -n '130,215p' vllm/multimodal/cache.py
sed -n '321,445p' vllm/multimodal/cache.py
sed -n '444,525p' vllm/multimodal/cache.pyRepository: vllm-project/vllm Length of output: 18334 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '===== cache wiring ====='
rg -n -C 8 \
'processor_cache_from_config|mm_processor_cache=|MultiModalProcessorOnlyCache|MultiModalProcessorSenderCache|MultiModalUUIDDict|uuid: str \| None' \
vllm/entrypoints vllm/renderers vllm/multimodal vllm/inputs \
--glob '*.py' | head -500
printf '%s\n' '===== UUID request fields ====='
sed -n '350,410p' vllm/entrypoints/openai/chat_completion/protocol.py
sed -n '225,255p' vllm/entrypoints/openai/responses/protocol.pyRepository: vllm-project/vllm Length of output: 32997 IDOR (CWE-639): Authorization Bypass Through User-Controlled Key (IDOR) Reachability: External Enforce tenant isolation for early UUID lookups. The API accepts caller-provided UUIDs, and the UUID is used as the cache identity without tenant or authorization context. In a multi-tenant deployment, a caller who knows another tenant’s UUID can select that tenant’s cached media when 🤖 Prompt for AI Agents
Jankwi marked this conversation as resolved.
Resolved
|
||
| # Maximum number of retries for fetching media (images, audio, video) | ||
| # from URLs. Each retry quadruples the timeout. Default is 3. | ||
| "VLLM_MEDIA_FETCH_MAX_RETRIES": lambda: int( | ||
|
|
@@ -2346,6 +2353,7 @@ def compile_factors() -> dict[str, object]: | |
| "VLLM_MEDIA_CACHE", | ||
| "VLLM_MEDIA_CACHE_MAX_SIZE_MB", | ||
| "VLLM_MEDIA_CACHE_TTL_HOURS", | ||
| "VLLM_EARLY_UUID_LOOKUPS", | ||
| "VLLM_MEDIA_FETCH_MAX_RETRIES", | ||
| "VLLM_MEDIA_URL_ALLOW_REDIRECTS", | ||
| "VLLM_MEDIA_LOADING_THREAD_COUNT", | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Keep the media URL available after an early UUID cache hit.
BaseMultiModalProcessorCache.is_cached_item()only checks membership and does not update eviction order. The image and video parsers then set the URL toNone, whileget_and_update()performs a second lookup with the same UUID hash. If eviction occurs between these operations, no cached media or URL fallback remains, and the request can fail. Retain the URL for a miss fallback or pin the entry until processing completes.🤖 Prompt for AI Agents