[Bugfix] Fix multimodal sender-cache invalidation race with queued renders by YueshenZ · Pull Request #62164 · vllm-project/vllm

YueshenZ · 2026-10-17T08:48:00Z

Overview

Fixes #62121. Apply sender-cache invalidations between generation renders on the single-worker multimodal executor so an engine cache-miss report cannot remove an entry between the processor's cache lookup and merge.

Claims

In-flight generation renders on the single-worker multimodal executor complete without the reported Expected a cached item assertion; already-queued renders consume pending invalidations before looking up cached data.
Unrelated outputs keep flowing while multimodal processing or cache invalidation is blocked.
Ignore uncached hashes and coalesce duplicate miss reports, avoiding unbounded notification growth on pre-rendered features traffic that never invokes the renderer.

Validation

Current review cleanup (20f04478): both helpers return when the cache is absent; enqueue also returns for empty input or an empty filtered pending set. The output handler delegates empty-list handling to the helper while keeping its outer cache guard and enqueue-before-publish ordering.

macOS arm64 CPU: pytest -p pytest_skipgc tests/multimodal/test_cache.py tests/renderers/test_warmup.py tests/renderers/test_executor_replace.py -k "not sleep" -q → 57 passed, 1 deselected. The cleanup opt-out is described below; test assertions were unchanged.
NVIDIA A800 80GB, Qwen2-VL-2B @ 895c3a49: the unmodified pytest tests/multimodal/test_cache.py::test_sleep_wake_preserves_mm_cache_consistency -s -vv -ra → 1 passed, 0 skipped. Logs confirm both generations and the intervening sleep/wake; sleep freed 38.05 GiB. Python sources were verified against 20f04478; native artifacts were reused from official wheel 7b6310c1 with PyTorch 2.14.0+cu130 / Transformers 5.19.0, and all model-file hashes matched the pinned revision.
A real-renderer check with the invalidation lock held confirmed that empty lists, empty tuples and fully filtered hashes return without waiting or changing an existing pending invalidation. The next render resends the payload, and the following render hits the cache.
Changed-file pre-commit checks passed, including ruff, format, typos, mypy, SPDX/import and configuration checks. actionlint was excluded because its local environment could not initialize; no workflow files changed. git diff --check passed.

The offline processor performance measurements compare a0e7137c with d2097ba5, not this cleanup: initial benchmark, three-repeat 1,024-prompt results. These synchronous measurements do not cover async miss-notification contention.

Coalescing implementation (d2097ba5), earlier macOS arm64 CPU validation, Python 3.13.15 / PyTorch 2.14.1:

pytest -p pytest_skipgc tests/multimodal/test_cache.py -k "not sleep" -q: 29 passed, 1 deselected. Includes LRU, SHM/IPC, the original forced-interleaving regression, coalescing/clear behavior, and a multi-hash failure with concurrent notifications.
pytest -p pytest_skipgc tests/renderers/test_warmup.py tests/renderers/test_executor_replace.py -q: 28 passed.
The local pytest_skipgc plugin applies the repository's skip_global_cleanup marker to avoid the previously observed macOS torch teardown crash; it does not modify test assertions. SHM tests ran with local shared-memory access enabled.
Changed-file pre-commit hooks passed, including ruff, format, typos, mypy and Buildkite test tethering. actionlint was skipped; no workflow file changed. git diff --check passed.
A method-level comparison using the real renderer/processor/LRU cache and the submitted dc1a962 queue/drain methods reproduced the backlog: 1,000 absent hashes retained 1,000 records before and 0 after; 200 notifications for one cached hash retained 200 before and 1 after. The next render resent the payload in both cases. This was not a full baseline-tree pytest run.

The original assertion regression previously failed on pristine a0e7137c and passed on submitted commit dc1a962. That submitted implementation also has Linux CPU validation (27 passed, 1 deselected) and A800 serving evidence below. The historical HTTP stress measurements below remain specific to dc1a962. The d2097ba5 offline benchmarks and 20f04478 sleep/wake test do not replace a rerun of that HTTP stress workload.

Historical A800 serving comparison for submitted commit dc1a962

Qwen2-VL-2B @ 895c3a49, A800 80GB PCIe, localhost HTTP using the reporter's pause/resume plus background-image load, no server-side hook. Five-minute trigger and two-minute control per arm:

arm	rounds	target asserts	image requests	ok	other 500	stream errors
main trigger	164	18	32,138	31,794	196	130
`dc1a962` trigger	164	0	31,934	31,565	199	170
main control	56	0	24,632	24,632	0	0
`dc1a962` control	56	0	25,070	25,070	0	0

Twelve fixed public-image responses were byte-identical for text, finish reason and completion-token count. Remaining errors were the existing P1-miss class; internal-error lines increased from 344 to 369 (1.07% to 1.16%). No overall error-rate improvement is claimed.

Details

The output handler previously invalidated the sender cache on the asyncio thread while the renderer performed its lookup and merge on its single-worker multimodal executor.

The handler now records missed hashes before publishing the associated outputs. Only hashes currently present in the sender cache are retained. When a processor cache is present, a short lock protects a pending set; each render swaps that set before its first lookup. Both helpers return immediately when there is no cache. Enqueue also returns for empty input or when filtering leaves no live hashes, before acquiring the lock. Membership checks and cache invalidation execute outside the lock. Failed invalidation restores the current and remaining hashes without discarding newly arrived notifications. Cache clear retains its existing behavior; stale notifications become no-ops at the next render boundary.

This avoids a second executor job and queue/set mirrored state. Pending records can briefly outlive the cache entry they reference, so this does not claim that their count always equals the instantaneous live cache size. Sync LLM retains direct invalidation.

This is distinct from #34749's assertion-to-error conversion and #61656's endpoint retries. P1 drift itself, automatic HTTP retries, and #62123's SHM handle issue are out of scope. The pre-existing pooling/embedding cross-executor race is also not addressed by this fix. The issue came from a public report and was investigated with constructed reproductions; it was not encountered in the submitter's production workload. AI assistance was used for investigation, implementation and validation, including this follow-up review.

Pull Request Checklist

I used vLLM's /pr-checklist skill.
AI assistance was used during the creation of this PR.
Design Fit: The change stays within renderer cache invalidation and output handling; no scheduler or model-runner changes.
Testing and Validation: CPU regression coverage and historical serving evidence are distinguished above; tests remain in the existing Buildkite-wired suite.
Code Quality and Style: Concise comments, a shared renderer fixture, and changed-file checks.
Pull Request Contents: Scope, evidence, remaining error class and validation limits are explicit.

…nders The AsyncLLM output handler invalidated the sender (P0) multimodal cache on the asyncio thread while the renderer reads that cache twice on its single-worker multimodal executor. An invalidation between the is_cached lookup and the get_and_update merge left the processor with hash-only input and no cached data, surfacing as 'AssertionError: Expected a cached item' (HTTP 500) under concurrent cache drift recovery. Queue missed hash strings on a per-renderer SimpleQueue before publishing the miss outputs, and drain the queue at every _process_multimodal entry before the first cache lookup. A running render completes undisturbed; a render queued before the miss observes the invalidation before processing. No extra executor job, Future callback or event-loop wait is introduced; unrelated outputs keep flowing while the multimodal worker is occupied. If invalidate raises, the current hash is requeued and the exception propagates to the render. Co-authored-by: Claude Code <noreply@anthropic.com> Signed-off-by: Shen <312058264+YueshenZ@users.noreply.github.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

DarkLight1337 · 2026-10-17T09:41:02Z

@claude review

claude

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Ignore miss hashes absent from the sender cache and coalesce repeat notifications in a pending set. Keep the lock limited to set updates and swaps so output handling does not wait for cache invalidation. Restore all unfinished hashes on failure without losing concurrent notifications. Cover absent and duplicate hashes, cache clear and resend, and failed invalidation with concurrent misses. Reuse the existing renderer fixture. Validated with 29 cache tests and 28 renderer tests on CPU; applicable changed-file pre-commit checks passed. This follow-up has no new GPU run. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Shen <312058264+YueshenZ@users.noreply.github.com>

DarkLight1337 · 2026-10-19T07:03:39Z

@claude review

DarkLight1337 · 2026-10-19T07:04:21Z

Could you run vllm bench mm-processor to ensure there are no performance regressions?

claude

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

YueshenZ · 2026-10-19T07:50:58Z

Ran vllm bench mm-processor comparing the PR base a0e7137c with current head d2097ba5 on an A800 80GB, Qwen2-VL-2B @ 895c3a49, BF16/TP=1/eager. The initial experiment uses three fresh processes per revision/cache setting, alternating order, 256 synthetic prompts after 16 warmups. All 3,072 measured requests completed.

Below are means across the three runs, with their min/max. The P95/P99 rows are averages of per-run quantiles.

Cache	Metric	Base mean [min, max]	Patch mean [min, max]	Change
4 GiB	Batch wall time (s)	5.3228 [5.3169, 5.3301]	5.3362 [5.3292, 5.3462]	+0.25%
4 GiB	Preprocessing mean (ms)	2.7862 [2.6805, 2.8712]	2.8160 [2.7919, 2.8339]	+1.07%
4 GiB	Preprocessing P95 (ms)	4.3179 [4.0639, 4.5351]	4.3649 [4.3162, 4.4063]	+1.09%
4 GiB	Preprocessing P99 (ms)	4.4947 [4.2661, 4.7086]	4.7631 [4.4970, 5.2637]	+5.97%
0 GiB	Batch wall time (s)	5.3268 [5.3096, 5.3414]	5.3314 [5.3216, 5.3379]	+0.09%
0 GiB	Preprocessing mean (ms)	2.4980 [2.4281, 2.5433]	2.5266 [2.4928, 2.5716]	+1.15%
0 GiB	Preprocessing P95 (ms)	3.8566 [3.7235, 3.9292]	3.8966 [3.8504, 3.9720]	+1.04%
0 GiB	Preprocessing P99 (ms)	3.9404 [3.8573, 4.0005]	4.4177 [3.9112, 5.1061]	+12.11%

The mean batch-time changes are small (+0.25% with caching, +0.09% without), but the initial preprocessing P99s were higher. I therefore ran a separate 1,024-prompt/32-warmup pair per cache setting; all 4,096 requests completed:

Cache	Metric	Base	Patch	Change
4 GiB	Batch wall time (s)	20.5814	20.6302	+0.24%
4 GiB	Preprocessing mean (ms)	2.8296	2.7502	-2.81%
4 GiB	Preprocessing P95 (ms)	4.5388	4.4108	-2.82%
4 GiB	Preprocessing P99 (ms)	4.8359	4.5907	-5.07%
0 GiB	Batch wall time (s)	20.4615	20.5256	+0.31%
0 GiB	Preprocessing mean (ms)	2.5320	2.4987	-1.32%
0 GiB	Preprocessing P95 (ms)	4.1531	4.0509	-2.46%
0 GiB	Preprocessing P99 (ms)	4.6277	4.2601	-7.94%

The larger-sample pairs did not reproduce the initial P99 increase. Tail results remain variable; these measurements show small observed mean-time differences in this tested configuration.

Since the internal stage timer excludes the added drain, I also timed complete real render_cmpl calls over three repetitions: mean changes were -0.24% for cache hits, +0.55% for unique misses and +0.58% with caching disabled. Those measurements include the new empty-drain/lock. These are offline/synchronous measurements; they do not exercise AsyncLLM's output loop, actual miss notifications, or contention.

Reproduction and environment

Linux, Python 3.12.3, PyTorch 2.14.0+cu130, Transformers 5.19.0, NVIDIA driver 580.105.08. Both exact Python source revisions use the same official 7b6310c1 precompiled artifacts; that wheel commit differs from the base only in two AMD DeepSeek model files. The pinned model files and changed-source hashes were checked.

For batch wall time, both benchmark copies identically export the existing total_time variable as an additional JSON field; the measured processing is unchanged. The source PR was not changed.

export OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 TOKENIZERS_PARALLELISM=false
export VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_USE_DEEP_GEMM=0 VLLM_PLUGINS=""
# MODEL is a local snapshot of Qwen/Qwen2-VL-2B-Instruct at
# 895c3a49bc3fa70a340399125c650a463535e71c.
# Run CACHE_GB=4 and 0; three fresh processes per revision/configuration.
vllm bench mm-processor \
  --model "$MODEL" --dataset-name random-mm \
  --num-prompts 256 --num-warmups 16 --seed 42 \
  --random-input-len 128 --random-output-len 1 --random-range-ratio 0 \
  --random-mm-base-items-per-request 1 --random-mm-num-mm-items-range-ratio 0 \
  --random-mm-limit-mm-per-prompt '{"image":1,"video":0}' \
  --random-mm-bucket-config '{(224,224,1):0.5,(672,672,1):0.5}' \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --max-model-len 2048 --max-num-seqs 32 --no-enable-prefix-caching \
  --enforce-eager --gpu-memory-utilization 0.5 \
  --mm-processor-cache-type lru --mm-processor-cache-gb "$CACHE_GB" \
  --metric-percentiles 50,95,99 --disable-tqdm --output-json "$RESULT"

The separate follow-up changes only --num-prompts 1024 --num-warmups 32.

DarkLight1337 · 2026-10-20T03:12:11Z

I therefore ran a separate 1,024-prompt/32-warmup pair per cache setting; all 4,096 requests completed

Could you also run each one three times so we could see the variance?

YueshenZ · 2026-10-20T04:41:56Z

Reran the 1,024-prompt/32-warmup configuration three times per revision and cache setting — 12 fresh processes run back-to-back (~14 minutes total), with arm order alternated between repetitions (base→patch, patch→base, base→patch). Every run used the same --seed 42, so all 12 processes measured an identical workload and the within-cell spread below is run-to-run noise rather than workload variance; the 32 warmup requests per run (generated with seed+1) are excluded from the reported stats and counters. All 12,288 measured requests completed (0 failed, 1,024 encoder calls per run), on the same machine and pinned revisions/environment as above.

Cache	Metric	Base mean [min, max] ±sd	Patch mean [min, max] ±sd	Change
4 GiB	Batch wall time (s)	20.9549 [20.9107, 20.9972] ±0.0432	20.9913 [20.9102, 21.1007] ±0.0984	+0.17%
4 GiB	Preprocessing mean (ms)	2.6666 [2.6308, 2.7185] ±0.0460	2.6487 [2.6349, 2.6750] ±0.0227	−0.67%
4 GiB	Preprocessing P95 (ms)	4.1552 [4.0838, 4.2659] ±0.0972	4.1041 [4.0898, 4.1317] ±0.0239	−1.23%
4 GiB	Preprocessing P99 (ms)	4.4172 [4.2597, 4.7121] ±0.2555	4.7795 [4.2109, 5.7618] ±0.8542	+8.20%
0 GiB	Batch wall time (s)	21.0546 [20.9100, 21.1993] ±0.1446	21.0481 [20.9224, 21.2288] ±0.1604	−0.03%
0 GiB	Preprocessing mean (ms)	2.4183 [2.3949, 2.4436] ±0.0244	2.4242 [2.4114, 2.4447] ±0.0179	+0.24%
0 GiB	Preprocessing P95 (ms)	3.7883 [3.7541, 3.8319] ±0.0398	3.8048 [3.7785, 3.8466] ±0.0366	+0.44%
0 GiB	Preprocessing P99 (ms)	4.3345 [3.9094, 5.0754] ±0.6439	4.5604 [3.9218, 5.8303] ±1.0998	+5.21%

(sd is the sample standard deviation of the three runs; Change is mean-of-three vs mean-of-three.)

Reading the variance:

Batch wall time and preprocessing mean/P95 differ by at most 1.23% between arms, sign-mixed, and every base–patch gap is smaller than the larger within-cell sd of the same row; the [min, max] ranges overlap in all eight cells.
The P99 means look higher with the patch (+8.2%/+5.2%), but that is driven by single elevated runs: the three largest P99s overall are 5.83/5.76 ms (patch, first repetition) and 5.08 ms (base, third repetition), while each cache setting's patch arm also contains its cell's smallest or second-smallest P99. Median-of-three P99 changes are +2.0% (4 GiB) and −2.2% (0 GiB).
Welch t-tests on all eight cells give |t| ≤ 0.88 (largest at 4 GiB P95; df 2.2–4.0, two-sided p ≥ 0.46); with three runs per cell nothing approaches significance.
Both arms' absolute wall times are ~1.7–2.9% higher than the October 19 single-pair run (e.g. 4 GiB base 20.95 s vs 20.58 s), so day-to-day drift is also larger than the arm difference.

Under this configuration the run-to-run variance dominates; no metric shows a patch effect beyond the spread of the three repetitions.

Per-run values, in execution order

#	Run	Wall (s)	Mean (ms)	P95 (ms)	P99 (ms)
1	4 GiB base, rep 1	20.9107	2.6308	4.0838	4.2799
2	4 GiB patch, rep 1	20.9629	2.6750	4.1317	5.7618
3	0 GiB base, rep 1	20.9100	2.4163	3.7788	4.0187
4	0 GiB patch, rep 1	21.2288	2.4447	3.8466	5.8303
5	4 GiB patch, rep 2	20.9102	2.6363	4.0909	4.2109
6	4 GiB base, rep 2	20.9972	2.7185	4.2659	4.7121
7	0 GiB patch, rep 2	20.9224	2.4114	3.7892	3.9291
8	0 GiB base, rep 2	21.1993	2.3949	3.7541	3.9094
9	4 GiB base, rep 3	20.9569	2.6506	4.1158	4.2597
10	4 GiB patch, rep 3	21.1007	2.6349	4.0898	4.3658
11	0 GiB base, rep 3	21.0546	2.4436	3.8319	5.0754
12	0 GiB patch, rep 3	20.9932	2.4164	3.7785	3.9218

…ache Hoist the cache checks in queue_mm_cache_invalidations and _drain_mm_cache_invalidations into early returns, per review: both helpers are no-ops when the mm processor cache is unset, so return before the pending-set work instead of guarding the queue body and the drain loop. Semantics are unchanged: the cache reference is only assigned during renderer initialization and never goes back to None, so pending entries cannot exist while the cache is unset. Validated with tests/multimodal/test_cache.py and the sparse-tensor and warmup renderer tests on an offline A800 host; pass sets match the parent revision exactly. ruff check and format pass. Co-authored-by: Claude Code <noreply@anthropic.com> Signed-off-by: Shen <312058264+YueshenZ@users.noreply.github.com>

Return before locking when incoming hashes or filtered pending entries are empty. Let output handling call the helper directly while retaining the cache guard and enqueue-before-publish ordering. Validated with 57 CPU cache/renderer regressions, the unchanged GPU sleep/wake test on A800, and a real-renderer probe showing empty calls do not block on the invalidation lock or discard existing notifications. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Shen <312058264+YueshenZ@users.noreply.github.com>

DarkLight1337

Thanks for your patience!

YueshenZ · 2026-10-20T23:00:00Z

/ci run

github-actions · 2026-10-20T23:00:34Z

✅ Triggered Buildkite CI #96285 for commit da2339077b28.

ℹ️ This PR is 225 commits behind upstream main. Merged upstream main into this branch as da2339077b28 before starting CI. Pull the branch before pushing more changes.

YueshenZ · 2026-10-21T01:51:54Z

Thanks again for the review, @DarkLight1337! The Buildkite checks are now green. The GitHub pre-commit workflow is still gated by pre-run-check because my account has fewer than four merged PRs. Could you please add the appropriate CI approval label when you have a chance? Thanks!

DarkLight1337 · 2026-10-21T03:35:22Z

/ci run

github-actions · 2026-10-21T03:35:28Z

✅ @YueshenZ, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

github-actions · 2026-10-21T03:35:41Z

✅ Triggered Buildkite CI #96323 for commit bcd85c99778d.

YueshenZ · 2026-10-21T06:16:27Z

/ci retry

github-actions · 2026-10-21T06:16:59Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #96323.

…nders (vllm-project#62164) Signed-off-by: Shen <312058264+YueshenZ@users.noreply.github.com> Co-authored-by: Shen <312058264+YueshenZ@users.noreply.github.com> Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: vllm-agent <claw@inferact.ai> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

YueshenZ requested review from DarkLight1337, NickLucche, njhill, shen-shanshan and ywang96 as code owners October 17, 2026 08:48

claude Bot reviewed Oct 17, 2026

View reviewed changes

mergify Bot added multi-modality Related to multi-modality (#4194) bug Something isn't working labels Oct 17, 2026

claude Bot reviewed Oct 17, 2026

View reviewed changes