[aoshen02]

Purpose

BaseMultiModalProcessor._apply_prompt_updates assumes the prompt it receives is in the pre-update form (one placeholder token per item) and never checks that assumption. When the prompt is already expanded, the compact target still matches the first token of the expanded run, so a second copy of the replacement is spliced in — silently.

Reproduced on Qwen3-VL-2B-Instruct with one 224×224 image, calling the processor directly (no server, no GPU):

compact prompt      : 11 tokens,  <|image_pad|> run = [1]
after apply #1      : 74 tokens,  run = [64]   placeholders = offset=4, length=64   ← correct
feed the 74 back in :
after apply #2      : 137 tokens, run = [127]  placeholders = offset=4, length=64   ← wrong

The prompt grows by 63 tokens and the placeholder range is not updated to match. The 64 image features are written over the first 64 positions of a 127-token run; the remaining 63 positions keep the <|image_pad|> row of the embedding table, which is never trained. The request returns 200 with no warning.

This is reachable from every entry point that pairs a token sequence with media, not just one endpoint:

• /inference/v1/generate with content_parts ([Frontend] Add content_parts to /inference/v1/generate for raw multim… vllm-project/vllm#51478) — the multi-turn RL pattern, where the ids returned by /inference/v1/generate ([Frontend] Return prompt metadata from /inference/v1/generate vllm-project/vllm#53187) are fed back with the same images;
• offline LLM.generate(prompt_token_ids=..., multi_modal_data=...);
• any caller that re-submits a rendered prompt.

For context, I checked how other engines behave when handed an already-expanded prompt plus media: TensorRT-LLM raises ValueError, LMDeploy raises assert/IndexError, SGLang errors out or falls back with a warning. vLLM is the one that corrupts the prompt and returns success.

Changes

A new _find_applied_updates, next to the _find_mm_placeholders it delegates to, returns the placeholders only when every item's post-update content is already present, and None otherwise. _apply_prompt_updates returns the prompt untouched on a hit and is unchanged on a miss:

applied = self._find_applied_updates(token_ids, mm_prompt_updates)
if applied is not None:
    return token_ids, applied

Two properties worth checking, because they are what makes this safe to put in a function shared by all models:

1. The change is strictly a skip. Content matching tells you what the prompt contains, not whether it has been processed: with a replacement of [1] → [1, 1], the prompt [1, 1] is both "two untouched targets" and "one applied content", and no amount of local inspection separates them. So an incomplete match is treated as no evidence at all and falls through to the normal path. Nothing that is updated today stops being updated, and nothing that succeeds today starts failing. test_apply_prompt_updates_expands_targets_aliasing_one_content pins exactly that [1, 1] case.

2. Detection does not assume a content is a run of one repeated token. A content like [1, 2] expands a prompt [1] to [1, 2], and the check has to recognise that on the way back in. Pinned by test_apply_prompt_updates_is_idempotent_for_mixed_content. Relatedly, a content of length 1 is identical before and after the update, so it is excluded from the evidence entirely (test_apply_prompt_updates_still_expands_single_token_items).

Known limitation, unchanged by this PR: a prompt that mixes expanded and unexpanded items is still mis-expanded, because it lands in the ambiguous case above. Fixing it needs either run-boundary-aware matching or an explicit caller-side contract, and both are out of scope here. vllm-project#57781 is the complementary backstop: with both applied, a compact prompt expands, a fully expanded one is a no-op, and a mixed one raises. I verified that combination end to end (results below). Callers that need mixed input normalise on their side today (OpenRLHF's dedup_media_tokens, verl's qwen2_5_vl_dedup_image_tokens, TRL vllm-project#6718/vllm-project#6719, NeMo-RL vllm-project#4182 all collapse expanded runs back to compact before calling vLLM).

Cost

The scan is preceded by a cheap necessary condition: a content cannot be a slice of the prompt unless the prompt holds its most repeated token at least as many times as the content does. This is a lower bound for any content shape, so it can never reject a prompt that really is expanded — at worst it fails to skip and the scan runs anyway. It costs one list.count() per distinct probe, which keeps the common (unexpanded) path off the scan entirely:

prompt len	with the guard	scan unconditionally	saved
203	12.0 µs	34.5 µs	3×
8003	98.5 µs	742.0 µs	8×
32003	218.9 µs	3378.8 µs	15×

Per _apply_prompt_updates call on Qwen3-VL-2B-Instruct, against the rest of the work:

images	prompt len	form	added code	rest of apply	full apply()
1	223	compact	3.9	14.8	5.0 ms
1	8803	compact	82.6	50.3	4.1 ms
1	33003	compact	276.0	166.9	4.7 ms
8	8824	compact	79.5	123.2	88.3 ms
1	8998	expanded	79.3	skipped	4.1 ms
8	10384	expanded	124.9	skipped	88.3 ms

Against the full multimodal apply() (dominated by image preprocessing) that is ~2% on a typical 8.8k prompt. Items of the same modality share a probe token, so the guard does not grow with item count. In the already-expanded case the apply is skipped entirely.

I surveyed the 83 in-tree _get_prompt_updates implementations for contents where the guard degenerates — i.e. where the most repeated token is an ordinary token with a low count, so the condition always holds and the scan always runs. All of them are video/audio paths where frame separators or timestamp text dominate: qwen3_vl.py get_video_repl under EVS pruning, nano_nemotron_vl's per-frame prose separators, llava_onevision2's codec video runs, and moss_audio's time markers. In each the scan is a few ms against an apply() dominated by media decoding, and correctness is unaffected either way.

Why this is not a duplicate

Searched open PRs/issues for _apply_prompt_updates, find_mm_placeholders, prompt_token_ids_are_expanded, already expanded placeholder, and PRs referencing vllm-project#51472. Nothing overlaps. Adjacent but distinct: vllm-project#57781 is the converse direction (a placeholder token the caller wrote themselves stealing an item's position) and composes with this — see above. vllm-project#53169 (WIP) hits the same hazard on the features path and works around it by running the processor over a synthetic placeholder-only prompt — its workaround becomes unnecessary once this lands, but it does not fix the processor. vllm-project#57740 / vllm-project#57763 are instances of the vllm-project#57781 direction on Qwen2.5-VL.

Test Plan

• tests/multimodal/test_processing.py: four new unit tests on the existing _text_fallback_processor harness — idempotence, idempotence for a content that is not a single repeated token, the aliasing case that must still expand, and the single-token-content regression guard.
• End-to-end on the real Qwen3-VL-2B-Instruct processor (CPU only), alone and with [Bugfix][Multimodal] Reject prompts with placeholder tokens outside the recorded placeholders vllm-project/vllm#57781 applied on top.
• Full tests/multimodal/ suite, compared against the same suite on a clean tree.

Test Result

.venv/bin/python -m pytest tests/multimodal/test_processing.py -q
# 90 passed, 21 failed

The 21 failures are all llava-hf/llava-v1.6-mistral-7b-hf config downloads hitting a PermissionError in this machine's HF cache; they are unrelated to the change and reproduce on a clean tree. The full tests/multimodal/ failure set is identical to the clean-tree baseline.

Real-processor check, two images expanding to 64 and 196 tokens:

1. compact -> expanded   : 12 -> 270 tokens, ranges [offset=4 len=64, offset=72 len=196]
2. expanded -> unchanged : PASS (270 tokens, identical ranges)
3. mixed (1st expanded)  : 75 -> 333 tokens, no error   (unchanged from main)
4. mixed (2nd expanded)  : 207 -> 465 tokens, no error  (unchanged from main)

With vllm-project#57781 applied on top (it patches cleanly, no conflicts; both PRs' unit tests pass together, 37 passed):

1. compact -> expanded   : 12 -> 270 tokens
2. expanded -> unchanged : PASS (270 tokens)
3. mixed (1st expanded)  : VLLMValidationError - 323 <|image_pad|> tokens, 260 accounted for
4. mixed (2nd expanded)  : VLLMValidationError - 455 <|image_pad|> tokens, 260 accounted for

Model evaluation is not applicable to the fixed path: today it produces a corrupted prompt, so there is no meaningful "before" quality number. Every other path is byte-identical.

Developed with AI assistance (Claude Code). The human submitter has reviewed every changed line.

🤖 Generated with Claude Code