Skip to content

[Perf] Avoid repeated multimodal prompt update scans - #51774

Merged
DarkLight1337 merged 3 commits into
vllm-project:mainfrom
gty111:perf/batched-mm-prompt-updates
Aug 11, 2026
Merged

DarkLight1337 merged 3 commits into
vllm-project:mainfrom
gty111:perf/batched-mm-prompt-updates

Conversation

@gty111

@gty111 gty111 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Avoid quadratic prompt-update planning when many multimodal items share the
same target.

For non-empty replacements, the previous implementation applies one item per
round. Each round still walks all unresolved items and rebuilds the match list,
resulting in roughly O(N²) item processing.

This change compiles updates with identical ordered (mode, target) choices
into FIFO queues. Matching only examines the next item in each queue and
consumes items in their original priority order. The common shared-target path
becomes O(prompt length + output length + item count).

Alternative targets, insertion and replacement modes, PromptIndex,
overlapping matches, and modality priority retain their existing behavior.

This is not a duplicate: no open PR addresses repeated shared-target update
rounds; #50716 optimizes individual scans but does not remove the per-item
rounds.

AI assistance was used.

Performance

Synthetic apply_token_matches benchmark:

  • 10,000 image placeholders
  • 1,000,000 input tokens
  • 50 replacement tokens per image
  • 1,490,000 output tokens
Implementation Time
Previous implementation 85.97 s
This PR 0.086 s

This is approximately a 995x speedup. The expanded tokens and match results
are identical.

Test Plan

.venv/bin/python -m pytest tests/multimodal/test_processing.py -q
.venv/bin/python -m pytest tests/models/multimodal/processing/test_common.py \
  -k 'Qwen2-VL or llava-1.5 or SmolVLM2-2.2B' -q
pre-commit run ruff-check --files \
  vllm/multimodal/processing/processor.py tests/multimodal/test_processing.py
pre-commit run ruff-format --files \
  vllm/multimodal/processing/processor.py tests/multimodal/test_processing.py
pre-commit run mypy-3.12 --files \
  vllm/multimodal/processing/processor.py tests/multimodal/test_processing.py \
  --hook-stage manual

Test Result

  • 95 processing tests passed.
  • 9 model processing tests passed.
  • 20,000 randomized differential cases matched the previous behavior.
  • All staged pre-commit hooks passed.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the multi-modality Related to multi-modality (#4194) label Aug 11, 2026
Comment thread vllm/multimodal/processing/processor.py
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Comment thread tests/multimodal/test_processing.py Outdated
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>

@DarkLight1337 DarkLight1337 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM now

@DarkLight1337
DarkLight1337 enabled auto-merge (squash) August 11, 2026 07:32
@DarkLight1337

Copy link
Copy Markdown
Member

/ci run

@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 11, 2026
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83315 for commit c390e7c284c8.

@DarkLight1337
DarkLight1337 merged commit 3fb7bb4 into vllm-project:main Aug 11, 2026
78 checks passed
zyp2014 pushed a commit to zyp2014/vllm that referenced this pull request Aug 21, 2026
)

Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

multi-modality Related to multi-modality (#4194) ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants