[bugfix] fix minicpmv mm prompt placeholder parse error by yma11 · Pull Request #49162 · vllm-project/vllm

yma11 · 2026-07-20T07:53:34Z

Purpose

minicpm-v gets prompt placeholder parse error in latest main:

File "/workspace/mayan/vllm-714/vllm/entrypoints/offline_utils.py", line 151, in _preprocess_cmpl_one
    (engine_input,) = self._preprocess_cmpl(
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/entrypoints/offline_utils.py", line 139, in _preprocess_cmpl
    return renderer.render_cmpl(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/renderers/base.py", line 1003, in render_cmpl
    self.process_for_engine(prompt, arrival_time, skip_mm_cache=skip_mm_cache)
  File "/workspace/mayan/vllm-714/vllm/renderers/base.py", line 955, in process_for_engine
    engine_input = self._process_singleton(prompt, skip_mm_cache=skip_mm_cache)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/renderers/base.py", line 876, in _process_singleton
    return self._process_tokens(prompt, skip_mm_cache=skip_mm_cache)  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/renderers/hf.py", line 1192, in _process_tokens
    engine_input = super()._process_tokens(prompt, skip_mm_cache=skip_mm_cache)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/renderers/base.py", line 781, in _process_tokens
    engine_input = self._process_multimodal(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/renderers/base.py", line 762, in _process_multimodal
    mm_inputs = mm_processor.apply(mm_processor_inputs, mm_timing_ctx)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/multimodal/processing/processor.py", line 1689, in apply
    prompt_ids, mm_placeholders = self._maybe_apply_prompt_updates(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/mayan/vllm-714/vllm/multimodal/processing/processor.py", line 1659, in _maybe_apply_prompt_updates
    self._validate_mm_placeholders(mm_placeholders, mm_item_counts)
  File "/workspace/mayan/vllm-714/vllm/multimodal/processing/processor.py", line 1628, in _validate_mm_placeholders
    raise RuntimeError(
RuntimeError: Expected there to be 1 prompt placeholders corresponding to 1 image items, but instead found 0 prompt placeholders! Make sure the implementation of `_call_hf_processor` and `_get_mm_fields_config` are consistent with each other.

This PR also remove vendor minicpm processor as shifted to transformers 5.13.1

Test Plan

 python ./examples/generate/multimodal/vision_language_offline.py -m minicpmv

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

yma11 · 2026-07-20T09:19:56Z

@hmellor Please help take a look. Thanks.

hmellor · 2026-07-20T13:25:19Z

@@ -555,6 +557,7 @@ def get_hf_processor(self, **kwargs: object):
            version=self.get_model_version(),
        )
        hf_processor = vendored_processor
+        """


Why is this here?

Suggested change

intermediate diff applied by mistaken. Updated and addressed your comments. Thanks.

hmellor · 2026-07-20T13:26:11Z

+        try:
+            module = importlib.import_module(module_name)
+        except ModuleNotFoundError as e:
+            # Let getattr(processors, name, None) gracefully fall back.
+            raise AttributeError(
+                f"module 'processors' has no attribute '{name}'"
+            ) from e


Why? This should be unreachable because of the if name in _CLASS_TO_MODULE check

hmellor · 2026-07-20T13:28:53Z

+        # placeholder text is tokenized differently inside chat templates.
+        # Keep dummy startup prompts aligned with the real online prompt shape
+        # so encoder-budget initialization exercises the same path.
+        if self.info.get_model_version() in {(4, 0), (4, 5), (4, 6)}:


This file does not define 4.6

yma11 · 2026-07-22T15:06:00Z

@hmellor can you review again? Thanks.

mergify · 2026-07-28T01:25:31Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yma11.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

hmellor · 2026-07-29T09:47:19Z

@@ -66,8 +64,6 @@
    "KimiAudioProcessor": "vllm.transformers_utils.processors.kimi_audio",
    "KimiK25Processor": "vllm.transformers_utils.processors.kimi_k25",
    "MiMoOmniProcessor": "vllm.transformers_utils.processors.mimo_v2_omni",
-    "MiniCPMOProcessor": "vllm.transformers_utils.processors.minicpmo",


Shouldn't you delete the processor for minicpmo too?

hmellor

We could also delete the comment:

        # MiniCPM-V 4.6 keeps the native transformers MiniCPMV4_6Processor:
        # this model has its own image/video handling and prompt-update logic
        # below, so it does not need (and is incompatible with) the vendored
        # MiniCPMVProcessor used by 2.x/4.0/4.5, whose __init__ assumes a
        # legacy `image_processor.version` attribute that 4.6 no longer has.

from MiniCPMV4_6ProcessingInfo.

Signed-off-by: Yan Ma <yan.ma@intel.com>

yma11 · 2026-08-06T12:59:47Z

Close as PR #48413 fixed this issue.

claude Bot reviewed Jul 20, 2026

View reviewed changes

mergify Bot added the bug Something isn't working label Jul 20, 2026

yma11 marked this pull request as draft July 20, 2026 08:04

yma11 force-pushed the minicpm-v branch from 3482737 to 7e6eb6b Compare July 20, 2026 08:36

yma11 marked this pull request as ready for review July 20, 2026 08:36

hmellor reviewed Jul 20, 2026

View reviewed changes

yma11 force-pushed the minicpm-v branch from 9caf1bf to eefd8b5 Compare July 21, 2026 02:24

mergify Bot added the needs-rebase label Jul 28, 2026

yma11 force-pushed the minicpm-v branch from eefd8b5 to b8978fa Compare July 28, 2026 07:41

mergify Bot removed the needs-rebase label Jul 28, 2026

hmellor reviewed Jul 29, 2026

View reviewed changes

yma11 force-pushed the minicpm-v branch from b8978fa to 4f75792 Compare August 5, 2026 01:49

yma11 added 3 commits August 6, 2026 02:45

[bugfix] fix minicpmv mm prompt placeholder parse error

27cfde2

Signed-off-by: Yan Ma <yan.ma@intel.com>

address comments

8ab5f21

Signed-off-by: Yan Ma <yan.ma@intel.com>

update

2753eb2

Signed-off-by: Yan Ma <yan.ma@intel.com>

yma11 force-pushed the minicpm-v branch from 4f75792 to 2753eb2 Compare August 6, 2026 02:45

DarkLight1337 closed this Aug 6, 2026

Navigation Menu

Uh oh!

[bugfix] fix minicpmv mm prompt placeholder parse error - #49162

[bugfix] fix minicpmv mm prompt placeholder parse error#49162
yma11 wants to merge 3 commits into
vllm-project:mainfrom
yma11:minicpm-v

yma11 commented Jul 20, 2026 •

edited by github-actions Bot

Loading

Uh oh!

claude Bot left a comment

Uh oh!

yma11 commented Jul 20, 2026

Uh oh!

hmellor Jul 20, 2026

Uh oh!

yma11 Jul 21, 2026

Uh oh!

hmellor Jul 20, 2026

Uh oh!

hmellor Jul 20, 2026

Uh oh!

yma11 commented Jul 22, 2026

Uh oh!

mergify Bot commented Jul 28, 2026

Uh oh!

hmellor Jul 29, 2026

Uh oh!

hmellor left a comment

Uh oh!

yma11 commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

yma11 commented Jul 20, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

yma11 commented Jul 20, 2026

Uh oh!

hmellor Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

yma11 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

hmellor Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

hmellor Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

yma11 commented Jul 22, 2026

Uh oh!

mergify Bot commented Jul 28, 2026

Uh oh!

hmellor Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

yma11 commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yma11 commented Jul 20, 2026 •

edited by github-actions Bot

Loading