Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 22 additions & 11 deletions docs/contributing/model/multimodal.md
Original file line number Diff line number Diff line change
Expand Up @@ -454,17 +454,17 @@ return a schema of the tensors outputted by the HF processor that are related to
??? code

```python
def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
# Mistral3Processor requires text corresponding to the images
return self.dummy_inputs.get_dummy_text(mm_counts)

def _postprocess_hf_mm_data(
self,
mm_data: Mapping[str, object],
hf_processor_mm_kwargs: Mapping[str, object],
hf_data: Mapping[str, object],
hf_kwargs: Mapping[str, object],
processed_data: BatchFeature,
) -> BatchFeature:
if not mm_data:
if not hf_data:
return processed_data

pixel_values = processed_data.get("pixel_values")
Expand All @@ -486,25 +486,36 @@ return a schema of the tensors outputted by the HF processor that are related to
calls the HF processor on the multi-modal data without passing any text.
If the HF processor instead requires text corresponding to the multi-modal items,
you should override
[_get_hf_processor_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_processor_text]
[_get_hf_mm_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_text]
to return the dummy text from
[BaseDummyInputsBuilder.get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text]
like in the example above. If the HF processor expects the multi-modal data
under different keys than those provided by the multi-modal items
(e.g. `audio` instead of `audios`), or requires additional keyword arguments
under different keys than those provided by the multi-modal items,
or requires additional keyword arguments
(e.g. `sampling_rate`), you should override
[_preprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._preprocess_hf_mm_data].
[_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs].
If you need to modify the output of the HF processor,
you should override
[_postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data].
Passthrough fields have already been merged into `processed_data` when this
hook is called.
If you need to change how the normalized inputs are passed to the HF
processor (for example, to call a method other than `__call__`, or to use
different kwargs to construct and invoke it), override
[_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor].
If the HF processor applies additional transformations to the prompt tokens
that are not reflected in its multi-modal outputs (e.g. prepending a BOS token),
you should override
[_postprocess_prompt][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_prompt]
to replicate them.
For even more control over how the HF processor is called, you can override
Only processors that require multiple HF calls or manually construct the
complete output should override
[_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main]
directly.
directly. Such implementations must obtain their normalized inputs through
[_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs]
and return through
[_finalize_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._finalize_hf_mm_data]
so passthrough fields and model-specific postprocessing are applied consistently.

Since `pixel_values` is now a list with one tensor per image, we can override
[_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] as follows:
Expand Down Expand Up @@ -725,7 +736,7 @@ Examples:

### Custom HF processor

Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and return it from [BaseProcessingInfo.get_hf_processor][vllm.multimodal.processing.BaseProcessingInfo.get_hf_processor]. It is then applied to the multi-modal data via [InputProcessingContext.call_hf_processor][vllm.multimodal.processing.InputProcessingContext.call_hf_processor] inside [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main].
Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and return it from [BaseProcessingInfo.get_hf_processor][vllm.multimodal.processing.BaseProcessingInfo.get_hf_processor]. The default [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor] applies it to the normalized multi-modal data via [InputProcessingContext.call_hf_processor][vllm.multimodal.processing.InputProcessingContext.call_hf_processor]. Override `_call_hf_processor` if the custom processor instead exposes a different callable interface.

Examples:

Expand Down
11 changes: 9 additions & 2 deletions docs/design/mm_processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,16 @@ In vLLM's rendering pipeline (see [BaseRenderer][vllm.renderers.base.BaseRendere

Since Transformers 5.10, `ProcessorMixin` now allows multi-modal inputs to be passed by themselves. However, certain subclasses (such as `ChameleonProcessor`) and older out-of-tree implementations may still define their own `__call__` method that assumes the presence of text with corresponding placeholder tokens. This causes a problem as we don't have the original text anymore to pass to these processors.

To work around this, each model defines how to generate dummy text based on the number of multi-modal inputs, via [get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text], which its override of [_get_hf_processor_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_processor_text] returns so that [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main] passes it to the HF processor together with the multi-modal inputs to obtain the processed multi-modal data.
To work around this, each model defines how to generate dummy text based on the number of multi-modal inputs, via [get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text], which its override of [_get_hf_mm_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_text] returns. [_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs] adds this text and `truncation=False` to the normalized HF inputs. The truncation setting is attached only to calls that use dummy text, so unrelated processor calls do not receive a text-specific option.

Similarly, since the multi-modal data extracted by vLLM may not match what a specific HF processor expects, [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main] allows each model to adapt the inputs via [_preprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._preprocess_hf_mm_data] (e.g. renaming keys like `audios` to `audio` or injecting extra keyword arguments such as `sampling_rate`) and the outputs via [_postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data], without having to reimplement the entire method.
HF processing is split into four overridable stages:

1. [_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs] extracts and normalizes processor data, kwargs, and passthrough data. It centrally renames vLLM's `audios` field to the standard HF `audio` keyword; processors that require another spelling can adapt it in an override.
2. [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor] invokes the HF processor. Models can override it when they use a different callable or need separate construction and invocation kwargs.
3. [_finalize_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._finalize_hf_mm_data] merges passthrough fields into the processed output. It lazily creates an empty output when there is no processor data.
4. [_postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data] performs model-specific output transformations after passthrough fields have been merged.

This division lets most models customize one stage without reimplementing [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main]. Implementations that genuinely require multiple processor calls or manually construct their complete output may still override that method, but must use `_get_hf_mm_inputs` and `_finalize_hf_mm_data` to retain the common input and output behavior.

## Prompt Update Detection

Expand Down
14 changes: 1 addition & 13 deletions vllm/model_executor/models/audioflamingo3.py
Original file line number Diff line number Diff line change
Expand Up @@ -377,21 +377,9 @@ def _parse_audio_data(
class AudioFlamingo3MultiModalProcessor(
BaseMultiModalProcessor[AudioFlamingo3ProcessingInfo]
):
def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
return self.dummy_inputs.get_dummy_text(mm_counts)

def _preprocess_hf_mm_data(
self,
mm_data: Mapping[str, object],
hf_processor_mm_kwargs: Mapping[str, object],
) -> tuple[Mapping[str, object], Mapping[str, object]]:
mm_data = dict(mm_data)
audios = mm_data.pop("audios", None)
if audios is not None:
mm_data["audio"] = audios

return mm_data, hf_processor_mm_kwargs

def _postprocess_hf_mm_data(
self,
mm_data: Mapping[str, object],
Expand Down
2 changes: 1 addition & 1 deletion vllm/model_executor/models/chameleon.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ def get_dummy_mm_data(


class ChameleonMultiModalProcessor(BaseMultiModalProcessor[ChameleonProcessingInfo]):
def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
# ChameleonProcessor requires text corresponding to the images
return self.dummy_inputs.get_dummy_text(mm_counts)

Expand Down
2 changes: 1 addition & 1 deletion vllm/model_executor/models/cohere2_vision.py
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ def get_dummy_mm_data(
class Cohere2VisionMultiModalProcessor(
BaseMultiModalProcessor[Cohere2VisionProcessingInfo]
):
def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
return self.dummy_inputs.get_dummy_text(mm_counts)

def _postprocess_hf_mm_data(
Expand Down
48 changes: 19 additions & 29 deletions vllm/model_executor/models/cohere_asr.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import torch
import torch.nn.functional as F
from torch import nn
from transformers import BatchFeature, PretrainedConfig
from transformers import PretrainedConfig

from vllm.compilation.decorators import support_torch_compile
from vllm.config import CacheConfig, ModelConfig, SpeechToTextConfig, VllmConfig
Expand Down Expand Up @@ -50,7 +50,10 @@
PromptUpdate,
TimingContext,
)
from vllm.multimodal.processing.processor import MultiModalProcessingInfo
from vllm.multimodal.processing.processor import (
HFMultiModalInputs,
MultiModalProcessingResult,
)
from vllm.renderers import TokenizeParams
from vllm.tokenizers import cached_tokenizer_from_config
from vllm.transformers_utils.processors.cohere_asr import (
Expand Down Expand Up @@ -1944,42 +1947,29 @@ def create_encoder_prompt(
) -> list[int]:
return [0]

def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
return self.dummy_inputs.get_dummy_text(mm_counts)

def _preprocess_hf_mm_data(
def _get_hf_mm_inputs(
self,
mm_data: Mapping[str, object],
hf_processor_mm_kwargs: Mapping[str, object],
) -> tuple[Mapping[str, object], Mapping[str, object]]:
feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)

mm_data = dict(mm_data)
mm_data["audio"] = mm_data.pop("audios")

hf_processor_mm_kwargs = dict(
**hf_processor_mm_kwargs,
sampling_rate=feature_extractor.sampling_rate,
mm_items: MultiModalDataItems,
hf_kwargs: Mapping[str, object],
) -> HFMultiModalInputs:
hf_inputs = super()._get_hf_mm_inputs(mm_items, hf_kwargs)

feature_extractor = self.info.get_feature_extractor(**hf_kwargs)
return hf_inputs._replace(
hf_kwargs=dict(
hf_inputs.hf_kwargs,
sampling_rate=feature_extractor.sampling_rate,
)
)

return mm_data, hf_processor_mm_kwargs

def _postprocess_hf_mm_data(
self,
mm_data: Mapping[str, object],
hf_processor_mm_kwargs: Mapping[str, object],
processed_data: BatchFeature,
) -> BatchFeature:
if "labels" in processed_data:
processed_data["input_ids"] = processed_data.pop("labels")

return processed_data

def _cached_apply_hf_processor(
self,
inputs: ProcessorInputs,
timing_ctx: TimingContext,
) -> MultiModalProcessingInfo:
) -> MultiModalProcessingResult:
# Dithering injects noise into the extracted features, so the
# feature extractor is not a pure function of its input. Since the
# processing cache assumes that processor outputs are invariant
Expand Down
14 changes: 7 additions & 7 deletions vllm/model_executor/models/colmodernvbert.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,15 +157,14 @@ class ColModernVBertMultiModalProcessor(
def _apply_hf_processor_main(
self,
mm_items: MultiModalDataItems,
hf_processor_mm_kwargs: Mapping[str, object],
hf_kwargs: Mapping[str, object],
) -> BatchFeature:
valid_mm_items = mm_items.select(
{k for k, c in mm_items.get_all_counts().items() if c > 0}
mm_data, hf_kwargs, passthrough_data = self._get_hf_mm_inputs(
mm_items, hf_kwargs
)
mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)

if not mm_data:
return BatchFeature(dict(passthrough_data))
return self._finalize_hf_mm_data(mm_data, hf_kwargs, passthrough_data)

prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())

Expand Down Expand Up @@ -193,8 +192,9 @@ def _apply_hf_processor_main(
result.update(image_outputs)

processed_data = result
processed_data.update(passthrough_data)
return processed_data
return self._finalize_hf_mm_data(
mm_data, hf_kwargs, passthrough_data, processed_data
)

def _get_mm_fields_config(
self,
Expand Down
25 changes: 7 additions & 18 deletions vllm/model_executor/models/colpali.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,15 @@

import torch
import torch.nn as nn
from transformers import BatchFeature, PaliGemmaProcessor
from transformers import PaliGemmaProcessor

from vllm.config import VllmConfig
from vllm.model_executor.layers.pooler.tokwise import pooler_for_token_embed
from vllm.model_executor.layers.pooler.tokwise.heads import TokenPoolerHead
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
from vllm.multimodal import MULTIMODAL_REGISTRY
from vllm.multimodal.parse import MultiModalDataItems
from vllm.multimodal.processing.processor import HFMultiModalInputs

from .interfaces import SupportsLateInteraction
from .interfaces_base import default_pooling_type
Expand Down Expand Up @@ -60,31 +61,19 @@ def get_hf_processor(self, **kwargs: object) -> PaliGemmaProcessor:
class ColPaliMultiModalProcessor(PaliGemmaMultiModalProcessor):
"""Multimodal processor for ColPali."""

def _apply_hf_processor_main(
def _get_hf_mm_inputs(
self,
mm_items: MultiModalDataItems,
hf_processor_mm_kwargs: Mapping[str, object],
) -> BatchFeature:
valid_mm_items = mm_items.select(
{k for k, c in mm_items.get_all_counts().items() if c > 0}
)
mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)

if not mm_data:
return BatchFeature(dict(passthrough_data))
hf_kwargs: Mapping[str, object],
) -> HFMultiModalInputs:
hf_inputs = super()._get_hf_mm_inputs(mm_items, hf_kwargs)

# The ColPali tokenizer_config.json ships with a small default
# max_length (50) that truncates the 1024 image tokens inserted
# by PaliGemmaProcessor, causing a token-count mismatch.
# vLLM enforces its own max_model_len, so we disable HF
# truncation to keep all image + text tokens intact.
processed_data = self.info.ctx.call_hf_processor(
self.info.get_hf_processor(**hf_processor_mm_kwargs),
mm_data,
dict(**hf_processor_mm_kwargs, truncation=False),
)
processed_data.update(passthrough_data)
return processed_data
return hf_inputs._replace(hf_kwargs=dict(hf_inputs.hf_kwargs, truncation=False))


@default_pooling_type(seq_pooling_type="CLS", tok_pooling_type="ALL")
Expand Down
31 changes: 2 additions & 29 deletions vllm/model_executor/models/deepseek_ocr.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@
from vllm.sampling_params import SamplingParams
from vllm.sequence import IntermediateTensors
from vllm.tokenizers import cached_tokenizer_from_config
from vllm.tokenizers.hf import HfTokenizer
from vllm.transformers_utils.configs.deepseek_vl2 import DeepseekVLV2Config
from vllm.transformers_utils.processors.deepseek_ocr import (
BASE_SIZE,
Expand Down Expand Up @@ -284,40 +283,14 @@
}


class DeepseekOCRMultiModalProcessor(
BaseMultiModalProcessor[DeepseekOCRProcessingInfo]
):
def _apply_hf_processor_main(
self,
mm_items: MultiModalDataItems,
hf_processor_mm_kwargs: Mapping[str, object],
) -> BatchFeature:
valid_mm_items = mm_items.select(
{k for k, c in mm_items.get_all_counts().items() if c > 0}
)
mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)

prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())

if mm_data:
processed_data = self.info.ctx.call_hf_processor(
self.info.get_hf_processor(**hf_processor_mm_kwargs),
dict(prompt=prompt_text, **mm_data),
hf_processor_mm_kwargs,
)

else:
tokenizer = self.info.get_tokenizer()
assert isinstance(tokenizer, HfTokenizer)
processed_data = tokenizer(
prompt_text, add_special_tokens=True, return_tensors="pt"
)

processed_data.update(passthrough_data)
return processed_data
def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
return self.dummy_inputs.get_dummy_text(mm_counts)

def _get_mm_fields_config(
self,

Check failure on line 293 in vllm/model_executor/models/deepseek_ocr.py

View check run for this annotation

Claude / Claude Code Review

DeepSeek-OCR processor call sends text= instead of required prompt=

DeepseekOCRMultiModalProcessor (and DeepseekOCR2MultiModalProcessor, and UnlimitedOCRMultiModalProcessor which subclasses it) dropped their custom `_apply_hf_processor_main` override and now rely on the base HF-call path, which invokes `hf_processor(text=prompt_text, images=..., **kwargs)`. But `DeepseekOCRProcessor.__call__` requires a keyword-only `prompt` argument and has no `text` parameter, so `text` is silently swallowed by `**kwargs` and the call raises `TypeError: __call__() missing 1 re
Comment thread
DarkLight1337 marked this conversation as resolved.
Resolved
hf_inputs: BatchFeature,
hf_processor_mm_kwargs: Mapping[str, object],
) -> Mapping[str, MultiModalFieldConfig]:
Expand Down
Loading
Loading