[Bugfix][Multi Modal] Fix incorrect Molmo token processing

[sangho-vision]

Purpose

When serving a Molmo model online using chat completion, the vLLM code first applies Molmo’s chat template to the input text and tokenizes it. It then calls the custom _apply_hf_processor_tokens_only method in MolmoMultiModalProcessor for further processing.
However, _apply_hf_processor_tokens_only internally calls the get_tokens_input of the Molmo's HF processor, which applies the chat template once again, resulting in double templating.

This behavior can be verified by running the following code:

from vllm import LLM
from vllm.sampling_params import SamplingParams
from transformers import AutoProcessor

model = LLM(
    model="allenai/Molmo-7B-D-0924",
    tensor_parallel_size=torch.cuda.device_count(),
    trust_remote_code=True,
    dtype='bfloat16',
    gpu_memory_utilization=0.95,
)

processor = AutoProcessor.from_pretrained(
    "allenai/Molmo-7B-D-0924",
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
)

sampling_params = SamplingParams(max_tokens=448, temperature=0)

image_url = "https://www.visitscotland.com/binaries/content/gallery/visitscotland/cms-images/2022/06/24/clashnessie-bay-car-road"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Describe the image."
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": image_url
                }
            },
        ],
    },
]

outputs = model.chat(messages, sampling_params=sampling_params)
prompt = processor.tokenizer.decode(outputs[0].prompt_token_ids, skip_special_tokens=True)
print(prompt)

This prints:

 User: User: Describe the image. Assistant: Assistant:

The double "User:" and "Assistant:" indicate that the chat template is applied twice.

This PR fixes this double templating issue.

Changes Made

Make sure that _apply_hf_processor_tokens_only uses the "none" message format instead of the model's default configuration:

        # The chat template is already applied to the prompt tokens
        # Use message_format="none" to avoid applying it again
        # Prepend an empty space if `always_start_with_space` is True
        tokens = processor.processor.get_tokens_input(  # type: ignore
            self.info.get_tokenizer().decode(prompt_tokens),
            message_format="none",
            always_start_with_space=processor.always_start_with_space,
        )

Test Plan

Run the same code snippet above.

Test Result

 User: Describe the image. Assistant:

The double templating is resolved.

[gemini-code-assist]

Code Review

This pull request correctly addresses a bug where the Molmo chat template was being applied twice, leading to incorrect model inputs. The fix is simple and effective. My review includes a suggestion to optimize the token processing logic to avoid an inefficient decode-then-encode cycle, which can improve performance for long prompts.

[gemini-code-assist]

While this correctly fixes the double-templating issue, calling get_tokens_input here incurs a decode-then-encode cycle for the entire prompt on every request. This can be inefficient for long prompts.

Since the main purpose of this call (with message_format="none") is to enforce the always_start_with_space logic, we can optimize this by inlining a more efficient version of that logic. The suggestion below avoids re-encoding the prompt if it already starts with a space, which can provide a significant performance improvement.

Suggested change

	# The chat template is already applied to the prompt tokens
	# Use message_format="none" to avoid applying it again
	# Prepend an empty space if `always_start_with_space` is True
	tokens = processor.processor.get_tokens_input( # type: ignore
	self.info.get_tokenizer().decode(prompt_tokens),
	message_format=processor.message_format,
	message_format="none",
	always_start_with_space=processor.always_start_with_space,
	)
	tokenizer = self.info.get_tokenizer()
	# The chat template is already applied. The logic below is an
	# optimized reimplementation of `processor.get_tokens_input`
	# with `message_format="none"`. It avoids a decode-encode cycle
	# if the prompt already starts with a space, improving performance.
	if processor.always_start_with_space:
	decoded_prompt = tokenizer.decode(prompt_tokens)
	if not decoded_prompt.startswith(" "):
	tokens = tokenizer.encode(" " + decoded_prompt,
	add_special_tokens=False)
	else:
	tokens = prompt_tokens
	else:
	tokens = prompt_tokens

[DarkLight1337]

Thanks for fixing