[Bugfix][Multi Modal] Fix incorrect Molmo image processing

[sangho-vision]

Purpose

Restore correct Molmo outputs by re-introducing image patch re-ordering using image_input_idx.
In PR #12966, the re-ordering that maps image features to their corresponding patch tokens was removed and replaced with a boolean mask (feat_is_patch). That mask only indicates whether each image feature is used as a patch token in the multimodal input sequence, not where each valid patch feature should be placed.

In addition, the method get_num_image_tokens in MolmoProcessingInfo did not account for image start/end tokens as well as column separator tokens.

This PR fixes that regression by:

• Restoring correct patch-token placement using image_input_idx
• Correcting token-count computation to include start/end and column tokens.

FIX #26518

Note
PR #26451 also addresses the patch re-ordering issue, but it does not include the correction for get_num_image_tokens (which ensures image start/end and column tokens are properly counted).
This PR provides a more complete fix covering both aspects.

Background

• Issue [Bug]: Molmo produces incorrect outputs #26451 reported incorrect pointing results in Molmo.
• As discussed in [Bugfix][Multi Modal] Fix incorrect output in Molmo #26518, removing image_input_idx broke the alignment between image features and patch-token positions.
• The boolean array feat_is_patch lacks the per-feature index mapping needed to reconstruct the original patch order.

Changes made

1. Replace feat_is_patch field in MolmoImageInputs TensorSchema with image_input_idx
2. Update multimodal field configuration to include image_input_idx in the processing pipeline
3. Re-order patches back to their correct spatial positions in the final sequence using image_input_idx
4. Fix get_num_image_tokens to include start/end and column separator tokens in the total count

Test Plan

Used the same test script from #26518:

from vllm import LLM
from vllm.sampling_params import SamplingParams
import requests
from PIL import Image

model = LLM(
    model="allenai/Molmo-7B-D-0924",
    trust_remote_code=True,
    dtype='bfloat16',
    gpu_memory_utilization=0.95,
)

sampling_params = SamplingParams(max_tokens=448, temperature=0)

image_url = "https://www.visitscotland.com/binaries/content/gallery/visitscotland/cms-images/2022/06/24/clashnessie-bay-car-road"
image = Image.open(requests.get(image_url, stream=True).raw)

inputs = [{
    "prompt": "Point to the car.",
    "multi_modal_data": {"image": image},
}]

outputs = model.generate(inputs, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

Test Result

 <point x="69.0" y="48.6" alt="car">car</point>

The coordinates correctly align with the car in the image.

Related

• See discussion in [Bugfix][Multi Modal] Fix incorrect output in Molmo #26518
• Concurrent bug fix: [Bug]: Molmo produces incorrect outputs #26451
• Reverts the unintended effect introduced by [VLM] Merged multi-modal processor for Molmo #12966

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request provides a crucial bugfix for Molmo image processing. It correctly restores the patch re-ordering mechanism by re-introducing image_input_idx, which was a regression from a previous change. Additionally, it fixes an issue in get_num_image_tokens to accurately account for special tokens, ensuring correct token count estimation. The changes are well-implemented, logical, and directly address the reported bugs. The code is clear and the fix appears to be complete and correct. I have no further comments.

[DarkLight1337]

Thanks, can you fix pre-commit?

[sangho-vision]

I fixed the pre-commit.

[DarkLight1337]

Thanks, LGTM