[Bugfix] Correct max tokens for non-contiguous embeds

[milesial]

Fixes a hang on some VLM models with large images.

Some models like Pixtral derivative do not have contiguous multimodal image embeddings, but insert text tokens in between chunks of image row embeddings.

The encoder cache manager did not take this into account, and underestimated the size of the cache to allocate. At inference time, some requests with a large image that exceed the cache size never got scheduled.

FIX #18329

Example

vllm serve mistralai/Pixtral-12B-2409  --no-enable-prefix-caching --tokenizer_mode mistral --config_format mistral --load_format mistral --max_model_len 32000

curl -X 'POST' \
'http://0.0.0.0:8000/v1/chat/completions' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
          "model": "mistralai/Pixtral-12B-2409",
          "messages": [
            {
              "role": "user",
              "content": [
                {
                  "type": "image_url",
                  "image_url":
                    {
                      "url": "https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fsvs.gsfc.nasa.gov%2Fvis%2Fa000000%2Fa003600%2Fa003657%2Fsc09cloud_still_persp_c1440.0500.jpg&f=1&nofb=1&ipt=3aece6fc933b224b28bcab2de0586cabb5e0693be3e04d6b7815d69e7b7b265d"
                    }
                },
                {
                  "type": "text",
                  "text": "What is in this image?"
                }
              ]
            }
          ],
          "stream": false,
          "max_tokens": 256,
          "temperature": 0
    }'

Before this MR:

INFO 07-28 18:58:14 [gpu_model_runner.py:2238] Encoder cache will be initialized with a budget of 4096 tokens, and profiled with 1 image items of the maximum feature size.   

Request stuck forever, never scheduled.

After this MR

INFO 07-28 19:22:40 [gpu_model_runner.py:2238] Encoder cache will be initialized with a budget of 4160 tokens, and profiled with 1 image items of the maximum feature size.   

Request runs as expected

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request addresses a bug where the encoder cache size for multimodal models with non-contiguous embeddings was underestimated, causing some inference requests to hang. The fix correctly calculates the required token space by using the full length of the placeholder (item.length) instead of just the number of embedding tokens (item.get_num_embeds()). This change is correct and effectively resolves the described issue. The change is localized and doesn't seem to have unintended side effects.

[milesial]

+ @DarkLight1337 for review

[DarkLight1337]

cc @ywang96 @Isotr0py

[DarkLight1337]

I feel that we should define a separate method for this definition. For example we can add MultiModalProfiler.get_max_placeholder_tokens which explicitly includes both multi-modal and text inside the placeholder tokens.

[milesial]

Added this new function that uses the existing one with an additional flag to include text tokens or not

[ywang96]

Really appreciate the fix! I left some comments

[ywang96]

LGTM!