[qthequartermasterman]

Overview

Fix chat requests that mix prompt embeddings and images when the prompt-embedding placeholder ID is outside the model vocabulary. Consume these temporary IDs during preprocessing instead of passing them to token validation and embedding lookup.

Discovered while helping validate #58658 and #63190, but this is a distinct issue: the fix consumes temporary placeholder IDs in the renderer, rather than changing tower/connector LoRA or prompt-embedding passthrough handling in the runner.

Claims

• Replace only prompt-embedding placeholder spans with safe dummy token IDs after locating all spans.
• Preserve surrounding text IDs, supplied embedding tensors, and embedding offsets.
• Fix the request rejection and avoid the TP=1 out-of-bounds embedding lookup, without tokenizer-cache, model-runner, or model-specific changes.

Validation

HF_HUB_OFFLINE=1 .venv/bin/python -m pytest \
  tests/renderers/test_hf.py::test_apply_prompt_embeds_keeps_engine_token_ids_in_vocabulary -q

PATH="$PWD/.venv/bin:$PATH" CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 \
  .venv/bin/python -m pytest \
  tests/entrypoints/openai/chat_completion/test_chat_completion_with_prompt_embeds.py::test_prompt_embeds_with_image -v

Validation	Before	After
Renderer regression	Fails the vocabulary-bound assertion without the two-line fix	Passes; all IDs are in vocabulary, text IDs are unchanged, and both embedding spans retain their offsets
SmolVLM-256M serving, TP=1	Mixed image/embedding request returns HTTP 400	Generates one token successfully; invalid ordinary completion token IDs still return HTTP 400
Gemma 4, TP=1 and TP>2	Original HTTP 400 reproduced on this model	Submitter confirmed the local reproducer succeeds at TP=1 and TP>2
Nemotron 3 Nano Omni	Original request returns HTTP 400	Submitter confirmed the local reproducer succeeds

The serving regression was also run with only the two-line normalization change present, with no tokenizer-cache changes: 1 passed. The relocated renderer regression passed: 1 passed. All commit hooks passed, including Ruff, mypy, Markdown lint, and test-tethering checks.

Model validation is limited to the generation smoke test and the submitter's Gemma and Nemotron reproducers; no accuracy or performance benchmark was run.

The original failure was reproduced locally with:

• nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
• Google/gemma-4-E2B-it
• HuggingFaceTB/SmolVLM-256M-Instruct

The original failure was not observable with Qwen/Qwen3.5-0.8B, Qwen/Qwen3.5-2B, LiquidAI/LFM2.5-VL-450M, or llava-hf/llava-onevision-qwen2-0.5b-ov-hf. Qwen3.5 and LFM2.5 have model-vocabulary headroom for the newly registered sentinel, so the stale cached vocabulary bound does not reject it. These are locally checked examples, not an exhaustive affected-model list.

Thanks to @Gatedd and @ajf-codes for helping me identify the issue!

Manual reproducer: mixed prompt embeddings and image input

Start a server with --enable-prompt-embeds, using the same model as MODEL_NAME below. TP=1 also exposes the unsafe embedding lookup if the validation rejection is bypassed without consuming the sentinel.

vllm serve Google/gemma-4-E2B-it \
  --enable-prompt-embeds \
  --tensor-parallel-size 1

Save the following script locally and run it with .venv/bin/python mre.py. It downloads the tokenizer and the checkpoint shard containing the token embedding weights, constructs real prompt embeddings, and sends them alongside an image. The image URL requires network access. Set MODEL_NAME to another listed model and restart the server to compare models.

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Send multimodal + prompt embeds to a local vLLM server and print the response."""

import json
import pprint

import httpx
import torch
import transformers
from huggingface_hub import hf_hub_download
from huggingface_hub.errors import RemoteEntryNotFoundError
from safetensors import safe_open

from vllm.utils.serial_utils import tensor2base64

# MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
MODEL_NAME = "Google/gemma-4-E2B-it"
# MODEL_NAME = "Qwen/Qwen3.5-0.8B"
# MODEL_NAME = "Qwen/Qwen3.5-2B"
# MODEL_NAME = "deepseek-ai/DeepSeek-OCR-2"
# MODEL_NAME = "LiquidAI/LFM2.5-VL-450M"
# MODEL_NAME = "HuggingFaceTB/SmolVLM-256M-Instruct"
# MODEL_NAME = "llava-hf/llava-onevision-qwen2-0.5b-ov-hf"
SERVER_URL = "http://localhost:8000/v1/chat/completions"


key_name_lookup = {
    "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16": "language_model.backbone.embeddings.weight",
    "Qwen/Qwen3.5-0.8B": "model.language_model.embed_tokens.weight",
    "Qwen/Qwen3.5-2B": "model.language_model.embed_tokens.weight",
    "deepseek-ai/DeepSeek-OCR-2": "model.embed_tokens.weight",
    "Google/gemma-4-E2B-it": "model.language_model.embed_tokens.weight",
    "LiquidAI/LFM2.5-VL-450M": "model.language_model.embed_tokens.weight",
    "HuggingFaceTB/SmolVLM-256M-Instruct": "model.text_model.embed_tokens.weight",
    "llava-hf/llava-onevision-qwen2-0.5b-ov-hf": "language_model.model.embed_tokens.weight",
}


def load_embedding_layer(
    model_name: str,
    key: str | None = None,
) -> torch.nn.Embedding:
    if key is None:
        key = key_name_lookup[model_name]

    try:
        index_path = hf_hub_download(model_name, "model.safetensors.index.json")
    except RemoteEntryNotFoundError:
        shard_name = "model.safetensors"
    else:
        with open(index_path) as f:
            index = json.load(f)
        shard_name = index["weight_map"][key]

    shard = hf_hub_download(model_name, shard_name)
    with safe_open(shard, framework="pt") as f:
        weight = f.get_tensor(key)
    return torch.nn.Embedding.from_pretrained(weight)


def encode(
    tokenizer: transformers.PreTrainedTokenizerBase, embedding_layer, text: str
) -> str:
    token_ids = tokenizer(text, return_tensors="pt", add_special_tokens=False).input_ids
    prompt_embeds = embedding_layer(token_ids).squeeze(0)
    return tensor2base64(prompt_embeds)


def main() -> None:
    tokenizer = transformers.AutoTokenizer.from_pretrained(
        MODEL_NAME, trust_remote_code=True
    )
    embedding_layer = load_embedding_layer(MODEL_NAME)

    messages = [
        {
            "role": "user",
            "content": "This part is text. The part that follows will be prompt embeddings. Please tell me the color they say.",
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "prompt_embeds",
                    "data": encode(
                        tokenizer,
                        embedding_layer,
                        "This part is prompt embeddings. The color is purple. End prompt embeds.",
                    ),
                },
            ],
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://fastly.picsum.photos/id/373/200/300.jpg?hmac=GXSHLvl-WsHouC5yVXzXVLNnpn21lCdp5rjUE_wyK-8"
                    },
                },
            ],
        },
        {
            "role": "user",
            "content": "Please describe the image to me. Then tell me my favorite color.",
        },
    ]

    request = {
        "model": MODEL_NAME,
        "max_tokens": 1024,
        "temperature": 0.0,
        "messages": messages,
    }

    r = httpx.post(SERVER_URL, json=request, timeout=600)

    print("-" * 30)
    if r.status_code == 200:
        response_json = r.json()
        print("-" * 30)
        pprint.pprint(response_json)
        print("-" * 30)
        print(response_json["choices"][0]["message"]["content"])
    else:
        print(r.status_code, r.text)
    print("-" * 30)


if __name__ == "__main__":
    main()

Details

The mixed-input path already records the positions and tensors needed for embedding splicing, but leaves the temporary sentinel IDs in prompt_token_ids. A newly registered sentinel can exceed the cached tokenizer's vocabulary bound, causing HTTP 400. If validation accepts the sentinel, an out-of-range model embedding lookup can instead crash before the supplied embedding is merged. TP>1 shard masking can hide that latter failure.

After locating all spans, replace their token IDs with 0 and write the updated list back to the engine input. The existing multimodal merge overwrites those dummy embeddings with the supplied tensors. Ordinary token IDs are not changed, so their validation remains intact.

The production change is two added lines. Tests cover two embedding spans of different lengths and real TP=1 image/embedding serving; documentation describes mixed image and prompt-embedding inputs.

Duplicate-work checks found no matching open upstream PR. The related #51123 addresses LoRA batch-shape handling in the runner, not mixed-input sentinel consumption in the renderer.

AI assistance was used to investigate, implement, and test this change.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--64367.org.readthedocs.build/en/64367/

[Copilot]

🟢 Approval recommended

No active review issues remain for the changes supporting mixed prompt-embedding and image requests.

No open issues 🎉

What changed in this PR

This PR fixes mixed prompt-embedding and image requests by replacing temporary embedding sentinel IDs in the Hugging Face renderer before token validation and model execution.

Changes:

• Normalize only prompt-embedding spans to safe dummy token IDs, preserving surrounding text, embedding tensors, and offsets.
• Add renderer and serving regressions covering vocabulary bounds, multiple embedding spans, and rejection of invalid ordinary token IDs.
• Document support for mixing prompt embeddings with images.

File	Description
vllm/​renderers/​hf.py	Replaces embedding sentinel IDs with zeros in engine input.
tests/​renderers/​test_hf.py	Checks safe token IDs, unchanged text, and preserved embedding offsets.
tests/​entrypoints/​openai/​chat_completion/​test_chat_completion_with_prompt_embeds.py	Tests mixed image/embedding serving and invalid-token rejection.
docs/​features/​prompt_embeds.md	Explains mixed image inputs and dummy-token replacement.

_{Review effort: Balanced}

[qthequartermasterman]

@DarkLight1337 This is a simple (2 line) patch in the HF Renderer to better handle the added prompt embedding placeholder token id in a few places (when validating token ids and also when sending those token ids to the parallel Vocab Embedding layer). I added several tests because this is kinda a weird code path. A bunch of independent conditions can each mask this issue (TP, sending only prompt embeds and/or text, a MM model that has extra token ids reserved in the model definition that aren't in the tokenizer, etc...), so I wanted to have very precise regression tests to make sure that this was exercised.

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #98204 for commit d1a66bff8a79.

ℹ️ This PR is 14 commits behind upstream main. Merged upstream main into this branch as d1a66bff8a79 before starting CI. Pull the branch before pushing more changes.