[Renderer] Move InputPreprocessor into Renderer (2/2)

[DarkLight1337]

Purpose

Towards #22880

• Transform embedding and multi-modal inputs in Renderer, removing the need for Input Preprocessor.
  • Input Preprocessor is retained for back-compatibility, but internally it just calls Renderer.
• Update all APIs (except Score API) to pass ProcessorInputs to the LLM engine instead of raw prompts, using Renderer to process the raw prompts into ProcessorInputs where applicable.
• Simplify beam search implementation to avoid duplicate MM processing.
• Fix additional audio BOS/EOS tokens being added for Qwen2-Audio (discovered when checking beam search results).

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request continues the significant refactoring of moving the input preprocessing logic from the engine into the renderer layer. The changes are extensive and affect many files, but they appear consistent and well-aligned with the goal of centralizing input processing. This refactoring improves the separation of concerns and should make the engine core cleaner.

I've identified one potential high-severity issue in vllm/beam_search.py where the logic to differentiate between token-only and multimodal prompts could be brittle. I've provided a suggestion to make it more robust by checking the prompt's type field directly.

Overall, this is a good step forward in improving the architecture of the input processing pipeline.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request continues the significant refactoring of moving input preprocessing logic from the InputPreprocessor into the Renderer. The changes are extensive, touching many files across the entrypoints and engine components to adapt to the new ProcessorInputs data structure and the updated rendering flow. The InputPreprocessor is now a thin wrapper for backward compatibility, and the core logic for tokenization, multimodal processing, and preparing engine inputs resides within the BaseRenderer. This is a positive change for code organization and clarity.
I've found one critical issue in the handling of text-only encoder-decoder models which I've commented on. Other than that, the refactoring appears solid and consistent with the stated goals.

[mergify]

Hi @DarkLight1337, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @DarkLight1337, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @DarkLight1337, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint