[Frontend] Use new Renderer for Completions and Tokenize API

[DarkLight1337]

Purpose

Follow-up to #30200
Towards #22880

• Add render_completions and tokenize_prompt to Renderer API.
  • Async versions are backed by AsyncMicrobatchTokenizer which is internally managed by Renderer.
• Remove legacy CompletionRenderer and related code from serving engine.
• Add _preprocess_completion to serving engine which has the same function as _preprocess_chat.
• Introduce ChatParams and TokenizeParams which are passed to the Renderer to apply the corresponding steps, greatly reducing the number of arguments that need to be passed to _preprocess_completion and _preprocess_chat.
  • Each request definition should implement build_chat_params and/or build_tok_params for the corresponding steps.
  • Replace _validate_truncation_size with TokenizeParams handling.
  • Deprecate passing truncate_prompt_tokens and params.truncate_prompt_tokens to offline APIs; users should pass tokenization_kwargs instead which are internally mapped to TokenizeParams.

After this PR:

• Add wrapper methods over preprocess_chat and _preprocess_completion.
• Use Renderer for IO Processor, Score API and Beam Search API.
• Incorporate InputProcessor into Renderer, including MM part.
• Allow per-prompt tokenization_kwargs by adding it to _CommonKeys and handling it accordingly (similar to mm_processor_kwargs ).

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[DarkLight1337]

cc @noooop @chaunceyjiang what do you think so far of this design?

[gemini-code-assist]

Code Review

The pull request introduces a new rendering architecture by refactoring the prompt processing logic into vllm/renderers/protocol.py and vllm/renderers/params.py. This change centralizes the handling of tokenization, chat template formatting, and multimodal input, leading to a more modular and maintainable codebase. The RendererLike protocol now defines the interface for rendering and tokenizing prompts, and request objects implement build_tok_params and build_chat_params methods to provide necessary configuration. While the overall refactoring is positive, there are a few critical issues related to parameter handling in chat template construction and error handling that need to be addressed.

[DarkLight1337]

It was added fairly recently by #29794, I think we can change the interface later to pass TokenizeParams directly, similar to SamplingParams. Another way would be to let users pass any keyword arguments to the LLM APIs and try to separate them into the individual params (which is how HF does it)