[Renderer] Move InputPreprocessor into Renderer (1/2)

[DarkLight1337]

Purpose

Towards #22880

• Define default TokenizeParams in the Renderer, and let each MM model define its own defaults via BaseProcessingInfo.get_default_tok_params. This allows tokenization for MM models to be handled by Renderer instead of inside MM processor, even for offline APIs. It should also solve the minor accuracy discrepency between offline and online APIs because of different handling of add_special_tokens.
• Update InputPreprocessor._tokenize_prompt and InputProcessor._truncate_inputs to use Renderer.
• Moved contains_only_strings check from InputPreprocessor to InputProcessor.

Misc changes:

• Move StreamingInput from inputs file to engine protocol to avoid circular import.
• Move get_timing_stats_from_engine_client to benchmarks since it's only used there.
• Renamed InputProcessor.close to InputProcessor.shutdown.
• Add convenience method MultiModalCacheStats.record.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request is a significant refactoring that moves the logic from InputPreprocessor into the Renderer class. This centralization of input processing logic is a good architectural improvement. The changes include defining default tokenization parameters in the Renderer and allowing multimodal models to provide their own defaults, which helps resolve inconsistencies. The movement of StreamingInput to avoid circular imports and the relocation of benchmark-specific utilities are also positive changes.

I've found one issue where a method is called redundantly, which could impact performance. Please see my specific comment for details.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a significant refactoring that moves the responsibility of tokenization parameter management and multi-modal processing from various components into the Renderer. This centralization simplifies the LLM and InputProcessor classes and improves the overall code structure. The changes are well-executed and consistent with the goal of the PR. I've identified a few places where a None check for the multi-modal processor is missing, which could lead to a crash if multimodal data is passed to a non-multimodal model. Please see my detailed comments.