[Frontend] Support Responses API input in tokenize endpoint

[SiluPanda]

Fixes #40556

Summary

• add a Responses-style /tokenize request schema
• reuse the existing Responses API rendering path so token IDs match the prompt sent to /v1/responses
• wire the Responses handler into tokenization and keep renderer-only multimodal cache bypass behavior
• tighten the Responses renderer typing and fail loudly if generate-state wiring regresses
• update tokenizer endpoint docs with concrete Responses examples
• add focused unit, route-level, renderer-context, and online parity coverage

Duplicate-work check

• Reviewed issue [Feature]: Support responses api in tokenize endpoint #40556 and its discussion.
• Searched open PRs for 40556, responses tokenize, and tokenize responses api; no other open PR adds Responses API input support to /tokenize.

Tests

• uvx ruff check on all changed Python files — passed
• uvx ruff format --check on all changed Python files — passed
• python3 -m py_compile on all changed Python files — passed
• PYTHONPATH=$PWD .../.venv/bin/python -m pytest -q tests/entrypoints/serve/tokenize/test_serving_tokenization.py tests/entrypoints/openai/responses/test_serving_responses.py — passed (42 passed, 1 xfailed)
• git diff --check — passed

The online tokenizer test module was collected but cannot run end-to-end on this macOS arm64 host because vLLM cannot infer a supported runtime device. The focused CPU-runnable tests above cover schema discrimination, route parsing, handler wiring, Responses rendering, previous-response context, tool forwarding, error propagation, and token strings.

Rebase update

Rebased on current main after the online-serving layout refactor. The implementation now uses OnlineRenderer, the current vllm.entrypoints.generate.api_router path, and the reorganized online-serving docs.

AI assistance disclosure

AI assistance was used to analyze the code paths, rebase and draft this revision, and strengthen tests/docs. This PR is now a draft pending the human submitter reviewing every changed line and being able to defend the change end-to-end, as required by AGENTS.md.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--41510.org.readthedocs.build/en/41510/

[gemini-code-assist]

Code Review

This pull request extends the /tokenize endpoint to support the Responses API, allowing users to get token IDs for rendered model inputs from raw completion prompts, chat-completion messages, or Responses API inputs. The implementation refactors the OpenAIServingResponses class to extract input rendering into a shared render_response_inputs method, which is now utilized by both the Responses API and the tokenization service. Additionally, the PR introduces the TokenizeResponsesRequest protocol and includes comprehensive unit tests for the new functionality. I have no feedback to provide as there were no review comments to assess.

[noooop]

PTAL: #38084

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @SiluPanda.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

[SiluPanda]

Rebased onto current main and resolved the online-serving refactor conflicts. Updated the implementation/tests to use OnlineRenderer and the current generate router, moved the docs change to the reorganized online-serving guide, and reran the CPU-runnable focused suite: 42 passed, 1 xfailed. The end-to-end server tests still require a supported vLLM device and cannot run on this macOS arm64 host.

[SiluPanda]

Policy/status correction: the latest rebase and test/doc updates used AI assistance. I have converted this PR back to draft so it is not presented as review-ready before the human submitter has reviewed every changed line and can defend the revision end-to-end, per the repository's current AGENTS.md accountability requirements.

The duplicate-work checks and test evidence remain recorded in the PR body. No maintainer CI label is being requested while the human-review gate is outstanding.