[hickeyma]

Overview

Adds output_mode="derender" to /inference/v1/generate, so the generate server returns the parsed assistant message itself. This is Phase 2 of #56851 and builds on #60073.

Claims

• output_mode="derender" returns message (content, reasoning, tool calls) on every choice without a separate /derender call
• message, finish_reason and logprobs match coupled /v1/chat/completions even for parsers whose adjust_request changes decoding. /derender can't match those
• /v1/chat/completions/render and /v1/messages/render now return parse_context. Other output modes ignore it, including fields added by a newer render, so prefill legs don't change
• A bad parse_context gets a 400 before generation. When a parser hides the reasoning, token_ids are dropped along with logprobs, like coupled chat
• Batch /derender now supports parallel_tool_calls=false, like coupled chat

Validation

pytest tests/entrypoints/cohere/test_api_router.py -v
pytest tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py -v 
pytest tests/entrypoints/scale_out/token_in_token_out/test_protocol.py -v
pytest tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py -v
pytest tests/entrypoints/scale_out/derender/test_derender_stream.py -v 
pytest tests/entrypoints/scale_out/derender/test_derender_parity.py -v 

The parity tests run R1-Distill with hermes and deepseek_r1 against coupled chat. They cover plain output, reasoning shown and hidden, tool_choice auto, required and named, parallel_tool_calls=false, a stop string, n=2 and logprobs with top_logprobs=3 and logprob_token_ids. Every case checks finish_reason and the tool call ID shape.

Details

The derender level parses the engine's own output.text. That text already has the stop string cut and was decoded with the flags adjust_request set which is where /derender drifts from coupled chat (#42729).

• Chat context. /render puts a ParseContext on the GenerateRequest with the tools, tool_choice, parallel_tool_calls, include_reasoning, resolved chat_template_kwargs and the history tool call count. It's only read when output_mode="derender". I went with always sending it, rather than only on request, so a rendered request works as is (decision 2 on [RFC]: Request level text and derender output on /inference/v1/generate #56851). parse_context is validated in serve_tokens before the request reaches the engine. Unknown fields fail the request only when output_mode="derender" because that's the only level that reads them. At the other levels they're logged once and ignored, so an older generate node keeps serving a newer render. chat_template_kwargs is typed as JSON values to make sure whatever goes in it has to survive the wire. The trade-off is that every rendered request carries the tools list, prefill legs included.
• Parsing. The generate server builds a slim ChatCompletionRequest from parse_context and reuses _parse_and_assemble from [Frontend] Extract parse and assembly out of batch chat derender #60073. /render resolves the template kwargs once and uses them for both reasoning_parser_kwargs and parse_context. A handler can pass its _engine_chat_template_kwargs to keep API server only state out of parse_context. Cohere uses this for _position_to_source and friends. Tool call IDs keep the parser's tc.id ([Bugfix][Frontend] Keep the parser's tool call IDs in batch derender #61852).
• finish_reason goes through resolve_finish_reason ([Frontend] Make finish_reason and Responses item status truthful #59324), the same as coupled chat and batch /derender.
• Logprobs are chat shaped and cut with chat_top_logprobs_limit ([Frontend] Apply each endpoint's top_logprobs cut in derender #59518). They're on whenever num_logprobs is set, so logprob_token_ids works. When a parser hides the reasoning, logprobs and token_ids are both dropped. The tokens and text levels don't change.

Limitations:

• Streaming returns a 400 for now. That's Phase 3.
• The text level still ignores logprob_token_ids. That gap was already there.
• [Core][Frontend] SamplingParams.sampled_logprobs_only: sampled-token logprob fast path #57442 and [Frontend] generate: packed top-k logprobs (packed_top_k_logprobs) #60915 add logprob flags this level can't support. Neither is on main yet, so there's a TODO and whichever PR lands second adds the 400.
• Cohere citations from the derender level come back without document sources. The position-to-source map can't go over JSON. ([Feature]: Cohere grounding citations on the derender paths #63164)
• return_tokens_as_token_ids is ignored on derender paths. ([Bug]: return_tokens_as_token_ids is ignored on derender paths #63162)

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[mergify]

Documentation preview: https://vllm--60082.org.readthedocs.build/en/60082/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hickeyma.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hickeyma.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[hickeyma]

I am looking at the ruff error now which was caused by rebase.

[mergify]

Hi @hickeyma, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[hickeyma]

Ruff error is fixed now.

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #96393 for commit 7769f8284161.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 7769f8284161 before starting CI. Pull the branch before pushing more changes.

[hickeyma]

/ci run

[github-actions]

✅ Triggered Buildkite CI #96409 for commit 7b605335ea60.