[Frontend] Add derender output mode to generate

docs/serving/online_serving/derenderer.md

@@ -89,6 +89,8 @@ The derenderer builds its tool and reasoning parsers from its own server flags p
 - Start the render server with the same `--tool-call-parser`, `--reasoning-parser`, `--enable-auto-tool-choice`, `--chat-template` and `--default-chat-template-kwargs` you'd give `vllm serve` for this model. If `/render` and `/derender` run on different servers, give both the same values.
 - Send the full `chat_request` that went to `/render`, not just `messages` and `tools`. Fields like `chat_template_kwargs`, `reasoning_effort`, `tool_choice` and `include_reasoning` change how the output is parsed.
 
+`/derender` decodes with the flags in `chat_request` which can differ from what the engine ran with. For parsers whose `adjust_request` changes decoding, use [`output_mode: "derender"`](token_in_token_out.md#derender) on `/inference/v1/generate`. It parses the engine's own text and matches coupled chat.
+
 A mismatch doesn't fail. The parser just splits `reasoning`, `content` and `tool_calls` differently from what `vllm serve` would return.
 
 `/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. From `completion_request` it reads `skip_special_tokens`, and `logprobs` and `logprob_token_ids` to cut `top_logprobs` the way `/v1/completions` does.

docs/serving/online_serving/token_in_token_out.md

@@ -54,6 +54,7 @@ The request field `output_mode` selects how much of the postprocessing the serve
 | --- | --- | --- |
 | `tokens` (default) | nothing, token IDs only | nothing |
 | `text` | `text` on every choice and decoded tokens with `bytes` in `logprobs` | a tokenizer |
+| `derender` | the parsed assistant `message` (`content`, `reasoning`, `tool_calls`) on every choice, plus `logprobs` shaped like coupled chat's | a tokenizer and the parser flags you'd give `vllm serve` |
 
 `text` returns text and token IDs in a single call, without a separate [derender](derenderer.md) hop per chunk. This suits latency sensitive streaming, where the derender hop lands on every output token.
 
@@ -83,6 +84,18 @@ print(response["choices"][0]["token_ids"])
 - `/derender` only accepts `output_mode: "tokens"` responses and returns a 400 for text responses which are already detokenized.
 - With `stream: true`, each choice's `text` is the delta since the previous chunk. A chunk is sent whenever the engine output carries new text or a `finish_reason`, even with no new token IDs. That covers text held back for stop string matching and the final output after an abort.
 
+### Derender
+
+`derender` returns the assistant `message` a coupled `/v1/chat/completions` would, without a separate [derender](derenderer.md) call. `/v1/chat/completions/render` and `/v1/messages/render` return `parse_context` on the `GenerateRequest`. Send it back with `output_mode: "derender"`. It carries the tools, `tool_choice`, `include_reasoning`, resolved JSON only `chat_template_kwargs` and the history tool call count. A `parse_context` field this server doesn't know fails a `derender` request and is ignored at the other levels.
+
+- The parser reads the engine's `text`, so it sees the stop string cut and the post `adjust_request` decode flags. That makes it match coupled chat for parsers whose `adjust_request` changes decoding which `/derender` can't.
+- A server with a tool or reasoning parser needs `parse_context`. Without a parser, `message.content` is the plain text.
+- `parse_context` is ignored at other levels, so a rendered request can go to a prefill leg unchanged.
+- `logprobs` follow coupled chat. `top_logprobs` is cut to `sampling_params.logprobs`, `logprob_token_ids` is honored. When a parser hides the reasoning with `include_reasoning: false`, `logprobs`, `token_ids` and `sampling_mask` are dropped so they don't expose it.
+- `finish_reason` follows coupled chat too. It's `tool_calls` when the message has tool calls except with a named `tool_choice` or a `length` cut.
+- `stream: true` isn't supported yet.
+- Cohere grounding citations aren't returned. The parser strips the citation spans from `content`, and the document map can't go over JSON in `parse_context`.
+
 ### Logprobs
 
 With `output_mode: "tokens"`, logprobs are `GenerateLogProbs` with an integer `token_id` and `rank` per entry and no `token` or `bytes` (see [Generate Output Logprobs](renderer.md#generate-output-logprobs)). With `output_mode: "text"`, they are `ChatCompletionLogProbs` carrying the decoded token strings and their UTF-8 `bytes` for the sampled token and every entry in `top_logprobs`; a server started with `--return-tokens-as-token-ids` returns `token_id:N` placeholders there instead, as `/v1/completions` does.
@@ -91,8 +104,10 @@ With `output_mode: "tokens"`, logprobs are `GenerateLogProbs` with an integer `t
 
 The server returns a 400 for:
 
-- `output_mode: "text"` on a server without a tokenizer (`--tokens-only` or `--skip-tokenizer-init`).
-- `output_mode: "text"` with `sampling_params.detokenize: false`.
+- `output_mode: "text"` or `"derender"` on a server without a tokenizer (`--tokens-only` or `--skip-tokenizer-init`).
+- `output_mode: "text"` or `"derender"` with `sampling_params.detokenize: false`.
+- `output_mode: "derender"` with `stream: true`.
+- `output_mode: "derender"` without `parse_context` on a server with a tool or reasoning parser or with a named `tool_choice` in `parse_context` that matches none of its `tools`. This is checked before generating.
 - An unsupported `output_mode` value.
 
 For using `output_mode` with separate prefill and decode pools, see [Disaggregated Prefilling](../../features/disagg_prefill.md#generate-api-output-modes).

pyproject.toml

@@ -199,6 +199,7 @@ extend-exclude = ["tests/models/fixtures/*", "tests/prompts/*", "tests/tokenizer
     "tests/entrypoints/speech_to_text/transcription/test_transcription_validation.py",
     "docs/governance/process.md", "docs/community/reviewers.md", "docs/assets/contributing/vllm_bench_serve_timeline.html", 
     "tests/v1/engine/test_fast_incdec_prefix_err.py", ".git/*", "csrc/cpu/sgl-kernels/*",
+    "tests/entrypoints/scale_out/token_in_token_out/test_protocol.py",
     "rust/src/chat/src/renderer/deepseek_v32/fixtures/*", "rust/src/parser/**",
     "rust/src/text/src/output/decoded.rs",
     "rust/src/tokenizer/src/incremental.rs",

tests/entrypoints/cohere/test_api_router.py

@@ -122,9 +122,11 @@ class _RenderHandler:
     def __init__(self, result):
         self.result = result
         self.seen_request = None
+        self.seen_kwargs_filter = None
 
-    async def render_chat_request(self, request):
+    async def render_chat_request(self, request, engine_chat_template_kwargs=None):
         self.seen_request = request
+        self.seen_kwargs_filter = engine_chat_template_kwargs
         if isinstance(self.result, Exception):
             raise self.result
         return self.result
@@ -488,22 +490,27 @@ def test_cohere_request_surface_survives_conversion(self):
         assert kwargs["safety_mode"] == "strict"
 
     def test_api_server_only_kwargs_not_forwarded(self):
-        """As on ``/cohere/v2/chat``, the engine's reasoning parser must not
-        receive the template kwargs that only the API server reads."""
+        """As on ``/cohere/v2/chat``, the engine's reasoning parser and the
+        derender ``parse_context`` must not receive the template kwargs that
+        only the API server reads. Their tuple keys can't be JSON encoded."""
+        kwargs = {
+            "safety_mode": "strict",
+            **dict.fromkeys(_API_SERVER_ONLY_TEMPLATE_KWARGS, {(0, 0): "x"}),
+        }
         result = _generate_request()
         result.reasoning_parser_kwargs = ReasoningParserKwargs(
-            chat_template_kwargs={
-                "safety_mode": "strict",
-                **dict.fromkeys(_API_SERVER_ONLY_TEMPLATE_KWARGS, "x"),
-            }
+            chat_template_kwargs=kwargs
         )
-        app = _build_render_app(_RenderChatHandler(), _RenderHandler(result))
+        render_handler = _RenderHandler(result)
+        app = _build_render_app(_RenderChatHandler(), render_handler)
         with TestClient(app) as client:
             r = client.post("/cohere/v2/chat/render", json=_minimal_request_body())
         assert r.status_code == HTTPStatus.OK
         assert r.json()["reasoning_parser_kwargs"] == {
             "chat_template_kwargs": {"safety_mode": "strict"}
         }
+        assert render_handler.seen_kwargs_filter is not None
+        assert render_handler.seen_kwargs_filter(kwargs) == {"safety_mode": "strict"}
 
     def test_501_when_chat_handler_missing(self):
         app = _build_render_app(None, _RenderHandler(_generate_request()))

tests/entrypoints/scale_out/derender/test_derender_parity.py

@@ -878,3 +878,162 @@ async def test_text_parity_stop_string(client, stream):
     if coupled["stop_reason"] != stop:
         pytest.skip("Model did not emit the stop string")
     assert derendered[len(inline_text) :].startswith(stop)
+
+
+# ---------------------------------------------------------------------------
+# Inline derender level: generate(output_mode="derender") == chat completions
+# ---------------------------------------------------------------------------
+
+
+async def _inline_derender(
+    client: httpx.AsyncClient, messages: list[dict], **extra
+) -> dict:
+    """Render a chat request, then generate it at the derender level with
+    the `parse_context` that `/render` returned."""
+    render = await client.post(
+        "/v1/chat/completions/render",
+        json={
+            "model": MODEL,
+            "messages": messages,
+            "temperature": 0,
+            "max_tokens": 128,
+            **extra,
+        },
+    )
+    assert render.status_code == 200, render.text
+    generate_request = render.json()
+    assert generate_request["parse_context"] is not None
+
+    resp = await client.post(
+        "/inference/v1/generate",
+        json={**generate_request, "output_mode": "derender"},
+    )
+    assert resp.status_code == 200, resp.text
+    data = resp.json()
+    assert data["output_mode"] == "derender"
+    return data
+
+
+def _assert_inline_parity(coupled: dict, inline: dict) -> None:
+    assert len(inline["choices"]) == len(coupled["choices"])
+    for c, i in zip(coupled["choices"], inline["choices"]):
+        if c["token_ids"] is None:
+            # Hidden reasoning suppresses token ids on both paths, so the two
+            # generations can't be shown to match and their text may differ.
+            assert i["token_ids"] is None
+            assert i["message"].get("reasoning") is None
+            assert c["message"].get("reasoning") is None
+            continue
+        assert i["token_ids"] == c["token_ids"], (
+            "greedy or seeded generation was expected to be deterministic "
+            "across the coupled and inline calls"
+        )
+        assert i["message"]["content"] == c["message"]["content"]
+        assert i["message"].get("reasoning") == c["message"].get("reasoning")
+        assert _tool_sig(i) == _tool_sig(c)
+        assert _tool_id_shapes(i) == _tool_id_shapes(c)
+        assert i["finish_reason"] == c["finish_reason"]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    ("content", "extra"),
+    [
+        pytest.param(TEXT_MESSAGES[0]["content"], {}, id="plain"),
+        pytest.param(
+            "What is 17 times 23? Think it through.",
+            {"include_reasoning": True, "max_tokens": 256},
+            id="reasoning",
+        ),
+        pytest.param(
+            "What is 17 times 23? Think it through.",
+            {"include_reasoning": False, "max_tokens": 256},
+            id="reasoning-hidden",
+        ),
+        pytest.param(
+            "What's the weather in Paris?",
+            {"tools": TOOLS, "tool_choice": "auto", "max_tokens": 1024},
+            id="tool-auto",
+        ),
+        pytest.param(
+            "What's the weather in Paris?",
+            {"tools": TOOLS, "tool_choice": "required", "max_tokens": 1024},
+            id="tool-required",
+        ),
+        pytest.param(
+            "What's the weather in Paris?",
+            {"tools": TOOLS, "tool_choice": FORCE_WEATHER_TOOL, "max_tokens": 1024},
+            id="tool-named",
+        ),
+        pytest.param(
+            "What's the weather in Paris and in Rome?",
+            {
+                "tools": TOOLS,
+                "tool_choice": "required",
+                "parallel_tool_calls": False,
+                "max_tokens": 1024,
+            },
+            id="tool-no-parallel",
+        ),
+        pytest.param(
+            "Write two short lines about the sea.", {"stop": ["\n"]}, id="stop"
+        ),
+        pytest.param(
+            TEXT_MESSAGES[0]["content"],
+            {"n": 2, "temperature": 0.8, "seed": 42},
+            id="n2",
+        ),
+    ],
+)
+async def test_inline_derender_parity(client, content, extra):
+    """The derender level parses the engine's own text, so it matches coupled
+    chat, including for hermes with tools whose `adjust_request` changes the
+    decode flags."""
+    messages = [{"role": "user", "content": content}]
+    coupled = await _coupled(client, messages, **extra)
+    inline = await _inline_derender(client, messages, **extra)
+    _assert_inline_parity(coupled, inline)
+
+    if extra.get("parallel_tool_calls") is False:
+        assert len(_tool_sig(inline["choices"][0])) <= 1
+
+
+def _top_tokens(entry: dict) -> list[str]:
+    return [top["token"] for top in entry["top_logprobs"]]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    ("include_reasoning", "extra"),
+    [
+        pytest.param(True, {"top_logprobs": 3}, id="top3"),
+        pytest.param(False, {"top_logprobs": 3}, id="top3-reasoning-hidden"),
+        pytest.param(True, {"logprob_token_ids": [15, 16, 17]}, id="token-ids"),
+    ],
+)
+async def test_inline_derender_logprobs(client, include_reasoning, extra):
+    """Logprobs match coupled chat, including the `top_logprobs` cut and
+    `logprob_token_ids`, and are dropped when reasoning is hidden."""
+    messages = [{"role": "user", "content": "What is 2+2?"}]
+    extra = {"logprobs": True, "include_reasoning": include_reasoning, **extra}
+    inline = (await _inline_derender(client, messages, **extra))["choices"][0]
+
+    if not include_reasoning:
+        assert inline["logprobs"] is None
+        assert inline["token_ids"] is None
+        return
+
+    coupled = (await _coupled(client, messages, **extra))["choices"][0]
+    i_content = inline["logprobs"]["content"]
+    c_content = coupled["logprobs"]["content"]
+    assert [(e["token"], e["bytes"]) for e in i_content] == [
+        (e["token"], e["bytes"]) for e in c_content
+    ]
+    if "top_logprobs" in extra:
+        # Near ties can reorder candidates across the two generations, so
+        # compare the cut, not the candidates.
+        assert all(len(e["top_logprobs"]) == 3 for e in i_content)
+    else:
+        assert [sorted(_top_tokens(e)) for e in i_content] == [
+            sorted(_top_tokens(e)) for e in c_content
+        ]

tests/entrypoints/scale_out/derender/test_derender_stream.py

@@ -35,6 +35,7 @@
     DeltaFunctionCall,
     DeltaMessage,
     DeltaToolCall,
+    FunctionCall,
     PerRequestMetrics,
     SpeculativeDecodingMetrics,
 )
@@ -55,6 +56,7 @@
     GenerateTokensResponse,
     GenerateTokensStreamChoice,
     GenerateTokensStreamResponse,
+    ParseContext,
 )
 from vllm.entrypoints.serve.engine.protocol import ErrorResponse, UsageInfo
 from vllm.parser import Parser
@@ -427,6 +429,56 @@ def batch_content(token_ids: list[int], finish_reason: str) -> str:
     assert stream.choices[0].delta.content == expected
 
 
+_WEATHER_TOOLS = [
+    {
+        "type": "function",
+        "function": {"name": "get_weather", "parameters": {"type": "object"}},
+    }
+]
+
+
+@pytest.mark.parametrize(
+    "tool_choice",
+    ["auto", "required", {"type": "function", "function": {"name": "get_weather"}}],
+)
+def test_derender_text_choices_builds_request_from_parse_context(
+    derenderer, monkeypatch, tool_choice
+):
+    """Every tool_choice survives the synthetic request validators, the
+    history count is seeded and parallel_tool_calls=false keeps one call."""
+    parser = MagicMock()
+    parser.parse.return_value = (
+        None,
+        None,
+        [
+            FunctionCall(name="get_weather", arguments="{}"),
+            FunctionCall(name="get_weather", arguments="{}"),
+        ],
+    )
+    monkeypatch.setattr(derenderer, "parser", MagicMock(return_value=parser))
+    parse_context = ParseContext.model_validate(
+        {
+            "tools": _WEATHER_TOOLS,
+            "tool_choice": tool_choice,
+            "parallel_tool_calls": False,
+            "history_tool_call_cnt": 3,
+        }
+    )
+    output = MagicMock(text="raw", token_ids=[1, 2], finish_reason="stop")
+
+    ((message, finish_reason),) = derenderer._derender_text_choices(
+        [output], parse_context.to_chat_request(), 3, [11]
+    )
+
+    chat_request = parser.parse.call_args.args[1]
+    assert chat_request.tool_choice == parse_context.tool_choice
+    parser.set_prompt_token_ids.assert_called_once_with([11])
+    parser.set_history_tool_call_count.assert_called_once_with(3)
+    assert len(message.tool_calls) == 1
+    # A forced named tool reports "stop", like coupled chat.
+    assert finish_reason == ("stop" if isinstance(tool_choice, dict) else "tool_calls")
+
+
 class TestDetokenizeDelta:
     """_detokenize_delta: chunked decode must equal one shot decode."""
 
@@ -2439,7 +2491,7 @@ async def test_stream_parsed_matches_batch_tool_call(parser_client, parser_token
     )
     assert batch_resp.status_code == 200, batch_resp.text
     batch_choice = batch_resp.json()["choices"][0]
-    if not batch_choice["message"]["tool_calls"]:
+    if not batch_choice["message"].get("tool_calls"):
         pytest.skip("Model did not emit a tool call")
 
     for chunk_sizes in ([len(output_ids)], [1] * len(output_ids)):

tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py

@@ -37,6 +37,7 @@
 )
 from vllm.outputs import CompletionOutput, RequestOutput, SamplingMask
 from vllm.renderers import renderer_from_config
+from vllm.renderers.online_derenderer import OnlineDerenderer
 from vllm.renderers.online_renderer import OnlineRenderer
 from vllm.sampling_params import SamplingParams
 from vllm.v1.engine.async_llm import AsyncLLM
@@ -122,6 +123,7 @@ def _build_serving_tokens(engine: AsyncLLM, **kwargs) -> ServingTokens:
         engine,
         models,
         online_renderer=online_renderer,
+        online_derenderer=MagicMock(spec=OnlineDerenderer),
         request_logger=None,
         **kwargs,
     )