[Rust Frontend] Add /derender endpoints: detokenization and state (phase 1/3) by ezhoureal · Pull Request #53223 · vllm-project/vllm

ezhoureal · 2026-08-21T07:00:36Z

Purpose

Phase 1/3 — split per @sagearc's review, mirroring the Python derender phasing in #42729 (#43606 detok → #45919 parsing → #48617 streaming). Follow-ups: #53418 (parsing), #53419 (streaming + GPU e2e).

Add POST /v1/chat/completions/derender and POST /v1/completions/derender to the Rust frontend, wire-compatible with the Python implementation in vllm/entrypoints/scale_out/derender/ + vllm/renderers/online_derenderer.py. The endpoints turn a GenerateResponse from /inference/v1/generate back into an OpenAI ChatCompletionResponse / CompletionResponse without a GPU, completing the render/derender pair used by disaggregated serving. The Rust frontend already has the render side (/v1/*/render); this adds the inverse.

Phase 1 ships the shared detokenization and state foundations plus the plain non-streaming endpoints — the equivalent of Python #43606:

derender/detok.rs: the incremental detokenization window (prev_tokens/prefix_offset/read_offset, capped at 1024) porting detokenize_incrementally; shared with streaming in phase 3.
derender/types.rs: non-streaming wire types plus DerenderStreamState (with validation) as the parsing-ready state; stream: true bodies are rejected with 400 until phase 3.
derender/logprobs.rs: token_id:N logprob placeholders are resolved back to decoded tokens (with the U+FFFD byte-fallback repair), matching Python's _resolve_logprobs.
Non-streaming chat derender always plain-detokenizes in this phase (reasoning/tool parsing lands in phase 2, exactly like Python [Render] Add /derender endpoints for disaggregated postprocessing #43606 → [Render] Add reasoning/tool parsing to /derender + fix byte-fallback FFFD #45919); completion derender flattens generate_responses × choices with re-indexing from 0, converts chat logprobs to the flat completion shape, aggregates usage, and passes kv_transfer_params through (dropped with a warning when responses disagree).
Caller payloads are bounds-checked (max_model_len, max_logprobs, sequence counts) before any tokenizer work, mirroring _validate_derender_bounds.
Both endpoints are registered on the render-only server and the engine-backed server, as Python registers them on both the full API server and the render-only launcher.

Related: #42729 (derender endpoints RFC) and #47161 (streaming derender RFC) — this stack ports both the non-streaming and streaming derender contract to the Rust frontend; the broader RFC scope stays open.

Duplicate-work check: searched open PRs via gh pr list --search for derender, derender path:rust, rust, frontend, and related keywords. No open PR implements derender in rust/. The open derender PRs (#50550 stream reasoning/tool calls, #47931 tool_calls finish reason) modify only the Python implementation; they do not overlap with this Rust port, and this stack deliberately matches current Python semantics so those Python changes can be ported as follow-ups.

AI assistance: this PR was implemented with AI assistance (Kimi Code CLI). I have reviewed the full diff and run the tests below myself.

Test Plan

31 route tests in rust/src/server/src/routes/tests.rs port the non-parsing, non-streaming behaviors asserted by tests/entrypoints/scale_out/derender/test_derender.py: text roundtrips, request_id echo, usage forwarding (supplied/omitted prompt_tokens), prompt_logprobs/kv_transfer_params passthrough, logprob placeholder resolution with bytes, all 400/404 error and bounds cases, and plain-detok fallback without chat_request.

cargo nextest run -p vllm-server -p vllm-chat -p vllm-text -p vllm-tokenizer
cargo clippy -p vllm-server -p vllm-chat -p vllm-text -p vllm-tokenizer --all-targets
cargo fmt --all --check
pre-commit run --files <changed files>

Test Result

cargo nextest run (4 touched crates): 825 passed, 1 skipped — includes all 31 phase-1 derender tests.
cargo clippy --all-targets: clean.
cargo fmt --check: clean.
pre-commit run on the changed files: all applicable hooks passed (SPDX headers, Rust fmt, etc.).

Model evals: N/A — this adds new endpoints to the Rust frontend without changing any existing serving behavior or model outputs; correctness is covered by the route-level tests above and wire compatibility with the existing Python derender contract. End-to-end GPU validation of the full stack (incl. Python↔Rust derender parity on real generations) lands with phase 3 (#53419).

Known deviations from Python (intentional)

Embedded chat_request/completion_request are validated on lowering (e.g. unknown model → 404); Python passes them to the parser unchecked.
null token_ids → 400 at JSON parse; completion-side empty token_ids is a consistent 400 (Python leaks a 500 there).
MAX_N_SEQUENCES is a constant (16384) with a TODO to plumb VLLM_MAX_N_SEQUENCES through server config.
convert_tokens_to_string is approximated via piece→id roundtrip + one-shot decode (exact for byte-level BPE tokenizers).

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d410bec728

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-21T07:15:39Z

+
+    let events = vec![
+        Ok(DecodedTextEvent::Start {
+            prompt_token_ids: Arc::from([]),


Pass real prompt IDs into parser replay

When parsed derender is used, this synthetic Start initializes the chat parser with an empty prompt. Several parsers derive their starting mode from the rendered prompt tail (for example Kimi K3 starts generation inside a prefilled think/response channel and does not re-emit the opener), so those models will classify generated reasoning/response text as the wrong stream when derendering even though the normal chat pipeline would parse it correctly. Please render/tokenize the supplied chat_request and pass the actual prompt token IDs into the replayed stream before parsing.

Useful? React with 👍 / 👎.

Fixed in 4cfeb02 — the parser replay now renders and tokenizes the embedded chat_request and feeds the real prompt_token_ids into the Start event, so prompt-dependent parsers start in the right mode. Added a regression test with deepseek_r1 where a prompt ending in now yields pure content instead of misclassified reasoning.

chatgpt-codex-connector · 2026-08-21T07:15:39Z

+
+    for choice in &request.generate_chunk.choices {
+        let (new_text, updated_state) =
+            detok::detokenize_delta(&ctx.tokenizer, &choice.token_ids, &state, skip_special)?;


Bound streaming derender chunks before decoding

The streaming handlers only validate state and choice count before calling detokenize_delta, unlike the non-streaming path which rejects token_ids longer than max_model_len before decoding. A caller can therefore POST a single streaming derender chunk with an oversized token_ids vector and force token-by-token detokenization past the server's configured resource bound. Please apply the same max_model_len check to generate_chunk before this decode step.

Useful? React with 👍 / 👎.

Fixed in 4cfeb02 — streaming chunks now get the same max_model_len/max_logprobs bounds check before decoding. This intentionally goes beyond Python, which only validates the non-streaming payloads.

chatgpt-codex-connector · 2026-08-21T07:15:39Z

+        choices.push(ChatCompletionChoice {
+            index: choice.index,
+            message,
+            logprobs: resolved_logprobs,


Drop logprobs when hidden reasoning is suppressed

When parsed derender is used with chat_request.include_reasoning=false and the generate response includes choice.logprobs, the message hides the reasoning text but this still attaches logprobs for the entire generated token stream. The regular chat completion path omits output metadata in that case to avoid leaking hidden reasoning tokens through per-token data, so this endpoint should apply the same gate before returning logprobs.

Useful? React with 👍 / 👎.

Fixed in 4cfeb02 — verified the normal chat path gates this way and matched it: choice logprobs are now dropped when reasoning was parsed but hidden via include_reasoning=false.

chatgpt-codex-connector · 2026-08-21T07:15:39Z

+    tokenizer: &DynTokenizer,
+    pieces: &[String],
+) -> Result<String, ApiError> {
+    let ids: Vec<u32> = pieces.iter().filter_map(|piece| tokenizer.token_to_id(piece)).collect();


Avoid lossy piece round-trips for streaming detok

This reconstructs text by converting id_to_token pieces back through token_to_id, but the tiktoken backend stores base-token byte pieces as lossy UTF-8 when the token bytes are not valid alone. For streamed output where a UTF-8 character is split across byte-fallback tokens, those pieces become replacement characters or fail to map back and are dropped, so derendered streaming text no longer matches one-shot decode. Please carry token IDs in the state or use a lossless byte-piece conversion for this path.

Useful? React with 👍 / 👎.

Fixed in 4cfeb02 — confirmed tiktoken stores base-token pieces via from_utf8_lossy. DerenderStreamState now optionally carries prev_token_ids alongside prev_tokens, and the detok window reconstructs from IDs; Python ignores the extra field, and foreign states without IDs fall back to the old piece mapping, so cross-frontend compatibility holds either way.

chatgpt-codex-connector · 2026-08-21T07:15:39Z

+        choices,
+        usage: Usage {
+            prompt_tokens,
+            total_tokens: prompt_tokens + completion_tokens,


Validate prompt token counts before adding usage

Because prompt_tokens is supplied by the request body, a caller can set it near usize::MAX; adding completion_tokens here then panics in checked builds and wraps in release builds, producing an invalid usage object. Please reject oversized prompt counts or use checked/saturating arithmetic anywhere derender aggregates caller-provided usage.

Useful? React with 👍 / 👎.

Fixed in 4cfeb02 — all usage aggregation sites now use checked arithmetic and return 400 on overflow.

chatgpt-codex-connector · 2026-08-21T07:15:40Z

+) -> Result<DerenderChatStreamResponse, ApiError> {
+    check_model(ctx, request.model.as_deref())?;
+
+    if ctx.has_parser() {


Allow plain streaming for tool-only parser models

This rejects streaming chat derender whenever the model has any parser registered, but normal chat output only enables the tool parser when the request actually has tools/tool_choice. For a tool-capable model with no tools in this request (or no chat_request), there is no tool markup to protect and the normal pipeline would be plain text, so this makes simple streaming derender return 400 unnecessarily. Please fail closed for reasoning parsers, but gate tool-parser rejection on the request actually enabling tool parsing.

Useful? React with 👍 / 👎.

Fixed in 4cfeb02 — streaming now fails closed only for reasoning parsers. Tool-only parser models stream plainly unless the request actually engages tool parsing (tools present with tool_choice != none, or required/named). Deliberate deviation from Python, which rejects whenever any parser is configured; noted in the module docs.

chatgpt-codex-connector · 2026-08-21T07:15:40Z

+
+            choices.push(CompletionChoice {
+                index,
+                text: decoded_text,


Honor completion echo in derender output

When the embedded completion_request has echo=true (especially the max_tokens=0 prompt-only case), returning only decoded_text drops the prompt that the regular completions path prefixes, and prompt-only requests will expose the internal generated token instead of just the prompt. This means render → generate → derender does not reproduce /v1/completions for supported echo requests; please use completion_request.echo/prompt-only handling when constructing the derendered choice text and logprob offsets.

Useful? React with 👍 / 👎.

Fixed in 4cfeb02 — completion derender now honors echo, reusing completion_echo_text from the normal path, including the max_tokens=0 prompt-only case and prompt_logprobs with shifted text_offsets. Python's derender ignores echo entirely, so this closes a parity gap that exists there too.

sagearc

The python derender RFC split this work into phases so each contract could be validated independently. I think it would help to keep the same structure here, starting with shared detokenization and state, then adding tool and reasoning parsing and streaming.

…ase 1/3) Phase 1 of 3 splitting the Rust derender endpoints per review on vllm-project#53223, mirroring the Python phasing in vllm-project#43606 (detokenization first; parsing and streaming follow). Related issues: vllm-project#42729, vllm-project#47161. Adds /v1/chat/completions/derender and /v1/completions/derender to both the render-only and engine-backed servers: shared incremental detokenization foundation, DerenderStreamState, logprob placeholder resolution, and the plain non-streaming derender paths with bounds validation. Reasoning/ tool-call parsing (phase 2) and streaming (phase 3) are intentionally absent; a request body with stream=true fails deserialization with 400. This change was developed with AI assistance (Kimi Code CLI). Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com>

ezhoureal · 2026-08-23T02:08:10Z

@sagearc Good call — split into phases mirroring the Python derender structure from #42729:

This PR is now phase 1/3: shared detokenization + DerenderStreamState + plain non-streaming derender (the [Render] Add /derender endpoints for disaggregated postprocessing #43606 equivalent). stream: true bodies are rejected with 400 until phase 3, and chat derender plain-detokenizes until phase 2 — so each contract can be validated independently.
Phase 2/3 ([Rust Frontend] /derender: reasoning and tool-call parsing (phase 2/3) #53418): reasoning/tool-call parsing via the production chat output pipeline (the [Render] Add reasoning/tool parsing to /derender + fix byte-fallback FFFD #45919 equivalent).
Phase 3/3 ([Rust Frontend] /derender: streaming derender + two-process e2e test (phase 3/3) #53419): streaming derender with the client-carried state protocol (the [Render] Add round trip parity test and docs for derender #48617 equivalent), plus a two-process GPU e2e test (real engine + vllm-rs render) that also pins Python↔Rust derender parity on identical greedy token IDs.

Each phase keeps its own route-test subset green (cargo nextest: 825/829/847 passed across the phases).

mergify · 2026-08-29T08:44:14Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ezhoureal.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…ase 1/3) Phase 1 of 3 splitting the Rust derender endpoints per review on vllm-project#53223, mirroring the Python phasing in vllm-project#43606 (detokenization first; parsing and streaming follow). Related issues: vllm-project#42729, vllm-project#47161. Adds /v1/chat/completions/derender and /v1/completions/derender to both the render-only and engine-backed servers: shared incremental detokenization foundation, DerenderStreamState, logprob placeholder resolution, and the plain non-streaming derender paths with bounds validation. Reasoning/ tool-call parsing (phase 2) and streaming (phase 3) are intentionally absent; a request body with stream=true fails deserialization with 400. This change was developed with AI assistance (Kimi Code CLI). Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com>

…ase 1/3) Phase 1 of 3 splitting the Rust derender endpoints per review on vllm-project#53223, mirroring the Python phasing in vllm-project#43606 (detokenization first; parsing and streaming follow). Related issues: vllm-project#42729, vllm-project#47161. Adds /v1/chat/completions/derender and /v1/completions/derender to both the render-only and engine-backed servers: shared incremental detokenization foundation, DerenderStreamState, logprob placeholder resolution, and the plain non-streaming derender paths with bounds validation. Reasoning/ tool-call parsing (phase 2) and streaming (phase 3) are intentionally absent; a request body with stream=true fails deserialization with 400. This change was developed with AI assistance (Kimi Code CLI). Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com> Signed-off-by: Tianer Zhou <ezhoureal@gmail.com>

Phase 2 of the derender split per review on vllm-project#53223, mirroring the Python implementation from vllm-project#45919. Stacked on phase 1 (detokenization + state). Restores non-streaming chat reasoning/tool-call parsing: when a parser is configured and `chat_request` is supplied, generated tokens are replayed through the production chat output pipeline so the configured parser splits them into reasoning, content and tool calls; otherwise the endpoint falls back to plain detokenization. Hidden reasoning also suppresses per-token logprobs, matching the normal chat path. Streaming endpoints remain phase 3: `stream: true` bodies still fail deserialization with a 400. Related: vllm-project#42729 Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com> Signed-off-by: Tianer Zhou <ezhoureal@gmail.com>

…(phase 3/3) Phase 3 of the 3-phase split per review on vllm-project#53223, stacked on phase 2 (rust-derender-2-parsing). Restores the remaining streaming functionality of the Rust /derender endpoints — streaming wire types and union variants, derender_chat_stream / derender_completion_stream handlers with validate_stream_bounds / stream_usage / tool_parsing_would_engage, stream dispatch in derender/mod.rs, and removal of the phase-1/2 #[allow(dead_code)] gates — and adds the two-process Python e2e test (RemoteRustRenderServer + test_derender_rust_e2e.py). Implements the client-carried DerenderStreamState protocol, mirroring the Python implementation from vllm-project#48617. Related: vllm-project#42729, vllm-project#47161. This change was made with AI assistance (Kimi Code CLI). Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com> Signed-off-by: Tianer Zhou <ezhoureal@gmail.com>

ezhoureal requested review from BugenZhao and njhill as code owners August 21, 2026 07:00

claude Bot reviewed Aug 21, 2026

View reviewed changes

mergify Bot added the rust label Aug 21, 2026

chatgpt-codex-connector Bot reviewed Aug 21, 2026

View reviewed changes

ezhoureal mentioned this pull request Aug 22, 2026

[RFC]: Derender Endpoints to Provide Detokenization for Disaggregated Serving #42729

Open

1 task

sagearc reviewed Aug 22, 2026

View reviewed changes

ezhoureal force-pushed the rust-derender-endpoints branch from 4cfeb02 to 57d4a24 Compare August 23, 2026 02:04

This was referenced Aug 23, 2026

[Rust Frontend] /derender: reasoning and tool-call parsing (phase 2/3) #53418

Draft

[Rust Frontend] /derender: streaming derender + two-process e2e test (phase 3/3) #53419

Draft

ezhoureal changed the title ~~[Rust Frontend] Add /derender endpoints for disaggregated postprocessing~~ [Rust Frontend] Add /derender endpoints: detokenization and state (phase 1/3) Aug 23, 2026

mergify Bot added the needs-rebase label Aug 29, 2026

ezhoureal force-pushed the rust-derender-endpoints branch from 57d4a24 to f25fe4f Compare August 30, 2026 01:57

ezhoureal force-pushed the rust-derender-endpoints branch from f25fe4f to f69f098 Compare August 30, 2026 01:58

mergify Bot removed the needs-rebase label Aug 30, 2026

BugenZhao mentioned this pull request Sep 2, 2026

[Perf][Rust Frontend] Coalesce decoded chunks per engine update #55012

Merged

4 tasks

ezhoureal marked this pull request as draft September 11, 2026 07:36

hickeyma mentioned this pull request Sep 14, 2026

[RFC]: Request level text and derender output on /inference/v1/generate #56851

Open

1 task

Uh oh!

Uh oh!

Conversation

ezhoureal commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Known deviations from Python (intentional)

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

ezhoureal Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

ezhoureal Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

ezhoureal Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

ezhoureal Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

ezhoureal Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

ezhoureal Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

ezhoureal Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

sagearc left a comment

Choose a reason for hiding this comment

Uh oh!

ezhoureal commented Aug 23, 2026

Uh oh!

mergify Bot commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ezhoureal commented Aug 21, 2026 •

edited

Loading