Conversation
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
…ase 1/3) Phase 1 of 3 splitting the Rust derender endpoints per review on vllm-project#53223, mirroring the Python phasing in vllm-project#43606 (detokenization first; parsing and streaming follow). Related issues: vllm-project#42729, vllm-project#47161. Adds /v1/chat/completions/derender and /v1/completions/derender to both the render-only and engine-backed servers: shared incremental detokenization foundation, DerenderStreamState, logprob placeholder resolution, and the plain non-streaming derender paths with bounds validation. Reasoning/ tool-call parsing (phase 2) and streaming (phase 3) are intentionally absent; a request body with stream=true fails deserialization with 400. This change was developed with AI assistance (Kimi Code CLI). Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com> Signed-off-by: Tianer Zhou <ezhoureal@gmail.com>
Phase 2 of the derender split per review on vllm-project#53223, mirroring the Python implementation from vllm-project#45919. Stacked on phase 1 (detokenization + state). Restores non-streaming chat reasoning/tool-call parsing: when a parser is configured and `chat_request` is supplied, generated tokens are replayed through the production chat output pipeline so the configured parser splits them into reasoning, content and tool calls; otherwise the endpoint falls back to plain detokenization. Hidden reasoning also suppresses per-token logprobs, matching the normal chat path. Streaming endpoints remain phase 3: `stream: true` bodies still fail deserialization with a 400. Related: vllm-project#42729 Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com> Signed-off-by: Tianer Zhou <ezhoureal@gmail.com>
…(phase 3/3) Phase 3 of the 3-phase split per review on vllm-project#53223, stacked on phase 2 (rust-derender-2-parsing). Restores the remaining streaming functionality of the Rust /derender endpoints — streaming wire types and union variants, derender_chat_stream / derender_completion_stream handlers with validate_stream_bounds / stream_usage / tool_parsing_would_engage, stream dispatch in derender/mod.rs, and removal of the phase-1/2 #[allow(dead_code)] gates — and adds the two-process Python e2e test (RemoteRustRenderServer + test_derender_rust_e2e.py). Implements the client-carried DerenderStreamState protocol, mirroring the Python implementation from vllm-project#48617. Related: vllm-project#42729, vllm-project#47161. This change was made with AI assistance (Kimi Code CLI). Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com> Signed-off-by: Tianer Zhou <ezhoureal@gmail.com>
…ping, reasoning assertion - _resolve_vllm_rs_binary: prefer rust/target builds over the wheel-packaged vllm-rs, which may be built from an older main snapshot without derender. - _generate: strip explicit null sampling params emitted by the Rust render server, which the Python /inference/v1/generate msgspec validation rejects (pre-existing render-side wire incompatibility, unrelated to derender). - test_e2e_chat_roundtrip: accept output in reasoning when a reasoning parser is configured. Verified on RTX 3070 (8GB, WSL2): 4 passed, 1 skipped (tool-call parity skip: model did not emit a tool call, gated by design). Co-authored-by: Kimi Code CLI Signed-off-by: zireael <zireael@users.noreply.github.com> Signed-off-by: Tianer Zhou <ezhoureal@gmail.com>
ezhoureal
force-pushed
the
rust-derender-3-streaming
branch
from
August 30, 2026 02:05
2c674df to
11e04db
Compare
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Stacked on #53418 (phase 2/3) — review only the top commit(s). Split per @sagearc's review on #53223, mirroring the Python derender phasing in #42729 (#43606 detok → #45919 parsing → #48617 streaming).
Phase 3: streaming derender for both
/v1/chat/completions/derenderand/v1/completions/derender, mirroring Python #48617 — plus a two-process GPU end-to-end test.DerenderStreamStateprotocol: bounded incremental detok window (prev_tokens/prefix_offset/read_offset, capped at 1024) portingdetokenize_incrementally, role emitted once, usage forwarding. Chunked derendering produces byte-identical text to one-shot decode, including multibyte characters split across chunk boundaries.NotImplementedErrorpath.tests/entrypoints/scale_out/derender/test_derender_rust_e2e.py: boots a real GPU engine (vllm serve) and the Rust render server (vllm-rs render) as two processes and verifies render →/inference/v1/generate→ derender roundtrips, Python↔Rust derender parity on identical greedy token IDs, and streaming chunked == one-shot over real tokens. AddsRemoteRustRenderServertotests/utils.py(skips cleanly when novllm-rsbinary is built).Duplicate-work check: no open PR implements streaming derender in
rust/; #50550 extends the Python streaming derender only. Related: #42729, #47161.AI assistance: implemented with AI assistance (Kimi Code CLI). I have reviewed the full diff and run the tests below myself.
Test Plan
Test Result
cargo nextest run: 847 passed, 1 skipped (includes all 21 streaming derender tests).cargo clippy --all-targets: clean.cargo fmt --check: clean.pre-commit runon the Python files: all hooks pass.deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5Bwith hermes/deepseek_r1 parsers): 4 passed, 1 skipped in ~95 s — the skip istest_derender_parity_tool_callgated on the model emitting a tool call (same gating astest_derender_parity.py).Model evals: N/A — adds endpoints to the Rust frontend without changing existing serving behavior or model outputs.
Notes for reviewers
nulls, which the Python/inference/v1/generatemsgspec validation rejects. The test strips them before the generate hop with a comment; a render-side fix can be a separate PR.VLLM_WSL2_ENABLE_PIN_MEMORY=1andVLLM_USE_FLASHINFER_SAMPLER=0(no CUDA toolkit); not test-code concerns.