[franciscojavierarceo]

Overview

Add the stateless Responses API (POST /v1/responses, streaming and non-streaming) to the Rust frontend on top of the existing ChatLlm pipeline.

The original stack split this endpoint into non-streaming (2/3, this PR) followed by streaming (#63903). The Rust frontend is streaming-first: ChatEventStream is the primary interface, and non-streaming Chat Completions is a fold over it. As agreed with @franciscojavierarceo, this PR now carries both layers in that order: streaming first, then non-streaming as a fold over the same state machine. It supersedes #63903. Layer 1/3 (#63901) has merged.

Commits

Each commit builds and passes its tests on its own.

1. da7eeffe37 (@franciscojavierarceo) — stateless streaming Responses: request lowering into ChatRequest, OutputItemStreamer for the item lifecycle, store-dependent features rejected or 404. Squashes the streaming-first part of the original stack (listed in the commit message); non-streaming returns 400 at this point.
2. 7e41889ed4 — typed wire DTOs and stream events: tools and tool choice reuse the Chat Completions definitions, input items deserialize into typed items (a message item may omit type, like Python), and SSE events are a typed enum with type and sequence_number added by a wrapper.
3. 2ea078d012 — defer to shared layers: the response echoes the sampling values the engine actually uses (model defaults applied, max_output_tokens clamped to the context), like Python's ResponsesResponse.from_request(sampling_params), via ChatEventStream::sampling().
4. 1c1bbe1fe5 — the streamer is the only source of output; output indices and the deferred output_item.done derive from completed items.
5. 10448da198 (co-authored by @franciscojavierarceo) — non-streaming as a fold: drive the same OutputItemStreamer, discard its events, and return the terminal response. The Responses terminal event carries the complete response, so output follows one set of item rules in both modes. Errors still map to HTTP statuses (engine errors 400/503, a broken stream 500) instead of a failed response.
6. 8fbae250f3 — per-request timing metrics, as in Python Responses.
7. 4da559a6db — a truncated reasoning item is also marked incomplete, as in Python.
8. f28f024629 — remaining vLLM request controls: forward min_p; reject spaces_between_special_tokens: false, media_io_kwargs, mm_processor_kwargs, and enable_response_messages: true instead of dropping them.
9. a1d8bd8364 — unset response fields serialize as null, as in Python.
10. 99446a2d75 — report usage.input_tokens_details.cache_write_tokens, as Python has since [Bugfix][Responses API] Complete cache-write token accounting #60973; the OpenAI SDK Response type requires it. Found by the e2e below.

The CI bot merged main into the branch as be0cafa775 before commit 10.

Behavior

• Stateless: clients replay the full history in input. store: true executes without storing (Python's implicit downgrade); background: true is 400; previous_response_id, retrieval, and cancellation are 404.
• max_tool_calls limits built-in tool calls only, so with function tools alone it is accepted and echoed.
• reasoning.summary, text.verbosity, include: ["reasoning.encrypted_content"], and service_tier are accepted compatibility inputs (Codex sends them by default) without producing summaries, applying verbosity, encrypting reasoning, or scheduling guarantees.
• Deliberate difference from Python: a generation failure during streaming ends with response.failed carrying the partial output, instead of a generic error frame.
• Not supported: built-in, custom, and namespace tools, allowed_tools, and output logprobs (rejected with 400).

Validation

Rebased onto eb239e34c2.

• Every commit: cargo fmt --check, cargo clippy --workspace --all-targets --all-features, and the server Responses tests pass (45 → 52 tests across the stack).
• Tip: cargo test --workspace has 2483 passed; the same 27 tests also fail on main in this environment (grammar replay fixtures and two environment-dependent tests).
• Route tests cover streaming event order, non-streaming fold, truncation (incomplete on the last item only), engine errors mapped to 400/503, 404s for stored-response features, and typed-client (async-openai) streaming.
• cache_write_tokens: a route assertion fails without the fix and passes with it.

E2E on GB200 at a1d8bd8364 (before commit 10), Qwen/Qwen3.5-27B-FP8 on one GPU, 64k context, --language-model-only, qwen3 reasoning and qwen3_coder tool parsers, Python engine from the matching main wheel:

• 12/12 HTTP probes directly against the Rust frontend, validating every SSE event and terminal response against the openai 3.42.1 types: JSON and streaming completion; echoed temperature/top_p equal the model's generation config when unset; a token limit marks the response and its last item incomplete; every streamed output_item.done equals the terminal output with matching IDs and contiguous sequence_numbers; function call and function_call_output replay in both modes; 404 for retrieval and previous_response_id. The only schema gap was the missing cache_write_tokens, fixed in commit 10.
• At temperature 0, /v1/responses and /v1/chat/completions produce identical prompt token counts, reasoning, and text for the same message.
• Real Codex CLI 0.163.0, Responses transport, function-only tool catalog, no agentic-api in between: 2/2 runs executed cat nonce.txt and replied with the exact nonce, ending in turn.completed. Each second request replayed two leading developer messages, the user turn, a plaintext reasoning item, the function_call, and its function_call_output; all requests returned 200 with response.completed.

This PR is 17 files, +4478 / -33; route tests are about 1.2k of that.

Original description (@franciscojavierarceo)

Overview

add stateless, non-streaming POST /v1/responses through the existing vllm-chat pipeline, with input replay, function tools, reasoning, and usage.

review order: 1/3: renderer #63901 → 2/3: non-streaming #63902 → 3/3: streaming #63903. all three upstream review PRs target main; later diffs include earlier layers.

this PR includes and depends on #63901. the fork layer shows its isolated diff.

Claims

• lower Responses input and function-call/result replay through the shared chat path, reusing compatible chat/audio/tool primitives.
• validate request controls and preserve model-resolved sampling defaults.
• use the current frontend's typed engine failures and numeric HTTP errors, with real route coverage for 400/503 failures and unknown-response 404s.
• reject stream: true with HTTP 400 on param: stream until layer 3.

Validation

Latest upstream synchronization — October 28

current head: d5b7df30aa; rebased onto upstream f4fd1ab68e. the fork keeps its native stacked bases; upstream PRs still target main.

• resolved the shared error-handler overlap with upstream's missing-LoRA 404 change. both NotFound and ResponseNotFound retain their existing wire envelopes; the LoRA/gRPC and Responses HTTP regressions pass.
• cargo test --locked -p vllm-server -p vllm-text --lib -- --test-threads=1: 579 server + 121 text passed, one existing text test ignored.
• strict Clippy for server/chat/text/CLI, applicable pre-commit hooks, and git diff --check: passed. cargo nextest remains unavailable.
• range-diff shows only the error-handler conflict adaptation; all other feature/fix patches are unchanged. all branch tips passed merge-tree checks against f4fd1ab68e.
• the old umbrella [Rust Frontend] Add OpenAI Responses API endpoint #58783 was synchronized without rewriting its original history. its source tree equals the tested complete stack and it stays draft.

the model-backed HTTP/Codex evidence below predates this synchronization (complete-stack head 3db4c9abf0); it was not repeated after this rebase. today's evidence covers compilation and deterministic wire/route/renderer regressions, not new model qualification. CI reruns on these new heads are separate from the earlier successful workflows. agentic-api was not changed or pushed.

Earlier review and runtime evidence

reviewed and revised on October 27; non-streaming head: 91a782aa08.

• a regression first reproduced the bug: token-limited responses had overall incomplete status but a final message item marked completed.
• this layer's cargo test --locked -p vllm-server -p vllm-text --lib -- --test-threads=1: 552 server + 118 text passed, one existing text test ignored.
• added a route regression for reasoning followed by a truncated function call: earlier output stays completed; only the final message/function item becomes incomplete.
• strict Clippy, applicable pre-commit hooks, formatting, and git diff --check: passed.
• all three tips merge cleanly with upstream f8f7634cb4; the fork's isolated stack bases are preserved.

commands ran from rust/ with OPENSSL_DIR pointing to the local OpenSSL build. cargo nextest was unavailable. full-stack tests ran serially because of the previously reproduced shared chat audio-metrics collector collision.

fresh model-backed HTTP and real Codex checks passed on the revised complete stack, as recorded in #63903. this intermediate branch was tested separately with the in-process HTTP route harness; it does not contain streaming.

Details

Responses owns its wire envelopes and conversions while reusing the chat generation path and matching shared audio/tool primitives. the rebase preserves upstream's other route registrations and error serialization rather than restoring the older envelope.

• omitted or null stream selects non-streaming. omitted/null watermarking and explicit overrides follow current Chat Completions semantics.
• metadata is a string map limited to 16 pairs, 64-character keys, and 512-character values.
• omitted temperature/top-p use model defaults; explicit overrides are preserved and checked against the engine request.
• unknown service tiers, invalid summary/verbosity values, unsupported include outputs, and nonzero top_logprobs are rejected.
• unknown previous_response_id uses the current shared NotFoundError envelope: HTTP 404, param: response_id, numeric code: 404.

the frontend remains stateless: clients replay history, and agentic-api owns persistence and tool execution. the shared native adapter tracked by vllm-project/agentic-api#359 and vllm-project/agentic-api#382 is still unfinished. accepted reasoning-summary, text-verbosity, and encrypted-reasoning inputs are documented compatibility no-ops; valid service tiers are echoed without scheduling guarantees. persistence, retrieval/cancellation, built-in/custom/namespace tool execution, Harmony rendering, output logprobs, and allowed_tools are out of scope. the rendered Responses documentation describes the supported subset.

these are the existing #58783 layers opened upstream at @BugenZhao's request, rather than competing implementations. the original authored history and discussion remain in #58783. @gcanlin's #53380 implementation is credited in the endpoint and streaming commits. #56124 covers render endpoints and #51535 covers chat allowed_tools; neither duplicates this endpoint.

this layer is 13 files, +2987 / -1 relative to #63901. tests stay under the test-only route module. agentic-api compatibility follow-ups remain local and unpushed. Codex assistance was used; human review of every changed line and independent validation are required before merge.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @franciscojavierarceo.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

Documentation preview: https://vllm--63902.org.readthedocs.build/en/63902/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @franciscojavierarceo.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[BugenZhao]

Thanks, @franciscojavierarceo! I think the overall shape looks good. However, I’d prefer to support streaming first and then implement non-streaming on top of it. In the Rust frontend, all the interfaces have been streaming-first from the start. We’d rather treat non-streaming as a degenerate case of streaming instead of having the two follow different paths.

Would you mind if I merged PR 3 into this one so I can rework non-streaming to use the streaming path, rather than the other way around?

[franciscojavierarceo]

Thanks, @franciscojavierarceo! I think the overall shape looks good. However, I’d prefer to support streaming first and then implement non-streaming on top of it. In the Rust frontend, all the interfaces have been streaming-first from the start. We’d rather treat non-streaming as a degenerate case of streaming instead of having the two follow different paths.

Would you mind if I merged PR 3 into this one so I can rework non-streaming to use the streaming path, rather than the other way around?

yeah sgtm

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @franciscojavierarceo.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[BugenZhao]

I pushed several commits to make the DTOs more strongly typed and to share as much code as possible between the non-streaming and streaming paths.

Thanks again @franciscojavierarceo!

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99513 for commit be0cafa77575.

ℹ️ This PR is 5 commits behind upstream main. Merged upstream main into this branch as be0cafa77575 before starting CI. Pull the branch before pushing more changes.

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99516 for commit 99446a2d751b.