[Feat] sgl-router: Serve the OpenAI Responses API on /v1/responses by whybeyoung · Pull Request #42314 · sgl-project/sglang

whybeyoung · 2026-10-03T01:34:31Z

Motivation

Clients built on the OpenAI Responses API (the openai SDK's responses.*, Codex) cannot use sgl-router today: it only serves chat completions. This adds POST /v1/responses to the router.

Modifications

A Responses request is converted to a chat request at ingress and served by the existing chat_completions handler, so it is routed exactly like chat (tokenization, cache-aware routing, admission, retry, PD). The reply is converted back on the outer body, streaming or not.

Request: string or item input; messages (developer → system), reasoning, function calls and outputs; text and image parts; instructions, max_output_tokens, reasoning.effort, text.format (JSON schema), tool_choice. Conversion works on serde_json::Value, so sglang chat extensions (top_k, chat_template_kwargs, …) pass through.
Tools: function, plus the shapes Codex sends: namespace groups (flattened to <ns>__<name>) and custom freeform tools (a function with one input string, grammar appended to the description). Replies map back to namespaced function_call / custom_tool_call items, and history items of both kinds are accepted.
Streaming: the spec event sequence with a gapless sequence_number (response.created → in_progress → item / part / delta / done events → completed, incomplete or failed); usage, including cached_tokens, is always present. An engine abort ends the Response as failed.
Unsupported, rejected with 400: previous_response_id, conversation, background and stored prompts (the router is stateless; send the full conversation), and top_logprobs. reasoning.effort is passed through to the engine unchanged.
Errors: engine errors are re-wrapped into the {"error": {...}} envelope; an upstream reply that cannot be read or parsed is a 502 via ApiError::UpstreamStatus.
src/protocol/sse.rs is a small generic layer that rewrites a chat SSE body into another protocol's events (SseTransducer, transduce_body). It wraps the body produced by proxy/sse.rs, so timeouts, abort-on-disconnect and TTFT/ITL metrics are unchanged.

Built-in tools such as web_search are rejected with 400.

The companion PR for Anthropic /v1/messages (#42316) carries a byte-identical protocol/sse.rs; whichever lands second only needs a one-line rebase in protocol/mod.rs.

Accuracy Tests

N/A: no model or kernel change.

Unit tests cover conversion, buffered and streamed replies, and the Codex tool shapes; tests/proxy/responses_api.rs drives requests end to end through a mock worker. cargo test (lib + proxy) and cargo clippy --all-targets -D warnings pass.

Also validated against a live deployment: the official openai Python SDK's types (strict validation of every Response object and stream event), and Codex-style namespace / custom tool calls, streamed and buffered, including history replay.

Speed Tests and Profiling

N/A: no change to the chat path.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

🤖 Generated with Claude Code

CI States

Latest PR Test (Base): ✅ Run #40508465628
Latest PR Test (Extra): ❌ Run #40508465466
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

A Responses request is converted to a chat request at ingress and served by `chat_completions`, so it is routed exactly like chat: tokenization, cache-aware routing, admission, retry and PD all apply unchanged. The reply is converted back on the outer body. - Conversion works on `serde_json::Value`, so sglang chat extensions (`top_k`, `chat_template_kwargs`, ...) pass through. - Input items: messages (`developer` becomes `system`), reasoning, function calls and outputs; text and image parts only. - Tools: `function`, plus Codex's `namespace` groups (flattened) and `custom` freeform tools (a function with one `input` string). Replies map back to namespaced `function_call` / `custom_tool_call` items. - Streaming emits the spec event sequence with a gapless `sequence_number`, ending in `response.completed`, `.incomplete` or `.failed`; usage is always present. - Stateless: `previous_response_id`, `conversation`, `background` and stored prompts are rejected with 400. Engine errors are re-wrapped into `{"error": {...}}`. The SSE conversion layer (`protocol::SseTransducer`, `transduce_body`) is generic so other protocols can reuse it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- An engine abort (`finish_reason: "abort"`) ends the Response as `failed`, not `completed`. - A successful upstream reply that cannot be read or parsed answers 502 via `ApiError::UpstreamStatus`, not 200 with an error body. - `reasoning.effort` is passed through: the engine accepts every tier. - Reject `top_logprobs` (not translated) and tools whose chat names collide. - Move the SSE layer to `protocol/sse.rs`; share the status → OpenAI error type mapping with `ApiError`; narrow visibility. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sherlockwu · 2026-10-05T17:28:52Z

+                args,
+                ..
+            } => {
+                let item = call_item(&id, &call_id, &name, &args, "completed", &self.echo.tools);


[P2] Tool calls are always marked completed, even when the output was cut off (length) or aborted, so a truncated call looks complete. Use finish.item_status() here and in response.rs:66?

sherlockwu · 2026-10-05T17:28:52Z

+    fn tool_delta(&mut self, call: &Value, out: &mut Vec<u8>) {
+        let tool_index = call.get("index").and_then(Value::as_u64).unwrap_or(0);
+        let same =
+            matches!(&self.open, Some(Open::Function { tool_index: t, .. }) if *t == tool_index);


[P2] Only one call is tracked, matched by index, which breaks two ways:

Interleaved indices (0, 1, 0, 1) become four calls, the later two with made-up IDs, empty names and partial arguments.

SGLang's Gemma 4 parser reuses one index for repeated calls to the same tool, so two separate calls merge into one.

Track calls by id, and attach chunks without an id to the call with that index?

sherlockwu · 2026-10-05T17:28:52Z

+                Ok(json!({"type": "function", "function": {"name": name}}))
+            }
+            // Keep the mode, drop the allow-list.
+            Some("allowed_tools") => Ok(o


[P2] Dropping the allow-list lets the model call tools the client excluded. Filter tools to the list, or reject?

sherlockwu · 2026-10-05T17:28:52Z

+        self.output.push(item);
+    }
+
+    fn emit_failed(&mut self, message: &str, out: &mut Vec<u8>) {


[P2] The open item is dropped on failure: after streaming "partial answer", response.failed has output: [] and the item never gets output_item.done. Close it as incomplete first?

sherlockwu · 2026-10-05T17:28:53Z

+fn merge_leading_system(messages: &mut Vec<Value>) {
+    let n = messages
+        .iter()
+        .take_while(|m| m["role"] == "system" && m["content"].is_string())


Only string system messages are merged. instructions plus a developer item with two input_text parts gives two leading system messages, which Qwen3.5 rejects. Flatten text-only parts before merging?

sherlockwu · 2026-10-05T17:28:53Z

+                    "user" | "system" | "developer" => {
+                        pending.flush(messages);
+                        // Chat templates know `system`; some drop `developer` messages.
+                        let role = if role == "developer" { "system" } else { role };


A developer item after the first turn becomes a system message mid-conversation, which some templates reject (e.g. Qwen3.5). Moving it to the top would reorder the history, so maybe send it as user instead?

sherlockwu · 2026-10-05T17:28:53Z

+                        }
+                        let text = content_text(content, i)?;
+                        match &mut pending.content {
+                            Some(c) => c.push_str(&text),


nit: adjacent assistant messages are joined with no separator ("A." + "B." → "A.B."). Same for reasoning below.

sherlockwu · 2026-10-05T17:28:53Z

+                }
+                json!({"type": "image_url", "image_url": image_url})
+            }
+            "image_url" => part.clone(),


nit: input is borrowed, so image parts (often large base64) are copied here and at L444. Taking input by value would avoid the copy.

Tool-call items were keyed by the chunk's `index`, so a reused index (Gemma 4 repeats it per tool name) merged two calls into one item, and text arriving mid-call closed the item and split the call in two. A chunk with a function name now starts a new item; argument-only chunks extend the open one. Reasoning or text arriving while a call is open is held and emitted after it (whitespace alone is dropped). A string `error` in the stream keeps its message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

main now carries the Messages API (#42316) with the shared SSE layer. Conflicts: protocol/mod.rs declares both protocols; protocol/sse.rs takes main's version (`data_payload` returns `Data`), and the Responses stream reads `Data::Json` only for now; app.rs and tests/proxy/main.rs keep both sides. WorkerSpec gained `services`; the Responses test fills the rest with `..Default::default()`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Stream: several function calls stay open at once. A chunk with a name starts a new call; others attach by `id`, else to the latest call with their `index`, so interleaved indices (0, 1, 0, 1) stay apart and a reused index (Gemma 4) or index-derived id (Kimi) starts a new call. - A call cut off by `length` or an abort is `incomplete`, streamed and buffered; only the last call can be cut off. - On failure, open items are closed as `incomplete` and kept in `response.failed`. `[DONE]` completes the response; a non-JSON line fails it. - `allowed_tools` restricts `tools` to its list; an unknown name is 400. - Leading system / developer messages with text-only parts are merged. - A `developer` item after the first turn is sent as `user`, since some templates reject a mid-conversation system message. - Adjacent assistant and reasoning items are joined with newlines. - `input` is taken by value, so image parts are moved, not copied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

whybeyoung · 2026-10-15T03:28:13Z

/tag-and-rerun-ci

…l helpers - A turn whose calls have no outputs yet stays open: the stream emits text or reasoning that arrived during a call after the call, and sent back as input it no longer splits into two assistant turns. - Error paths keep the upstream reply's parts (access-log context) with a specific 502; every non-2xx body gets the OpenAI envelope, and 5xx or in-stream engine text is scrubbed as on /v1/messages. - Shared by both protocols: `new_id`, `scrub_message`, `openai_error_type` and `upstream_error` in `protocol`, and the reply helpers (`is_sse`, `rebuild`, `MAX_REPLY_BYTES`) in `routes::reply`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

main moved the router to the top-level sgl-router/ (#45414); the Responses files move with it (protocol/responses/*, routes/responses.rs, routes/reply.rs, tests/proxy/responses_api.rs). Conflicts: server/error.rs takes main's restructured IntoResponse (client_message, gRPC status) and keeps only the shared `protocol::openai_error_type` for the type mapping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

whybeyoung · 2026-10-28T09:02:28Z

/tag-and-rerun-ci

whybeyoung · 2026-10-28T09:05:15Z

/tag-and-rerun-ci

whybeyoung and others added 2 commits October 3, 2026 09:29

This was referenced Oct 3, 2026

[sgl-router] Serve the Anthropic Messages API on /v1/messages #42316

Merged

[Feat] sgl-router: Add API profiles, a vendor API contract as one YAML file #42345

Open

sherlockwu requested changes Oct 5, 2026

View reviewed changes

whybeyoung and others added 3 commits October 6, 2026 10:06

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 15, 2026

whybeyoung and others added 2 commits October 15, 2026 15:20

whybeyoung requested review from Kangyan-Zhou, ShangmingCai and hzh0425 as code owners October 27, 2026 13:01

This comment has been minimized.

Sign in to view

whybeyoung changed the title ~~[sgl-router] Serve the OpenAI Responses API on /v1/responses~~ [Feat] sgl-router: Serve the OpenAI Responses API on /v1/responses Oct 28, 2026

Navigation Menu

Repository navigation

[Feat] sgl-router: Serve the OpenAI Responses API on /v1/responses - #42314

[Feat] sgl-router: Serve the OpenAI Responses API on /v1/responses#42314
whybeyoung wants to merge 7 commits into
mainfrom
feat/router-responses-api

whybeyoung commented Oct 3, 2026 •

edited by github-actions Bot

Loading

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

sherlockwu Oct 5, 2026

Uh oh!

whybeyoung commented Oct 15, 2026

Uh oh!

This comment has been minimized.

whybeyoung commented Oct 28, 2026

Uh oh!

whybeyoung commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

whybeyoung commented Oct 3, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

CI States

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whybeyoung commented Oct 15, 2026

Uh oh!

This comment has been minimized.

whybeyoung commented Oct 28, 2026

Uh oh!

whybeyoung commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

whybeyoung commented Oct 3, 2026 •

edited by github-actions Bot

Loading