feat(api-server): stream model reasoning as reasoning_content in chat completions - #52606
Conversation
… completions
The /v1/chat/completions streaming path wired stream_delta, tool_start, and
tool_complete callbacks but never reasoning_callback, so a model's reasoning
(e.g. Claude extended thinking) was dropped on the floor for OpenAI-compatible
frontends. Open WebUI renders a collapsible "thinking" block from
delta.reasoning_content chunks, but nothing emitted them.
Thread reasoning_callback through _create_agent and _run_agent, add an
_on_reasoning callback in the streaming branch that enqueues a tagged
("__reasoning__", text) item, and route that item in _emit to a
chat.completion.chunk carrying delta.reasoning_content. Visible content,
tool-progress events, and the end-of-stream sentinel are unchanged; empty
reasoning chunks are skipped.
Co-authored-by: ladyada <limor@ladyada.net>
|
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the focused Chat Completions reasoning-streaming fix. The premise is confirmed on current main: _handle_chat_completions forwards only text and tool callbacks at gateway/platforms/api_server.py:2311-2321, while _write_sse_chat_completion only special-cases tool-progress tuples at gateway/platforms/api_server.py:2499-2510. The agent-side callback is live: Anthropic thinking_delta reaches _fire_reasoning_delta in agent/chat_completion_helpers.py:2675-2679.
Problems
- This PR changes only
gateway/platforms/api_server.py;gh pr diff 52606contains no regression test. Add coverage intests/gateway/test_api_server.pyfor callback forwarding and the emitteddelta.reasoning_contentchunk. - The supplied discussion points to open #57094 as the broader active implementation, covering this path plus non-streaming Chat and Responses streaming. This PR remains a valid narrow fix, but consolidation should avoid landing overlapping callback plumbing independently.
Suggested changes
- Mock
_run_agent, invoke its receivedreasoning_callback, and assert reasoning and visible text serialize into their respective delta fields. - Add a forwarding-seam test through
_run_agentand_create_agent.
This is an automated hermes-sweeper review.
| @@ -2007,6 +2020,7 @@ def _on_tool_complete(tool_call_id, function_name, function_args, function_resul | |||
| stream_delta_callback=_on_delta, | |||
There was a problem hiding this comment.
Please add an API-server SSE regression test that invokes this callback through the mocked _run_agent call and verifies it serializes as delta.reasoning_content rather than delta.content; the current PR changes no test files.
|
Thanks for the review. #57094 (disclosure: my PR) already has tests covering both streaming delta.reasoning_content and non-streaming reasoning_content in the response body, plus Responses API support. It's been rebased onto latest main and all 198 tests pass. Happy to close this in favor of #57094 if that helps consolidate. One ask: could the duplicate label be removed from #57094 so it gets a proper review? #13401 was already closed by its author in favor of #57094. |
Summary
The
/v1/chat/completionsstreaming path wiredstream_delta_callback,tool_start_callback, andtool_complete_callback, but neverreasoning_callback. As a result, a model's reasoning stream (e.g. Claude extended thinking,thinking_delta→agent._fire_reasoning_delta) was silently dropped for OpenAI-compatible frontends.Open WebUI (and other OpenAI-compatible UIs) render a collapsible "thinking" block from
delta.reasoning_contentchunks — but nothing on this path emitted them.Change
Thread
reasoning_callbackthrough the existing call chain and route reasoning deltas to a dedicated SSE chunk:_create_agentand_run_agentgain areasoning_callbackparameter, forwarded toAIAgent(which already accepts and fires it via_fire_reasoning_delta)._handle_chat_completions, a new_on_reasoning(text)enqueues a tagged("__reasoning__", text)item (mirroring the existing("__tool_progress__", …)pattern)._emitgains anelifthat renders that tagged item as achat.completion.chunkcarryingdelta.reasoning_content.Visible
delta.content, thehermes.tool.progressevents, and the end-of-stream sentinel are all unchanged. Empty/None reasoning chunks are skipped.Real behavior / verification
_create_agentand_run_agentsignatures carryreasoning_callback(verified viainspect.signature).thinking_delta(chat_completion_helpers.py) →agent._fire_reasoning_delta→self.reasoning_callback(assigned inagent_init.py) →_on_reasoning→_stream_q→_emit._emitbranch confirms a tagged reasoning item serializes to{"delta": {"reasoning_content": "..."}}while normal content still serializes to{"delta": {"content": "..."}}.Scope / safety