Skip to content

feat(api_server): stream model reasoning as reasoning.delta on /v1/runs/events - #61259

Open
studying-end wants to merge 1 commit into
NousResearch:mainfrom
studying-end:feat/api-server-reasoning-delta
Open

feat(api_server): stream model reasoning as reasoning.delta on /v1/runs/events#61259
studying-end wants to merge 1 commit into
NousResearch:mainfrom
studying-end:feat/api-server-reasoning-delta

Conversation

@studying-end

Copy link
Copy Markdown

What does this PR do?

The HTTP API Server (/v1/runs) never wired reasoning_callback, so the model's real reasoning — DeepSeek reasoning_content, codex response.reasoning.*.delta, Bedrock thinking, inline <think> — was captured internally (on_reasoning_delta_fire_reasoning_delta) but dropped on the API Server codepath. As a result /v1/runs/{id}/events only ever streamed the answer (message.delta), and the one-shot reasoning.available event carried an answer-derived snippet (first 500 chars of assistant_message.content), not real reasoning.

The CLI/TUI already display real reasoning (they set reasoning_callback). This PR brings the API Server to parity: wire reasoning_callback through _create_agent and emit a reasoning.delta SSE event from the /v1/runs handler (mirrors the existing pattern in tui_gateway/server.py). Clients now receive the real reasoning stream ahead of the answer:

reasoning.delta × N     ← real reasoning, token stream (NEW)
message.delta × N       ← answer
reasoning.available     ← unchanged (answer-derived snippet; clients may ignore)
run.completed

Backward compatible: reasoning.available is unchanged; clients that don't handle reasoning.delta are unaffected. Only /v1/runs is changed — /v1/chat/completions and /v1/responses are untouched.

Related Issue

N/A (no existing issue; happy to open one if preferred).

Type of Change

  • ✨ New feature (non-breaking change that adds functionality)

Changes Made

  • gateway/platforms/api_server.py_create_agent now accepts and forwards reasoning_callback; the /v1/runs handler (_run_and_close) wires a _reasoning_cb that pushes a reasoning.delta SSE event (payload field delta, same shape as message.delta).
  • tests/gateway/test_api_server_runs.py — added TestRunEvents::test_events_stream_emits_reasoning_delta: asserts reasoning_callback is wired through _create_agent and that its output reaches the SSE stream as reasoning.delta.

Additive only: +60 lines across 2 files, no change to existing events.

How to Test

  1. Unit: uv run --extra dev pytest tests/gateway/test_api_server_runs.py -q → all pass, including the new test_events_stream_emits_reasoning_delta.
  2. Live: run the API server with a reasoning model (e.g. DeepSeek reasoner, reasoning_effort != none), POST /v1/runs, then GET /v1/runs/{id}/events → confirm reasoning.delta events arrive ahead of message.delta and contain real reasoning (distinct from the final answer).

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commitsfeat(api_server): stream model reasoning as reasoning.delta on /v1/runs/events
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass — ran the affected file tests/gateway/test_api_server_runs.py (24 passed); the full tests/ suite was not run on this base
  • I've added tests for my changes — test_events_stream_emits_reasoning_delta
  • I've tested on my platform: Windows 11 — unit tests pass; live SSE verified end-to-end against a DeepSeek reasoning model via an API Server client

Documentation & Housekeeping

  • I've updated relevant documentation — or N/A (no docs surface changed; the new reasoning.delta event mirrors tui_gateway's existing one)
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A (no new config keys)
  • I've updated CONTRIBUTING.md or AGENTS.md — or N/A
  • I've considered cross-platform impact (Windows, macOS) — or N/A (pure Python, no platform-specific code)
  • I've updated tool descriptions/schemas — or N/A

Screenshots / Logs

/v1/runs/{id}/events with a reasoning model — before this PR only message.delta + reasoning.available were emitted; after, reasoning.delta streams first:

event: reasoning_delta
data: {"event":"reasoning.delta","run_id":"...","delta":"<real reasoning chunk>"}
... (more reasoning.delta) ...
event: message_delta
data: {"event":"message.delta","run_id":"...","delta":"<answer chunk>"}
...
event: run_completed

…ns/events

The HTTP API Server (/v1/runs) never wired reasoning_callback, so the model's real reasoning (DeepSeek reasoning_content, codex reasoning, Bedrock thinking, inline <think>) was captured internally but dropped on this codepath. Wire reasoning_callback through _create_agent and emit a reasoning.delta SSE event from the /v1/runs handler (mirrors what tui_gateway already does). Clients now receive the real reasoning stream ahead of the answer.

Changes: gateway/platforms/api_server.py (_create_agent accepts/forwards reasoning_callback; /v1/runs _run_and_close adds _reasoning_cb pushing reasoning.delta, payload field 'delta', same shape as message.delta); tests/gateway/test_api_server_runs.py (TestRunEvents::test_events_stream_emits_reasoning_delta).
@alt-glitch alt-glitch added type/feature New feature or request comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have duplicate This issue or pull request already exists labels Jul 9, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #15169 (earliest open PR for this fix). Both wire reasoning_callback through api_server.py::_create_agent and add the same _reasoning_cb pushing reasoning.delta SSE events via loop.call_soon_threadsafe(q.put_nowait, ...) on the /v1/runs handler — same code-site, same event shape, same endpoint. #55163 is another open sibling of the same cluster. Marking this a duplicate of the canonical #15169; a maintainer should pick one of the cluster to merge.

@teknium1

Copy link
Copy Markdown
Contributor

The implementation addresses a verified current-main gap: gateway/platforms/api_server.py:1234-1363 does not thread reasoning_callback into AIAgent, and /v1/runs only wires message.delta at gateway/platforms/api_server.py:4263-4298. The proposed callback matches the existing agent mechanism (run_agent.py:4733-4740).

Problems

  • The new reasoning.delta is a public SSE event, but the Runs API documentation only describes /v1/runs/{run_id}/events generically at website/docs/user-guide/features/api-server.md:267-269; clients have no documented payload contract.

Suggested changes

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 10, 2026
@teknium1 teknium1 added the area/streaming Streaming responses: gateway delivery, provider wire label Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/streaming Streaming responses: gateway delivery, provider wire comp/gateway Gateway runner, session dispatch, delivery duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants