Skip to content

fix(gateway): stream real reasoning_content instead of answer echo - #75562

Open
huanshan5195 wants to merge 3 commits into
NousResearch:mainfrom
huanshan5195:fix/reasoning-stream-echo
Open

fix(gateway): stream real reasoning_content instead of answer echo#75562
huanshan5195 wants to merge 3 commits into
NousResearch:mainfrom
huanshan5195:fix/reasoning-stream-echo

Conversation

@huanshan5195

@huanshan5195 huanshan5195 commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What & Why

When a thinking model (GLM-5.2, DeepSeek-V4, Kimi, Qwen) answers through the HTTP gateway, the reasoning panel shows the model's answer text (truncated to 500 chars) instead of its real chain-of-thought. Two coupled bugs cause this; this PR fixes the parts that #15169 does not cover.

Bug 1 — this PR (root cause of #60634)

conversation_loop.py emits reasoning.available from assistant_message.content[:500] for all agents, including top-level ones. That fallback was designed for subagent delegation (a child's content IS the parent's reasoning surface), but a missing _delegate_depth > 0 guard lets top-level agents publish their own answer as a fake "thinking" block.

This is the bug users reported in #60634 ("reasoning text is identical to the assistant answer").

Bug 2 — covered by #15169

_create_agent / _run_agent never forward reasoning_callback to AIAgent, so _fire_reasoning_delta fires into a no-op and real reasoning deltas are silently dropped. #15169 fixes this for the /v1/runs SSE path.

Scope — complementary to #15169

Fix This PR #15169
conversation_loop.py answer-echo guard ✅ ❌
/v1/chat/completions delta.reasoning_content ✅ ❌
_create_agent / _run_agent reasoning_callback plumbing ✅ (shared foundation) ✅
/v1/runs reasoning.delta SSE events included for end-to-end testability ✅ (primary)
Regression tests ✅ (6 tests, 374 lines) ❌

The /v1/runs _reasoning_cb → reasoning.delta wiring is included here so the full callback chain is testable end-to-end, but #15169 is the primary vehicle for that path. If maintainers merge #15169 first, this PR can rebase onto it and drop the /v1/runs block — only conversation_loop.py + /v1/chat/completions + tests remain.

Under #15169 alone, a top-level agent would still publish its answer as a fake thinking block (Bug 1 is untouched). That's the gap this PR closes.

Changes

agent/conversation_loop.py (+12/-12)

Added _delegate_depth > 0 guard so only subagents relay their content as reasoning. Top-level agents no longer emit reasoning.available from assistant_message.content. Subagent delegation behavior (_thinking first-line relay) is preserved unchanged.

gateway/platforms/api_server.py (+46/-1)

Non-thinking models never fire reasoning_callback, so no spurious reasoning block appears.

tests/gateway/test_api_server_reasoning.py (new, 374 lines, 6 tests)

  • _create_agent forwards reasoning_callback to AIAgent (defaults to None for back-compat).
  • Multi-chunk reasoning.delta ordering preserved before run.completed; None/empty payloads suppressed.
  • reasoning_callback errors do not break the run stream.
  • /v1/chat/completions streams delta.reasoning_content separated from delta.content, reasoning preceding content (the answer-echo regression).
  • Non-thinking models emit no reasoning_content.

Relationship to other PRs / issues

Happy to rebase onto #15169 or merge into a single vehicle — maintainers' call.

How to test

  1. Configure a thinking model (e.g. GLM-5.2) and send a turn through /v1/chat/completions (stream=true). A client reading delta.reasoning_content should see real reasoning, distinct from delta.content.
  2. Switch to a non-thinking model: no reasoning_content chunks should appear.
  3. pytest tests/gateway/test_api_server_reasoning.py — 6 passed. tests/gateway/test_api_server_runs.py — 16 passed (no regression).

Platforms tested

  • Linux (WSL2) gateway against a GLM-5.2 thinking endpoint.
  • pytest: 6 new + 16 existing tests green.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery area/streaming Streaming responses: gateway delivery, provider wire needs-decision Awaiting maintainer decision before any implementation sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 31, 2026
@alt-glitch

alt-glitch commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #60634, #15169, and #48024. This broader patch fixes the top-level content echo and streams real reasoning through both Runs and chat-completions; its Runs event contract should be reconciled with the existing proposals.

teknium1
teknium1 reviewed Jul 31, 2026
View reviewed changes

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing both the fake reasoning fallback and the dropped callback path. The current-main premise is valid: agent/conversation_loop.py:5529-5545 publishes answer content as reasoning.available, while gateway/platforms/api_server.py:2653-2656 does not pass reasoning_callback into AIAgent.

Problems

  • gateway/platforms/api_server.py:6245 emits every callback chunk as reasoning.available. That is fallback/snapshot semantics: ui-tui/src/app/turnController.ts:715-723 accepts only the first available text, whereas :766-785 appends reasoning.delta chunks. The incremental Runs contract should use reasoning.delta, as related PR #15169 does.
  • The diff adds no regression tests. In particular, it does not prove multi-chunk Runs ordering, Chat delta.reasoning_content separation, or the _run_agent → _create_agent → AIAgent callback forwarding chain.

Suggested changes

  • Emit reasoning.delta for Runs and add an endpoint test covering multiple chunks before run.completed.
  • Add focused forwarding and Chat SSE serialization regression tests in tests/gateway/.

This is an automated hermes-sweeper review.

Comment thread gateway/platforms/api_server.py Outdated
return
try:
loop.call_soon_threadsafe(_put_event_if_active, {
"event": "reasoning.available",

@teknium1 teknium1 Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reasoning_callback is invoked per provider chunk, but reasoning.available is the one-shot fallback event: the existing UI keeps only its first nonempty value. Emit reasoning.delta here so the Runs stream preserves every real reasoning chunk, matching #15169's contract.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 31, 2026
@huanshan5195

huanshan5195 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review - both points addressed in 92dfdd8.

1. Runs event type (api_server.py:6245)
_reasoning_cb now emits reasoning.delta instead of reasoning.available, so every real reasoning chunk reaches the client in order rather than being collapsed by the one-shot snapshot semantics the TUI applies to reasoning.available (turnController.ts:715-723). The legacy reasoning.available snapshot path (event_cb at api_server.py:6047, and the chat tool_progress path at :3557) is kept intact for post-run/legacy consumers - only the live callback wiring switches to reasoning.delta, matching the #15169 contract.

2. Regression tests
Added tests/gateway/test_api_server_reasoning.py:

  • _create_agent forwards reasoning_callback to AIAgent (default None) - the _run_agent -> _create_agent -> AIAgent chain.
  • Multi-chunk reasoning.delta ordering preserved before run.completed over /v1/runs; None/empty suppressed.
  • reasoning_callback errors don't break the run stream.
  • /v1/chat/completions delta.reasoning_content separated from delta.content, reasoning preceding content.
  • Non-thinking models emit no reasoning_content.

6 passed; test_api_server_runs.py (16) still green.

On overlap with #15169: this PR now aligns with its reasoning.delta contract on /v1/runs, and additionally fixes the conversation_loop.py top-level content echo and adds /v1/chat/completions delta.reasoning_content separation (both out of scope for #15169). Happy to reconcile/merge into a single vehicle if preferred - those two pieces are the additive scope here.

@GottZ GottZ mentioned this pull request Aug 3, 2026
Draft
@huanshan5195
huanshan5195 force-pushed the fix/reasoning-stream-echo branch from 92dfdd8 to ba28884 Compare August 21, 2026 19:48
huanshan5195 and others added 2 commits August 22, 2026 03:49
@huanshan5195 Easy PI
fix(gateway): stream real reasoning_content instead of answer echo
cdd1d4f 
The /v1/runs and /v1/chat/completions SSE paths showed the model's answer content (truncated to 500 chars) as the "thinking" block, so the reasoning panel was an exact echo of the response. Two coupled fixes:

1. agent/conversation_loop.py: top-level agents no longer emit reasoning.available from assistant_message.content. That fallback was meant for subagent delegation (a child's content IS the parent's reasoning surface) but was also firing for top-level agents, duplicating the answer as a fake chain-of-thought.

2. gateway/platforms/api_server.py: wire the agent's reasoning_callback (already fired by _fire_reasoning_delta from reasoning_content/reasoning deltas of GLM/DeepSeek/Kimi/Qwen thinking models) through to the SSE transports: /v1/runs emits reasoning.available events with the real thinking text; /v1/chat/completions emits delta.reasoning_content chunks. Non-thinking models never fire the callback, so no spurious reasoning block is shown.

Tested with GLM-5.2 via /v1/runs: reasoning panel now shows the model's chain-of-thought, distinct from the answer; non-thinking models show no reasoning block. Python syntax verified (py_compile).
@huanshan5195 Easy PI
test(gateway): assert reasoning.delta contract + chat reasoning_conte…
8e24a55 
…nt separation

Address review on NousResearch#75562:

- /v1/runs: _reasoning_cb now emits `reasoning.delta` (incremental) instead
  of `reasoning.available` (one-shot snapshot). The snapshot path
  (event_cb `reasoning.available` at api_server.py:6047, and the chat
  tool_progress `reasoning.available` at :3557) is intentionally preserved
  for post-run/legacy consumers; only the live callback wiring switches to
  `reasoning.delta`, matching the NousResearch#15169 contract and
  ui-tui/src/app/turnController.ts:766-785 delta-appending behavior. Without
  this, turnController.ts:715-723 kept only the first non-empty
  `reasoning.available` value and dropped every subsequent real chunk.

- Add tests/gateway/test_api_server_reasoning.py covering:
  * _create_agent forwards reasoning_callback through to AIAgent (and
    defaults to None for back-compat) - the _run_agent -> _create_agent ->
    AIAgent forwarding chain.
  * Multi-chunk reasoning.delta ordering is preserved before run.completed
    over /v1/runs; None/empty payloads are suppressed.
  * reasoning_callback errors do not break the run stream.
  * /v1/chat/completions streams delta.reasoning_content separated from
    delta.content, with reasoning preceding content (the answer-echo
    regression this PR fixes).
  * Non-thinking models (reasoning_callback wired but never fired) emit no
    reasoning_content, so no spurious reasoning block is shown.
@huanshan5195
huanshan5195 force-pushed the fix/reasoning-stream-echo branch from ba28884 to 8e24a55 Compare August 21, 2026 19:49
@huanshan5195

huanshan5195 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Updated for the review: branch rebased onto current main (0 behind, conflict resolved — the stream queue is now ThreadSafeAsyncQueue, and the reasoning callback uses put_threadsafe to match).

On the two points:

  1. Runs path reasoning.delta — since the review, main itself introduced structured /v1/runs run events (cc2b56b26), which emit per-chunk reasoning.delta (current api_server.py:7656); the old reasoning.available fallback remains only for the non-stream path. So the Runs contract concern is already satisfied upstream. This PR's remaining scope is the Chat SSE path: forwarding reasoning_callback through _run_agent → AIAgent and emitting real delta.reasoning_content chunks instead of the answer-echo, with multi-chunk ordering preserved.

  2. Regression tests — added in the second commit (tests/gateway/test_api_server_reasoning.py): Runs reasoning.delta event ordering, Chat delta.reasoning_content separation from content chunks, non-thinking-model emitting no reasoning, and callback-error resilience. Verified locally: pytest tests/gateway/test_api_server_reasoning.py -q → 6 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teknium1 teknium1 teknium1 left review comments

Assignees

No one assigned

Labels

area/streaming Streaming responses: gateway delivery, provider wire comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@huanshan5195 @alt-glitch @teknium1