fix(api-server): stream run reasoning deltas

[xingj404-lab]

Summary

• wire the agent reasoning callback into the /v1/runs execution path
• emit additive reasoning.delta SSE events for structured reasoning chunks
• preserve the existing message, tool, error, and completion event behavior
• add regression coverage proving reasoning frames arrive before run.completed

Root cause

The provider stream already invokes the agent's reasoning callback, but the API server's runs adapter did not pass such a callback when constructing AIAgent. As a result, reasoning could be persisted and visible in session history after the run while remaining absent from the live runs SSE stream.

Compatibility

This is an additive SSE event. Existing clients that ignore unknown event types continue to behave as before.

Validation

• scripts/run_tests.sh tests/gateway/test_api_server_runs.py
• 34 tests passed

Fixes #60634

[alt-glitch]

This was generated by AI during triage.

Duplicate of #15169: the live patch uses the same /v1/runs reasoning-callback wiring and reasoning.delta SSE mechanism.

[teknium1]

Thanks for the focused regression coverage. The underlying gap is real on current main: gateway/platforms/api_server.py:1705-1834 does not pass a reasoning callback into AIAgent, while run_agent.py:5037-5049 only delivers reasoning chunks through self.reasoning_callback.

Problems

• This duplicates open PR fix(api_server): stream reasoning.delta events over /v1/runs SSE #15169 (293be5c04643bd4a7eaa111f010c9117678d4de2), which implements the same /v1/runs callback wiring and reasoning.delta SSE path. The member triage comment also identifies fix(api_server): stream reasoning.delta events over /v1/runs SSE #15169 as the live duplicate patch.
• The two patches choose different event payload keys: this PR adds delta; fix(api_server): stream reasoning.delta events over /v1/runs SSE #15169 uses text. website/docs/user-guide/features/api-server.md:275-283 documents the stream generically, not a reasoning-event schema.

Suggested changes

• Consolidate on one canonical patch and settle the wire shape before salvage, preserving the earlier contribution where feasible.

Automated hermes-sweeper review.

[GottZ]

This was generated by AI during triage.

Summary

Twenty-six PRs address or reference this issue complex across four causes: incorrect answer-derived reasoning snapshots, missing live reasoning callbacks in Runs/Chat/Responses SSE, provider-specific reasoning-marker leakage, and an upstream LLM Gateway SSE parsing failure. The diffs either correct the reasoning source, wire structured reasoning to API transports, add delimiter scrubbers, or introduce a JSONDecodeError fallback; several combine or supersede those narrower changes.

Related pull requests

• feat(api_server): wire tool_progress and reasoning callbacks into SSE stream #4265 [closed] related — (+176/-2) — n/a: Wires tool-progress and reasoning callbacks into Chat Completions SSE, but also includes unrelated body-limit and multimodal processing changes; closed in favor of the narrower feat(api-server): stream chat reasoning chunks #13401.
• fix(api_server): stream reasoning tokens as hermes.reasoning.delta SSE events #11482 related — (+107/-0) — n/a: Wires Chat Completions reasoning callbacks and emits custom hermes.reasoning.delta events with serialization tests, but its tests bypass the production forwarding seam and its wire format competes with delta.reasoning_content.
• feat(api-server): stream chat reasoning chunks #13401 [closed] related — (+70/-0) — n/a: Implements focused Chat Completions callback forwarding and delta.reasoning_content SSE chunks; closed at the author's request in favor of the broader feat(api_server): stream reasoning_content deltas via SSE #57094, despite its high-salvage keep-open review.
• fix(api_server): stream reasoning.delta events over /v1/runs SSE #15169 best fix — (+239/-0) — n/a: Adds the missing /v1/runs reasoning callback, emits incremental reasoning.delta events with a text field, and tests forwarding and SSE ordering; it remains the earlier recorded best fix, with one test-name/error-path correction requested by the keep-open review.
• feat(api_server): emit hermes.reasoning.delta SSE event in /v1/chat/completions stream #22364 [closed] related — (+20/-0) — n/a: Adds the same Chat Completions callback plumbing as fix(api_server): stream reasoning tokens as hermes.reasoning.delta SSE events #11482/feat(api-server): stream chat reasoning chunks #13401 but emits custom hermes.reasoning.delta events; closed after the author reverted the feature.
• feat(api_server): stream inline <think> as delta.reasoning_content (unify A/B-class reasoning) #23638 partial — (+299/-93) — n/a: Broadens Chat Completions reasoning support by recovering inline `` content and combining it with structured reasoning as delta.reasoning_content, but the keep-open review requires endpoint and split-stream regression coverage.
• fix(api_server): stream reasoning deltas via reasoning_callback (closes #24518) #24535 [closed] best fix — (+28/-6) — n/a: Suppresses answer-derived reasoning.available snapshots and emits real Runs reasoning through reasoning_callback; closed by its author for contribution-path reasons but still a recorded best-fix reference for api_server: reasoning.available event carries final response text instead of reasoning_content #24518.
• fix(run_agent): gate reasoning.available on reasoning_content, not content #24566 [closed] best fix — (+208/-18) — n/a: Separates subagent _thinking from top-level reasoning.available, sourcing the latter only from reasoning_content with regression tests; closed because conflicts required a fresh rebase, but retained as a recorded best-fix reference.
• fix: use reasoning_content for reasoning.available event #24571 [closed] partial — (+18/-6) — n/a: Changes reasoning.available to prefer reasoning fields but falls back to visible content, preserving the reported answer-echo failure for models without structured reasoning; closed as a duplicate of the more thoroughly decoupled fix(run_agent): gate reasoning.available on reasoning_content, not content #24566.
• fix: disable streaming on JSONDecodeError when tools are present #24572 [closed] best fix — (+14/-1) — n/a: Disables streaming after a tool-bearing JSONDecodeError so retry can use the non-streaming path; closed as a duplicate of fix(agent): fall back to non-streaming on JSONDecodeError #24532 and remains a recorded best-fix reference, while the issue's upstream gateway fix makes the workaround obsolete for updated LLM Gateway deployments.
• fix(agent): reasoning event source + streaming JSONDecodeError with tools #24852 [closed] partial — (+32/-7) — n/a: Combines the fix: use reasoning_content for reasoning.available event #24571-style reasoning-source fallback with the same tool-stream JSONDecodeError fallback as fix: disable streaming on JSONDecodeError when tools are present #24572; closed as a duplicate of fix(agent): fall back to non-streaming on JSONDecodeError #24532/fix(api_server): stream reasoning deltas via reasoning_callback (closes #24518) #24535 rather than a canonical combined fix.
• [codex] Expose chat completions reasoning SSE #35083 related — (+68/-2) — n/a: Wires Chat Completions reasoning to a new custom hermes.reasoning event and adds an endpoint test, but the keep-open review requires alignment with the requested delta.reasoning_content contract and documentation.
• fix: unpack StreamingThinkScrubber tuple in streaming delta paths #37382 [closed] related — (+12/-2) — n/a: Unpacks tuple-returning scrubber results in two streaming call sites, but current main's scrubber returns strings; closed because it is an incomplete stacked fragment of feat(api_server): stream inline <think> as delta.reasoning_content (unify A/B-class reasoning) #23638 that would fail independently.
• fix(agent): add Chinese reasoning tags to think-block filter for MiniMax M3 #43836 [closed] partial — (+162/-2) — n/a: Adds bare Chinese MiniMax reasoning markers to three scrubber surfaces with tests; closed after a blocking contributor design decision rejected provider-specific output scrubbing in favor of reasoning routing.
• fix(stream): scrub MiniMax M3 Chinese reasoning markers #43839 [closed] best fix — (+84/-10) — n/a: Provides the most structured, boundary-gated Chinese-marker scrubber with shared constants and broad tests, but despite its recorded best-fix status it was closed under the same blocking maintainer policy against bespoke delimiter parsing.
• fix(agent): add Chinese reasoning tags to think block filters (#43827) #43932 [closed] partial — (+15/-2) — n/a: Removes Chinese marker words through direct replacements and extends CLI/gateway tag lists; closed as superseded by fix(agent): add Chinese reasoning tags to think-block filter for MiniMax M3 #43836, with the contributor duplicate finding outweighing the non-contributor approval.
• feat(api-server): stream model reasoning as reasoning_content in chat completions #52606 related — (+23/-0) — n/a: Narrowly adds Chat Completions reasoning_callback forwarding and delta.reasoning_content, but has no regression test and overlaps the broader tested feat(api_server): stream reasoning_content deltas via SSE #57094 implementation.
• fix(think_scrubber): recover final answer from unclosed <think> block #53391 related — (+118/-8) — n/a: Attempts to recover a final line from an unclosed `` block, but the keep-open review identifies a reasoning-leak risk because generic multiline reasoning is indistinguishable from the claimed malformed answer format.
• feat(gateway): stream reasoning.delta over Runs API SSE #55163 duplicate — (+20/-0) — n/a: Adds Runs reasoning.delta callback wiring with a delta payload, but the keep-open review requires the current _put_event_if_active transport guard and a focused SSE regression test.
• feat(api_server): stream reasoning_content deltas via SSE #57094 related — (+399/-158) — n/a: Covers streaming and non-streaming Chat reasoning plus Responses response.reasoning.delta, tests all three surfaces, and documents them; it supersedes closed feat(api-server): stream chat reasoning chunks #13401, though its diff also contains unrelated runtime/fallback changes that should be split or rebased away.
• fix: include final reasoning in run events #60664 [closed] partial — (+135/-5) — n/a: Reconciles final persisted reasoning into Runs SSE, completion, and status, but was closed because it duplicates uncapped reasoning across three locations and changes the existing event shape.
• fix(api): include final reasoning in runs SSE #60906 best fix — (+125/-2) — n/a: Replays normalized final assistant reasoning before run.completed and includes the safe turn transcript for refresh parity; it is the recorded best fix for final reconciliation and has a high-salvage keep-open verdict with later clean rebase and CI evidence.
• feat(api_server): stream model reasoning as reasoning.delta on /v1/runs/events #61259 duplicate — (+60/-0) — n/a: Adds the same Runs callback and reasoning.delta path as fix(api_server): stream reasoning.delta events over /v1/runs SSE #15169 but uses a delta payload and adds an endpoint test; the keep-open review calls for documentation and canonical schema selection.
• fix(api-server): stream run reasoning deltas #67339 best fix — (+56/-0) — n/a: Adds guarded Runs callback wiring, reasoning.delta events with a delta field, and ordering coverage; despite its recorded best-fix status and keep-open review, both the review and contributor triage identify its live mechanism as a duplicate of earlier recorded best fix fix(api_server): stream reasoning.delta events over /v1/runs SSE #15169.
• feat(api-server): surface reasoning_content on /v1/chat/completions #74692 related — (+115/-6) — n/a: Surfaces answer-derived reasoning.available previews in streaming and non-streaming Chat responses, but the blocking contributor review confirms it does not wire the provider reasoning_callback and therefore misses the reported structured-reasoning source.
• fix(gateway): stream real reasoning_content instead of answer echo #75562 best fix — (+432/-13) — n/a: Removes the top-level answer-echo fallback and wires real reasoning into both Runs reasoning.delta and Chat delta.reasoning_content with endpoint and forwarding tests; it remains a recorded best fix for api_server: reasoning.available event carries final response text instead of reasoning_content #24518/Runs SSE exposes incomplete reasoning compared with session history raw messages #60634, with the review-requested Runs event correction reflected in the shown diff.

Duplicates

Runs live-delta cluster: #15169, #55163, #61259, and #67339, with the same callback mechanism also included in broader #75562; Chat callback cluster: #11482, #13401, #22364, #35083, #52606, and the Chat portions of #57094/#75562. Reasoning-source duplicates are #24566/#24571 and the corresponding parts of #24535/#24852/#75562; Chinese-marker scrubbers are #43836/#43839/#43932; JSONDecodeError fallbacks are #24572/#24852.

Suggested consolidation

Close #67339 as a duplicate of #15169: despite the keep-open review and recorded best-fix verdict on #67339, its diff implements the same Runs callback and reasoning.delta path as the earlier #15169, which is itself a recorded best fix, while differing only in the unsettled delta versus text payload key. Author action on #15169: rebase onto current main, use the active-stream guard, settle and document the payload schema, and correct the review-identified error-path test; keep #60906 open for complementary final-history reconciliation, keep #75562 open only as a salvage path for the distinct answer-echo removal and Chat transport work, and consolidate the remaining live Runs and Chat duplicates into those canonical scopes.

Complex graph

flowchart TD
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I24518(["issue #24518 (closed)"])
    I60634(["issue #60634 (open)"])
    subgraph Dup15169 ["PRs duplicating each other"]
        P15169["PR #15169 (open)"]
        P24535["PR #24535 (closed)"]
        P55163["PR #55163 (open)"]
        P61259["PR #61259 (open)"]
        P67339["PR #67339 (open)"]
        P75562["PR #75562 (open)"]
    end
    P67339 -.->|partial| I24518
    P67339 -->|best fix| I60634
    class I24518 closed
    class I60634 open
    class P15169 open
    class P24535 closed
    class P55163 open
    class P61259 open
    class P67339 open
    class P75562 open
    class P15169 best
    class P24535 best
    class P67339 best
    class P75562 best
    class P75562 best
    class P67339 target
    click I24518 "https://github.com/NousResearch/hermes-agent/issues/24518"
    click I60634 "https://github.com/NousResearch/hermes-agent/issues/60634"
    click P15169 "https://github.com/NousResearch/hermes-agent/pull/15169"
    click P24535 "https://github.com/NousResearch/hermes-agent/pull/24535"
    click P55163 "https://github.com/NousResearch/hermes-agent/pull/55163"
    click P61259 "https://github.com/NousResearch/hermes-agent/pull/61259"
    click P67339 "https://github.com/NousResearch/hermes-agent/pull/67339"
    click P75562 "https://github.com/NousResearch/hermes-agent/pull/75562"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 26 pull requests and 5 issues in this complex. Each diff was read against this issue; Assessment working set: 230 kB of PR diffs, 89 kB of issue/PR text, 42 kB of discussion (64 comments), 40 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.