fix(streaming): dedupe content, recover reasoning, unique tool_call IDs in deferred flush - #9470
Merged
Merged
Conversation
When tool calls are discovered only during final parsing (after the
streaming token callback returns), processTools' default switch branch
used to emit the full accumulated content alongside the tool_call args
chunk. Clients that accumulate delta.content per the OpenAI streaming
contract end up showing every narration line twice. Three related bugs
in the same flush path:
1. Content duplication: the args chunk carried Content:textContentToReturn
even though the text had already been streamed token-by-token via
the token callback, so delta.content was both the running total and
bundled with tool_calls in one delta (two spec violations).
2. Reasoning drop: when the C++ autoparser surfaces reasoning only as
a final aggregate (no incremental tokens), the callback never emits
it and the flush branch didn't either, silently losing it.
3. tool_call ID collision: empty ss.ID fell back to the request id, so
multiple empty-ID calls in the same turn all shared the same id,
breaking tool_result matching by tool_call_id.
Extracted the block into buildDeferredToolCallChunks (pure function,
unit-testable) and added 19 Ginkgo specs covering streamed vs.
not-streamed content/reasoning, single vs. multi call, and
incremental-vs-deferred emission. Every case asserts the invariant
that no delta carries both non-empty Content/Reasoning and non-empty
ToolCalls.
Fix summary:
- emit reasoning in its own leading chunk when !reasoningAlreadyStreamed
- emit role+content in their own chunks when !contentAlreadyStreamed
- drop Content from the tool_call args chunk
- fallback to fmt.Sprintf("%s-%d", id, i) for empty ss.ID so calls stay
uniquely addressable
Reproduced live against qwen3.6-35b-a3b-apex served by LocalAI with
the C++ autoparser; the full-content replay chunk that preceded each
tool_calls block is gone after the fix.
Assisted-by: Claude:claude-opus-4-7 go vet
mudler
force-pushed
the
fix/stream-tool-calls-dup-content
branch
from
April 21, 2026 13:23
635eacd to
4ec7a74
Compare
extractor.Reasoning() returns only the Go-side extractor's lastReasoning accumulator (pkg/reasoning/extractor.go:129). ChatDelta reasoning coming through ProcessChatDeltaReasoning lives in a separate accumulator (cdLastStrippedReasoning) that Reasoning() does not expose. The "reasoning != \"\" && extractor.Reasoning() == \"\"" guard therefore fires exactly when the autoparser streamed reasoning incrementally via the callback — producing a duplicate final delivery. Replace both guard sites in the noActionToRun branch with the sentReasoning flag introduced in the previous commit. Extract the closing-chunk logic into buildNoActionFinalChunks so the refactor is testable; the helper mirrors buildDeferredToolCallChunks. Add Ginkgo coverage for both the content-streamed and content-not-streamed paths: reasoning is dropped when it was streamed, delivered once when it arrived only as a final aggregate, and omitted when empty. Metadata invariants carried over from the sibling helper. Assisted-by: Claude:claude-opus-4-7 go vet
The previous condition only looked at functionResults[0].Name, which misbehaved when a real tool call followed a noAction sentinel — the noAction shadowed the real call and the whole turn was treated as a question to answer, silently dropping the tool call. The mirror case, [realCall, noActionCall], fell into the default branch and emitted the noAction entry as if it were a real tool_call. Replace with hasRealCall, which scans the slice and returns true as soon as it finds a non-noAction entry. noActionToRun now matches the semantic intent: "every entry is the noAction sentinel (or the slice is empty)". Note: this does not change incremental emission, where noAction entries may still be forwarded as tool_call chunks by the XML/JSON iterative parsers. That is a separate layer (functions.Parse*) and addressing it requires threading noAction through the parser APIs — out of scope for this change. Assisted-by: Claude:claude-opus-4-7 go vet
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When tool calls are discovered only during final parsing (after the streaming token callback returns), processTools' default switch branch used to emit the full accumulated content alongside the tool_call args chunk. Clients that accumulate delta.content per the OpenAI streaming contract end up showing every narration line twice. Three related bugs in the same flush path:
Extracted the block into buildDeferredToolCallChunks (pure function, unit-testable) and added 19 Ginkgo specs covering streamed vs. not-streamed content/reasoning, single vs. multi call, and incremental-vs-deferred emission. Every case asserts the invariant that no delta carries both non-empty Content/Reasoning and non-empty ToolCalls.
Fix summary:
Reproduced live against qwen3.6-35b-a3b-apex served by LocalAI with the C++ autoparser; the full-content replay chunk that preceded each tool_calls block is gone after the fix.