Skip to content

fix(streaming): dedupe content, recover reasoning, unique tool_call IDs in deferred flush - #9470

Merged
mudler merged 3 commits into
masterfrom
fix/stream-tool-calls-dup-content
Apr 21, 2026
Merged

fix(streaming): dedupe content, recover reasoning, unique tool_call IDs in deferred flush#9470
mudler merged 3 commits into
masterfrom
fix/stream-tool-calls-dup-content

Conversation

@mudler

@mudler mudler commented Apr 21, 2026

Copy link
Copy Markdown
Owner

When tool calls are discovered only during final parsing (after the streaming token callback returns), processTools' default switch branch used to emit the full accumulated content alongside the tool_call args chunk. Clients that accumulate delta.content per the OpenAI streaming contract end up showing every narration line twice. Three related bugs in the same flush path:

  1. Content duplication: the args chunk carried Content:textContentToReturn even though the text had already been streamed token-by-token via the token callback, so delta.content was both the running total and bundled with tool_calls in one delta (two spec violations).
  2. Reasoning drop: when the C++ autoparser surfaces reasoning only as a final aggregate (no incremental tokens), the callback never emits it and the flush branch didn't either, silently losing it.
  3. tool_call ID collision: empty ss.ID fell back to the request id, so multiple empty-ID calls in the same turn all shared the same id, breaking tool_result matching by tool_call_id.

Extracted the block into buildDeferredToolCallChunks (pure function, unit-testable) and added 19 Ginkgo specs covering streamed vs. not-streamed content/reasoning, single vs. multi call, and incremental-vs-deferred emission. Every case asserts the invariant that no delta carries both non-empty Content/Reasoning and non-empty ToolCalls.

Fix summary:

  • emit reasoning in its own leading chunk when !reasoningAlreadyStreamed
  • emit role+content in their own chunks when !contentAlreadyStreamed
  • drop Content from the tool_call args chunk
  • fallback to fmt.Sprintf("%s-%d", id, i) for empty ss.ID so calls stay uniquely addressable

Reproduced live against qwen3.6-35b-a3b-apex served by LocalAI with the C++ autoparser; the full-content replay chunk that preceded each tool_calls block is gone after the fix.

When tool calls are discovered only during final parsing (after the
streaming token callback returns), processTools' default switch branch
used to emit the full accumulated content alongside the tool_call args
chunk. Clients that accumulate delta.content per the OpenAI streaming
contract end up showing every narration line twice. Three related bugs
in the same flush path:

1. Content duplication: the args chunk carried Content:textContentToReturn
   even though the text had already been streamed token-by-token via
   the token callback, so delta.content was both the running total and
   bundled with tool_calls in one delta (two spec violations).
2. Reasoning drop: when the C++ autoparser surfaces reasoning only as
   a final aggregate (no incremental tokens), the callback never emits
   it and the flush branch didn't either, silently losing it.
3. tool_call ID collision: empty ss.ID fell back to the request id, so
   multiple empty-ID calls in the same turn all shared the same id,
   breaking tool_result matching by tool_call_id.

Extracted the block into buildDeferredToolCallChunks (pure function,
unit-testable) and added 19 Ginkgo specs covering streamed vs.
not-streamed content/reasoning, single vs. multi call, and
incremental-vs-deferred emission. Every case asserts the invariant
that no delta carries both non-empty Content/Reasoning and non-empty
ToolCalls.

Fix summary:
- emit reasoning in its own leading chunk when !reasoningAlreadyStreamed
- emit role+content in their own chunks when !contentAlreadyStreamed
- drop Content from the tool_call args chunk
- fallback to fmt.Sprintf("%s-%d", id, i) for empty ss.ID so calls stay
  uniquely addressable

Reproduced live against qwen3.6-35b-a3b-apex served by LocalAI with
the C++ autoparser; the full-content replay chunk that preceded each
tool_calls block is gone after the fix.

Assisted-by: Claude:claude-opus-4-7 go vet
@mudler
mudler force-pushed the fix/stream-tool-calls-dup-content branch from 635eacd to 4ec7a74 Compare April 21, 2026 13:23
mudler added 2 commits April 21, 2026 13:25
extractor.Reasoning() returns only the Go-side extractor's lastReasoning
accumulator (pkg/reasoning/extractor.go:129). ChatDelta reasoning
coming through ProcessChatDeltaReasoning lives in a separate
accumulator (cdLastStrippedReasoning) that Reasoning() does not
expose. The "reasoning != \"\" && extractor.Reasoning() == \"\"" guard
therefore fires exactly when the autoparser streamed reasoning
incrementally via the callback — producing a duplicate final delivery.

Replace both guard sites in the noActionToRun branch with the
sentReasoning flag introduced in the previous commit. Extract the
closing-chunk logic into buildNoActionFinalChunks so the refactor is
testable; the helper mirrors buildDeferredToolCallChunks.

Add Ginkgo coverage for both the content-streamed and
content-not-streamed paths: reasoning is dropped when it was streamed,
delivered once when it arrived only as a final aggregate, and omitted
when empty. Metadata invariants carried over from the sibling helper.

Assisted-by: Claude:claude-opus-4-7 go vet
The previous condition only looked at functionResults[0].Name, which
misbehaved when a real tool call followed a noAction sentinel — the
noAction shadowed the real call and the whole turn was treated as a
question to answer, silently dropping the tool call. The mirror case,
[realCall, noActionCall], fell into the default branch and emitted the
noAction entry as if it were a real tool_call.

Replace with hasRealCall, which scans the slice and returns true as
soon as it finds a non-noAction entry. noActionToRun now matches the
semantic intent: "every entry is the noAction sentinel (or the slice
is empty)".

Note: this does not change incremental emission, where noAction
entries may still be forwarded as tool_call chunks by the XML/JSON
iterative parsers. That is a separate layer (functions.Parse*) and
addressing it requires threading noAction through the parser APIs —
out of scope for this change.

Assisted-by: Claude:claude-opus-4-7 go vet
@mudler
mudler merged commit 7d8c1d5 into master Apr 21, 2026
41 checks passed
@mudler
mudler deleted the fix/stream-tool-calls-dup-content branch April 21, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant