Skip to content

fix(gemini): keep tool_call JSON valid across streaming SSE replays - #25046

Open
Alex-Electron wants to merge 1 commit into
NousResearch:mainfrom
Alex-Electron:fix/gemini-native-stream-tool-calls-aggregation
Open

fix(gemini): keep tool_call JSON valid across streaming SSE replays#25046
Alex-Electron wants to merge 1 commit into
NousResearch:mainfrom
Alex-Electron:fix/gemini-native-stream-tool-calls-aggregation

Conversation

@Alex-Electron

Copy link
Copy Markdown

Summary

  • The native Gemini adapter emits json.dumps(args) on every SSE event that carries a functionCall. Gemini re-sends the same call across repeated events and may mutate args mid-stream, so the consumer (which concatenates delta.tool_calls[].function.arguments by index per OpenAI streaming protocol) ends up with invalid JSON like {"q": "A"}{"q": "AB"}, silently discarding the tool call.
  • The previous args_str.startswith(last_arguments) prefix-grow guard rarely fires on JSON: the trailing " / } mean a longer args dict almost never has the shorter one as a textual prefix (e.g. {"q":"abcdef"} doesn't start with {"q":"abc"}), so the fallback path emits the full new args.
  • Switch to a flush-on-finish strategy: on first sighting of a (part_index, name, signature) slot, emit a header chunk with name/id and empty arguments; cache the latest args across subsequent events; when finishReason arrives, emit one chunk per slot with the final full args, then the finish chunk. Concatenation by the consumer yields a single valid JSON string per slot, regardless of how often Gemini re-emits or mutates the call.

Test plan

  • tests/agent/test_gemini_native_adapter.py — full file passes (13/13)
  • New test_stream_event_translation_handles_non_prefix_args_change — reproduces the bug (would FAIL on main: json.decoder.JSONDecodeError: Extra data: line 1 column 11 (char 10) on '{"q": "A"}{"q": "AB"}') and PASSES with this fix
  • New test_stream_event_translation_handles_prefix_grow_args — final concat equals final args for the growing-args case
  • test_stream_event_translation_emits_tool_call_delta_with_stable_index rewritten to assert on protocol-level concat instead of fragile per-chunk ordering — still passes
  • test_stream_event_translation_keeps_identical_calls_in_distinct_parts — unchanged, still passes
  • End-to-end smoke test on a live Telegram-bot deployment: a single user turn that triggers mcp_context7_resolve_library_id + web_search + delegate_task in one assistant response. Pre-fix: tool-call args garbled into concatenated JSON, agent silently fell back to default behaviour. Post-fix: all three calls dispatched correctly and the structured answer returned.

Notes

The header chunk carries arguments: "" so consumers can open the tool-call slot (and UIs that surface "agent is calling X…" still get the signal), while the actual args land in a single chunk just before the finish chunk. This sidesteps both the dedup and the partial-args cases without needing the consumer to handle non-prefix-preserving fragments.

The native Gemini adapter emitted `json.dumps(args)` on every streaming SSE
event that carried a functionCall. Gemini re-sends the same call across
repeated events (and occasionally mutates its args mid-stream), so the
consumer ended up concatenating fragments like:

    {"q": "A"}{"q": "AB"}

This is invalid JSON and breaks downstream tool_call parsing, silently
discarding the call. The previous prefix-grow guard didn't help: JSON's
closing quote/brace means a longer args dict is almost never a textual
prefix of a shorter one (e.g. `{"q":"abcdef"}` does NOT start with
`{"q":"abc"}`), so the fallback path emitted the full new args.

Switch to a flush-on-finish strategy:

- First sighting of a (part_index, name, signature) slot → emit a header
  chunk carrying name/id and empty arguments. UI sees the call opening.
- Subsequent events update the slot's cached args; no further deltas are
  emitted until the finishReason event.
- On finishReason → emit one chunk per slot with the final full args,
  then the finish chunk.

The OpenAI streaming protocol concatenates `delta.tool_calls[].function
.arguments` by `index`; this scheme produces a single valid JSON string
per slot regardless of how often Gemini re-emits or mutates the call.

Tests:
- New `test_stream_event_translation_handles_non_prefix_args_change`
  reproduces the bug — concat of emitted fragments must be parseable
  JSON even when args change mid-stream.
- New `test_stream_event_translation_handles_prefix_grow_args` covers
  growing-args case (final concat == final args).
- `test_stream_event_translation_emits_tool_call_delta_with_stable_index`
  rewritten to assert on protocol-level concat instead of fragile
  per-chunk ordering.
- `test_stream_event_translation_keeps_identical_calls_in_distinct_parts`
  unchanged and still passing.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists provider/gemini Google Gemini (AI Studio, Cloud Code) comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 13, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused provider-boundary fix. The underlying failure is present on current main: agent/gemini_native_adapter.py:704-720 can emit two full argument objects for a changed replay, while agent/chat_completion_helpers.py:2319-2320 concatenates argument fragments for the same slot.

Problems

  • Commit 44aa08a's non-prefix regression test accepts either the initial or final argument object. That is weaker than the stated flush-on-finish contract, which caches and emits the latest arguments.

Suggested changes

  • Require the final replayed value ({"q": "AB"}) in test_stream_event_translation_handles_non_prefix_args_change; this prevents a future implementation from preserving valid JSON while dispatching stale arguments.

GitHub reports this branch as conflicting, but the target streaming logic is still present on current main. This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform area/streaming Streaming responses: gateway delivery, provider wire labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/streaming Streaming responses: gateway delivery, provider wire comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/gemini Google Gemini (AI Studio, Cloud Code) sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants