fix(bedrock): filter internal json_tool_call when mixed with real tools

[jquinter]

Summary

• Fixes [Bug]: Bedrock returns fake json_tool_call tool when using response_format + tools together, breaking OpenAI Agents SDK #18381: When using both tools and response_format with Bedrock Converse API, LiteLLM internally adds a fake tool called json_tool_call (RESPONSE_FORMAT_TOOL_NAME) to handle structured output. Bedrock may return both this internal tool AND real user-defined tools, causing consumers like OpenAI Agents SDK to break trying to dispatch json_tool_call as a real tool.
• Supersedes When using multiple workers with uvicorn, use MultiProcessCollector for Prometheus #11067 and builds on the approach from Fix Bedrock Converse API returning both json_tool_call and real tools when tools and response_format are used #18384 by @haggai-backline, extending it to cover the streaming case and fixing the optional_params.pop() mutation issue.
• Replaces PR fix(bedrock): filter internal json_tool_call when mixed with real tools #20916 with properly scoped changes (Bedrock only, no unrelated files)

Changes

Non-streaming fix (converse_transformation.py)

• Extracted _filter_json_mode_tools() static method that handles 3 scenarios:
  • Only json_tool_call → convert to content
  • Mixed with real tools → filter out json_tool_call, keep real tools
  • No json_tool_call → pass through unchanged
• Changed optional_params.pop("json_mode") → .get("json_mode") to avoid mutating caller's dict

Streaming fix (invoke_handler.py)

• Added json_mode parameter and _current_tool_name tracking to AWSEventStreamDecoder
• When json_mode=True, suppresses json_tool_call start/stop chunks and converts delta chunks to text content instead of tool call arguments
• Real tool chunks pass through normally

Plumbing (converse_handler.py)

• Pass json_mode to AWSEventStreamDecoder in make_call and make_sync_call

Test plan

• test_transform_response_with_both_json_tool_call_and_real_tool — Bedrock returns both json_tool_call AND get_weather, verifies only get_weather remains
• test_transform_response_does_not_mutate_optional_params — Verifies optional_params still contains json_mode after _transform_response()
• test_streaming_filters_json_tool_call_with_real_tools — Streaming chunks with both tools, verifies json_tool_call becomes text content while real tool passes through
• test_streaming_without_json_mode_passes_all_tools — Backward compat: json_mode=False passes all tools through unchanged
• All 63 existing tests in test_converse_transformation.py continue to pass

Credits to @haggai-backline for the original investigation and non-streaming approach in PR #18384.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 28, 2026 3:25am

[greptile-apps]

Greptile Overview

Greptile Summary

This PR fixes issue #18381 where using both tools and response_format with Bedrock Converse API caused the internal json_tool_call (RESPONSE_FORMAT_TOOL_NAME) to leak through alongside real user-defined tools, breaking consumers like OpenAI Agents SDK.

Key changes:

• Non-streaming: Extracted _filter_json_mode_tools() in converse_transformation.py that correctly handles three scenarios: json_tool_call only (converts to content), mixed with real tools (filters out json_tool_call, preserves as text content), and no json_tool_call (pass-through). Also fixed optional_params.pop() → .get() to prevent mutating the caller's dict.
• Streaming: Added json_mode and _current_tool_name state tracking to AWSEventStreamDecoder. When json_mode=True, json_tool_call start/stop chunks are suppressed and delta chunks are emitted as text content instead of tool call arguments.
• Plumbing: json_mode is now passed through to AWSEventStreamDecoder in both sync and async paths.
• All changes are properly scoped to the Bedrock provider directory (litellm/llms/bedrock/), following the project's architecture guidelines.
• 4 new mock tests cover the mixed tools scenario, optional_params mutation fix, streaming filtering, and backward compatibility.

Confidence Score: 4/5

• This PR is safe to merge — well-scoped fix with comprehensive test coverage for both streaming and non-streaming paths.
• Score of 4 reflects: properly scoped Bedrock-only changes, clear separation of concerns with the extracted _filter_json_mode_tools method, good backward compatibility (json_mode=False passes everything through unchanged), the .pop() to .get() mutation fix, and 4 well-designed mock tests. Minor deductions for duplicated properties-unwrapping logic and the inherent complexity of streaming state management, though the implementation handles it correctly.
• litellm/llms/bedrock/chat/invoke_handler.py has the most complex changes with streaming state management via _current_tool_name tracking — worth a careful look at edge cases.

Important Files Changed

Filename	Overview
litellm/llms/bedrock/chat/converse_transformation.py	Added _filter_json_mode_tools() static method to handle 3 scenarios (json_tool_call only, mixed, none). Changed .pop() to .get() to avoid mutating optional_params. Clean refactoring of existing logic with good separation of concerns.
litellm/llms/bedrock/chat/invoke_handler.py	Added json_mode and _current_tool_name tracking to AWSEventStreamDecoder for streaming. Suppresses json_tool_call start/stop chunks and converts delta chunks to text when json_mode is active. State management is correct for sequential block processing.
litellm/llms/bedrock/chat/converse_handler.py	One-line change passing json_mode to AWSEventStreamDecoder constructor in make_sync_call. Minimal and correct plumbing change.
tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py	Added 4 well-structured mock tests covering mixed tools, optional_params mutation, streaming filtering, and backward compatibility. No real network calls - all properly mocked.

Sequence Diagram

sequenceDiagram
    participant Client
    participant LiteLLM
    participant Bedrock

    Note over Client,Bedrock: Non-Streaming (Mixed tools + json_mode)
    Client->>LiteLLM: completion(tools=[get_weather], response_format=json)
    LiteLLM->>Bedrock: Converse API (tools=[json_tool_call, get_weather])
    Bedrock-->>LiteLLM: Response with json_tool_call + get_weather tool calls
    LiteLLM->>LiteLLM: _filter_json_mode_tools()
    Note over LiteLLM: json_tool_call → message.content<br/>get_weather → tool_calls
    LiteLLM-->>Client: Response(tool_calls=[get_weather], content=json_data)

    Note over Client,Bedrock: Streaming (Mixed tools + json_mode)
    Client->>LiteLLM: completion(stream=True, tools=[get_weather], response_format=json)
    LiteLLM->>Bedrock: Converse API (stream)
    Bedrock-->>LiteLLM: start: json_tool_call
    LiteLLM->>LiteLLM: Suppress start (json_mode=True)
    Bedrock-->>LiteLLM: delta: json_tool_call input
    LiteLLM->>LiteLLM: Convert to text delta
    LiteLLM-->>Client: chunk(content=json_data)
    Bedrock-->>LiteLLM: stop: json_tool_call
    LiteLLM->>LiteLLM: Suppress stop
    Bedrock-->>LiteLLM: start: get_weather
    LiteLLM-->>Client: chunk(tool_calls=[get_weather])
    Bedrock-->>LiteLLM: delta: get_weather input
    LiteLLM-->>Client: chunk(tool_calls=[get_weather args])

Loading

_{Last reviewed commit: 01636aa}

[greptile-apps]

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[Sanady]

Any updates on this pull request? We are facing similar issue and if this is fixing this issue it would be beneficial.