fix(websearch): wrap agentic loop response in fake stream for streaming requests

[mateo-berri]

Relevant issues

Copy of #27449 by @vokako, re-targeted onto an internal branch so it can run through CircleCI. Original authorship is preserved on the commit

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Fully end to end against a live proxy with real web search, no mocks. A proxy runs websearch_interception on the anthropic provider wired to a real Tavily search_tools backend, then a streaming /v1/messages request whose prompt forces the model to call web_search, so the agentic loop runs a real search and synthesizes a grounded answer. The only thing that changes between before and after is how that final answer is framed back to the client: one JSON body, or an SSE stream

Setup is identical for before and after (any configured search_provider works; Tavily shown here):

cat > config.yaml <<'YAML'
model_list:
  - model_name: anthropic-haiku-4-5
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

search_tools:
  - search_tool_name: tavily-search
    litellm_params:
      search_provider: tavily
      api_key: os.environ/TAVILY_API_KEY

general_settings:
  master_key: sk-1234

litellm_settings:
  callbacks: ["websearch_interception"]
  websearch_interception_params:
    enabled_providers: ["anthropic"]
    search_tool_name: tavily-search
YAML

litellm --config config.yaml --port 4000

Reproduction request (streaming, native web_search tool, prompt forces a tool call so the agentic loop runs):

curl -sS -N -D /tmp/headers.txt -o /tmp/body.txt \
  -X POST http://localhost:4000/v1/messages \
  -H 'content-type: application/json' \
  -H 'x-api-key: sk-1234' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{
        "model": "anthropic-haiku-4-5",
        "max_tokens": 1024,
        "stream": true,
        "tools": [{"type": "web_search_20250305", "name": "web_search", "max_uses": 3}],
        "messages": [{"role": "user", "content": "Use the web_search tool to find who won the 2022 FIFA World Cup, then tell me. You must call web_search."}]
      }'

grep -i '^content-type' /tmp/headers.txt
grep -cE '^(event|data):' /tmp/body.txt   # how many SSE lines came back

Before (on litellm_internal_staging, without this PR)

The real Tavily search runs and the model answers (note the live web_search_result citations), but the whole thing comes back as a single JSON body with content-type: application/json, so an SSE client reads zero events

$ grep -i '^content-type' /tmp/headers.txt
content-type: application/json

$ grep -cE '^(event|data):' /tmp/body.txt
0

$ cat /tmp/body.txt
{"model":"anthropic-haiku-4-5","id":"msg_01Nx...","type":"message","role":"assistant",
 "content":[{"type":"web_search_tool_result","tool_use_id":"toolu_01AL...","content":[
     {"type":"web_search_result","url":"https://en.wikipedia.org/wiki/2022_FIFA_World_Cup","title":"2022 FIFA World Cup - Wikipedia"},
     {"type":"web_search_result","url":"https://www.fifa.com/en/tournaments/mens/worldcup/qatar2022","title":"..."}]},
   {"type":"text","text":"... Argentina won the 2022 FIFA World Cup ..."}],
 "stop_reason":"end_turn","usage":{"input_tokens":3170,"output_tokens":119}}

After (with this PR)

Same request, same real search, but the response is re-wrapped as an Anthropic SSE stream with content-type: text/event-stream and the grounded answer streams as text_delta events

$ grep -i '^content-type' /tmp/headers.txt
content-type: text/event-stream; charset=utf-8

$ grep -cE '^(event|data):' /tmp/body.txt
14

$ cat /tmp/body.txt
event: message_start
data: {"type": "message_start", "message": {"id": "msg_01Cx...", "role": "assistant", "model": "claude-haiku-4-5", "usage": {"input_tokens": 3170, "output_tokens": 0}}}

event: content_block_start
data: {"type": "content_block_start", "index": 1, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "text_delta", "text": "**Argentina** won the 2022 FIFA World Cup! They defeated France in the final on December 18, 2022, in Qatar ... 4-2 on penalty kicks ... Lionel Messi's first World Cup championship."}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 1}

event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}}

event: message_stop
data: {"type": "message_stop", "usage": {"input_tokens": 3170, "output_tokens": 118}}

Type

🐛 Bug Fix

Changes

When websearch_interception is enabled and a client sends a streaming request via /v1/messages, the handler converts stream=True to stream=False internally to execute the search. After the agentic loop completes, the non-streaming dict response was returned directly to the client expecting SSE events, resulting in empty streams

In _call_agentic_completion_hooks(), the FakeAnthropicMessagesStreamIterator wrapping previously only ran when no agentic loop executed. When the agentic loop did run, the return paths returned the dict directly without wrapping

This adds a _maybe_wrap_in_fake_stream() helper that checks the websearch_interception_converted_stream flag and wraps dict responses in FakeAnthropicMessagesStreamIterator, applied to all return paths in _call_agentic_completion_hooks (async_run_agentic_loop, _execute_anthropic_agentic_plan, plan.response_override, and plan.terminate)

The helper is gated on api_surface == "anthropic_messages"; the converted-stream flag is only ever set by anthropic-messages websearch interception and the wrapper rebuilds an Anthropic SSE stream, so other surfaces (e.g. the responses API) are left untouched on every return path. logging_obj is typed Optional to match the None call sites

This copy resolves merge conflicts against litellm_internal_staging; the base had since added a responses API-surface path, which is kept intact while the anthropic_messages path gets the fake-stream wrapping

Tests drive the helper across all branches and additionally exercise the legacy, response_override, and terminate return paths of _call_agentic_completion_hooks end to end, asserting the dict response is wrapped only when the converted-stream flag is set on the anthropic_messages surface

Files touched: litellm/llms/custom_httpx/llm_http_handler.py and tests/test_litellm/integrations/websearch_interception/test_websearch_streaming_wrap.py

---

Note

Medium Risk
Touches agentic completion hook return behavior for Anthropic messages and websearch interception; scope is narrow (gated flag + surface) but affects streaming contract for clients using that path.

Overview
Fixes empty SSE streams when websearch interception runs an agentic loop on /v1/messages requests that were originally stream=True but forced to non-streaming internally.

Introduces _maybe_wrap_in_fake_stream() on BaseLLMHTTPHandler, which reads websearch_interception_converted_stream on the logging object and, for api_surface == "anthropic_messages" dict responses, returns a FakeAnthropicMessagesStreamIterator instead of the raw message dict.

That helper is now applied on every exit path from _call_agentic_completion_hooks (legacy async_run_agentic_loop, plan response_override, terminate, _execute_anthropic_agentic_plan, and the no-loop tail), replacing logic that only wrapped when no agentic loop ran. Non-anthropic surfaces (e.g. responses API) are unchanged.

Adds test_websearch_streaming_wrap.py covering the helper branches and hook return paths with and without the converted-stream flag.

^{Reviewed by Cursor Bugbot for commit bec6c87. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[mateo-berri]

@greptileai

---

Generated by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes streaming responses for Anthropic Messages websearch interception. The main changes are:

• Adds _maybe_wrap_in_fake_stream() to wrap converted non-streaming dict responses as Anthropic SSE streams.
• Applies the wrapper across agentic hook return paths, including legacy loop, response override, terminate, plan execution, and no-loop paths.
• Keeps the wrapper gated to api_surface == "anthropic_messages" so responses API behavior stays unchanged.
• Adds focused tests for the helper and affected hook branches.

Confidence Score: 4/5

The change is narrowly scoped to websearch interception wrapping for Anthropic Messages streaming requests and is covered across the affected return paths.

The implementation is gated by API surface and the converted-stream flag, and focused tests exercise helper behavior plus the relevant hook branches. Remaining risk is limited to integration behavior around live provider streaming and websearch interception.

No specific files need additional attention beyond normal CI coverage for litellm/llms/custom_httpx/llm_http_handler.py and the new websearch streaming wrap tests.

T-Rex Logs

What T-Rex did

• I ran the baseline tests to compare pre-change behavior and observed that the base path lacked the _maybe_wrap_in_fake_stream helper and converted-stream agentic loop paths returned raw dicts for legacy, plan override, terminate, and anthropic plan execution, while the existing no-loop tail path returned FakeAnthropicMessagesStreamIterator and yielded Anthropic SSE bytes such as event: message_start/content_block_delta.
• I then ran the post-change tests and observed that head returns FakeAnthropicMessagesStreamIterator for flagged anthropic_messages dict responses across all exercised paths, yields Anthropic SSE event bytes, and leaves unflagged and responses-surface paths as raw dicts; both probe and targeted pytest exited with code 0 on head.

_{Ran code and verified through T-Rex}

_{Reviews (4): Last reviewed commit: "test(websearch): cover _execute_anthropi..." | Re-trigger Greptile}

[mateo-berri]

Both findings were valid and are addressed in 2a8e484. _maybe_wrap_in_fake_stream now takes api_surface and only wraps when it is anthropic_messages, so the responses surface (and any future reuse of the flag) is left untouched on every return path, not just the tail; logging_obj is now typed Optional["LiteLLMLoggingObj"]. Added regression tests that drive the legacy, response_override, and terminate paths of _call_agentic_completion_hooks end to end and assert the wrapping.

@greptileai

---

Generated by Claude Code

[mateo-berri]

Added two more regression tests in bec6c87 covering the remaining _execute_anthropic_agentic_plan and tail return paths so every converted-stream path is exercised; this also brings patch coverage on the diff to 100%. No production code changed since the 5/5 review.

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit bec6c87. Configure here.}