fix(websearch): websearch_interception agentic loop fixes for chat completions and anthropic messages - #31669
Conversation
maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller. Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path. Regression test added.
When the original request forces tool_choice to litellm_web_search, the follow-up request after search execution inherited that tool_choice, causing the model to call the search tool again instead of synthesizing an answer from the results.
… messages Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's synthesis call after executing Exa/Perplexity searches) were missing api_key because the named api_key param in async_anthropic_messages_handler was never merged into the kwargs dict forwarded downstream. Result: every /v1/messages websearch follow-up failed with "x-api-key header is required" and the caller received the raw tool_use response instead of the synthesized answer.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
https://github.com/BerriAI/litellm/blob/45b60e088d6493c564b6fefc9994b3ada335e96e/litellm_core_utils/chat_completion_agentic_loop.py#L221-L222
This gate now skips any callback that only implements the generic agentic-loop hooks, but litellm/integrations/code_interpreter_interception/handler.py is one of those callbacks: it overrides async_should_run_agentic_loop and explicitly branches on _agentic_loop_api_surface == CHAT_COMPLETION_AGENTIC_SURFACE to handle chat-completions. For /v1/chat/completions requests using the code-execution tool, the callback is no longer invoked, so the server-side tool execution path is bypassed and callers get the raw tool call instead of the executed result. Please keep the generic hook path as a fallback or add matching chat-completion hook implementations for existing generic integrations.
https://github.com/BerriAI/litellm/blob/45b60e088d6493c564b6fefc9994b3ada335e96e/litellm_core_utils/chat_completion_agentic_loop.py#L146
Dropping tool_choice from optional_params_clean does not actually remove it from the follow-up request because both chat-completion executors start from the original optional_params and then merge the patch params over it. In the scenario this commit calls out, where the initial request forced tool_choice to litellm_web_search, the original forced choice survives the merge and the synthesis call is forced to call web search again, causing a repeated-tool safety failure or the legacy OpenAI path to recurse instead of returning the synthesized answer. Please delete it after the merge or otherwise make the patch explicitly clear the original value.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Greptile SummaryThis PR fixes the websearch interception agentic loop for chat completions and Anthropic Messages. The main changes are:
Confidence Score: 5/5The changes are narrowly scoped to websearch interception hook routing and follow-up request construction, with focused tests covering the fixed paths. No blocking correctness issues were identified in the changed code, and the added tests exercise the chat-completion hook dispatch, forced tool choice removal, and Anthropic API key forwarding behavior.
 T-Rex LogsWhat T-Rex did
Reviews (4): Last reviewed commit: "fix(websearch): always strip tool_choice..." | Re-trigger Greptile |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…on-specific hooks CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with _agentic_loop_api_surface to handle both surfaces from one hook. The chat completion loop must also check _gate_overridden so callbacks using the unified hook pattern still fire for chat completions.
|
|
|
@greptileai review |
krrish-berri-2
left a comment
krrish-berri-2
left a comment
There was a problem hiding this comment.
I think touching chat_completion_agentic_loop.py is the right layer for the chat-completions hook dispatch bug. /v1/chat/completions reconverges through the provider-agnostic dispatcher after provider-specific routing, so a callback that only implements async_should_run_chat_completion_agentic_loop needs that dispatcher to call the chat-specific gate. The latest commit also keeps the generic-hook fallback for integrations like code interpreter, so the core-loop change itself does not look suspicious to me
I do think the tool_choice fix still needs a follow-up before merge. The patch builder now omits tool_choice, but both follow-up execution paths start from the original request params and merge the patch over them. Omission does not clear the original forced value, so a request forced to litellm_web_search can still force the synthesis call back into search. The fix should explicitly clear tool_choice after the merge, or add an explicit clear/delete signal to the patch contract, and cover it with a regression test that captures the follow-up litellm.acompletion kwargs
| if k | ||
| not in { | ||
| "tools", | ||
| "tool_choice", |
There was a problem hiding this comment.
This removes tool_choice from the patch, but it does not clear the original value during follow-up execution. Both the typed dispatcher and the legacy direct path start from the original optional_params and then merge request_patch.optional_params, so omission here leaves a forced litellm_web_search choice in place. Please explicitly clear tool_choice after merging, or encode a clear/delete signal in the request patch, and add a regression test that proves the follow-up litellm.acompletion call has no tool_choice when the original request forced web search
…up call The _execute_chat_completion_agentic_loop path merged original optional_params (which includes forced tool_choice) into follow-up params without explicit removal. _build_chat_completion_request_patch already excluded tool_choice from its optional_params output, but dict.update() with a missing key leaves the original value intact. Explicit pop after the merge removes it.
|
@greptileai review |
The tool_choice removal was gated on patch.tools is not None. WebSearch sets tools via patch.optional_params not patch.tools, so the gate was False and forced tool_choice from the original request survived into the synthesis call. Move the pop outside the patch.tools branch so it applies unconditionally.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
@greptileai review |
| if not _gate_overridden(callback): | ||
|
|
||
| uses_chat_completion_hooks = _chat_completion_gate_overridden(callback) | ||
| uses_unified_hooks = not uses_chat_completion_hooks and _gate_overridden(callback) |
There was a problem hiding this comment.
This seems like a lot of conditional logic is there a cleaner way we can do this ?
Replace three-variable conditional logic with one use_chat_hooks bool. Gate dispatch (gate fn selection) and plan dispatch both key off the same bool, with no intermediate uses_unified_hooks variable needed.
Remove the if/else branching between chat-completion-specific hooks and unified hooks in maybe_run_chat_completion_agentic_loop. The dispatcher now exclusively uses _gate_overridden + async_should_run_agentic_loop. WebSearchInterceptionLogger routes to the appropriate internal method based on _agentic_loop_api_surface, keeping the separation of concerns inside the callback rather than in the dispatcher.
|
@ishaan-jaff ready for re-review. Addressed your feedback - dispatcher now has zero branching on hook type; routing logic moved inside WebSearchInterceptionLogger based on _agentic_loop_api_surface. E2E confirmed with real Exa searches on both /v1/chat/completions and /v1/messages. |
…mpletions and anthropic messages (BerriAI#31669) * fix(websearch): wire chat completion agentic loop to correct hooks maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller. Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path. Regression test added. * fix(websearch): strip tool_choice from follow-up request When the original request forces tool_choice to litellm_web_search, the follow-up request after search execution inherited that tool_choice, causing the model to call the search tool again instead of synthesizing an answer from the results. * fix(websearch): inject api_key into agentic hook kwargs for anthropic messages Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's synthesis call after executing Exa/Perplexity searches) were missing api_key because the named api_key param in async_anthropic_messages_handler was never merged into the kwargs dict forwarded downstream. Result: every /v1/messages websearch follow-up failed with "x-api-key header is required" and the caller received the raw tool_use response instead of the synthesized answer. * ci: trigger CI run * fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with _agentic_loop_api_surface to handle both surfaces from one hook. The chat completion loop must also check _gate_overridden so callbacks using the unified hook pattern still fire for chat completions. * fix(websearch): strip tool_choice from legacy chat completion follow-up call The _execute_chat_completion_agentic_loop path merged original optional_params (which includes forced tool_choice) into follow-up params without explicit removal. _build_chat_completion_request_patch already excluded tool_choice from its optional_params output, but dict.update() with a missing key leaves the original value intact. Explicit pop after the merge removes it. * fix(websearch): always strip tool_choice from plan-path follow-up params The tool_choice removal was gated on patch.tools is not None. WebSearch sets tools via patch.optional_params not patch.tools, so the gate was False and forced tool_choice from the original request survived into the synthesis call. Move the pop outside the patch.tools branch so it applies unconditionally.
4 participants
Relevant issues
Fixes websearch_interception callback not triggering for chat completion requests (LLM returned finish_reason: tool_calls with litellm_web_search but search never executed).
Linear ticket
Pre-Submission checklist
@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewScreenshots / Proof of Fix
Start the proxy from the repo root (the config sets
callbacks: ["websearch_interception"]andsearch_toolswith Exa AI):OpenAI /v1/chat/completions
Response (real Exa search result, not a tool_calls turn):
{ "choices": [{ "finish_reason": "stop", "message": { "content": "As of today, the current price of Bitcoin (BTC) is approximately **$58,607.99**. The price has seen a decrease of around **2.9%** over the last 24 hours. Bitcoin has a market capitalization of about **$1.17 trillion**, with a circulating supply of around **20.05 million BTC**." } }] }Anthropic /v1/messages
Response (real Exa search result, not a tool_use turn):
{ "stop_reason": "end_turn", "content": [{ "type": "text", "text": "Based on the current search results, here is the current price of Bitcoin today:\n\n## Bitcoin Price Today\n\n**$58,500 - $58,700 USD**\n\nThe exact price varies slightly depending on the exchange and time of day, but major sources report:\n\n- **CoinMarketCap**: $58,598.17 USD\n- **CoinGecko**: $58,617.65 USD\n- **Kraken**: $58,509.00 USD\n- **Coinbase**: Approximately $58,700 USD\n..." }] }Both paths return real search results with
finish_reason: stop/stop_reason: end_turninstead of raw tool call turns.Type
Bug Fix
Refactoring
Changes
Three bugs fixed in the websearch_interception agentic loop, plus a refactor to address the PR review feedback.
Bug 1: chat completion requests never intercepted
maybe_run_chat_completion_agentic_loop was checking _gate_overridden (which inspects async_should_run_agentic_loop) but WebSearchInterceptionLogger only overrode async_should_run_chat_completion_agentic_loop. The gate check never matched so the hook was never called.
Bug 2: tool_choice leaked into synthesis follow-up (both paths)
When the original request had tool_choice forced to litellm_web_search, the follow-up call after executing the search inherited that tool_choice, causing the model to call the search tool again instead of synthesizing an answer.
Legacy path: params.update(request_patch.optional_params) didn't clear the original tool_choice. Fixed with explicit pop after merge.
Plan path: the pop was gated on if patch.tools is not None, which is False for websearch (no tool swap needed). Fixed by moving the pop outside that gate.
Bug 3: api_key missing from Anthropic Messages agentic hooks
On the llm_http_handler non-streaming and streaming paths, api_key was a named local variable that never got merged into kwargs before _call_agentic_completion_hooks. The follow-up call had no credentials.
Refactor: single unified dispatch flow
Per PR review feedback, eliminated the if/else branching in maybe_run_chat_completion_agentic_loop that selected between chat-completion-specific hooks and unified hooks. The dispatcher now uses only _gate_overridden + async_should_run_agentic_loop. WebSearchInterceptionLogger's async_should_run_agentic_loop and async_build_agentic_loop_plan now route internally to the chat-completion or Anthropic path based on _agentic_loop_api_surface.