fix: short-circuit websearch for github_copilot provider - #24143
Conversation
…lot) For providers like github_copilot that don't natively support web search, Claude Code's search sub-conversations were falling through to the adapter path which strips the web_search tool and has no stream reconversion. Instead of routing search requests through the full LLM pipeline, detect web-search-only requests early (all tools are web_search, simple prompt) and execute the search directly via Tavily/Perplexity, returning a synthetic Anthropic response. No adapter, no backend LLM call needed. Fixes #21733
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR fixes web search interception for the Key changes:
Minor issues found:
Confidence Score: 4/5
|
| Filename | Overview |
|---|---|
| litellm/integrations/websearch_interception/handler.py | Adds try_short_circuit_search() to detect web-search-only requests and execute the search directly. Provider guard via ProviderConfigManager.get_provider_anthropic_messages_config correctly prevents the short-circuit from firing for bedrock/vertex_ai (which use the full agentic loop). Zero-token synthetic response is a known limitation. |
| litellm/llms/anthropic/experimental_pass_through/messages/handler.py | Adds _try_websearch_short_circuit() called after pre-request hooks. Correctly captures original_stream before hooks convert it, and adds a get_llm_provider() fallback to derive custom_llm_provider when not supplied. Short-circuit return bypasses anthropic_messages_handler but the @client decorator will still fire success hooks with the zero-token synthetic response. |
| tests/test_litellm/integrations/websearch_interception/test_websearch_short_circuit.py | 18 new unit tests covering short-circuit detection, provider guards, streaming vs non-streaming, and entry-point behaviour. All tests use AsyncMock — no real network calls, compliant with the test-folder rule. An empty # Query extraction tests section is a leftover placeholder with no tests. |
Sequence Diagram
sequenceDiagram
participant Client as Claude Code Client
participant AM as anthropic_messages()
participant PRH as _execute_pre_request_hooks()
participant WSI as WebSearchInterceptionLogger
participant SC as try_short_circuit_search()
participant Search as Tavily / Perplexity
participant LLM as Backend LLM (github_copilot)
Client->>AM: POST /v1/messages\n(stream=True, tools=[web_search_20250305])
Note over AM: original_stream = True
AM->>PRH: run pre-request hooks
PRH->>WSI: async_pre_request_hook()
Note over WSI: converts web_search_20250305 → litellm_web_search\nconverts stream=True → stream=False
WSI-->>PRH: modified request_kwargs
PRH-->>AM: request_kwargs (stream=False, tools=[litellm_web_search])
AM->>AM: derive custom_llm_provider via get_llm_provider()
AM->>SC: _try_websearch_short_circuit(stream=original_stream=True)
SC->>WSI: try_short_circuit_search()
Note over WSI: provider in enabled_providers? ✓\nprovider has native AnthropicMessagesConfig? ✗ (github_copilot)\nall tools are web search? ✓\nquery extracted from last user message
WSI->>Search: _execute_search(query)
Search-->>WSI: search result text
WSI-->>SC: synthetic AnthropicMessagesResponse dict
SC->>SC: stream=True → wrap in FakeAnthropicMessagesStreamIterator
SC-->>AM: FakeAnthropicMessagesStreamIterator
AM-->>Client: SSE stream (synthetic response)\n[LLM never called]
Note over LLM: ← Backend LLM bypassed entirely
Last reviewed commit: "fix: guard short-cir..."
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| short_circuit_response = await _try_websearch_short_circuit( | ||
| model=model, | ||
| messages=messages, | ||
| tools=tools, | ||
| custom_llm_provider=custom_llm_provider, | ||
| stream=stream, | ||
| ) | ||
| if short_circuit_response is not None: | ||
| return short_circuit_response |
There was a problem hiding this comment.
The async_pre_request_hook in WebSearchInterceptionLogger converts stream=True to stream=False for providers in enabled_providers (which includes github_copilot). This conversion happens inside _execute_pre_request_hooks() at line 190, before _try_websearch_short_circuit() is called here.
As a result, by the time _try_websearch_short_circuit is called with the extracted stream value (line 215), stream is already False for any github_copilot streaming request. This means:
_try_websearch_short_circuitwill always return a plaindict(never aFakeAnthropicMessagesStreamIterator) for the primary use case this PR targets.- A client that sent
stream=Truewill receive a non-streaming JSON body instead of the expected SSE format, likely causing a protocol error.
The _websearch_interception_converted_stream flag is stored in kwargs but is never checked by _try_websearch_short_circuit. The fix would be to read the original stream value before the hooks run and pass it to the short-circuit:
# Save original stream flag before hooks can convert it
original_stream = stream
request_kwargs = await _execute_pre_request_hooks(
model=model,
messages=messages,
tools=tools,
stream=stream,
custom_llm_provider=custom_llm_provider,
**kwargs,
)
tools = request_kwargs.pop("tools", tools)
stream = request_kwargs.pop("stream", stream)
request_kwargs.pop("litellm_params", None)
kwargs.update(request_kwargs)
short_circuit_response = await _try_websearch_short_circuit(
model=model,
messages=messages,
tools=tools,
custom_llm_provider=custom_llm_provider,
stream=original_stream, # use original, not the hook-converted value
)This bug is not covered by the new tests because test_returns_stream_iterator_when_streaming passes stream=True directly to _try_websearch_short_circuit, bypassing the hook conversion step entirely.
There was a problem hiding this comment.
Fixed — we now save original_stream before hooks run and pass that to the short-circuit.
| short_circuit_response = await _try_websearch_short_circuit( | ||
| model=model, | ||
| messages=messages, | ||
| tools=tools, | ||
| custom_llm_provider=custom_llm_provider, | ||
| stream=stream, |
There was a problem hiding this comment.
custom_llm_provider not updated after pre-request hook extraction
When a caller passes custom_llm_provider=None but embeds the provider in the model string (e.g., model="github_copilot/claude-sonnet-4"), _execute_pre_request_hooks() derives the provider internally and stores it in request_kwargs["litellm_params"]["custom_llm_provider"]. However, that key is immediately popped (request_kwargs.pop("litellm_params", None)) and the outer custom_llm_provider local variable is never updated.
This means _try_websearch_short_circuit receives custom_llm_provider=None, leading to provider_str = "" in try_short_circuit_search, which will never match any entry in enabled_providers, silently skipping the short-circuit.
Since anthropic_messages_handler (called further down) independently calls litellm.get_llm_provider() to derive the provider, it would be straightforward to derive it once here and pass the result to both call sites:
# Extract modified parameters
tools = request_kwargs.pop("tools", tools)
stream = request_kwargs.pop("stream", stream)
# Propagate the provider derived inside the pre-request hooks, if not already set
if not custom_llm_provider:
custom_llm_provider = request_kwargs.get("litellm_params", {}).get(
"custom_llm_provider"
)
request_kwargs.pop("litellm_params", None)
kwargs.update(request_kwargs)There was a problem hiding this comment.
Fixed — we extract custom_llm_provider from litellm_params before popping, with a fallback to litellm.get_llm_provider().
| if not tools: | ||
| return None | ||
|
|
||
| # Check if provider is in enabled list | ||
| provider_str = custom_llm_provider or "" | ||
| if ( | ||
| self.enabled_providers is not None | ||
| and provider_str not in self.enabled_providers | ||
| ): | ||
| return None | ||
|
|
||
| # All tools must be web search tools | ||
| if not all(is_web_search_tool(t) for t in tools): | ||
| return None | ||
|
|
||
| # Extract search query from the last user message | ||
| query = self._extract_search_query(messages) | ||
| if not query: | ||
| return None |
There was a problem hiding this comment.
try_short_circuit_search() is called after _execute_pre_request_hooks() has already run async_pre_request_hook, which converts native web_search_20250305 tools to the litellm_web_search format. At the point this method is called, tools may therefore contain litellm_web_search entries rather than the original native tools.
The is_web_search_tool() check (line 109) correctly recognises both formats, so the detection still works. However, the _extract_search_query() call (line 113) extracts the query from the messages (not the tool inputs), so it's unaffected.
The subtle risk is that on a non-github_copilot provider where the pre-hook does NOT run (tools remain as web_search_20250305), the check still fires — meaning the short-circuit could trigger for any provider whose tools happen to all be web search tools, not just the github_copilot path. The provider guard (provider_str not in self.enabled_providers) prevents this, but it relies entirely on the user having configured enabled_providers correctly. A comment here clarifying that the tools at this point may already be in the litellm_web_search format would reduce future confusion.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Acknowledged — is_web_search_tool() handles both formats correctly, and the provider guard prevents false triggers. No change needed.
|
|
| # Build synthetic Anthropic response | ||
| from uuid import uuid4 | ||
|
|
||
| response: Dict[str, Any] = { | ||
| "id": f"msg_{uuid4().hex[:24]}", | ||
| "type": "message", | ||
| "role": "assistant", | ||
| "model": model, | ||
| "content": [{"type": "text", "text": search_result_text}], | ||
| "stop_reason": "end_turn", | ||
| "stop_sequence": None, | ||
| "usage": {"input_tokens": 0, "output_tokens": 0}, | ||
| } |
There was a problem hiding this comment.
The synthetic response hardcodes input_tokens: 0 and output_tokens: 0. This flows into the @client decorator's success hooks, which use these values for SpendLog entries and rate-limit accounting. For paid providers like github_copilot, every short-circuited search will appear as a zero-cost call in usage dashboards, making it impossible to correlate actual search costs.
The existing _execute_agentic_loop path has the same limitation (it also returns whatever the backend reports), but since the short-circuit completely bypasses the LLM, there's no natural token count to fall back on. Consider at minimum estimating tokens from the query length and result length, or adding a dedicated usage.search_calls field:
# Rough token estimate: 1 token ≈ 4 chars
estimated_input = max(1, len(query) // 4)
estimated_output = max(1, len(search_result_text) // 4)
response: Dict[str, Any] = {
...
"usage": {"input_tokens": estimated_input, "output_tokens": estimated_output},
}There was a problem hiding this comment.
Intentional — no LLM call is made, so 0 tokens is accurate. Search provider costs are tracked separately.
| # All tools must be web search tools | ||
| if not all(is_web_search_tool(t) for t in tools): | ||
| return None |
There was a problem hiding this comment.
enabled_providers includes None-like strings
is_web_search_tool(t) correctly recognises both web_search_20250305-typed tools AND tools already converted by async_pre_request_hook to the litellm_web_search format. However, the short-circuit is only meant for providers that cannot natively handle web search (like github_copilot). For providers that ARE listed in enabled_providers and DO handle the Anthropic agentic loop (e.g., bedrock), hitting this short-circuit would bypass the full agentic loop that these providers rely on.
The root of the issue is that enabled_providers serves double duty: it controls both the existing agentic-loop path and now the new short-circuit path. A bedrock user who configured enabled_providers=["bedrock"] would have their web-search requests short-circuited by this new code, skipping the follow-up LLM call that the agentic loop is designed to produce.
Consider introducing a separate short_circuit_providers list (or a flag like short_circuit_unsupported=True) to distinguish between providers where the existing agentic loop should run versus providers where the short-circuit is appropriate.
There was a problem hiding this comment.
Fixed — the short-circuit now checks get_provider_anthropic_messages_config() and skips providers with native agentic loop support.
Addresses Greptile review feedback: - Save original stream flag before pre-request hooks convert it, so streaming callers get SSE events instead of a plain dict - Propagate custom_llm_provider derived inside _execute_pre_request_hooks when it was not explicitly passed by the caller - Add tests covering both scenarios
- Replace hand-rolled _extract_search_query with existing get_last_user_message from common_utils - Use full UUID (str(uuid.uuid4())) to match codebase convention - Move uuid import to module level per CLAUDE.md
| # Check if provider is in enabled list | ||
| provider_str = custom_llm_provider or "" | ||
| if ( | ||
| self.enabled_providers is not None | ||
| and provider_str not in self.enabled_providers | ||
| ): | ||
| return None |
There was a problem hiding this comment.
enabled_providers), bypassing the agentic loop
The provider guard uses enabled_providers, which defaults to ["bedrock"]. If a user has enabled_providers=["bedrock"] (the default) or enabled_providers=["bedrock", "github_copilot"], any bedrock request whose tools consist solely of web-search tools will hit this short-circuit and receive raw search text — skipping the follow-up LLM call that the existing agentic loop produces.
For bedrock the expected flow is:
- Intercept web-search tool_use in the model response
- Execute the search
- Make a follow-up request to the LLM with the results
- Return the LLM-generated answer
The short-circuit skips step 3 entirely, so callers get raw Tavily/Perplexity text instead of a synthesized answer. This is a behavioural regression for any existing bedrock user who has deployed WebSearchInterceptionLogger.
The PR description says "zero regression risk for Bedrock/Vertex" but the code does not include a guard to restrict short-circuiting to providers that genuinely lack native web-search support (e.g., a separate short_circuit_providers list or a capability flag in model_prices_and_context_window.json).
# Example: introduce a separate set to distinguish short-circuit providers
# from agentic-loop providers
if provider_str not in self.short_circuit_providers:
return NoneThere was a problem hiding this comment.
Fixed — added a guard that checks ProviderConfigManager.get_provider_anthropic_messages_config(). Providers with native support (bedrock, vertex_ai, etc.) skip the short-circuit and use the existing agentic loop.
| # Propagate the provider derived inside pre-request hooks, if not already set | ||
| if not custom_llm_provider: | ||
| custom_llm_provider = request_kwargs.get("litellm_params", {}).get( | ||
| "custom_llm_provider" | ||
| ) |
There was a problem hiding this comment.
custom_llm_provider propagation can still be None when the model string contains the provider
_execute_pre_request_hooks derives custom_llm_provider from the model string internally and stores it in request_kwargs["litellm_params"]["custom_llm_provider"]. The new code reads it back correctly here. However, _execute_pre_request_hooks constructs request_kwargs as:
request_kwargs = {
"tools": tools,
"stream": stream,
"litellm_params": {"custom_llm_provider": custom_llm_provider},
**kwargs, # ← if kwargs contains "litellm_params", this key wins
}If the caller passes a litellm_params key in **kwargs (which is common in the proxy path), the **kwargs expansion overwrites the "litellm_params" key that was explicitly set above, and the derived custom_llm_provider is lost. In that scenario the if not custom_llm_provider: branch here reads back None or the caller-supplied value, not the model-derived value.
The root-cause fix is to merge rather than overwrite in _execute_pre_request_hooks, but at minimum the propagation here should fall back to calling litellm.get_llm_provider(model=model) directly when request_kwargs["litellm_params"] is missing custom_llm_provider:
if not custom_llm_provider:
custom_llm_provider = request_kwargs.get("litellm_params", {}).get(
"custom_llm_provider"
)
if not custom_llm_provider:
try:
_, custom_llm_provider, _, _ = litellm.get_llm_provider(model=model)
except Exception:
passThere was a problem hiding this comment.
Fixed — added fallback to litellm.get_llm_provider(model=model) when litellm_params extraction returns None.
- Skip short-circuit for providers that have a BaseAnthropicMessagesConfig (bedrock, vertex_ai, azure_ai, anthropic) — they use the agentic loop which includes a follow-up LLM synthesis step. Short-circuiting would return raw search text instead of an LLM-synthesized answer. - Add fallback to litellm.get_llm_provider() for custom_llm_provider derivation when litellm_params is overwritten by kwargs. - Add test for bedrock guard. Addresses Greptile review comments #3 and #4.
61dde5e
into
BerriAI:litellm_oss_staging_03_19_2026
…uit-copilot fix: short-circuit websearch for github_copilot provider
2 participants
Summary
Fixes #21733 —
websearch_interceptiondoesn't work withgithub_copilotprovider.Root cause: Claude Code sends web search as a separate, standalone
/v1/messagesrequest (simple prompt, singleweb_search_20250305tool). Forgithub_copilot, this request falls to the adapter path which:web_search_options: {}→ Copilot API ignores itFix: Detect web-search-only requests early at the
/v1/messagesentry point and short-circuit: extract the query, call Tavily/Perplexity directly via the existing_execute_search()method, and return a syntheticAnthropicMessagesResponse. No adapter, no backend LLM call, no agentic loop.This approach:
github_copilotand any other non-Anthropic provider_execute_search()(search provider resolution) andFakeAnthropicMessagesStreamIterator(SSE wrapping)stream: trueandstream: falseChanges
litellm/integrations/websearch_interception/handler.pytry_short_circuit_search()+_extract_search_query()litellm/llms/anthropic/experimental_pass_through/messages/handler.py_try_websearch_short_circuit(), called before adapter dispatchtests/.../test_websearch_short_circuit.pyTest plan
github_copilotprovider + Tavily + Claude Code web search