fix(websearch_interception): end the turn when the agentic loop hits its ceiling - #37911
Conversation
…its ceiling When the bounded loop cap or the repeated tool-call fingerprint guard refused a rerun, the raise escaped the parent agentic frame and the client got the raw model turn back: HTTP 200 carrying an unresolved tool_use block for the internal litellm_web_search tool and stop_reason "tool_use". The client never declared that tool, so it had no way to answer it and the conversation could not continue The safety check now raises AgenticLoopSafetyError, a ValueError subclass, and _call_agentic_completion_hooks catches it and returns a finalized response: the blocks belonging to the refused tool calls are dropped, and stop_reason is closed out to end_turn when nothing the client declared is still waiting. Refused blocks are matched by the ids and names of the tool calls the rail refused rather than by hardcoding the web search tool name Only the non-streaming anthropic messages path ends the turn this way. A streaming caller has already sent the original message by the time the hooks run, so a finalized turn would arrive as a second message rather than replace the first, and the responses surface carries a pydantic model this finalizer does not rewrite. Both keep raising, exactly as they did before Also adds max_agentic_loops to websearch_interception_params so the ceiling can be set once for the whole feature. A per deployment litellm_params.max_agentic_loops still wins over it, and the field stays on the proxy's untrusted root list so a client cannot raise its own ceiling
Greptile SummaryThis PR makes capped Anthropic Messages web-search loops return a terminal response instead of leaking LiteLLM’s internal search call.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/llms/custom_httpx/llm_http_handler.py | Finalizes capped Anthropic Messages turns by removing refused internal tool calls while retaining client-owned tool calls and existing behavior on unsupported surfaces. |
| litellm/integrations/websearch_interception/handler.py | Adds validated feature-level loop configuration and applies it when a deployment-specific ceiling is absent. |
| litellm/litellm_core_utils/agentic_loop_settings.py | Centralizes loop-ceiling coercion, defaults, and validation while explicitly rejecting booleans and values below one. |
| litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py | Emits matching content-block starts for otherwise unrecognized Anthropic blocks, preventing malformed reconstructed streams. |
| litellm/proxy/proxy_server.py | Validates per-deployment loop ceilings during proxy configuration loading. |
| tests/test_litellm/integrations/websearch_interception/test_websearch_agentic_loop_cap.py | Covers loop termination, internal-versus-client tool filtering, reconstructed streaming, configuration precedence, and validation behavior. |
Reviews (3): Last reviewed commit: "fix: keep accepting a loop ceiling that ..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
ARCHITECTURE.md promised the clean end_turn for every intercepted request. A request that streams all the way through and one on /v1/responses both still hand back the internal tool call, so say that plainly instead. Also note that where the refused call was the only block left, the turn can come back with no text in it. On AgenticLoopSafetyError, note that the chat completions loop still raises a plain ValueError from its own copy of the rails, so nobody writes an except for this type expecting it to cover that surface too.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 7c02c08. Configure here.
|
Mateo Wang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
…itellm_fix_agentic_loop_cap_response
A capped turn on a streaming request is rebuilt into SSE by FakeAnthropicMessagesStreamIterator. It emitted content_block_stop for every block but content_block_start only for text, thinking, redacted_thinking and tool_use, so a web search turn's server_tool_use and web_search_tool_result blocks produced stops with no matching start. Anthropic's SDK accumulator appends on content_block_start and then indexes content[event.index] on content_block_delta, so the orphan stops shifted every later index and client.messages.stream() raised IndexError on the text block. Unknown block types now pass through with a start of their own, which keeps position equal to index. Also corrects two claims that said no current caller reaches the loop with stream=True. AgenticStreamingIterator does, and it keeps raising, because its events are already on the wire.
The ceiling was only checked at the feature level, on
litellm_settings.websearch_interception_params. The per-deployment
litellm_params.max_agentic_loops, which wins over it, went straight into
int(kwargs.get("max_agentic_loops", 3) or 3), so a 0 was swallowed by the
falsy fallback and read as the default 3. Asking for the tightest ceiling
handed you the loosest one. A non-integer booted the proxy and then failed
every request to that model with "invalid literal for int() with base 10".
Both settings now share one validator, which names the field it rejected,
and the per-deployment value is checked while the model list is read at
startup so a bad value stops the proxy rather than surfacing per request.
The check sits in load_config rather than on LiteLLM_Params because the
proxy builds its router with ignore_invalid_deployments=True, where a
validation error drops the deployment silently instead of refusing to
start. This is the same placement the complexity_router_config plugin
check already uses.
Chat completions read the same key through a separate path that turned 0
into 1 and true into a ceiling of 1, so it now shares the validator too
and the key means one thing on both surfaces.
The ceiling used to go through `int(... or 3)`, so anything `int()` accepted worked. Tightening the new shared validator to `isinstance(int)` turned a config that boots today into a proxy that refuses to start, because `max_agentic_loops: os.environ/MAX_AGENTIC_LOOPS` is resolved to a string before it reaches either check, and a YAML-quoted "5" is a string too. Accept ints, integral floats, and strings that parse to a whole number. Keep refusing bools, fractional floats, words, and anything below 1.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit b103edb. Configure here.
TLDR
Problem this solves:
How it solves it:
/v1/messagesturn now ends withstop_reason: end_turnlitellm_web_searchblock is dropped from the responsetool_usesurvives, since blocks are matched by idmax_agentic_loopssetting on the feature, validated at proxy startupUser Flow
Before
The turn dies inside the client, and the developer never gets an answer.
max_agentic_loops: 5underlitellm_settings.websearch_interception_params; the proxy starts fine and the key does nothing, so the ceiling stays at the default of 3 unless they repeat it under every model'slitellm_params. A per-model value the loop cannot honor,max_agentic_loops: 0, also starts fine and quietly runs at 3POST https://litellm-proxy/v1/messageswith"stream": true,"model": "claude-sonnet-4-5"and"tools": [{"type": "web_search_20250305", "name": "web_search"}], then iterates the streamed eventsmessage_start, and then the Anthropic SDK raisesIndexError: list index out of rangewhile assembling the message, 6.1 seconds in. The stream closed content blocks it never opened. The app dies with a stack trace and no answerPOST https://litellm-proxy/v1/messagesreturnsHTTP 200, so it looks like it worked"stop_reason": "tool_use"and a content block{"type": "tool_use", "id": "toolu_01...", "name": "litellm_web_search", "input": {"query": "..."}}. The app never declared a tool by that name, so it cannot run anything or hand back a matchingtool_resultPOST https://litellm-proxy/v1/messageswith atool_resultHTTP 200comes back with anotherlitellm_web_searchcall, so a client that loops whilestop_reasonistool_usenever exitsPOST /v1/messagesrequests and ends in an apology rather than an answer, with nothing to say the proxy had stopped searching on purposeAfter
The turn ends on a real answer, and the conversation carries on.
max_agentic_loops: 5underlitellm_settings.websearch_interception_paramsnow applies to every intercepted model, and a value the loop cannot honor (0,-1,"five",true) stops the proxy at startup with a message naming the key instead of being quietly ignored. A per-modellitellm_params.max_agentic_loopsstill wins where it is set, and is checked the same wayPOST https://litellm-proxy/v1/messagesrequest"stop_reason": "end_turn", and no block namedlitellm_web_searchappears anywhere in the body, so there is nothing left for the app to answer. The reply can be thinner than it would have been with more rounds, since the searching stopped at the ceiling, andmax_agentic_loopsis the knob for thatPOST https://litellm-proxy/v1/messagesreturnsHTTP 200in 1.9 seconds with a plain answer: "Based on my search results, the cheapest on-demand H100 price I found was $2.00 per hour from GMI Cloud.""stop_reason": "tool_use"carrying their ownrecord_findingcall, not the proxy'sRelevant issues
Related to #35334, which reports the same leaked
litellm_web_searchblock on the same endpoint but from adifferent trigger, a forced
tool_choicesurviving into the follow-up call. This PR removes the leak once theceiling trips, so that symptom stops being reachable, but the
tool_choiceinheritance it names is untouchedand still worth fixing on its own
Linear ticket
Resolves LIT-5673
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Shared setup for every case below. Two proxies, one per commit, each in its own worktree with its own venv on
its own random high port, against a real Bedrock
claude-sonnet-4-5deployment and a real Tavily searchbackend. Real provider calls, real spend, no mocks and no source edits on either side. The client is Anthropic's
Python SDK, the same one the reporting app uses. The re-stamped runs at the head commit ran the proxy with
--num_workers 2, so every result below is from a multi-worker topology rather than a single process.The prompt is the User Flow's question with explicit per-provider search instructions appended, so the model
keeps searching past the ceiling instead of stopping at three rounds on its own.
Before (76d4627)
Capped non-streaming /v1/messages
HTTP 200with"stop_reason": "tool_use"and the internal call leaked into the body:web_search, so it has no tool by that name and cannot produce atool_resultContinuing the conversation
The max_agentic_loops setting
max_agentic_loops: 5underlitellm_settings.websearch_interception_paramsand send the prompt"stop_reason": "tool_use"carrying the leaklitellm_paramsand send the same prompt"stop_reason": "end_turn"and no leak, so only the per deployment key ever workedmax_agentic_loops: "five"and observe the proxy start healthy, the bad value silently droppedInterceptor-converted stream at the ceiling
"stream": true, which interception converts to non-streamingcontent_block_startnaminglitellm_web_searchat index 3/v1/responses at the ceiling
POST http://127.0.0.1:<port>/v1/responsesand trip the ceilingHTTP 200whoseoutputcarries{"type": "function_call", "name": "litellm_web_search", ...}with an empty
output_textA streaming regression this PR introduced, and fixed
Worth stating plainly rather than burying: the first version of this fix broke the streaming path, and review
caught it. Dropping the internal
tool_useblock means a capped turn now ends inserver_tool_useandweb_search_tool_resultblocks, and the fake stream iterator had nocontent_block_startbranch for either,while emitting
content_block_stopfor both. Orphan stops. The Anthropic SDK's accumulator appends on start andindexes by
event.indexon delta, so the indexes no longer line up and it walks off the end of the list.At
6760379b4a,client.messages.stream()against a capped turn dies 6.1 seconds in having seen exactly oneevent:
At
0485b3fcd4, the same request against the same 2-worker proxy on the same port:Every start is now paired with a stop, and the ceiling did fire on that run: the proxy logged
Exceeded max_agentic_loops=1and exactly one Tavily search went out. The fix is a passthroughcontent_block_startfor any block type the iterator does not special-case, so an unrecognised block can nolonger produce a stop without a start. Four tests cover it; each fails with the fix stashed.
After (19e077a)
Real Claude Code against the proxy
WebSearchwhose query needs sixlookups: "on-demand hourly H100 GPU price, look up AWS first, then Google Cloud, then Azure, then Oracle
Cloud, then Lambda Labs, then CoreWeave, searching each provider separately one at a time and not answering
until all six are searched"
wall clock = 31s, fourPOST /v1/messagesseen by the proxy,with the cap confirmed fired:
1 Exceeded max_agentic_loops=1searches=1with notrip. Claude Code sends one request per search round, so depth resets each turn; what trips the ceiling from
a real client is a single
WebSearchwhose own query needs several lookupsCapped non-streaming /v1/messages
Continuing the conversation
HTTP 400before:A client's own tool still survives the cap
record_findingtoolThe ceiling holds across workers
The ceiling is read once at config load and the depth counter rides in the request, so there is no per-process
state for two workers to split. Measured rather than argued: ten capped requests across a 2-worker proxy, with
per-PID CPU-time deltas confirming both workers served traffic.
Every one of the ten came back
stop_reason=end_turnwithleak=Falseand a loggedExceeded max_agentic_loops=<ceiling>.The max_agentic_loops setting
Boot with
max_agentic_loops: 5underlitellm_settings.websearch_interception_paramsand send the promptObserve the key take effect on every intercepted model, measured by search count and token spend:
Observe the per model
1win over the feature-level5, which is the precedence this PR claims. Re-verifiedat the head commit with feature-level
6against per-model1:HTTP 200in 9 seconds, exactly one Tavilysearch,
['server_tool_use', 'web_search_tool_result', 'text'],stop_reason: end_turn, zerotool_useblocks and zero occurrences of
litellm_web_search. The trip is visible to the developer without reading alog, since the text ends on the model announcing the search the ceiling then denied it: "Let me search for
GMI Cloud's founding year as it appears to offer the lowest stable on-demand rate at $2.00/hr"
Boot with each rejected feature-level value and observe the proxy refuse to start, with nothing ever served.
The messages are unchanged at the head commit after the validator moved behind the shared function:
The same check on a per-deployment ceiling
Put each rejected value on a single model's
litellm_paramsand boot the 2-worker proxyObserve the proxy refuse to start, each message logged once per worker and naming the offending model:
Confirm nothing is served rather than trusting the logs: both workers print
Application startup failed. Exiting., the parent printsChild process failed to start, stopping the parent process, and pollingGET /health/livelinessonce a second across the whole boot window returns 24 consecutive no-responses withnothing left listening on the port afterward. The old "boots anyway and quietly runs at 3" outcome is
unreachable
Boot the happy path with no
max_agentic_loopsanywhere and observe the default ceiling still at 3: a 5-deepchain makes exactly 3 Tavily calls and logs
Exceeded max_agentic_loops=3, while an ordinary questionanswers normally in 8 seconds with one search and no trip
One honest wrinkle on the exit code, which is not this PR's doing. A startup failure exits 3 at one worker but
exits 0 at two, so an orchestrator reads a typo'd ceiling as a clean shutdown. Three controls pin it to the
worker count rather than to this change: per-model
0at one worker exits 3, feature-level0at two workersexits 0, feature-level
0at one worker exits 3. Identical on both validation paths, so it is uvicorn'smultiprocess supervisor swallowing the children's status, not anything added here. Filed separately as #37948.
Nothing is served in either case, so the safety property this PR is claiming holds; it is the exit code an
orchestrator sees that is wrong.
Interceptor-converted stream at the ceiling
Covered in the regression section above: at the head commit the rebuilt stream pairs every
content_block_startwith itscontent_block_stop, the Anthropic SDK accumulates the turn without raising,and
litellm_web_searchappears nowhere in it./v1/responses at the ceiling
POST http://127.0.0.1:<port>/v1/responsesand trip the ceilingHTTP 200whoseoutputstill carries{"type": "function_call", "name": "litellm_web_search"},unchanged from before, since the terminal turn is gated on the anthropic messages surface
Deviations
Two, both stated rather than papered over.
The
/v1/responsescase had to run against direct Anthropic rather than Bedrock, because that surface bridgesto the Bedrock converse handler, which ignores
AWS_BEARER_TOKEN_BEDROCKand falls through to an expired localAWS SSO profile. Every other case ran on Bedrock as written.
The re-stamped runs at the head commit ran with no Postgres attached:
DATABASE_URLunset,STORE_MODEL_IN_DB=False, all config from YAML, which both workers read identically. This was not a freechoice. The local Postgres is saturated at 108 connections against
max_connections=100, held by otherlong-lived proxies on this machine (39 idle, the oldest idle two days), and a 2-worker boot with the DB attached
dies on
FATAL: sorry, too many clients alreadywithChild process failed to start. Raisingmax_connectionsor killing other backends was not mine to do. The surface under test holds no DB-backed state, so a shared
database would not have exercised anything the YAML path does not.
Where the run is anchored
The live run above is anchored at
19e077ab510240a3d0c9993e27e8d635fff6d318. One commit has landed since,b103edb588, which only widens what the ceiling validator accepts: a value that spells a whole number, such asthe string an
os.environ/ceiling resolves to, is read rather than refused. It rejects strictly less thanbefore and changes nothing about a capped turn. Every rejection message quoted above is byte-identical at the
new tip, asserted by unit tests rather than by eye:
Type
🐛 Bug Fix
🆕 New Feature
Caveats (if any)
max_agentic_loops: 6sent six searches and came back carrying only round 1'sweb_search_tool_result, with no refusals involved. That is how the agentic loop has always assembled its final response, ceiling or no ceiling, and this PR leaves it alone. It is worth naming anyway, because paying for six searches and seeing one result set surprises peopletool_useblock, so nothing is left over when that block was all the model had emittedmax_agentic_loopsat startup is a behavior change. A deployment carryingmax_agentic_loops: 0boots today and quietly runs at the default ceiling of 3; after this change the proxy refuses to start until the value is fixed. That is deliberate, since 0 currently hands the loosest ceiling to whoever asked for the tightest, but it does mean an existing config can stop booting on upgrade. Only values the loop cannot honor are refused: anything spelling a whole number is still accepted, including the string anos.environ/ceiling resolves to, so a config that works today keeps working unless its ceiling was already meaninglessPOST /model/newor a DB reload skips config load, so a bad ceiling there is caught at request time instead, with an error naming the field. Validating in those two endpoints is a follow-up/v1/responsesstill returns the internal call at the ceiling, unchanged either way, since the terminal turn is gated on the anthropic messages surface/v1/chat/completionskeeps its own rails and still raises at the ceiling. Moving that surface over is blocked on fix(agentic loop): return a real stream when code-interpreter interception converts the request #37657Final Attestation
Note
Medium Risk
Changes agentic-loop control flow and
/v1/messagesresponse shape when the loop cap trips, plus startup validation that can refuse to boot on a badmax_agentic_loops./v1/responsesand chat completions still raise instead of finalizing.Overview
When web-search interception hits
max_agentic_loops(or a repeated tool-call fingerprint), a non-streaming/v1/messagesturn now ends instead of leaking the internallitellm_web_searchtool_useblock. Refused blocks are dropped,stop_reasonbecomesend_turnunless a client-declared tool remains, and interceptor-converted streams are rebuilt from that finalized turn.max_agentic_loopscan be set onwebsearch_interception_params(default still 3) or per deployment, with the deployment value winning. Both knobs go through shared validation at config load: values that are not a whole number ≥ 1 stop the proxy rather than being read as 3 or failing at request time. Env-sourced string ceilings like"5"still work.The fake Anthropic stream iterator now emits
content_block_startfor unrecognized blocks (server_tool_use,web_search_tool_result), so capped SSE no longer closes blocks it never opened./v1/responsesand chat-completions still raise at the ceiling; live streaming (AgenticStreamingIterator) is unchanged.Reviewed by Cursor Bugbot for commit b103edb. Bugbot is set up for automated code reviews on this repo. Configure here.