Skip to content

fix(websearch_interception): end the turn when the agentic loop hits its ceiling - #37911

Merged
mateo-berri merged 9 commits into
litellm_internal_stagingfrom
litellm_fix_agentic_loop_cap_response
Aug 22, 2026
Merged

fix(websearch_interception): end the turn when the agentic loop hits its ceiling#37911
mateo-berri merged 9 commits into
litellm_internal_stagingfrom
litellm_fix_agentic_loop_cap_response

Conversation

@mateo-berri

@mateo-berri mateo-berri commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • A capped search loop returns an internal tool call clients cannot answer
  • Answering that call returns another one, so a retrying client never exits
  • The loop ceiling was settable per deployment only, and undocumented

How it solves it:

  • A capped non-streaming /v1/messages turn now ends with stop_reason: end_turn
  • The internal litellm_web_search block is dropped from the response
  • A client's own tool_use survives, since blocks are matched by id
  • New max_agentic_loops setting on the feature, validated at proxy startup

User Flow

Before

The turn dies inside the client, and the developer never gets an answer.

  1. A platform engineer runs a LiteLLM proxy with web search interception turned on for their Bedrock Claude models, pointed at their search provider, and rolls Claude Code out to their developers against it. They try to raise the search loop's ceiling with max_agentic_loops: 5 under litellm_settings.websearch_interception_params; the proxy starts fine and the key does nothing, so the ceiling stays at the default of 3 unless they repeat it under every model's litellm_params. A per-model value the loop cannot honor, max_agentic_loops: 0, also starts fine and quietly runs at 3
  2. A developer points their app at the proxy and asks a question whose answer needs a chain of searches, each query built out of the previous answer so the model cannot batch them: cheapest on-demand H100 provider, then that company's founding year, then that year's flagship NVIDIA data-center GPU, and so on
  3. Their app calls POST https://litellm-proxy/v1/messages with "stream": true, "model": "claude-sonnet-4-5" and "tools": [{"type": "web_search_20250305", "name": "web_search"}], then iterates the streamed events
  4. The proxy runs the searches on the developer's behalf and asks the model again after each round, up to the ceiling, and the model asks for one more search after the last allowed round
  5. One event arrives, message_start, and then the Anthropic SDK raises IndexError: list index out of range while assembling the message, 6.1 seconds in. The stream closed content blocks it never opened. The app dies with a stack trace and no answer
  6. The developer turns streaming off and retries. POST https://litellm-proxy/v1/messages returns HTTP 200, so it looks like it worked
  7. The body carries "stop_reason": "tool_use" and a content block {"type": "tool_use", "id": "toolu_01...", "name": "litellm_web_search", "input": {"query": "..."}}. The app never declared a tool by that name, so it cannot run anything or hand back a matching tool_result
  8. Their app does what the API contract says and answers it, sending another POST https://litellm-proxy/v1/messages with a tool_result
  9. HTTP 200 comes back with another litellm_web_search call, so a client that loops while stop_reason is tool_use never exits
  10. The same question in Claude Code against the same proxy spends about 31 seconds and four POST /v1/messages requests and ends in an apology rather than an answer, with nothing to say the proxy had stopped searching on purpose
  11. Every one of those extra round trips spends another upstream model call against the developer's own budget, for as long as their client keeps retrying

After

The turn ends on a real answer, and the conversation carries on.

  1. Same setup, except max_agentic_loops: 5 under litellm_settings.websearch_interception_params now applies to every intercepted model, and a value the loop cannot honor (0, -1, "five", true) stops the proxy at startup with a message naming the key instead of being quietly ignored. A per-model litellm_params.max_agentic_loops still wins where it is set, and is checked the same way
  2. The same question from the same developer
  3. The same POST https://litellm-proxy/v1/messages request
  4. The proxy runs the searches on the developer's behalf and asks the model again after each round, up to the ceiling, and the model asks for one more search after the last allowed round
  5. Every content block is opened before it is closed, so the SDK assembles the message in 7.9 seconds and the app receives the whole turn: the search the proxy ran, its results, and 387 characters of written answer
  6. The turn ends on "stop_reason": "end_turn", and no block named litellm_web_search appears anywhere in the body, so there is nothing left for the app to answer. The reply can be thinner than it would have been with more rounds, since the searching stopped at the ceiling, and max_agentic_loops is the knob for that
  7. The developer asks a follow-up on top of that turn. POST https://litellm-proxy/v1/messages returns HTTP 200 in 1.9 seconds with a plain answer: "Based on my search results, the cheapest on-demand H100 price I found was $2.00 per hour from GMI Cloud."
  8. When their app does declare a tool of its own, that tool still comes back for them to run: the response ends on "stop_reason": "tool_use" carrying their own record_finding call, not the proxy's
  9. The same question in Claude Code finishes the turn instead of retrying, and the developer keeps chatting in the same session
  10. One question now costs a bounded number of upstream model calls, because the client is never handed something it has to answer to make progress

Relevant issues

Related to #35334, which reports the same leaked litellm_web_search block on the same endpoint but from a
different trigger, a forced tool_choice surviving into the follow-up call. This PR removes the leak once the
ceiling trips, so that symptom stops being reachable, but the tool_choice inheritance it names is untouched
and still worth fixing on its own

Linear ticket

Resolves LIT-5673

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for every case below. Two proxies, one per commit, each in its own worktree with its own venv on
its own random high port, against a real Bedrock claude-sonnet-4-5 deployment and a real Tavily search
backend. Real provider calls, real spend, no mocks and no source edits on either side. The client is Anthropic's
Python SDK, the same one the reporting app uses. The re-stamped runs at the head commit ran the proxy with
--num_workers 2, so every result below is from a multi-worker topology rather than a single process.

model_list:
  - model_name: claude-sonnet-4-5
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
      aws_region_name: us-west-2
  - model_name: claude-sonnet-4-5-permodel
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0
      aws_region_name: us-west-2
      max_agentic_loops: 1

search_tools:
  - search_tool_name: tavily-search
    litellm_params:
      search_provider: tavily
      api_key: os.environ/TAVILY_API_KEY

litellm_settings:
  callbacks: ["websearch_interception"]
  websearch_interception_params:
    enabled_providers: ["bedrock"]
    search_tool_name: "tavily-search"
    max_agentic_loops: 5

The prompt is the User Flow's question with explicit per-provider search instructions appended, so the model
keeps searching past the ceiling instead of stopping at three rounds on its own.

Before (76d4627)

Capped non-streaming /v1/messages

  1. Send the request:
    POST http://127.0.0.1:<port>/v1/messages
    {"model":"claude-sonnet-4-5","max_tokens":4096,
     "tools":[{"type":"web_search_20250305","name":"web_search"}],
     "messages":[{"role":"user","content":"compare current on-demand H100 pricing ..."}]}
    
  2. Observe HTTP 200 with "stop_reason": "tool_use" and the internal call leaked into the body:
    {"type":"tool_use","id":"toolu_01...","name":"litellm_web_search",
     "input":{"query":"Azure ND96isr H100 v5 on-demand pricing current 2024"}}
    
  3. The app declared only web_search, so it has no tool by that name and cannot produce a tool_result

Continuing the conversation

  1. Append that assistant turn verbatim and send the next message on the same history
  2. Observe the conversation refuse to continue:
    HTTP 400
    messages.1: `tool_use` ids were found without `tool_result` blocks immediately after
    
  3. The only way forward is to discard the turn and start over

The max_agentic_loops setting

  1. Boot with max_agentic_loops: 5 under litellm_settings.websearch_interception_params and send the prompt
  2. Observe the key ignored: the response still comes back "stop_reason": "tool_use" carrying the leak
  3. Move the same value to the model's litellm_params and send the same prompt
  4. Observe "stop_reason": "end_turn" and no leak, so only the per deployment key ever worked
  5. Boot with max_agentic_loops: "five" and observe the proxy start healthy, the bad value silently dropped

Interceptor-converted stream at the ceiling

  1. Send the same request with "stream": true, which interception converts to non-streaming
  2. Observe the stream carry content_block_start naming litellm_web_search at index 3

/v1/responses at the ceiling

  1. Send a web search request to POST http://127.0.0.1:<port>/v1/responses and trip the ceiling
  2. Observe HTTP 200 whose output carries {"type": "function_call", "name": "litellm_web_search", ...}
    with an empty output_text

A streaming regression this PR introduced, and fixed

Worth stating plainly rather than burying: the first version of this fix broke the streaming path, and review
caught it. Dropping the internal tool_use block means a capped turn now ends in server_tool_use and
web_search_tool_result blocks, and the fake stream iterator had no content_block_start branch for either,
while emitting content_block_stop for both. Orphan stops. The Anthropic SDK's accumulator appends on start and
indexes by event.index on delta, so the indexes no longer line up and it walks off the end of the list.

At 6760379b4a, client.messages.stream() against a capped turn dies 6.1 seconds in having seen exactly one
event:

  RESULT                 : RAISED after 6.13s
  event sequence         : ['message_start']
  File ".../anthropic/lib/streaming/_messages.py", line 513, in accumulate_event
    content_block = current_snapshot.content[event.index]
IndexError: list index out of range

At 0485b3fcd4, the same request against the same 2-worker proxy on the same port:

  RESULT                 : accumulated OK in 7.90s
  accumulated block types: ['server_tool_use', 'web_search_tool_result', 'text']
  stop_reason            : end_turn
  litellm_web_search leak: False
  text chars             : 387
  event sequence         : ['message_start', 'content_block_start', 'content_block_stop',
                            'content_block_start', 'content_block_stop', 'content_block_start',
                            'content_block_delta', 'text', 'content_block_stop', 'message_delta',
                            'message_stop']

Every start is now paired with a stop, and the ceiling did fire on that run: the proxy logged
Exceeded max_agentic_loops=1 and exactly one Tavily search went out. The fix is a passthrough
content_block_start for any block type the iterator does not special-case, so an unrecognised block can no
longer produce a stop without a start. Four tests cover it; each fails with the fix stashed.

After (19e077a)

Real Claude Code against the proxy

  1. Run Claude Code under tmux pointed at the proxy at ceiling 1, and ask one WebSearch whose query needs six
    lookups: "on-demand hourly H100 GPU price, look up AWS first, then Google Cloud, then Azure, then Oracle
    Cloud, then Lambda Labs, then CoreWeave, searching each provider separately one at a time and not answering
    until all six are searched"
  2. Observe the turn finish rather than retry, wall clock = 31s, four POST /v1/messages seen by the proxy,
    with the cap confirmed fired: 1 Exceeded max_agentic_loops=1
  3. Note that the three-provider version of the same question never reaches the ceiling, searches=1 with no
    trip. Claude Code sends one request per search round, so depth resets each turn; what trips the ceiling from
    a real client is a single WebSearch whose own query needs several lookups

Capped non-streaming /v1/messages

  1. Send the capped request and observe it terminate cleanly:
    HTTP_STATUS=200 TIME=6.395819s BYTES=8352
      search rounds observed : 1
      stop_reason            : end_turn
      content types          : ['server_tool_use', 'web_search_tool_result', 'text']
      litellm_web_search leak: False
      has non-empty text     : True
    
    trip: Exceeded max_agentic_loops=1
    

Continuing the conversation

  1. Append that assistant turn verbatim and send the next message on the same history
  2. Observe the follow-up succeed where the same replay returned HTTP 400 before:
    follow-up turn: HTTP 200 in 1.93s
      content types: ['text']
      stop_reason  : end_turn
      answer       : 'Based on my search results, the cheapest on-demand H100 price I found
                      was **$2.00 per hour from GMI Cloud**.'
      leak         : False
    

A client's own tool still survives the cap

  1. Send the same capped request with a client-declared record_finding tool
  2. Observe the client's tool come back for it to run, while the internal one stays gone:
    HTTP_STATUS=200 TIME=6.589382s
      stop_reason  : tool_use
      content types: ['server_tool_use', 'web_search_tool_result', 'text', 'tool_use']
      [3] tool_use  name='record_finding'  id=toolu_bdrk_0116ep3Z8mu9mBei34cWrzSn
                    input={"provider": "GMI Cloud", "usd_per_hour": 2.0}
      leak         : False
    
  3. Blocks are matched by id, so dropping the internal call cannot take a client's own call with it

The ceiling holds across workers

The ceiling is read once at config load and the depth counter rides in the request, so there is no per-process
state for two workers to split. Measured rather than argued: ten capped requests across a 2-worker proxy, with
per-PID CPU-time deltas confirming both workers served traffic.

Configured ceiling Requests Searches per request Deviation
3 6 3 none
1 4 1 none

Every one of the ten came back stop_reason=end_turn with leak=False and a logged
Exceeded max_agentic_loops=<ceiling>.

The max_agentic_loops setting

  1. Boot with max_agentic_loops: 5 under litellm_settings.websearch_interception_params and send the prompt

  2. Observe the key take effect on every intercepted model, measured by search count and token spend:

    Config Searches Input tokens
    No key set 3 9181
    Feature-level 5 5 14319
    Feature 5, per model 1 2 3317
  3. Observe the per model 1 win over the feature-level 5, which is the precedence this PR claims. Re-verified
    at the head commit with feature-level 6 against per-model 1: HTTP 200 in 9 seconds, exactly one Tavily
    search, ['server_tool_use', 'web_search_tool_result', 'text'], stop_reason: end_turn, zero tool_use
    blocks and zero occurrences of litellm_web_search. The trip is visible to the developer without reading a
    log, since the text ends on the model announcing the search the ceiling then denied it: "Let me search for
    GMI Cloud's founding year as it appears to offer the lowest stable on-demand rate at $2.00/hr"

  4. Boot with each rejected feature-level value and observe the proxy refuse to start, with nothing ever served.
    The messages are unchanged at the head commit after the validator moved behind the shared function:

    TypeError:  websearch_interception_params.max_agentic_loops must be an integer, got 'three'
    ValueError: websearch_interception_params.max_agentic_loops must be at least 1, got 0
    ValueError: websearch_interception_params.max_agentic_loops must be at least 1, got -1
    TypeError:  websearch_interception_params.max_agentic_loops must be an integer, got True
    

The same check on a per-deployment ceiling

  1. Put each rejected value on a single model's litellm_params and boot the 2-worker proxy

  2. Observe the proxy refuse to start, each message logged once per worker and naming the offending model:

    ValueError: litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be at least 1, got 0
    TypeError:  litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be an integer, got 'three'
    ValueError: litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be at least 1, got -1
    TypeError:  litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be an integer, got True
    
  3. Confirm nothing is served rather than trusting the logs: both workers print Application startup failed. Exiting., the parent prints Child process failed to start, stopping the parent process, and polling
    GET /health/liveliness once a second across the whole boot window returns 24 consecutive no-responses with
    nothing left listening on the port afterward. The old "boots anyway and quietly runs at 3" outcome is
    unreachable

  4. Boot the happy path with no max_agentic_loops anywhere and observe the default ceiling still at 3: a 5-deep
    chain makes exactly 3 Tavily calls and logs Exceeded max_agentic_loops=3, while an ordinary question
    answers normally in 8 seconds with one search and no trip

One honest wrinkle on the exit code, which is not this PR's doing. A startup failure exits 3 at one worker but
exits 0 at two, so an orchestrator reads a typo'd ceiling as a clean shutdown. Three controls pin it to the
worker count rather than to this change: per-model 0 at one worker exits 3, feature-level 0 at two workers
exits 0, feature-level 0 at one worker exits 3. Identical on both validation paths, so it is uvicorn's
multiprocess supervisor swallowing the children's status, not anything added here. Filed separately as #37948.
Nothing is served in either case, so the safety property this PR is claiming holds; it is the exit code an
orchestrator sees that is wrong.

Interceptor-converted stream at the ceiling

Covered in the regression section above: at the head commit the rebuilt stream pairs every
content_block_start with its content_block_stop, the Anthropic SDK accumulates the turn without raising,
and litellm_web_search appears nowhere in it.

/v1/responses at the ceiling

  1. Send a web search request to POST http://127.0.0.1:<port>/v1/responses and trip the ceiling
  2. Observe HTTP 200 whose output still carries {"type": "function_call", "name": "litellm_web_search"},
    unchanged from before, since the terminal turn is gated on the anthropic messages surface

Deviations

Two, both stated rather than papered over.

The /v1/responses case had to run against direct Anthropic rather than Bedrock, because that surface bridges
to the Bedrock converse handler, which ignores AWS_BEARER_TOKEN_BEDROCK and falls through to an expired local
AWS SSO profile. Every other case ran on Bedrock as written.

The re-stamped runs at the head commit ran with no Postgres attached: DATABASE_URL unset,
STORE_MODEL_IN_DB=False, all config from YAML, which both workers read identically. This was not a free
choice. The local Postgres is saturated at 108 connections against max_connections=100, held by other
long-lived proxies on this machine (39 idle, the oldest idle two days), and a 2-worker boot with the DB attached
dies on FATAL: sorry, too many clients already with Child process failed to start. Raising max_connections
or killing other backends was not mine to do. The surface under test holds no DB-backed state, so a shared
database would not have exercised anything the YAML path does not.

Where the run is anchored

The live run above is anchored at 19e077ab510240a3d0c9993e27e8d635fff6d318. One commit has landed since,
b103edb588, which only widens what the ceiling validator accepts: a value that spells a whole number, such as
the string an os.environ/ ceiling resolves to, is read rather than refused. It rejects strictly less than
before and changes nothing about a capped turn. Every rejection message quoted above is byte-identical at the
new tip, asserted by unit tests rather than by eye:

ValueError: litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be at least 1, got 0
ValueError: litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be at least 1, got -1
TypeError:  litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be an integer, got True
TypeError:  litellm_params.max_agentic_loops on model 'bedrock-sonnet-45' must be an integer, got 'three'

Type

🐛 Bug Fix
🆕 New Feature

Caveats (if any)

  • The client sees fewer search results than it is billed for. An uncapped run at max_agentic_loops: 6 sent six searches and came back carrying only round 1's web_search_tool_result, with no refusals involved. That is how the agentic loop has always assembled its final response, ceiling or no ceiling, and this PR leaves it alone. It is worth naming anyway, because paying for six searches and seeing one result set surprises people
  • A capped turn is not guaranteed to carry text. When the cap trips deep in a multi-hop chain the turn often comes back with no text at all, four of five runs on a 5-deep sequential chain, and where text did survive it was a mid-research note promising a search that never arrives. Shallower turns are fine: every Claude Code turn and every ordinary question produced real prose. The capped assistant message loses its tool_use block, so nothing is left over when that block was all the model had emitted
  • A capped turn is silently truncated. The client gets a well-formed terminal response and no signal that the model wanted to keep searching, so a caller cannot tell a finished answer from a cut-off one. Surfacing that is a follow-up
  • Rejecting a bad max_agentic_loops at startup is a behavior change. A deployment carrying max_agentic_loops: 0 boots today and quietly runs at the default ceiling of 3; after this change the proxy refuses to start until the value is fixed. That is deliberate, since 0 currently hands the loosest ceiling to whoever asked for the tightest, but it does mean an existing config can stop booting on upgrade. Only values the loop cannot honor are refused: anything spelling a whole number is still accepted, including the string an os.environ/ ceiling resolves to, so a config that works today keeps working unless its ceiling was already meaningless
  • A rejected ceiling stops the proxy but does not always look like a failure. At two or more workers the parent exits 0 rather than 3, so an orchestrator reads a typo'd ceiling as a clean shutdown instead of a config error. Nothing is served either way, and controls that vary only the worker count pin this to uvicorn's multiprocess supervisor rather than to this change: the same split shows up whichever validator raised, and the exit code tracks the worker count alone. Filed as Proxy exits 0 when a worker fails startup validation under --num_workers > 1 #37948, not fixed here
  • Runtime-added deployments are not covered by the startup check. A model added through POST /model/new or a DB reload skips config load, so a bad ceiling there is caught at request time instead, with an error naming the field. Validating in those two endpoints is a follow-up
  • Cost attribution shifts, in the client's favor, on the non-streaming path. Before this change the discarded depth-1 call was billed with nothing to show for it; now that turn is what the client receives. The capped streaming path is still not cost attributed
  • /v1/responses still returns the internal call at the ceiling, unchanged either way, since the terminal turn is gated on the anthropic messages surface
  • /v1/chat/completions keeps its own rails and still raises at the ceiling. Moving that surface over is blocked on fix(agentic loop): return a real stream when code-interpreter interception converts the request #37657
  • The anthropic path never runs the agentic loop cleanup. Pre-existing, untouched here

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Medium Risk
Changes agentic-loop control flow and /v1/messages response shape when the loop cap trips, plus startup validation that can refuse to boot on a bad max_agentic_loops. /v1/responses and chat completions still raise instead of finalizing.

Overview
When web-search interception hits max_agentic_loops (or a repeated tool-call fingerprint), a non-streaming /v1/messages turn now ends instead of leaking the internal litellm_web_search tool_use block. Refused blocks are dropped, stop_reason becomes end_turn unless a client-declared tool remains, and interceptor-converted streams are rebuilt from that finalized turn.

max_agentic_loops can be set on websearch_interception_params (default still 3) or per deployment, with the deployment value winning. Both knobs go through shared validation at config load: values that are not a whole number ≥ 1 stop the proxy rather than being read as 3 or failing at request time. Env-sourced string ceilings like "5" still work.

The fake Anthropic stream iterator now emits content_block_start for unrecognized blocks (server_tool_use, web_search_tool_result), so capped SSE no longer closes blocks it never opened. /v1/responses and chat-completions still raise at the ceiling; live streaming (AgenticStreamingIterator) is unchanged.

Reviewed by Cursor Bugbot for commit b103edb. Bugbot is set up for automated code reviews on this repo. Configure here.

…its ceiling

When the bounded loop cap or the repeated tool-call fingerprint guard refused a
rerun, the raise escaped the parent agentic frame and the client got the raw model
turn back: HTTP 200 carrying an unresolved tool_use block for the internal
litellm_web_search tool and stop_reason "tool_use". The client never declared that
tool, so it had no way to answer it and the conversation could not continue

The safety check now raises AgenticLoopSafetyError, a ValueError subclass, and
_call_agentic_completion_hooks catches it and returns a finalized response: the
blocks belonging to the refused tool calls are dropped, and stop_reason is closed
out to end_turn when nothing the client declared is still waiting. Refused blocks
are matched by the ids and names of the tool calls the rail refused rather than by
hardcoding the web search tool name

Only the non-streaming anthropic messages path ends the turn this way. A streaming
caller has already sent the original message by the time the hooks run, so a
finalized turn would arrive as a second message rather than replace the first, and
the responses surface carries a pydantic model this finalizer does not rewrite.
Both keep raising, exactly as they did before

Also adds max_agentic_loops to websearch_interception_params so the ceiling can be
set once for the whole feature. A per deployment litellm_params.max_agentic_loops
still wins over it, and the field stays on the proxy's untrusted root list so a
client cannot raise its own ceiling
@greptile-apps

greptile-apps Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes capped Anthropic Messages web-search loops return a terminal response instead of leaking LiteLLM’s internal search call.

  • Adds shared validation and feature-level configuration for max_agentic_loops, with per-deployment precedence.
  • Removes refused internal tool-use blocks while preserving client-declared tool calls.
  • Rebuilds valid Anthropic stream events for previously unsupported content-block types.
  • Adds regression coverage for terminal responses, streaming, configuration propagation, and startup validation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
litellm/llms/custom_httpx/llm_http_handler.py Finalizes capped Anthropic Messages turns by removing refused internal tool calls while retaining client-owned tool calls and existing behavior on unsupported surfaces.
litellm/integrations/websearch_interception/handler.py Adds validated feature-level loop configuration and applies it when a deployment-specific ceiling is absent.
litellm/litellm_core_utils/agentic_loop_settings.py Centralizes loop-ceiling coercion, defaults, and validation while explicitly rejecting booleans and values below one.
litellm/llms/anthropic/experimental_pass_through/messages/fake_stream_iterator.py Emits matching content-block starts for otherwise unrecognized Anthropic blocks, preventing malformed reconstructed streams.
litellm/proxy/proxy_server.py Validates per-deployment loop ceilings during proxy configuration loading.
tests/test_litellm/integrations/websearch_interception/test_websearch_agentic_loop_cap.py Covers loop termination, internal-versus-client tool filtering, reconstructed streaming, configuration precedence, and validation behavior.

Reviews (3): Last reviewed commit: "fix: keep accepting a loop ceiling that ..." | Re-trigger Greptile

Comment thread litellm/integrations/websearch_interception/ARCHITECTURE.md
@codecov

codecov Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.97980% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...itellm/litellm_core_utils/agentic_loop_settings.py 95.83% 1 Missing ⚠️
litellm/llms/custom_httpx/llm_http_handler.py 97.95% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

ARCHITECTURE.md promised the clean end_turn for every intercepted request.
A request that streams all the way through and one on /v1/responses both
still hand back the internal tool call, so say that plainly instead. Also
note that where the refused call was the only block left, the turn can come
back with no text in it.

On AgenticLoopSafetyError, note that the chat completions loop still raises
a plain ValueError from its own copy of the rails, so nobody writes an
except for this type expecting it to cover that surface too.
@mateo-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 7c02c08. Configure here.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ Mateo Wang


Mateo Wang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codspeed-hq

codspeed-hq Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_fix_agentic_loop_cap_response (b103edb) with litellm_internal_staging (7a1afa1)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (28887f1) during the generation of this report, so ff20a4b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

A capped turn on a streaming request is rebuilt into SSE by
FakeAnthropicMessagesStreamIterator. It emitted content_block_stop for
every block but content_block_start only for text, thinking,
redacted_thinking and tool_use, so a web search turn's server_tool_use
and web_search_tool_result blocks produced stops with no matching start.

Anthropic's SDK accumulator appends on content_block_start and then
indexes content[event.index] on content_block_delta, so the orphan stops
shifted every later index and client.messages.stream() raised IndexError
on the text block. Unknown block types now pass through with a start of
their own, which keeps position equal to index.

Also corrects two claims that said no current caller reaches the loop
with stream=True. AgenticStreamingIterator does, and it keeps raising,
because its events are already on the wire.
The ceiling was only checked at the feature level, on
litellm_settings.websearch_interception_params. The per-deployment
litellm_params.max_agentic_loops, which wins over it, went straight into
int(kwargs.get("max_agentic_loops", 3) or 3), so a 0 was swallowed by the
falsy fallback and read as the default 3. Asking for the tightest ceiling
handed you the loosest one. A non-integer booted the proxy and then failed
every request to that model with "invalid literal for int() with base 10".

Both settings now share one validator, which names the field it rejected,
and the per-deployment value is checked while the model list is read at
startup so a bad value stops the proxy rather than surfacing per request.
The check sits in load_config rather than on LiteLLM_Params because the
proxy builds its router with ignore_invalid_deployments=True, where a
validation error drops the deployment silently instead of refusing to
start. This is the same placement the complexity_router_config plugin
check already uses.

Chat completions read the same key through a separate path that turned 0
into 1 and true into a ceiling of 1, so it now shares the validator too
and the key means one thing on both surfaces.
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

The ceiling used to go through `int(... or 3)`, so anything `int()` accepted
worked. Tightening the new shared validator to `isinstance(int)` turned a
config that boots today into a proxy that refuses to start, because
`max_agentic_loops: os.environ/MAX_AGENTIC_LOOPS` is resolved to a string
before it reaches either check, and a YAML-quoted "5" is a string too.

Accept ints, integral floats, and strings that parse to a whole number. Keep
refusing bools, fractional floats, words, and anything below 1.
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit b103edb. Configure here.

@mateo-berri
mateo-berri enabled auto-merge August 22, 2026 18:40
@mateo-berri
mateo-berri merged commit abf99e3 into litellm_internal_staging Aug 22, 2026
74 checks passed
@mateo-berri
mateo-berri deleted the litellm_fix_agentic_loop_cap_response branch August 22, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants