fix(router): stop retries and fallbacks from re-running executed MCP tools

[Atharva-Kanherkar]

TLDR

Problem this solves:

• A failed model call after an MCP tool re-ran the tool
• One request wrote to an auto-approved MCP tool three times

How it solves it:

• Tag errors raised after the MCP session sent a tool call
• The router neither retries nor falls back on a tagged error
• Covers streaming chat, whose tools run inside the stream
• Calls that never reach a server keep today's retries

Intentional product change: when the model call that follows an auto-approved MCP tool call fails after that call was sent to its MCP server, the client now gets the error right away. Before, the router retried or fell back to another model group, which re-ran the whole MCP loop and executed the tool again on every attempt. Requests where no tool call reached a server, for example because a guardrail blocked every call or a per-user credential was missing, are retried exactly as before

User Flow

Before: an app that records entries through an auto-approved MCP tool gets one error and three writes

1. The proxy admin sets num_retries: 2 and adds an MCP server ledger whose commit_write tool appends a line to a file
2. The app sends POST http://localhost:4000/v1/chat/completions asking the model to record entry alpha with that tool (/v1/responses and /v1/messages behave the same)
3. The model calls the tool, then the model call that should produce the final answer fails with a 500
4. The app gets HTTP 500, and the ledger file now has three alpha lines

After: the same request gets the same 500 and exactly one write

1. The proxy admin sets num_retries: 2 and adds an MCP server ledger whose commit_write tool appends a line to a file
2. The app sends POST http://localhost:4000/v1/chat/completions asking the model to record entry alpha with that tool (/v1/responses and /v1/messages behave the same)
3. The model calls the tool, then the model call that should produce the final answer fails with a 500
4. The app gets HTTP 500, and the ledger file has one alpha line, so the app decides whether to try again

Relevant issues

Fixes #43153

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The test files covering my change pass locally: pytest tests/test_litellm/test_router_retry_non_retryable_errors.py tests/test_litellm/experimental_mcp_client/test_mcp_client.py tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_max_concurrent_requests.py (1207 passed). The tests only extend existing test files. The end-to-end ones drive a real Router through the MCP gateway with respx at the HTTP boundary, without patching private class methods. They fail on the merge base, and each earlier revision of this PR fails the tests added for the finding raised on it
• My PR passes all required CI/CD checks (waiting on the run for 99b3c06. On 4ad4a4d every check passed except two tests in the fork-only part of mcp-integration, test_mcp_client_unit.py::test_call_tool and test_mcp_server.py::test_sse_mcp_handler_mock. They fail the same way on main at 61953318 without this PR, because feat(mcp): share compatibility-aware result conversion across tool surfaces #43089 added allow_input_required to the session call and moved SSE to serve_loop, and these CircleCI-only tests still expect the old calls. Every script of the code quality workflow, ruff check, ruff format --check, the strict ruff, type discipline and test quality gates, the MCP operation boundary check, and the circular import and import safety checks pass locally, with no new basedpyright errors in any changed file)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (the fifth review scored 4/5 and called the replay protection sound; its only finding, routine comments, is fixed in 99b3c06, and a new review is requested)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: the proxy runs from this branch's merge base and from its head with python litellm/proxy/proxy_cli.py --config config.yaml --port 4000. Every model call is a real, paid gpt-5.4-mini call to OpenAI, relayed by a small local forwarder on 127.0.0.1:8765 that logs every request as initial or follow_up. It forces the otherwise rare transient failures with flag files: while fault_on exists it answers every follow-up request (the one that carries the tool result) with HTTP 500 instead of relaying it, fault_once does the same for one follow-up and is then removed, and drop_stream makes it send the headers of a streamed follow-up and then close the connection. The MCP server is a real FastMCP streamable HTTP server (mcp==1.30.0) on 127.0.0.1:8766 whose one tool, commit_write, appends its entry to ledger.txt. The block-mcp guardrail is a documented custom guardrail that rejects every MCP tool call before it runs; a request opts in with "guardrails": ["block-mcp"]. The streaming case uses config-stream.yaml, which is config.yaml plus a router fallback from repro-model to backup-model. With an OpenAI model the proxy sends /v1/messages upstream as a Responses call, so that case logs /v1/responses

model_list:
  - model_name: repro-model
    litellm_params:
      model: openai/gpt-5.4-mini
      api_base: http://127.0.0.1:8765/v1
      api_key: os.environ/OPENAI_API_KEY
  - model_name: backup-model
    litellm_params:
      model: openai/gpt-5.4-mini
      api_base: http://127.0.0.1:8765/v1
      api_key: os.environ/OPENAI_API_KEY

mcp_servers:
  ledger:
    url: http://127.0.0.1:8766/mcp
    transport: http

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

litellm_settings:
  num_retries: 2

guardrails:
  - guardrail_name: block-mcp
    litellm_params:
      guardrail: custom_guardrail.BlockMCPTools
      mode: pre_mcp_call
      default_on: false

# config-stream.yaml only:
# router_settings:
#   fallbacks: [{"repro-model": ["backup-model"]}]

TOOLS='[{"type":"mcp","server_label":"ledger","server_url":"litellm_proxy","require_approval":"never"}]'
ASK='Call the commit_write tool exactly once with entry alpha, then reply done.'
CHAT="{\"model\":\"repro-model\",\"messages\":[{\"role\":\"user\",\"content\":\"$ASK\"}],\"tools\":$TOOLS}"
RESP="{\"model\":\"repro-model\",\"input\":\"$ASK\",\"tools\":$TOOLS}"
MSGS="{\"model\":\"repro-model\",\"max_tokens\":1024,\"messages\":[{\"role\":\"user\",\"content\":\"$ASK\"}],\"tools\":$TOOLS}"
BLOCK='"guardrails":["block-mcp"],'
post() { curl -s -o body.json -w 'HTTP %{http_code}\n' "http://localhost:4000$1" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d "$2"; }
# before each endpoint case: : > ledger.txt; : > upstream.log; touch fault_on

MCP server, forwarder and guardrail

# mcp_ledger.py, run as: python mcp_ledger.py 8766 ledger.txt
import sys
from pathlib import Path

from mcp.server.fastmcp import FastMCP

ledger = Path(sys.argv[2])
server = FastMCP("ledger", host="127.0.0.1", port=int(sys.argv[1]))


@server.tool()
def commit_write(entry: str) -> str:
    """Append one entry to the ledger file."""
    with ledger.open("a") as f:
        f.write(entry + "\n")
    return f"recorded {entry}"


server.run(transport="streamable-http")

# forwarder.py, run as: python forwarder.py 8765 upstream.log fault_on fault_once drop_stream
import json
import sys
from pathlib import Path

from aiohttp import ClientSession, web

port, log = int(sys.argv[1]), Path(sys.argv[2])
fault_flag, fault_once_flag, drop_stream_flag = Path(sys.argv[3]), Path(sys.argv[4]), Path(sys.argv[5])


def is_follow_up(body: dict) -> bool:
    messages = body.get("messages") or []
    items = body.get("input") if isinstance(body.get("input"), list) else []
    return any(m.get("role") == "tool" for m in messages if isinstance(m, dict)) or any(
        i.get("type") == "function_call_output" for i in items if isinstance(i, dict)
    )


async def forward(request: web.Request) -> web.Response:
    raw = await request.read()
    body = json.loads(raw)
    kind = "follow_up" if is_follow_up(body) else "initial"
    inject = kind == "follow_up" and (fault_flag.exists() or fault_once_flag.exists())
    drop = kind == "follow_up" and body.get("stream") is True and drop_stream_flag.exists()
    outcome = "injected_500" if inject else "dropped_stream" if drop else "forwarded"
    with log.open("a") as f:
        f.write(f"{request.path} {kind} {outcome}\n")
    if inject:
        fault_once_flag.unlink(missing_ok=True)
        return web.json_response({"error": {"message": "injected follow-up failure", "type": "server_error"}}, status=500)
    if drop:
        response = web.StreamResponse(headers={"Content-Type": "text/event-stream"})
        await response.prepare(request)
        request.transport.close()
        return response
    headers = {k: v for k, v in request.headers.items() if k.lower() in ("authorization", "content-type")}
    async with ClientSession() as session:
        async with session.post(f"https://api.openai.com{request.path}", data=raw, headers=headers) as upstream:
            return web.Response(body=await upstream.read(), status=upstream.status, content_type=upstream.content_type)


app = web.Application()
app.router.add_post("/v1/chat/completions", forward)
app.router.add_post("/v1/responses", forward)
web.run_app(app, host="127.0.0.1", port=port, print=None)

# custom_guardrail.py, next to config.yaml
from litellm.exceptions import GuardrailRaisedException
from litellm.integrations.custom_guardrail import CustomGuardrail


class BlockMCPTools(CustomGuardrail):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        raise GuardrailRaisedException(message="MCP tool calls are blocked", blocked_content=True)

Before (6195331)

/v1/chat/completions

1. Send the request and read the error

   $ post /v1/chat/completions "$CHAT"
   HTTP 500
   $ jq -c '.error | {message, code}' body.json
   {"message":"litellm.InternalServerError: InternalServerError: OpenAIException - injected follow-up failure\n\nLiteLLM: model group 'repro-model' failed with the error above. No fallback was attempted.","code":"500"}

2. Count the tool runs and the upstream calls: three writes, and the whole loop ran three times

   $ wc -l < ledger.txt
   3
   $ sort upstream.log | uniq -c
      3 /v1/chat/completions follow_up injected_500
      3 /v1/chat/completions initial forwarded

/v1/responses

1. Send the request and read the error

   $ post /v1/responses "$RESP"
   HTTP 500
   $ jq -c '.error | {message, code}' body.json
   {"message":"litellm.InternalServerError: InternalServerError: OpenAIException - {\"error\": {\"message\": \"injected follow-up failure\", \"type\": \"server_error\"}}\n\nLiteLLM: model group 'repro-model' failed with the error above. No fallback was attempted.","code":"500"}

2. Count the tool runs and the upstream calls: three writes, and the whole loop ran three times

   $ wc -l < ledger.txt
   3
   $ sort upstream.log | uniq -c
      3 /v1/responses follow_up injected_500
      3 /v1/responses initial forwarded

/v1/messages

1. Send the request and read the error

   $ post /v1/messages "$MSGS"
   HTTP 500
   $ jq -c '.error | {message, code}' body.json
   {"message":"litellm.InternalServerError: InternalServerError: OpenAIException - {\"error\": {\"message\": \"injected follow-up failure\", \"type\": \"server_error\"}}\n\nLiteLLM: model group 'repro-model' failed with the error above. No fallback was attempted.","code":null}

2. Count the tool runs and the upstream calls: three writes, and the whole loop ran three times

   $ wc -l < ledger.txt
   3
   $ sort upstream.log | uniq -c
      3 /v1/responses follow_up injected_500
      3 /v1/responses initial forwarded

Guardrail blocks the tool and the follow-up fails once

1. Send each request with the guardrail selected, failing its first follow-up call

   $ rm fault_on; : > ledger.txt; : > upstream.log
   $ for pair in "/v1/chat/completions|$CHAT" "/v1/responses|$RESP" "/v1/messages|$MSGS"; do touch fault_once; post "${pair%%|*}" "{$BLOCK${pair#*|\{}"; done
   HTTP 200
   HTTP 200
   HTTP 200

2. Count the tool runs and the upstream calls: no write, and each request was retried once after its failed follow-up

   $ wc -l < ledger.txt
   0
   $ sort upstream.log | uniq -c
      1 /v1/chat/completions follow_up forwarded
      1 /v1/chat/completions follow_up injected_500
      2 /v1/chat/completions initial forwarded
      2 /v1/responses follow_up forwarded
      2 /v1/responses follow_up injected_500
      4 /v1/responses initial forwarded

No injected failure

1. Send each request once without any fault

   $ rm -f fault_on fault_once; : > ledger.txt; : > upstream.log
   $ post /v1/chat/completions "$CHAT"; post /v1/responses "$RESP"; post /v1/messages "$MSGS"
   HTTP 200
   HTTP 200
   HTTP 200

2. Count the tool runs and the upstream calls: one write and one initial plus one follow-up call per request

   $ wc -l < ledger.txt
   3
   $ sort upstream.log | uniq -c
      1 /v1/chat/completions follow_up forwarded
      1 /v1/chat/completions initial forwarded
      2 /v1/responses follow_up forwarded
      2 /v1/responses initial forwarded

Streaming chat whose follow-up stream drops, with a router fallback

1. Start the proxy with config-stream.yaml, stream the chat request, and let the forwarder drop the follow-up stream right after its headers

   $ rm -f fault_on fault_once; : > ledger.txt; : > upstream.log; touch drop_stream
   $ STREAM="{\"stream\":true,${CHAT#\{}"
   $ curl -sN -o stream.txt http://localhost:4000/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d "$STREAM"
   $ grep -c '^data: {"id"' stream.txt
   14
   $ grep '^data: {"error"' stream.txt | cut -c1-150
   data: {"error": {"message": "litellm.APIConnectionError: APIConnectionError: OpenAIException - Response payload is not completed: <TransferEncodingErr

2. Count the tool runs and the upstream calls: two writes, because the fallback re-ran the whole loop and its chunks were spliced after the original ones

   $ wc -l < ledger.txt
   2
   $ sort upstream.log | uniq -c
      2 /v1/chat/completions follow_up dropped_stream
      2 /v1/chat/completions initial forwarded

After (99b3c06)

/v1/chat/completions

1. Send the request and read the error

   $ post /v1/chat/completions "$CHAT"
   HTTP 500
   $ jq -c '.error | {message, code}' body.json
   {"message":"litellm.InternalServerError: InternalServerError: OpenAIException - injected follow-up failure","code":"500"}

2. Count the tool runs and the upstream calls: one write, one initial call and one follow-up call

   $ wc -l < ledger.txt
   1
   $ sort upstream.log | uniq -c
      1 /v1/chat/completions follow_up injected_500
      1 /v1/chat/completions initial forwarded

/v1/responses

1. Send the request and read the error

   $ post /v1/responses "$RESP"
   HTTP 500
   $ jq -c '.error | {message, code}' body.json
   {"message":"litellm.InternalServerError: InternalServerError: OpenAIException - {\"error\": {\"message\": \"injected follow-up failure\", \"type\": \"server_error\"}}","code":"500"}

2. Count the tool runs and the upstream calls: one write, one initial call and one follow-up call

   $ wc -l < ledger.txt
   1
   $ sort upstream.log | uniq -c
      1 /v1/responses follow_up injected_500
      1 /v1/responses initial forwarded

/v1/messages

1. Send the request and read the error

   $ post /v1/messages "$MSGS"
   HTTP 500
   $ jq -c '.error | {message, code}' body.json
   {"message":"litellm.InternalServerError: InternalServerError: OpenAIException - {\"error\": {\"message\": \"injected follow-up failure\", \"type\": \"server_error\"}}","code":null}

2. Count the tool runs and the upstream calls: one write, one initial call and one follow-up call

   $ wc -l < ledger.txt
   1
   $ sort upstream.log | uniq -c
      1 /v1/responses follow_up injected_500
      1 /v1/responses initial forwarded

Guardrail blocks the tool and the follow-up fails once

1. Send each request with the guardrail selected, failing its first follow-up call

   $ rm fault_on; : > ledger.txt; : > upstream.log
   $ for pair in "/v1/chat/completions|$CHAT" "/v1/responses|$RESP" "/v1/messages|$MSGS"; do touch fault_once; post "${pair%%|*}" "{$BLOCK${pair#*|\{}"; done
   HTTP 200
   HTTP 200
   HTTP 200

2. Count the tool runs and the upstream calls: the same as the merge base, so a request where no tool call reached a server is still retried

   $ wc -l < ledger.txt
   0
   $ sort upstream.log | uniq -c
      1 /v1/chat/completions follow_up forwarded
      1 /v1/chat/completions follow_up injected_500
      2 /v1/chat/completions initial forwarded
      2 /v1/responses follow_up forwarded
      2 /v1/responses follow_up injected_500
      4 /v1/responses initial forwarded

No injected failure

1. Send each request once without any fault

   $ rm -f fault_on fault_once; : > ledger.txt; : > upstream.log
   $ post /v1/chat/completions "$CHAT"; post /v1/responses "$RESP"; post /v1/messages "$MSGS"
   HTTP 200
   HTTP 200
   HTTP 200

2. Count the tool runs and the upstream calls: one write and one initial plus one follow-up call per request

   $ wc -l < ledger.txt
   3
   $ sort upstream.log | uniq -c
      1 /v1/chat/completions follow_up forwarded
      1 /v1/chat/completions initial forwarded
      2 /v1/responses follow_up forwarded
      2 /v1/responses initial forwarded

Streaming chat whose follow-up stream drops, with a router fallback

1. Start the proxy with config-stream.yaml, stream the chat request, and let the forwarder drop the follow-up stream right after its headers

   $ rm -f fault_on fault_once; : > ledger.txt; : > upstream.log; touch drop_stream
   $ STREAM="{\"stream\":true,${CHAT#\{}"
   $ curl -sN -o stream.txt http://localhost:4000/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d "$STREAM"
   $ grep -c '^data: {"id"' stream.txt
   7
   $ grep '^data: {"error"' stream.txt | cut -c1-150
   data: {"error": {"message": "litellm.APIConnectionError: APIConnectionError: OpenAIException - Response payload is not completed: <TransferEncodingErr

2. Count the tool runs and the upstream calls: one write, and the client gets the error after the original chunks instead of a fallback that runs the tool again

   $ wc -l < ledger.txt
   1
   $ sort upstream.log | uniq -c
      1 /v1/chat/completions follow_up dropped_stream
      1 /v1/chat/completions initial forwarded

Type

Bug Fix

Caveats (if any)

Medium

• A failure after a tool call was sent now surfaces
  • Before, a retry could hide it by running the tool again
  • Retrying only the follow-up call could be a later improvement

Low

• The router's "No fallback was attempted" note is skipped here
  • The existing disable_fallbacks exit skips it the same way
• Responses streams failing after output are handled in fix(router): stop Responses fallback after delivered output #42958
• The SDK's own num_retries wrapper, used without a router, is unchanged

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds tool execution tracking to prevent retry loops from re-running MCP tools.

The PR appears safe to merge; no outstanding review finding remains.

Summary

The PR marks errors after an MCP tool call has been dispatched and prevents the router from retrying or falling back when doing so could execute the tool again.

• Applies the replay guard to chat completions, Responses, and Anthropic Messages, including streaming chat follow-up errors.
• Preserves retries when a tool call was blocked or failed before dispatch.

_{Reviews (6) · Last reviewed commit: "style(mcp): drop routine comments from t..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing Atharva-Kanherkar:fix-mcp-retry-tool-reexecution (8b1c0a2) with main (85dc7cb)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Atharva-Kanherkar]

@greptileai please review again: the tag now requires an executed tool, the Responses re-listing is guarded, and tests avoid private patches

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[Atharva-Kanherkar]

@greptileai please review again: tool calls now count once sent to the MCP server, and streamed chat follow-ups are guarded too

[Atharva-Kanherkar]

@greptileai please review again: a tool call now counts only when the MCP session sends it, so unsent calls keep retries

[Atharva-Kanherkar]

@greptileai please review again: the replay tests now extend the existing router retry test file instead of adding a new one

[Atharva-Kanherkar]

@greptileai please review again: the routine comments are gone, and the branch now merges the latest main

[Atharva-Kanherkar]

@joshua-berri I have resolved the conflicts with main, could you please take a look when you get a chance?

[Atharva-Kanherkar]

The misc failures are unrelated to this PR: the 4 failing tests in tests/unit/interactions/test_openapi_compliance.py fetch the live spec from ai.google.dev, which no longer has the CreateModelInteractionParams schema and now uses /interactions/{interactionsId} instead of /interactions/{id}.

[Atharva-Kanherkar]

can anyone in the maintainers see this?