fix(mcp): tools/call no longer 404s on a worker that has not served tools/list by devin-ai-integration[bot] · Pull Request #42072 · BerriAI/litellm

devin-ai-integration · 2026-09-20T01:50:24Z

TLDR

Problem this solves:

Since the mcp 2 upgrade (refactor(mcp): upgrade SDK2 while preserving legacy gateway behavior #41718), tools/call 404s on any worker that has not served tools/list
Hits every true_passthrough and OAuth-discovery server on multi-worker and multi-pod deployments
A client that listed once still fails whenever its next call lands on another worker

How it solves it:

On a resolution miss, the worker lists that one server with the caller's own credentials, then resolves
Each worker lists a server at most once; a failed list falls through to the existing 404
Unknown tools, servers the key cannot access, and startup-mapped servers behave as before

User Flow

Before: a developer whose agent (Claude Code, in the runs below) calls tools through a true_passthrough MCP server gets "Tool not found" from every worker that has not served a tools/list since the last restart

The proxy admin restarts a proxy running with --num_workers 2 and an MCP server rig_passthrough configured with auth_type: true_passthrough
The developer's MCP client sends POST http://localhost:16733/mcp initialize with Authorization: Bearer sk-1234 and gets 200 with an mcp-session-id header
It sends POST http://localhost:16733/mcp notifications/initialized with that session id and gets 202
It sends POST http://localhost:16733/mcp tools/call with {"name":"rig_passthrough-add","arguments":{"a":1,"b":2}} and gets {"content":[{"type":"text","text":"Error: Tool 'add' not found"}],"isError":true}. Every fresh session gets the same answer on both workers, and a client that sent tools/list once still gets it whenever a later call lands on the other worker
Calling the same tool through POST http://localhost:16733/mcp-rest/tools/call with {"name":"add","server_id":"<server id>","arguments":{"a":1,"b":2}} gets 500 An unexpected error occurred: Tool add not found from that worker

After: the same calls return the tool result on every worker, whether or not it has served a tools/list

The proxy admin restarts a proxy running with --num_workers 2 and an MCP server rig_passthrough configured with auth_type: true_passthrough
The developer's MCP client sends POST http://localhost:16733/mcp initialize with Authorization: Bearer sk-1234 and gets 200 with an mcp-session-id header
It sends POST http://localhost:16733/mcp notifications/initialized with that session id and gets 202
It sends POST http://localhost:16733/mcp tools/call with {"name":"rig_passthrough-add","arguments":{"a":1,"b":2}} and gets {"content":[{"type":"text","text":"3"}],"isError":false} on every fresh session, on both workers, with no tools/list first
POST http://localhost:16733/mcp-rest/tools/call with {"name":"add","server_id":"<server id>","arguments":{"a":1,"b":2}} gets 200 with "text":"3" from both workers
A tool the server does not have, rig_passthrough-nope, still gets Error: Tool 'nope' not found

Relevant issues

Affected release

Regression on main since #41718 (eda1fab); not in any rc or dev tag yet, so no backport needed

Linear ticket

Resolves LIT-8212

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both legs: one proxy booted with --num_workers 2 from the named commit (mcp 2.2.0), Postgres behind it, and three MCP servers pointing at the same local upstream, which serves add, request_headers, and fail_tool

mcp_servers:
  rig_http:
    transport: http
    url: http://127.0.0.1:16833/mcp
    allow_all_keys: true
  rig_oauth_cc:
    transport: http
    url: http://127.0.0.1:16833/mcp
    auth_type: oauth2
    oauth2_flow: client_credentials
    client_id: rig-client
    client_secret: rig-secret
    token_url: http://127.0.0.1:16833/oauth/token
    allow_all_keys: true
  rig_passthrough:
    transport: http
    url: http://127.0.0.1:16833/mcp
    auth_type: true_passthrough
    allow_all_keys: true

Every "cold" iteration below is a fresh MCP session against a freshly restarted proxy: initialize (captures mcp-session-id), notifications/initialized, then tools/call for <server>-add with Authorization: Bearer sk-1234, and never a tools/list first. The pid= lines under each run come from a custom logging callback in the rig config that records which uvicorn worker served the call, so both workers are shown handling requests

H=(-H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -H 'MCP-Protocol-Version: 2025-06-18' -H 'Authorization: Bearer sk-1234')
sid=$(curl -sS -D - -o /dev/null http://127.0.0.1:16733/mcp "${H[@]}" -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"cold","version":"1"}}}' | grep -i '^mcp-session-id:' | tr -d '\r' | cut -d' ' -f2)
curl -sS -o /dev/null http://127.0.0.1:16733/mcp "${H[@]}" -H "Mcp-Session-Id: $sid" -d '{"jsonrpc":"2.0","method":"notifications/initialized"}'
curl -sS http://127.0.0.1:16733/mcp "${H[@]}" -H "Mcp-Session-Id: $sid" -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"rig_passthrough-add","arguments":{"a":1,"b":2}}}'

The Claude Code case drives the real Claude Code TUI (v2.1.278) under tmux against the same 2-worker proxy, started with claude --mcp-config mcp.json --strict-mcp-config, where mcp.json is

{"mcpServers":{"litellm":{"type":"http","url":"http://127.0.0.1:16733/mcp","headers":{"Authorization":"Bearer sk-1234"}}}}

and, on each leg, a prompt that asks it to call rig_passthrough-add with a=1 and b=2 six separate times, one call at a time, and report each call's result

Before (`82ddab2`)

Claude Code calls the true_passthrough tool six times

Restarted the proxy, started Claude Code with the config above, and sent the prompt
Claude Code's own tools/list at startup was served by worker pid 95887, and all six tools/call requests landed on worker pid 95886, which had not served a list
What the pane showed: six "Error: Tool 'add' not found", and Claude Code concluding the tool "is not registered on the server"

Cold tools/call on the true_passthrough server

Ran the three requests above six times, each in a fresh session, right after a restart
Output:

before [main]  1: Error: Tool 'add' not found
before [main]  2: Error: Tool 'add' not found
before [main]  3: Error: Tool 'add' not found
before [main]  4: Error: Tool 'add' not found
before [main]  5: Error: Tool 'add' not found
before [main]  6: Error: Tool 'add' not found
before [main] TOTAL ok=0 not_found=6
   2 logger.failure pid=22254 call_mcp_tool
   4 logger.failure pid=22263 call_mcp_tool

Raw body of one failing call:

data: {"jsonrpc":"2.0","id":2,"result":{"content":[{"text":"Error: Tool 'add' not found","type":"text"}],"isError":true}}

One tools/list, then cold tools/call again

Sent one tools/list (served by one worker), then eight more cold calls
Output: only the worker that served the list resolves the tool; the call that landed on the other worker still fails

after-list [main]  1: 3
after-list [main]  2: 3
after-list [main]  3: 3
after-list [main]  4: 3
after-list [main]  5: 3
after-list [main]  6: Error: Tool 'add' not found
after-list [main]  7: 3
after-list [main]  8: 3
after-list [main] TOTAL ok=7 not_found=1
   1 logger.failure pid=22254 call_mcp_tool
   7 logger.success pid=22263 call_mcp_tool

REST tools/call with server_id on a cold worker

curl -sS http://127.0.0.1:16733/mcp-rest/tools/call -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-1234' -d '{"name":"add","arguments":{"a":1,"b":2},"server_id":"a3298f3a30d401988d10b2c011283f74"}' repeated until a request landed on the worker that had not served a list
Output on that worker:

{"detail":{"error":"internal_server_error","message":"An unexpected error occurred: Tool add not found"}}

Cold tools/call on the plain http and client_credentials servers (control)

Same six cold calls against rig_http-add and rig_oauth_cc-add
Output: both resolve on every worker, since the startup mapping covers them

control [main] TOTAL ok=6 not_found=0
   3 logger.success pid=22254 call_mcp_tool
   3 logger.success pid=22263 call_mcp_tool
before-oauth [main] TOTAL ok=6 not_found=0
   1 logger.success pid=22254 call_mcp_tool
   5 logger.success pid=22263 call_mcp_tool

Cold tools/call for a tool the server does not have

Not run on this leg; the passthrough server answers "not found" for every tool here, so the case is only meaningful after the fix

After (`a3e9ed3`)

Claude Code calls the true_passthrough tool six times

Fresh proxy, started Claude Code (v2.1.278) with the config above, and sent the prompt
Claude Code's own tools/list at startup was served by worker pid 17328, and all six tools/call requests landed on worker pid 17330, which had never served a list
What the pane showed: six calls, each returning 3, and Claude Code concluding "All six sequential calls to rig_passthrough-add with a=1 and b=2 returned 3 each time, with no errors"

This is the direct mirror of the Before leg: the same one-list-then-six-calls-on-the-other-worker split, now every call resolving instead of every call 404ing

Cold tools/call on the true_passthrough server

Ran the cold request above six times, each in a fresh session, right after a restart
Output:

after [aftertip]  1: 3
after [aftertip]  2: 3
after [aftertip]  3: 3
after [aftertip]  4: 3
after [aftertip]  5: 3
after [aftertip]  6: 3
after [aftertip] TOTAL ok=6 not_found=0
   6 logger.success pid=57883 call_mcp_tool

On a fresh proxy no worker has listed the passthrough server (the startup fill skips it), so iteration 1 is itself a cold-worker call and it resolves

One tools/list, then cold tools/call again

Covered by the Claude Code run above, which is exactly one tools/list (worker pid 17328) followed by six tools/call on the never-listed worker pid 17330
All six resolve, so the Before leg's "1 in 8 still fails" is gone

REST tools/call with server_id on a cold worker

Fresh proxy, the REST call sent eight times, the first before any tools/list, so it lands on a worker that has not listed
Output: 200 with the tool result, and both workers resolve

rest-after 1: {"content":[{"type":"text","text":"3",...}],"structuredContent":{"result":3},"isError":false} HTTP=200
rest-after 2..7: identical, HTTP=200
rest-after 8: {"content":[{"type":"text","text":"3",...}],"structuredContent":{"result":3},"isError":false} HTTP=200
   1 logger.success pid=10086 call_mcp_tool
   7 logger.success pid=10087 call_mcp_tool

Cold tools/call on the plain http and client_credentials servers (control)

Same six cold calls against rig_http-add and rig_oauth_cc-add
Output: unchanged, still resolve on every worker

control [aftertip] TOTAL ok=6 not_found=0
   5 logger.success pid=57883 call_mcp_tool
   1 logger.success pid=57884 call_mcp_tool
after-oauth [aftertip] TOTAL ok=6 not_found=0
   6 logger.success pid=57883 call_mcp_tool

Cold tools/call for a tool the server does not have

Four cold calls for rig_passthrough-nope
Output: still "not found" on both workers, so an unknown tool never dispatches by name alone

after-unknown [aftertip]  1: Error: Tool 'nope' not found
after-unknown [aftertip]  2: Error: Tool 'nope' not found
after-unknown [aftertip]  3: Error: Tool 'nope' not found
after-unknown [aftertip]  4: Error: Tool 'nope' not found
after-unknown [aftertip] TOTAL ok=0 not_found=4
   2 logger.failure pid=57883 call_mcp_tool
   2 logger.failure pid=57884 call_mcp_tool

What the run saw next to the fix

One worker lists, six calls on the other, all resolve
REST and MCP cold calls resolve on never-listed workers
Unknown tool re-lists the server once, then answers not found
Error text drops the server prefix; pre-existing, left alone
REST returns 500 not 404 for a missing tool; pre-existing

Type

🐛 Bug Fix

Caveats (if any)

Low: an unknown tool name, or a server that exposes no tools to the caller, costs one extra upstream tools/list on that worker before the call answers "not found". This is the accepted cost of listing lazily. It runs on the error path only, never on a successful call, and is bounded to one re-list per call
Low: the per-tool (not per-server) gating is proven by a mutation-tested unit test rather than the live A/B, because the rig's upstream hands every caller the same catalog, so a live run cannot show two callers holding different tool sets on one server. The happy path and the unknown-tool path are both proven live
Low: the Responses API embedded-MCP tool call runs through the manager's call_tool directly, not through this pre-call listing, so a cold worker there can still answer "not found" for a passthrough tool it never listed across a multi-turn worker split. This PR neither causes nor changes that path, its resolution is byte-identical before and after, so it is pre-existing and left alone, a candidate follow-up if that surface needs the same guarantee

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

The startup tool-name fill skips servers whose upstream wants the caller's own token (true_passthrough, OAuth discovery), and mcp 2 no longer runs the list handler before an uncached tools/call, so every uvicorn worker that had not served tools/list answered 404 "Tool not found" for prefixed tools/call and the REST server_id route on those servers. On a resolution miss, execute_mcp_tool now lists the prefix-matched (or server_id-requested) server once, with the caller's credentials, through the existing tools/list path, then resolves as before. Listing failures fall through to the existing 404, a worker that already listed the server never re-lists it, and a server outside the caller's allowed set is never listed.

devin-ai-integration · 2026-09-20T01:50:27Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

codspeed · 2026-09-20T01:52:56Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_mcp_cold_worker_tools_call (a3e9ed3) with main (5417abd)¹}

No successful run was found on main (09e14a4) during the generation of this report, so 5417abd was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

greptile-apps · 2026-09-20T01:53:26Z

Confidence Score: 5/5

The PR appears safe to merge; no new blocking failures were found.

Findings

A successful empty tools/list creates no mapping row, so every later call repeats the upstream listing before returning 404, adding latency and load. ▶

Summary

This PR repairs cold-worker MCP tool calls by lazily listing the targeted server with the caller’s credentials before resolving an uncached tool.

Adds server-prefix ownership and per-tool exposure helpers.
Restricts lazy listing to servers available to the caller.
Preserves REST server_id routing and prefixed MCP tool-call behavior.
Adds focused coverage for cold workers, unknown tools, caller access restrictions, credential-specific catalogs, and prefix ownership.

_{Reviews (2) · Last reviewed commit: "fix(mcp): gate the pre-call listing per ..."}

greptile-apps · 2026-09-20T01:53:29Z

+    def has_listed_tools(self, server: MCPServer) -> bool:
+        """True once this worker holds at least one tool row for ``server``."""
+        owned: Final = self._owned_mapping_values(server)
+        return any(
+            normalize_server_name(owner) in owned for owner in self.tool_name_to_mcp_server_name_mapping.values()
+        )


A successful empty tools/list creates no mapping row, so every later call repeats the upstream listing before returning 404, adding latency and load.

codecov · 2026-09-20T01:59:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

A tools/call on a cold worker listed the target server once and then never again, so a later caller whose credentials expose a wider upstream catalog got 404 for tools the first caller never had. Gate the pre-call listing on whether this worker already exposes the requested tool, so callers with different catalogs no longer mask each other. Removing the per-server guard also drops the empty-listing case that re-listed on every call.

mateo-berri · 2026-09-20T02:30:51Z

@greptileai

mateo-berri · 2026-09-20T02:30:52Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a3e9ed3. Configure here.}

mateo-berri

LGTM

devin-ai-integration Bot requested a review from a team September 20, 2026 01:50

devin-ai-integration Bot assigned mateo-berri Sep 20, 2026

greptile-apps Bot reviewed Sep 20, 2026

View reviewed changes

mateo-berri added the run-ci label Sep 20, 2026

cursor Bot reviewed Sep 20, 2026

View reviewed changes

mateo-berri approved these changes Sep 20, 2026

View reviewed changes

mateo-berri merged commit ef7da9b into main Sep 20, 2026
97 of 98 checks passed

mateo-berri deleted the litellm_mcp_cold_worker_tools_call branch September 20, 2026 03:04

seanb4t mentioned this pull request Sep 28, 2026

[Bug]: MCP tools/call on a single-server endpoint returns 404 "Tool not found" when a different replica served tools/list (oauth_passthrough servers) #42655

Closed

alejandrosanz-stratio mentioned this pull request Oct 6, 2026

[Bug]: MCP tools/call with an unprefixed tool name on /{server_name}/mcp resolves via the global tool-name mapping, not the server in the path (404 when cold, 403 when another server has a tool with the same name) #44831

Open

This was referenced Oct 7, 2026

fix(container): update image ghcr.io/berriai/litellm-non_root (v1.104.0 → v1.104.3) vrozaksen/home-ops#3155

Closed

feat(container): update image ghcr.io/berriai/litellm-non_root (v1.104.0 → v1.105.0) vrozaksen/home-ops#3186

Merged

Navigation Menu

Repository navigation

fix(mcp): tools/call no longer 404s on a worker that has not served tools/list - #42072

fix(mcp): tools/call no longer 404s on a worker that has not served tools/list#42072
mateo-berri merged 2 commits into
mainfrom
litellm_mcp_cold_worker_tools_call

devin-ai-integration Bot commented Sep 20, 2026 •

edited by mateo-berri

Loading

Uh oh!

devin-ai-integration Bot commented Sep 20, 2026 •

edited by mateo-berri

Loading

Uh oh!

codspeed Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

greptile-apps Bot Sep 20, 2026

Uh oh!

codecov Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

mateo-berri commented Sep 20, 2026

Uh oh!

mateo-berri commented Sep 20, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

mateo-berri left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

devin-ai-integration Bot commented Sep 20, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (82ddab2)

Claude Code calls the true_passthrough tool six times

Cold tools/call on the true_passthrough server

One tools/list, then cold tools/call again

REST tools/call with server_id on a cold worker

Cold tools/call on the plain http and client_credentials servers (control)

Cold tools/call for a tool the server does not have

After (a3e9ed3)

Claude Code calls the true_passthrough tool six times

Cold tools/call on the true_passthrough server

One tools/list, then cold tools/call again

REST tools/call with server_id on a cold worker

Cold tools/call on the plain http and client_credentials servers (control)

Cold tools/call for a tool the server does not have

What the run saw next to the fix

Type

Caveats (if any)

Final Attestation

Uh oh!

devin-ai-integration Bot commented Sep 20, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

greptile-apps Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Findings

Summary

Uh oh!

Uh oh!

greptile-apps Bot Sep 20, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Sep 20, 2026

Uh oh!

mateo-berri commented Sep 20, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Sep 20, 2026 •

edited by mateo-berri

Loading

Before (`82ddab2`)

After (`a3e9ed3`)

devin-ai-integration Bot commented Sep 20, 2026 •

edited by mateo-berri

Loading

codspeed Bot commented Sep 20, 2026 •

edited

Loading

greptile-apps Bot commented Sep 20, 2026 •

edited

Loading

codecov Bot commented Sep 20, 2026 •

edited

Loading