feat(mcp): bound outbound tool-call concurrency per MCP server

[tin-berri]

Relevant issues

• Root cause: LiteLLM dispatches MCP tool calls to each upstream with unbounded concurrency, so backends that process requests in batches get overwhelmed and time out under load
• Fix: add an optional per-server max_concurrent_requests cap that queues excess outbound calls so no MCP server ever gets more than N in flight at once

Linear ticket

Resolves LIT-2749

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

When LiteLLM needs to make several MCP tool calls it dispatches them with an unbounded asyncio.gather, so every call hits the upstream server at the same instant. Backends that process requests in batches get overwhelmed and time out, and raising the per-call timeout does not help because the burst still lands all at once.

Repro uses a small instrumented MCP server (streamable-http) exposing one tool, slow_tool, that reports how many calls it is handling at the same moment. It is registered on a live proxy via config

mcp_servers:
  batch_backend:
    url: "http://127.0.0.1:9100/mcp"
    transport: "http"

then six tool calls are fired concurrently through the proxy

for i in $(seq 1 6); do
  curl -s -X POST http://127.0.0.1:4000/mcp-rest/tools/call \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"server_id":"55c09d47c38f455f356f325a176b49e3","name":"slow_tool","arguments":{}}' &
done; wait

Before, with no cap, the backend sees all six at once

in_flight_at_entry -> 1 2 3 4 5 6   (peak concurrency 6)

After, with max_concurrent_requests: 2 on that server, the backend never sees more than two and the rest queue until a slot frees

in_flight_at_entry -> 1 1 1 2 2 2   (peak concurrency 2)

Type

🐛 Bug Fix

Changes

Adds an optional per-server max_concurrent_requests field, mirroring the existing timeout field across the request models, the DB table, the Prisma schema (with a migration), and the runtime MCPServer. The MCPServerManager keeps one asyncio.Semaphore per server_id, created lazily from that value, and the outbound tool-call path acquires it before calling the upstream. Calls beyond the cap wait for a slot rather than being rejected, so a batch backend is never sent more than the configured number of simultaneous requests. Leaving the field unset preserves today's unbounded behavior, and a non-positive value is treated as unlimited so it can never deadlock on a zero-permit semaphore. The limiter is keyed on server_id, so two servers never throttle each other and the cap survives the registry atomic-swap on config reload.

The semaphore is in-process, so the cap is enforced per worker. The MCP gateway runs single-worker today, so per-process is the effective global cap; a Redis-backed distributed limiter is the natural follow-up when the gateway becomes multi-worker.

Known limitation: the semaphore is created once per server_id and its permit count is fixed for the process lifetime, so changing max_concurrent_requests on an already-active server does not take effect until the proxy restarts. Config-defined caps and newly created servers always reflect the configured value; only a runtime update to a live server is deferred to restart. Refreshing the limiter when the stored limit changes is a small follow-up.

OpenAPI-backed MCP servers

The same cap is enforced for OpenAPI-spec servers (spec_path), which dispatch through a separate handler. Verified live with a real OpenAPI spec pointing at a plain HTTP backend that reports its in-flight count, registered on the proxy and driven with six concurrent calls.

Before, no cap

in_flight_at_entry -> 1 2 3 4 5 6   (peak concurrency 6)

After, with max_concurrent_requests: 2

in_flight_at_entry -> 1 2 2 2 2 2   (peak concurrency 2)

[tin-berri]

@greptileai

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds an optional outbound concurrency cap for MCP server tool calls. The main changes are:

• Adds max_concurrent_requests to MCP server schemas, request models, and runtime models.
• Applies a per-server semaphore around regular MCP and OpenAPI-backed outbound tool calls.
• Keeps unset and non-positive values unbounded for backward-compatible behavior.
• Adds tests for capped, uncapped, non-positive, per-server, and OpenAPI concurrency behavior.

Confidence Score: 5/5

The change is narrowly scoped and covered by targeted tests for capped, uncapped, non-positive, per-server, and OpenAPI-backed MCP tool-call behavior.

The implementation consistently threads the new field through schemas, models, persistence, and runtime enforcement while preserving existing unbounded behavior when unset or non-positive.

T-Rex Logs

What T-Rex did

• Ran the MCP concurrency cap base test and observed in_flight_at_entry values 1 through 6 with peaks up to 6, indicating unbounded upstream concurrency.
• Ran the MCP concurrency cap head test and observed batched entries in groups of two, with peaks at 2 and six successful results, indicating queueing rather than rejection.
• Ran the MCP concurrency edge-case test and observed per-server isolation peaks (srv-a=1, srv-b=1) with a global peak of 2, plus semaphore_reused behavior of peak 2.
• Observed pre-change API fields: base output showed all models valid (200 OK) but the with_field dump reported has_field=false and value=null for max_concurrent_requests.
• Observed post-change API fields: head output showed with_field=true and value=2 for several MCP server request models, while the after state maintained 200 OK with value=null for absent fields.
• Compared pre-migration schema against head migration: an initial commit had no max_concurrent_requests column and inserts failed with 'no column named max_concurrent_requests'.
• Applied the head migration and confirmed max_concurrent_requests exists as INTEGER across schema copies; Python model uses Optional[int] = None; rows persist NULL and positive values; Prisma validation succeeds with DATABASE_URL.

_{Ran code and verified through T-Rex}

_{Reviews (5): Last reviewed commit: "feat(mcp): bound outbound tool-call conc..." | Re-trigger Greptile}

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[tin-berri]

Good catch on the OpenAPI path. Fixed in the latest commit: the limiter now wraps both dispatch branches in call_tool. OpenAPI-backed servers (spec_path) acquire the same per-server semaphore via _call_openapi_tool_handler, so max_concurrent_requests is honored regardless of MCP server type. Added a regression test (test_openapi_backed_server_also_respects_the_cap) that drives the OpenAPI branch and asserts peak in-flight stays at the cap; it fails if the wrap is removed.

@greptileai

[mateo-berri]

We gotta be careful to bump proxy extras for this migration. Otherwise LGTM, thanks!