fix(mcp): stop 404ing internal-only MCP servers behind a load balancer

[mateo-berri]

Relevant issues

Linear ticket

Resolves LIT-3964

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Type

🐛 Bug Fix

Changes

Regression: with use_x_forwarded_for: true and no mcp_trusted_proxy_ranges configured, every internal-only MCP server (available_on_public_internet: false) started returning 404 as soon as an X-Forwarded-For header was present, which is what every load balancer sends. So this hit real MCP clients (Cursor, mcp-inspector, etc.) during discovery and connection in production while a direct localhost call without XFF looked fine

Root cause is IPAddressUtils.get_mcp_client_ip in litellm/proxy/auth/ip_address_utils.py. When XFF was enabled and present but no trusted ranges were configured, it returned "" to "fail closed". That empty string then flowed straight into the IP access-control gate, where is_internal_ip("") is False, so the caller was classified external; the internal-only server got filtered out of the registry and the named-server lookup (get_mcp_server_by_name) raised 404

This was introduced in #28008 (merged to the release line as its internal-staging copy #28356, commit 6d6eda81). Before that change the same branch fell through to parse XFF, which is why 1.88.0 worked and 1.89.0 / 1.89.2 did not

The fix honours X-Forwarded-For when the direct peer is itself internal (a private reverse proxy in front of the gateway), and only ignores it, classifying by the direct connection IP, when the peer is a public address or falls outside the configured trusted ranges. This restores access for internal clients behind a load balancer while actually closing a gap the pre-1.89 behaviour had: a public caller connecting directly to the gateway and spoofing an internal X-Forwarded-For is now classified by its real public peer instead of being trusted. The hardened path is unchanged; set mcp_trusted_proxy_ranges to validate the proxy explicitly

A follow-up commit hardens the unknown-peer case raised in review (request.client is None, possible with Unix-socket transports and some ASGI test clients). The check is now framed as peer_is_private_proxy, and the untrusted path returns direct_ip or "" so a request with no identifiable peer fails closed to the external "" sentinel rather than returning None, which would mean "no filtering" downstream and leak internal access

A further commit makes the private-proxy check classify the direct peer with the same mcp_internal_ip_ranges the access-control gate uses, instead of the RFC1918 defaults. Otherwise a reverse proxy that is internal only under a custom CIDR was treated as untrusted, its X-Forwarded-For dropped and its own IP returned, so a public client forwarded through it could inherit the proxy's internal classification and reach internal-only servers

A final commit corrects the one-shot use_x_forwarded_for warning text, which still claimed access-control client IPs would use the proxy's literal request values; with the carve-out, access-control now honours X-Forwarded-For when the direct peer is internal, so the message would have misled the very operators this fix helps

Tests

tests/test_litellm/proxy/auth/test_mcp_ip_filtering.py:

• test_internal_client_behind_private_proxy_without_trusted_ranges_is_internal is the direct regression for LIT-3964; it would return "" (and 404) before this change
• test_internal_client_behind_proxy_can_reach_internal_only_server ties the two halves together end to end the way production does (extract client IP, then filter_server_ids_by_ip) and asserts the internal-only server survives the filter
• test_public_direct_peer_spoofing_xff_stays_external and test_external_client_behind_private_proxy_stays_external keep the security assertions of the two cases Support MCP OAuth passthrough and issuer-scoped JWT auth #28008 added; both callers are still classified external, just by their real address rather than by the empty-string sentinel
• test_unknown_direct_peer_spoofing_xff_stays_external and test_unknown_peer_spoofing_xff_cannot_reach_internal_only_server cover the request.client is None case: the first asserts the result is non-None and external, the second asserts the internal-only server is dropped by filter_server_ids_by_ip so a forged XFF from an unidentifiable peer cannot reach it
• test_xff_honoured_when_peer_internal_under_custom_ranges and test_public_client_via_custom_internal_proxy_cannot_reach_internal_only_server cover custom mcp_internal_ip_ranges: the first asserts the forwarded client IP is honoured when the peer is internal only under a custom CIDR, the second asserts a public forwarded client cannot reach an internal-only server through such a proxy

Screenshots / Proof of Fix

Minimal config (internal-only server, XFF on, no trusted ranges):

general_settings:
  master_key: sk-1234
  use_x_forwarded_for: true

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: gpt-4o-mini

mcp_servers:
  internal_math:
    transport: stdio
    command: python3
    args:
      - tests/mcp_tests/mcp_server.py
    allow_all_keys: true
    available_on_public_internet: false

python litellm/proxy/proxy_cli.py --config .yaml --detailed_debug --reload 2>&1 | tee litellm.log

# WITH an X-Forwarded-For header (what a load balancer sends)
curl -s -o /dev/null -w "with XFF:    %{http_code}\n" \
  -H "X-Forwarded-For: 10.0.0.9" \
  "http://localhost:4000/internal_math/authorize?redirect_uri=http://example/cb"

# WITHOUT the header
curl -s -o /dev/null -w "without XFF: %{http_code}\n" \
  "http://localhost:4000/internal_math/authorize?redirect_uri=http://example/cb"

On 1.89.x the with XFF line returns 404 (server filtered out); after this fix both lines return the same non-404 (e.g. 400 client_id required, meaning the server was found), matching 1.88.0

---

Note

Medium Risk
Changes MCP access-control IP extraction and trust rules for X-Forwarded-For; mistakes could expose internal-only servers or break clients behind proxies, but behavior is heavily tested and tightens some spoofing cases.

Overview
Fixes LIT-3964: with use_x_forwarded_for on and no mcp_trusted_proxy_ranges, get_mcp_client_ip no longer always returns "" when X-Forwarded-For is present. That empty string made every caller look external and internal-only MCP servers 404 behind load balancers.

get_mcp_client_ip now trusts X-Forwarded-For when the direct peer is an internal reverse proxy (mcp_internal_ip_ranges, same rules as access control) and trusted ranges are not pinned. Public, out-of-range, or missing request.client peers still ignore forged XFF and classify by the direct connection (direct_ip or ""), so spoofing cannot reach available_on_public_internet=false servers and unknown peers do not return None (which would skip filtering).

The one-shot operator warning text is updated to describe this carve-out vs explicit mcp_trusted_proxy_ranges. Tests cover the regression, spoofing, custom internal CIDRs, and end-to-end filter_server_ids_by_ip.

^{Reviewed by Cursor Bugbot for commit a02e058. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

---

Generated by Claude Code

[greptile-apps]

Greptile Summary

Fixes LIT-3964: get_mcp_client_ip no longer returns "" for every request that carries an X-Forwarded-For header when use_x_forwarded_for is enabled but mcp_trusted_proxy_ranges is not configured, which was causing internal-only MCP servers to 404 behind any load balancer.

• Core logic change (ip_address_utils.py): when is_request_from_trusted_proxy returns False and no explicit mcp_trusted_proxy_ranges is set, a new peer_is_private_proxy check determines whether the direct peer is internal under mcp_internal_ip_ranges; if it is, XFF is honoured (the fix); otherwise the function returns direct_ip or "", ensuring public callers, out-of-range callers, and unknown peers (request.client is None) all fail closed and cannot spoof an internal X-Forwarded-For to reach available_on_public_internet=false servers.
• Test coverage (test_mcp_ip_filtering.py): eight new tests cover the regression path, public-peer spoofing, unknown-peer spoofing, external clients behind private proxies, custom CIDR ranges, and two end-to-end filter_server_ids_by_ip checks; two prior tests are renamed and their assertions updated to match the improved return value (real IP rather than sentinel "").

Confidence Score: 5/5

Safe to merge — the change is tightly scoped to the MCP client-IP extraction path, all affected branches are covered by new and updated unit tests, and the spoofing-prevention properties are explicitly verified end-to-end.

The fix correctly handles all four peer cases (public direct, internal direct, unknown, custom CIDR) and returns direct_ip or "" rather than None on the untrusted path, preserving the fail-closed contract downstream. Existing security assertions are retained; updated assertions reflect a stricter return value (actual IP instead of sentinel "") which does not weaken any security guarantee.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/auth/ip_address_utils.py	Core fix: get_mcp_client_ip now trusts XFF when the direct peer is an internal private proxy (using mcp_internal_ip_ranges), restoring access for internal MCP clients behind load balancers. Public, out-of-range, and unknown peers still ignore XFF and fail closed to direct_ip or "". Docstring and one-shot warning text updated to match new behaviour. Logic is correct across all cases.
tests/test_litellm/proxy/auth/test_mcp_ip_filtering.py	Adds eight new tests covering: the LIT-3964 regression path, the spoofing prevention cases (public direct peer, unknown peer), external clients via private proxies, custom mcp_internal_ip_ranges, and two end-to-end filter_server_ids_by_ip scenarios. Existing tests renamed and assertions updated to match the more-precise (public IP rather than "") return value while retaining the is_internal_ip security assertion.

_{Reviews (4): Last reviewed commit: "docs(mcp): correct the XFF-without-trust..." | Re-trigger Greptile}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Custom internal ranges bypass XFF
  • The branch already contains commit 87f7fc7 which parses mcp_internal_ip_ranges and passes it into the peer_is_private_proxy is_internal_ip call so the private-proxy carve-out and the downstream access-control gate share one definition of internal, plus regression tests covering this exact scenario.

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a02e058. Configure here.}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

This would cause a security regression. Closing