fix(mcp): stop 404ing internal-only MCP servers when XFF is on without trusted proxy ranges

[mateo-berri]

Relevant issues

Fixes LIT-3964 (regression: all internal-only MCP servers return 404 Not Found)

Linear ticket

https://linear.app/litellm-ai/issue/LIT-3964/regression-all-internal-only-mcp-servers-are-returning-404-not-found

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Minimal repro config (internal-only server, XFF on, no trusted ranges):

general_settings:
  master_key: sk-1234
  use_x_forwarded_for: true
  # deliberately NO mcp_trusted_proxy_ranges

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: gpt-4o-mini

mcp_servers:
  internal_math:
    transport: stdio
    command: python3
    args:
      - tests/mcp_tests/mcp_server.py
    allow_all_keys: true
    available_on_public_internet: false   # internal-only

Run it:

python litellm/proxy/proxy_cli.py --config <that_file>.yaml --detailed_debug --reload 2>&1 | tee litellm.log

Probe the named-server resolution path with and without an X-Forwarded-For header (what a load balancer sends):

curl -s -o /dev/null -w "with XFF:    %{http_code}\n" \
  -H "X-Forwarded-For: 10.0.0.9" \
  "http://localhost:4000/internal_math/authorize?redirect_uri=http://example/cb"

curl -s -o /dev/null -w "without XFF: %{http_code}\n" \
  "http://localhost:4000/internal_math/authorize?redirect_uri=http://example/cb"

Before this fix (v1.89.0 / v1.89.2): with XFF: 404 (internal-only server hidden), without XFF: non-404. After this fix: both are non-404 (the server resolves and the request proceeds to normal auth handling), matching v1.88.0.

Type

🐛 Bug Fix

Changes

Root cause

IPAddressUtils.get_mcp_client_ip returned "" ("fail closed") when use_x_forwarded_for: true was set but mcp_trusted_proxy_ranges was not configured. That empty IP made is_internal_ip("") return False, so every available_on_public_internet=false server was classified as external, hidden by _is_server_accessible_from_ip, and get_mcp_server_by_name returned None; the MCP discovery / SSE / streamable-HTTP routes then turned that None into a 404. The combination of XFF enabled plus no trusted ranges is very common (LiteLLM behind a load balancer), so it broke every internal-only server while looking fine on naive localhost testing (no XFF header means request.client.host is 127.0.0.1).

The fail-closed branch was introduced in #28356 (squash 6d6eda8101), shipped in v1.89.0; v1.88.x never contained it, matching the working-vs-broken versions reported.

Fix

Honor X-Forwarded-For whenever use_x_forwarded_for is enabled, so a client behind a reverse proxy is classified by its real (forwarded) IP rather than the proxy's peer address. This matches pre-1.89 behavior and how the rest of the proxy already treats use_x_forwarded_for (e.g. _check_valid_ip). The spoof defense is kept only when mcp_trusted_proxy_ranges is configured: an untrusted direct peer still falls back to its literal IP so a forged header cannot grant internal access. OAuth discovery URL building is unaffected; it keeps failing closed via is_request_from_trusted_proxy and get_request_base_url.

The two existing unit tests that encoded the fail-closed empty-string behavior as intended were corrected to assert the restored behavior, and a TestInternalOnlyServerResolutionRegression class was added that drives the exact 404 boundary (get_mcp_client_ip -> get_mcp_server_by_name) for both a forwarded internal client (now resolves) and a forwarded external client (still hidden). A mutation check confirms all four new/updated assertions fail against the old return "" code and pass against the fix.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes a regression introduced in v1.89.0 where all available_on_public_internet=false MCP servers returned 404 whenever use_x_forwarded_for: true was set without mcp_trusted_proxy_ranges. The root cause was a fail-closed branch in get_mcp_client_ip that returned "" in that configuration, causing is_internal_ip to classify every caller as external.

• ip_address_utils.py: Removes the fail-closed return "" branch. The new logic only falls back to the direct peer's IP when mcp_trusted_proxy_ranges IS configured and the direct peer is outside the trusted range; otherwise it calls _get_request_ip_address with use_x_forwarded_for=True, honoring the forwarded header unconditionally. This restores pre-v1.89 behavior.
• test_mcp_ip_filtering.py: Two tests that were asserting the (now-removed) fail-closed "" return value are updated to assert the actual forwarded IP; a new TestInternalOnlyServerResolutionRegression class drives the full get_mcp_client_ip → get_mcp_server_by_name path for both an internal and an external forwarded caller.

Confidence Score: 3/5

Safe to merge for the 404 regression fix, but the access-control path now trusts X-Forwarded-For without validation or a warning whenever mcp_trusted_proxy_ranges is absent, which is worth resolving before this reaches production deployments that expose the proxy directly.

The core logic change in get_mcp_client_ip restores pre-v1.89 behavior and fixes a real regression, and the new regression tests are well-constructed. However, the revised code trusts the raw XFF header for access-control decisions (deciding which available_on_public_internet=false servers are visible) with no warning to operators, while the only existing warning about this configuration targets the OAuth URL-building path and says XFF will NOT be trusted — the opposite of what now happens for IP classification.

litellm/proxy/auth/ip_address_utils.py — specifically the get_mcp_client_ip function and the relationship between its XFF-trust behavior and the is_request_from_trusted_proxy warning message.

Security Review

• IP spoofing / access control bypass (ip_address_utils.py lines 197–206): When use_x_forwarded_for=True and mcp_trusted_proxy_ranges is absent, get_mcp_client_ip now trusts X-Forwarded-For unconditionally. An attacker with direct TCP access to the proxy can forge X-Forwarded-For: 10.0.0.1 and be classified as internal, making every available_on_public_internet=false MCP server visible to them. No warning is emitted on this code path; the existing _warned_xff_without_trusted_ranges warning fires only from the OAuth URL-building path (is_request_from_trusted_proxy), not from the access-control path.

Important Files Changed

Filename	Overview
litellm/proxy/auth/ip_address_utils.py	Reverts the v1.89 fail-closed behavior in get_mcp_client_ip so that XFF is honored whenever use_x_forwarded_for=True; fixes the 404 regression but reintroduces unconditional XFF trust (no spoof defense, no warning) when mcp_trusted_proxy_ranges is absent
tests/test_litellm/proxy/auth/test_mcp_ip_filtering.py	Two existing tests updated to reflect the restored XFF-trusting behavior (assertions changed from "" to the actual forwarded IPs); new TestInternalOnlyServerResolutionRegression class added covering the full get_mcp_client_ip → get_mcp_server_by_name path for both internal and external forwarded clients

Comments Outside Diff (2)

1. litellm/proxy/auth/ip_address_utils.py, line 147-156 (link)

   Warning message is misleading for access-control callers

   The existing one-shot warning says "X-Forwarded-Host/Proto will NOT be trusted when building MCP OAuth discovery URLs." This is accurate for OAuth URL construction, but it is the opposite of what happens in get_mcp_client_ip under the same configuration: the X-Forwarded-For header IS trusted (and used for access control) when no trusted ranges are configured. An operator who sees this log line could reasonably conclude that XFF is completely ignored, while in reality it is deciding which available_on_public_internet=false servers their callers can reach. The warning should explicitly state that X-Forwarded-For will still be used for IP-based access control in this configuration.

2. litellm/proxy/auth/ip_address_utils.py, line 197-206 (link)

   XFF trusted blindly for access control when no trusted-proxy ranges are set

   When use_x_forwarded_for=True and mcp_trusted_proxy_ranges is not configured, the new condition short-circuits and the code falls through to _get_request_ip_address(request, use_x_forwarded_for=True), which reads the raw X-Forwarded-For header without any peer validation. An external attacker with direct access to the proxy can send X-Forwarded-For: 192.168.1.1 and be classified as an internal host, bypassing every available_on_public_internet=false restriction. No warning is emitted on this code path — the _warned_xff_without_trusted_ranges warning only fires from is_request_from_trusted_proxy (the OAuth URL-building path), so access-control callers get no indication that XFF is being trusted unconditionally. A one-shot warning analogous to _warned_xff_without_trusted_ranges should fire here so operators know their available_on_public_internet=false access control is based on an unvalidated header.

_{Reviews (1): Last reviewed commit: "style: apply black formatting to match C..." | Re-trigger Greptile}

[mateo-berri]

Thanks @greptileai. Both points are about observability of the same configuration; addressed in 3d24ec2 without changing the trust model.

On the P1 security note (unconditional XFF trust for access control when mcp_trusted_proxy_ranges is unset): this is intentional. use_x_forwarded_for: true already means "trust X-Forwarded-For" everywhere else in the proxy (_check_valid_ip / _get_request_ip_address), and it is exactly the pre-1.89 behavior this PR restores; second-guessing XFF only for MCP would diverge from that contract and, depending on the proxy's peer IP, can still leave internal-only servers 404ing. The explicit spoof defense is mcp_trusted_proxy_ranges, which validates the proxy CIDR before XFF is honored. To close the visibility gap you flagged, I added a dedicated one-shot warning (_warn_xff_trusted_for_access_control) that fires from the access-control path stating XFF is trusted as-is for IP-based access control and that a direct caller could spoof an internal IP, pointing operators to set mcp_trusted_proxy_ranges.

On the P2 misleading-message note: the existing warning is now scoped to OAuth URL building ("when building MCP OAuth discovery URLs"), and the new access-control warning explicitly states XFF IS used for access control, so an operator in this config sees both facts rather than concluding XFF is fully ignored. Covered by test_warns_once_that_xff_is_trusted_for_access_control_without_ranges and test_no_access_control_warning_when_trusted_ranges_configured.

[greptile-apps]

Good follow-up. Your additions in commit 3d24ec2 properly separate the warnings and clarify what's happening in each path.

On the security stance: Your justification is sound—use_x_forwarded_for: true already establishes a trust contract for XFF elsewhere in the proxy (like _check_valid_ip), and this PR simply aligns MCP access control to that same contract while restoring the pre-1.89 behavior. The dedicated warning you added (_warn_xff_trusted_for_access_control) makes the security boundary explicit, which is the right observability tradeoff.

On the warning improvements: The refactored warnings now properly distinguish OAuth URL building (XFF not trusted without ranges) from access control (XFF is trusted). That removes the misleading implication. The one-shot pattern prevents log spam, and the test coverage (test_warns_once_that_xff_is_trusted_for_access_control_without_ranges + test_no_access_control_warning_when_trusted_ranges_configured) confirms both the presence and absence conditions.

The regression tests targeting the exact 404 path (TestInternalOnlyServerResolutionRegression) drive the reported bug end-to-end, which is solid coverage for preventing future regressions on this boundary.

Approved. The observability gaps are closed without changing the underlying trust model, which correctly matches the proxy-wide XFF semantics.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[mateo-berri]

This would cause a security regression. Closing