fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls

[ryan-crabbe-berri]

Relevant issues

Supersedes #30411, which removed the same phantom 401 with a token-shape heuristic (sk- / JWT). This replaces that heuristic with the existing per-server delegate_auth_to_upstream flag, so the no-validation passthrough is gated on an explicit operator opt-in rather than a guess about the token's shape.

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

• Branch creation CI run
  Link:
• CI run for the last commit
  Link:
• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Verified against a live proxy on localhost:4000 (LITELLM_OTEL_V2=1, OTLP exporter to Jaeger) with a real Atlassian MCP server (auth_type=oauth2, delegate_auth_to_upstream=true) and the official MCP Inspector driving real OAuth plus a real tools/list call.

After the fix, a delegated server skips LiteLLM validation, so the upstream bearer is never mistaken for a virtual key and no failed-auth span is emitted:

curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:4000/atlassian/mcp \
  -H "Authorization: Bearer <atlassian-token>" \
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"c","version":"1"}}}'
# -> 200; proxy log shows no "Virtual Key expected" and no OTEL failure handler

The phantom is the validation that used to run regardless of server type. With the flag off you can see exactly what was being logged on every delegated call before this change: the bearer is validated as a virtual key, fails, and the failure is recorded to the tracer:

user_api_key_auth.py:1662 HTTPException: 401: LiteLLM Virtual Key expected. Received=...
OpenTelemetry Logger: Failure Handler ... 'status': 'failure', 'error_code': '401'

The real tools/list call through the MCP Inspector produces a clean trace in Jaeger (service litellm): a single POST /{mcp_server_name}/mcp span, HTTP 200, with no errored auth span.

Before

After

Type

Bug Fix

Changes

In process_mcp_request, the delegate_auth_to_upstream check now runs before user_api_key_auth for an Authorization-only request. A delegated oauth2 server (oauth2 with the flag set, non client_credentials) skips LiteLLM validation entirely and forwards the bearer upstream, so a tool call that succeeds no longer carries a failed-auth 401 span. The check is consolidated into one branch that covers both the no-credential discovery request and the authenticated call carrying the upstream bearer, and the now-unused _target_servers_use_oauth2 helper is removed. The auth_type=none pass-through cold-start path is unchanged.

The tests update the OAuth2 and delegate suites to the new contract and add regression coverage: a delegated server with a bearer never calls user_api_key_auth; a non-delegated oauth2 server whose validation fails surfaces a real 401 for both HTTPException and ProxyException; an Authorization bearer on a delegated server is treated as an upstream token while x-litellm-api-key still resolves identity.

Behavior change worth calling out for reviewers. A non-delegated oauth2 server that previously relied on implicit bearer passthrough will now return a real 401 until the operator sets delegate_auth_to_upstream: true; that is the same change that closes the over-broad passthrough which admitted any non-sk bearer on any oauth2 server as anonymous. And on a delegated server the Authorization header is now always an upstream token, even when it is sk-shaped, so callers who want LiteLLM identity or spend tracking on a delegated server must send x-litellm-api-key instead.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes the phantom 401 OTEL span that appeared on successful tool calls to delegated oauth2 MCP servers. The root cause was that process_mcp_request previously ran user_api_key_auth on every Authorization bearer — including upstream tokens that were never LiteLLM keys — before the delegate_auth_to_upstream gate fired. The fix moves that gate above the auth call for all requests that don't carry an explicit x-litellm-api-key, so a delegated server's bearer is forwarded upstream without ever entering LiteLLM validation.

• The delegate_auth_to_upstream check now covers both the no-credential discovery request and the authenticated upstream-bearer request in one consolidated branch, replacing the now-removed _target_servers_use_oauth2 helper.
• Non-delegated oauth2 servers lose their implicit bearer-passthrough fallback: a failed LiteLLM auth now propagates as a real 401 unless the server is a passthrough cold-start target (_is_mcp_passthrough_cold_start).
• The behavioral change for Authorization: Bearer <litellm-sk-key> on delegated servers is intentional and documented — callers needing LiteLLM identity or spend tracking must use x-litellm-api-key instead.

Confidence Score: 4/5

The fix is correctly scoped and improves security for non-delegate oauth2 servers, but the Authorization header semantics on delegated servers have changed in a way that silently breaks existing callers who relied on sending LiteLLM sk-keys via that header for spend tracking.

The delegate gate is narrowly correct — it only fires for servers where every target has auth_type=oauth2 and delegate_auth_to_upstream=true, correctly fails closed for client-credentials and mixed-target requests. The remaining concern is that existing setups using Authorization: Bearer sk-... against a delegated server for LiteLLM identity resolution will silently lose spend tracking after upgrade, with no opt-out path short of migrating all callers to x-litellm-api-key. The project rule for this class of change asks for a user-controlled feature flag rather than a hard cutover, and that flag is absent.

litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py — specifically the delegate branch at lines 225-240 where Authorization-bearer behavior on delegated servers changes without a transition flag.

Important Files Changed

Filename	Overview
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py	Core auth routing refactored to skip LiteLLM validation for all requests to delegated-upstream servers; removes _target_servers_use_oauth2, consolidates delegate branch, and tightens the fallback to auth_type=none cold-start only. Introduces a documented breaking contract change for Authorization bearers on delegated servers.
tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py	Tests updated to match the new contract: delegated servers assert user_api_key_auth is never called; non-delegate oauth2 servers now expect real 401 propagation. The regression guard for sk-key-via-Authorization on delegated servers is reversed to document the new intended behavior.

_{Reviews (2): Last reviewed commit: "fix(mcp): drop phantom 401 span on deleg..." | Re-trigger Greptile}

[greptile-apps]

Test regression guard reversed without a bug fix

test_litellm_key_via_authorization_header_not_bypassed was an explicit regression guard: it verified that an sk-shaped key sent via Authorization on a delegated server still triggered user_api_key_auth and resolved identity. The replacement test documents the exact opposite — mock_auth.assert_not_called() and user_id is None — meaning the guard has been removed rather than the underlying invariant fixed. If LiteLLM validation (and spend tracking) for sk-keys via Authorization on delegated servers is now intentionally dropped, the backwards-incompatible change needs a feature flag before the guard can be removed; otherwise this test hides a real regression in spend-tracking coverage.

Rule Used: What: Flag any modifications to existing tests and... (source)

[ryan-crabbe-berri]

@greptileai the entire point of delegate auth to upstream is the proxy is a passthrough and we dont do any auth on our end though so we shouldnt do user_api_key_auth because this is going to fail if its an oauth token and the tracer will stamp it as a 401 even though we are going to eventually have a successful tool call

the code prebviously did this and the happy path for this usage worked for customers but it just meant that the mcp would work but all the traces woudl show as 401's

[greptile-apps]

Yes, exactly right — that's precisely what this PR fixes. The root cause was that the old code only skipped user_api_key_auth for the no-credential delegated case (not litellm_api_key AND delegate_auth_to_upstream). When Authorization: Bearer <upstream-token> was present, the check fell through to the oauth2_headers branch, which always attempted LiteLLM auth first. That auth was guaranteed to fail for any real OAuth token, producing the phantom 401 span in the tracer — even though the call would succeed end-to-end.

The fix reorders the branches so delegate_auth_to_upstream is checked before oauth2_headers, covering both the no-credential discovery request and the authenticated call with the upstream bearer. The trade-off worth noting: on a delegated server, Authorization is now always treated as an upstream token — even sk--shaped ones. Callers who need LiteLLM identity/spend tracking on a delegated server must supply x-litellm-api-key instead (see test_explicit_litellm_key_takes_precedence_over_delegate). The diff and the renamed test test_authorization_bearer_on_delegate_server_treated_as_upstream document this contract change explicitly.

[yuneng-berri]

@greptile re-review this given the new context Ryan provided and give a score out of 5

[tin-berri]

LGTM! Great Job Ryan!