feat(mcp): migrate the token_exchange (OBO) arm to the v2 resolver by tin-berri · Pull Request #31526 · BerriAI/litellm

tin-berri · 2026-06-27T20:32:22Z

Relevant issues

Part of the MCP Gateway v2 migration; the next outbound-credential mode after authorization_code (which landed via #31473 and #31493, now merged into litellm_internal_staging). This branch is rebased onto staging, so the diff is just the token_exchange changes.

Linear ticket

N/A

Pre-Submission checklist

I have added meaningful tests
My PR passes all CI/CD checks (lint, format, unit tests) locally
My PR's scope is as isolated as possible; it only migrates the token_exchange arm
I have requested a Greptile review and received a Confidence Score of at least 4/5

What this is

The RFC 8693 token exchange (OBO) mode, migrated end to end onto the v2 outbound-credential resolver, the same way authorization_code was. A server declaring oauth2_token_exchange now resolves its upstream credential through resolve_credentials instead of v1's resolve_mcp_auth/TokenExchangeHandler.

The flow is the true on-behalf-of swap: the gateway takes the caller's inbound token, POSTs the RFC 8693 grant to the configured exchange endpoint authenticating as its own OAuth client, and forwards the returned upstream-bound token to the MCP server. The caller's token only ever reaches the IdP exchange endpoint, never the upstream.

Changes

Rfc8693TokenExchanger (new) is the pure core: it builds the RFC 8693 form, runs the exchange through an injected HTTP edge, and returns the upstream token as a typed Result[OAuthToken, CredError]. The exchanged token is cached and single-flighted per caller token, server, and exchange config (the cache key hashes the subject token together with the config that minted it), reusing the shared in-process foundation, so a repeated caller token skips the IdP round-trip and concurrent calls collapse to one exchange; a rotated caller token or a rotated server config hashes to a new key and re-exchanges rather than serving a stale token. v1's exchanged-token cache is per-process too, so there is no cross-replica machinery here.

TokenExchangeConfig gains an audience field that is forwarded only when the operator configured one. Both audience and resource are optional in RFC 8693 and the authorization server applies its own default when neither is sent, so fabricating an audience from the URL risks invalid_target; sending it only when set matches both the spec's intent and v1.

The resolver arm reads the caller's inbound token and swaps it via the injected TokenExchanger. to_server_spec maps a complete oauth2_token_exchange server (an endpoint plus client credentials, mirroring v1's has_token_exchange_config) to TokenExchangeConfig, and defers an incomplete one to v1. The egress wires the LazyTokenExchanger into the provider.

One deliberate behavior change, gated behind the v2 resolver flag: a token_exchange server hit without a caller token now fails closed with a plain 401 rather than falling through to v1's client_credentials grant. The v2 contract is that a configured mode whose credential is absent errors rather than silently using a weaker source, so the call site now scopes the per-server browser-OAuth challenge to authorization_code and lets token_exchange raise its own 401.

Errors are modeled as values throughout: a missing endpoint or client credential is misconfigured, an IdP that returns no usable token is upstream_unavailable, and the I/O lives only in the injected post adapter.

Two hardening additions land on top of the migration. The exchanger now validates the RFC 8693 response token_type before minting the upstream header: an exchanged token whose token_type is present and not Bearer (for example N_A, which RFC 8693 uses for a token that is not a standalone access token) fails closed as upstream_unavailable rather than being forwarded as a bogus Bearer, and an absent token_type still defaults to Bearer per RFC 6750 so IdPs that omit the field keep working

TokenExchangeConfig also gains token_endpoint_auth_method, threaded from the server config, so an OBO server can authenticate to the exchange endpoint with client_secret_basic (HTTP Basic, the OIDC default) as well as the existing client_secret_post. The exchanger builds its client auth through the shared build_token_endpoint_client_auth helper, so Basic sends the credentials in the Authorization header and keeps them out of the body while post keeps them in the form, matching how the v1 token endpoints already apply the two methods

Tests

New test_token_exchanger.py and test_token_exchange_provider.py pin the exchange and form, the per-caller-token cache, single-flight under concurrency, TTL expiry on an injected clock, the rotated-token re-exchange, and the misconfigured/upstream_unavailable mappings. test_resolver.py covers the arm (OBO success, no-token 401, error propagation, fail-closed default) and test_adapter.py covers the mapping (complete vs incomplete config, audience present vs omitted, scopes, subject_token_type). The full outbound_credentials suite is green (213), including the non-Bearer token_type fail-closed cases and the client_secret_basic vs client_secret_post client-auth cases, ruff and format are clean, and basedpyright adds no new errors over the baseline.

Screenshots / Proof of Fix

Verified end to end against a real IdP doing RFC 8693, on the branch with --use_v2_migration_resolver.

Setup: Keycloak 26.2 (standard token exchange, GA) in Docker with realm mcp, a caller-client (alice's source token), a confidential litellm-gateway client (token exchange enabled), an upstream-mcp audience client, and audience mappers so the gateway may exchange alice's token for the upstream-mcp audience. A mock upstream MCP server records the Authorization it receives. The proxy runs an oauth2_token_exchange MCP server pointing token_exchange_endpoint at Keycloak and url at the mock.

Proof 1, Keycloak performs the real RFC 8693 swap (the exact request the gateway makes):

# alice's own token, then the gateway exchanges it for an upstream-mcp-audience token
curl -s -X POST $KC/realms/mcp/protocol/openid-connect/token \
  -d grant_type=urn:ietf:params:oauth:grant-type:token-exchange \
  -d client_id=litellm-gateway -d client_secret=$SECRET \
  -d subject_token=$SUBJECT -d subject_token_type=urn:ietf:params:oauth:token-type:access_token \
  -d audience=upstream-mcp

subject token claims:   {'azp': 'caller-client',   'aud': ['litellm-gateway','account'], 'sub': '3e54...'}
exchange response:      {'issued_token_type': 'urn:ietf:params:oauth:token-type:access_token', 'token_type': 'Bearer', 'expires_in': 600}
exchanged token claims: {'azp': 'litellm-gateway', 'aud': 'upstream-mcp',                  'sub': '3e54...'}
=> distinct tokens; audience swapped to upstream-mcp, same subject (alice)

Proof 2, the v2 token_exchange arm is live on the proxy and fails closed without a caller token (this exact message is from the new _token_exchange arm):

LiteLLM:WARNING mcp_server_manager.py - Failed to get tools from server keycloak_obo:
  401: unauthorized: Token exchange requires a caller token to exchange (OBO).

Proof 3, the real resolve_credentials (with the production build_token_exchanger, which POSTs to the live Keycloak) forwards the EXCHANGED token to the upstream, not alice's caller token:

resolve_credentials -> httpx.Auth: StaticHeaderAuth
alice SUBJECT token (presented to LiteLLM): {'azp': 'caller-client',   'aud': ['litellm-gateway','account']}
token the UPSTREAM received (forwarded):    {'azp': 'litellm-gateway', 'aud': 'upstream-mcp', 'sub': 'alice'}
forwarded == caller token ? False     # true OBO: the caller token never reaches the upstream
upstream audience swapped to 'upstream-mcp' ? True

Note on discovery: the aggregator's tools/list path does not thread the caller token, so an oauth2_token_exchange server's tools are not discoverable through the aggregator on the v2 path (v1 masked this with its no-subject client_credentials fallback, which v2 intentionally drops). The arm and the egress are correct as shown; making OBO usable end to end through the aggregator additionally needs the caller token threaded into discovery, tracked as a follow-up.

Type

New Feature

Changes

See above.

Note

High Risk
Changes authentication and RFC 8693 token exchange for upstream MCP calls, including fail-closed behavior and what bearer reaches upstream; mistakes could leak caller tokens or break OBO.

Overview
RFC 8693 on-behalf-of (OBO) for MCP servers with oauth2_token_exchange now runs through the v2 UpstreamCredentialProvider instead of v1 resolve_mcp_auth / TokenExchangeHandler.

The gateway takes the caller’s inbound bearer, POSTs the token-exchange grant to the configured IdP (as its own OAuth client), caches and single-flights the result per hashed (subject token + exchange config), and attaches the exchanged upstream bearer. to_server_spec maps complete server config to TokenExchangeConfig (endpoint or token_url, client creds, optional audience, token_endpoint_auth_method, scopes); incomplete configs still defer to v1. TokenExchangeConfig documents optional audience (no fabricated default) and adds audience + token_endpoint_auth_method on the model.

The resolver’s token_exchange arm returns 401 when there is no caller token (no v1 client-credentials fallback). Browser OAuth challenges on unauthorized are limited to authorization_code so OBO errors surface as plain 401s. Response hardening rejects non-Bearer token_type and clearly non-access issued_token_type before forwarding.

build_token_exchanger() wires Rfc8693TokenExchanger at MCP egress construction with the shared httpx POST adapter and in-process cache TTLs matching existing MCP OAuth constants.

^{Reviewed by Cursor Bugbot for commit 346088a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Deliberately not implemented

Scope notes from the OBO behavior-contract audit, recorded so review does not mistake these for oversights. The RFC 8707 resource parameter is not supported; targeting is via audience only, and none is fabricated when unset since the authorization server applies its own default. Client authentication to the exchange endpoint supports client_secret_basic and client_secret_post only; private_key_jwt and mTLS are out of scope for now. The response's token_type and issued_token_type are validated when present (a non-Bearer token_type or a refresh/id/saml issued_token_type fails closed) but their absence is tolerated rather than treated as a protocol error. A narrower scope granted in the response is not read back. The exchange endpoint URL is not restricted to https, so local and test IdPs work. The exchanged-token cache is in-process only, matching v1; durable or cross-instance persistence is a possible later step for multi-replica deployments. The cache key binds subject token, tenant, endpoint, audience, scopes, and client, but does not parse the subject to key on iss, and per-issuer exchange config routing does not exist since a server has exactly one configured IdP. Transient IdP failures surface as a retryable 503 with no gateway-side backoff loop

tin-berri · 2026-06-27T20:32:37Z

@greptileai

codecov · 2026-06-27T20:36:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-06-27T20:36:57Z

Greptile Summary

Migrates the oauth2_token_exchange (RFC 8693 OBO) mode onto the v2 outbound-credential resolver, completing the pattern established by the earlier authorization_code migration. Complete server configs now resolve through UpstreamCredentialProvider / Rfc8693TokenExchanger instead of v1's TokenExchangeHandler, while incomplete configs (missing endpoint or client creds) continue to defer to v1.

Rfc8693TokenExchanger is the new pure core: it POSTs the RFC 8693 form to the configured exchange endpoint (with injected HTTP), caches and single-flights the result per (hash(subject_token, full_config), server_id), validates token_type and issued_token_type before minting the upstream Bearer, and returns typed Result[OAuthToken, CredError] — never raises.
TokenExchangeConfig gains audience (sent only when operator-configured) and token_endpoint_auth_method.
The unauthorized-error handler in mcp_server_manager.py now scopes the browser-OAuth challenge to authorization_code only; token_exchange missing-caller-token errors surface as a plain 401.

Confidence Score: 5/5

Safe to merge; all four concerns raised in the prior review round are resolved and no new correctness issues were found.

The cache key now hashes the full exchange config so a config rotation forces a fresh exchange; eager construction via build_token_exchanger() eliminates the lazy-init race; _post_exchange_endpoint guards non-dict JSON bodies before field parsing; the LazyTokenExchanger wrapper is gone. Core logic is correct: RFC 8693 form is built properly, token_type and issued_token_type are validated before minting a Bearer upstream header, the browser-OAuth challenge is correctly scoped to authorization_code only, and the fail-closed path is exercised by tests.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/_experimental/mcp_server/outbound_credentials/token_exchanger.py	New pure core for RFC 8693 OBO: config-inclusive cache key, single-flight coordinator, Bearer/issued_token_type validation, and client-auth delegation. Clean.
litellm/proxy/_experimental/mcp_server/outbound_credentials/token_exchange_provider.py	Composition root: wires exchanger to real httpx post, guards non-dict JSON bodies, maps transport failures to None. No lazy-init race. Clean.
litellm/proxy/_experimental/mcp_server/outbound_credentials/resolver.py	Adds _token_exchange arm: requires inbound_token, delegates to injected TokenExchanger, propagates typed errors. Null-default exchanger returns misconfigured. Correct.
litellm/proxy/_experimental/mcp_server/outbound_credentials/adapter.py	Adds _token_exchange_spec: defers incomplete configs to v1, forwards audience only when set, uses token_url as fallback endpoint. Clean mapping.
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py	Wires build_token_exchanger() at egress and scopes browser-OAuth challenge to authorization_code only. Correct fail-closed behavior.
litellm/proxy/_experimental/mcp_server/outbound_credentials/types.py	TokenExchangeConfig gains audience and token_endpoint_auth_method. Backward-compatible.
tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_token_exchanger.py	Comprehensive: OBO form, caching, rotation, single-flight, TTL expiry, token_type/issued_token_type checks. 344 lines of new tests.
tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_token_exchange_provider.py	Tests build_token_exchanger and HTTP edge (transport errors, success, non-object JSON). No real network calls.
tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_resolver.py	Adds token_exchange resolver tests. Existing tests reformatted only, not weakened.
tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_adapter.py	Adds full-config mapping, token_url fallback, audience-omission, and incomplete-config defer-to-v1 cases.

_{Reviews (9): Last reviewed commit: "feat(mcp): reject a non-access issued_to..." | Re-trigger Greptile}

tin-berri · 2026-06-27T20:46:32Z

@greptileai addressed both findings: the exchanged-token cache key now binds the exchange config (so a config rotation re-exchanges instead of serving a stale token), and the lazy provider wrapper is gone in favor of eager build-once construction. Please re-review.

tin-berri · 2026-06-27T21:16:10Z

@greptileai all three inline findings are now addressed as of 745c9df: (1) the exchanged-token cache key binds the full exchange config so a config rotation re-exchanges, (2) the lazy provider wrapper is removed in favor of eager build-once construction, and (3) the post adapter validates the JSON shape and maps a non-object body to a typed upstream_unavailable instead of a 500. Each has a regression test; both new files are at 100% coverage. Please re-review.

tin-berri · 2026-06-27T21:53:56Z

@greptileai

mateo-berri · 2026-07-01T19:23:04Z

@greptileai

mateo-berri · 2026-07-01T19:23:07Z

bugbot run

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Forwarded Authorization bypasses OBO exchange
- Updated the shared Authorization stripping decision to cover migrated oauth2_token_exchange servers and added a regression test for configured header forwarding.

_{You can send follow-ups to the cloud agent here.}

mateo-berri

Is the bugbot concern legit? Just reping me once you've gotten 5/5 greptile with "No files need special attention", no veria concerns, and no bugbot concerns on the last commit. Or if there are any false positives, a response to each of them as to why it's a false positive

CLAassistant · 2026-07-01T19:31:59Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ tin-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…mode Adds the pure Rfc8693TokenExchanger plus its composition root: the OBO exchange POSTs the RFC 8693 grant through an injected HTTP edge and returns the upstream-bound token as a typed Result, caching and single-flighting per (subject_token, server) so a repeated caller token skips the IdP round-trip. The audience is carried on TokenExchangeConfig and sent only when the operator set one, matching the spec default behavior. Errors are values: a missing endpoint or client credential is misconfigured, an IdP that returns no usable token is upstream_unavailable.

Routes RFC 8693 OBO servers through the v2 resolver: the resolver arm reads the caller's inbound token and swaps it via the injected TokenExchanger, to_server_spec maps a complete oauth2_token_exchange server (endpoint plus client credentials) to TokenExchangeConfig, and the egress wires the LazyTokenExchanger in. A token_exchange server with no caller token fails closed with a plain 401 rather than v1's fall-through to client_credentials, so the call site now scopes the per-server browser-OAuth challenge to authorization_code and lets other modes raise their own.

The exchanged-token cache was keyed only by (subject_token, server_id), so rotating a server's audience, scope, endpoint, client_id, or secret kept serving a token minted for the old config until TTL. The key now hashes the caller token together with the config that minted it, so a config change forces a fresh exchange. Everything is hashed, so no secret is held in the key.

…rapper The token exchanger reads no runtime global at build time (its httpx client is acquired per call), unlike the per-user store, so it does not need lazy first-use construction. Building it once at egress construction removes the first-use init path entirely and keeps the process-lifetime cache.

The post adapter annotated the parsed body as a dict without checking it, so a valid-but-non-object JSON response (list/string/number) was returned as-is and crashed the field parsing with an AttributeError. It now validates the shape at the boundary and returns None for a non-object body, so a malformed IdP response surfaces as a typed upstream_unavailable rather than a server error.

…the v2 OBO exchange

tin-berri · 2026-07-03T19:11:05Z

bugbot run

tin-berri · 2026-07-03T19:11:10Z

@greptileai rereview

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Cache key omits auth method
- Included token_endpoint_auth_method in the exchange cache key and added a regression test for auth-method-only rotation.

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 346088a. Configure here.}

mateo-berri

LGTM; thanks!

greptile-apps Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread litellm/proxy/_experimental/mcp_server/outbound_credentials/token_exchanger.py Outdated

This comment was marked as outdated.

Sign in to view

greptile-apps Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread litellm/proxy/_experimental/mcp_server/outbound_credentials/token_exchange_provider.py Outdated

greptile-apps Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread litellm/proxy/_experimental/mcp_server/outbound_credentials/token_exchange_provider.py Outdated

greptile-apps Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread litellm/proxy/_experimental/mcp_server/outbound_credentials/token_exchanger.py

Base automatically changed from litellm_mcp_v2_authz_code_xrepl_2 to litellm_internal_staging June 27, 2026 23:27

tin-berri force-pushed the litellm_mcp_v2_token_exchange branch from 745c9df to c5dc4af Compare June 27, 2026 23:41

This was referenced Jun 29, 2026

feat(mcp): make token_exchange (OBO) production-ready - discovery threading + audit hardening + RFC 9728 challenge #31622

Merged

feat(mcp): discover the OBO token endpoint via RFC 9728 -> RFC 8414 (no IdP guessing) #31762

Merged

cursor Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py

mateo-berri requested changes Jul 1, 2026

View reviewed changes

tin-berri mentioned this pull request Jul 2, 2026

feat(mcp): add entra_obo profile to the token_exchange (OBO) arm #31983

Merged

4 tasks

tin-berri force-pushed the litellm_mcp_v2_token_exchange branch from e9e752c to 967b2f5 Compare July 3, 2026 17:37

tin-berri mentioned this pull request Jul 3, 2026

feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard #31772

Merged

4 tasks

tin-berri added 7 commits July 3, 2026 11:22

fix(mcp): fail closed on a non-Bearer token_type in the OBO exchange

80c3d95

feat(mcp): honor token_endpoint_auth_method (client_secret_basic) in …

5bcf4aa

…the v2 OBO exchange

tin-berri force-pushed the litellm_mcp_v2_token_exchange branch from 967b2f5 to 5bcf4aa Compare July 3, 2026 18:44

feat(mcp): reject a non-access issued_token_type in the OBO exchange

346088a

cursor Bot reviewed Jul 3, 2026

View reviewed changes

Comment thread litellm/proxy/_experimental/mcp_server/outbound_credentials/token_exchanger.py

fix: include token endpoint auth method in exchange cache key

eaf9fd7

tin-berri requested a review from mateo-berri July 3, 2026 22:27

mateo-berri approved these changes Jul 3, 2026

View reviewed changes

tin-berri merged commit f19bf2c into litellm_internal_staging Jul 3, 2026
124 checks passed

tin-berri deleted the litellm_mcp_v2_token_exchange branch July 3, 2026 23:05

mateo-berri mentioned this pull request Jul 4, 2026

feat(mcp): backport the OBO token endpoint discovery stack (#31526, #31622, #31762) to 1.91.0rc1 #32099

Closed

4 tasks

Uh oh!

Uh oh!

Conversation

tin-berri commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

What this is

Changes

Tests

Screenshots / Proof of Fix

Type

Changes

Deliberately not implemented

Uh oh!

tin-berri commented Jun 27, 2026

Uh oh!

codecov Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

tin-berri commented Jun 27, 2026

Uh oh!

Uh oh!

tin-berri commented Jun 27, 2026

Uh oh!

Uh oh!

tin-berri commented Jun 27, 2026

Uh oh!

mateo-berri commented Jul 1, 2026

Uh oh!

mateo-berri commented Jul 1, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

CLAassistant commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tin-berri commented Jul 3, 2026

Uh oh!

tin-berri commented Jul 3, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tin-berri commented Jun 27, 2026 •

edited

Loading

codecov Bot commented Jun 27, 2026 •

edited

Loading

greptile-apps Bot commented Jun 27, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

CLAassistant commented Jul 1, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading