test(e2e): cover MCP OAuth SSO and cold restart persistence

[devin-ai-integration]

TLDR

Problem this solves:

• OAuth coverage omitted aggregate SSO and cold gateway restart
• Whole-flow retries could hide immediate post-consent failures
• The OAuth test did not execute in CI

How it solves it:

• One shared scenario with four focused live OAuth variants
• Assert semantic tools/list and tools/call without flow retries
• Verify encrypted ownership and reconnect after an owned restart
• Observe bearer equality through the existing live forwarding edge
• Run the protected live job on relevant pull requests, requiring every variant to pass

User Flow

Before: an engineer who breaks the MCP OAuth login, or the reuse of the stored upstream token, still gets a green PR, because no CI job walks the real journey

1. They open a PR that touches the gateway's MCP OAuth code
2. CI runs the unit suites and the changed e2e selection. No job connects an MCP SDK client to http://<gateway>/mcp, logs in through gateway SSO, approves Linear's real consent screen, and then calls tools/list and tools/call
3. No job restarts the gateway and reconnects with a fresh client
4. Every check is green and the break merges

After: the same PR gets a red MCP OAuth happy path / oauth check that names the failing phase

1. They open the same PR
2. The MCP OAuth happy path workflow starts and waits for an e2e-changed approval from a LiteLLM engineer
3. Once approved, it registers a Linear server with POST http://127.0.0.1:<port>/v1/mcp/server, then connects an MCP SDK client to http://127.0.0.1:<port>/mcp using gateway SSO, and to http://127.0.0.1:<port>/<alias>/mcp using a JWT in x-litellm-api-key
4. Each client completes gateway login and real Linear consent, then immediately calls tools/list and <alias>-list_teams and expects real team data with isError=false
5. GET http://127.0.0.1:<port>/v1/mcp/server/{server_id}/user-credentials shows one oauth2 credential for that user, the gateway restarts, and a fresh client reconnects and gets the same team data with no second Linear consent
6. A break turns the check red, and the log names the failing phase and exception class, never the token or the exception message

The aggregate client uses normal SDK authentication without injecting a gateway header. The per-server initial consent deliberately uses x-litellm-api-key and is labeled accordingly; its reconnect uses only the gateway JWT. This does not claim an Authorization-only per-server OAuth dance succeeds

Relevant issues

The Linear login is provisioned as E2E_LINEAR_STORAGE_STATE_B64 in the protected e2e-changed environment. This PR has no provisioning dependency on LIT-4973. Renew an expired session with the existing capture helper and update that same secret

LIT-7737 retains the dependency/Python compatibility matrix. LIT-4506 owns broader isolation/failure regressions and LIT-3559 owns broader auth coverage. No production code, dependency constraints or lockfiles change

Affected release

Linear ticket

Resolves LIT-3467

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This PR only adds tests, so the proof is the new live job running against a real gateway, a real Keycloak login and real Linear consent, with no mocks, plus a probe showing the job turns red when the gateway stops reading back the stored upstream token. The branch has merged current main (974e4f109c), which already carries the MCP SDK 2 gateway upgrade and the shared E2E lockout fix from #42008, so the After run exercises that gateway

Before (974e4f1)

1. git ls-tree --name-only 974e4f109c tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py tests/e2e/mcp/oauth_gateway.py .github/workflows/test-mcp-oauth-e2e.yml prints nothing: the happy path test, its owned gateway helper and the workflow do not exist
2. git ls-tree --name-only 974e4f109c tests/e2e/mcp/ | grep -i oauth lists only oauth_chat_client.py and test_mcp_chat_completion_oauth_e2e.py, which cover the explicit-header route and no gateway restart
3. No workflow on that commit runs a live OAuth login, so a PR that breaks it shows no failing check

After (d5ac850)

1. The MCP OAuth happy path / oauth job ran on the pull request at this commit after an e2e-changed approval: https://github.com/BerriAI/litellm/actions/runs/35472707002
2. Its step "Run every required OAuth variant without retries" ran tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py with E2E_REQUIRED_TEST_COUNT: 4 and finished in about two minutes with every step successful
3. Its step "Report JUnit results and reject skipped or missing cases" printed tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py: 4 collected, 0 skipped with no failed cases
4. Each of the four cases ran real gateway login and Linear consent, immediate tools/list and <alias>-list_teams, the one-credential ownership check, the upstream bearer comparison at the forwarding edge, and a fresh-client reconnect after a verified cold gateway restart that contacted no linear.app host
5. Locally at this commit, uv run --no-sync pytest tests/code_coverage_tests/test_e2e_changed_gate.py -q reports 46 passed, and from tests/e2e, uv run --no-sync pytest test_provider_edge.py test_idp.py test_proxy_client.py coverage_registry/test_collector.py -q reports 369 passed
6. All 96 other PR checks at this commit are green; buildkite/e2e-tests, which is not required, was still running
7. The same job also passed on a throwaway merge of this tip onto current main (7651d6b550, 43 commits past the merge base, none touching this PR's files): https://github.com/BerriAI/litellm/actions/runs/35474368195, again 4 collected, 0 skipped

Probe: the job turns red on a broken gateway

To show the User Flow's last step, a throwaway branch off this tip made the gateway's stored-credential read return nothing (_read_credential in litellm/proxy/_experimental/mcp_server/outbound_credentials/per_user_oauth_store.py returning None, every other file unchanged), and the workflow was dispatched on it with the same e2e-changed approval: https://github.com/BerriAI/litellm/actions/runs/35473850324

1. The oauth job went red about a minute after the pytest step started, and the step "Report JUnit results and reject skipped or missing cases" ended with every selected file must execute a passing test, with no failures or errors
2. That step printed tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py: 4 collected, 0 skipped followed by four failed: cases, each with oauth_failure_phase: call
3. The two aggregate_sso cases report oauth_exception_type: AssertionError with frames ending at test_mcp_oauth_happy_path_e2e.py:182:test_consent_list_call_and_cold_restart and test_mcp_oauth_happy_path_e2e.py:56:assert_tool_result, so the first tools/call after consent came back as an error; the two explicit_header_jwt cases report oauth_exception_type: ExceptionGroup with frames ending at oauth_chat_client.py:304:_list_and_call
4. The public log carries no assertion message and no token, because pytest's output goes to the private file the job removes on exit
5. The throwaway branch was deleted after the run

90687ae597 closed Bugbot's reconnect-guard finding: the browser helper now records every request across the browser context and fails the reconnect when any linear.app host was contacted, so a fast upstream auto-approve can no longer pass as a consent-free reconnect. d5ac850feb adds -I to the gate test's reporter subprocess for the test-quality rule that landed on main

Earlier runs on this branch: b7bab56d4d passed (https://github.com/BerriAI/litellm/actions/runs/35465677503), df7e1fe69d passed (https://github.com/BerriAI/litellm/actions/runs/35462954636), and 4e8a4d4b61 failed all four cases within about a minute of the test starting (https://github.com/BerriAI/litellm/actions/runs/35463907681). That failure's private pytest log was removed on exit and the public log carried no assertion text, and the only source change from the passing run was the chat-test constant restore, which the happy-path test does not import. The b7bab56d4d commit publishes only failure phase, exception class and code locations for future failures

Actual selection against main now includes only the five access-control/JWT canary files; the unchanged OAuth chat test is not selected. No skip gate was weakened and no new exclusion remains for that existing test. A real loopback probe verified gateway header injection and off-origin stripping, including redirects

Management APIs prove credential existence and canonical ownership. A narrow read-only DB inspection after consent and after restart proves encryption and enables exact bearer comparison; the API does not expose plaintext tokens. Observation remains at the live forwarding boundary, not inside Linear

Type

✅ Test

Caveats (if any)

Medium

• The oauth job is not a required check and waits for an e2e-changed approval, like every e2e job on this repo today
  • A break can still merge when nobody approves the run; a follow-up tracks making it required, which needs a job that still reports on PRs the path filter skips
• One dispatched run at 4e8a4d4b61 failed all four variants before the failure-location reporter existed and never reproduced
  • The five runs since, at four later commits plus the merged tree, passed 4/4; the next such failure names its phase and exception class

Low

• Saved upstream sessions expire; the existing chat OAuth test already depends on the same E2E_LINEAR_STORAGE_STATE_B64, so this adds no new dependency
  • Renew with tests/e2e/mcp/linear_session_capture.py and update that secret
• The unchanged test_mcp_chat_completion_oauth_e2e.py did not run live at this tip, since the selector picks changed test files rather than helpers
  • Every SDK request in its flow targets the gateway origin, so the same-origin header rule is inert there, and the explicit_header_jwt variants exercise the same transport
• OAuthObservation.gateway_token is a JWT the gateway never receives on the aggregate_sso route, so its leak check is inert there; the stored-token equality check still covers the forwarded bearer
• stored_oauth reads the persisted credential through the proxy's decrypt_value_helper, so a refactor of that helper fails this test rather than the product
• Exact bearer measurement is at the live forwarding boundary, not inside the upstream
• /team/member_add leaves a stale team roster cache for up to 60 seconds; the test orders calls around it

QA runbook

Shared setup: use the existing database/Prisma and Keycloak setup, a real E2E license, a permanent test salt, and a captured test-account Linear session. Build and stage the dashboard at the tested commit. See tests/e2e/CONTRIBUTING.md under "MCP OAuth happy path" for the exact invocation. The dedicated workflow provisions this environment and requires four non-skipped passes

• tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py::TestMcpOauthHappyPath::test_consent_list_call_and_cold_restart[direct-aggregate_sso] - an SDK client on the aggregate /mcp route logs in with gateway SSO, consents once at Linear, and still gets team data after a cold gateway restart
  • Register an HTTP authorization-code Linear server, grant the canonical test user/team access, and confirm no stored upstream credentials
  • Connect a fresh SDK client to /mcp, complete gateway SSO and the server card's real upstream consent, then finish connecting
  • Immediately list and call the alias-prefixed read-only tool; require isError=false and valid nonempty teams
  • Verify one encrypted OAuth credential under the intended user/server, restart the owned gateway, and repeat with a fresh SDK client without upstream consent
  • Confirm no gateway header restoration or whole-flow retry makes the assertion pass
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py::TestMcpOauthHappyPath::test_consent_list_call_and_cold_restart[direct-explicit_header_jwt] - the same journey on the per-server route with the JWT sent in x-litellm-api-key, reconnecting after the restart with only the gateway JWT
  • Repeat the setup and immediate list/call through /<alias>/mcp, with the JWT configured in x-litellm-api-key from client creation
  • After the real restart, reconnect using only the gateway JWT and require the same semantic result without consent
  • Confirm this explicit configuration is not presented as an Authorization-only OAuth dance
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py::TestMcpOauthHappyPath::test_consent_list_call_and_cold_restart[observed-aggregate_sso] - the aggregate journey through the live forwarding edge, proving Linear receives the stored upstream token and never the gateway token
  • Repeat the aggregate scenario through the existing live edge forwarding to real Linear
  • Before and after restart, require actual tools/list and tools/call observations whose bearer equals the decrypted owned upstream token and contains no test gateway JWT
  • Confirm direct variants independently cover discovery; no replay or token rewriting is used
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/mcp/test_mcp_oauth_happy_path_e2e.py::TestMcpOauthHappyPath::test_consent_list_call_and_cold_restart[observed-explicit_header_jwt] - the explicit JWT journey through the same observer, with the same bearer proof before and after the restart
  • Repeat the explicit JWT scenario through the same live observer
  • Require matching bearer observations for both operations before and after restart
  • Confirm missing observations or wrong tokens fail; do not infer correctness solely from a nonempty response
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/9c61b6dde4e64e83a6f5fe8af173a9dd
Open in Devin Desktop: https://app.devin.ai/desktop/session/9c61b6dde4e64e83a6f5fe8af173a9dd?variant=devin

---

Note

Low Risk
Test and CI harness changes only; no production gateway or auth logic is modified in this diff.

Overview
Adds live MCP OAuth acceptance for gateway-managed Linear authorization-code flows, with a dedicated protected CI job instead of folding the scenario into the generic changed-e2e selector.

New scenario (test_mcp_oauth_happy_path_e2e.py) — one parametrized test runs four variants: aggregate gateway SSO vs explicit x-litellm-api-key JWT, each direct to Linear or observed through the live provider edge (bearer must match decrypted stored upstream token, no gateway leak). Flow covers consent (captured Playwright session), immediate tools/list + list_teams, credential ownership APIs, cold restart of an owned source-built gateway (oauth_gateway.py, no Redis), and reconnect without a second upstream consent. Opt-in via mcp_oauth_live / E2E_MCP_OAUTH_LIVE; tests using oauth_gateway skip the shared-proxy liveness gate.

Harness & CI — test-mcp-oauth-e2e.yml provisions DB/Keycloak, builds consent UI, materializes E2E_LINEAR_STORAGE_STATE, runs pytest with --reruns 0 and E2E_REQUIRED_TEST_COUNT=4. assert_tests_ran.py fails on wrong count or skips; on OAuth failures it prints only sanitized oauth_failure_phase, oauth_exception_type, and oauth_frame (conftest attaches these; gate test asserts no token leakage). The happy-path file is excluded from select_tests.py so it only runs in this workflow. Registry adds mcp.call_tool.oauth.persists_across_processes; oauth_chat_client, models, proxy_client.update_team, and provider-edge request observation support the new paths.

^{Reviewed by Cursor Bugbot for commit d5ac850. Bugbot is set up for automated code reviews on this repo. Configure here.}

• d5ac850 passes /live-pr-risk

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ joshua-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding findings at the unchanged latest tip.

Summary

Adds a dedicated live MCP OAuth end-to-end workflow and supporting test harness.

• Covers aggregate SSO and explicit per-server JWT flows against Linear.
• Verifies semantic tool listing and invocation before and after an owned gateway restart.
• Checks persisted encrypted credentials and forwarded bearer identity.
• Adds CI selection, required-case enforcement, sanitized diagnostics, contributor guidance, and coverage-registry metadata.
• The current tip is identical to the previous review SHA, so no new changes require findings.

_{Reviews (5) · Last reviewed commit: "test(e2e): isolate diagnostic reporter s..."}

[mateo-berri]

@greptileai

[joshua-berri]

bugbot run

[joshua-berri]

@veria-ai Please review the current tip 742a3ad. All four OAuth cases passed with zero skips in https://github.com/BerriAI/litellm/actions/runs/35467606351. Scope is LIT-3467 E2E tests, necessary helpers, CI and documentation. (aside)

[cursor]

Stale Bugbot comment from a previous run.

[joshua-berri]

bugbot run

[joshua-berri]

@veria-ai @greptileai Please review 90687ae. The reconnect guard now records Linear requests across the browser context and checks them even after a fast callback. A real Chromium probe reproduced the old false pass and now rejects it; permitted initial consent and gateway-only reconnect still pass. Full lint passed; fresh live CI is pending. (aside)

[cursor]

Stale Bugbot comment from a previous run.

[joshua-berri]

bugbot run

[joshua-berri]

@veria-ai @greptileai Please refresh review at d5ac850 after integrating main and its shared E2E lockout fix #42008. No new production diff; the only accompanying adjustment adds isolated Python execution to our diagnostic reporter test. All 147 affected tests and full lint passed locally. (aside)

[joshua-berri]

bugbot run

[joshua-berri]

@veria-ai Please review the latest PR tip, including the integration of #42008. Review only; no implementation changes requested. (aside)

[joshua-berri]

@greptileai Please review the latest PR tip after the main-branch integration. (aside)

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d5ac850. Configure here.}

[mateo-berri]

LGTM