feat: add auth-failure retry with forced credential refresh for MCP tool calls

[Pratham-Mishra04]

Summary

When a live MCP tool call returns a 401/403 or an "unauthorized"/"forbidden" error despite Bifrost's own credential bookkeeping considering the credential still valid, the upstream server and Bifrost's local state have diverged. Previously this surfaced as an opaque tool call failure with no recovery attempt. This PR adds automatic auth-failure recovery: for per-user connections, it forces a credential refresh, re-acquires a fresh connection, and retries the call once synchronously; for shared persistent connections (where a synchronous reconnect could take minutes), it fails the current call immediately and triggers a background reconnect so the next call succeeds.

Changes

• Added isAuthFailureErrorText to core/mcp/utils.go to detect raw upstream 401/403/unauthorized/forbidden error text from mcp-go's flattened error strings. Deliberately separate from isTransientError (opposite polarity — same text class is a permanent signal there, a positive retry trigger here) and from isOAuth2TokenExpiredErrorText (which matches Bifrost's own internal sentinel, not raw upstream rejections).
• Added attemptAuthFailureRecovery and triggerBackgroundReconnect to ToolsManager in toolmanager.go. Per-user path: force refresh → re-acquire connection → retry once synchronously. Shared path: fail fast, fire background goroutine that force-refreshes then calls ReconnectClient. Destructive, non-idempotent tools are opted out of auto-retry to avoid double side effects.
• Added ReconnectClient(id string) error to the ClientManager interface so ToolsManager can trigger a reconnect without a direct dependency on MCPManager.
• Added ForceRefresh(ctx, config) to the MCPCredentialStore interface and implemented it across all resolvers in core/mcp/credstore: no-op for none, static headers, and per-user headers; delegates to OAuth2Provider.ForceRefreshAccessToken for shared and per-user OAuth.
• Added ForceRefreshAccessToken(ctx, config) to the OAuth2Provider interface and implemented it in framework/oauth2/main.go. Branches on AuthType: shared OAuth resolves the token via OauthConfigID and calls RefreshAccessToken directly (bypassing the ExpiresAt gate that GetAccessToken applies); per-user OAuth derives (mode, identity) from context via the new ctx.MCPIdentity(mode) helper and does the same. Inactive (needs_reauth) shared tokens short-circuit with ErrOAuth2TokenExpired rather than attempting a doomed live refresh.
• Moved identityForMCPAuthMode from core/mcp/credstore/utils.go into BifrostContext as the MCPIdentity(mode) method, making it available to both the credstore resolvers and the OAuth2 provider without duplication. Deleted credstore/utils.go.
• Added auth_retry_test.go with end-to-end coverage of ExecuteTool against a real *client.Client wired to a fake transport.Interface: per-user success on second attempt, per-user retry-also-fails, shared fail-fast with background goroutine verification, destructive-non-idempotent opt-out, destructive-but-idempotent still retries, and non-auth failure never triggers recovery.
• Added framework/oauth2/force_refresh_test.go covering ForceRefreshAccessToken for shared OAuth (resolves and refreshes), missing OauthConfigID, inactive token short-circuit, per-user OAuth (resolves and refreshes), no identity in context, and unsupported auth type.
• Updated all test doubles (MockClientManager, MockAutoClientManager, mockToolClientManager, testClientManager, expiredOAuthCredStore, genericFailureCredStore, fakeOAuth2Provider) to satisfy the expanded interfaces.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

go test ./core/mcp/... ./core/schemas/... ./core/mcp/credstore/... ./framework/oauth2/...

The new auth_retry_test.go tests use a fake transport.Interface wired to a real *client.Client (via client.WithSession() to skip the MCP initialize handshake), so they exercise the actual CallTool error text path that isAuthFailureErrorText evaluates — not a mocked shortcut. The background-goroutine test (TestExecuteTool_AuthFailureRetry_Shared_FailsFastAndTriggersBackgroundReconnect) uses signal channels with a 2-second timeout to assert the goroutine actually runs without sleeping.

Breaking changes

• Yes
• No

ClientManager gains ReconnectClient(id string) error and MCPCredentialStore gains ForceRefresh(ctx *BifrostContext, config *MCPClientConfig) error. Any external implementations of either interface must add these methods. OAuth2Provider gains ForceRefreshAccessToken(ctx *BifrostContext, config *MCPClientConfig) error with the same requirement.

Security considerations

The auto-retry is gated behind isAuthFailureErrorText, which only matches explicit HTTP 401/403 status codes or the literal strings "unauthorized"/"forbidden" in the error text. Destructive, non-idempotent tools (as declared via MCPToolAnnotations) are explicitly excluded from auto-retry to prevent unintended double execution of side-effecting operations. The shared-connection path never retries synchronously — it only triggers a background reconnect — so there is no risk of amplifying requests against a rate-limited or actively-rejecting upstream.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[Pratham-Mishra04]

• feat: add stack-absorb Claude skill for manual Graphite stack change distribution #5786 : 2 dependent PRs (#5787 , #5812 )
• docs+feat: expand MCP connection states into reference table and add tooltip to State column header #5785
• feat: add token-exchange demo MCP server for testing against real identity providers #5730
• docs: add token exchange (on-behalf-of) MCP auth documentation #5729
• docs: document make-before-break reconnection and shared-client retry behavior #5728
• feat: make-before-break reconnect for HTTP/SSE MCP clients with generation-guarded stale write protection #5727
• feat: retry shared-connection MCP auth failures after bounded reconnect wait with concurrent dedup #5726
• feat: add ui for token_exchange (On-Behalf-Of) auth type MCP clients #5725
• feat: add token_exchange MCP auth type with delegated identity-provider token exchange #5724
• docs: add needs_reauth state, reauthorize endpoint, per-client tool sync/timeout, allowed_extra_headers, and OAuth config rotation to MCP OpenAPI spec #5723
• docs: add OAuth reauthorization, admin discovery credentials, auth failure recovery, and credential rotation docs #5722
• docs: add env./vault. reference support for OAuth client_id/client_secret and update OpenAPI EnvVar schema #4860
• docs: add pending_verification state and initiate-verification / verify-headers MCP client endpoints to OpenAPI spec #4858
• docs: add config.json bootstrap support for auth-aware MCP clients (oauth, per_user_oauth, per_user_headers) #3876
• feat: use *SecretVar for OAuth2Config credentials to preserve env./vault. #4859
• feat: add LRU credential cache to MCP headers provider with scoped eviction #5721
• feat: add in-memory LRU cache for per-user MCP OAuth access tokens with targeted eviction #5720
• feat: add generation-guarded LRU cache with single-flight fills and secondary index #5719
• feat: add per-user OAuth/headers admin discovery credential repair flow with registry needs_reauth projection #5718
• feat: retain admin bootstrap credential for periodic per-user MCP tool discovery refresh #5717
• feat: add MCPAuthModeAdmin + reauthorize endpoint for shared OAuth MCP clients #5716
• feat: replace OAuth config pin-and-warn with in-place rotation and needs_reauth cascade #5715
• feat: add needs_reauth state and replace per-user connection badge with sessions link #5714
• feat: add auth-failure retry with forced credential refresh for MCP tool calls #5713 👈 (View in Graphite)
• feat: add NeedsReauth MCP connection state for dead OAuth2 credentials #5712
• refactor: make GetExpiringOauthTokens auth modes caller-configurable via AuthModes field #5711
• refactor: unify shared and per-user OAuth refresh under a single RefreshAccessToken(tokenID) path, dropping oauth_configs.token_id FK shortcut #5710
• refactor: migrate OAuth flow state/PKCE fields from oauth_configs to new mcp_oauth_flows table #5709
• refactor: merge oauth_tokens/oauth_user_tokens into unified mcp_oauth_tokens table #5708
• feat: add per_user_headers bootstrap verification flow via submitHandler abstraction #3875
• feat: add verify-headers endpoint and pending_verification state for per-user-headers MCP clients #3874
• fix: extend pending_verification bootstrap path to cover per_user_oauth clients #3873
• feat: add OAuth authorization flow for config.json-bootstrapped pending_verification MCP clients #3872
• feat: add pending_verification state and initiate-verification endpoint for config.json shared-OAuth MCP clients #3871
• docs: document flow mode access rules, temp token toggle, and per-server tool manager overrides #3870 : 1 other dependent PR (#3882 )
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added automatic recovery for MCP tool calls affected by authentication failures.
  • OAuth credentials can now be force-refreshed for shared and per-user connections.
  • Shared connections refresh credentials and reconnect in the background.

• Bug Fixes

  • Safe, read-only, or idempotent calls retry once after credential refresh.
  • Destructive, non-idempotent, or unannotated actions are not automatically retried.
  • Shared connections fail fast while recovery proceeds asynchronously.
  • Unrelated failures continue through the existing error-handling path.

Walkthrough

MCP tool execution now detects authentication failures, force-refreshes credentials, retries safe per-call connections, and asynchronously reconnects shared connections. OAuth providers refresh shared or per-user tokens through new credential and client-manager contracts.

Changes

MCP authentication recovery

Layer / File(s)	Summary
Refresh and identity contracts core/schemas/context.go, core/schemas/mcp.go, core/schemas/oauth.go, core/mcp/toolmanager.go	Adds MCP identity lookup, forced credential refresh, OAuth token refresh, and shared-client reconnection contracts.
Credential and OAuth refresh core/mcp/credstore/*, framework/oauth2/main.go, framework/oauth2/force_refresh_test.go, core/mcp/reauth_state_test.go	Implements shared and per-user credential refresh, no-op refresh behavior for static credentials, and validation coverage.
Tool authentication recovery core/mcp/toolmanager.go, core/mcp/utils.go, core/mcp/auth_retry_test.go, core/mcp/*_test.go	Classifies authentication failures, suppresses unsafe retries, retries per-call connections once, and repairs shared connections in the background.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• maximhq/bifrost#5722: Documents the MCP authentication recovery and retry behavior.
• maximhq/bifrost#5726: Extends shared-connection recovery with bounded reconnect coordination.
• maximhq/bifrost#5710: Refactors OAuth token refresh behavior used by this refresh flow.

Suggested reviewers: akshaydeo, danpiths, roroghost17

Sequence Diagram(s)

sequenceDiagram
  participant MCPToolManager
  participant MCPTransport
  participant CredStore
  participant ClientManager
  MCPToolManager->>MCPTransport: Execute tools/call
  MCPTransport-->>MCPToolManager: Authentication failure
  MCPToolManager->>CredStore: ForceRefresh credentials
  CredStore-->>MCPToolManager: Refresh result
  MCPToolManager->>ClientManager: Acquire refreshed connection
  ClientManager-->>MCPToolManager: New connection
  MCPToolManager->>MCPTransport: Retry tools/call once
  MCPTransport-->>MCPToolManager: Recovered tool result

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 74.07% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the primary change: automatic authentication-failure retries with forced credential refresh for MCP tool calls.
Description check	✅ Passed	The description covers the purpose, implementation, testing, affected areas, breaking changes, security considerations, and checklist; only non-critical sections are incomplete.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 07-29-feat_reactively_retry_mcp_tool_calls_on_auth_failure

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/mcp/toolmanager.go`:
- Around line 844-866: Create the recovery timeout context before the
ForceRefresh call, using toolExecutionTimeout, and pass it to ForceRefresh,
AcquireClientConn, and the retry conn.CallTool invocation. Remove the later
retryCtx creation so all recovery operations share the same deadline, while
preserving cancellation and existing error handling.
- Around line 745-747: Update the retry response path in
attemptAuthFailureRecovery to pass retryResponse.IsError as the required third
argument to createToolResponseMessage, preserving the MCP tool-error state while
leaving the existing response text and return values unchanged.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3c447f93-2f1e-4c56-ab50-a550488d85a2

📥 Commits

Reviewing files that changed from the base of the PR and between fe261ef and 6277f49.

📒 Files selected for processing (20)

• core/mcp/agent_test.go
• core/mcp/auth_retry_test.go
• core/mcp/codemode/starlark/starlark_test.go
• core/mcp/credstore/credstore.go
• core/mcp/credstore/none.go
• core/mcp/credstore/per_user_headers.go
• core/mcp/credstore/per_user_oauth.go
• core/mcp/credstore/per_user_oauth_test.go
• core/mcp/credstore/shared_headers.go
• core/mcp/credstore/shared_oauth.go
• core/mcp/credstore/utils.go
• core/mcp/reauth_state_test.go
• core/mcp/toolmanager.go
• core/mcp/toolmanager_test.go
• core/mcp/utils.go
• core/schemas/context.go
• core/schemas/mcp.go
• core/schemas/oauth.go
• framework/oauth2/force_refresh_test.go
• framework/oauth2/main.go

💤 Files with no reviewable changes (1)

• core/mcp/credstore/utils.go

🚧 Files skipped from review as they are similar to previous changes (17)

• core/mcp/credstore/none.go
• core/mcp/utils.go
• core/mcp/credstore/shared_oauth.go
• core/mcp/credstore/per_user_oauth.go
• core/mcp/toolmanager_test.go
• core/mcp/credstore/per_user_oauth_test.go
• core/mcp/reauth_state_test.go
• framework/oauth2/main.go
• core/schemas/oauth.go
• core/mcp/credstore/shared_headers.go
• core/schemas/mcp.go
• core/mcp/credstore/per_user_headers.go
• core/mcp/codemode/starlark/starlark_test.go
• core/mcp/agent_test.go
• core/mcp/credstore/credstore.go
• core/schemas/context.go
• core/mcp/auth_retry_test.go

[Pratham-Mishra04]

Merge activity

• Aug 8, 8:47 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Aug 8, 9:15 AM UTC: Graphite rebased this pull request as part of a merge.
• Aug 8, 9:16 AM UTC: @Pratham-Mishra04 merged this pull request with Graphite.

The base branch was changed.