feat: add MCP state-change callback and degraded cluster-aggregate state with per-instance drill-down

[Pratham-Mishra04]

Summary

Adds a stateChangeCallback hook to MCPManager that fires on every reactive (non-admin-driven) MCP client connection-state transition, and introduces a degraded aggregate state for distributed deployments where multiple instances report conflicting states for the same client.

Changes

• State-change callback: A new stateChangeCallback field on MCPManager is invoked whenever a client's state changes through ClientConnectionChecker.setState (periodic checker transitions) or failConnectAttempt (reactive connect-failure classification into NeedsReauth). Deliberately excluded from admin-driven transitions (CloseAndMarkNeedsReauth, DisableClient/EnableClient, UpdateClient) since those already have their own call sites. The callback is always fired outside the manager lock to avoid holding m.mu across arbitrary caller-supplied work.
• SetMCPStateChangeCallback on Bifrost: Exposes the callback registration through the top-level Bifrost type, no-op when MCP is not configured.
• MCPConnectionStateDegraded: A new read-time aggregate MCPConnectionState value ("degraded") that is never a node's own local state — it signals that instances in a distributed deployment currently disagree about a client's state. Single-instance deployments never produce it.
• MCPClusterStateAggregator interface: An optional interface the configured MCPManager may implement. When present, getMCPClientsPaginated calls AggregateMCPClientState and, if the result is Degraded, replaces the response state and attaches a node_states per-instance breakdown map.
• MCPClientResponse.NodeStates: New optional field (node_states) on the HTTP response carrying the per-instance state breakdown when state is "degraded".
• UI — StateBadge component: Replaces the inline <Badge> in the MCP clients table with a StateBadge that renders a plain badge for all states except "degraded", which additionally gets a hover Popover drill-down. summarizeNodeStates groups instance IDs by their reported state so the drill-down reads as counts ("2 instances: Healthy, 1 instance: Unstable") rather than a raw ID list.
• UI — degraded color token: Added degraded: "bg-blue-100 text-blue-800" to MCP_STATUS_COLORS — a distinct blue/purple rather than amber/red to signal "instances disagree" rather than "one instance is unhealthy."
• Tests: Added TestFailConnectAttempt_NeedsReauthTransition_FiresStateChangeCallback, TestFailConnectAttempt_NoStateChange_DoesNotFireCallback, and TestSetState_FiresStateChangeCallbackOnGenuineTransition covering genuine transitions firing the callback exactly once and no-op transitions not firing it.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

# Core/Transports
go test ./core/mcp/... ./core/...

# UI
cd ui
pnpm i
pnpm build

To exercise the degraded state manually, implement MCPClusterStateAggregator on a custom MCPManager that returns MCPConnectionStateDegraded with a populated nodeStates map for a given client, register it, and call GET /api/mcp/clients. The response for that client should include "state": "degraded" and a populated node_states object. In the UI, hovering the "Degraded" badge should show the per-instance breakdown grouped by state.

Screenshots/Recordings

The "Degraded" badge renders in blue/purple and, on hover, shows a popover listing instance counts per state (e.g. "2 instances: Healthy, 1 instance: Unstable").

Breaking changes

• Yes
• No

Related issues

Security considerations

The stateChangeCallback is caller-supplied and may perform arbitrary work including I/O. It is always invoked outside m.mu to prevent lock contention or deadlocks. No secrets or PII are passed through the callback — only client ID, name, and state enum values.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[Pratham-Mishra04]

• docs: add token_exchange auth type, per-client TLS/sync/timeout fields, tool list persistence, and reconnect per-call clarifications to MCP docs and OpenAPI spec #5977
• docs: rename MCP states to healthy/unstable, add degraded and needs_session_stickiness, and expand 409/OAuth flow descriptions #5976
• docs: add connections page with full state table, connection-mode matrix, and per-auth-type lifecycle; rename connected→healthy, disconnected→unstable #5975
• docs: add needs_session_stickiness field, session stickiness behavior, and per-call vs sticky connection docs #5974
• fix: gate verify-headers replay guard on pre-request pending_verification state, project needs_reauth over unstable, and block premature shared-OAuth completion #5973
• fix: defer SetClientTools until after DB persistence to prevent runtime/DB state drift #5972
• feat: add tools-change callback funnel with content-hash gating and targeted DB persist for all MCP discovery paths #5971
• fix: enable per-call tool discovery for shared oauth/headers/none MCP clients and rename UpdateClientConnection to UpdateClientCredentials #5970
• feat: add virtual key and user identity filters to OAuth2 grants sidebar #5969
• feat: add virtual_key_id and user_id multi-select filters to MCP sessions #5921
• feat: add MCP state-change callback and degraded cluster-aggregate state with per-instance drill-down #5920 👈 (View in Graphite)
• feat: add compare-and-swap guard on refresh_token to RefreshOauthTokenFieldsIfActive #5919
• feat: add MCP server filter to MCP sessions sidebar #5918
• feat: replace OAuth grants filter bar with collapsible sidebar #5917
• feat: replace MCP sessions inline filter bar with collapsible sidebar #5916
• feat: add needs_session_stickiness to MCP HTTP clients for per-call vs persistent connection control #5915
• refactor: replace ClientHealthMonitor+ClientToolSyncer with unified ClientConnectionChecker and rename states connected→healthy, disconnected→unstable #5914
• refactor: extract shared SectionHeader, replace accordions with bordered sections in MCP client form #5913
• refactor: extract TLS, OAuth, and token exchange fields into shared components and reorganize MCP client sheet into tabs #5912
• feat: add tiered retry configs for MCP connect, per-call connect, and tool-call paths #5911
• refactor: replace OAuth2 token-expired text matching with errors.Is and preserve last-known tool map on disconnect #5910
• feat: add -jwt flag for JWT bearer auth to mcp-test-client #5909
• refactor: extract shared MCP authorizer dialog chrome into authorizerUi.tsx #5908
• feat: add shouldRun guard to skills orphan cleanup worker for multi-instance deployments #5813
• feat: add SetShouldRefreshGate to gate OAuth token sweep ticks #5812
• feat: add stack-absorb Claude skill for manual Graphite stack change distribution #5786 : 1 other dependent PR (#5787 )
• docs+feat: expand MCP connection states into reference table and add tooltip to State column header #5785
• feat: add token-exchange demo MCP server for testing against real identity providers #5730
• docs: add token exchange (on-behalf-of) MCP auth documentation #5729
• docs: document make-before-break reconnection and shared-client retry behavior #5728
• feat: make-before-break reconnect for HTTP/SSE MCP clients with generation-guarded stale write protection #5727
• feat: retry shared-connection MCP auth failures after bounded reconnect wait with concurrent dedup #5726
• feat: add ui for token_exchange (On-Behalf-Of) auth type MCP clients #5725
• feat: add token_exchange MCP auth type with delegated identity-provider token exchange #5724
• docs: add needs_reauth state, reauthorize endpoint, per-client tool sync/timeout, allowed_extra_headers, and OAuth config rotation to MCP OpenAPI spec #5723
• docs: add OAuth reauthorization, admin discovery credentials, auth failure recovery, and credential rotation docs #5722
• docs: add env./vault. reference support for OAuth client_id/client_secret and update OpenAPI EnvVar schema #4860
• docs: add pending_verification state and initiate-verification / verify-headers MCP client endpoints to OpenAPI spec #4858
• docs: add config.json bootstrap support for auth-aware MCP clients (oauth, per_user_oauth, per_user_headers) #3876
• feat: use *SecretVar for OAuth2Config credentials to preserve env./vault. #4859
• feat: add LRU credential cache to MCP headers provider with scoped eviction #5721
• feat: add in-memory LRU cache for per-user MCP OAuth access tokens with targeted eviction #5720
• feat: add generation-guarded LRU cache with single-flight fills and secondary index #5719
• feat: add per-user OAuth/headers admin discovery credential repair flow with registry needs_reauth projection #5718
• feat: retain admin bootstrap credential for periodic per-user MCP tool discovery refresh #5717
• feat: add MCPAuthModeAdmin + reauthorize endpoint for shared OAuth MCP clients #5716
• feat: replace OAuth config pin-and-warn with in-place rotation and needs_reauth cascade #5715
• feat: add needs_reauth state and replace per-user connection badge with sessions link #5714
• feat: add auth-failure retry with forced credential refresh for MCP tool calls #5713
• feat: add NeedsReauth MCP connection state for dead OAuth2 credentials #5712
• refactor: make GetExpiringOauthTokens auth modes caller-configurable via AuthModes field #5711
• refactor: unify shared and per-user OAuth refresh under a single RefreshAccessToken(tokenID) path, dropping oauth_configs.token_id FK shortcut #5710
• refactor: migrate OAuth flow state/PKCE fields from oauth_configs to new mcp_oauth_flows table #5709
• refactor: merge oauth_tokens/oauth_user_tokens into unified mcp_oauth_tokens table #5708
• feat: add per_user_headers bootstrap verification flow via submitHandler abstraction #3875
• feat: add verify-headers endpoint and pending_verification state for per-user-headers MCP clients #3874
• fix: extend pending_verification bootstrap path to cover per_user_oauth clients #3873
• feat: add OAuth authorization flow for config.json-bootstrapped pending_verification MCP clients #3872
• feat: add pending_verification state and initiate-verification endpoint for config.json shared-OAuth MCP clients #3871
• docs: document flow mode access rules, temp token toggle, and per-server tool manager overrides #3870 : 1 other dependent PR (#3882 )
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added reactive notifications for MCP connection-state changes.
  • Added Needs Reauthorization and Degraded connection states.
  • MCP client responses now include per-instance node states when connections are degraded.
  • Updated MCP client status displays with badges and hover details summarizing distributed connection states.

• Bug Fixes

  • Prevented duplicate notifications when a connection state remains unchanged.

• Tests

  • Added coverage for genuine state transitions and callback behavior.

Walkthrough

The PR adds MCP connection-state callbacks for actual transitions. It also adds the distributed degraded state, per-node state aggregation in HTTP responses, and UI popovers showing node-state counts.

Changes

MCP state observability

Layer / File(s)	Summary
Callback registration core/mcp/interface.go, core/mcp/mcp.go, core/bifrost.go	Bifrost and MCP manager APIs now register or clear state-change callbacks.
Transition notifications and tests core/mcp/clientmanager.go, core/mcp/connectionchecker.go, core/mcp/*_test.go	Callbacks run once for actual state changes after mutex release. Tests cover NeedsReauth, Healthy, Unstable, and unchanged states.

Distributed MCP state reporting

Layer / File(s)	Summary
Aggregation and UI presentation core/schemas/mcp.go, transports/bifrost-http/handlers/mcp.go, ui/lib/types/mcp.ts, ui/lib/constants/config.ts, ui/app/workspace/mcp-registry/views/mcpClientsTable.tsx	HTTP responses can expose aggregated degraded state and per-node states. The UI displays node-state counts in a popover.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ClientConnectionChecker
  participant MCPManager
  participant StateChangeCallback
  ClientConnectionChecker->>MCPManager: Report a connection-state transition
  MCPManager->>StateChangeCallback: Send client identifiers and old/new states

Loading

sequenceDiagram
  participant HTTPHandler
  participant MCPClusterStateAggregator
  participant MCPRegistryUI
  HTTPHandler->>MCPClusterStateAggregator: AggregateMCPClientState(clientID, localState)
  MCPClusterStateAggregator-->>HTTPHandler: Return aggregate state and node states
  HTTPHandler-->>MCPRegistryUI: Return MCPClientResponse
  MCPRegistryUI->>MCPRegistryUI: Render degraded state popover

Loading

Possibly related PRs

• maximhq/bifrost#5914: Adds related ClientConnectionChecker state-transition handling.
• maximhq/bifrost#5712: Adds related NeedsReauth connection-state handling.
• maximhq/bifrost#5727: Modifies related MCP connection-failure handling.

Suggested reviewers: akshaydeo, danpiths

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the callback, degraded aggregate state, and per-instance drill-down introduced by the pull request.
Description check	✅ Passed	The description follows the template, explains the design, testing, affected areas, security, and breaking-change status; empty related issues is non-critical.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 08-06-feat_add_mcp_client_state-change_callback_hook

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/mcp/clientmanager.go`:
- Around line 2185-2201: Preserve callback delivery order by routing committed
state-transition events through one bounded, lifecycle-safe ordered dispatcher
instead of invoking callbacks directly after unlock. Update the
failure-transition path around clientmanager.go:2185-2201 to enqueue its event
in commit order, and update the checker-transition path around
connectionchecker.go:377-386 to use the same dispatcher; both files require
changes. Keep shared state race-safe and add a concurrent test covering Healthy
→ Unstable → NeedsReauth that verifies observer order.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: be253bc7-2f8c-4a37-a7f2-7b143dcc0696

📥 Commits

Reviewing files that changed from the base of the PR and between ad8cffb and 569aa6e.

📒 Files selected for processing (12)

• core/bifrost.go
• core/mcp/clientmanager.go
• core/mcp/clientmanager_test.go
• core/mcp/connectionchecker.go
• core/mcp/connectionchecker_test.go
• core/mcp/interface.go
• core/mcp/mcp.go
• core/schemas/mcp.go
• transports/bifrost-http/handlers/mcp.go
• ui/app/workspace/mcp-registry/views/mcpClientsTable.tsx
• ui/lib/constants/config.ts
• ui/lib/types/mcp.ts

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[Pratham-Mishra04]

Merge activity

• Aug 8, 10:33 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Aug 8, 11:28 AM UTC: Graphite rebased this pull request as part of a merge.
• Aug 8, 11:29 AM UTC: @Pratham-Mishra04 merged this pull request with Graphite.

The base branch was changed.