feat(mcp): per-server circuit breaker threshold via breaker_threshold config

[igitur]

Adds per-server breaker_threshold config key under mcp_servers.<name> to override the global _CIRCUIT_BREAKER_THRESHOLD (default: 3).

Problem

The MCP circuit breaker uses a hardcoded threshold of 3 consecutive failures before marking a server unreachable. For unstable-but-recoverable MCP servers (e.g. email/IMAP servers that flap on Gmail rate limiting), 3 failures is too aggressive — the server gets locked out for 60s even though it would recover on the next call.

Changes

• New _circuit_breaker_thresholds: Dict[str, int] module-level dict for per-server overrides
• _bump_server_error() and the tool handler guard check both now resolve threshold per-server, falling back to _CIRCUIT_BREAKER_THRESHOLD (3)
• register_mcp_servers() populates the dict from srv_cfg.get("breaker_threshold") when it's a positive int

Usage

mcp_servers:
  himalaya:
    command: node
    args: [dist/index.js]
    breaker_threshold: 10   # takes 10 failures to circuit-break, not 3

Omit breaker_threshold to keep the default of 3.

Closes #44172

[teknium1]

Thanks for adding a narrowly scoped MCP configuration path. The premise is still present on current main: both breaker transitions use the global threshold at tools/mcp_tool.py:3609 and tools/mcp_tool.py:4692.

Problems

• tools/mcp_tool.py:5181 only writes valid overrides into the new module-level map. It never removes an existing entry. The normal reload path calls shutdown_mcp_servers() and then fresh discovery (cli.py:11367-11371), so removing or invalidating breaker_threshold would leave the prior threshold active instead of restoring the global default.
• This user-facing config key is not documented in the MCP server-key reference (website/docs/reference/mcp-config-reference.md:44-68).

Suggested changes

• Reconcile or clear the override map on reload, and add a test covering override removal/invalid values after a prior valid override.
• Document the positive-integer constraint and default fallback in the MCP config reference.
• The PR addresses only one portion of #44172; please make the issue reference partial rather than Closes.

Automated hermes-sweeper review.

[teknium1]

This map needs a removal/reconciliation path. After a valid threshold is configured, then removed or made invalid in config.yaml, MCP reload will leave the old value here and no longer fall back to _CIRCUIT_BREAKER_THRESHOLD. Please clear absent/invalid entries and add a reload regression test.

[GottZ]

This was generated by AI during triage.

Summary

Twenty PRs address or reference four related MCP reliability and hardening issues. Their diffs cover completed-response versus transport-failure classification, idle keepalive, dead-transport recovery, breaker messaging and locking, proactive reconnects, and per-server threshold configuration; #68455 addresses only the configurable-threshold portion of #44172.

Related pull requests

• fix: MCP circuit breaker should only count infrastructure errors #11128 partial — (+43/-2) — n/a: Classifies exception strings and stops returned error envelopes from incrementing the breaker, but uses pass instead of resetting prior strikes; its keep_open review requires _reset_server_error() and current-state regression tests.
• fix: MCP circuit breaker recovery and HTTP keepalive (#16788, #17003) #17016 [closed] fixes — (+81/-20) — n/a: Added half-open probing and HTTP keepalive while still counting returned error envelopes; it remains relevant as the closed predecessor whose keepalive was salvaged by fix(mcp): add periodic keepalive to _wait_for_lifecycle_event (salvage #17016) #20209 and whose recovery scope was superseded by fix(mcp): never permanently wedge the circuit breaker on a dead transport #53599.
• fix(mcp): add periodic keepalive to _wait_for_lifecycle_event (salvage #17016) #20209 [merged] best fix — (+33/-4) — n/a: Merged reference implementation and recorded best fix for MCP HTTP connections go stale after extended idle periods #17003, isolating fix: MCP circuit breaker recovery and HTTP keepalive (#16788, #17003) #17016's 180-second list_tools() keepalive and reconnect-on-failure path without changing tool-error classification.
• fix(mcp): reconnect dead stdio before breaker probe #21006 [closed] fixes — (+110/-1) — n/a: Detected dead stdio child PIDs before probing and requested reconnect; it remains relevant because closed fix(mcp): reconnect dead stdio before breaker probe #21006 was superseded by transport-agnostic merged fix fix(mcp): never permanently wedge the circuit breaker on a dead transport #53599.
• fix(mcp): don't trip circuit breaker on tool-level semantic errors #32728 partial — (+144/-2) — n/a: Resets the breaker only for listed semantic-error substrings, leaving unlisted completed errors able to trip it; the keep_open review calls for the structural completed-response boundary.
• fix(mcp): don't bump circuit breaker for tool-level errors #40951 [closed] partial — (+6/-1) — n/a: Removes the increment for returned tool errors but uses pass, so prior transport strikes remain; this closed direct-path variant is incomplete relative to the consecutive-failure cause.
• fix(mcp): never permanently wedge the circuit breaker on a dead transport #53599 [merged] best fix — (+343/-6) — n/a: Merged reference implementation and recorded best fix for MCP circuit breaker open state can permanently wedge gateway when stdio subprocess dies #16788, parking exhausted server tasks, reviving them through reconnect signaling, and resetting breaker state after session initialization.
• fix: soften MCP circuit-breaker error messages to avoid model over-adherence #56833 related — (+34/-9) — n/a: Softens transient breaker messages and tests against directive-like wording without changing which failures feed the breaker; the shown diff addresses the visible keep_open review's wording and coverage requests.
• fix(mcp): add lock protection to circuit breaker error counters #60666 related — (+144/-8) — n/a: Locks breaker count/timestamp mutations and routes successful session-expiry retries through _reset_server_error(); its concurrent and retry-branch tests address the visible keep_open review.
• fix(mcp): semantic tool errors must not trip circuit breaker #61555 best fix — (+54/-9) — n/a: Recorded best fix for MCP circuit breaker treats tool-level errors (DNS failure, HTTP 4xx/5xx) as server failures, triggers false circuit break #11113 that resets after completed direct tool RPCs, but its keep_open review shows delivered errors in both recovery helpers can still be discarded and counted.
• fix(mcp): proactively reconnect on transport-exception breaker trip #61556 partial — (+47/-0) — n/a: Signals reconnect when transport exceptions trip the breaker despite a surviving session object; its keep_open review requires synchronized signal delivery and task-lifecycle proof.
• fix(mcp): don't count tool-level errors toward the circuit breaker (#11113) #67340 [closed] partial — (+134/-9) — n/a: Implements the same direct completed-RPC reset as fix(mcp): semantic tool errors must not trip circuit breaker #61555 with additional strike-reset and transport-exception tests; it remains relevant as a closed duplicate of fix(mcp): semantic tool errors must not trip circuit breaker #61555.
• feat(mcp): per-server circuit breaker threshold via breaker_threshold config #68455 best fix — (+232/-2) — n/a: Recorded best fix for the configurable-threshold portion of [Bug]: MCP client hardening: no result-size cap, no per-turn call budget, reconnect/circuit-breaker constants not configurable #44172, but its keep_open review identifies stale overrides after reload, missing configuration documentation, and an overbroad Closes #44172 claim.
• fix(mcp): don't trip the circuit breaker on application-level tool errors #68669 partial — (+94/-9) — n/a: Resets after completed direct calls but leaves auth and session-recovery error envelopes able to count; despite its MAINTAINER-BOT keep_open verdict, fix(tools): stop counting app-level MCP tool errors as server unreachability #74045 contains those missing sibling fixes and handler-level tests.
• fix(mcp): stop counting app-level tool errors as server unreachability #74042 [closed] partial — (+103/-25) — n/a: Applies the completed-RPC rule to direct calls and both recovery helpers; it remains relevant as the closed predecessor superseded by convention-compliant replacement fix(tools): stop counting app-level MCP tool errors as server unreachability #74045.
• fix(tools): stop counting app-level MCP tool errors as server unreachability #74045 best fix — (+240/-28) — n/a: Recorded best fix for MCP circuit breaker treats tool-level errors (DNS failure, HTTP 4xx/5xx) as server failures, triggers false circuit break #11113 that resets after completed direct, OAuth-retry, and session-retry RPCs and tests all three through handler-level paths; its MAINTAINER-BOT verdict is keep_open.
• fix(mcp): do not trip server breaker on tool-level errors #74606 partial — (+44/-9) — n/a: Fixes and tests only the direct CallToolResult.isError path; its MAINTAINER-BOT keep_open verdict identifies the missing shared recovery changes already present in fix(tools): stop counting app-level MCP tool errors as server unreachability #74045.
• fix(mcp): don't trip circuit breaker on tool errors #74718 [closed] fixes — (+145/-28) — n/a: Implements the same direct-plus-recovery rule as fix(tools): stop counting app-level MCP tool errors as server unreachability #74045; it remains relevant because it was closed solely for recreation under the intended contributor account as fix(mcp): don't trip circuit breaker on tool errors #74795.
• fix(mcp): don't trip circuit breaker on tool errors #74795 fixes — (+145/-28) — n/a: Matches fix(tools): stop counting app-level MCP tool errors as server unreachability #74045's production sites, but its keep_open review finds it redundant and notes that its recovery tests bypass the handler-level CallToolResult.isError conversion covered by fix(tools): stop counting app-level MCP tool errors as server unreachability #74045.
• fix(mcp): keep functional errors out of transport breaker #75511 fixes — (+500/-32) — n/a: Its latest diff applies the completed-response rule to direct, OAuth, and session-recovery paths with handler-level coverage and a real event-loop lock; despite the earlier keep_open review now being technically addressed, its production behavior duplicates smaller fix(tools): stop counting app-level MCP tool errors as server unreachability #74045.

Duplicates

#67340 is a closed duplicate of #61555; #11128 and #32728 are classifier-based alternatives, while #40951, #61555, #68669, and #74606 overlap on the direct completed-RPC reset. #74042 was superseded by #74045, #74718 was replaced by #74795, and #74795 and #75511 duplicate #74045's direct-plus-auth/session-recovery behavior.

Suggested consolidation

Author action: rebase #68455 onto main, or split out the part that can merge; reconcile removal or invalidation of breaker_threshold during reload, add the reload regression and MCP config-reference documentation, and change Closes #44172 to a partial reference because the diff implements only threshold configuration. Keep recorded best fixes #61555 and #74045 open with their documented salvage paths; close #68669, #74606, #74795, and #75511 as duplicates of #74045—despite their visible keep_open reviews where applicable, the diffs show #74045 already covers the same direct behavior plus both recovery helpers with stronger handler-level coverage.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I11113(["issue #11113 (open)"])
    I44172(["issue #44172 (open)"])
    P68455["PR #68455 (open)"]
    P68455 -.->|partial| I11113
    P68455 -->|best fix| I44172
    class I11113 open
    class I44172 open
    class P68455 open
    class P68455 best
    class P68455 target
    click I11113 "https://github.com/NousResearch/hermes-agent/issues/11113"
    click I44172 "https://github.com/NousResearch/hermes-agent/issues/44172"
    click P68455 "https://github.com/NousResearch/hermes-agent/pull/68455"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 20 pull requests and 4 issues in this complex. Each diff was read against this issue; Assessment working set: 156 kB of PR diffs, 61 kB of issue/PR text, 48 kB of discussion (62 comments), 33 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.