fix(mcp): don't trip circuit breaker on tool-level semantic errors - #32728
fix(mcp): don't trip circuit breaker on tool-level semantic errors#32728Rolync217 wants to merge 1 commit into
Conversation
The per-server circuit breaker in tools/mcp_tool.py bumps its counter on
ANY response containing an "error" key, regardless of whether the error
is a transport-level failure (server crashed, timeout, parse error) or
a tool-level semantic error (e.g. "page_not_found", "invalid_params",
"permission_denied").
This conflates two distinct concerns. A successful MCP call that returns
a structured negative result is NOT a sign the server is broken — the
server is healthy and the tool ran correctly; the caller's request just
referenced something that doesn't exist or isn't allowed.
The bug bites any agent flow that does check-then-write: "get_page → if
missing, put_page → cross-link". On a fresh brain, the first 3 entity
probes return page_not_found (legitimate), the breaker trips, and every
subsequent tool call to that server (including the put_page that would
have created the missing page) short-circuits with "unreachable" for
the cooldown window. The agent falls back to plan-B paths or fails the
task, while the MCP server is sitting there perfectly healthy.
Discovered debugging a gbrain voice-note-ingest pipeline where every
voice memo got stashed in a Kanban TODO list instead of becoming a
proper brain page, because the routine "does this entity page exist?"
probe pattern was tripping the breaker on every fresh entity.
## Fix
Inside the json-parse path at tools/mcp_tool.py:2398, when the response
contains an "error" key, check the error string against a list of known
semantic-error patterns:
- page_not_found, not_found, no such, does not exist, doesn't exist
- invalid_params, invalid_input, invalid_argument
- validation_error, validation failed
- permission_denied, unauthorized, forbidden
- already_exists, conflict, duplicate
- rate_limit, rate_limited, too_many_requests
Substring match (case-insensitive) — matches whether the tool returned
a plain string or a JSON envelope as its error_text. If the error
matches any pattern, reset the breaker counter; the tool ran fine, just
reported a structured "no" — the server is healthy.
Unknown error patterns continue to bump the breaker (fail-closed).
Transport errors caught by the except block at line 2429 still bump
unconditionally, preserving the original "server broken" detection.
## Tests
tests/tools/test_mcp_circuit_breaker.py extends with two new cases:
- test_semantic_error_envelope_does_not_bump_breaker
Hits a stub server 5 times with isError=True / page_not_found
envelopes (above the threshold of 3). Asserts the counter stays
at zero throughout — none of the semantic errors counted.
- test_unknown_error_envelope_still_bumps_breaker
Hits a stub server 3 times with an error string that matches no
semantic pattern. Asserts the counter bumps to threshold,
preserving fail-closed behavior for unrecognized errors.
All 5 tests in the file pass (3 existing + 2 new).
|
Competing fix with #11128 for the same MCP circuit breaker issue (#11113). #11128 uses |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating a real MCP breaker failure mode. Current main still has the underlying issue: tools/mcp_tool.py:3943-3956 receives a completed MCP CallToolResult and serializes isError=True as an error payload, then tools/mcp_tool.py:4001-4002 increments the server-health breaker for every such payload.
Problems
- The proposed substring blocklist is not a sound health boundary. A completed
CallToolResult(isError=True)already proves the MCP transport and server responded. The proposed unknown-error test instead preserves a false trip for valid tool errors whose text is not in the list. - The current handler is now at
tools/mcp_tool.py:3993-4006; GitHub reports this PR as conflicting, so the patch needs to be salvaged into the current control flow.
Suggested changes
- Classify at the structural
CallToolResultboundary, not by error-message text: do not increment (and reset) the breaker after any completedisError=Trueresponse; preserve increments for the exception/no-session transport paths. - Add an arbitrary
isError=Trueregression case above threshold and retain a separate transport-exception case that increments the breaker.
Automated hermes-sweeper review.
| "validation_error", "validation failed", | ||
| "permission_denied", "unauthorized", "forbidden", | ||
| "already_exists", "conflict", "duplicate", | ||
| "rate_limit", "rate_limited", "too_many_requests", |
There was a problem hiding this comment.
This still makes breaker state depend on tool error text. The JSON path is reached only after _call_once() completes, and _call() creates this envelope from a returned CallToolResult.isError; that is already evidence the MCP server responded. Please reset/avoid incrementing for every completed isError result, then keep breaker increments in the exception/transport paths instead.
What does this PR do?
Fixes a circuit-breaker bug in
tools/mcp_tool.pythat mis-classifies tool-level semantic errors (e.g.page_not_found,invalid_params,permission_denied) as server-health failures, and trips the breaker as if the MCP server itself were unreachable.The breaker treats ANY response containing an
"error"key as a failure. But a successful MCP call that returns a structured negative result (the tool ran, the tool said "no such resource") is not a sign the server is broken. The server is healthy and responded correctly; the caller just asked for something that doesn't exist or isn't allowed.Real-world impact: any agent flow that does check-then-write (e.g.
get_page → if missing, put_page) trips the breaker after 3 cache misses and locks the entire server for the cooldown window. I discovered this debugging a gbrain voice-note-ingest pipeline — every voice memo my Hermes received got stashed in a Kanban TODO list instead of becoming a brain page, because the routine "doespeople/<name>exist yet?" probe pattern was tripping the breaker on every fresh entity. The gbrain MCP server was healthy and reachable the entire time.Related Issue
I didn't open one separately — the title + this body should be enough to reproduce. Happy to file an issue first if maintainers prefer.
Fixes #
Type of Change
Changes Made
tools/mcp_tool.py:2398-2402— inside the json-parse path, when the response contains an"error"key, check the error string against a substring list of known semantic-error patterns. If it matches, reset the breaker counter (the tool ran fine, the server is healthy). If it doesn't match, continue to bump (preserves fail-closed behavior for unrecognized errors). The transport-error path at:2429is unchanged.tests/tools/test_mcp_circuit_breaker.py— extends with two new cases:test_semantic_error_envelope_does_not_bump_breaker— hits a stub server 5x withisError=True+page_not_foundenvelopes (above the threshold of 3). Asserts counter stays at zero throughout.test_unknown_error_envelope_still_bumps_breaker— hits a stub server 3x with an error string matching no semantic pattern. Asserts counter bumps to threshold (regression guard for the "fail-closed on unknown errors" contract).How to Test
pytest tests/tools/test_mcp_circuit_breaker.py -v→ all 5 tests pass (3 existing + 2 new). Tests use the same stub-server pattern as the existing breaker tests in this file.Real-world repro (the path I discovered the bug on): connect any MCP server that has a
get_Xtool which returns a structured{"error": "not_found"}envelope when the resource doesn't exist. Configure an agent flow that doesget_X → if missing, put_X. Pre-fix: after 3 misses, every subsequent call to that server short-circuits with "MCP server X is unreachable after 3 consecutive failures." Post-fix: the breaker stays closed, theput_Xcall goes through, the resource gets created.Stable on my Mac mini for 24+ hours now. The full voice-note-ingest → gbrain pipeline that originally failed (Hermes ↔ gbrain stdio MCP, Telegram inbound, faster-whisper STT, local Ollama fallback) now creates brain pages correctly end-to-end.
Checklist
Code
fix(mcp): ...)pytest tests/tools/test_mcp_circuit_breaker.py -q— all 5 tests passDocumentation & Housekeeping
cli-config.yaml.example(no config keys added)CONTRIBUTING.md/AGENTS.md(no architecture or workflow changes)Screenshots / Logs
Before (pre-patch — circuit breaker tripping on legitimate get_page misses):
put_pagereturns in0.00sbecause the breaker is short-circuiting before the call reaches the (perfectly healthy) MCP subprocess.After (post-patch — same agent flow, voice memo arrives, brain page lands):