fix(mcp): add lock protection to circuit breaker error counters - #60666
fix(mcp): add lock protection to circuit breaker error counters#60666isheng-eqi wants to merge 1 commit into
Conversation
teknium1
left a comment
There was a problem hiding this comment.
Thanks for addressing a real MCP circuit-breaker race. Current main still has the unlocked read-modify-write at tools/mcp_tool.py:3001-3004, and the helper locking plus session-expiry helper calls target that defect directly.
Problems
tests/tools/test_mcp_circuit_breaker.py:572only invokes_bump_server_errorand_reset_server_error. Current main already clears both breaker maps in_reset_server_error(tools/mcp_tool.py:3014-3015), so the test passes without either production change in this PR. It does not cover the changed successful-retry branches attools/mcp_tool.py:3414and3417.
Suggested changes
- Exercise
_handle_session_expired_and_retryafter a tripped breaker and assert a successful retry clears both the count and timestamp. - Add deterministic concurrent-bump coverage so the locking regression is observable.
Automated hermes-sweeper review.
| @@ -569,3 +569,29 @@ async def _run_stdio(self, config): | |||
| run_task.cancel() | |||
|
|
|||
| asyncio.run(_scenario()) | |||
|
|
|||
There was a problem hiding this comment.
This test passes on current main because _reset_server_error already sets the count to zero and removes the timestamp (tools/mcp_tool.py:3014-3015). Please exercise the successful _handle_session_expired_and_retry retry path, which is where this PR changes reset behavior.
_bump_server_error and _reset_server_error performed read-modify-write on shared dicts without holding _lock, risking lost updates under concurrent access from MCP loop thread and caller thread. Also replaced two direct _server_error_counts assignments in _handle_session_expired_and_retry with _reset_server_error calls to ensure breaker timestamp is properly cleared.
5e6b227 to
9367f94
Compare
|
Thanks for the detailed review! Added two tests addressing both concerns:
Please take another look! |
What does this PR do?
Adds thread-safety to MCP circuit breaker error tracking.
_bump_server_errorand_reset_server_errorwere accessing shared dicts (_server_error_counts,_server_breaker_opened_at) without holding_lock, risking lost updates under concurrent access from the MCP loop thread and caller thread.Also fixes
_handle_session_expired_and_retrywhich was directly assigning_server_error_counts[name] = 0instead of calling_reset_server_error(name), leaving stale breaker timestamps.Type of Change
Changes Made
tools/mcp_tool.py: wrapped_bump_server_errorand_reset_server_errorbodies inwith _lock. Replaced two direct dict assignments with_reset_server_errorcalls.tests/tools/test_mcp_circuit_breaker.py: added test verifying reset clears both count and timestamp.How to Test