Skip to content

fix(mcp): reconnect message-less closed transports - #66547

Closed
diffen77 wants to merge 4 commits into
NousResearch:mainfrom
diffen77:fix/mcp-transport-reconnect
Closed

fix(mcp): reconnect message-less closed transports#66547
diffen77 wants to merge 4 commits into
NousResearch:mainfrom
diffen77:fix/mcp-transport-reconnect

Conversation

@diffen77

Copy link
Copy Markdown

Summary

  • recognize message-less AnyIO ClosedResourceError, BrokenResourceError, and EndOfStream failures (including nested exception groups) as stale MCP transports
  • preserve interruption precedence, rebuild the existing server-owned transport, wait for a distinct ready session, and retry the interrupted operation once
  • reset both circuit-breaker count and open timestamp after successful recovery without rebuilding registered handlers or tool schemas
  • exercise the real MCPServerTask.run() lifecycle for both stdio and Streamable HTTP configurations

This is a follow-up to #13383: the existing text-based recovery handles explicit "expired session" responses, but AnyIO transport-close exceptions are commonly message-less and therefore bypassed that classifier.

Verification

  • scripts/run_tests.sh tests/tools/test_mcp_tool_session_expired.py tests/tools/test_mcp_circuit_breaker.py tests/tools/test_mcp_reconnect_signal.py tests/tools/test_mcp_reconnect_retry_reset.py -j 4 — 37 passed
  • scripts/run_tests.sh -j 8 <all tests/tools/test_mcp*.py + test_refresh_agent_mcp_tools.py + tests/tui_gateway/*.py + tests/test_tui_gateway*.py + test_tui_mcp_late_refresh.py> — 77 files, 1426 passed
  • ruff check . — passed
  • git diff --check — passed
  • operational stdio smoke: a real FastMCP child intentionally exited during the first call; the already-registered handler transparently returned from a fresh session on the single retry, with the same registry entry, handler identity, and schema, and breaker count reset to zero

Recovery contract reviewed

  • bounded reconnect lifecycle/backoff remains unchanged
  • per-server RPC lock and reconnect event preserve serialized/single-flight behavior
  • exactly one handler retry is attempted
  • cancellation/interruption is never reclassified as transport expiry
  • tool registration/schema identity remains stable across reconnect

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/mcp MCP client and OAuth P2 Medium — degraded but workaround exists labels Jul 17, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #65111: both address closed MCP transports, while this PR additionally handles nested AnyIO exception groups and reconnect lifecycle/breaker reset behavior. The overlapping recovery paths need maintainer consolidation.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for covering the message-less AnyIO path. Current main still rejects empty exception text in tools/mcp_tool.py:3553-3562, while the handler reaches session recovery through tools/mcp_tool.py:4299-4304, so the premise is valid.

Problems

  • tools/mcp_tool.py:3587 recursively classifies nested exceptions without the cycle/depth guard used by the new interruption helper. The new cycle test only exercises _exception_tree_contains_interruption (tests/tools/test_mcp_tool_session_expired.py:142-154), so a cyclic or deeply nested .exceptions graph can still recurse during classification.
  • This overlaps with #65111, as the maintainer comment notes. Its proposed classifier also covers BrokenPipeError and EOFError; consolidate the transport-type coverage rather than retaining parallel recovery classifiers.

Suggested changes

  • Use one iterative, identity-tracked exception-tree traversal for interruption precedence and transport-leaf detection; add classifier-level deep/cyclic tests.
  • Fold in #65111's built-in pipe/EOF coverage while preserving the existing text fallback and one-retry contract.

Automated hermes-sweeper review.

Comment thread tools/mcp_tool.py Outdated
# AnyIO task groups can wrap the transport exception in an
# ExceptionGroup whose own string omits the message-less leaf details.
nested = getattr(exc, "exceptions", ())
if nested and any(_is_session_expired_error(child) for child in nested):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This recursive walk does not share the cycle/depth protection added in _exception_tree_contains_interruption. A cyclic or deeply nested .exceptions graph can recurse here before classification; traverse the exception graph iteratively with an identity-based visited set, and add a classifier-level regression test.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 18, 2026
teknium1 added a commit that referenced this pull request Jul 21, 2026
For the #66547 and #66981 salvage cherry-picks in this branch.
teknium1 added a commit that referenced this pull request Jul 21, 2026
Builds on diffen77's #66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.

Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
teknium1 added a commit that referenced this pull request Jul 21, 2026
For the #66547 and #66981 salvage cherry-picks in this branch.
teknium1 added a commit that referenced this pull request Jul 21, 2026
Builds on diffen77's #66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.

Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
teknium1 added a commit that referenced this pull request Jul 21, 2026
For the #66547 and #66981 salvage cherry-picks in this branch.
teknium1 added a commit that referenced this pull request Jul 21, 2026
Builds on diffen77's #66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.

Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
@teknium1

Copy link
Copy Markdown
Contributor

Merged via #68660. Your message-less closed-transport classification was cherry-picked (4 commits, authorship preserved); we made the cause/context traversal cycle-guarded on top.

@teknium1 teknium1 closed this Jul 21, 2026
@diffen77
diffen77 deleted the fix/mcp-transport-reconnect branch July 22, 2026 17:40
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Builds on diffen77's NousResearch#66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.

Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
Builds on diffen77's NousResearch#66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.

Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/mcp MCP client and OAuth type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants