Skip to content

fix(mcp): silence Event loop is closed on reload parked teardown - #69564

Closed
buckster123 wants to merge 1 commit into
NousResearch:mainfrom
buckster123:fix/mcp-parked-reload-loop-closed
Closed

buckster123 wants to merge 1 commit into
NousResearch:mainfrom
buckster123:fix/mcp-parked-reload-loop-closed

Conversation

@buckster123

Copy link
Copy Markdown

Summary

  • /reload-mcp (and any full MCP teardown path) could print a noisy ignored exception when tearing down parked MCP servers:
Exception ignored in: <coroutine object MCPServerTask.run ...>
  File tools/mcp_tool.py, in _wait_for_reconnect_or_shutdown
    t.cancel()
RuntimeError: Event loop is closed
  • Happens when run() parks after a permanent/initial connect failure: start() raises, the server is not added to _servers, then reload closes the MCP loop while the parked wait helper's finally still calls Future.cancel().

Changes

  1. _safe_cancel_await — cancel + await wait-helper futures without raising if the loop is already closed. Used by:
    • _wait_for_lifecycle_event
    • _wait_for_reconnect_or_shutdown
    • _wait_for_lazy_reconnect
  2. _parked_failed_servers — if start() fails but leaves run() alive (parked for revival), track it so full MCP teardown still calls server.shutdown().
  3. _stop_mcp_loop — cancel leftover tasks on the loop before loop.stop(), so any remaining orphans exit while the loop is still alive.

Test plan

  • uv run --extra dev python -m pytest tests/tools/test_mcp_stability.py -q30 passed
  • Manual: configure a failing MCP stdio server (bad command), let it park, run /reload-mcp — no Event loop is closed traceback; healthy servers reconnect.

Context

Observed while hot-swapping Hermes MCP config (dropping failed servers + swapping Cerebro to a Rust binary). Reload still succeeded (Reconnected: blender, cerebro, sonus); the traceback was cosmetic but scary.

Parked MCPServerTask.run helpers cancelled child wait tasks in finally
after the shared MCP loop was already closed during reload, which
surfaced as:

  Exception ignored in: <coroutine object MCPServerTask.run ...>
  RuntimeError: Event loop is closed

Root causes:
1. start() can raise after run() parks for revival, leaving an orphaned
   task never recorded in _servers so reload never reaps it.
2. wait-helper finally blocks called Future.cancel() without guarding
   against a closed loop.

Fix:
- _safe_cancel_await for lifecycle / parked / lazy-reconnect wait helpers
- track parked-failed starts in _parked_failed_servers and reap them on
  full MCP teardown
- cancel leftover loop tasks before stop in _stop_mcp_loop
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/mcp MCP client and OAuth P2 Medium — degraded but workaround exists needs-decision Awaiting maintainer decision before any implementation labels Jul 22, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Related: overlaps #61466 (safe cancellation) and #60104 (drain before loop close), but this patch also tracks failed-start parked servers for explicit teardown. These are competing lifecycle mechanisms; maintainer choice is needed.

@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the careful MCP lifecycle investigation. This is now implemented on current main through a stronger combined lifecycle solution.

  • c00a1d58d505123a64fef7e2f0500f6e483e8193 retains recoverably parked failed-start tasks in _servers for shutdown/revival at tools/mcp_tool.py:5861-5873.
  • ab0d3fac3d4cde14333d4107d8b5ae1475399a04 drains pending MCP-loop tasks on their owning loop before closing it at tools/mcp_tool.py:6781-6822.
  • tests/tools/test_mcp_stability.py:526-612 covers an unregistered parked task and verifies its cleanup completes before loop.stop().

This is an automated hermes-sweeper review.

@teknium1 teknium1 closed this Jul 30, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main tool/mcp MCP client and OAuth type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants