Skip to content

fix(mcp): catch RuntimeError when canceling tasks during event loop close - #64114

Open
NorethSea wants to merge 2 commits into
NousResearch:mainfrom
NorethSea:fix/mcp-event-loop-closed-shutdown
Open

fix(mcp): catch RuntimeError when canceling tasks during event loop close#64114
NorethSea wants to merge 2 commits into
NousResearch:mainfrom
NorethSea:fix/mcp-event-loop-closed-shutdown

Conversation

@NorethSea

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes a benign but noisy RuntimeError: Event loop is closed traceback that appears on stderr when using /quit to exit a conversation with active MCP servers.

Root cause: _stop_mcp_loop() closes the MCP event loop, but MCPServerTask coroutines parked in _wait_for_lifecycle_event / _wait_for_reconnect_or_shutdown may reach t.cancel() inside their finally blocks after the loop is already dead. Task.cancel() internally calls loop.call_soon(), which raises RuntimeError("Event loop is closed").

The existing _mcp_loop_exception_handler suppresses this error only when routed through the loop's exception handler — not when the coroutine throws it directly, leaving an Exception ignored in: message from Python's GC.

Fix: Catch RuntimeError around t.cancel() in both finally blocks. The loop-closed case is a harmless race — no connection or state is lost.

Related Issue

Fixes #60197

Type of Change

  • 🐛 Bug fix

Changes Made

  • tools/mcp_tool.py — wrap t.cancel() with try/except RuntimeError in _wait_for_lifecycle_event and _wait_for_reconnect_or_shutdown

How to Test

  1. Start Hermes with at least one MCP server configured
  2. Enter a conversation and issue /quit
  3. Observe stderr — before the fix: Exception ignored in: <coroutine object MCPServerTask.run...> with RuntimeError: Event loop is closed; after the fix: clean exit

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs — [Bug] RuntimeError: Event loop is closed during /exit (MCPServerTask.shutdown) #60197 reports the same issue with no fix yet
  • My PR contains only changes related to this fix
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes — N/A (race condition, hard to reliably reproduce in test)
  • I've tested on my platform: macOS 26.5.2, Python 3.11

Documentation & Housekeeping

  • N/A — no documentation, config, architecture, or tool schema changes

…lose

When /quit stops the MCP event loop via _stop_mcp_loop(),
MCPServerTask coroutines parked in _wait_for_lifecycle_event or
_wait_for_reconnect_or_shutdown may reach t.cancel() after the loop
is closed. Task.cancel() internally calls loop.call_soon(), which
raises RuntimeError('Event loop is closed') on a dead loop.

The existing _mcp_loop_exception_handler suppresses this error when
it arrives via the loop's exception handler, but not when the
coroutine throws it directly and the exception surfaces to GC as
'Exception ignored in: <coroutine object MCPServerTask.run>'.

Catch RuntimeError around t.cancel() in both finally blocks so the
race is silently handled.

Fixes: NousResearch#60197
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/mcp MCP client and OAuth P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists labels Jul 14, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #60032 — same two sites (_wait_for_lifecycle_event and _wait_for_reconnect_or_shutdown) and the same mechanism (wrap t.cancel() in try/except RuntimeError to swallow the benign 'Event loop is closed' race on exit). #60032 is the earliest open canonical fix for #60197; competing alternate approach in #60104 (drain parked tasks before loop close).

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Comment

Changes

Fixes event loop shutdown race in tools/mcp_tool.py. When the event loop is already closed (RuntimeError), the code now catches it instead of propagating. Two similar finally-block fixes in _wait_for_lifecycle_event and _wait_for_reconnect_or_shutdown.

Assessment

  • Correctness: Fixes a benign shutdown race where RuntimeError is raised when awaiting a cancelled task whose event loop has already closed. Now correctly catches and ignores. The logic is equivalent — either we awaited successfully or caught the error.
  • Safety: No security implications. No behavior change for正常运行 paths.
  • Code quality: Clean, targeted fix. Comments explain the benign shutdown race. No debug artifacts.

Reviewed by Hermes Agent

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the cancellation race. Current main still has the two unguarded t.cancel() calls targeted here at tools/mcp_tool.py:2115-2121 and tools/mcp_tool.py:2159-2165, so the report is grounded.

Problems

  • The equivalent pending-task cleanup in _wait_for_lazy_reconnect() remains unguarded at tools/mcp_tool.py:3118-3124.
  • The lifecycle race remains at the loop owner: shutdown_mcp_servers() proceeds to _stop_mcp_loop() after its bounded wait (tools/mcp_tool.py:5661-5667), and that function closes the loop (tools/mcp_tool.py:5822-5826). Guarding two leaf cancellations suppresses only part of the class.
  • This diff adds no regression for the parked-server shutdown route.

Suggested changes

  • Prefer a bounded pending-task drain before loop close, or consistently cover all equivalent waiter cleanup paths.
  • Add an end-to-end regression through shutdown_mcp_servers() with a parked MCPServerTask.

Automated hermes-sweeper review.

Comment thread tools/mcp_tool.py
await t
except (asyncio.CancelledError, Exception):
t.cancel()
except RuntimeError:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard is needed here, but the same cancel()-then-await cleanup remains in _wait_for_lazy_reconnect() on current main (tools/mcp_tool.py:3118-3124). Please cover that equivalent waiter too if this approach is retained.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching the equivalent cleanup path. I pushed c6bead7, which adds the same Event loop is closed guard to _wait_for_lazy_reconnect() and a bounded pending-task drain before _stop_mcp_loop() closes the loop. It also adds regression coverage for the waiter cleanup and shutdown_mcp_servers() with a parked MCPServerTask. tests/tools/test_mcp_tool.py passes all 214 tests, and the related MCP tests pass all 24 tests.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 16, 2026
@andrexibiza

Copy link
Copy Markdown
Contributor

Rebase-on-behalf delivery — this PR's branch was rebased onto current main (9d6c5a920c7).

Rebased branch: andrexibiza:rebase-on-behalf/64114 (2 commits, original authorship preserved)

Conflict resolution (3 files):

  • tools/mcp_tool.py (3 regions) — deliberate divergence documented: this PR converts _drain_mcp_loop_tasks from async to sync (run_coroutine_threadsafe). Main's evolved async architecture (_drain_and_stop_mcp_loop + safe_schedule_threadsafe, which keeps drain+stop in one loop-owned sequence) supersedes it — the sync conversion is the design that creates the RuntimeError race this PR's commit 2 patches. Main's async version kept; the PR's .codegraph/ gitignore entry and shutdown tests preserved.
  • tests/tools/test_mcp_tool.py — this PR's shutdown tests added (HEAD side was empty). Verified: 0 main tests lost.
  • .gitignore — this PR's .codegraph/ entry added to main's file.

To adopt (author):

git fetch https://github.com/andrexibiza/hermes-agent.git rebase-on-behalf/64114
git checkout <your-branch-name>
git reset --hard FETCH_HEAD
git push --force-with-lease origin <your-branch-name>

py_compile verified; test-coverage superset verified. This unblocks the mcp_tool extraction window (hunk gate).

@andrexibiza

Copy link
Copy Markdown
Contributor

Interlock binding — this PR addresses the mcp_tool.py extraction window of EPIC #78647 (shard #78642). Rebase-on-behalf delivery: andrexibiza:rebase-on-behalf/64114.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/mcp MCP client and OAuth type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] RuntimeError: Event loop is closed during /exit (MCPServerTask.shutdown)

5 participants