Skip to content

fix(mcp): drain the MCP loop and retain parked startup tasks for clean shutdown - #74139

Merged
kshitijk4poor merged 7 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/60197-mcp-parked-shutdown
Jul 29, 2026
Merged

fix(mcp): drain the MCP loop and retain parked startup tasks for clean shutdown#74139
kshitijk4poor merged 7 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/60197-mcp-parked-shutdown

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Summary

Ends the RuntimeError: Event loop is closed shutdown noise from parked MCP tasks (#60197) by combining the two complementary community fixes: the MCP loop owner now drains pending tasks before closing the loop, and a server parked after a recoverable initial connect failure is retained in _servers so shutdown owns it and /mcp-refresh revival actually works.

Root cause (per @shady2k's analysis on the issue): a server that exhausts _MAX_INITIAL_CONNECT_RETRIES parks its run() task to self-probe, but start() raises before _servers[name] is written — so shutdown_mcp_servers() never reaps the task, _stop_mcp_loop() closes the loop with the coroutine still suspended, and GC finalization drives t.cancel() into the closed loop. Under systemd this turns a clean SIGTERM stop into exit code 1 / failed unit state (see issue thread).

What's salvaged

Validation

Check Result
E2E reproducer (deterministic failing stdio server, real imports, subprocess) on upstream/main dirty: RuntimeError: Event loop is closed ×2, Task was destroyed ×3, server absent from _servers
Same reproducer on this branch clean: 0 noise lines, broken-test retained in _servers, exit 0
MCP test suites (tests/tools/ -k mcp, 812 tests) all pass
New regressions (ownership/revival/terminal/probe + drain direct/bounded/blocked-loop + e2e shutdown) all pass
ruff / git diff --check clean
ty check tools/mcp_tool.py 10 diagnostics, identical to upstream/main baseline

Credit

Closes #60197

seppegadeyne and others added 7 commits July 29, 2026 16:59
_stop_mcp_loop() stopped and closed the background loop without reaping
the tasks still on it. A task left suspended is resumed later by the GC,
whose finalizer drives its cleanup against the now-closed loop:

    Exception ignored in: <coroutine object MCPServerTask.run ...>
      File "tools/mcp_tool.py", line 2947, in run
        parked = await self._wait_for_reconnect_or_shutdown(
      File "tools/mcp_tool.py", line 2161, in _wait_for_reconnect_or_shutdown
        t.cancel()
    RuntimeError: Event loop is closed

shutdown_mcp_servers() only reaps servers held in _servers, so a server
that parked after exhausting its initial-connect budget — never inserted
there, because start() raises _error before the caller registers it — has
no owner to signal it and stays suspended until the loop is gone.

Drain the loop the way asyncio.run() does: cancel the remaining tasks and
gather them while the loop is still open, so each runs its own finally.
Cancel alone is not enough — Task.cancel() only schedules the throw.

This resolves the reported traceback, but not the ownership bug that
strands the task in the first place; that needs a follow-up. Deliberately
not using "Fixes" so NousResearch#60197 stays open for it.

Addresses NousResearch#60197
Addresses NousResearch#66113

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…reap

Follow-up to the salvaged NousResearch#62026 ownership fix, folding in NousResearch#72054's
CancelledError rule by @adurham: start() already cancels/reaps its own
run task when the caller's connect timeout cancels start() itself, so
_connect_server() must propagate cancellation without awaiting a
redundant shutdown() inside a cancelled context. Non-cancellation
failures on the unclaimed (standalone probe) path still reap the parked
task, now with the reap failure logged instead of raising over the real
error.

Also maps mrz@mrzlab630.pw for the attribution check.

Co-authored-by: Adam Durham <amdnative@gmail.com>
@kshitijk4poor
kshitijk4poor enabled auto-merge (rebase) July 29, 2026 14:56
@kshitijk4poor
kshitijk4poor disabled auto-merge July 29, 2026 14:57
@alt-glitch alt-glitch added type/bug Something isn't working comp/tools Tool registry, model_tools, toolsets tool/mcp MCP client and OAuth P2 Medium — degraded but workaround exists labels Jul 29, 2026
@kshitijk4poor
kshitijk4poor merged commit 1f70ba6 into NousResearch:main Jul 29, 2026
41 checks passed
adurham added a commit to adurham/hermes-agent that referenced this pull request Jul 31, 2026
…s on all 7 PRs

Documents the real external engagement on the 7 upstream PRs filed
2026-07-26, and the 3 substantive follow-up fixes pushed in response:

- NousResearch#72054 closed as superseded, but merged anyway via NousResearch#74139 (contributor
  CrowLoki's reconciliation with NousResearch#62026, credited via Co-authored-by).
- NousResearch#72087, NousResearch#72151, NousResearch#72152, NousResearch#72153, NousResearch#72155, NousResearch#72164 all reviewed by the
  repo's automated sweeper -- keep_open/high on all 6.
- Fixed NousResearch#72087 (payload-proportional test assertions, catching a future
  allowlist-regression risk flagged by both the sweeper and an
  independent contributor who measured it precisely on their own fork).
- Fixed NousResearch#72152 (extracted ProfileRail's focus/visibilitychange wiring
  into a tested hook, matching the directory's own established
  use-profile-prewarm.ts pattern).
- Rebased NousResearch#72155 past a real merge conflict (an unrelated upstream
  test-pruning pass removed 3 tests my diff's context touched).

All fixes verified by simulating the exact regression each review was
warning about and confirming the new tests catch it, then restoring the
real fix.

Also noted a real environment issue found this session: the `upstream`
remote's SSH URL intermittently fails to connect from this network; a
one-off HTTPS fetch into a separate ref works around it without touching
the configured remote.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kshitijk4poor
kshitijk4poor deleted the salvage/60197-mcp-parked-shutdown branch August 5, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists tool/mcp MCP client and OAuth type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] RuntimeError: Event loop is closed during /exit (MCPServerTask.shutdown)

4 participants