Skip to content

fix(mcp): reconnect dead stdio before breaker probe - #21006

Closed
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/mcp-stdio-breaker-reconnect-16788
Closed

fix(mcp): reconnect dead stdio before breaker probe#21006
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/mcp-stdio-breaker-reconnect-16788

Conversation

@LeonSGP43

@LeonSGP43 LeonSGP43 commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • track the child PIDs owned by each stdio MCP session
  • detect when a circuit-breaker half-open probe would reuse a stdio session whose child process is already gone
  • request a transport reconnect and return a clean stale-transport error instead of writing into the dead pipe
  • reset the MCP server error breaker when a stdio session successfully initializes

Why

Fixes #16788. Main already has the generic half-open cooldown behavior, but the issue's distinct failure mode remains: a dead stdio subprocess can leave a stale session object behind, so every half-open probe retries the same closed pipe and re-opens the breaker forever. This patch keeps the existing breaker behavior, but refuses to probe a known-dead stdio transport.

Scope

This only touches the stdio MCP transport liveness path and the existing MCP circuit-breaker handler. HTTP/streamable HTTP behavior is unchanged.

Tests

  • scripts/run_tests.sh tests/tools/test_mcp_circuit_breaker.py tests/tools/test_mcp_tool_session_expired.py tests/tools/test_mcp_tool_401_handling.py tests/tools/test_mcp_tool.py -> 210 passed, 4 warnings
  • python -m py_compile tools/mcp_tool.py -> passed
  • git diff --check -> passed

Fixes #16788

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists tool/mcp MCP client and OAuth labels May 7, 2026
@alt-glitch alt-glitch mentioned this pull request May 15, 2026
Closed
@teknium1 teknium1 mentioned this pull request Jun 27, 2026
Merged
teknium1 added a commit that referenced this pull request Jun 27, 2026
@teknium1 @LeonSGP43 @srojk34
fix(mcp): never permanently wedge the circuit breaker on a dead trans…
88c0246 
…port (#53599)

A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (#16788).

Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.

- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
  instead of returning when the reconnect budget is exhausted, so the task
  stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
  long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
  parked/dead task and respawning a dead stdio subprocess) and returns a
  clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
  (stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling

Builds on the closed-PR cluster for this issue: keeps #49255's deregister-on-
exhaustion insight and #21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.

Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
@teknium1

teknium1 commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Superseded by #53599 (merged: 88c0246), which fixes #16788 at the root.

Your "signal a reconnect instead of probing a dead transport" insight is the heart of the fix and you're credited via Co-authored-by on the merge commit. The merged version generalizes it: rather than os.kill(pid, 0) PID polling in the handler (racy under PID reuse, POSIX-only), the run loop now owns transport liveness — it parks as a dormant listener when the reconnect budget is exhausted (instead of exiting and orphaning the server), and the half-open probe signals _reconnect_event whenever the session is absent. That revives a dead stdio subprocess and works transport-agnostically (stdio/HTTP/SSE) with zero PID bookkeeping. Thanks for the contribution.

@teknium1 teknium1 closed this Jun 27, 2026
@teknium1 teknium1 mentioned this pull request Jun 27, 2026
Closed
pai-scaffolde pushed a commit to pai-scaffolde/hermes-agent that referenced this pull request Jun 28, 2026
@teknium1 @LeonSGP43
@srojk34 @pai-scaffolde
fix(mcp): never permanently wedge the circuit breaker on a dead trans…
1d6cc6d 
…port (NousResearch#53599)

A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (NousResearch#16788).

Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.

- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
  instead of returning when the reconnect budget is exhausted, so the task
  stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
  long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
  parked/dead task and respawning a dead stdio subprocess) and returns a
  clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
  (stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling

Builds on the closed-PR cluster for this issue: keeps NousResearch#49255's deregister-on-
exhaustion insight and NousResearch#21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.

Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
@teknium1 @LeonSGP43 @srojk34
fix(mcp): never permanently wedge the circuit breaker on a dead trans…
4023bce 
…port (NousResearch#53599)

A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (NousResearch#16788).

Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.

- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
  instead of returning when the reconnect budget is exhausted, so the task
  stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
  long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
  parked/dead task and respawning a dead stdio subprocess) and returns a
  clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
  (stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling

Builds on the closed-PR cluster for this issue: keeps NousResearch#49255's deregister-on-
exhaustion insight and NousResearch#21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.

Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
@teknium1 @LeonSGP43 @srojk34
fix(mcp): never permanently wedge the circuit breaker on a dead trans…
96967cd 
…port (NousResearch#53599)

A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (NousResearch#16788).

Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.

- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
  instead of returning when the reconnect budget is exhausted, so the task
  stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
  long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
  parked/dead task and respawning a dead stdio subprocess) and returns a
  clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
  (stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling

Builds on the closed-PR cluster for this issue: keeps NousResearch#49255's deregister-on-
exhaustion insight and NousResearch#21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.

Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
@teknium1 @LeonSGP43 @srojk34
fix(mcp): never permanently wedge the circuit breaker on a dead trans…
a6f7fee 
…port (NousResearch#53599)

A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (NousResearch#16788).

Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.

- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
  instead of returning when the reconnect budget is exhausted, so the task
  stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
  long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
  parked/dead task and respawning a dead stdio subprocess) and returns a
  clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
  (stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling

Builds on the closed-PR cluster for this issue: keeps NousResearch#49255's deregister-on-
exhaustion insight and NousResearch#21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.

Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
@teknium1 @LeonSGP43 @srojk34
fix(mcp): never permanently wedge the circuit breaker on a dead trans…
a417f56 
…port (NousResearch#53599)

A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (NousResearch#16788).

Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.

- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
  instead of returning when the reconnect budget is exhausted, so the task
  stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
  long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
  parked/dead task and respawning a dead stdio subprocess) and returns a
  clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
  (stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling

Builds on the closed-PR cluster for this issue: keeps NousResearch#49255's deregister-on-
exhaustion insight and NousResearch#21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.

Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
This was referenced Aug 1, 2026
Open
Open
Open
Open
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
@teknium1 @LeonSGP43 @srojk34
fix(mcp): never permanently wedge the circuit breaker on a dead trans…
1cee79d 
…port (NousResearch#53599)

A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (NousResearch#16788).

Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.

- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
  instead of returning when the reconnect budget is exhausted, so the task
  stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
  long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
  parked/dead task and respawning a dead stdio subprocess) and returns a
  clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
  (stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling

Builds on the closed-PR cluster for this issue: keeps NousResearch#49255's deregister-on-
exhaustion insight and NousResearch#21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.

Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

P2 Medium — degraded but workaround exists tool/mcp MCP client and OAuth type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

MCP circuit breaker open state can permanently wedge gateway when stdio subprocess dies

3 participants

@LeonSGP43 @teknium1 @alt-glitch