Skip to content

fix(mcp): un-invert the stdio children liveness check - #94339

Merged
kshitijk4poor merged 2 commits into
NousResearch:mainfrom
liuhao1024:liuhao/cron-bugfix-94335
Aug 27, 2026
Merged

kshitijk4poor merged 2 commits into
NousResearch:mainfrom
liuhao1024:liuhao/cron-bugfix-94335

Conversation

@liuhao1024

@liuhao1024 liuhao1024 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

_stdio_children_dead() returned True ("all children dead") on the first live pid — the intended False sat dead-coded directly beneath it. In any spawn path that captures child PIDs (observed in hermes -z oneshots), the #81995 pre-call fast-fail then raised TimeoutError: MCP stdio subprocess ... has exited; failing the call fast on every tools/call while the subprocess was demonstrably alive (the reporter's child-watcher poll caught it running at the moment of failure). Long-lived gateway/dashboard sessions were unaffected only because _stdio_child_pids stays empty there (the if not pids short-circuit) — which is also why production gateways never noticed.

This PR restores the documented decision table at the fast-fail boundary — no captured PIDs / HTTP transport ⇒ fail open; every tracked child dead ⇒ True; any live tracked child ⇒ False — and additionally fails open when psutil is unavailable or a PID probe raises, so unknown liveness can never authorize the destructive fast-fail. The destructive consumer _watch_stdio_children() stays pending while a child is alive and resolves once every tracked child has exited.

Related Issue

Fixes #94335

Type of Change

  • 🐛 Bug fix

Changes Made

  • tools/mcp_tool.py (MCPServerTask._stdio_children_dead) — restore the aggregate liveness polarity and fail open when psutil is unavailable or a PID probe raises; unknown liveness cannot authorize the pre-call fast-fail.
  • tests/tools/test_mcp_stdio_children_dead.py — eight regressions covering live, all-dead, mixed, absent-PID/HTTP, missing-psutil, probe-error, watcher-live, and watcher-all-dead behavior.

Provenance and Credit

How to Test

  • Run: .venv/bin/python -m pytest tests/tools/test_mcp_stdio_children_dead.py -q
  • Observed result: 8 passed. On unfixed main the same file reports 5 failed, 3 passed — failing exactly the live-child, mixed-liveness, missing-psutil, probe-error, and watcher-live pins.
  • Adjacent MCP fast-fail/cancellation sweep (test_mcp_bridge_single_failure.py, test_mcp_circuit_breaker.py, test_mcp_cancelled_error_propagation.py, test_mcp_failure_classification.py): 30 passed.

Checklist

  • Code follows the project's style guidelines
  • Self-review of the completed code performed
  • New and existing unit tests pass locally
  • Changes are backward-compatible (the all-dead verdict and unknown/fail-open paths are preserved; the live-pid verdict flips to correct and dependency/probe failures now fail open)

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/mcp MCP client and OAuth labels Aug 25, 2026
@tanao521

Copy link
Copy Markdown

Verified end-to-end on Windows 11 (hermes-agent 0.20.5, commit bc47fcd, editable install).

Reproduced the bug in a desktop-app CLI session with a stdio MCP server (chrome-devtools via npx): every tools/call failed fast with TimeoutError: MCP stdio subprocess for 'chrome-devtools' has exited while the child process was verifiably alive — matching the inverted-liveness failure mode exactly (repro details in #94335).

Verified this fix approach locally: applied the corrected liveness loop, and a fresh-session call to a real MCP tool returned correct results immediately, no other changes needed. The server itself was healthy throughout — a standalone NDJSON MCP probe completed initialize → tools/list → tools/call against the same subprocess while Hermes-side calls kept failing, confirming the inverted check was the only broken path.

The regression tests cover the matrix that matters here: live → not dead, all-dead → dead, mixed → not dead, plus the no-pids/HTTP fail-open cases. This unblocks every stdio MCP server on affected installs — worth merging promptly.

@liuhao1024

Copy link
Copy Markdown
Contributor Author

Thanks @tanao521 — this is a textbook end-to-end verification, and every data point maps directly onto what the fix changes:

  • Reproduced the inverted-liveness failure mode on Windows 11 (TimeoutError: ... has exited while the child process was verifiably alive): exactly the symptom shape from [Bug] _stdio_children_dead() inverted liveness check fail-fasts every stdio MCP call in oneshot (-z) sessions #94335 — the exit was inferred from the wrong side of the check, so a healthy subprocess was reported dead.
  • The control experiment is the strong part: a standalone NDJSON probe completing initialize → tools/list → tools/call against the same subprocess, while Hermes-side calls kept failing, isolates the broken path to the client-side liveness check alone — the server, the transport, and the spawn were all sound.
  • Fix-applied result: a fresh-session tools/call against a real stdio MCP server returns correct results immediately with no other changes, which is precisely the contract the corrected liveness loop restores.

For the maintainers: this independently confirms the fix on the platform where the issue was reported (Windows 11, editable install, npx-spawned stdio server) — beyond the mocked-subprocess tests, it exercises a real stdio MCP server through the desktop CLI session and shows the failure disappearing with only this change applied. As noted, the test matrix (live → not dead, all-dead → dead, mixed → not dead, plus the no-pids/HTTP fail-open cases) covers the decision table the old check got wrong.

@andrexibiza andrexibiza left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head c1b39a86bad20786f135b7620f6b87784ea839e9 against current main regression #94637 and the newer duplicate #94661.

The implementation restores the documented decision table at the actual pre-call fast-fail boundary: no captured PIDs / HTTP transport => fail open; every tracked child dead => true; any live tracked child => false. The four focused regressions pin exactly those cases, and the exact head is hosted-green on CI 32798064054, Docker 32798063539, and Nix 32798063527. Independent Windows 11 end-to-end evidence on this PR additionally reproduced a live stdio child being misclassified dead and showed real tools/call recovery with only this polarity correction.

The later #94661 carries the same primary fix plus an ImportError/probe-error fail-open guard. psutil==7.2.2 is an exact core dependency on current main, so that extra guard is not required to close the released regression; it is a defensiveness extension, not a reason to keep two permanent owners. Preserve any unique test/provenance evidence from #94661, but keep this older fix as the canonical liveness-polarity owner and do not merge both implementations.

No blocker found on this head.

@andrexibiza andrexibiza left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Canonicalization review — keep #94339 as the sole runtime owner

I re-verified exact head c1b39a86bad20786f135b7620f6b87784ea839e9 against current main (1bbb6e5bce56e721ab685af4cd87df21bbff4d35) and the overlapping carriers #94521, #94586, and #94661.

The polarity repair here is the correct canonical implementation: one live tracked child means the aggregate is not dead. Do not merge four implementations of the same boolean/state-machine defect.

Before merge, absorb only the genuinely unique closure evidence from the siblings:

1. Watcher-consumer proof from #94521

The current suite pins _stdio_children_dead() directly, but the destructive consumer is _watch_stdio_children(), which races the RPC and cancels it when the aggregate verdict becomes true. Add deterministic tests proving both edges:

def test_watcher_does_not_resolve_while_a_child_is_alive():
    async def _run():
        with patch("psutil.pid_exists", return_value=True):
            with pytest.raises(asyncio.TimeoutError):
                await asyncio.wait_for(
                    _task_with_pids([60634])._watch_stdio_children(),
                    timeout=0.05,
                )

    asyncio.run(_run())


def test_watcher_resolves_when_all_children_are_dead():
    async def _run():
        with patch("psutil.pid_exists", return_value=False):
            await asyncio.wait_for(
                _task_with_pids([111, 222])._watch_stdio_children(),
                timeout=0.1,
            )

    asyncio.run(_run())

This preserves #94521's unique state-machine coverage without /bin/sh, real-process timing, or platform-specific fixtures.

2. Explicit unknown/fail-open proof from #94661

The helper documents unknown → don't fail fast, but this head still performs a bare dependency import and an unguarded probe. Dependency/probe failure is not evidence that every child exited and therefore cannot authorize the destructive fast-fail:

        try:
            import psutil
        except ImportError:
            return False  # unknown → don't fail fast
        for pid in pids:
            try:
                alive = psutil.pid_exists(pid)
            except Exception:
                return False  # unknown → don't fail fast
            if alive:
                return False  # at least one child alive → not all dead
        return True  # every tracked child has exited

Add direct tests for missing psutil and a pid_exists() probe exception. The existing regression file will need asyncio, builtins, and pytest imports.

Carrier disposition

  • #94339: canonical runtime owner; correct polarity and strongest focused base suite.
  • #94521: duplicate implementation; unique watcher evidence distilled above.
  • #94586: duplicate implementation; no unique regression evidence.
  • #94661: duplicate implementation; unique dependency/probe fail-open evidence distilled above.

Please rebase this carrier onto current main, add only those four tests plus the local fail-open guard, and run the focused file plus adjacent MCP fast-fail/cancellation tests. Once that exact head is green, the other three carriers should remain closed as duplicates.

Contributor credit is preserved: watcher cases come from #94521; dependency/probe fail-open cases come from #94661. This remains one canonical fix for one aggregate-liveness defect.

Copy link
Copy Markdown
Contributor

Apply-ready absorption patch for exact head c1b39a86

This is the bounded delta from the canonicalization review: deterministic watcher-consumer tests distilled from #94521, plus dependency/probe fail-open handling and tests distilled from #94661. It does not import either sibling's alternate runtime carrier.

Local receipts against the reviewed source shape:

  • git apply --check: pass
  • py_compile for both changed files: pass
  • isolated focused regression file: 8 passed
  • patch SHA-256: 805b522bff4e1f1564fcc51209682f7e2bf9c9436b19ff0b4be4057b8063d4b4
diff --git a/tests/tools/test_mcp_stdio_children_dead.py b/tests/tools/test_mcp_stdio_children_dead.py
index b8f02c6..09398d5 100644
--- a/tests/tools/test_mcp_stdio_children_dead.py
+++ b/tests/tools/test_mcp_stdio_children_dead.py
@@ -1,13 +1,20 @@
-"""Regression tests for ``_stdio_children_dead`` (#94335).
+"""Regression tests for MCP stdio aggregate liveness (#94335 / #94637).
 
-The liveness check was inverted: the loop returned True ("all children dead")
-on the first LIVE pid, so the #81995 pre-call fast-fail raised
-``TimeoutError: MCP stdio subprocess ... has exited`` for every tools/call in
-oneshot (-z) sessions even though the subprocess was demonstrably alive.
+The #81995 fast-fail gate consumes ``_stdio_children_dead`` as a boolean
+state machine: True means every tracked child is gone; False means at least
+one child is alive or liveness is unknown. The live-child branch was inverted,
+so healthy stdio RPCs were cancelled while their subprocesses were still alive.
+
+Watcher-consumer cases are distilled from #94521. Dependency/probe fail-open
+cases are distilled from #94661 into the canonical #94339 carrier.
 """
 
+import asyncio
+import builtins
 from unittest.mock import patch
 
+import pytest
+
 from tools.mcp_tool import MCPServerTask
 
 
@@ -39,3 +46,49 @@ def test_no_captured_pids_stays_fail_open():
     """Unknown (no tracked pids / HTTP transport) must not fail fast."""
     assert _task_with_pids([])._stdio_children_dead() is False
     assert _task_with_pids([1], http=True)._stdio_children_dead() is False
+
+
+def test_psutil_unavailable_stays_fail_open():
+    """Missing probe support is unknown, never proof of child death."""
+    real_import = builtins.__import__
+
+    def _without_psutil(name, *args, **kwargs):
+        if name == "psutil":
+            raise ImportError("psutil unavailable")
+        return real_import(name, *args, **kwargs)
+
+    with patch("builtins.__import__", side_effect=_without_psutil):
+        assert _task_with_pids([1])._stdio_children_dead() is False
+
+
+def test_pid_probe_error_stays_fail_open():
+    """A failed probe cannot authorize the destructive fast-fail."""
+    with patch("psutil.pid_exists", side_effect=OSError("probe failed")):
+        assert _task_with_pids([1])._stdio_children_dead() is False
+
+
+def test_watcher_does_not_resolve_while_a_child_is_alive():
+    """The watcher must not cancel an RPC while any child is still live."""
+
+    async def _run():
+        with patch("psutil.pid_exists", return_value=True):
+            with pytest.raises(asyncio.TimeoutError):
+                await asyncio.wait_for(
+                    _task_with_pids([60634])._watch_stdio_children(),
+                    timeout=0.05,
+                )
+
+    asyncio.run(_run())
+
+
+def test_watcher_resolves_when_all_children_are_dead():
+    """The watcher completes only when the aggregate verdict is all-dead."""
+
+    async def _run():
+        with patch("psutil.pid_exists", return_value=False):
+            await asyncio.wait_for(
+                _task_with_pids([111, 222])._watch_stdio_children(),
+                timeout=0.1,
+            )
+
+    asyncio.run(_run())
diff --git a/tools/mcp_tool.py b/tools/mcp_tool.py
--- a/tools/mcp_tool.py
+++ b/tools/mcp_tool.py
@@ -2987,18 +2987,20 @@ def _stdio_children_dead(self) -> bool:
         pids = getattr(self, "_stdio_child_pids", None)
         if not pids or self._is_http():
             return False
-        for pid in pids:
-            # windows-footgun: ok — psutil.pid_exists handles Windows; the
-            # os.kill probe below only runs when psutil is unavailable.
+        try:
             import psutil
-
-            if not psutil.pid_exists(pid):
-                continue  # this one is dead
-            return False  # at least one child alive (signal permission
-            # irrelevant for liveness) — inverting this fail-fasted every
-            # stdio call in oneshot (-z) sessions while the subprocess was
-            # demonstrably alive (#94335)
-        return True
+        except ImportError:
+            return False  # unknown → don't fail fast
+        for pid in pids:
+            # pid_exists handles Windows without signal-permission noise; a
+            # probe failure is unknown, not proof that every child exited.
+            try:
+                alive = psutil.pid_exists(pid)
+            except Exception:
+                return False  # unknown → don't fail fast
+            if alive:
+                return False  # at least one child alive → not all dead
+        return True  # every tracked child has exited
 
     async def _watch_stdio_children(self) -> None:
         """Poll child liveness while a stdio RPC is in flight (#81995).

After applying, rerun the repository's focused file and adjacent MCP fast-fail/cancellation suite on the resulting exact head. This keeps #94339 as the one implementation owner while preserving the unique evidence and contributor credit from the duplicate carriers.

@Finn763

Finn763 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Agreed — closing in favor of #94339 per the canonicalization. For the record, here are the two unique closures this carrier contributed, offered for absorption into #94339 before merge:

1. Unknown → fail-open contract (_stdio_children_dead): missing psutil or a failed pid_exists() probe is not evidence that every tracked child exited. Concretely, on top of the polarity fix:

try:
    import psutil
except ImportError:
    return False  # unknown liveness → don't fail fast (best-effort contract)

for pid in self._stdio_child_pids:
    try:
        if psutil.pid_exists(pid):
            return False  # at least one child alive
    except Exception:
        return False  # probe error → unknown → don't fail fast
return True

2. Regression suite (7 cases, RED on pre-fix / GREEN post-fix, sabotage-verified): live pid → not all dead; all dead → fast-fail eligible; mixed live+dead; real spawned live child; psutil missing (ImportError → False); probe raises (→ False); stale pid list after child exit. Available at tests/tools/test_mcp_stdio_liveness.py on Finn763:fix/issue-94637-stdio-liveness if useful.

Happy to rebase/contribute these directly onto #94339 if that's easier.

@alt-glitch alt-glitch added P1 High — major feature broken, no workaround and removed P2 Medium — degraded but workaround exists labels Aug 25, 2026
@fred0m

fred0m commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Verified the same fix on macOS (hermes-agent v2026.8.19-826, installed at ~/.hermes/hermes-agent, stdio MCP).

Reproduced the exact failure mode after today's update: every tools/call to a stdio MCP failed instantly with TimeoutError: MCP stdio subprocess ... has exited, while the server itself stayed healthy (hermes mcp test connected and discovered tools fine). The inverted liveness check in _stdio_children_dead() matches this perfectly.

Applied the same polarity change locally (live child → return False) and confirmed recovery after restarting the Hermes parent process: the MCP call returned real data again. This also confirms all stdio MCP servers are affected, not just one.

LGTM from a macOS production-install perspective.

@chenyuan35

Copy link
Copy Markdown

Independent real-world reproduction + fix verification (Windows 11, packaged desktop install, x@latest stdio server):

  • Symptom matched exactly: after the Aug 25 update, every MCP tool call fast-failed within ~10ms with MCP stdio subprocess ... has exited while the npx child was demonstrably alive and serving.
  • Root cause confirmed on main (1bbb6e5bc): _stdio_children_dead() returns True on the first live pid — the inverted polarity this PR fixes at the same line.
  • Fix equivalence: I applied this PR's exact change locally; all calls work again end-to-end through a live browser-extension session (browser_tabs list round-trips successfully).
  • Regression coverage: I also ran an independent red/green check — the PR's polarity assertion fails on the buggy parent and passes with the fix; my extra cases (real spawned-and-exited child pids via subprocess on Windows, HTTP transport fail-open) behave per the documented decision table.

One note for whoever merges: the same inverted block still exists verbatim on main today (tools/mcp_tool.py, _stdio_children_dead), so please merge before the next release cuts — every Windows stdio user on current main is hitting this.

@alt-glitch alt-glitch added P2 Medium — degraded but workaround exists and removed P1 High — major feature broken, no workaround labels Aug 25, 2026
@sr99622

sr99622 commented Aug 25, 2026

Copy link
Copy Markdown

Confirming the issue and fix as others here

@Sedrak-Hovhannisyan

Copy link
Copy Markdown

Independent macOS fixture confirmation

Verified on macOS 26.6.2 against the affected v0.20.5 source and upstream base 02c7ae956e42891d5e337a921b45de0a6067146d.

  • Used a temporary HERMES_HOME plus a dependency-free local stdio JSON-RPC fixture; no production configuration, Zoho endpoint, mailbox data, account IDs, or credentials were read.
  • The real Hermes path completed initializenotifications/initializedtools/list, registered the fixture tool, and tracked one stdio child.
  • On the affected base, _stdio_children_dead() returned true before the first call and the handler produced the exact immediate error: MCP stdio subprocess for 'fixture' has exited; failing the call fast instead of waiting 15s. The fixture's tool body was never reached.
  • With this PR head (c1b39a86b), the pre-call verdict is false and the same first call returns the fixture's fixed result (fixture-ok).

Also ran the canonical targeted suites at this PR head: 4/4 new liveness tests and 119/119 related MCP tests pass; Ruff passes for the changed files.

This confirms the defect is the liveness-predicate polarity, not Streamable HTTP, mcp-remote, deferred-tool dispatch, or early MCP-session teardown.

@alt-glitch alt-glitch added P1 High — major feature broken, no workaround P2 Medium — degraded but workaround exists and removed P2 Medium — degraded but workaround exists P1 High — major feature broken, no workaround labels Aug 25, 2026
@SJBanister

SJBanister commented Aug 25, 2026

Copy link
Copy Markdown

Independent macOS/Moraine reproduction and control-path confirmation:

  • Hermes v0.20.5 connected to the Moraine stdio server and discovered all four tools successfully.
  • The first real search_sessions call then failed immediately with MCP stdio subprocess for 'moraine' has exited; failing the call fast instead of waiting 300s.
  • psutil/process inspection showed the tracked watchdog, moraine run mcp, and moraine-mcp processes were still alive.
  • ClickHouse /ping, moraine status, ingest, monitor, and the standalone MCP socket were healthy.
  • Sending the same initialize + tools/call search_sessions JSON-RPC sequence directly to moraine run mcp returned real search results.
  • Changing the live-PID branch to return False restored the intended contract; a focused local MCP suite passed (19 passed).

This independently confirms the PR fixes the exact discovery-green / first-call-fast-fail behavior on macOS arm64 with Moraine 0.6.4.

@liuhao1024

Copy link
Copy Markdown
Contributor Author

This is an exemplary close-out — thank you for doing the work before the PR rather than after it. The state matrix (live / dead-Windows / dead-Linux-zombie / dead-Linux-reaped) covering every way the walk could differ, measured on both platforms, is exactly the evidence a "provably no-op" claim needs, and declining to spend your tested-by-author provenance on dead code in a core file is the right call. The coupled-lifetime observation on Windows (child holds the shim's stdio pipes) is a particularly useful detail — it closes the escape hatch in the failure direction too.

One addition from my side that further supports your conclusion: since your earlier comment, upstream landed a spawn-ledger change (e1e72f109c, "register stdio MCP helper children in the spawn ledger") on the tracking side — helper children spawned by stdio MCP servers are now registered into _stdio_child_pids directly rather than being discovered later. That means the scenario your walk was designed for (a shim's real server not being in the tracked set) is increasingly addressed at the source: the tracked set itself is becoming authoritative, so the liveness check doesn't need to walk the process tree to find stragglers. Combined with your re-parenting findings on the query side, both halves of the gap are converging on "track it at spawn, don't chase it at check time."

Your re-open condition (a launcher topology where the real server survives the shim's exit AND remains reachable via the shim's PID) is well-framed — a pidfile/snapshot variant captured at spawn time would also slot naturally into the ledger if such an environment ever reports in. Either way, the try-import / fail-open baseline from this PR's head is the contract everything else builds on.

@47Hunter47

Copy link
Copy Markdown

Thanks — glad the evidence landed. The spawn-ledger piece (e1e72f109c) is a nice confirmation: "track it at spawn, don't chase it at check time" is exactly the right framing, and it means the tracking half of the gap is now closed at the source.

With both halves in place (polarity fix here + spawn ledger upstream), the re-open condition for the walk becomes very narrow: a launcher topology where the real server outlives the shim and the shim's PID is still the only handle to it at check time. If that ever shows up in the wild, I'll bring a pidfile/snapshot variant that slots into the ledger.

For now, the try-import / fail-open baseline is the contract, and the polarity fix is the full solution for the observed environment. No further action needed from my side.

@Stone441

Copy link
Copy Markdown

Confirmed and validated the fix end-to-end on macOS 26.6.1 with Hermes v0.20.5 (local HEAD dcfdc8de) and a real scheduled cron workflow using mcp-apple-mail==3.2.0 over uvx stdio.

Before the patch:

  • hermes mcp test apple-mail connected successfully and discovered the tools.
  • The watchdog → uvx → mcp-apple-mail process tree was demonstrably alive.
  • Normal agent-visible tools/call dispatch still failed instantly with:
    MCP stdio subprocess for 'apple-mail' has exited; failing the call fast instead of waiting 300s.
  • A minimal live-PID assertion reproduced the inversion deterministically: psutil.pid_exists(pid) == True while _stdio_children_dead() == True.
  • Two scheduled runs failed closed without advancing their application cursor.

I then applied the current #94339 patch locally, including the fail-open handling for unavailable/failed PID probes.

Verification:

  • Regression test was observed red before the code change: 5 failed / 3 passed.
  • Focused MCP liveness/watchdog/circuit-breaker suite: 17 passed.
  • Broader test_mcp_tool.py + focused suite: 115 passed (2 existing un-awaited-coroutine warnings from the _watch_children() predicate, separate from this polarity fix).
  • Fresh-process production-path validation via hermes cron run succeeded using the normal agent-visible MCP handlers, not hermes mcp test or a direct client:
    • mcp__apple_mail__list_mailboxes completed successfully.
    • Multiple mcp__apple_mail__search_emails calls completed successfully.
    • The real incremental workflow processed two new messages, passed SQLite integrity verification, and advanced its saved cursor.

One operational detail: a Desktop/serve process that had imported the old module continued to exhibit the old failure until process reload, while a fresh CLI cron process and restarted Gateway loaded the patch and succeeded. This is expected Python module lifetime behavior, but it explains why merely editing the file does not repair already-running sessions.

This independently confirms that #94339 fixes the real long-lived cron/gateway use case on macOS, not only a unit-level or one-shot reproduction.

@47Hunter47

Copy link
Copy Markdown

@Stone441 — thank you, this is the second independent end-to-end confirmation (after @liuhao1024's Windows run), and the real-cron macOS data point is exactly what this fix needed: the cursor-advancing workflow through the agent-visible handlers, not hermes mcp test.

One note on commit state: you tested dcfdc8de; the PR head is now 633d4fc9cf, which adds a small hardening on top of the same polarity fix — the psutil import and the per-PID probe are now wrapped in try/except so an unavailable or failing probe returns False (unknown → don't fail fast) instead of propagating. That is precisely the fail-open behavior you described validating, so your results carry over to the head as-is.

Your module-lifetime observation is expected and consistent with what we saw in the Windows validation — an already-running process that imported the old module keeps the old behavior until reload; fresh processes pick up the patch. Good callout for anyone upgrading with a long-lived gateway.

@C-Huai-Jie

Copy link
Copy Markdown

Independent confirmation, desktop-app scenario (complements the existing gateway confirmations).

Environment: Windows 11, Hermes desktop app (git install, f751a8c54; same buggy code on current origin/main), two unrelated stdio MCP servers.

Repro: every real tools/call to any stdio MCP server failed instantly with the exact error in this PR:

  • zcode-bridge
  • codebase-memory-mcp

Both server subprocesses stayed alive throughout (Get-Process/tasklist), and the same bridge answered every call when spawned from a normal terminal — identical to this PR's description. This is the desktop-app path (captured child PIDs), not the long-lived gateway path (if not pids short-circuit), so it hits exactly the polarity inversion this PR fixes.

Fix confirmed: applying the polarity change (alive child → return False) locally made both servers return live data on the very next tools/call — no other changes needed. Matches this PR's decision table exactly.

Also worth noting: one of the two servers runs through a Windows uv-venv shim that re-execs into a uv-managed CPython, matching the Windows-specific descendant-walk concern already raised here — so that extension PR is genuinely needed, not just theoretical. Happy to share the local diff if useful.

@47Hunter47

Copy link
Copy Markdown

Thanks @C-Huai-Jie — the desktop-app data point is a useful complement to the gateway/oneshot confirmations: two unrelated stdio servers, identical fast-fail while both children were verifiably alive, and both recovered by the polarity change alone.

On the descendant-walk point: your shim → uv CPython re-exec observation matches what was measured in this thread — the premise is real and was confirmed on the very gateway that hit the bug (comment 5434311617). The conclusion there was that the walk is a provably no-op on every reachable state:

  • Live tracked PID → the polarity check already returns False; the walk adds live children that are also live → False. Identical.
  • Dead tracked PID, WindowsProcess(dead_pid).children() raises NoSuchProcess (the cached child list dies with the process, even with a detached live grandchild) → the walk adds nothing → True. Identical.
  • Dead tracked PID, Linux → the kernel reparents the grandchild to init/subreaper the moment the shim dies, so children() returns [] in the zombie window or NoSuchProcess after reap. Identical.
  • Probe errors → both paths fail open. Identical.

Measured across the full live/dead/mixed matrix on Windows plus the zombie/reap matrix on Linux — every case returns the same verdict under both implementations. One more Windows-specific detail: the child holds the stdio pipes the shim owns, so the shim cannot exit while the real server keeps running — the "shim exits, server survives" scenario does not occur even in the failure direction.

And since that measurement, upstream landed e1e72f109c ("register stdio MCP helper children in the spawn ledger and reap orphans", #61514) on the tracking side: helper children are now registered into _stdio_child_pids at spawn time, so the real server is tracked directly and the liveness check doesn't need to chase it through the process tree. With both halves in place (polarity fix here + spawn ledger upstream), the re-open condition for the walk stays narrow: a launcher topology where the real server outlives the shim and the shim's PID is the only handle to it at check time.

So the polarity fix in this PR is the complete fix for the topology you describe — no extension PR needed. If that narrow re-open condition ever shows up in the wild, a pidfile/snapshot variant would slot into the ledger.

@kshitijk4poor
kshitijk4poor merged commit ef46ec0 into NousResearch:main Aug 27, 2026
37 checks passed
@kshitijk4poor

Copy link
Copy Markdown
Contributor

Merged with your commits preserved via rebase (merge commit ef46ec0). This was the canonical carrier for the stdio liveness inversion — the fail-open hardening (psutil/probe failures = unknown, never fast-fail) and the 8-test suite made it the clear winner over the later duplicates (#95947's MCP half closed pointing here). Thanks for the disciplined rebase and for shepherding the verification wave.

@47Hunter47

Copy link
Copy Markdown

Independent verification confirmed by @lilaflo on issue #95150 (v0.20.5, commit 786f370): after this fix, MCP server (stdio) tool registration appears at startup and all stdio MCP tool calls work end-to-end; before it, every call failed fast with TimeoutError despite healthy server processes. Thanks for the clean 2-line fix.

teknium1 pushed a commit that referenced this pull request Sep 2, 2026
…k probe

The fast-fail gate probed the stdio child watcher by CALLING it —
inspect.isawaitable(_watch_children()) — creating a fresh coroutine on
every stdio MCP tool call that was never awaited (RuntimeWarning spam +
gc churn). Inspect the function instead of invoking it.

Salvaged (unique hunk only) from PR #96044; the bundled
_stdio_children_dead polarity fix was already on main via #94339.
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…k probe

The fast-fail gate probed the stdio child watcher by CALLING it —
inspect.isawaitable(_watch_children()) — creating a fresh coroutine on
every stdio MCP tool call that was never awaited (RuntimeWarning spam +
gc churn). Inspect the function instead of invoking it.

Salvaged (unique hunk only) from PR NousResearch#96044; the bundled
_stdio_children_dead polarity fix was already on main via NousResearch#94339.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P1 High — major feature broken, no workaround tool/mcp MCP client and OAuth type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] _stdio_children_dead() inverted liveness check fail-fasts every stdio MCP call in oneshot (-z) sessions