Skip to content

fix(terminal): deadlock-proof _bash_starts probe on Windows - #78023

Closed
ffouladgar-ux wants to merge 1 commit into
NousResearch:mainfrom
ffouladgar-ux:fix/terminal-bash-probe-deadlock
Closed

fix(terminal): deadlock-proof _bash_starts probe on Windows#78023
ffouladgar-ux wants to merge 1 commit into
NousResearch:mainfrom
ffouladgar-ux:fix/terminal-bash-probe-deadlock

Conversation

@ffouladgar-ux

Copy link
Copy Markdown

What does this PR do?

Fixes a permanent hang of the bash availability probe (_bash_starts) on Windows. The probe previously used subprocess.run(timeout=15), whose Windows cleanup path has a fatal flaw: when the timeout fires, Python kills only the direct child and then runs an unbounded communicate() cleanup. If MSYS bash (Git-for-Windows) hangs while spawning an external program (/usr/bin/true; /usr/bin/cat --version), orphaned grandchild processes keep the stdout pipe open, so the cleanup communicate() blocks forever.

Because _bash_starts is called during every LocalEnvironment init, the hang stalled every tool that uses the local environment (terminal, file reads) indefinitely. Inside the ACP adapter process this meant ACP turns never completed and every delegated task hung forever.

The fix replaces subprocess.run with a Popen + explicit communicate(timeout=15) pair, and on timeout:

  • kills the whole process tree via taskkill /PID <pid> /T /F on Windows (not just the direct child),
  • does a bounded 5s post-kill read — never an unbounded cleanup.

Verified end-to-end on Windows 11: standalone ACP probes (read_file -> 11.6s full reply; terminal echo hi -> 3.3s, exit 0) and a real Multica issue that completed in 1 minute and posted its finalization comment — the exact action that had hung for days.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • tools/environments/local.py — rewrite _bash_starts probe:
    • Popen with stdin=DEVNULL, stdout/stderr=PIPE (was capture_output=True),
    • explicit communicate(timeout=15),
    • on TimeoutExpired: taskkill /PID <pid> /T /F tree-kill on Windows + proc.kill() fallback, then a bounded 5s communicate() read,
    • on success path, returncode read from the Popen object.

How to Test

  1. Windows host with Git-for-Windows/MSYS bash installed.
  2. Reproduce the hang: with the old code, run a Hermes ACP task under conditions where MSYS bash hangs in external-program spawn (system-wide Mandatory ASLR enabled reproduces it reliably); observe the tool thread blocking forever.
  3. Apply this PR; repeat — the probe now times out at 15s, tree-kills, and returns False; tools initialize and complete.
  4. Regression: on macOS/Linux, _bash_starts still returns True for a healthy bash.

Checklist

  • My commit messages follow Conventional Commits
  • My PR contains only changes related to this fix (no unrelated commits)
  • I have tested on my platform: Windows 11 (E2E ACP + terminal) and macOS (existing behavior unchanged)

Replace subprocess.run(timeout=15) with Popen + explicit
communicate(timeout=15). On Windows, subprocess.run's timeout cleanup
kills only the direct child then runs an UNBOUNDED communicate(); when
MSYS bash hangs in external-program spawn, orphaned grandchild
processes hold the stdout pipe open and the cleanup blocks forever,
stalling every tool that inits a LocalEnvironment (terminal, file
reads) indefinitely inside the ACP adapter process.

On timeout: taskkill /PID <pid> /T /F tree-kill on Windows + bounded
5s post-kill read. Verified end-to-end on Windows: ACP probes complete
in 3-12s and real Multica issue tasks finalize.
@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard tool/terminal Terminal execution and process management backend/local Local shell execution platform/windows Native Windows-specific behavior or breakage P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Aug 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #69083. Both repair the same Windows _bash_starts timeout cleanup deadlock; #69083 has the maintained shared helper, corrected process-tree kill ordering, and coverage.

@ffouladgar-ux

Copy link
Copy Markdown
Author

Thanks for the triage — agreed, this is a duplicate of #69083, which is the more complete fix (shared helper, corrected kill ordering, tests). Closing this one in favor of it.

One data point to add for #69083 since I hit this independently: the hang is confirmed end-to-end in the wild — a Multica-spawned hermes acp task on Windows stalled for days at the first tool call (terminal/file both route through _bash_starts), and the tree-kill approach (taskkill /T /F) + bounded post-kill read fixed it. Verified with standalone ACP probes (tools complete in 3-12s) and a real issue task that finalized in ~1 min. Happy to help test #69083 when it lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/local Local shell execution comp/cli CLI entry point, hermes_cli/, setup wizard duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants