Skip to content

fix(tools): prevent terminal probe hang and kill orphaned processes on Windows - #103402

Open
ericmaddox wants to merge 1 commit into
NousResearch:mainfrom
ericmaddox:fix/windows-gitbash-probe-hang
Open

ericmaddox wants to merge 1 commit into
NousResearch:mainfrom
ericmaddox:fix/windows-gitbash-probe-hang

Conversation

@ericmaddox

Copy link
Copy Markdown

What does this PR do?

When testing whether candidate Git Bash executables on Windows can successfully launch MSYS child processes, _bash_starts runs an external program probe (/usr/bin/true; /usr/bin/cat --version >/dev/null).

On Windows, Git for Windows' <root>\bin\bash.exe is a wrapper shim that spawns <root>\usr\bin\bash.exe. When the probe child deadlocks (e.g. MSYS2 fork deadlock), subprocess.run(..., timeout=15) times out and calls proc.kill(). However, on Windows proc.kill() (via TerminateProcess) only terminates the direct wrapper process, leaving the re-exec'd usr\bin\bash.exe process alive and holding open the stdout/stderr pipe handles.

Because Python's standard library subprocess.run implementation calls proc.communicate() without a timeout inside its except TimeoutExpired block to drain pipes, it blocks indefinitely waiting for EOF on the still-open pipes. This causes the entire terminal environment creation to hang for minutes past the timeout.

Fix

  1. Implement _run_probe_command with explicit process tree termination on Windows (taskkill /F /T /PID <pid>) upon timeout before killing the process, ensuring all child/grandchild processes are stopped and pipes are closed.
  2. Tighten probe timeout from 15s to 5s (a healthy probe finishes in <0.2s).
  3. Update _windows_bash_candidates to probe <root>\usr\bin\bash.exe directly alongside <root>\bin\bash.exe, avoiding the extra wrapper process layer.
  4. Expand _MSYS_SPAWN_FAILURE_MARKERS to recognize timeout/deadlock probe diagnostics as MSYS spawn failures, ensuring actionable remediation guidance is presented if all candidates fail.

Related Issue

Fixes #103398

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tools/environments/local_gitbash_probe.py: Replace unbounded subprocess.run with _run_probe_command which executes taskkill /F /T /PID <pid> on Windows timeout. Tighten timeout to 5s. Add timeout/deadlock markers to _MSYS_SPAWN_FAILURE_MARKERS.
  • tools/environments/local.py: In _windows_bash_candidates, include usr/bin bash candidate paths and resolve direct usr�in�ash.exe for custom paths.
  • tests/tools/test_find_shell.py: Update unit tests to mock _run_probe_command and add test_probe_timeout_records_deadlock_diagnostic asserting that probe timeouts fail cleanly and record MSYS failure diagnostics.

How to Test

  1. Run unit tests covering shell resolution and probe execution:
    pytest tests/tools/test_find_shell.py -v
  2. Verify all tests pass and test_probe_timeout_records_deadlock_diagnostic validates that a hung probe times out and registers as an MSYS failure.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11 / Python 3.11

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

…n Windows

When checking whether Git Bash candidates can launch MSYS2 child processes on Windows,
`_bash_starts` used `subprocess.run(timeout=15)`. When a candidate deadlocked during probe
(e.g. MSYS fork deadlock), `subprocess.run`'s timeout kill only terminated the parent wrapper
(`bin\bash.exe`), leaving the re-exec'd child (`usr\bin\bash.exe`) alive. Because the child
held the stdout/stderr pipe handles open, Python's `communicate()` inside `subprocess.run`
hung indefinitely past the timeout.

Fix this by:
1. Replacing unbounded communicate/kill with `_run_probe_command` which executes taskkill /F /T
   on Windows to terminate the entire process tree on timeout.
2. Tightening probe timeout from 15s to 5s.
3. Adding direct `usr\bin\bash.exe` candidate paths to `_windows_bash_candidates` so Git for
   Windows installations can be probed directly.
4. Adding timeout and deadlock markers to `_MSYS_SPAWN_FAILURE_MARKERS` to surface actionable
   remediation instructions if all candidates fail.

Closes NousResearch#103398
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tools Tool registry, model_tools, toolsets tool/terminal Terminal execution and process management backend/local Local shell execution platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Sep 5, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Related: #83413 (earlier open PR, same tree-kill approach for the orphaned usr\bin\bash.exe grandchild, plus stdin handling under ACP embedding) and issues #74982 / #103398. This PR additionally probes usr\bin\bash.exe directly and tightens the probe timeout. Competing fixes for the same hang — flagging so a maintainer can consolidate.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/local Local shell execution comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: terminal tool hangs for minutes on trivial commands — bash startup probe deadlocks, and the probe child survives subprocess.run's kill

2 participants