Skip to content

fix(tools): bound env probe subprocess so a Windows inherited pipe can't wedge sessions (#67964) - #67991

Closed
PRATHAMESH75 wants to merge 1 commit into
NousResearch:mainfrom
PRATHAMESH75:fix/env-probe-windows-pipe-deadlock
Closed

PRATHAMESH75 wants to merge 1 commit into
NousResearch:mainfrom
PRATHAMESH75:fix/env-probe-windows-pipe-deadlock

Conversation

@PRATHAMESH75

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #67964. On native Windows, the local Python toolchain probe can deadlock every new session's system-prompt build.

tools/env_probe.py::_run captures subprocess output with capture_output=True (OS pipes). A console-script launcher such as pip.exe spawns a descendant (python.exe … pip --version) that inherits the captured stdout/stderr write ends and can outlive its parent. With pipes, the reader threads inside subprocess.communicate() then block until that grandchild closes the write end — which the 3s timeout does not bound, because killing the direct child leaves the grandchild holding the pipe. In the reported incident the warm-probe thread stayed blocked ~28 min.

Because get_environment_probe_line() holds the module-level _CACHE_LOCK across the probe, every new session that needs a system prompt then waits on that lock indefinitely — a partial outage (new chats stuck working, zero API calls; older sessions with a persisted prompt unaffected).

Fix

Capture through tempfile.TemporaryFile() instead of capture_output pipes. Temp files have no reader threads, so subprocess's timeout/wait() only ever waits on the direct child. A lingering grandchild holding an inherited handle can no longer block the parent, the documented 3s timeout genuinely bounds the whole call, and the probe fails open — satisfying the issue's stated expectation. This mirrors the "don't wait on a Windows-inherited pipe handle" pattern already used in #67373 for BaseEnvironment.execute().

Single-purpose change confined to _run; the happy-path return contract (returncode, stdout, stderr) is unchanged, so _CACHE_LOCK is now held for at most the bounded probe duration.

Testing

  • New TestRunBoundedByTimeout in tests/tools/test_env_probe.py: a direct child prints ok, spawns a 20s-sleeping grandchild inheriting its stdout, then exits. Asserts _run returns rc=0, out="ok" in < 3.0s. Verified this test fails on the old capture_output path (blocks the full 3s → rc=-1) and passes with temp-file capture.
  • Full tests/tools/test_env_probe.py (11) + tests/run_agent/test_run_agent.py — 447 passed.
  • ruff check clean (incl. PLW1514); check-windows-footguns.py and check_subprocess_stdin.py pass.

The arm64-fork-Docker CI job is expected to fail on fork PRs.

@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/tools Tool registry, model_tools, toolsets platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Jul 20, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Related: focused fix PR for open #67964's Windows inherited-pipe environment-probe deadlock; the targeted env-probe suite passes.

@teknium1

Copy link
Copy Markdown
Collaborator

Merged via #68045 — your commit was cherry-picked onto current main with your authorship preserved in git log (e77ffdc).

Context: while your PR was open, #67999 landed a different fix for the same deadlock (#67964) — process-tree kill on timeout plus a fail-open background worker so prompt builds never wait on a stuck probe. Your temp-file capture approach was the better mechanism for _run itself: no reader threads means the timeout genuinely bounds the whole call, and a fast probe with a lingering descendant now returns its real output in milliseconds instead of burning the timeout and losing the line. So we replaced the pipe-based _run from #67999 with yours, kept the fail-open caller layer, and carried your regression test along.

Thanks for the well-aimed fix — first submission of the capture mechanism, and it made the final design better.

@teknium1 teknium1 closed this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P1 High — major feature broken, no workaround platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Windows env probe can deadlock all new sessions when an orphaned pip child keeps capture pipes open

3 participants