Skip to content

fix(env-probe): stuck Windows probe can no longer deadlock system-prompt builds - #67999

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-1a618f9e
Jul 20, 2026
Merged

fix(env-probe): stuck Windows probe can no longer deadlock system-prompt builds#67999
teknium1 merged 1 commit into
mainfrom
hermes/hermes-1a618f9e

Conversation

@teknium1

@teknium1 teknium1 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

A stuck local-toolchain probe can no longer deadlock system-prompt construction — a wedged pip --version subprocess (Windows orphaned-descendant pipe hang, #67964) now degrades only the probe line itself, never new sessions.

Root cause: on Windows, subprocess.run(capture_output=True, timeout=3) reacts to TimeoutExpired by calling communicate() a second time with no timeout to collect partial output. That call joins the pipe reader threads unboundedly. When the probed pip.exe launcher left an orphaned python.exe descendant holding the inherited stdout/stderr write handles, the pipes never hit EOF, the warm-probe thread hung inside subprocess._communicate while holding the module-level _CACHE_LOCK, and every new session's prompt build blocked at get_environment_probe_line() — zero API calls, working forever (28 min in the reported incident until manual orphan kill).

Changes

  • tools/env_probe.py::_run — Popen + communicate(timeout) instead of subprocess.run; on timeout: kill the process tree (taskkill /T on Windows), bounded wait(1) reap, and never a second pipe read. Bounded always; abandoned daemon reader threads are capped (≤6/process, result cached).
  • tools/env_probe.py::get_environment_probe_line — the probe now runs in a single background worker that publishes via threading.Event. Callers wait at most 10s and fail open with "" (prompt just omits the toolchain line). After one full timeout, later callers only peek (~50ms) — a permanently stuck probe costs the wait once, not per-session. If the stuck worker ever finishes, its line resumes appearing in new prompts (matches the incident: killing the orphan un-wedged everything with no restart).
  • warm_environment_probe_async — same single-worker start path; a generation counter keeps test resets from racing stale workers.
  • tests/tools/test_env_probe.py — regression tests: hung probe + 4 concurrent callers return bounded and fail open; late recovery publishes the line; repeat callers skip the full wait; _run() returns promptly against a real pipe-holding descendant subprocess (the reporter's suggested deterministic repro, cross-platform).

Validation

Before After
Orphaned pip descendant holds pipes (E2E, real pip shim on PATH) warm thread wedged holding lock; all prompt builds block indefinitely probe worker finishes in 3.2s; 4 concurrent "sessions" all return in 3.2s
_run() vs pipe-holding grandchild sleeping 20s unbounded post-kill communicate (Windows) returns timeout in ~1s
tests/tools/test_env_probe.py 10 tests 14 tests, all green via scripts/run_tests.sh
tests/run_agent/test_run_agent.py (probe wiring) — green

Fixes #67964

Infographic

env-probe-deadlock-proof

@teknium1
fix(env-probe): stuck Windows probe can no longer deadlock system-pro…
b151861 
…mpt builds

An orphaned pip descendant holding the probe's inherited stdout/stderr
pipe handles wedged subprocess.run's post-timeout communicate() (which
joins the pipe reader threads with NO timeout on Windows). The warm
probe thread then hung holding the module-level _CACHE_LOCK, so every
new session's prompt build blocked indefinitely (#67964).

Two layers:

- _run(): replace subprocess.run with Popen + communicate(timeout); on
  TimeoutExpired kill the process TREE (taskkill /T on Windows) and
  reap the direct child bounded — never re-read the pipes, so an
  orphaned descendant holding them open can't block us.
- get_environment_probe_line(): the probe now runs in a single
  background worker publishing via a threading.Event; callers wait at
  most _PROBE_WAIT_TIMEOUT (10s) then fail open with "". After one
  full timeout, later callers only peek. If the stuck worker ever
  finishes, its line resumes appearing in new prompts.

Regression tests: hung probe with 4 concurrent callers returns bounded;
late recovery publishes; repeat callers skip the wait; _run() returns
promptly despite a pipe-holding descendant (real subprocess E2E).

Fixes #67964
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows needs-decision Awaiting maintainer decision before any implementation labels Jul 20, 2026
@alt-glitch

alt-glitch commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to open #67991 and #67964. Both address the same Windows env-probe deadlock; this PR additionally uses process-tree cancellation plus a bounded shared worker/fail-open path, so maintainers should choose or consolidate rather than treat it as a duplicate.

@GottZ GottZ mentioned this pull request Jul 20, 2026
Closed
@teknium1
teknium1 merged commit b9dba7e into main Jul 20, 2026
33 checks passed
@teknium1
teknium1 deleted the hermes/hermes-1a618f9e branch July 20, 2026 12:38
This was referenced Jul 20, 2026
Merged
Closed
teknium1 added a commit that referenced this pull request Jul 23, 2026
@teknium1
fix(windows): re-fit env_probe console suppression to the temp-file _…
914ddcf 
…run + add no-window tests (#67690 follow-up)

Follow-up to the #67690 salvage (@m4r13y). The PR's tools/env_probe.py
hunk was written against the old capture_output=True _run(); #67964/#67999
rewrote _run to temp-file capture on July 20, so that hunk no longer
applied — but the rewritten _run still lacked creationflags and kept
flashing one console per probe (~5 per kanban worker start) from
windowless parents. Re-implement the one-line fix against the current
shape: creationflags=windows_hide_flags() on the temp-file subprocess.run,
preserving the #67964 grandchild-can't-wedge-the-pipe contract.

Also add the tests the PR didn't ship, in
tests/test_windows_subprocess_no_window_flags.py:
- env_probe._run passes CREATE_NO_WINDOW and keeps temp-file (non-PIPE)
  stdout/stderr + DEVNULL stdin
- lazy_deps uv install / pip --version probe / pip install fallback /
  ensurepip bootstrap all pass CREATE_NO_WINDOW
- suppress_platform_ver_console: POSIX no-op (platform._syscmd_ver
  untouched, win32_ver() still returns), and simulated-Windows stubbing
  (echo stub installed, idempotent, never raises)
teknium1 added a commit that referenced this pull request Jul 24, 2026
@teknium1
fix(windows): re-fit env_probe console suppression to the temp-file _…
9235802 
…run + add no-window tests (#67690 follow-up)

Follow-up to the #67690 salvage (@m4r13y). The PR's tools/env_probe.py
hunk was written against the old capture_output=True _run(); #67964/#67999
rewrote _run to temp-file capture on July 20, so that hunk no longer
applied — but the rewritten _run still lacked creationflags and kept
flashing one console per probe (~5 per kanban worker start) from
windowless parents. Re-implement the one-line fix against the current
shape: creationflags=windows_hide_flags() on the temp-file subprocess.run,
preserving the #67964 grandchild-can't-wedge-the-pipe contract.

Also add the tests the PR didn't ship, in
tests/test_windows_subprocess_no_window_flags.py:
- env_probe._run passes CREATE_NO_WINDOW and keeps temp-file (non-PIPE)
  stdout/stderr + DEVNULL stdin
- lazy_deps uv install / pip --version probe / pip install fallback /
  ensurepip bootstrap all pass CREATE_NO_WINDOW
- suppress_platform_ver_console: POSIX no-op (platform._syscmd_ver
  untouched, win32_ver() still returns), and simulated-Windows stubbing
  (echo stub installed, idempotent, never raises)
teknium1 added a commit that referenced this pull request Jul 24, 2026
@teknium1
fix(windows): re-fit env_probe console suppression to the temp-file _…
0b17d4d 
…run + add no-window tests (#67690 follow-up)

Follow-up to the #67690 salvage (@m4r13y). The PR's tools/env_probe.py
hunk was written against the old capture_output=True _run(); #67964/#67999
rewrote _run to temp-file capture on July 20, so that hunk no longer
applied — but the rewritten _run still lacked creationflags and kept
flashing one console per probe (~5 per kanban worker start) from
windowless parents. Re-implement the one-line fix against the current
shape: creationflags=windows_hide_flags() on the temp-file subprocess.run,
preserving the #67964 grandchild-can't-wedge-the-pipe contract.

Also add the tests the PR didn't ship, in
tests/test_windows_subprocess_no_window_flags.py:
- env_probe._run passes CREATE_NO_WINDOW and keeps temp-file (non-PIPE)
  stdout/stderr + DEVNULL stdin
- lazy_deps uv install / pip --version probe / pip install fallback /
  ensurepip bootstrap all pass CREATE_NO_WINDOW
- suppress_platform_ver_console: POSIX no-op (platform._syscmd_ver
  untouched, win32_ver() still returns), and simulated-Windows stubbing
  (echo stub installed, idempotent, never raises)
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@teknium1
fix(env-probe): stuck Windows probe can no longer deadlock system-pro…
3201af9 
…mpt builds (NousResearch#67999)

An orphaned pip descendant holding the probe's inherited stdout/stderr
pipe handles wedged subprocess.run's post-timeout communicate() (which
joins the pipe reader threads with NO timeout on Windows). The warm
probe thread then hung holding the module-level _CACHE_LOCK, so every
new session's prompt build blocked indefinitely (NousResearch#67964).

Two layers:

- _run(): replace subprocess.run with Popen + communicate(timeout); on
  TimeoutExpired kill the process TREE (taskkill /T on Windows) and
  reap the direct child bounded — never re-read the pipes, so an
  orphaned descendant holding them open can't block us.
- get_environment_probe_line(): the probe now runs in a single
  background worker publishing via a threading.Event; callers wait at
  most _PROBE_WAIT_TIMEOUT (10s) then fail open with "". After one
  full timeout, later callers only peek. If the stuck worker ever
  finishes, its line resumes appearing in new prompts.

Regression tests: hung probe with 4 concurrent callers returns bounded;
late recovery publishes; repeat callers skip the wait; _run() returns
promptly despite a pipe-holding descendant (real subprocess E2E).

Fixes NousResearch#67964
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@teknium1
fix(windows): re-fit env_probe console suppression to the temp-file _…
2bf6d4c 
…run + add no-window tests (NousResearch#67690 follow-up)

Follow-up to the NousResearch#67690 salvage (@m4r13y). The PR's tools/env_probe.py
hunk was written against the old capture_output=True _run(); NousResearch#67964/NousResearch#67999
rewrote _run to temp-file capture on July 20, so that hunk no longer
applied — but the rewritten _run still lacked creationflags and kept
flashing one console per probe (~5 per kanban worker start) from
windowless parents. Re-implement the one-line fix against the current
shape: creationflags=windows_hide_flags() on the temp-file subprocess.run,
preserving the NousResearch#67964 grandchild-can't-wedge-the-pipe contract.

Also add the tests the PR didn't ship, in
tests/test_windows_subprocess_no_window_flags.py:
- env_probe._run passes CREATE_NO_WINDOW and keeps temp-file (non-PIPE)
  stdout/stderr + DEVNULL stdin
- lazy_deps uv install / pip --version probe / pip install fallback /
  ensurepip bootstrap all pass CREATE_NO_WINDOW
- suppress_platform_ver_console: POSIX no-op (platform._syscmd_ver
  untouched, win32_ver() still returns), and simulated-Windows stubbing
  (echo stub installed, idempotent, never raises)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint needs-decision Awaiting maintainer decision before any implementation P1 High — major feature broken, no workaround platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Windows env probe can deadlock all new sessions when an orphaned pip child keeps capture pipes open

2 participants

@teknium1 @alt-glitch