Skip to content

fix(tools): prevent worker deadlock when agent backgrounds server via shell & (#68915) - #70549

Closed
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/68915-worker-deadlock-bg
Closed

webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/68915-worker-deadlock-bg

Conversation

@webtecnica

Copy link
Copy Markdown
Contributor

Problem

When LLM issues A && B & via terminal, bash creates a subshell that holds stdout pipe open → worker deadlocks permanently (requires kill -9).

Root Cause

The existing _rewrite_compound_background fix only ran in the foreground path (base.py:execute()). The background spawn_local() path in process_registry.py bypassed it entirely.

Fix

  • Added _rewrite_compound_background call in spawn_local() before Popen/PTY spawn
  • Both PTY and Popen paths now use rewritten command
  • 4 regression tests added
  • Simple cmd & (without &&) preserved unchanged

Fixes #68915

…ker deadlock on server backgrounding

Issue NousResearch#68915: when the agent runs a compound command with trailing & (e.g.
`cd /app && node server.js &`), bash parses it as `(A && B) &` — a subshell
that holds the stdout pipe open forever when B is a long-running server.
The existing _rewrite_compound_background in terminal_tool.py correctly
rewrites this to `A && { B & }` to avoid the subshell fork, but it was only
applied in the foreground execute() path (tools/environments/base.py).

The background spawn_local() path bypasses base.py entirely and passed the
raw command directly to Popen/PTY, leaving the deadlock unmitigated.

Fix: apply _rewrite_compound_background in spawn_local() before the command
is passed to Popen or PTY spawn. Uses a lazy import to avoid circular
dependency (terminal_tool imports process_registry).

- PTY spawn path: now uses safe_command (rewritten)
- Popen spawn path: now uses safe_command (rewritten)
- Session.command still stores the original (unrewritten) command for display
- Simple `cmd &` is left unchanged (no subshell bug)

Tests: 4 regression tests verifying (1) compound is rewritten, (2) simple bg
is preserved, (3) multi-line compounds are rewritten, (4) session.command
stores original.
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/tools Tool registry, model_tools, toolsets tool/terminal Terminal execution and process management sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Jul 24, 2026
@kshitijk4poor

Copy link
Copy Markdown
Contributor

Closing as duplicate of #70884 (salvage of #70546). Both your spawn_local fix and the telegram connect fix were salvaged together. Your authorship preserved via cherry-pick.

Note: the commits in #70546 were authored under "Hermes Agent" identity — added contributor email mapping (agent@hermes.dev -> webtecnica) in the salvage PR.

kshitijk4poor added a commit to kshitijk4poor/hermes-agent that referenced this pull request Jul 24, 2026
Follow-up to salvaged PR NousResearch#70549:
- Replace fragile 'or' assertions with single precise checks that catch
  partial-rewrite regressions (would have masked a missing closing brace)
- Add test_pty_path_uses_rewritten_command covering the PTY spawn path
  that was modified but previously untested
kshitijk4poor added a commit that referenced this pull request Jul 24, 2026
Follow-up to salvaged PR #70549:
- Replace fragile 'or' assertions with single precise checks that catch
  partial-rewrite regressions (would have masked a missing closing brace)
- Add test_pty_path_uses_rewritten_command covering the PTY spawn path
  that was modified but previously untested
@kshitijk4poor

Copy link
Copy Markdown
Contributor

Merged via #70876 — your fix was cherry-picked with authorship preserved, and the original fix landed on main (commit d7512c8).

Your PR identified that spawn_local() in tools/process_registry.py was bypassing the _rewrite_compound_background rewriter already applied in base.py:execute(), causing the P1 worker deadlock. The follow-up PR added test coverage for the PTY spawn path and tightened the regression assertions.

Thanks for the contribution!

randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Follow-up to salvaged PR NousResearch#70549:
- Replace fragile 'or' assertions with single precise checks that catch
  partial-rewrite regressions (would have masked a missing closing brace)
- Add test_pty_path_uses_rewritten_command covering the PTY spawn path
  that was modified but previously untested
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
Follow-up to salvaged PR NousResearch#70549:
- Replace fragile 'or' assertions with single precise checks that catch
  partial-rewrite regressions (would have masked a missing closing brace)
- Add test_pty_path_uses_rewritten_command covering the PTY spawn path
  that was modified but previously untested
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
Follow-up to salvaged PR NousResearch#70549:
- Replace fragile 'or' assertions with single precise checks that catch
  partial-rewrite regressions (would have masked a missing closing brace)
- Add test_pty_path_uses_rewritten_command covering the PTY spawn path
  that was modified but previously untested
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P1 High — major feature broken, no workaround sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Worker deadlocks when agent backgrounds a server via shell & (orphaned subshell holds stdout pipe open)

3 participants