Skip to content

fix(shell): stop shell_execute hang on background children - #1887

Merged
Aaronontheweb merged 5 commits into
netclaw-dev:devfrom
Aaronontheweb:skunkworks/shell-daemon-drain-fix
Aug 18, 2026
Merged

fix(shell): stop shell_execute hang on background children#1887
Aaronontheweb merged 5 commits into
netclaw-dev:devfrom
Aaronontheweb:skunkworks/shell-daemon-drain-fix

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Problem

shell_execute hangs when a command starts a background process. One
example is a command that starts itself as a daemon. A bare nginx call
returns at once in a real shell. Through shell_execute, the same call
hung for about 715 seconds.

The direct process exits fast. A forked child, such as a daemon or a
cmd & background job, inherits the stdout/stderr pipe. The child keeps
the pipe write end open after the direct process exits.

Root cause

The output drain read the pipe with CancellationToken.None. No token
could stop that read. The drain waited for the pipe to reach end of file.
End of file came only when the background child also exited. Neither the
caller token nor the command timeout could stop the read.

Fix

src/Netclaw.Actors/Tools/ShellTool.cs and
src/Netclaw.Actors/Tools/BoundedOutputReader.cs carry the fix.

  • The drain now reads with a real CancellationToken. The token links to
    the caller token and the command timeout, the same as before.
  • Once the direct process exits, the tool arms a 500 ms grace period on
    that token. The grace period lets buffered output flush. The drain then
    stops, and the tool returns.
  • The fix covers both output-capture paths in ShellTool: the buffered
    path (ExecuteCoreAsync) and the path that streams tool output
    (ExecuteStreamCoreAsync).
  • A grace period can cut a capture before end of file. This happens when a
    background child still holds the pipe open.
    BoundedOutputReader.DrainToWindowAsync now returns a Cancelled flag
    for this case. ShellTool logs a warning on a grace cut and adds a note
    to the result text. The note tells the agent that a background process
    held the pipe open. The note tells the agent that capture stopped at
    that point.

Tests

Two regression tests reproduce the hang with sleep 20 & exit 0. The
direct bash process exits at once. The backgrounded sleep holds the pipe
open for 20 seconds. Each test asserts the tool returns within five
seconds.

  • ShellToolTests.Direct_process_exit_with_backgrounded_child_holding_pipe_open_returns_promptly
  • ShellToolStreamingTests.Direct_process_exit_with_backgrounded_child_holding_pipe_open_streams_promptly

Both tests failed before the fix. The tool took about 20 seconds. The
tool missed the five-second bound. Both tests are POSIX-only, because
they use & background-job syntax.

A second pair of tests proves the grace-cut marker:

  • A normal command that reaches end of file shows no note.
  • A backgrounded child that outlives the direct process shows the note.

Validation

  • dotnet test src/Netclaw.Actors.Tests (full project): 3152 passed, 1
    skipped (a Windows-only test), 0 failed.
  • dotnet slopwatch analyze: 0 issues.
  • ./scripts/Add-FileHeaders.ps1 -Verify: all files have headers.

Review record

A non-author agent reviewed this change in an isolated worktree. The agent
tried to break the fix. The agent found no correctness, security, or
data-loss defect. Verdict: APPROVE WITH NITS. The review comment is on
this PR.

The agent found one nit with real impact. A grace cut did not tell the
agent that capture stopped early. A follow-up commit on this PR fixed
that nit. The commit added the Cancelled flag and the result-text note
described above. The remaining nits are minor and stay open by design
(wall-clock grace timer, fixed 500 ms cost on a backgrounded command).

@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Adversarial review — non-author agent

Verdict: APPROVE WITH NITS.

A non-author agent reviewed this change in an isolated worktree. The agent tried to break the fix.
The agent found no correctness, security, or data-loss defect.

Proof points:

  • The new tests are real. The agent reverted only the two production files. The tests then failed with
    a 20.0s hang. The agent restored the fix. The tests then passed in about 0.5s.
  • Both capture paths (streaming and non-streaming) are fixed and each has a test.
  • Output is not lost. A 200,000-char burst before a backgrounded child exits captured 200000/200000
    chars. Over-budget capture self-signals with an embedded marker.
  • No regression. The grace timer arms only after WaitForExitAsync returns, so a long foreground
    command still runs to its real timeout.
  • Gates (agent re-ran them): slopwatch 0 issues; ShellTool and streaming suites 39 passed, 1
    Windows-only skip.

Open nits (minor; deferred to the upstream decision):

  1. The grace-cut path drops the Truncated flag (ShellTool.cs:214), so a cut of an under-budget
    lingering child's stream carries no signal. That stream is not the command result, so the impact is
    cosmetic. A one-line signal would close the no-silent-fallback rule fully.
  2. The grace uses CancellationTokenSource.CancelAfter (a real timer), not TimeProvider, so the two
    new tests use wall-clock margins. This matches the file's existing style.
  3. A command of the form cmd & exit 0 now pays a fixed ~500ms grace before it returns.

This PR is a do-not-merge focus PR for cherry-pick evaluation.

@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

This change fixes the silent-cut finding from the adversarial review.

The post-exit grace window can cut the output drain before EOF. A background child can hold the pipe open after the direct process exits. The tool did not report this cut before now.

BoundedOutputReader.DrainToWindowAsync and the streaming pipe drain in ShellTool now report a Cancelled flag. The flag marks a grace cut, not a budget cut. ShellTool logs a warning on a grace cut and appends a short note to the result text. The note tells the agent that a background process held the pipe open and that the tool did not capture output after that point.

Tests prove both cases. A normal command that reaches EOF shows no note. A backgrounded child that outlives the direct process shows the note. The full Netclaw.Actors.Tests suite passes, and dotnet slopwatch analyze reports 0 new issues.

Commit: 11562b9

A command that daemonizes leaves shell_execute stuck. The direct
process exits at once. A forked or backgrounded child keeps the
stdout/stderr pipe open. The tool waited for that pipe to close,
with no bound on the wait.

The output drain read the pipe with CancellationToken.None. A real
token could not stop the read. The tool then hung until the child
exited on its own, for example about 715 seconds for a bare nginx
call.

The fix links a real token into the drain. The token cancels with
the caller token and the command timeout, the same as before. Once
the direct process exits, the tool arms a short grace period on
that token. The grace period lets buffered output flush. Then the
drain stops. The tool returns. The fix applies to both the
streaming path and the non-streaming path in ShellTool.

A foreground command keeps its existing timeout. Only the drain
after process exit gets the short grace period.

Two new regression tests reproduce a backgrounded child that holds
the pipe open. Each test asserts the tool returns within five
seconds, not twenty.
The post-exit grace window can cut the output drain before EOF. A
background child can hold the pipe write end open after the direct
process exits. The tool did not report this cut before now. The
result text looked complete, but the capture could be partial.

BoundedOutputReader.DrainToWindowAsync now returns a Cancelled flag.
The streaming pipe drain in ShellTool now returns the same flag. The
flag marks a grace cut. It does not mark a budget cut.

ShellTool logs a warning on a grace cut. ShellTool also appends a
note to the result text on a grace cut. The note tells the agent that
a background process held the pipe open. The note tells the agent
that the tool did not capture output after that point.

Tests prove both cases. A normal command that reaches EOF shows no
note. A backgrounded child that outlives the direct process shows the
note.
@Aaronontheweb
Aaronontheweb force-pushed the skunkworks/shell-daemon-drain-fix branch from 11562b9 to 20508c5 Compare August 13, 2026 01:55
@Aaronontheweb Aaronontheweb changed the title [skunkworks/do-not-merge] fix(shell): stop shell_execute hang on background children fix(shell): stop shell_execute hang on background children Aug 13, 2026
@Aaronontheweb
Aaronontheweb marked this pull request as ready for review August 13, 2026 01:56
@Aaronontheweb Aaronontheweb added bug Something isn't working shell Issues related to the shell tool, since it has the largest security perimeter. labels Aug 13, 2026
The follow-up commit widened DrainToWindowAsync from a two-element
tuple to a three-element tuple. It added a Cancelled flag. Two
benchmark files still deconstructed the old two-element shape. The
build failed with error CS8132 on all three OS test jobs.

This commit updates both deconstructions to the three-element shape.
The benchmarks discard the new Cancelled flag; they do not need it.

ShellTool.cs had a catch block with only a comment and no statement.
Slopwatch treats a comment-only catch block as empty and reports
error SW003. This commit adds a Debug.WriteLine call that records the
cancellation. Other catch blocks in the same file use this pattern.

BoundedOutputReaderTests.cs uses an infinite Task.Delay in a test
helper. The helper stands in for a pipe that never reaches end of
file. Slopwatch reports this delay as warning SW004. This commit adds
a SlopwatchSuppress attribute with the reason.
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

CI broke for two reasons.

The follow-up commit widened DrainToWindowAsync to a three-element
tuple. Two benchmark files still deconstructed the old two-element
shape, so the build failed with error CS8132 on all three OS jobs.
ShellTool.cs also had a catch block with only a comment, which
Slopwatch reports as an empty catch block (error SW003).

Fix in b16baf3:

  • Update both benchmark deconstructions to the three-element shape.
  • Add a Debug.WriteLine call to the catch block, matching the
    pattern other catch blocks in the same file already use.
  • Add a SlopwatchSuppress attribute to the intentional infinite
    Task.Delay test helper, to clear the SW004 warning.

dotnet build -c Release, dotnet test src/Netclaw.Actors.Tests,
dotnet slopwatch analyze, and Add-FileHeaders.ps1 -Verify all
pass locally.

@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Repair commit f346408 addresses the follow-up review.

  • The fix removes the unused logger dependency and all raw command logging.
  • The agent-visible grace-cut marker remains in buffered and streamed results.
  • The original public three-parameter ShellTool constructor remains available.
  • A reflection test locks that constructor signature.
  • The fix removes a duplicate IsPosix member that caused all platform builds to fail.

Validation:

  • Release build: passed with zero warnings.
  • BoundedOutputReader tests: 14 passed.
  • ShellTool buffered tests: 31 passed and one Windows-only test skipped.
  • ShellTool streaming tests: 10 passed.
  • Full Netclaw.Actors.Tests: 3,407 passed and one Windows-only test skipped.
  • Header verification: passed.
  • git diff --check: passed.
  • Slopwatch: exited 0. It reported one existing warning in unchanged PowerShellHostProbeTests.cs.

This change does not add command text to OTLP logs. The tool result remains the operator-visible signal for a grace cut.

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb Aaronontheweb added the reliability Retries, resilience, graceful degradation label Aug 18, 2026
@Aaronontheweb
Aaronontheweb enabled auto-merge (squash) August 18, 2026 00:15
@Aaronontheweb
Aaronontheweb merged commit 265d606 into netclaw-dev:dev Aug 18, 2026
17 checks passed
@Aaronontheweb Aaronontheweb mentioned this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working reliability Retries, resilience, graceful degradation shell Issues related to the shell tool, since it has the largest security perimeter.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant