Skip to content

fix(shell): harden PowerShell host probe against cold-start timeouts - #1859

Merged
Aaronontheweb merged 4 commits into
devfrom
fix/pwsh-probe-resilience
Aug 10, 2026
Merged

fix(shell): harden PowerShell host probe against cold-start timeouts#1859
Aaronontheweb merged 4 commits into
devfrom
fix/pwsh-probe-resilience

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Problem

Windows CI flake: TestShellEnvironment's static ctor resolves the shell environment by probing pwsh.exe with a hard 5s timeout. On a loaded Windows runner the cold pwsh.exe spawn (Defender scan, first-run init) exceeded 5s, the resolver threw (Failed did not fall back), and every dependent test class died with TypeInitializationException — 123 tests in run 31417450377.

The same risk exists in production: daemon startup resolves the shell env (Program.cs:133), so a slow pwsh could brick startup.

Changes

  1. Probe retries once on timeout. PowerShellHostProbe.ProbeAsync retries (250ms backoff) when the only failure is Timeout; other failures return immediately. Each attempt spawns a fresh process. Worst-case probe latency doubles (~12.5s per executable) but only on the rare timeout path — healthy hosts see zero added latency.

  2. Probe failure is fallback-eligible. The resolver no longer hard-throws when the pwsh probe fails. It probes powershell.exe 5.1 and records PowerShellFallbackReason.PreferredHostProbeFailed (startup logs a warning). Both-fail still throws — fail-closed is preserved, and the combined message now lists both failures (the old early-throw hid the preferred-failure detail).

  3. CI warm-up. pr_validation.yml pre-warms the probe on Windows runners (both pwsh and the powershell.exe fallback) so cold-start cost doesn't eat the in-test timeout budget.

Tests

  • Resolver test updated: pwsh probe Failed now selects Windows PowerShell 5.1 with PreferredHostProbeFailed; new both-fail test asserts the combined error.
  • Probe retry tests (fake-time driven): timeout→retry→recover, timeout→timeout returns Failed(Timeout), non-timeout failure does not retry.
  • Full Netclaw.Daemon.Tests: 1000/1000 pass on Linux.
  • Windows CI on this PR is the definitive check.

Note

Worst-case Windows daemon startup when both hosts are slow is now ~25s before failing with a clear combined error (vs ~6s before). Intentional tradeoff: slow-but-successful, or slow-but-clear failure, instead of bricked startup on a transient cold-start timeout.

Windows CI flake: TestShellEnvironment's static ctor resolves the shell
environment by probing pwsh.exe with a hard 5s timeout. On a loaded
Windows runner the cold pwsh.exe spawn (Defender scan, first-run init)
exceeded 5s, the resolver threw (Failed did not fall back), and every
dependent test class died with TypeInitializationException. The same
risk exists in production: daemon startup resolves the shell env, so a
slow pwsh could brick startup.

Changes:
- ProbeAsync retries once (250ms backoff) when the only failure is
  Timeout; other failures return immediately. Each attempt spawns a
  fresh process.
- Resolver treats a pwsh probe failure as fallback-eligible: it probes
  powershell.exe 5.1 and records PowerShellFallbackReason
  .PreferredHostProbeFailed. Both-fail now throws a combined message
  listing both failures instead of the old early-throw that hid the
  preferred-failure detail.
- CI: pre-warm the probe on Windows runners (pwsh + powershell.exe
  fallback) so cold-start cost doesn't eat the in-test timeout budget.
…mmand

The previous warm-up passed $PSVersionTable.PSVersion.ToString() inside a
double-quoted -Command string; the outer pwsh shell interpolated the
variable into its type name and the child powershell.exe failed to parse
the command. Use -Command "exit 0" (no variables) and guard the step
with a final exit 0 so a missing powershell.exe can't fail CI.
@Aaronontheweb
Aaronontheweb enabled auto-merge (squash) August 10, 2026 21:21
@Aaronontheweb Aaronontheweb added bug Something isn't working github_actions Pull requests that update GitHub Actions code reliability Retries, resilience, graceful degradation shell Issues related to the shell tool, since it has the largest security perimeter. labels Aug 10, 2026
@Aaronontheweb
Aaronontheweb merged commit 8682049 into dev Aug 10, 2026
29 of 30 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/pwsh-probe-resilience branch August 10, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working github_actions Pull requests that update GitHub Actions code reliability Retries, resilience, graceful degradation shell Issues related to the shell tool, since it has the largest security perimeter.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant