fix(providers): detect a mid-stream LLM stall within seconds

[Aaronontheweb]

Problem

A dead or half-open LLM connection can send a few tokens, then stop. No
error appears. The connection stays open. A benchmark run hit this case.
A stream stalled for about 605 seconds. The stall lost a race against an
outer 900-second kill. The run lost most of its time budget.

Netclaw catches this stall today with one coarse per-call watchdog
(FirstTokenTimeout, 600 seconds by default). This bound must stay
generous, because a self-hosted backend can stay silent for minutes
during cold prefill.

Five sidecar LLM call paths have no inter-delta stall guard at all:

• title generation
• memory extraction
• compaction observation
• memory distillation
• memory curation

Each sidecar path has only a flat absolute deadline (10 to 450 seconds).
A stall on a sidecar path can run to the full deadline before Netclaw
detects it.

Prior art: closed PR #1272

PR #1272 proposed TCP keepalive on the LLM HttpClient. The author
closed the PR after this finding. TCP keepalive proves only that the
socket between Netclaw and its reverse proxy (Caddy) stays alive. TCP
keepalive does not detect a stuck upstream stream. A self-hosted backend
can hold an established connection open, send zero bytes, and still
answer keepalive probes at the TCP layer.

The fix needs a signal from the application layer, not the transport
layer.

Mechanism

This PR adds StreamStallGuardChatClient, a new IChatClient decorator
at PipelineChatClientFactory.Compose. This seam wraps every LLM call
path: the main session, the sub-agent path, and all five sidecar paths.

The guard arms an inactivity timer only after the first substantive
update in a stream. ChatStreamUpdateClassifier.IsSubstantiveUpdate is
the one public predicate that decides "substantive." StreamingResponseReader
and this guard both call the same predicate, so the two components share
one rule.

Note: a reasoning-only delta with text counts as substantive. It arms
the timer, the same as a text or tool-call delta. This behavior matches
the watchdog rule used elsewhere in the codebase. Only a content-free
keepalive leaves the timer unarmed.

Once armed, every later update resets the timer. Only a stream that
produces no more output at all trips it.

The timer measures provider silence only. The guard disarms the timer
right after each update arrives, and arms it again just before the next
read. A slow downstream consumer that holds an update past the window
does not trigger a false abort.

On expiry the guard raises a TimeoutException. RetryPolicy.ShouldRetry
classifies this exception as retryable, the same as any other transient
failure. No new retry path exists. By the time this guard catches a
stall, the stream already sent at least one chunk. The transport layer
cannot silently retry a partial stream.

Config

The new timeout lives on RetryPolicy: StreamInactivityTimeout. The
default value is 45 seconds. An operator can set this value to zero to
turn the guard off.

src/Netclaw.Configuration/Schemas/netclaw-config.v1.schema.json now
covers Session.Tuning.StreamingRetryPolicy, so an operator can set
this value through netclaw.json.

Relationship to other work

This PR does not remove or change ProcessingWatchdog. The main session
path and the sub-agent path keep ProcessingWatchdog. On those two
paths, StreamStallGuardChatClient runs as a uniform guard under the
current watchdog.

PR #1893 adds a separate, later half: turn-level resume after a caught
stall. This PR only shortens the time to catch a stall. PR #1893
changes what happens after Netclaw catches one.

Tests

StreamStallGuardChatClientTests.cs uses a FakeTimeProvider. No test
uses a real Task.Delay or Thread.Sleep wait. The tests prove:

• A stream that yields two updates, then stalls forever, times out at
  the inactivity window.
• A healthy stream that paces updates under the window completes with
  no abort.
• A stall before the first update stays under the current watchdog, not
  this guard.
• A stream with only keepalive updates, then a stall, stays unguarded
  until a substantive update arrives.
• A slow consumer that holds an update past the window does not trigger
  an abort.
• A zero timeout turns the guard off.

PipelineChatClientFactoryTests.cs adds two end-to-end tests. These
tests prove the same behavior through the full composed pipeline:
Logging → Retry → StreamStallGuard → leaf.

ConfigSchemaDoctorCheckTests.cs adds one test. This test proves a
config that sets Session.Tuning.StreamingRetryPolicy passes schema
validation.

SessionConfigDefaultsTests.cs adds one test. This test proves the new
config keys bind to the runtime RetryPolicy.

Validation

This branch is rebased onto the current dev branch (upstream/dev at
b1832884). The rebase had no conflicts.

• dotnet build Netclaw.slnx — clean, 0 warnings, 0 errors
• dotnet test src/Netclaw.Daemon.Tests — 1027 passed
• dotnet test src/Netclaw.Configuration.Tests — 596 passed
• dotnet test src/Netclaw.Cli.Tests --filter ConfigSchemaDoctorCheck —
  24 passed
• dotnet test src/Netclaw.Actors.Tests — 3149 passed, 1 skipped (a
  Windows-only test)
• dotnet slopwatch analyze — 0 issues
• Add-FileHeaders.ps1 -Verify — all files have headers

Review record

Two adversarial review cycles checked this branch. A non-author agent
ran each cycle in an isolated worktree, and re-ran the gates itself.

The first cycle found three issues:

• The guard armed on any update, not only a substantive one.
• The timer stayed armed across a slow consumer read.
• The new config knob had no schema coverage.

A follow-up commit fixed all three.

The second cycle found two more issues:

• A production-to-production InternalsVisibleTo grant was wider than
  the guard needed.
• A comment claimed a reasoning-only delta never arms the timer. This
  claim was false.

A follow-up commit narrowed the grant to one public predicate
(ChatStreamUpdateClassifier). The same commit fixed the comment.

All five findings are fixed on this branch. The full review trail is in
the PR comments.

[Aaronontheweb]

Adversarial review — non-author agent

Verdict: REQUEST CHANGES.

A non-author agent reviewed this change in an isolated worktree and re-ran the gates itself: slopwatch
0 issues; Netclaw.Daemon.Tests 1022 passed; Netclaw.Configuration.Tests 519 passed. The tests are
real — the agent neutralized only the arm line and the stall test hung to a 90s kill; the agent
restored the code and the 10 tests passed in 35ms. Time is virtualized (FakeTimeProvider).

Good news first: the highest-risk trap is SAFE. A stream that emits tokens then stalls is NOT retried
(RetryingChatClient retries only pre-first-chunk) and the actor treats the TimeoutException as
terminal, so there is no partial-output double emission.

Blocking findings

F1 — The guard duplicates a more-capable watchdog Netclaw already owns, and it arms on the wrong
signal. The guard sets seenFirstUpdate on the first update of ANY kind — a content-free keepalive or
a reasoning-only delta — then arms the 45s timer. That contradicts the guard's own stated invariant
(time-to-first-byte stays governed by the coarser watchdog) and can abort a healthy cold prefill.
Netclaw already draws the correct distinction: StreamingResponseReader.IsSubstantiveUpdate +
ProcessingWatchdog.OnStreamProgress promote to a tight inter-delta budget only on the first
SUBSTANTIVE delta and keep a keepalive-immune NoProgressTimeout; SessionConfig.FirstTokenTimeout
(default 600s) is the existing inter-delta budget. The new decorator is a parallel, less-capable copy —
the constitution's "reuse before you add" rule. Suggested direction: tune or extend the existing
two-phase watchdog (lower the inter-delta budget), not a parallel guard. At minimum, arm on the first
SUBSTANTIVE update. Test gap: StallBeforeFirstDelta never exercises the keepalive-arms-guard path, so
the invariant is unverified.

F2 — The inactivity timer stays armed across the yield return, so it measures provider silence PLUS
downstream consumer time. If a consumer holds an update longer than the window, the timer fires and
throws a false TimeoutException on a healthy stream. Latent at 45s (current consumers are fast), but
live if the timeout is lowered or a consumer does heavier per-update work. Suggested: disarm after a
successful MoveNextAsync, re-arm before the next one — measure provider silence only.

F3 (discussable) — StreamInactivityTimeout is not representable in the config schema.
Session.Tuning.StreamingRetryPolicy is absent from the schema and Session.Tuning uses
additionalProperties: false, so an operator cannot set the knob — including the "set to zero to
disable" claim. It does NOT break default startup (the subtree is not written by default). Either add
schema coverage for StreamingRetryPolicy or drop the disable-via-config claim.

Verified sound

Partial-output-on-retry (no double emission); decorator ordering; timer lifetime (uses
TimeProvider.CreateTimer, disposed on every path); constitution (no global::, required deps, fails
loud, headers present).

Nits

Stale "Logging -> Retry" comment (DaemonProviderServiceExtensions.cs:75); long non-STE comments.

This PR is a do-not-merge focus PR. Core insight for the upstream decision: Netclaw already owns the
right mechanism (the two-phase ProcessingWatchdog + FirstTokenTimeout); the fix is to tune that
existing budget, not add a parallel guard.

[Aaronontheweb]

Adversarial re-review — non-author agent (rework, commit ced240d)

Verdict: REQUEST CHANGES — but the core mechanism is verified SOLID. Both findings are design/doc,
not runtime correctness bugs.

Verified solid (revert experiments + own gate runs)

• F1 works. A keepalive-only stream that stalls before any substantive content does NOT arm the
  tight timer. The reviewer reverted only the F1 gate; the new test failed with the expected
  TimeoutException; restored, it passed. The decorator calls StreamingResponseReader.IsSubstantiveUpdate
  directly, so the semantics match by construction.
• F2 works, NO unguarded window. The timer re-arms before every provider read and disarms after each
  success, so it measures provider silence only. The pre-first-token window is left to ProcessingWatchdog's
  prefill budget by design. Reverting the disarm line made the slow-consumer test fail (false abort);
  restored, it passed.
• F3 correct end-to-end. StreamingRetryPolicy + StreamInactivityTimeout added with defaults;
  schema -> SessionConfig -> RetryPolicy -> guard binding holds; schema-doctor 24/24.
• Gates (re-run): Daemon StreamStallGuard + Pipeline 12/12; Configuration defaults 8/8; slopwatch 0;
  headers clean. TimeProvider used; no silent fallback; the TimeSpan.Zero disable is explicit.

Findings

F-A [design/coupling] — InternalsVisibleTo netclawd opens the whole Actors internal surface.
src/Netclaw.Actors/Netclaw.Actors.csproj grants EVERY internal type/member of Netclaw.Actors to
production netclawd, only to reuse one predicate. This is the first production->production internal
grant (before, only test/benchmark assemblies held it). A later change to IsSubstantiveUpdate for the
session reader would silently alter the daemon's stall-guard arm condition, with no compiler gate.
Recommendation: narrow the seam — expose only the small classification as a PUBLIC predicate that both
callers use, not blanket internal access.

F-B [doc/behavior mismatch] — the comments deny a real behavior. The comments say a reasoning-only
delta "never arms" the timer, but IsSubstantiveUpdate returns true for a reasoning delta (and empty
reasoning falls through to default: true), so a reasoning-only delta ALWAYS arms it. The behavior is
consistent with ProcessingWatchdog system-wide (not a NEW abort risk), so the defect is the misleading
doc. Fix the comments to state that reasoning deltas arm the timer — or, if the intent is to exclude
reasoning, that needs a design decision (and argues against reusing IsSubstantiveUpdate).

Nits

A 45s-deadline timer TOCTOU (astronomically unlikely, retryable, self-heals); the TimeSpan schema
strings lack a pattern (matches the existing convention).

Decision for this skunkworks focus PR

The core fix is correct and adversarially verified. Both findings are design/doc, not correctness, and
each has more than one valid answer (narrow-the-seam vs duplicate-the-predicate; fix-the-doc vs
change-the-intent). They are documented here for the upstream/OpenSpec decision rather than further
iterated on a do-not-merge PR.

[Aaronontheweb]

Adversarial review follow-up

This commit fixes both findings from the adversarial review.

Finding 1: narrow the InternalsVisibleTo grant

Netclaw.Actors.csproj added InternalsVisibleTo netclawd. This was
the first production-to-production internal grant in the repo. The
grant existed only so StreamStallGuardChatClient could call the
internal StreamingResponseReader.IsSubstantiveUpdate.

Fix: the predicate now lives in a new public class,
ChatStreamUpdateClassifier, in Netclaw.Actors. Both
StreamingResponseReader and StreamStallGuardChatClient call this
one public method. The logic is unchanged — one implementation, two
callers. The InternalsVisibleTo grant to netclawd is removed. The
three grants that pre-date this branch (Netclaw.Actors.Tests,
Netclaw.Daemon.Tests, Netclaw.Benchmarks) stay.

Finding 2: fix the doc/behavior mismatch

Doc comments in StreamStallGuardChatClient.cs and RetryPolicy.cs,
and one line in the PR description, claimed a reasoning-only delta
never arms the tight stall timer. This claim was false. A reasoning
delta with text is substantive, so it arms the timer, the same as a
text or tool-call delta. Only a content-free keepalive leaves the
timer unarmed. This matches ProcessingWatchdog's existing rule
system-wide.

Fix: the comments now state the true rule. The predicate and the
guard's timing do not change — this is a documentation fix only.

Verification

• dotnet build Netclaw.slnx — clean
• dotnet test src/Netclaw.Daemon.Tests — 1024 passed
• dotnet test src/Netclaw.Configuration.Tests — 520 passed
• dotnet test src/Netclaw.Actors.Tests — 3052 passed, 1 pre-existing
  platform skip
• dotnet test src/Netclaw.Cli.Tests --filter ConfigSchemaDoctorCheck
  — 24 passed
• dotnet slopwatch analyze — 0 issues
• Add-FileHeaders.ps1 -Verify — all files have headers

[Aaronontheweb]

The macOS test failure is a pre-existing flake, not this PR

Two specialist analyses (Akka.NET + .NET concurrency, independent verification) root-caused the
Test-macos-26 failure (FailedStdioStartup_IsReportedBeforeLeaseAssertions):

• The test embeds a random GUID in a fake command name; this run's GUID contained the substring
  401 (632401b4..., visible in the job's own log).
• McpClientManager.FindHttpStatus does a bare Contains("401")/Contains("403") over exception
  text that embeds the command name, so the stdio spawn failure was misclassified as HTTP 401 and
  produced a different error string than the test asserts. Probability ~1.4% per run; deterministic
  given the GUID; platform is irrelevant.
• The full publish/read path is sequentially awaited with a volatile pair — no race exists, and this
  PR touches no MCP code.

The test landed in #1904 (2026-08-12) and flaked the same day. This is also a small production
diagnostics bug (a server command path that contains 401/403 misreports as an HTTP failure in
mcp list/doctor). A separate fix PR follows. The failed job has been re-run.

[Aaronontheweb]

This fix is stupid and is going to create problems with self-hosters who have a long TFT window. Closing it.