feat(sessions): resume the turn after a mid-stream LLM timeout

[Aaronontheweb]

Problem

A dead or half-open LLM stream ends the whole turn today. In headless
mode, a failed turn is a failed session. No external process retries
a failed session.

A benchmark run exposed the size of this gap. The run had 13 turn
failures. Correlated provider stall storms caused 9 of them. Each
stall showed the same pattern.

The model sent a few tokens. Then it went silent. The watchdog
waited the full inter-delta budget, FirstTokenTimeout, 600 seconds
by default. Then the turn failed. Each stall burned close to 600
seconds this way.

PR #1888 makes this wait shorter. It does not change the outcome.
See Relationship below.

Mechanism

LlmSessionActor now has a method: TryResumeAfterTimeout. This
method runs on two paths: a watchdog expiry for the LLM call, and a
call failure whose cause is a timeout.

On a timeout, the actor takes these steps, in order:

1. It checks for a restart drain. A coordinated daemon restart, in
   progress, makes the actor fail the turn at once. This matches
   every other mid-turn continuation under a restart drain.
2. It checks the retry budget. A spent budget makes the actor fail
   the turn.
3. It sends a TextStreamDiscarded output. See Consumer correctness
   below.
4. It discards the dead call. This step needs no rollback code. A
   call that never completes never writes to session state.
5. It reissues the same call, with the same message list and the
   same tool-use flag. The user message stays the same. The session
   history stays the same.
6. It logs a warning. The warning states the attempt number, the
   budget, and the estimated input size of the discarded call.

The actor resumes safely because of its own structure. It needs no
separate gate for this safety. The actor dispatches a tool call only
from a fully completed response. A call that times out mid-stream
never reaches that point, on any iteration.

So a timed-out call can never dispatch a tool call in the same turn.
The actor can safely resume on any call in the turn. This covers a
call after an earlier tool round completed, too. All 9 stalls behind
this change follow this pattern.

A discarded call still costs input tokens. The provider never
returns usage for a call that fails before it completes. The actor
now reports this fact honestly. UsageOutput carries two new
fields: DiscardedResumeEstimatedInputTokens and
DiscardedResumeAttempts. Both fields stay null when the actor does
not resume in the turn. The estimate reuses the real input count
from the last completed call, an honest proxy, not a fabricated
guess.

Consumer correctness

The dead call sent zero or more delta events before it went silent.
The resumed call sends its full answer again, from the start. A
subscriber that joins delta text for the turn would glue the dead
call's partial text onto the resumed call's answer.

Three consumers join delta text this way:

• the headless JSON envelope (HeadlessChannel)
• channel and webhook delivery (ExecutionOutputAccumulator)
• the chat TUI (ChatPage)

The fix adds a new output: TextStreamDiscarded. The system always
delivers this output. A subscriber's output filter cannot block it.

HeadlessChannel and ExecutionOutputAccumulator now track a
committed-length marker. The marker records how much of the buffer
holds text from a call that already reached its own TextOutput
this turn. TextStreamDiscarded trims the buffer back to that
marker. Only the dead call's own, unsent text gets removed. An
earlier, completed call's text stays intact.

TextOutput gained a new field: IsCallBoundary. This field marks
the point where one call's own text output is complete. A subscriber
uses this point to move its commit marker forward.

One exception applies. The approval-prompt-expired notice, and its
sibling notices, also send as a TextOutput. These notices set
IsCallBoundary to false. A notice like this can fire mid-turn,
while an unrelated call still runs. It must not move a subscriber's
commit marker over live, unsent text.

Config

A new config value bounds the retry count:
Session.Tuning.TimeoutResumeRetryBudget. Its default value is 2.

When the budget runs out, the turn fails. It fails the same way it
failed before this change. There is no infinite loop. There is no
silent fallback.

The schema file, netclaw-config.v1.schema.json, has the new entry.
The entry has a default value and a description.

Relationship

PR #1888 adds a fast, 45-second stall detector at the provider seam.
That change shortens the wait before the session catches a stall. It
does not change the outcome. The turn still fails. This PR adds the
recovery half. Detection and recovery now work as one pair.

Both PRs add a sibling entry to the same Session.Tuning schema
object. #1888 adds StreamingRetryPolicy. This PR adds
TimeoutResumeRetryBudget. Whichever PR merges second will hit a
small, one-line rebase conflict there. The fix is trivial: keep both
entries.

Tests

New tests prove each behavior with a fake stream, not a real
provider:

• Multi-delta discard: a resumed call, after several dead-call
  deltas, keeps the completed calls' text. It removes only the dead
  call's own partial text.
• Watchdog budget on a resumed call: a resumed call that already
  sent real content stays on the tighter, promoted budget through a
  later keepalive. A resumed call that died during prefill, with no
  content, keeps the full prefill budget.
• Silent resumed call: a resumed call that sends no further
  deltas still reports the full answer. It does not report an empty
  one.
• Retry budget exhaustion: once the retry budget runs out, the
  turn fails exactly as it did before this change.
• Restart drain: a timeout during a coordinated restart drain
  fails the turn. The actor does not resume.
• Discarded-usage honesty: the reported estimate matches the
  real input count from the last completed call. The estimate
  reports null when no call completed yet this session.

Validation

All four required suites pass on the rebased branch (upstream/dev
at 19ed641e, includes #1890 and #1896):

• Netclaw.Actors.Tests: 3163 passed, 1 prior Windows-only
  skip, 0 failed.
• Netclaw.Daemon.Tests: 1020 passed, 0 failed.
• Netclaw.Configuration.Tests: 597 passed, 0 failed.
• Netclaw.Cli.Tests: 1367 passed, 0 failed.
• dotnet build Netclaw.slnx -c Release: succeeds. This build covers
  the full solution, benchmarks included.
• dotnet slopwatch analyze -d .: 0 new issues. One prior issue
  remains in an untouched file (PowerShellHostProbeTests.cs, from
  PR fix(shell): harden PowerShell host probe against cold-start timeouts #1859), outside this PR's diff.
• Add-FileHeaders.ps1 -Verify: all files have headers.

The native tape harness also ran, on this PR's head commit, on a live
host. Per the PR comment: ./scripts/smoke/run-smoke.sh light —
"All smoke checks passed (23 tapes, 9 scenarios)". This result
closes the one open item from the review record below.

Review record

This PR went through four non-author review passes before this
rewrite:

1. A full adversarial review found 2 critical, 1 high, and 2 medium
   defects. The next revision fixed all five.
2. A second full adversarial review found 2 new high-severity
   data-loss paths. It also found one prior fix, only half done, and
   one performance defect. The next revision fixed all four.
3. A focused verify pass confirmed the four fixes held. It then
   found 3 narrower issues. The next revision fixed all three.
4. A targeted re-verify confirmed all three fixes. It found no new
   defect.

Every issue, across all four passes, got fixed on the branch. The
full trail sits in the PR comments.

Scope

Sub-agent calls (SubAgentActor) keep their own watchdog logic.
This PR does not touch that path.

[Aaronontheweb]

Adversarial review — non-author agent

Verdict: REQUEST CHANGES. Two critical defects; one high; two medium. Gates all green (reviewer
re-ran them). The stale-message races, budget termination, startup-context rollback, and schema were
verified CORRECT.

C1 — The discarded partial text reaches the user's answer (silent corruption)

The dead call's deltas were already emitted as TextDeltaOutput. The resume emits a second full delta
stream. Every delta-accumulating consumer concatenates BOTH: HeadlessChannel._responseBuffer (the
headless JSON envelope — the motivating mode), ExecutionOutputAccumulator (channels/webhooks/jobs),
and the TUI ChatPage. Proven by replaying real emitted outputs through the accumulator:
"The answer is STALLED_PARTIAL_MARKERResumed answer after timeout." The PR's test misses this because
the single-update fake never emits a delta before the buffered-first-delta trick, and it asserts on
TextOutput, which these consumers discard once a delta arrived. Fix: emit a stream-discard output
that tells accumulators to drop the buffer before resume, or hold deltas until call completion.

C2 — The safety gate is per-TURN, so it refuses almost every real stall

if (_turnState.ToolIterationCount > 0) — the counter resets only at turn end. All 9 motivating
stalls happened after earlier tool iterations; the gate refuses every one. The fix as built recovers
approximately none of its target class. The reviewer verified a per-call gate is safe BY CONSTRUCTION:
tool dispatch happens only on a fully-completed response; the delta handler never dispatches; all
tools run locally; the watchdog stops before dispatch, so no stale expiry can fire post-dispatch.
Fix: DELETE the ToolIterationCount check (the invariant is structural); keep the retry budget as the
only bound; invert Timeout_after_tool_call_dispatched_does_not_resume.

H3 — Resume restarts on PrefillTimeout and ignores the restart drain

Each resume resets _anyContentStreamed, so attempt N waits PrefillTimeout (1800s), not the
promoted 600s — a stalled turn now fails after ~30 min instead of ~10, and TryResumeAfterTimeout
does not check _restartDrainRequested (every other mid-turn continuation does). Fix: refuse resume
during a restart drain; consider an overall turn deadline or arm resumed calls on the promoted budget.

M4 — The new tests fail by HANGING, not asserting

WaitForStreamInvocationAsync has no timeout; with resume disabled the test hangs the runner (blame
dump). Add an explicit timeout.

M5 — Token/cost accounting drops the dead call

The provider bills the discarded call; the session counts nothing. Up to 3 billed input contexts
reported as 1. Count the dead call's usage.

Nits

No file-to-runtime round-trip test for the new knob (Cross-Boundary Contract Rule); message-list
assertion compares Role+Text only; STE drift in the new XML doc.

This is a do-not-merge focus PR; findings will be fixed on the branch and re-reviewed.

[Aaronontheweb]

This revision fixes all five findings from the adversarial review.

• C1 (critical): the dead call's partial deltas corrupted the
  final answer in every delta-accumulating consumer. Fixed with a
  new TextStreamDiscarded output — HeadlessChannel,
  ExecutionOutputAccumulator (webhooks/reminders), and ChatPage
  all clear buffered text on receipt. Proven by a real multi-delta
  test that asserts on the delta-accumulated result.
• C2 (critical): the ToolIterationCount gate refused resume for
  a call after any completed tool iteration — the dominant real-world
  case. Deleted. Safety is structural: tool dispatch only happens on
  a fully completed response, so a call that times out mid-stream can
  never have dispatched one. The test that checked the old (wrong)
  behavior is inverted to prove resume now works.
• H3: resume now refuses when a coordinated daemon restart is
  draining the session, and a resumed call arms the watchdog on the
  promoted budget instead of the full prefill budget, so the retry
  budget can no longer triple the time to a final failure.
• M4: the test helper that waits for the next streaming
  invocation now has a bounded timeout, so a regression fails the
  test instead of hanging it.
• M5: a discarded call's estimated input tokens and the resume
  attempt count are now reported on UsageOutput, as separate,
  clearly labeled fields — never blended into the real provider
  totals.

Full detail on each fix, including why the gate deletion is safe, is
in the PR description above. All touched suites are green
(Actors, Daemon, Configuration, Cli), slopwatch reports 0 issues,
and file headers are clean.

[Aaronontheweb]

Adversarial re-review (rework commit 32b72af) — non-author agent

Verdict: REQUEST CHANGES. Two new data-loss paths on the exact surface the PR targets; one prior
finding (H3) only half fixed; one hot-path performance defect. C2 (gate deletion) attacked and HELD.
All gates green (reviewer re-ran; numbers match the PR body).

D1 (High) — Over-discard deletes text from earlier COMPLETED calls in the turn

The headless and accumulator buffers are TURN-scoped; TextStreamDiscarded carries no call scope, so
Clear() wipes legitimate preamble from completed calls. Probe: expected "Checking the files now.
Done: the answer is X." — actual "Done: the answer is X.". The loss is permanent (call 1's message is
already in history; the model does not repeat it). ChatPage and Slack already do the correct
call-scoped thing — the accumulating consumers are the outliers. Fix direction: call/segment-scoped
discard (mirror ChatPage's segment semantics; consider a CallId on the discard output — N1).

D2 (High) — Headless envelope goes EMPTY when the resumed call does not stream deltas

The discard arm clears _responseBuffer but leaves _receivedTextDeltaInCurrentTurn true; a
single-chunk resumed response emits no deltas, and the TextOutput arm skips the append. Probe:
expected "Resumed answer", actual "". A successful turn reports an empty response — worse than the
pre-PR loud failure; a no-silent-fallback violation. The author fixed this exact bug in the
accumulator (_sawTextDelta = false) but did not port the one-line fix + test to HeadlessChannel.

D3 (Medium) — H3 arming dies on the first keepalive

FireLlmCall still resets _anyContentStreamed, so the first content-free keepalive re-arms the
liveness timer at PrefillTimeout (proven with a virtual-clock probe). A keepalive-emitting stall is
bounded by NoProgressTimeout (~50 min worst case), not the claimed promoted budget. Also: when the
dead call died during prefill with zero deltas, the promoted budget rests on no evidence and can kill
a genuinely slow failover prefill early. One fix for both: carry the dead call's
_anyContentStreamed forward instead of the one-shot override.

D4 (Medium) — EstimateInputTokens runs on every ContinueFireLlmCall

Only the resume path reads it, but it walks and re-stringifies the whole message list (JSON-serializes
every FunctionCallContent, ToString()s every result) on every tool iteration — quadratic in turn
length, on the actor thread. "Reuse before you add": _lastInputTokenCount already carries the
provider's REAL input count through this seam.

Nits

N1 discard lacks CallId (root of D1); N2 old-CLI unknown-type banner (pre-existing class); N3 the
USAGE: log line appends empty discard fields on every turn; N4 interrupt banner on stdout (non-JSON
path) — flagged for a conscious call; N5 STE drift (4 sites); N6 budget has no schema maximum.

Attacked and HELD

C2 gate deletion is SOUND on this commit (stale-CallId return before dispatch; watchdog stopped before
dispatch; both resume triggers cannot double-resume in either interleaving; no repeatable side effect
found). Tests are real (reviewer reverted each fix; matching tests failed). ErrorCorrelationTests
budget=0 edit legitimate. M5 usage honesty verified end-to-end. Startup-context rollback correct
across repeated resumes. DTO round-trip covered; outputs never persisted.

Open Definition-of-Done item

The ChatPage change is a Termina surface — the native tape harness (run-smoke.sh) has not run (needs
a live host). Flagged for pre-cherry-pick.

Findings will be fixed on the branch and re-verified. This remains a do-not-merge focus PR.

[Aaronontheweb]

This revision fixes all four findings from the second adversarial review.

• D1 (High): HeadlessChannel and ExecutionOutputAccumulator
  cleared the whole turn-scoped text buffer on TextStreamDiscarded,
  wiping an earlier completed call's already-delivered text along
  with the dying call's partial. Both consumers now track a
  committed-length marker and truncate back to it, matching
  ChatPage's call-scoped segment semantics. TextOutput already
  marks a call boundary, so the fix adds no CallId to the protocol.
• D2 (High): HeadlessChannel left its "saw a delta" flag
  turn-scoped, so a resumed call with zero deltas reported an empty
  JSON envelope for a successful turn. The flag now resets at every
  call boundary — the same fix that closes D1.
• D3 (Medium): a resumed call's watchdog arm reverted to the full
  prefill budget on the first content-free keepalive, and forced the
  promoted budget onto a call that died during prefill with zero
  content. FireLlmCall now carries the dead call's own
  _anyContentStreamed value forward instead of resetting it.
• D4 (Medium): EstimateInputTokens re-stringified the full
  message list on every ContinueFireLlmCall. The fix deletes that
  method and reuses _lastInputTokenCount, the provider's real input
  count already tracked for the compaction trigger. The estimate
  reports null, not a fabricated number, when no call in the
  session has reported real usage yet.

Each fix has a test that fails without it (reversion-tested by hand
against this commit). All touched suites are green: Netclaw.Actors.Tests
(3065 passed), Netclaw.Daemon.Tests (1016 passed), Netclaw.Cli.Tests
(1326 passed), Netclaw.Configuration.Tests (521 passed). Slopwatch
reports 0 new issues. File headers are clean.

The Termina tape harness (run-smoke.sh) still has not run — it needs
a live host, unavailable in this sandbox. This stays an open
pre-cherry-pick item.

[Aaronontheweb]

Focused verify pass (commit 3145635) — non-author agent

Verdict: REQUEST CHANGES — but all four D-fixes HOLD (each reversion-tested with the documented
symptom). Remaining findings are narrow:

• F1 (medium, regression): the [usage] console line lost its leading newline — the guard was
  re-pointed to the call-scoped flag, and the turn-final TextOutput always clears it first. The
  documented ^[usage] anchor contract is broken (evals survive; readability + contract). Fix: a
  separate turn-scoped flag for the newline guard only.
• F2 (low, narrow race): three approval-notice emitters (expired prompt / wrong requester /
  unavailable option) send TextOutput mid-stream, violating the new "TextOutput = call boundary"
  contract — a stale approval click during a live stream can advance the committed marker over
  in-flight text, letting a later discard remove nothing (glued answers). Reproduced with a probe.
  Fix: those notices must not advance the committed marker.
• F3 (nit): discarded_est_in= renders empty instead of being omitted when the estimate is null.
• F4/F5 (informational, accepted): the prefill-budget carry-forward lengthens the fully-silent-provider
  worst case (~60 min, bounded by NoProgressTimeout x budget) — a stated tradeoff; the USAGE debug-log
  line now shows discard fields only after a resume.

Verified holding: the committed-length keystone (no double-commit; delta-seen branch skips the append;
tool-only calls consistent; reasoning never enters the buffers; _committedLength <= _buffer.Length on
every path); all three consumers; D3's two-sided budget behavior; D4's honest-null end to end. Bonus:
D2 also fixed a PRE-EXISTING no-resume bug (a multi-call turn whose last call streamed no deltas
dropped that call's text from the envelope and the accumulator).

Gates (reviewer-run): Actors 3065, Cli 1326, Daemon 1016, Configuration 521 — 0 failures; slopwatch 0;
headers clean.

F1-F3 will be fixed on the branch; F4/F5 accepted as documented. Do-not-merge focus PR.

[Aaronontheweb]

Fixed the three verify-pass findings from the last review round.

• F1 (regression): the [usage] console line lost its leading newline
  after a streamed turn. Added a turn-scoped flag for the newline
  guard, separate from the call-scoped flag that drives the buffer
  and commit logic.
• F2 (race): the three approval notices (expired prompt, wrong
  requester, unavailable option) send TextOutput mid-stream and could
  glue a dead call's partial text onto a resumed answer. Added
  IsCallBoundary to TextOutput (default true). The notices set it
  false. HeadlessChannel, ExecutionOutputAccumulator, and ChatPage
  move their call-boundary marker only when IsCallBoundary is true.
  The DTO and mapper carry the new field.
• F3 (nit): discarded_est_in= printed as empty when no completed
  call had a real estimate. The console line now omits the token
  in that case and keeps discarded_attempts=.

Each fix has a test that fails without it. Touched suites are green:
Netclaw.Actors.Tests (3066 passed, 1 skipped), Netclaw.Cli.Tests
(1333 passed), Netclaw.Daemon.Tests (1016 passed),
Netclaw.Configuration.Tests (521 passed). Slopwatch reports 0 new
issues. File headers are clean.

New commit: 64a8701

[Aaronontheweb]

Targeted re-verify (commit 64a8701) — same non-author verifier

Verdict: F1, F2, F3 all CONFIRMED FIXED. No new defect.

• F1: the turn-scoped newline flag restores [usage] at start-of-line; the no-text, single-chunk, and
  two-turn shapes show no new blank-line regression. (One cosmetic note on a mixed delta+chunk turn —
  not a defect; the pre-PR baseline dropped that text entirely.)
• F2: IsCallBoundary gates all three marker consumers; the mid-stream-notice probe now yields the
  resumed answer, not the glued pair. Default-true wiring holds (normal call-final TextOutput still
  commits); the DTO maps null->true, so an OLDER payload keeps the old contract. Round-trip tested
  both directions.
• F3: null estimate -> token omitted; real estimate -> printed; attempts kept.

Every new test fails without its own hunk (reversion-tested). Gates: Actors 3066, Cli 1333 — 0
failures; slopwatch 0; headers clean.

Review record for this PR: 2 full adversarial reviews + 1 focused verify + 1 targeted re-verify, all
non-author. All findings fixed. The one OPEN Definition-of-Done item before any cherry-pick: the
Termina tape harness for the ChatPage change (needs a live host). This remains a draft, do-not-merge
focus PR for cherry-pick evaluation.

[Aaronontheweb]

Termina tape harness: PASS

The native smoke harness ran on this PR's head (64a8701) on a local Linux host:
./scripts/smoke/run-smoke.sh light — All smoke checks passed (23 tapes, 9 scenarios).

This closes the one open Definition-of-Done item (the ChatPage change is a Termina surface). Host
recipe notes: vhs 0.11.0 (repo-pinned SHA), ttyd 1.7.3 (1.7.7 is protocol-incompatible with vhs
0.11.0 — typing stalls after one key), VHS_NO_SANDBOX=true (hosts with
apparmor_restrict_unprivileged_userns=1 abort Chromium's zygote init), dockerized ollama behind a
CLI shim.

[Aaronontheweb]

Fable went a bit nuts with the comments here - going to review this now.

This PR was generated while running Netclaw through Terminal Bench 2.1 and we kept failing some tasks due to the Deepseek Platform API having network connectivity problems. We'd already had an earlier pull request where we tried implementing essentially the same fix to use the SocketHttpHandler to detect immediate network disruptions and trigger the retry-reconnect loop.