feat(runner): detect stalled agent runs via event inactivity - #6595
feat(runner): detect stalled agent runs via event inactivity#6595guyoron1 wants to merge 12 commits into
Conversation
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
1 similar comment
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
Site previewPreview: https://0ddee7a2-site.fullsend-ai.workers.dev Commit: |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
ffe01bf to
0339139
Compare
PR Summary by QodoDetect stalled agent runs from event-stream inactivity
AI Description
Diagram
High-Level Assessment
Files changed (10)
|
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| // StallTimeout terminates the run when the event stream stays silent for | ||
| // this long. Timeout is wall-clock and cannot tell a wedged agent from a | ||
| // thinking one, so without this a wedge is billed for the full window. | ||
| // Zero disables the watchdog. The CLI resolves it from |
There was a problem hiding this comment.
1. Runtime guide not consulted 📘 Rule violation ⛨ Security
The PR changes the runtime.Runtime execution contract by adding RunParams.StallTimeout and watchdog integration, but neither updates docs/contributing/runtime-implementation.md nor states in the PR description that the guide was consulted. The new event-inactivity and cancellation requirements are therefore undocumented for runtime implementers.
Agent Prompt
## Issue description
The runtime implementation guide does not document the new stall-watchdog contract, and the PR description does not state that the guide was consulted.
## Issue Context
`RunParams` now carries an event-inactivity timeout, and streaming runtime implementations must reset the watchdog from normalized events, disarm it when the stream ends, and use the `ExecStreamReader` cancellation path. Update the guide accordingly and amend the PR description to explicitly reference `docs/contributing/runtime-implementation.md`.
## Fix Focus Areas
- internal/runtime/runtime.go[49-55]
- internal/runtime/claude.go[110-135]
- internal/runtime/pi_run.go[488-528]
- docs/contributing/runtime-implementation.md[120-131]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
Code review by qodo was updated up to the latest commit 0339139 |
d81237a to
2b1e60b
Compare
|
/review |
PR Reviewer Guide 🔍Warning
Here are some key observations to aid the review process:
|
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 2b1e60b |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 957481a |
|
Any way we can fold this with the heartbeat? Or the other way around: any way to fold the heart beat into this? They serve similar purposes and the watchdog could be reporting "agent working: x seconds since last event" each 30 seconds (as the heartbeat interval is 30 seconds). |
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review sweep — 5 findings posted inline (no approval/request-changes; comment only):
- HIGH
internal/runtime/stall.go:46— the stall kill releases the local openshell client; nothing signals the agent inside the sandbox (verified against OpenShell v0.0.116 source) - HIGH
internal/runtime/pi_run.go:528— liveness counted at the AgentEvent level, so streaming tool output (pitool_execution_update, Claudeusertool_result) reads as silence - HIGH
internal/cli/run_overrides.go:115— default 10m equals Claude Code'sBASH_MAX_TIMEOUT_MSceiling; previously-successful runs now fail; not marked breaking - MEDIUM
docs/runtimes.md:88—FULLSEND_STALL_TIMEOUTdocumented as a CI repo-variable override but not in thesetup-agent-env.shallowlist - MEDIUM
internal/cli/run.go:1852— stall timeout not bounded by the run timeout; a no-op fortimeout_minutes <= 10harnesses (triage, prioritize)
| // agent is alive, so a wedged process is indistinguishable from a thinking | ||
| // one until the global timeout expires — and gets billed for the difference. | ||
| // Every event the stream parser emits is proof of life: note() records it, | ||
| // half a timeout of silence logs a warning, and a full timeout of silence |
There was a problem hiding this comment.
[HIGH] Stall kill releases the local openshell client, but nothing signals the agent inside the sandbox
The design claim here (stall.go:46-48, claude.go:110-112, pi_run.go:488-490, the PR body's "the existing kill path... No second termination mechanism", and docs/cli/run.md:63 "terminates the sandbox command — the same kill the global timeout uses") does not hold against the OpenShell source at the pinned v0.0.116.
cancel() from sandbox.ExecStreamReader (internal/sandbox/sandbox.go:1222-1226) cancels an exec.CommandContext, which SIGKILLs only the local openshell sandbox exec client. Server side, handle_exec_sandbox (crates/openshell-server/src/grpc/sandbox.rs:1189-1256) runs the exec in a detached tokio::spawn; stream_exec_over_relay -> run_exec_with_russh (sandbox.rs:2295) writes with let _ = tx.send(...) and leaves its loop only on ChannelMsg::Close/ExitStatus — the only tx.closed() check in that file is in the watch handler (sandbox.rs:1090), not the exec path. In the supervisor, spawn_pipe_exec (crates/openshell-supervisor-process/src/ssh.rs:1360) hands the std::process::Child to a wait() thread with no kill_on_drop, and channel_close (ssh.rs:534) / SshHandler::drop (ssh.rs:460) abort only main_output_task. The server-side timeout_seconds wrapper does not signal the child either — on expiry it drops the russh future and reports exit 124. So neither cancel path terminates the in-sandbox sh -c claude|pi ...; in both cases the process actually dies only when the deferred sandbox.Delete at internal/cli/run.go:1565-1581 tears the sandbox down.
Consequences for the stall path specifically: after a "stalled" verdict runAgent returns (run.go:2030) and collectOpenshellLogs plus the post-failure workspace download run against a still-live agent that is still writing the workspace, running hooks and spending tokens; under --keep-sandbox (run.go:1569) the agent keeps running in the kept sandbox with nothing left to stop it. Practical exposure in the normal path is bounded to the seconds before teardown, but the documented "no in-sandbox process survives the kill" property is inverted and the docs promise it. The PR body's non-goals (per-dimension timeouts, heartbeat, global timeout) don't defer this.
Suggestion: have the watchdog's kill terminate the process inside the sandbox first, then cancel the client. origin/main already has the primitive: killStrayProcesses/clearStrayProcesses in internal/runtime/stray_processes.go TERM->KILLs the sandbox user's processes via sandbox.Exec while sparing the keep-alive (mind its documented sandboxMu serialization; on main it only runs from ClearIterationArtifacts, i.e. the next iteration, which a stalled run never reaches). Alternatively send TERM to the exec'd process group via a short sandbox.Exec. Then reword stall.go:46-48, claude.go:110-112, pi_run.go:488-490, the PR body and docs/cli/run.md:63 to state what actually happens (client released; in-sandbox process killed by the sweep / torn down with the sandbox) — and note the same caveat applies to the global timeout today.
There was a problem hiding this comment.
You're right that the cancel SIGKILLs only the local exec client — verified. What terminates the agent is the deferred sandbox.Delete registered before the agent loop (run.go:1565-1582), which runs on every return path including the stall one, so teardown is prompt; 815582c documents that chain in stall.go and run.md instead of adding new kill machinery (stray_processes.go doesn't exist on this branch or main). Residual: --keep-sandbox skips Delete by design and leaves the agent running — identical to the global-timeout path today; happy to address that as a follow-up if you want it.
| var lastResult *ResultEvent | ||
| innerHandler := handler | ||
| handler = func(evt AgentEvent) { | ||
| stall.note() |
There was a problem hiding this comment.
[HIGH] Liveness is counted at the AgentEvent level, so actively streaming tool output is treated as silence
stall.note() is called only from the normalized-event handler (here and claude.go:135), not per stream line. On pi, tool_execution_update lines — emitted continuously while a tool streams output — are explicitly discarded by parsePiStream (internal/runtime/pi_progress.go:625: "Lifecycle / intermediate events — no AgentEvent mapping", alongside turn_start/turn_end), and pi's bash tool has no command timeout (docs/contributing/runtime-implementation.md:448). So a code-role run whose test suite streams output for 10+ minutes is killed with "no runtime events" while raw JSON lines are flowing, and the whole run fails (run.go:2030 returns; no retry iteration).
On Claude, parseClaudeStream (internal/runtime/claude_progress.go:149-283) has cases only for system, stream_event, result and assistant — user/tool_result lines produce nothing — and fullsend passes --verbose --output-format stream-json without --include-partial-messages (claude.go:324-325), so the silent window for one tool call is tool runtime + result round-trip + the entire next model turn including thinking.
docs/cli/run.md:63 ("without a single agent event") and the warning text therefore misdescribe these cases: the process is demonstrably alive and the watchdog reports it as wedged.
Suggestion: count liveness at the parser level, not the AgentEvent level: give both parsers a per-line liveness callback (e.g. onLine func() or a LivenessEvent{} AgentEvent the renderer/metrics ignore) invoked for every successfully unmarshalled line — including pi tool_execution_update/turn_* and Claude user tool_result messages — and call stall.note() from it, keeping the semantic events unchanged. Add tests that a stream of pi tool_execution_update lines and Claude user tool_result lines keeps the watchdog quiet, and fix the run.md wording.
There was a problem hiding this comment.
Fixed in 815582c at the root — both stream parsers now invoke a per-line hook after every successful envelope unmarshal (including pi tool_execution_update/turn_* and Claude tool_result lines) and Run passes stall.note, so any well-formed stream line resets the clock. Garbage/blank lines don't count; tests cover both runtimes.
| // a slow clone) is legitimately silent for minutes. The default is | ||
| // deliberately generous; repos that know their event cadence can tune it | ||
| // down with FULLSEND_STALL_TIMEOUT. | ||
| const defaultStallTimeout = 10 * time.Minute |
There was a problem hiding this comment.
[HIGH] Default-on 10m stall timeout equals Claude Code's bash ceiling and kills previously-successful runs; PR is not marked breaking
defaultStallTimeout = 10 * time.Minute is enabled by default, and the justifying comment ("a single long tool call ... is legitimately silent for minutes") never checks how long a tool call may legitimately run. Per the Claude Code environment-variables reference, BASH_MAX_TIMEOUT_MS ("Maximum timeout the model can set for long-running bash commands") defaults to 600000 ms — exactly 10 minutes — and the model routinely requests it for test suites; pi's bash tool has no timeout at all (docs/contributing/runtime-implementation.md:448).
Because tool completion is not counted as liveness (see the pi_run.go:528 thread), the observed silence for such a call is tool runtime + result round-trip + the next assistant turn, which always exceeds the default; the 30s poll cadence adds at most 30s of grace. A healthy run that uses the documented bash ceiling is therefore killed as stalled with the shipped default — the same run that succeeded before this change now fails with ErrStalled.
COMMITS.md ("Breaking changes") lists "Default values change in ways that alter existing behavior" as breaking and AGENTS.md:16 makes a missing ! an important-severity review finding, yet the title is a plain feat(runner) with no BREAKING CHANGE: trailer.
To be clear, the PR is right that there is no --include-partial-messages so events are per assistant message, and that system/api_retry is mapped to RetryEvent (claude_progress.go:161) so API backoff keeps the watchdog fed.
Suggestion: either (a) make tool completion / streaming output count as liveness (the parser-level fix) so the silent window is bounded by tool runtime alone, and document the relationship to BASH_MAX_TIMEOUT_MS in docs/cli/run.md, or (b) keep AgentEvent liveness but choose a default with headroom above the bash ceiling (e.g. 15m) and state the derivation in the comment. In either case mention BASH_MAX_TIMEOUT_MS in the run.md tuning paragraph so a repo that raises it knows to raise the watchdog, and if the default stays at/near 10m mark the PR feat(runner)! with a BREAKING CHANGE: trailer naming FULLSEND_STALL_TIMEOUT=0 as the opt-out.
There was a problem hiding this comment.
Fixed in 815582c — default raised to 15m with the BASH_MAX_TIMEOUT_MS 600000ms ceiling named as the derivation in the comment and docs. With line-level liveness (other thread) streaming tools never look silent; the 15m floor covers genuinely quiet calls above the bash ceiling.
| | Runtime | `--runtime` | `FULLSEND_RUNTIME` | `runtime:` on the agent's `agents:` entry | `runtime:` in `.fullsend/config.yaml` (repo default) | | ||
| | Model | `--model` | `FULLSEND_MODEL` (`FULLSEND_PI_MODEL` is a lower-precedence alias on pi) | `model:` on the agent's `agents:` entry | harness `model:`, then agent frontmatter `model:` | | ||
| | Effort | `--effort` | `FULLSEND_EFFORT` | `effort:` on the agent's `agents:` entry | harness `effort:` | | ||
| | Stall timeout | — | `FULLSEND_STALL_TIMEOUT` (default `10m`, `0` disables) | — | — | |
There was a problem hiding this comment.
[MEDIUM] This row presents FULLSEND_STALL_TIMEOUT as a CI repository variable, but the passthrough allowlist does not include it
This row sits directly above the sentence on line 91, "In CI these are repository variables of the same name, plain or role-prefixed (TRIAGE_FULLSEND_MODEL)". That passthrough is FULLSEND_REPO_VARS: ${{ toJSON(vars) }} in .github/workflows/reusable-dispatch.yml (lines 667/794/929/1209) feeding internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh, whose override_keys allowlist at line 45 is FULLSEND_RUNTIME FULLSEND_MODEL FULLSEND_EFFORT FULLSEND_FALLBACK_MODELS FULLSEND_PI_PROVIDER FULLSEND_PI_MODEL on both this branch and origin/main; only allowlisted keys reach GITHUB_ENV. A repo that sets a FULLSEND_STALL_TIMEOUT or CODE_FULLSEND_STALL_TIMEOUT Actions variable silently keeps the 10m default with no warning.
The script is scaffold-shipped from this repo (internal/scaffold/scaffold.go:125 and vendorcontent.go:136 special-case it; ADR 0035 lists setup-agent-env.sh as "upstream infrastructure ... referenced directly from upstream"), and fullsend-ai/agents has no setup-agent-env.sh under .github/scripts, so the fix is an in-repo edit.
Suggestion: add FULLSEND_STALL_TIMEOUT to override_keys in setup-agent-env.sh:45 (the value regex ^[A-Za-z0-9._/@:,-]+$ already admits Go durations such as 10m, 90s, 0), extend setup-agent-env-test.sh and the key list in internal/scaffold/scaffold_test.go:753 — or, if that is out of scope for this PR, move the row out of the override table / add a sentence that the stall timeout is currently process-env only and not yet a repository-variable override.
There was a problem hiding this comment.
Fixed in 815582c — FULLSEND_STALL_TIMEOUT added to override_keys (role-prefixed variants work via the generic per-key handling); setup-agent-env-test.sh and scaffold_test.go extended.
| timeout = 30 * time.Minute | ||
| } | ||
|
|
||
| stallTimeout, stallErr := resolveStallTimeout(os.Getenv) |
There was a problem hiding this comment.
[MEDIUM] Stall timeout is not bounded by the run's own timeout, so the watchdog is a no-op for harnesses with timeout_minutes <= 10
timeout is derived from h.TimeoutMinutes at 1847-1850 and stallTimeout from resolveStallTimeout here, with no relationship between them. ExecStreamReader wraps the command in context.WithTimeout(ctx, timeout) (sandbox.go:1222), so when stallTimeout >= timeout the global context fires first, the stream ends, stall.stop() runs before Wait (claude.go:162, pi_run.go:554) and the run reports a plain timeout, never ErrStalled — the watchdog can only ever emit its half-way warning.
This is not hypothetical for the fleet: fullsend-ai/agents harness/triage.yaml and harness/prioritize.yaml both set timeout_minutes: 10, equal to the default, so for those two roles the kill can never fire before the global timeout — precisely the "wedged run burns its entire global timeout" case the PR body describes. Nothing in docs/cli/run.md tells a repo that FULLSEND_STALL_TIMEOUT must be strictly shorter than timeout_minutes to have any effect.
Suggestion: because equality is still a no-op (the ctx timer wins the race), a plain min(stallTimeout, timeout) is not enough: clamp the effective stall timeout to a fraction of timeout (e.g. timeout/2) when the configured value is not strictly shorter, or emit a startup StepWarn when stallTimeout >= timeout and document in run.md that the stall timeout must be shorter than timeout_minutes for the kill to ever fire.
There was a problem hiding this comment.
Fixed in 815582c — when the stall timeout is not strictly below the run timeout the watchdog is not armed and a StepInfo says so; no clamping, existing configs change only by the log line. Relationship documented in runtimes.md and run.md.
|
/agentic_review |
| if stallErr != nil { | ||
| printer.StepWarn(fmt.Sprintf("Stall watchdog: %v; using %s", stallErr, stallTimeout)) | ||
| } | ||
| if stallTimeout >= timeout { |
There was a problem hiding this comment.
1. stalltimeout guard lacks tests 📘 Rule violation ▣ Testability
The new branch that disables the watchdog when its timeout meets or exceeds the global run timeout has no corresponding behavioral test. Regressions could silently leave runs unprotected or disable valid watchdog configurations.
Agent Prompt
## Issue description
Add behavioral tests for the new stall-timeout deactivation condition.
## Issue Context
Verify that a stall timeout equal to or greater than the run timeout is disabled, while a smaller positive timeout remains enabled. Assert the resulting runtime parameters or observable behavior rather than only the log text.
## Fix Focus Areas
- internal/cli/run.go[1852-1864]
- internal/cli/run_test.go[1-1]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
There was a problem hiding this comment.
Fixed in 53ae715 — the decision is extracted to effectiveStallTimeout() and TestEffectiveStallTimeout asserts parameter behavior across below/equal/above/zero/default; no log-text assertions.
|
Code review by qodo was updated up to the latest commit 815582c |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 53ae715 |
53ae715 to
f02a4cf
Compare
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit f02a4cf |
|
@rh-hemartin They solve different problems, so I left them separate: the heartbeat ( Swapping the heartbeat's line from "time since start" to "time since last event" (using the watchdog's |
|
I understand the difference, I don't think that with your addition the heartbeat has any value. I would bring to discussion if we want the heartbeat after we merge this stall detection. |
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review sweep — 3 findings (no approval/request-changes; comment only).
[HIGH] Codex runtime is completely excluded from stall-watchdog coverage (internal/runtime/codex_run.go:483, not touched by this PR's diff, so posted here instead of inline)
Verified against PR head f02a4cf: CodexRuntime.Run (codex_run.go:413-537) uses the identical shape as ClaudeRuntime.Run/PiRuntime.Run — it calls sandbox.ExecStreamReader to get stdout/execCmd/cancel, wraps a handler, and drains via parseCodexStream(reader, handler) — but never calls startStallWatchdog and never references params.StallTimeout anywhere in the file (confirmed via grep: only claude.go and pi_run.go call startStallWatchdog(params.StallTimeout, ...)). RunParams' doc comment in runtime.go:54 says "Runtimes that stream no events ignore it", which is inaccurate for codex since it is architecturally a third streaming runtime with the same NDJSON-over-ExecStreamReader shape as pi — a wedged codex run still burns the full global timeout with no ErrStalled signal while claude/pi runs of the same scenario are now caught early.
Suggestion: either wire the same startStallWatchdog/stall.note()/stalledErr() pattern into CodexRuntime.Run (mirroring pi_run.go/claude.go), or explicitly scope codex out in the PR description's non-goals and correct the RunParams/runtime.go doc comment so the gap is documented instead of silently implied not to exist.
Two other findings are posted as inline comments on internal/cli/run.go:2175 and internal/cli/run_overrides.go:168.
| if runErr != nil { | ||
| attachIterationContent("error") | ||
| finalizeAgentSpan(agentSpan, runErr, iteration, exitCode, rt.System(), rt.Name(), &metrics, "") | ||
| if errors.Is(runErr, agentruntime.ErrStalled) { |
There was a problem hiding this comment.
[MEDIUM] run.go's stall-timeout integration wiring has no test coverage
Verified against PR head f02a4cf: grep for ErrStalled/Stalled in internal/cli/*_test.go shows only TestAggregateMetrics_StalledOmittedWhenFalse (a JSON-marshal/omitempty test) and the pure-function tests TestResolveStallTimeout/TestEffectiveStallTimeout in run_overrides_test.go. Nothing exercises the actual wiring inside runAgent itself: resolving FULLSEND_STALL_TIMEOUT via os.Getenv at run.go:2000, the StepInfo "watchdog inactive" branch at 2004-2009, passing StallTimeout into RunParams at 2146, or the errors.Is(runErr, agentruntime.ErrStalled) branch at 2175-2181 that sets aggMetrics.Stalled and prints the stall-specific StepFail message. This is distinct from the already-resolved thread on run.go:1856 (which was about testing the effectiveStallTimeout decision function in isolation, fixed via TestEffectiveStallTimeout) — the integration-level branches in runAgent remain unexercised by any test.
Suggestion: add a fake/dummy runtime that returns agentruntime.ErrStalled from Run() and assert runAgent sets aggMetrics.Stalled and emits the stall-specific message, to get direct coverage of the new branches in run.go rather than relying only on the pure-function unit tests.
| // timeout's context, so a stall timeout at or above it always loses the race | ||
| // to the global deadline. Below it, the configured value stands unchanged — | ||
| // no clamping or deriving, existing configs keep their behavior. | ||
| func effectiveStallTimeout(stall, run time.Duration) time.Duration { |
There was a problem hiding this comment.
[MEDIUM] effectiveStallTimeout's disable check ignores the watchdog's own poll-interval detection latency
Verified against PR head f02a4cf: stall.go's own comment (lines 21-26) states detection lands within roughly 5% of the threshold, capped by a 30s poll interval (stallMaxPoll = 30 * time.Second). effectiveStallTimeout (run_overrides.go:168-173) only disarms when stall >= run, not when stall + (poll latency) >= run. So a configuration where stall is just under run (e.g. stall=14m50s, run=15m) stays "armed" per this check, but the watchdog may not actually detect and fire until up to stallMaxPoll after the stall threshold is crossed — potentially after the global context deadline already won the race. In that near-boundary zone the watchdog appears active but often provides no real protection, which is inconsistent with the PR's own stated detection-latency model.
Suggestion: change the disable condition to account for detection latency, e.g. stall + stallMaxPoll >= run (or an equivalent margin), so "inactive" determination matches the watchdog's actual worst-case detection time rather than the nominal threshold.
f02a4cf to
811acf1
Compare
|
@waynesun09 @rh-hemartin rebased onto main (conflicts:
Deferred: the heartbeat, separate output and PR. |
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review sweep — 2 findings posted inline (docs/cli/run.md, internal/runtime/stall.go); 2 more below have no single diff line to anchor to (no approval/request-changes; comment only).
[MEDIUM] PR description is stale relative to the shipped diff (runtime coverage and default value)
Verified via gh pr view 6595 --json body: the PR description still says the watchdog covers "both streaming runtimes (claude, pi)" and defaults FULLSEND_STALL_TIMEOUT to "10m". Neither matches head 811acf1b: internal/runtime/codex_run.go now calls startStallWatchdog (wired at codex_run.go:493-561, added by commit 909517f), docs/cli/run.md:81 says the watchdog "covers claude, pi and codex — every runtime that streams", stall_test.go:355 hard-requires {claude.go, pi_run.go, codex_run.go} via streamingRuntimeFiles, and defaultStallTimeout in run_overrides.go:142 is 15 * time.Minute, also reflected in the docs. The rebase summary comment on this PR confirms "Codex arms the watchdog... the default stays 15m" but doesn't mention updating the description text, and it is indeed still unchanged.
Suggestion: update the PR description before merge to say the watchdog covers claude, pi, and codex, and that the default is 15m, so PR history/changelog readers don't undercount codex coverage or the default timeout.
[MEDIUM] Pi Agent children can look stalled — child timeout races the watchdog (internal/runtime/pi_extension/fullsend-agent.js:54, not touched by this PR's diff, so posted here instead of inline)
Verified against head 811acf1b: fullsend-agent.js's run() spawns a pi Agent child and awaits it up to timeoutMs = (agent.timeoutSeconds || DEFAULT_TIMEOUT_SECONDS) * 1000 (DEFAULT_TIMEOUT_SECONDS = 900 at line 54; also piAgentTimeoutSeconds = 900 in pi_bootstrap.go:61). The child's stdout is buffered and parsed internally but never written to the tool's own process.stdout, so nothing is emitted to the parent pi process's JSON stream while the child runs. pi_progress.go's own wire-format comment states tool_execution_update is "emitted continuously while a tool streams output" — a tool that doesn't itself stream (like this sub-agent tool, which just awaits a child process) produces no intermediate liveness between tool_execution_start and tool_execution_end. The parent stream — and therefore the stall watchdog's liveness clock — can go quiet for up to the full 900s child deadline, which is numerically equal to defaultStallTimeout (15m) in run_overrides.go:142, so a legitimately-running child agent races the kill (plus up to 30s of watchdog poll latency).
Suggestion: forward child pi-agent NDJSON lines to the parent stream (or otherwise emit periodic liveness from the extension while a child agent runs), or ensure the child budget cannot equal the parent stall timeout, or at minimum document the interaction in docs/cli/run.md so operators enabling pi sub-agents know the watchdog can misfire on long-running personas.
|
|
||
| The global run timeout is wall-clock, so a wedged agent looks exactly like a thinking one until it expires — and the run is billed for the difference. The watchdog watches the runtime output stream instead: every well-formed line the runtime writes counts as liveness, including lines that map to no agent event (pi's `tool_execution_update` while a tool streams output, Claude Code's `user` tool-result messages, codex's `item.started`/`item.updated`), so an actively streaming tool is never mistaken for a stall. It covers claude, pi and codex — every runtime that streams — and the scripted `dummy` runtimes ignore it. After half of `FULLSEND_STALL_TIMEOUT` of stream silence it warns once (`::warning::no agent events for 7m30s` in CI), and after the full duration it kills the run: first the agent inside the sandbox, through the same TERM-then-KILL sweep that clears stray processes between iterations, then the local `openshell sandbox exec` client. Cancelling the exec is all the global timeout does and it signals nothing inside the sandbox, so the sweep is what actually stops a wedged agent from writing the workspace and spending tokens — including under `--keep-sandbox`, where nothing else would. The run then fails with `agent stalled` and records `"stalled": true` in `metrics.json`. | ||
|
|
||
| `FULLSEND_STALL_TIMEOUT` takes a Go duration and defaults to `15m`; `0` disables the watchdog. The default sits above Claude Code's bash ceiling — `BASH_MAX_TIMEOUT_MS` defaults to 600000ms (10 minutes) and the model routinely requests the full ceiling for test suites — so a legitimately quiet long command is not killed as stalled; a repo that raises `BASH_MAX_TIMEOUT_MS` should raise the stall timeout with it. The value must clear the harness `timeout_minutes` by more than the watchdog's polling interval (a twentieth of the stall timeout, capped at 30s): any closer and the global timeout wins the race, so the watchdog is not armed and the run logs that stall protection is inactive. Harnesses with a short `timeout_minutes` — 10 minutes or less — therefore get no stall protection at the default; lower `FULLSEND_STALL_TIMEOUT` for those or accept that the global timeout is the only backstop. A value that is not a duration is reported in the run log and ignored, and the default applies. |
There was a problem hiding this comment.
[HIGH] Understates when the default stall watchdog is inactive
Verified against head 811acf1b (unchanged since): this line says harnesses with timeout_minutes "10 minutes or less" get no stall protection at the default. The actual code (effectiveStallTimeout in run_overrides.go:174: if stall+agentruntime.StallDetectionLatency(stall) >= run) disarms whenever run <= stall(900s) + latency(30s) = 930s = 15.5m. So harnesses with timeout_minutes of 15 or less get no default stall protection, not just 10 or less. This exact arithmetic was reworked by commit d0c05b8 ("fix(cli): size the stall watchdog against its detection latency") to account for poll latency, but this prose was never updated to match — it still cites the old, pre-fix "10 minutes" threshold.
Suggestion: change "10 minutes or less" to "15 minutes or less" (or phrase generically as "not more than the stall timeout plus its ~30s detection interval") to match the current effectiveStallTimeout arithmetic, and check docs/guides/user/bring-your-own-agent.md's timeout_minutes: 15 example isn't presented as a case that gets stall protection.
| // The sweep runs first so the stream reaches EOF on its own; cancel then | ||
| // releases the client whatever the sweep did. | ||
| // | ||
| // Unlike ClearIterationArtifacts' sweep this one is not serialized against |
There was a problem hiding this comment.
[MEDIUM] stallKill bypasses sandboxMu — truncated credential file possible under --keep-sandbox
Verified against head 811acf1b: this comment explicitly states the stray-process sweep stallKill runs is "not serialized against the credential refreshers' writes (internal/cli/run.go's sandboxMu is not reachable from here)", reasoning this is fine because the stalled run "is already tearing down" the sandbox. That reasoning is contradicted by run.go's own sandboxMu doc comment (run.go:4017-4030), which explains the identical TERM/KILL sweep in ClearIterationArtifacts is deliberately serialized against refreshOIDCToken's openshell sandbox upload (a tar xf that truncates its target on open) precisely because an unsynchronized kill mid-write leaves a truncated .gcp-oidc-token — and docs/cli/run.md now explicitly documents that the stallKill sweep runs "including under --keep-sandbox, where nothing else would." Under --keep-sandbox the sandbox persists (its teardown is what this comment relies on to make the race harmless), so a credential refresher killed mid-write by the unsynchronized sweep can leave a truncated credential file in a sandbox the operator is told to go re-enter and continue working in.
Suggestion: either take sandboxMu in stallKill before running the sweep (mirroring ClearIterationArtifacts), or explicitly document the --keep-sandbox caveat in docs/cli/run.md's stall watchdog section so operators know to re-check/re-auth after a stall-killed run with --keep-sandbox.
The heartbeat prints "Agent running (Xs elapsed)" whether or not the agent is alive, so a wedged process is indistinguishable from a thinking one and burns the entire global timeout before anyone learns it was dead. The global timeout is wall-clock; nothing watched the event stream. The runtime now runs a watchdog seated on the normalized event stream: every event is proof of life, half a timeout of silence warns once per stall episode (::warning:: in CI, the printer everywhere), and a full timeout of silence terminates the sandbox command through the cancel ExecStreamReader already returns -- the same kill the global timeout uses, not a second mechanism -- and fails the run with ErrStalled. runAgent maps that sentinel to a specific failure line and records "stalled": true in metrics.json. FULLSEND_STALL_TIMEOUT (Go duration, default 10m, 0 disables) is resolved by the CLI and handed to the runtime in RunParams, since runtimes do not read env themselves (fullsend-ai#6526). 10m rather than a Cloudflare-style 60s because fullsend does not request partial messages: Claude Code's stream-json emits one event per assistant message and per completed tool call, so a single long tool call is legitimately silent for minutes. The watchdog derives no context of its own and never rebinds the caller's ctx -- a body-scope `ctx, cancel := context.WithCancel(ctx)` is what once made every successful run report as "cancelled" -- and a source-reading regression test pins that for both streaming runtimes. Non-goals: per-dimension timeouts (dimensions run inside one CLI process the runner cannot see into), and no change to the heartbeat or the global timeout. Signed-off-by: guy oron <goron@redhat.com>
…c time stop() and the ticker's fire branch both wrote fired/lastEvent from separate atomics with no ordering between them, so a tick already past the select when stop() closed the channel could still set fired=true after disarm — misclassifying a healthy or just-finished run as stalled. Replaced fired+stopOnce with a single state (armed/stopped/ fired) that stop() and the fire branch both reach only via CompareAndSwap from armed, so exactly one wins. Also swapped the UnixNano/time.Unix round-trip for time.Since(start), which keeps Go's monotonic clock reading instead of discarding it, so a wall-clock step can no longer perturb the silence calculation. Signed-off-by: guy oron <goron@redhat.com>
The select in watch() can draw an already-queued tick even when the stopped channel is closed, and only the kill path was protected by the state CAS — so a run that had just completed normally could still emit a misleading inactivity warning or CI annotation. A bare state check before warning would shrink the window but keep a check-then-warn race, so stop()'s disarm and the warning's check+emit now serialize on a mutex: once stop() returns, no warning can follow. Signed-off-by: guy oron <goron@redhat.com>
- re-check lastEvent under mu before the half-timeout warning, so an event arriving after the tick computed its silence suppresses the stale warning - count liveness per well-formed stream line (parseClaudeStreamLines / parsePiStreamLines feed stall.note), so streaming tool output and lifecycle lines with no AgentEvent mapping keep the watchdog quiet - raise the default stall timeout to 15m: Claude Code's bash ceiling (BASH_MAX_TIMEOUT_MS, 600000ms) makes a 10m tool call legitimate, so the default needs headroom above it - allowlist FULLSEND_STALL_TIMEOUT in setup-agent-env.sh so the documented CI repository variable actually reaches the runner - skip arming the watchdog (with a log line) when the stall timeout is not below the run timeout, where the global deadline always fires first and the watchdog could never act - document the real kill chain: the cancel kills the local openshell exec client; the in-sandbox agent dies when the deferred sandbox teardown deletes the sandbox, and survives under --keep-sandbox Signed-off-by: guy oron <goron@redhat.com>
- count a fully consumed oversized stream line (> streamBufSize) as liveness in both parsers: it is excluded from semantic parsing, but a runtime writing megabytes is alive and must not be killed as stalled - extract the stall-vs-run-timeout disable decision into effectiveStallTimeout and cover it with a behavioral test (disabled at or above the run timeout, untouched below, 0 stays 0) - validate FULLSEND_STALL_TIMEOUT repository variables with a duration-shaped pattern in setup-agent-env.sh: the shared charset rejected valid Go durations (+5m, 1µs, 1μs), silently dropping the override; µ/μ are matched as literal alternations so the check stays byte-safe in any locale, and injection protection is preserved Signed-off-by: guy oron <goron@redhat.com>
- the stall kill only released the local `openshell sandbox exec` client, which is all the global timeout does and signals nothing inside the sandbox: the wedged agent kept writing the workspace and spending tokens until teardown, and indefinitely under --keep-sandbox. stallKill now runs the stray-process sweep over a second exec channel first (OpenShell exposes no signal API), then cancels; a failed sweep is reported and still releases the client - arm the watchdog in CodexRuntime.Run: it has the same ExecStreamReader -> handler -> parse shape as the other two streaming runtimes but never read params.StallTimeout, so a wedged codex run burned its whole global timeout with no ErrStalled. parseCodexStreamLines gives it the same per-line liveness hook, so item.started/item.updated progress lines are not mistaken for silence - derive the file list in the wiring tests from the ExecStreamReader call sites, so a fourth streaming runtime cannot ship unguarded the way codex did - export StallDetectionLatency (the poll interval watch() ticks at) as the one source of truth for how late the kill can land - correct the RunParams doc comment: it said runtimes that stream no events ignore StallTimeout, which read as "no streaming runtime is missing" Signed-off-by: guy oron <goron@redhat.com>
- effectiveStallTimeout compared the threshold, not the kill: the watchdog polls, so a stall just under the run timeout (14m50s against 15m) was reported as armed while the global deadline usually won the race. Compare against stall + StallDetectionLatency instead, and name the interval in the "inactive" line - cover the run.go wiring: fold the resolve/warn/disarm decision into runStallTimeout and the ErrStalled verdict into noteStalledRun, both exercised directly — the branches sat inside runAgent, which no test reaches past sandbox creation - test the case the fleet actually hits: harnesses with timeout_minutes: 10 get no watchdog at the 15m default Signed-off-by: guy oron <goron@redhat.com>
The kill was documented as leaving the in-sandbox agent running until teardown; it now terminates it. Name codex as a covered runtime, and say that the stall timeout must clear timeout_minutes by more than the poll interval — so a harness at 10 minutes or less has no stall protection at the default. Signed-off-by: guy oron <goron@redhat.com>
The unit tests cover the watchdog, stallKill and the line hooks in isolation; nothing showed they compose. Drive the real ClaudeRuntime.Run against a fake openshell on PATH -- the stub shape claude_test.go already uses -- and assert that a stream which goes quiet ends with the sandbox swept exactly once and ErrStalled returned. Deleting the sweep from stallKill, or the stalledErr check from claude.go, fails it. TestStreamingRuntimesArmTheWatchdog stays, now labelled for what it is: a source-shape guard covering all three runtimes at once, not a behavioural test. Its cost is a rename or a reflowed call breaking it; what it buys is catching a fourth streaming runtime shipped unguarded, which is how codex shipped unguarded here. Also: the malformed-value warning goes through the printer, which run.go points at stdout, so run.md saying "reported on stderr" was wrong. Signed-off-by: guy oron <goron@redhat.com>
effectiveStallTimeout disarms when timeout_minutes is not above the stall timeout plus its detection interval: at the default that is 15m + 30s, so harnesses at 15 minutes or less get no stall protection, not 10 or less as the prose still said from before the interval was accounted for. Signed-off-by: guy oron <goron@redhat.com>
…ntials stallKill's sweep is not serialized against the credential refreshers, and the comment justified that with the sandbox being torn down — which is not true under --keep-sandbox. State the actual reason: a kill path must not wait on a lock a wedged upload may hold, the only file it can truncate is the OIDC token, and that token is dead within five minutes of the run failing regardless. docs/cli/run.md now tells operators to re-seed credentials before reusing a stall-killed sandbox they kept. Signed-off-by: guy oron <goron@redhat.com>
The Agent tool consumes its child's stdout instead of forwarding it, so the parent pi's JSON stream is silent for the whole child — up to the child's timeoutSeconds (900s), which is exactly the runner's default FULLSEND_STALL_TIMEOUT. A legitimately long persona therefore raced the stall kill, and lost by the watchdog's poll latency. run() now emits a progress update through pi's onUpdate every 30s while a child is running; each one is a tool_execution_update line on the parent's stream, which the watchdog already counts as liveness. The interval is unref'd and cleared when the child is gone, and a child that is itself wedged stays bounded by its own timeout, as before. Signed-off-by: guy oron <goron@redhat.com>
811acf1 to
3c43f5f
Compare
|
@waynesun09 @rh-hemartin rebased onto main (a4b5c48), force-pushed, description updated below.
@rh-hemartin agreed — let's discuss dropping the heartbeat once this lands. |
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review sweep — 5 findings posted inline (no approval/request-changes; comment only). Verified against head 3c43f5ff2.
- MEDIUM
internal/runtime/codex_progress.go:380— codexitem.updatedis never emitted while a command streams, so the documented codex liveness source does not exist - MEDIUM
internal/cli/run.go:2308— A stall kill skips output-file, transcript and debug-log extraction that the global-timeout path preserves - MEDIUM
internal/runtime/pi_run.go:870— A stalled pi run drops completed sub-agent token/cost usage from metrics.json - MEDIUM
internal/runtime/stall.go:98— stall.go and run.md still claim the global timeout does nothing inside the sandbox; the #7042 sweep contradicts that, as does this PR's own contributing doc - MEDIUM
internal/runtime/pi_extension/fullsend-agent.js:64— No floor ties FULLSEND_STALL_TIMEOUT to the sub-agent extension's fixed 30s liveness tick
|
|
||
| // parseCodexStreamLines is parseCodexStream with a per-line liveness hook: | ||
| // onLine (nil ok) is called for every well-formed JSON line — including the | ||
| // item and lifecycle types that map to no AgentEvent, `item.updated` among |
There was a problem hiding this comment.
[MEDIUM] codex item.updated is never emitted while a command streams, so the documented codex liveness source does not exist
The comment here (and internal/runtime/codex_run.go:542, docs/cli/run.md:79) states that item.updated is "emitted while a command streams output" and therefore feeds the stall watchdog. Verified against the pinned codex (images/sandbox/Containerfile:227 = ARG CODEX_VERSION=0.152.1): in openai/codex at tag rust-v0.152.1, codex-rs/exec/src/event_processor_with_jsonl_output.rs pushes ThreadEvent::ItemUpdated in exactly one place (line 566), inside the ServerNotification::TurnPlanUpdated arm — plan/todo-list updates only. The PR's own live-capture note contradicts line 380 without needing the external source: codex_progress.go:351-355 says "command_execution and file_change both arrive started-then-completed with the same id" — i.e. nothing at all is written between item.started and item.completed for a command. So for codex a long-running command emits no stream lines, and stall.note is never called for its duration. What actually bounds codex silence is the unified-exec tool returning control to the model within 30s (MAX_YIELD_TIME_MS = 30_000 in codex-rs/core/src/unified_exec/mod.rs:77), and unified_exec is Stage::Stable, default_enabled: true at 0.152.1 (codex-rs/features/src/lib.rs:909-911). grep -rn unified_exec internal/ docs/ returns nothing: fullsend neither writes that feature into the codex config.toml it uploads (codex_bootstrap.go:150) nor names the assumption anywhere, so codex stall safety rests on an upstream default the repo does not pin, document, or re-check.
Suggestion: Correct the three passages: drop item.updated as a command-streaming liveness source (it is a plan/todo event), and state that codex liveness during a long command comes from unified-exec yields — repeated tool calls at most ~30s apart, default-on at 0.152.1. Add that assumption to the existing "Re-check on a CODEX_VERSION bump" section (docs/contributing/runtime-implementation.md:1323), or set the feature explicitly in the written config.toml, so a future pin that flips the default does not turn every long codex command into a false stall.
| if runErr != nil { | ||
| attachIterationContent("error") | ||
| finalizeAgentSpan(agentSpan, runErr, iteration, exitCode, rt.System(), rt.Name(), &metrics, "") | ||
| noteStalledRun(runErr, stallTimeout, &aggMetrics, printer) |
There was a problem hiding this comment.
[MEDIUM] A stall kill skips output-file, transcript and debug-log extraction that the global-timeout path preserves
noteStalledRun at run.go:2308 sits inside the if runErr != nil branch, which returns at run.go:2319 — before step "9b. Extract output files" (2382), "9c. Extract transcripts" (2398) and the tx.ExtractDebugLog block (2409-2417). ErrStalled reaches that branch: claude.go:207-209, pi_run.go:870-872 and codex_run.go all return exitCode, stallErr from stall.stalledErr(). On the global-timeout path the same runtimes fall through to return exitCode, nil (the only post-Wait error return is the ProcessState == nil guard), so runErr == nil, the loop reaches lastIterTimedOut at 2364, sweeps the sandbox under withSandboxLock (2365-2381), and then extracts. There is no deferred extraction anywhere in run.go to compensate (grep -n 'defer.*Extract' internal/cli/run.go is empty). So the exact scenario this PR targets — an agent that wedges after doing useful work — now loses whatever it already wrote under /sandbox/workspace/output, its transcripts, and its debug log, all of which the slower timeout path salvaged; the sandbox is then torn down by the deferred sandbox.Delete. TestClaudeRuntime_Run_StallSweepsTheSandboxOnce stops at the runtime boundary and cannot see this.
Suggestion: Treat a stalled iteration like a timed-out one: record the stall verdict and lastExitCode, then fall through to 9b/9c extraction with finish_reason=error before returning the ErrStalled failure — the artifacts are the only evidence of what the wedged agent was doing. If dropping them on a stall is deliberate, say so in docs/cli/run.md next to the "stalled": true row so operators do not go looking for transcripts that were never pulled.
| exitCode = execCmd.ProcessState.ExitCode() | ||
| } | ||
| // A stall is the cause of whatever Wait reports, so it is checked first. | ||
| if stallErr := stall.stalledErr(); stallErr != nil { |
There was a problem hiding this comment.
[MEDIUM] A stalled pi run drops completed sub-agent token/cost usage from metrics.json
if stallErr := stall.stalledErr(); stallErr != nil { return exitCode, stallErr } at pi_run.go:870 returns before the if m.Agent != nil && m.Agent.Enabled block at pi_run.go:892-905, which reads the extension's sub-agent usage file and calls foldPiSubagentUsage to fold children's tokens and cost into metrics. That block's own comment states the reason it cannot be skipped: "Children are separate pi processes, so none of their tokens reached the stream just parsed; the extension's usage file is the only record of what they spent." run.go still writes metrics on the stall path (writeMetricsJSON at 2317, after aggregateRunMetrics), so the file is produced — just with every completed child's spend missing, and with per_model_usage no longer summing to the totals (the comment at 897-899 flags exactly that invariant). Sub-agent stalls are now a realistic path: commit 3c43f5f added the 30s liveness tick precisely because parents wait on children, and a parent that stalls after several children finished has real, unrecorded spend.
Suggestion: Read and fold the sub-agent usage file best-effort before returning ErrStalled (same call, same non-fatal error handling), so completed child cost survives a parent stall. A test with recorded child usage followed by a parent stall would pin it.
|
|
||
| // stallKill is the kill every streaming runtime hands the watchdog. | ||
| // | ||
| // cancel — the context cancel from sandbox.ExecStreamReader, and the whole of |
There was a problem hiding this comment.
[MEDIUM] stall.go and run.md still claim the global timeout does nothing inside the sandbox; the #7042 sweep contradicts that, as does this PR's own contributing doc
stallKill's doc comment (stall.go:98-102) says cancel is "the whole of what the global timeout does" and that the agent "keeps writing the workspace and spending tokens until the run's deferred sandbox.Delete tears the sandbox down (and forever, under --keep-sandbox)". docs/cli/run.md:79 repeats it more strongly: "Cancelling the exec is all the global timeout does and it signals nothing inside the sandbox, so the sweep is what actually stops a wedged agent ... including under --keep-sandbox, where nothing else would." Both are stale. run.go:2364-2381 runs agentruntime.TerminateStrayProcesses under withSandboxLock whenever iterationTimedOut is true, before extraction and regardless of --keep-sandbox (#7042); this is not branch-only — git show origin/main:internal/cli/run.go has the same call at line 2352. The PR's own docs/contributing/runtime-implementation.md:193 already documents it correctly: "The same sweep runs as TerminateStrayProcesses right after an iteration ends at its budget." So two of the three passages describing the same mechanism disagree with the third and with the code.
Suggestion: Fix stall.go:98-102 and docs/cli/run.md:79 to match runtime-implementation.md:193 — the global-timeout path already TERM/KILLs the agent's sandbox processes (under the sandbox lock, --keep-sandbox included); what is unique to the stall path is that it does the sweep from the runtime layer because run.go's runErr != nil branch returns before the timeout sweep ever runs.
| // tool_execution_update line on the parent's stream, which the runner counts | ||
| // as liveness; 30s matches the watchdog's poll cap. A child that is itself | ||
| // wedged is bounded by its own timeoutSeconds, not by the watchdog. | ||
| const DEFAULT_LIVENESS_MS = 30_000; |
There was a problem hiding this comment.
[MEDIUM] No floor ties FULLSEND_STALL_TIMEOUT to the sub-agent extension's fixed 30s liveness tick
DEFAULT_LIVENESS_MS = 30_000 (fullsend-agent.js:64, used by the setInterval at 791-794) is a compile-time constant whose comment justifies itself as "30s matches the watchdog's poll cap" — it has no knowledge of the host-side FULLSEND_STALL_TIMEOUT. setInterval does not fire immediately, so the parent stream can be silent for a full 30s after a child starts and again between ticks. Meanwhile docs/cli/run.md:81 explicitly tells operators with short timeout_minutes to "lower FULLSEND_STALL_TIMEOUT for those", and resolveStallTimeout accepts any non-negative duration with no minimum. The concrete break point: at a 30s stall timeout effectiveStallTimeout still arms the watchdog (detection latency is timeout/20 = 1.5s, well under a multi-minute run timeout), so a healthy parent whose only activity is waiting on a child is killed as stalled. The plumbing to fix it already exists — createAgentTool takes livenessMs as an injectable option (fullsend-agent.js:487) — only the manifest wiring is missing.
Suggestion: Either pass the effective stall timeout (or a fraction of it) through the pi manifest into createAgentTool's existing livenessMs option so the tick scales with the configured timeout, or have resolveStallTimeout/effectiveStallTimeout warn or refuse below a documented floor (e.g. 2 * DEFAULT_LIVENESS_MS = 60s) and state that floor in docs/cli/run.md:81 next to the advice to lower the timeout.
3 participants
Heyaa : )
This one came out of watching a wedged run burn its entire global timeout while the heartbeat cheerfully printed "agent running" the whole time — the process was dead and nothing could tell.
The runtime now watches the runtime output stream instead: every well-formed line the stream parser reads is proof of life (including lines that map to no
AgentEvent— pi'stool_execution_update, Claude Code'susertool results, codex'sitem.started/item.updated), half a timeout of silence warns once per stall episode (::warning::in CI, printer everywhere), and a full timeout kills the run and fails it distinctly withErrStalled+"stalled": trueinmetrics.json.Config:
FULLSEND_STALL_TIMEOUT(Go duration, default15m,0disables), resolved by the CLI (runStallTimeoutininternal/cli/run_overrides.go) and handed to the runtime inRunParams.StallTimeout— runtimes don't read env themselves (#6526). A malformed value is reported and the default applies. The value is allowlisted insetup-agent-env.sh, so a CI repo variable can set it. The watchdog is disarmed — and the run says so — when the stall timeout plus its detection interval (a twentieth of the timeout, capped at 30s) is not below the harnesstimeout_minutes, since the global deadline would always win: at the default that means harnesses at 15 minutes or less get no stall protection.Coverage: claude, pi and codex — every runtime that streams — through one guard in the shared stream helper at each call site (
startStallWatchdog);dummy/opencodestream no events and ignore the field.RunParamsnames the covered runtimes and a source-shape test pins all three.Design points:
cancelfromsandbox.ExecStreamReaderonly SIGKILLs the localopenshell sandbox execclient; OpenShell has no signal API. So the kill first runs the same TERM-then-KILL stray-process sweep that clears the sandbox between iterations, over a second exec channel, then cancels — the wedged agent stops writing the workspace and spending tokens, under--keep-sandboxtoo. The sweep is not serialized against the credential refreshers (a kill path must not wait on a lock a wedged upload may hold); the only file it can truncate is the five-minute OIDC token, and the docs tell operators to re-seed before reusing a kept sandbox.BASH_MAX_TIMEOUT_MSceiling (600000ms), which the model routinely requests for test suites. Not a breaking change — previously successful runs stay successful; repos that raise the bash ceiling raise the stall timeout with it.timeoutSeconds(900s).ctx(a body-scope rebind once made every successful run report "cancelled"); the stop/fire race is closed with a CAS and silence is measured on the monotonic clock.Tests: the watchdog matrix (flowing lines never kill / silence kills exactly once /
0disables / warn-once-per-episode, rearmed by the next line / no annotations outside CI / no warning after disarm),resolveStallTimeout+effectiveStallTimeoutcases, the sweep-then-cancel order and its failure path,TestClaudeRuntime_Run_StallSweepsTheSandboxOncedriving the realRunpath, the source-shape guard for claude/pi/codex,stalled,omitemptymarshalling, and the Agent tool's liveness updates (node --test).go build,go vet, andinternal/runtime,internal/cli,internal/scaffoldpass.Non-goals: per-dimension timeouts (dimensions run inside one CLI process the runner can't see into), and no change to the heartbeat or the global timeout — the heartbeat's future is a separate discussion after merge.