Repository navigation
Conversation
A Claude Code secondmate on the Mac mini froze three times, each time needing a hand-run SIGKILL before it could be relaunched. The liveness library already probes every secondmate endpoint on the watcher tick and relaunches a dead one through the guarded spawn path, but a frozen agent classifies `alive` - correctly and permanently, since its process really is running - so it fell through every recovery path. Add `wedged` as a separate state for herdr-backed secondmates, computed from the two progress counters herdr already exposes (state_change_seq and scroll.max_offset_from_bottom). Both must stay unchanged for the whole configured window while the agent still reports `working`, which is the machine-readable form of the stopped TUI timer a human recognizes. Recovery captures non-destructive evidence first and records its path in the relaunch ledger - three freezes left no sample, which is why the mechanism behind them is still unproven - then SIGKILLs only pids that classify as a harness inside the pane's own process subtree, relaunches through the existing path, and raises the wedge alarm so an automatic kill of live work is never silent. Three limits guard against killing a legitimately quiet agent: both counters must be frozen, the window is configurable with a 15-minute default and a 10-minute floor, and the existing per-mate relaunch bound applies unchanged. An unreadable counter yields `unreadable`, never `wedged`, so a vendor shape change can only disable recovery; the watcher logs that case rather than letting it degrade silently. For a remote mate the counters, the capture, and the kill all run on its own host through two new control verbs; the parent never signals across hosts or derives a pid from a local reading. Supporting changes: extract the wedge alarm's channel dispatch from fm-supervise-daemon.sh into bin/fm-wedge-alarm-lib.sh so both callers share one notification path, and move the herdr adapter's pane process-tree walk into bin/fm-agent-process-lib.sh so the probe that proves a pane agent-free and the kill that acts on it cannot drift. Out of scope, noted as follow-ups rather than built: ordinary crewmates, non-herdr backends, the Stop hook's asyncRewake timeout, and holding a steer until the pane is idle.
… wedge kill Review of the wedged-secondmate recovery found three ways the detector could SIGKILL a healthy agent. All three are fixed here; the first carries the captain's explicit decision on detection policy. Two frozen progress counters do not separate a deadlock from a long, quiet turn. A healthy agent running one long Bash command or a subagent redraws its spinner in place and adds no scrollback, so neither counter moves and it would have been killed up to the relaunch bound. Require two further signals before a wedged verdict, both from the incident's own evidence that the frozen process sat at 0% CPU with no children: the agent's consumed CPU time must not grow across the window, and the agent must have no live child process other than its own MCP servers. Either signal unreadable now yields no verdict, so an unreadable process table can only disable recovery, never authorize a kill. The window was wall-clock age, not observed time. A machine that slept for 20 minutes with an agent mid-turn woke up holding a pre-sleep sample whose counters could not have moved, and only the session-start sweep re-based the window, so the first poll after wake would have killed live work. The record now carries the previous sample's own epoch and a gap larger than the detector could have watched restarts the window. The comment claiming suspend was already covered was wrong and is corrected. Local wedge recovery leaked a husk pane. The kill deliberately spares the pane shell, but the relaunch then spawns a new pane exactly as it does for a dead endpoint, so the agent-free pane was left behind. Close it in the relaunch path once nothing is running in it, rather than in the kill primitive, so the agent-only kill scope and the tests that prove it stay intact. A remote route needs nothing: its host-local launch already removes a confirmed agent-less endpoint. Tests cover each new signal with its own control on the same fixture, against real processes: a live tool child stays alive while an MCP-server child alone still wedges, rising CPU time stays alive, an unobserved gap restarts the window while continuously observed frozen time still wedges, and an unreadable counter, CPU column, or process table never wedges.
…nce, fix stand-ins
Owner
|
Speaking as Kun's firstmate: thanks @mitu1306. Three hard freezes that each needed a manual I compared head CI: I reviewed the diff first (no workflow or secret surface; Contract-class: new-default. With no config the window is 900s, so every Herdr secondmate gets automatic SIGKILL-and-relaunch unless the captain writes VISION.md, rule by rule
Waiting on you, the author: could you make it opt-in (absent |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Automatic freeze recovery: go-ahead to build it? - yes.
Context. The remote secondmate secondmate_outreach (Claude Code in herdr session fm-remote on host mac-mini) froze three times on 2026-10-06/07: the TUI stopped on "running PreToolUse hooks" with the timer stopped, the process alive at 0% CPU with no children, SIGTERM ignored; each time the captain had to SIGKILL it by hand before it could be relaunched.
The investigation report data/macmini-claude-freeze/report.md (in the firstmate home; read sections 7 and 8 in full) found an upstream Claude Code deadlock, not Firstmate hooks or a version regression, and recommended the fix approved here: bin/fm-secondmate-liveness-lib.sh already probes secondmate endpoints on each watcher tick and relaunches dead ones through FM_SPAWN_NO_GUARD=1 bin/fm-spawn.sh --secondmate under a per-mate lock with a bounded relaunch ledger, but fm_backend_herdr_agent_state classifies a frozen Claude as alive, so it is never recovered. herdr exposes progress counters (state_change_seq from
herdr agent get <pane>, scroll.max_offset_from_bottom fromherdr pane list) that stop advancing when the session freezes.The approved design: a distinct wedged state (agent working, no progress across a generous configurable window, suggested 10+ minutes, sampled on the existing watcher tick); on the first wedged verdict capture evidence non-destructively (macOS
sample <pid> 10pluslsof -p <pid>, Linux equivalent) into the secondmate state and record it in the relaunch ledger; then SIGKILL only that pane's agent process, never the herdr server, sibling panes or worker processes, and relaunch through the existing path, reusing the existing relaunch bound so a misdetection cannot kill-loop a healthy mate; raise the existing wedge alarm (docs/wedge-alarm.md) so a recovery is never silent.What Changed
wedgedliveness state for Herdr-backed secondmates (newbin/fm-herdr-wedge-lib.sh), checked on every watcher tick. A mate counts as wedged only if all of these stay frozen for the whole no-progress window while the agent still reportsworking: both herdr progress counters (state_change_seqandscroll.max_offset_from_bottom), the agent's CPU time, and its live non-MCP child processes. The window defaults to 900s, has a 600s floor, is set inconfig/secondmate-wedge-window, and can be turnedoff. It counts only observed time, so a long gap between samples (for example after a suspend) restarts it. Any signal that cannot be read gives no verdict, so it never leads to a kill.fm-watch.shlogs when the counters cannot be read.bin/fm-secondmate-liveness-lib.shfirst saves evidence (sample/lsofon macOS, or the Linux equivalent) and records its path in the relaunch ledger. It then SIGKILLs only the harness processes in that pane's own process subtree and relaunches through the existing guarded spawn path, inside the existing relaunch bound. Remote mates get newwedge-stateandwedge-recoververbs infm-remote-secondmate-control.sh, so the probe, capture and kill all run on the mate's own host. The supervising home still sets the window and the sample-gap limit.fm-teardown.shremoves the new per-mate wedge state file.fm-supervise-daemon.shinto a sharedbin/fm-wedge-alarm-lib.shwith a title each caller can set. Process-table and descendant helpers moved out ofbackends/herdr.shintofm-agent-process-lib.sh. Docs were updated (configuration, wedge-alarm, remote-secondmates, herdr-backend, scripts, architecture), and remote and local wedge cases were added totests/fm-secondmate-liveness.test.sh.🤖 Generated with Claude Code
Risk Assessment
✅ Low: The fix round correctly parses both BSD
[HH:]MM:SS.CCand Linux[DD-]HH:MM:SScputime to centiseconds and rejects anything else. It tags stored CPU values with a unit so older records restart the window instead of reading as equal. It derives the sample-gap limit from the watcher cadence, both locally and when the parent passes it to a remote mate. Tests use fm_agent_standin and include controls that would fail if the code were wrong, and nothing in the change contradicts the stated intent.Testing
In this test step I ran two isolated herdr labs against a real, SIGSTOP-frozen lab Claude. The first was a 12-minute watcher-tick run. It showed the full chain in order: wedged verdict at about 600s, evidence capture, ledger rows, an agent-only SIGKILL, pane close, relaunch call, and the wedge alarm. The second was a short kill-scope run. It showed that afterfm_secondmate_liveness_wedge_recoverthe pane shell, the pane, and a process outside the pane all survived; only the claude pid died. The 12-minute run took place earlier in this same test step, before my context restarted. I report it from the log and files it wrote at 22:47-22:58, which is after the HEAD commit (22:19). I also ran the targeted test file (40/40 pass), which covers the macOS BSD and Linux cputime formats and the stubbed remote transport. No pattern-based kills were used. Both labs were torn down, and the default and fm-remote sessions were untouched. This is a CLI/daemon change with no UI, so there are no screenshots; pane captures and logs are the evidence.Evidence: 12-minute driver log (ticks 0-11, wedged verdict, relaunch, kill, pane close)
Source: 12-minute driver log (ticks 0-11, wedged verdict, relaunch, kill, pane close)
Evidence: Lab driver script (replays the fm-watch liveness tick)
Source: Lab driver script (replays the fm-watch liveness tick)
Evidence: Evidence capture written before the kill (full run)
Source: Evidence capture written before the kill (full run)
Evidence: Relaunch ledger rows (attempt, wedge-capture, relaunched)
Source: Relaunch ledger rows (attempt, wedge-capture, relaunched)
1791475121 attempt 1791475122 wedge-capture /tmp/fm-lab.jSM1vw/state/.wedge-sample-smlab-1791475122.txt 1791475123 relaunchedEvidence: Recorded relaunch call
Source: Recorded relaunch call
22:58:43 spawn argv=[smlab --secondmate] FM_SPAWN_NO_GUARD=1Evidence: Wedge alarm text
Source: Wedge alarm text
secondmate smlab was wedged (running, no progress); killed agent pid(s) 309690 and relaunched. Evidence: /tmp/fm-lab.jSM1vw/state/.wedge-sample-smlab-1791475122.txtEvidence: Frozen lab pane before the full run (herdr shows working)
Source: Frozen lab pane before the full run (herdr shows working)
Evidence: Kill-scope: wedge_recover result and ledger
Source: Kill-scope: wedge_recover result and ledger
Evidence: Kill-scope: processes and pane after the kill
Source: Kill-scope: processes and pane after the kill
ps after kill (shell 865329, wrapper 878848, agent 878852, outside sentinel 868038) 865329 862634 Ss+ /usr/bin/zsh 868038 955 SN sleep 3600 herdr pane list after kill: {"pane_id":"w1:p1","agent_status":"unknown"} process-info: {"shell_pid":865329,"fg":[{"pid":865329,"name":"zsh"}]}Evidence: Kill-scope: pids while frozen (claude in T state)
Source: Kill-scope: pids while frozen (claude in T state)
Evidence: Kill-scope: frozen pane render and pane after kill
Source: Kill-scope: frozen pane render and pane after kill
Evidence: Targeted test file output (40/40 ok)
Source: Targeted test file output (40/40 ok)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed ✅
bin/fm-herdr-wedge-lib.sh:195- On macOS, wedge detection never fires, and macOS is the platform of the incident (mac-mini). BSD/macOSps -o cputime=printsminutes:seconds.centiseconds, for example0:12.34or125:03.45, not Linux's[dd-]hh:mm:ss. The awk parser splits on[:-]and requires every field to match^[0-9]+$, so the12.34field fails it and fm_herdr_wedge_cpu_seconds returns nonzero. fm_herdr_wedge_classify then printsunreadableon every tick (line 339), so a frozen secondmate on the mac-mini is never recovered. This breaks the intent's core requirement and the user's decision to 'read CPU time with ps cputime on macOS'. fm-watch.sh also logs 'progress counters unreadable' for every mate on every tick. The tests run the real Linuxps, so they do not catch it. Fix: accept the BSD shape. Allow an optional fractional part on the last field (drop it, or compare with centisecond precision) and treat a 2-fieldM:SSvalue as minutes:seconds. Add a test that feeds a0:12.34-style value through the FM_HERDR_PS_BIN seam.bin/fm-herdr-wedge-lib.sh:90- The 300s FM_HERDR_WEDGE_MAX_SAMPLE_GAP_SECS limit is fixed and does not depend on the watcher cadence. If a home sets FM_SECONDMATE_LIVENESS_SECS to 300 or more, or the watcher is often down for more than 5 minutes between wake cycles, every sample restarts the window and returnsprogressing. Wedge recovery is then turned off with no log line. This fails safe (no false kill), but it is the kind of silent degradation theunreadablelog in fm-watch.sh was added to prevent. Consider deriving the gap from the tick cadence (for example, a few ticks), or logging when a window is restarted because of a gap.🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Test** - 2 issues found → auto-fixed → no changes applied ✅
bin/fm-herdr-lab.sh provision/run/teardown fm-lab-wedge-live-*with realclaude --model haikuin two panes (default session untouched; tripwire held)Froze pane 1's real Claude mid-turn withsystemctl --user freeze <scope>and confirmed agent_status=working, frozen TUI timer, flat cputime, SIGTERM ignoredPane 2: real Claude running a silent foreground./build.sh(perl select 1500s) as a Bash tool call, agent_status=workingDriver sourcing bin/fm-secondmate-liveness-lib.sh:fm_secondmate_liveness_probefull sweep then poll ticks every 60s on both panes in abin/fm-lab-home.shlab home (stopped after tick 1)fm_herdr_wedge_agent_pids,fm_herdr_wedge_live_work,fm_herdr_wedge_cpu_centiseconds,fm_herdr_wedge_countersagainst the live lab panefm_herdr_wedge_classifyon the frozen pane with FM_HERDR_WEDGE_MAX_SAMPLE_GAP_SECS=5 (simulated suspend gap)fm_secondmate_liveness_probewith config/secondmate-wedge-window=off, andfm_herdr_wedge_windowwith 60 and a typobash tests/fm-secondmate-liveness.test.sh(colocated wedge suite, rc=0)🔧 Fix applied.
2 warnings still open:
sh -cwrapper so zsh job control does not take back the foreground, and herdr kept reportingworking. The driver recorded the baseline at tick 0. Then I stopped it by mistake about 50 seconds in, because apkill -fpattern matched it. Re-run the same lab setup and leave the driver alone for 11 minutes or more to close this gap.working, and the session-start sweep records a baseline instead of judgingbin/fm-herdr-lab.sh provision fm-lab-wedgefinal-3638766-936plusbin/fm-lab-home.sh create(isolated lab session and home)Started lab-onlyclaude --model haikuin lab pane w1:p1 undersh -c, sent a long prompt, thenkill -STOPandkill -TERMon its pid; herdragent getstill reportedworkingLab driver that repeats the watcher's secondmate_liveness_tick against the lab meta; tick 0 baseline recorded, then the driver was stopped by mistakebash tests/fm-secondmate-liveness.test.sh(40/40 ok)bin/fm-herdr-lab.sh teardown fm-lab-wedgefinal-3638766-936andrm -rfof the lab home🔧 No changes applied.
✅ Re-checked - no issues remain.
Full 12-minute lab run (driver started detached and left alone):last-attempt/driver.shreplays the fm-watch.sh liveness tick (probe -> relaunch bound -> fm_secondmate_liveness_relaunch) every 60s against lab pane fm-lab-wedge12m-301279-3492:w1:p1, with a lab-only claude frozen by SIGSTOP insidesh -c. The fm-spawn.sh it calls is a lab fakeroot that only records the call.Kill-scope lab:bin/fm-herdr-lab.sh provision fm-lab-killscope-862559-30509,pane run w1:p1 "sh -c 'claude --model haiku --permission-mode bypassPermissions; true'", prompt sent,kill -STOP <recorded claude pid>, thenfm_secondmate_liveness_wedge_recovercalled directly on a lab meta under a marked lab home frombin/fm-lab-home.sh createPost-kill checks:ps -o pid,ppid,stat,args -p <shell>,<wrapper>,<agent>,<outside sentinel>,herdr pane list,herdr pane process-info --pane w1:p1,herdr pane read w1:p1Teardown only by recorded pid and exact session name:kill 868038,bin/fm-herdr-lab.sh teardown fm-lab-killscope-862559-30509,rm -rf $LAB. Afterwards I confirmed the default and fm-remote sessions were unchanged.bash tests/fm-secondmate-liveness.test.sh(40/40 ok, including BSD and Linux cputime parsing and the stubbed remote wedge-state and wedge-recover cases)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.