Skip to content

fix(bin): keep acknowledged downtime retired when a take-over stop lands late - #6670

Open
Vela3202019 wants to merge 5 commits into
kunchenguid:mainfrom
Vela3202019:fm/firstmate-resurface-loop
Open

Vela3202019 wants to merge 5 commits into
kunchenguid:mainfrom
Vela3202019:fm/firstmate-resurface-loop

Conversation

@Vela3202019

@Vela3202019 Vela3202019 commented Oct 5, 2026 •

Copy link
Copy Markdown

Intent

Stop the supervision host from re-announcing 'check: rearm-resurface' on every turn end while attended.

Seen 2026-10-04 10:15-10:17 PDT on main at 241d461, attended, supervision host on, wake queue empty (.wake-queue 0 bytes, seq 76). Four consecutive empty 'check: rearm-resurface' wakes arrived about 28s apart, each handled with a drain plus --ack-through 0. state/.supervision-host.log repeats per turn: 'start gen=host-N' -> 'pass-through attended main-only check: rearm-resurface' -> (main turn ends) 'start gen=host-M' + 'take-over arm=' -> about 19s later 'pass-through a close without a wake' -> 'start' -> resurface again. state/.watch-cycle-exits.log shows the taken-over cycle ending with a signal-exit reason, an attached arm ending attached-cycle-ended, then a started cycle exiting actionable-check at once. The marker state/.watcher-down read acked:handling:, so the dying cycle's pending downtime was announced to main instead of being restored by fm_recovery_marker_handover_restore (bin/fm-wake-lib.sh; bin/fm-watch-arm.sh around lines 559-590). Suspect the host's restart after 'a close without a wake' re-arms with recovery pending, or the handover snapshot is taken before main's acknowledgement.

Find why an acknowledged, empty-queue home re-announces downtime on every take-over, fix it, and add a regression test. Cost today: one main turn per loop while attended.

The PR must also cover Greptile's P1 'Slow captures outlive takeover' (bin/fm-watch-arm.sh:149), so Greptile passes. Make the take-over path restore the acknowledged downtime marker even when the watcher's pane capture runs past the stop bound - bound that capture (or otherwise make take_over_cycle not give up before the restore), plus a regression test with a capture slower than the bound. Out of scope: bounding the other herdr calls in the watcher cycle body; that stays a separate task.

What Changed

  • take_over_cycle in bin/fm-watch-arm.sh used to give up after a fixed 5-second wait for the TERM'd watcher to exit. It now waits up to the stall bound (STALL_BOUND). A watcher stuck in a slow pane capture, poll sleep, or event wait now still dies inside the take-over. The handover restore then runs, so an acknowledged, empty-queue episode is no longer re-announced as check: rearm-resurface on every take-over.
  • Added two regression tests to tests/fm-watch-arm.test.sh. Both use a fake tmux capture-pane that blocks mid-capture: one takes 12s, and the other takes 30s against FM_POLL=1, so it runs well past the poll interval. Each test checks that the take-over waits for the watcher to die, restores the acknowledged marker, and keeps the fresh cycle quiet.
  • Updated docs/watcher-continuity.md and the --take-over header comment to describe the stall-bounded wait for the watcher's death. They also note that the short ledger-row wait still follows it. The slow-capture take-over is added to the arm test coverage list.

Risk Assessment

✅ Low: The fix round turns take_over_cycle's two waits into one wait bounded by STALL_BOUND, as the user decided, and removes WATCH_POLL, TAKE_OVER_STOP_BOUND and the "stop window" wording; behavior matches the previous combined wait, the restore still runs after a capture slower than the poll interval, and a watcher that outlives the stall bound still falls back to the plain attach.

Testing

I ran the five take-over tests in tests/fm-watch-arm.test.sh on HEAD, and all passed. The two new regression tests fail when the base fm-watch-arm.sh is swapped in, so they reproduce the bug. I then drove the real fm-watch-arm.sh and fm-watch.sh in disposable lab homes against a real tmux pane on a private socket, with a tmux shim that only slows capture-pane. The base code reproduced the incident end to end: the acknowledged marker became pending downtime, check: rearm-resurface printed again, and the ledger showed the same three exit reasons. On HEAD the take-over waited out both a 12s capture and a 30s capture with a 1s poll (the Greptile P1 case), kept the marker acked:handling, started a quiet fresh cycle, and linked it in the ledger. A watcher that outlived the stall bound still made the take-over give up on time and attach as a plain arm. The guard scenarios (queued work, self-exit downtime, an unowned cycle) were checked only by the repository's tests, not in a live lab run, so that scenario is marked untested. Every lab was torn down.

  • Live validation: ✅ go - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Main acknowledges a wake and the queue is empty; a --take-over lands while the watcher is in a 12s pane capture; the episode stays acked:handling and no check: rearm-resurface is printed ✅ pass live live-head-capture12s.log (base contrast: live-base-capture12s.log shows pending:downtime and then check: rearm-resurface)
Reproduce the reported loop on base 99da16d: the take-over gives up, the arm plain-attaches, the dying watcher reopens downtime, and the next re-arm re-announces check: rearm-resurface ✅ pass live live-base-capture12s.log: ledger reads signal-exit, then attached-cycle-ended, then actionable-check
Greptile P1: with FM_POLL=1, a pane capture of about 30s runs far past the poll window, and the take-over still restores the acknowledged marker and starts a quiet fresh cycle ✅ pass live live-head-capture30s.log; test_take_over_restores_after_a_capture_that_outlives_the_poll_interval passes on HEAD and fails on base
Adversarial: a watcher that outlives the stall bound does not hang the take-over; it gives up at the bound and plain-attaches ✅ pass live live-head-outlives-stall-bound.log (attached after 6s with FM_WATCHER_STALL_BOUND=6)
Guards still hold: a take-over still surfaces queued work and self-exit downtime, and does not stop a cycle the named arm does not own ⏸️ untested no The prior payload did not establish a live result for this scenario. It was checked only by the repository tests (test_take_over_owns_a_fresh_cycle_and_keeps_queued_work_surfacing, test_take_over_pres…
Evidence: Live run on base 99da16d: take-over gives up, marker reopens, re-arm prints check: rearm-resurface

Source: Live run on base 99da16d9: take-over gives up, marker reopens, re-arm prints check: rearm-resurface

== [base 99da16d9 (before fix)] capture delay 12s, lab=/var/folders/32/ng_xnctn3dj29mxdt024s0cc0000gn/T//fm-lab.IsSdg9
-- owner arm: watcher: started pid=31564 (beacon fresh)
-- watcher pid=31564 entered its slow pane capture: yes
-- after main's ack: .watcher-down=acked:handling:32295.1791354523.V1jjH5  queue_bytes=0
-- take-over arm output after 8s:
     watcher: attached pid=31564 (beacon 10s)
-- old watcher pid=31564 alive: yes
-- .watcher-down now: acked:handling:32295.1791354523.V1jjH5
-- after the old watcher died: .watcher-down=pending:downtime:31564.1791354535.fKo7V5
-- 4s later, take-over arm still waiting quietly: no; output now:
     watcher: attached pid=31564 (beacon 10s)
     watcher: FAILED - cycle ended without an actionable reason
-- take-over arm exited; the host re-arms as it does after 'a close without a wake':
     check: rearm-resurface
-- cycle ledger (.watch-cycle-exits.log):
     arm_pid=31537	watcher_pid=31564	origin=started	started_at=1791354522	ended_at=1791354535	exit_code=143	signal=TERM	reason=signal-exit	beacon_age=13	lock_before=pid:31564|identity:Tue Oct  6 23:28:42 2026     bash /tmp/fmbase.2qOP/bin/fm-watch.sh	lock_after=pid:none|identity:none	successor=none
     arm_pid=33301	watcher_pid=31564	origin=attached	started_at=1791354532	ended_at=1791354539	exit_code=unknown	signal=unknown	reason=attached-cycle-ended	beacon_age=17	lock_before=pid:31564|identity:Tue Oct  6 23:28:42 2026     bash /tmp/fmbase.2qOP/bin/fm-watch.sh	lock_after=pid:none|identity:none	successor=none
     arm_pid=37732	watcher_pid=37749	origin=started	started_at=1791354539	ended_at=1791354539	exit_code=0	signal=none	reason=actionable-check	beacon_age=0	lock_before=pid:none|identity:none	lock_after=pid:none|identity:none	successor=none
== [base 99da16d9 (before fix)] lab torn down
Evidence: Live run on HEAD, 12s capture: marker stays acked, fresh quiet cycle, ledger taken-over successor=started

Source: Live run on HEAD, 12s capture: marker stays acked, fresh quiet cycle, ledger taken-over successor=started

== [fixed HEAD 960390ae] capture delay 12s, lab=/var/folders/32/ng_xnctn3dj29mxdt024s0cc0000gn/T//fm-lab.r6Oa1b
-- owner arm: watcher: started pid=39061 (beacon fresh)
-- watcher pid=39061 entered its slow pane capture: yes
-- after main's ack: .watcher-down=acked:handling:40206.1791354547.DY4lBO  queue_bytes=0
-- take-over arm output after 12s:
     watcher: started pid=44264 (beacon fresh)
-- old watcher pid=39061 alive: no
-- .watcher-down now: acked:handling:40206.1791354547.DY4lBO
-- after the old watcher died: .watcher-down=acked:handling:40206.1791354547.DY4lBO
-- 4s later, take-over arm still waiting quietly: yes; output now:
     watcher: started pid=44264 (beacon fresh)
-- cycle ledger (.watch-cycle-exits.log):
     arm_pid=39038	watcher_pid=39061	origin=started	started_at=1791354546	ended_at=1791354559	exit_code=143	signal=TERM	reason=signal-exit	beacon_age=13	lock_before=pid:39061|identity:Tue Oct  6 23:29:06 2026     bash ~/.no-mistakes/worktrees/34bae1aacd62/01M49PKABF74NX0T7K3KJK59CB/bin/fm-watch.sh	lock_after=pid:none|identity:none	successor=none
     arm_pid=41142	watcher_pid=39061	origin=attached	started_at=1791354548	ended_at=1791354559	exit_code=unknown	signal=unknown	reason=taken-over	beacon_age=13	lock_before=pid:39061|identity:Tue Oct  6 23:29:06 2026     bash ~/.no-mistakes/worktrees/34bae1aacd62/01M49PKABF74NX0T7K3KJK59CB/bin/fm-watch.sh	lock_after=pid:none|identity:none	successor=started:44264
== [fixed HEAD 960390ae] lab torn down
Evidence: Live run on HEAD, FM_POLL=1 with a 30s capture (Greptile P1): restore still runs

Source: Live run on HEAD, FM_POLL=1 with a 30s capture (Greptile P1): restore still runs

== [fixed HEAD 960390ae, FM_POLL=1, capture 30s (Greptile P1)] capture delay 30s, lab=/var/folders/32/ng_xnctn3dj29mxdt024s0cc0000gn/T//fm-lab.WUODbf
-- owner arm: watcher: started pid=12025 (beacon fresh)
-- watcher pid=12025 entered its slow pane capture: yes
-- after main's ack: .watcher-down=acked:handling:13302.1791354446.mmvlOF  queue_bytes=0
-- take-over arm output after 30s:
     watcher: started pid=20860 (beacon fresh)
-- old watcher pid=12025 alive: no
-- .watcher-down now: acked:handling:13302.1791354446.mmvlOF
-- 4s later, take-over arm still waiting quietly: yes; output now:
     watcher: started pid=20860 (beacon fresh)
-- cycle ledger (.watch-cycle-exits.log):
     arm_pid=12000	watcher_pid=12025	origin=started	started_at=1791354446	ended_at=1791354476	exit_code=143	signal=TERM	reason=signal-exit	beacon_age=30	lock_before=pid:12025|identity:Tue Oct  6 23:27:26 2026     bash ~/.no-mistakes/worktrees/34bae1aacd62/01M49PKABF74NX0T7K3KJK59CB/bin/fm-watch.sh	lock_after=pid:none|identity:none	successor=none
     arm_pid=14296	watcher_pid=12025	origin=attached	started_at=1791354447	ended_at=1791354477	exit_code=unknown	signal=unknown	reason=taken-over	beacon_age=31	lock_before=pid:12025|identity:Tue Oct  6 23:27:26 2026     bash ~/.no-mistakes/worktrees/34bae1aacd62/01M49PKABF74NX0T7K3KJK59CB/bin/fm-watch.sh	lock_after=pid:none|identity:none	successor=started:20860
== [fixed HEAD 960390ae, FM_POLL=1, capture 30s (Greptile P1)] lab torn down
Evidence: Live adversarial run: watcher outlives the stall bound, take-over gives up at the bound and plain-attaches

Source: Live adversarial run: watcher outlives the stall bound, take-over gives up at the bound and plain-attaches

== [fixed HEAD, FM_WATCHER_STALL_BOUND=6, capture 25s (watcher outlives stall bound)] capture delay 25s, lab=/var/folders/32/ng_xnctn3dj29mxdt024s0cc0000gn/T//fm-lab.QebM2n
-- owner arm: watcher: started pid=48945 (beacon fresh)
-- watcher pid=48945 entered its slow pane capture: yes
-- after main's ack: .watcher-down=acked:handling:49748.1791354583.ULg3Gs  queue_bytes=0
-- take-over arm output after 6s:
     watcher: attached pid=48945 (beacon 7s)
-- old watcher pid=48945 alive: yes
-- .watcher-down now: acked:handling:49748.1791354583.ULg3Gs
-- after the old watcher died: .watcher-down=pending:downtime:48945.1791354607.ZElqO3
-- 4s later, take-over arm still waiting quietly: yes; output now:
     watcher: attached pid=48945 (beacon 7s)
-- cycle ledger (.watch-cycle-exits.log):
     arm_pid=48912	watcher_pid=48945	origin=started	started_at=1791354582	ended_at=1791354608	exit_code=143	signal=TERM	reason=signal-exit	beacon_age=26	lock_before=pid:48945|identity:Tue Oct  6 23:29:42 2026     bash ~/.no-mistakes/worktrees/34bae1aacd62/01M49PKABF74NX0T7K3KJK59CB/bin/fm-watch.sh	lock_after=pid:none|identity:none	successor=none
== [fixed HEAD, FM_WATCHER_STALL_BOUND=6, capture 25s (watcher outlives stall bound)] lab torn down
Evidence: Live scenario driver script (lab home + private tmux socket)

Source: Live scenario driver script (lab home + private tmux socket)

#!/usr/bin/env bash
# Live scenario: a real fm-watch-arm.sh owner arm + real fm-watch.sh watcher in a
# disposable lab FM_HOME, watching a real crew pane on a private tmux socket
# (fm-lab). A PATH wrapper delegates every tmux call to that socket and only
# delays capture-pane by CAPTURE_DELAY seconds. Main acknowledges, then a
# --take-over arm takes the cycle over mid-capture.
# Usage: live-takeover-scenario.sh <repo-bin-dir> <capture-delay> <label>
set -u
BIN=$1 DELAY=$2 LABEL=$3
WT=~/.no-mistakes/worktrees/34bae1aacd62/01M49PKABF74NX0T7K3KJK59CB
LAB=$(mktemp -d "${TMPDIR:-/tmp}/fm-lab.XXXXXX")
"$WT/bin/fm-lab-home.sh" create "$LAB" >/dev/null
mkdir -p "$LAB/tmux" "$LAB/shim"
export TMUX_TMPDIR="$LAB/tmux"; unset TMUX
REAL=/opt/homebrew/bin/tmux
cat > "$LAB/shim/tmux" <<SH
#!/usr/bin/env bash
if [ "\${1:-}" = capture-pane ]; then touch "$LAB/capture-started"; sleep $DELAY; fi
exec $REAL -L fm-lab "\$@"
SH
chmod +x "$LAB/shim/tmux"
$REAL -L fm-lab new-session -d -s fmlab -n crew 'printf "crew idle prompt\n"; exec cat'
printf 'window=fmlab:crew\nkind=ship\nharness=claude\n' > "$LAB/state/crew.meta"
E="env -u NO_MISTAKES_GATE -u FM_STATE_OVERRIDE -u FM_ROOT_OVERRIDE -u FM_DATA_OVERRIDE -u FM_CONFIG_OVERRIDE -u FM_PROJECTS_OVERRIDE PATH=$LAB/shim:$PATH FM_HOME=$LAB FM_POLL=1 FM_SIGNAL_GRACE=0 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 ${EXTRA_ENV:-}"
echo "== [$LABEL] capture delay ${DELAY}s, lab=$LAB"
$E FM_ARM_CONFIRM_TIMEOUT=30 "$BIN/fm-watch-arm.sh" --restart > "$LAB/owner.out" 2>&1 &
OWNER=$!
for _ in $(seq 600); do grep -q '^watcher: started' "$LAB/owner.out" && break; sleep 0.05; done
echo "-- owner arm: $(cat "$LAB/owner.out")"
W=$(cat "$LAB/state/.watch.lock/pid")
for _ in $(seq 400); do [ -e "$LAB/capture-started" ] && break; sleep 0.05; done
echo "-- watcher pid=$W entered its slow pane capture: $([ -e "$LAB/capture-started" ] && echo yes || echo no)"
# Main handles a wake and acknowledges it, exactly as the primary does.
$E bash -c '. "$1"; fm_wake_append signal take-over "signal: handled by main"' _ "$BIN/fm-wake-lib.sh"
$E "$BIN/fm-wake-drain.sh" >/dev/null 2>"$LAB/drain.err"
ACK=$(sed -n 's/^WAKE_ACK_REQUIRED:.*\(--ack-through .*\)$/\1/p' "$LAB/drain.err")
# shellcheck disable=SC2086
$E "$BIN/fm-wake-drain.sh" $ACK >/dev/null 2>&1
echo "-- after main's ack: .watcher-down=$(cat "$LAB/state/.watcher-down")  queue_bytes=$(wc -c < "$LAB/state/.wake-queue" | tr -d ' ')"
T0=$SECONDS
$E FM_ARM_CONFIRM_TIMEOUT=3 "$BIN/fm-watch-arm.sh" --take-over "$OWNER" > "$LAB/takeover.out" 2>&1 &
ARM=$!
for _ in $(seq 1800); do grep -q '^watcher:' "$LAB/takeover.out" && break; kill -0 $ARM 2>/dev/null || break; sleep 0.05; done
echo "-- take-over arm output after $((SECONDS-T0))s:"; sed 's/^/     /' "$LAB/takeover.out"
echo "-- old watcher pid=$W alive: $(kill -0 "$W" 2>/dev/null && echo yes || echo no)"
echo "-- .watcher-down now: $(cat "$LAB/state/.watcher-down" 2>/dev/null)"
for _ in $(seq 600); do kill -0 "$W" 2>/dev/null || break; sleep 0.1; done
sleep 4
echo "-- after the old watcher died: .watcher-down=$(cat "$LAB/state/.watcher-down" 2>/dev/null)"
echo "-- 4s later, take-over arm still waiting quietly: $(kill -0 $ARM 2>/dev/null && echo yes || echo no); output now:"; sed 's/^/     /' "$LAB/takeover.out"
if ! kill -0 $ARM 2>/dev/null; then
  echo "-- take-over arm exited; the host re-arms as it does after 'a close without a wake':"
  $E FM_ARM_CONFIRM_TIMEOUT=10 "$BIN/fm-watch-arm.sh" > "$LAB/rearm.out" 2>&1 &
  RA=$!; sleep 6; sed 's/^/     /' "$LAB/rearm.out"; kill -TERM $RA 2>/dev/null; wait $RA 2>/dev/null
fi
echo "-- cycle ledger (.watch-cycle-exits.log):"; sed 's/^/     /' "$LAB/state/.watch-cycle-exits.log"
kill -TERM $ARM 2>/dev/null; wait $ARM 2>/dev/null; kill -TERM $OWNER 2>/dev/null; wait $OWNER 2>/dev/null
for p in $(cat "$LAB/state/.watch.lock/pid" 2>/dev/null); do kill -TERM "$p" 2>/dev/null; done
sleep 1
$REAL -L fm-lab kill-server 2>/dev/null
rm -rf "$LAB"
echo "== [$LABEL] lab torn down"
Evidence: Take-over tests on HEAD (all ok)

Source: Take-over tests on HEAD (all ok)

ok - watch-arm: --take-over waits out a slow stop and keeps the acknowledged episode retired
ok - watch-arm: --take-over restores the acknowledged episode after a capture that outlives the poll interval
ok - watch-arm: --take-over owns a fresh cycle without a recovery wake and still surfaces queued work
watcher: secondmate liveness check failed
ok - watch-arm: takeover preserves self-exit downtime and surfaces a recovery wake
ok - watch-arm: --take-over attaches to a cycle the named arm does not own and leaves it running
Evidence: New regression tests against base fm-watch-arm.sh (both fail)

Source: New regression tests against base fm-watch-arm.sh (both fail)

not ok - --take-over did not own a fresh cycle after a slow stop: watcher: attached pid=72416 (beacon 10s)
watcher: FAILED - cycle ended without an actionable reason
exit=
not ok - --take-over did not own a fresh cycle after a capture that outlived the poll interval: watcher: attached pid=80377 (beacon 10s)
watcher: FAILED - cycle ended without an actionable reason
exit=

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • ⚠️ bin/fm-watch-arm.sh:588 - take_over_cycle now runs two wait loops back to back with the same condition (fm_pid_alive &#34;$pid&#34;). The first is bounded by TAKE_OVER_STOP_BOUND (FM_POLL+25). The second is bounded by STALL_BOUND, which is at least 900s by default (3 × max(300, POLL+60), fm-wake-lib.sh:136-154). Together they behave as one wait of POLL+25+STALL_BOUND. So the poll-derived stop window has no observable effect: no path acts when the first deadline passes. The leftover parts are WATCH_POLL with its validation and TAKE_OVER_STOP_BOUND (bin/fm-watch-arm.sh:151-153), the first loop (:579-582), and the comments and docs that describe a 'stop window' as a separate stage (fm-watch-arm.sh:79-82, :142-150, :571, :583-587; docs/watcher-continuity.md:236-237; the test comments at tests/fm-watch-arm.test.sh around the 'window = FM_POLL + 25' note). A single loop bounded by STALL_BOUND would behave the same and drop that code. The earlier round-1 decision asked for a poll-derived bound, and this commit makes that bound redundant, so removing it needs the user's sign-off. Recommended remedy: keep one wait bounded by STALL_BOUND and delete WATCH_POLL, TAKE_OVER_STOP_BOUND, and the 'stop window' wording. The other option is to keep the code and only fix the docs so they describe the effective bound.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Main acknowledges a wake and the queue is empty; a --take-over lands while the watcher is in a 12s pane capture; the episode stays acked:handling and no check: rearm-resurface is printed ✅ pass live live-head-capture12s.log (base contrast: live-base-capture12s.log shows pending:downtime and then check: rearm-resurface)
Reproduce the reported loop on base 99da16d: the take-over gives up, the arm plain-attaches, the dying watcher reopens downtime, and the next re-arm re-announces check: rearm-resurface ✅ pass live live-base-capture12s.log: ledger reads signal-exit, then attached-cycle-ended, then actionable-check
Greptile P1: with FM_POLL=1, a pane capture of about 30s runs far past the poll window, and the take-over still restores the acknowledged marker and starts a quiet fresh cycle ✅ pass live live-head-capture30s.log; test_take_over_restores_after_a_capture_that_outlives_the_poll_interval passes on HEAD and fails on base
Adversarial: a watcher that outlives the stall bound does not hang the take-over; it gives up at the bound and plain-attaches ✅ pass live live-head-outlives-stall-bound.log (attached after 6s with FM_WATCHER_STALL_BOUND=6)
Guards still hold: a take-over still surfaces queued work and self-exit downtime, and does not stop a cycle the named arm does not own ⏸️ untested no The prior payload did not establish a live result for this scenario. It was checked only by the repository tests (test_take_over_owns_a_fresh_cycle_and_keeps_queued_work_surfacing, test_take_over_pres…
  • bash &lt;runner sourcing tests/fm-watch-arm.test.sh&gt; test_take_over_waits_out_a_slow_stop_and_keeps_the_episode_acknowledged test_take_over_restores_after_a_capture_that_outlives_the_poll_interval test_take_over_owns_a_fresh_cycle_and_keeps_queued_work_surfacing test_take_over_preserves_downtime_from_watcher_self_exit test_take_over_attaches_to_a_cycle_the_named_arm_does_not_own on HEAD: all 5 ok
  • Same two new regression tests with base git show 99da16d9:bin/fm-watch-arm.sh swapped in temporarily (restored afterwards): both fail ('--take-over did not own a fresh cycle ... attached pid=... cycle ended without an actionable reason')
  • Live: live-takeover-scenario.sh &lt;base bin&gt; 12 in a disposable lab FM_HOME (fm-lab-home.sh create) with a real tmux crew pane on a private fm-lab socket, a 12s capture delay, then main's drain/ack, then fm-watch-arm.sh --take-over &lt;owner&gt;, then the host's follow-up re-arm
  • Live: same scenario on HEAD with a 12s capture
  • Live: same scenario on HEAD with FM_POLL=1 and a 30s capture (Greptile P1)
  • Live adversarial: HEAD with FM_WATCHER_STALL_BOUND=6 and a 25s capture (the watcher outlives the bound)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High risk] Changes watcher stop-and-takeover timing in the process supervisor.

The PR appears safe to merge; no new actionable issue was identified.

Reviews (2) · Last reviewed commit: "no-mistakes(review): Collapse take-over ..." · Reviewed by Greptile

Comment thread bin/fm-watch-arm.sh Outdated
@Vela3202019
Vela3202019 force-pushed the fm/firstmate-resurface-loop branch from 38dd791 to 960390a Compare October 7, 2026 06:31
@Vela3202019 Vela3202019 changed the title fix(bin): wait out slow take-over stops so acknowledged downtime stays retired fix(bin): keep acknowledged downtime retired when a take-over stop lands late Oct 7, 2026
…s retired

A watcher TERM'd mid-poll while blocked inside a pane-capture command
substitution defers its exit until the substitution completes, which can
outlive the take-over's old fixed five-second stop window. The abandoned
stop never ran the handover restore, so the dying watcher's downtime
publication minted a fresh pending generation from the acknowledged marker
and every later park re-announced 'check: rearm-resurface' to main.

The stop window now covers the slowest legitimate poll block: the
watcher's check timeout plus margin (FM_ARM_TAKE_OVER_STOP_BOUND), so the
stop is confirmed, the restore runs, and the fresh cycle stays quiet.

Regression: a slow-capture take-over keeps the acknowledged episode
retired and owns a fresh cycle (fails without the fix with the attached
cycle ending without an actionable reason).
…the stop window

A pane capture can run past the take-over stop window (FM_POLL + 25).
The stop is already in flight and the death is guaranteed once the capture
completes, so the taking arm now keeps waiting for the watcher to exit up
to the stall bound instead of abandoning the stop: the handover restore
still runs and the acknowledged downtime episode stays retired instead of
being re-announced on the next park.

Regression: a 30s capture against the 26s window derived from FM_POLL=1
fails without the extended wait (attached cycle ending without an
actionable reason) and stays quiet with it.
@Vela3202019
Vela3202019 force-pushed the fm/firstmate-resurface-loop branch from 960390a to 137568d Compare October 12, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant