fix(bin): stop the watcher from missing status signals written during or between cycles

[MrGTV-love]

Intent

I would like all bugs to be fixed so tomorrow can be focused entirely to Vernant and not problems preventning Vernant from getting built.

A recommended architecture should come with proof.

Context (the backlog record for this bug, as filed):

Watcher did not signal a scout's done event written while it was running (repo: firstmate) (kind: ship) (priority: 1) (since 2026-10-08)
Evidence 2026-10-08: fm-backlog-triage-20261008 (scout) appended 'done [at=1791490597]' (15:16:37 CDT) to state/fm-backlog-triage-20261008.status while watcher pid 15556 ran 14:49:57-15:23:41 (state/.watch-cycle-exits.log reason=actionable-check, delivered 'check: rearm-resurface'); state/.watch-deliveries.log has no signal for it, and the next drain (seq 23944) showed no wake annotation or unread status for it. Same for fm-nm-agent-pipe-buffer 'done [at=1791488399]' (14:39:59, during a Main handling turn between watchers 46565 and 15556). Main learned of both only by polling crew state ~15:30. Diagnose (diagnostic-reasoning): baseline-at-start race for events written between cycles, and/or the in-cycle classifier skipping these done lines; fix the root and add a behavioral test for both shapes. mode=no-mistakes yolo=off.
More evidence (same day): vernant-d2-custody 'needs-decision [key=main996-preservation-diagnosis] [at=1791488659]' (14:44), plus d2 key main992-gre2023-cleanup and d3 key analytics-gre1948-native-validation blocked lines, reached Main only through the open-loop ledger at ~15:38, never as signal wakes. So keyed needs-decision and blocked lines are also missed, not only done lines. Raise priority: Main is blind to lane decisions while this persists.

What Changed

• bin/fm-watch.sh now scans status signals (done, keyed needs-decision, blocked) at the start of each cycle, before the maintenance work. The maintenance work (ledger, reconcile, replies, liveness, relaunch, stall, process-event, recovery, inactive-outcome, due checks) moves into a checkpointed prelude_phase. state/.prelude-progress records the next step. After three signal wakes in a row put maintenance off, the next cycle runs maintenance first. Due checks and secondmate stall rows no longer exit on the first actionable result. The watcher queues each result and wakes once with the first reason.
• Before any non-signal wake exits, the watcher scans signals again. It also scans between blocking checks, after the check batch, and before it takes the wake-queue lock for process events. It queues any new status lines with that wake. The wake keeps its original headline and routing. Late signal files are saved in state/.watch-late-signals. The next cycle sends them as their own signal: wake, but only if their rows are still queued. If the marker cannot be saved, the signal wake goes out at once. wake() in bin/fm-push-transition-lib.sh gets before-output and after-output hooks for this. The grace sleep is skipped while the watcher holds the wake-queue lock.
• docs/configuration.md, docs/architecture.md, and the fm-merge-outcome-lib.sh header now describe the new scheduling and delivery order. tests/fm-watch-triage.test.sh adds regression tests for these cases: done and keyed lines written between and during cycles, a signal stuck behind a blocked check, routing of late signals, heartbeat scope, filtering of drained rows, output-failure retries, marker failure before process events, no grace sleep under the queue lock, an overdue ledger, the deferral limit, and a bad deferral marker.

Risk Assessment

✅ Low: The round-1 lock fix is two small lines that match the user's instruction, the grace sleep no longer runs under the queue lock on either race shape, the new test is behavioral and fails without the fix, and the only new finding is a test-coverage shift.

Testing

I drove six live lab scenarios with the real watcher and the real Main drain. Each one ran in its own disposable FM_HOME lab. They prove the fix for the reported bug. Done lines, keyed needs-decision lines, and blocked lines now reach Main as signal wakes, whether they are written between watcher cycles or during a cycle. A process-event wake no longer holds the wake-queue lock during the signal grace. A concurrent real queue writer waited at most 627ms against a 6000ms grace, and the late signal reached Main on the next arm. The new deterministic lock test passed 10 of 10 isolated runs under heavy host load (load average 46-71). It fails on the pre-F1 code and on a half fix, so it checks both parts of F1. The 10 targeted late-signal regression tests all passed. The test-determinism scenario (T1) is a test-suite property, not a live product run, so it is recorded as untested for live validation. The run produced no UI change, so there are no screenshots. The evidence is CLI transcripts. All labs and temporary trees were removed, and the worktree is clean.

• Live validation: ✅ go - 6 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Scout appends 'done [at=...]' while no watcher runs; the next armed watcher emits a signal wake and Main's drain shows the done line	✅ pass	live	r2-live-between-target.log: arm2 'signal: .../scout-a.status'; drain annotation 'done [at=1791588358]: backlog triage finished'
Vernant lanes append keyed needs-decision and keyed blocked lines; Main receives signal wakes and OPEN DECISIONS for both keys	✅ pass	live	r2-live-keyed-target.log: arm2 signal for vernant-d2/d3; drain lists main996-preservation-diagnosis and analytics-gre1948-native-validation
Scout appends done while the watcher runs a slow quiet check; the same watcher emits a signal wake, and the next arm stays quiet (no duplicate)	✅ pass	live	r2-live-incycle-quiet-target.log: arm2 'signal: .../scout-a.status', drain shows done; arm3 emitted nothing before its 90s timeout
Scout appends done while a slow check reports a result; Main gets both the check row and the done signal row in the same drain	✅ pass	live	r2-live-incycle-loud-target.log: drain rows seq 4 check + seq 6 signal with done annotation
Chatty fleet: a due check reports every round and a status line lands every round; signal wakes still reach Main (base never delivered one in 6 rounds)	✅ pass	live	r2-live-chatty-target.log: arm2, arm3, arm4, arm6 emit signal; prior base log live-chatty-base.log emitted only check wakes
Adversarial F1: a status line lands in the race window beside a queued process-event wake; a concurrent real queue writer never waits on the 6s grace, and the signal reaches Main on the next arm	✅ pass	live	r2-live-lockgrace-target.log: 24 appends, max lock wait 627ms vs 6000ms grace; signal row queued; arm3 'signal: .../scout-a.status'
T1: the lock-grace test proves the rule by mechanism; it passes 10/10 under heavy load and fails on the pre-F1 code and on a half fix	⏸️ untested	no	The prior payload did not establish a live result for this scenario: it was recorded with live=false. It is a test-suite determinism property driven through tests/fm-watch-triage.test.sh with a sleep…

Evidence: Deterministic lock test 10/10 under load on target

Source: Deterministic lock test 10/10 under load on target

commit fab06b960a4023eb10a75d7721900325e752a32e test_procevent_wake_never_lingers_under_the_queue_lock, 10 isolated runs
run 1 rc=0 31s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 46.49 46.59 48.66
run 2 rc=0 32s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 48.47 47.08 48.75
run 3 rc=0 33s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 57.00 49.41 49.54
run 4 rc=0 35s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 65.76 52.38 50.62
run 5 rc=0 35s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 71.35 55.37 51.78
run 6 rc=0 36s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 71.22 57.39 52.68
run 7 rc=0 37s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 68.18 58.35 53.25
run 8 rc=0 35s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 61.99 57.82 53.26
run 9 rc=0 36s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 64.00 59.09 53.92
run 10 rc=0 29s | ok - a process-event wake never holds the queue lock across the signal grace and still delivers the signal  | load averages: 61.00 58.66 53.93

Evidence: Lock test fails on pre-F1 code

Source: Lock test fails on pre-F1 code

pre-F1 product code (51774c6b bin/fm-watch.sh) + current test file; expect FAIL
run 1 rc=1 | not ok - before-lock slept the signal grace while holding the wake-queue lock 
run 2 rc=1 | not ok - before-lock slept the signal grace while holding the wake-queue lock 
run 3 rc=1 | not ok - before-lock slept the signal grace while holding the wake-queue lock 

Evidence: Lock test fails on half fix (grace under held lock)

Source: Lock test fails on half fix (grace under held lock)

half fix: flush moved before lock, but grace still slept under held lock (only the under-lock skip reverted); expect FAIL on under-lock shape
3227c3227
<     [ "$signal_append" = fm_wake_append_locked ] || sleep "$SIGNAL_GRACE"
---
>     sleep "$SIGNAL_GRACE"
rc=1 | not ok - under-lock slept the signal grace while holding the wake-queue lock 

Evidence: Live: done line written between cycles reaches Main

Source: Live: done line written between cycles reaches Main

[18:25:51] code=~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK scenario=between lab=~/tmp/fm-lab.fuwQXI
[18:25:55] arm1-baseline exit=0 output:
    watcher: started pid=70111 (beacon fresh)
    signal: ~/tmp/fm-lab.fuwQXI/state/scout-a.status
[18:25:57] arm1 Main drain (stdout):
    1791588354	2	signal	scout-a.status	signal: ~/tmp/fm-lab.fuwQXI/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: working: setup
[18:25:58] arm1 acked through seq 2
[18:25:58] no watcher running: scout appends done line
[18:26:01] arm2 exit=0 output:
    watcher: started pid=90073 (beacon fresh)
    signal: ~/tmp/fm-lab.fuwQXI/state/scout-a.status
[18:26:01] wake queue rows:
    3	signal	scout-a.status	signal: ~/tmp/fm-lab.fuwQXI/state/scout-a.status
    4	signal	scout-a.status	signal: ~/tmp/fm-lab.fuwQXI/state/scout-a.status
[18:26:04] arm2 Main drain (stdout):
    1791588361	4	signal	scout-a.status	signal: ~/tmp/fm-lab.fuwQXI/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: done [at=1791588358]: backlog triage finished
    STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
    scout-a done [at=1791588358]: backlog triage finished
[18:26:04] arm2 acked through seq 4
[18:26:04] lab removed

Evidence: Live: keyed needs-decision and blocked lines reach Main

Source: Live: keyed needs-decision and blocked lines reach Main

[18:26:04] code=~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK scenario=keyed lab=~/tmp/fm-lab.SrHJQa
[18:26:08] arm1-baseline exit=0 output:
    watcher: started pid=6785 (beacon fresh)
    signal: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
[18:26:10] arm1 Main drain (stdout):
    1791588367	3	signal	vernant-d2.status	signal: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
    1791588367	4	signal	vernant-d3.status	signal: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
    wake annotation: latest wake-EVENT observed at drain, not current state: vernant-d2.status: working: setup
    wake annotation: latest wake-EVENT observed at drain, not current state: vernant-d3.status: working: setup
[18:26:11] arm1 acked through seq 4
[18:26:11] no watcher running: lanes append keyed needs-decision and blocked lines
[18:26:16] arm2 exit=0 output:
    watcher: started pid=26398 (beacon fresh)
    signal: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
[18:26:16] wake queue rows:
    5	signal	vernant-d2.status	needs-decision: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
    6	signal	vernant-d3.status	signal: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
    7	signal	vernant-d2.status	needs-decision: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
    8	signal	vernant-d3.status	signal: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
[18:26:19] arm2 Main drain (stdout):
    1791588375	7	signal	vernant-d2.status	needs-decision: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
    1791588375	8	signal	vernant-d3.status	signal: ~/tmp/fm-lab.SrHJQa/state/vernant-d2.status ~/tmp/fm-lab.SrHJQa/state/vernant-d3.status
    wake annotation: latest wake-EVENT observed at drain, not current state: vernant-d2.status: needs-decision [key=main996-preservation-diagnosis] [at=1791588371]: preserve or drop?
    wake annotation: latest wake-EVENT observed at drain, not current state: vernant-d3.status: blocked [key=analytics-gre1948-native-validation]: need native validation access
    OPEN DECISIONS (still open, folded from the durable status logs - not just the latest line):
    vernant-d2 [key=main996-preservation-diagnosis] needs-decision: preserve or drop?
    vernant-d3 [key=analytics-gre1948-native-validation] blocked: need native validation access
    OPEN DECISIONS: close one by answering it: bin/fm-send.sh <task> --resolve-key <key> '<answer>'
[18:26:19] arm2 acked through seq 8
[18:26:19] lab removed

Evidence: Live: done during quiet slow check reaches Main

Source: Live: done during quiet slow check reaches Main

[18:26:20] code=~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK scenario=incycle-quiet lab=~/tmp/fm-lab.q6tOhF
registered: state/slow.check.sh
[18:26:24] arm1-baseline exit=0 output:
    watcher: started pid=44408 (beacon fresh)
    signal: ~/tmp/fm-lab.q6tOhF/state/scout-a.status
[18:26:26] arm1 Main drain (stdout):
    1791588383	2	signal	scout-a.status	signal: ~/tmp/fm-lab.q6tOhF/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: working: setup
[18:26:27] arm1 acked through seq 2
[18:26:27] settle: let Main consume the first open-loop-ledger wake
[18:26:29] settle exit=0 output:
    watcher: started pid=63825 (beacon fresh)
    check: open-loop-ledger (1 overdue assigned obligations; read state/open-loops.json)
[18:26:36] watcher running a slow check: scout appends done line
[18:26:44] arm2 exit=0 output:
    watcher: started pid=83798 (beacon fresh)
    signal: ~/tmp/fm-lab.q6tOhF/state/scout-a.status
[18:26:44] wake queue rows:
    4	signal	scout-a.status	signal: ~/tmp/fm-lab.q6tOhF/state/scout-a.status
    5	signal	scout-a.status	signal: ~/tmp/fm-lab.q6tOhF/state/scout-a.status
[18:26:47] arm2 Main drain (stdout):
    1791588403	5	signal	scout-a.status	signal: ~/tmp/fm-lab.q6tOhF/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: done [at=1791588396]: backlog triage finished
    STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
    scout-a done [at=1791588396]: backlog triage finished
[18:26:47] arm2 acked through seq 5
[18:26:47] next arm after Main handled the batch:
[18:28:18] arm3 exit=124 output:
    watcher: started pid=22225 (beacon fresh)
[18:28:18] lab removed

Evidence: Live: done during actionable slow check reaches Main with the check result

Source: Live: done during actionable slow check reaches Main with the check result

[18:28:18] code=~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK scenario=incycle-loud lab=~/tmp/fm-lab.tJ2K8h
registered: state/slow.check.sh
[18:28:21] arm1-baseline exit=0 output:
    watcher: started pid=86167 (beacon fresh)
    signal: ~/tmp/fm-lab.tJ2K8h/state/scout-a.status
[18:28:23] arm1 Main drain (stdout):
    1791588500	2	signal	scout-a.status	signal: ~/tmp/fm-lab.tJ2K8h/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: working: setup
[18:28:23] arm1 acked through seq 2
[18:28:23] settle: let Main consume the first open-loop-ledger wake
[18:28:25] settle exit=0 output:
    watcher: started pid=7134 (beacon fresh)
    check: open-loop-ledger (1 overdue assigned obligations; read state/open-loops.json)
[18:28:32] watcher running a slow check: scout appends done line
[18:28:40] arm2 exit=0 output:
    watcher: started pid=26139 (beacon fresh)
    check: ~/tmp/fm-lab.tJ2K8h/state/slow.check.sh: slow: upstream PR merged
[18:28:40] wake queue rows:
    4	check	~/tmp/fm-lab.tJ2K8h/state/slow.check.sh	check: ~/tmp/fm-lab.tJ2K8h/state/slow.check.sh: slow: upstream PR merged
    5	signal	scout-a.status	signal: ~/tmp/fm-lab.tJ2K8h/state/scout-a.status
    6	signal	scout-a.status	signal: ~/tmp/fm-lab.tJ2K8h/state/scout-a.status
[18:28:41] arm2 Main drain (stdout):
    1791588517	4	check	~/tmp/fm-lab.tJ2K8h/state/slow.check.sh	check: ~/tmp/fm-lab.tJ2K8h/state/slow.check.sh: slow: upstream PR merged
    1791588519	6	signal	scout-a.status	signal: ~/tmp/fm-lab.tJ2K8h/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: done [at=1791588512]: backlog triage finished
    STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
    scout-a done [at=1791588512]: backlog triage finished
[18:28:41] arm2 acked through seq 6
[18:28:41] next arm after Main handled the batch:
[18:30:11] arm3 exit=124 output:
    watcher: started pid=59390 (beacon fresh)
    ~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK/bin/fm-watch.sh: child setpgid (50970 to 50970): Operation not permitted
[18:30:12] lab removed

Evidence: Live: chatty fleet with loud check every round still gets signal wakes

Source: Live: chatty fleet with loud check every round still gets signal wakes

[18:30:12] code=~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK scenario=chatty lab=~/tmp/fm-lab.Rhb1vf
registered: state/slow.check.sh
[18:30:15] arm0-baseline exit=0 output:
    watcher: started pid=14870 (beacon fresh)
    signal: ~/tmp/fm-lab.Rhb1vf/state/scout-a.status
[18:30:16] arm0 Main drain (stdout):
    1791588614	2	signal	scout-a.status	signal: ~/tmp/fm-lab.Rhb1vf/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: working: setup
[18:30:17] arm0 acked through seq 2
[18:30:17] settle: let Main consume the first open-loop-ledger wake
[18:30:18] settle exit=0 output:
    watcher: started pid=29211 (beacon fresh)
    check: open-loop-ledger (1 overdue assigned obligations; read state/open-loops.json)
[18:30:22] round 1: fleet appended a status line; a due check also reports every round
[18:30:26] arm1 exit=0 output:
    watcher: started pid=41464 (beacon fresh)
    check: ~/tmp/fm-lab.Rhb1vf/state/slow.check.sh: slow: upstream PR merged
[18:30:29] round 2: fleet appended a status line; a due check also reports every round
[18:30:32] arm2 exit=0 output:
    watcher: started pid=59332 (beacon fresh)
    signal: ~/tmp/fm-lab.Rhb1vf/state/scout-a.status
[18:30:37] round 3: fleet appended a status line; a due check also reports every round
[18:30:41] arm3 exit=0 output:
    watcher: started pid=77126 (beacon fresh)
    signal: ~/tmp/fm-lab.Rhb1vf/state/scout-a.status
[18:30:46] round 4: fleet appended a status line; a due check also reports every round
[18:30:50] arm4 exit=0 output:
    watcher: started pid=1952 (beacon fresh)
    signal: ~/tmp/fm-lab.Rhb1vf/state/scout-a.status
[18:30:54] round 5: fleet appended a status line; a due check also reports every round
[18:30:59] arm5 exit=0 output:
    watcher: started pid=24573 (beacon fresh)
    check: ~/tmp/fm-lab.Rhb1vf/state/slow.check.sh: slow: upstream PR merged
[18:31:04] round 6: fleet appended a status line; a due check also reports every round
[18:31:07] arm6 exit=0 output:
    watcher: started pid=48723 (beacon fresh)
    signal: ~/tmp/fm-lab.Rhb1vf/state/scout-a.status
[18:31:10] lab removed

Evidence: Live: process-event wake does not block queue writers on grace; signal delivered next arm

Source: Live: process-event wake does not block queue writers on grace; signal delivered next arm

[18:31:10] code=~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK scenario=lockgrace lab=~/tmp/fm-lab.5KUjXs
[18:31:13] arm1-baseline exit=0 output:
    watcher: started pid=65388 (beacon fresh)
    signal: ~/tmp/fm-lab.5KUjXs/state/scout-a.status
[18:31:15] arm1 Main drain (stdout):
    1791588672	2	signal	scout-a.status	signal: ~/tmp/fm-lab.5KUjXs/state/scout-a.status
    wake annotation: latest wake-EVENT observed at drain, not current state: scout-a.status: working: setup
[18:31:16] arm1 acked through seq 2
[18:31:16] settle: let Main consume the first open-loop-ledger wake
[18:31:18] settle exit=0 output:
    watcher: started pid=81387 (beacon fresh)
    check: open-loop-ledger (1 overdue assigned obligations; read state/open-loops.json)
[18:31:20] queued one process-event result; arming with a 6s signal grace
[18:31:21] status line landed in the step-6..7 race window (injected=yes)
[18:31:29] arm2 exit=0 output:
    watcher: started pid=94606 (beacon fresh)
    check: process-event result captured: procevent:live-lab:1
[18:31:29] concurrent queue writer: 24 appends, max lock wait 627ms (signal grace = 6000ms)
    signal row queued: scout-a.status
    signal row queued: scout-a.status
[18:31:30] arm3 exit=0 output:
    watcher: started pid=16387 (beacon fresh)
    signal: ~/tmp/fm-lab.5KUjXs/state/scout-a.status
[18:31:30] lab removed

Evidence: Targeted late-signal regression tests on target

Source: Targeted late-signal regression tests on target

commit fab06b960a4023eb10a75d7721900325e752a32e
=== test_done_and_keyed_lines_surface_between_and_during_cycles: ok - done, keyed needs-decision, and keyed blocked lines surface whether written between cycles or during one 
=== test_signal_not_held_behind_a_blocked_check: ok - done and keyed decision signals appended during quiet and actionable checks are delivered 
=== test_flushed_signals_preserve_main_owned_routing: ok - late signals keep original stale routing and wake once independently, including Main-owned stale rows 
=== test_heartbeat_late_signal_retains_fleet_scope: ok - a late routine signal preserves heartbeat fleet scope and gets one task-scoped follow-up 
=== test_late_signal_follow_up_filters_drained_rows: ok - follow-ups filter drained rows and marker-store failures preserve one-shot check results in successor posture 
=== test_late_signal_output_failure_retries: ok - follow-up and end-of-prelude output failures preserve signal markers until one successful successor ring 
=== test_late_signal_marker_failure_before_procevent: ok - late marker failures deliver signals and leave interrupted process-event wakes owed 
=== test_signal_not_preempted_by_an_overdue_ledger: ok - the signal wake goes first and the overdue ledger surfaces on the next cycle 
=== test_cycle_work_runs_after_bounded_signal_deferrals: ok - maintenance debt survives early wakes until both stalled mates, process-events, and every due check run 
=== test_invalid_prelude_deferral_marker_restores_the_previous_order: ok - an invalid deferral marker restores the previous cycle order 

Evidence: Live driver script

Source: Live driver script

#!/usr/bin/env bash
# Live driver: run the real firstmate watcher (bin/fm-watch-arm.sh -> fm-watch.sh)
# and the real Main drain (bin/fm-wake-drain.sh) against a disposable lab home.
# Usage: driver.sh <code-dir> <scenario>
set -u
CODE=$1 SCEN=$2
LAB=$(mktemp -d "${TMPDIR:-/tmp}/fm-lab.XXXXXX")
"$CODE/bin/fm-lab-home.sh" create "$LAB" >/dev/null 2>&1 \
  || ~/.no-mistakes/worktrees/32d18ed9638d/01M4H906AJDNT83YE1DPHM1VRK/bin/fm-lab-home.sh create "$LAB" >/dev/null || exit 1
ST="$LAB/state"
mkdir -p "$LAB/projects/demo" "$LAB/drv"
CLEAN_ENV=(env -u FM_GATE_REFUSE_BYPASS -u FM_ROOT_OVERRIDE -u FM_STATE_OVERRIDE -u FM_DATA_OVERRIDE
  -u FM_CONFIG_OVERRIDE -u FM_PROJECTS_OVERRIDE FM_HOME="$LAB" FM_POLL=1 FM_SIGNAL_GRACE=1 FM_HEARTBEAT=999999)
ts() { date +%H:%M:%S; }
log() { printf '[%s] %s\n' "$(ts)" "$*"; }
task() { printf 'project=%s/projects/demo\nkind=%s\n' "$LAB" "$2" > "$ST/$1.meta"; printf 'working: setup\n' > "$ST/$1.status"; }
arm_fg() {  # <label> [env...]
  local label=$1; shift
  (cd "$CODE" && "${CLEAN_ENV[@]}" "$@" timeout 90 bin/fm-watch-arm.sh) > "$LAB/drv/$label.out" 2>&1
  log "$label exit=$? output:"; sed 's/^/    /' "$LAB/drv/$label.out"
}
arm_bg() {  # <label> [env...]
  local label=$1; shift
  (cd "$CODE" && "${CLEAN_ENV[@]}" "$@" timeout 90 bin/fm-watch-arm.sh) > "$LAB/drv/$label.out" 2>&1 &
  ARM_PID=$!
}
drain() {  # <label> : Main's real drain + acknowledgement
  local label=$1 seq gen
  "${CLEAN_ENV[@]}" "$CODE/bin/fm-wake-drain.sh" > "$LAB/drv/$label.drain" 2> "$LAB/drv/$label.drain.err"
  log "$label Main drain (stdout):"; sed 's/^/    /' "$LAB/drv/$label.drain"
  seq=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation .*$/\1/p' "$LAB/drv/$label.drain.err")
  gen=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$LAB/drv/$label.drain.err")
  if [ -n "$seq" ] && [ -n "$gen" ]; then
    "${CLEAN_ENV[@]}" "$CODE/bin/fm-wake-drain.sh" --ack-through "$seq" --recovery-generation "$gen" >/dev/null 2>&1
    log "$label acked through seq $seq"
  fi
}
settle() {  # consume the one-time first open-loop-ledger wake so later cycles reach the step under test
  log "settle: let Main consume the first open-loop-ledger wake"
  arm_fg settle "$@"; drain settle >/dev/null
}
queue() { log "wake queue rows:"; cut -f2- "$ST/.wake-queue" 2>/dev/null | sed 's/^/    /'; }
register_check() {  # check driven by $LAB/drv/check-mode
  cat > "$ST/slow.check.sh" <<SH
#!/usr/bin/env bash
mode=\$(cat "$LAB/drv/check-mode" 2>/dev/null)
case "\$mode" in
  slow-quiet) touch "$LAB/drv/check-started"; sleep 6 ;;
  slow-loud) touch "$LAB/drv/check-started"; sleep 6; echo 'slow: upstream PR merged'; echo off > "$LAB/drv/check-mode" ;;
  loud) echo 'slow: upstream PR merged' ;;
esac
exit 0
SH
  chmod 0700 "$ST/slow.check.sh"
  "${CLEAN_ENV[@]}" "$CODE/bin/fm-check-register.sh" slow || log "check registration FAILED"
}

log "code=$CODE scenario=$SCEN lab=$LAB"
case "$SCEN" in
  between)
    task scout-a scout
    arm_fg arm1-baseline; drain arm1
    log "no watcher running: scout appends done line"
    printf 'done [at=%s]: backlog triage finished\n' "$(date +%s)" >> "$ST/scout-a.status"
    arm_fg arm2; queue; drain arm2
    ;;
  keyed)
    task vernant-d2 ship; task vernant-d3 ship
    arm_fg arm1-baseline; drain arm1
    log "no watcher running: lanes append keyed needs-decision and blocked lines"
    printf 'needs-decision [key=main996-preservation-diagnosis] [at=%s]: preserve or drop?\n' "$(date +%s)" >> "$ST/vernant-d2.status"
    printf 'blocked [key=analytics-gre1948-native-validation]: need native validation access\n' >> "$ST/vernant-d3.status"
    arm_fg arm2; queue; drain arm2
    ;;
  incycle-quiet|incycle-loud)
    task scout-a scout; register_check; echo off > "$LAB/drv/check-mode"
    arm_fg arm1-baseline FM_CHECK_INTERVAL=1; drain arm1
    settle FM_CHECK_INTERVAL=1
    echo "slow-${SCEN#incycle-}" > "$LAB/drv/check-mode"; rm -f "$LAB/drv/check-started"; sleep 2
    arm_bg arm2 FM_CHECK_INTERVAL=1
    i=0; while [ ! -e "$LAB/drv/check-started" ] && [ $i -lt 600 ]; do sleep 0.1; i=$((i+1)); done
    sleep 1
    log "watcher running a slow check: scout appends done line"
    printf 'done [at=%s]: backlog triage finished\n' "$(date +%s)" >> "$ST/scout-a.status"
    wait "$ARM_PID"; log "arm2 exit=$? output:"; sed 's/^/    /' "$LAB/drv/arm2.out"
    queue; drain arm2
    log "next arm after Main handled the batch:"
    arm_fg arm3 FM_CHECK_INTERVAL=1
    ;;
  chatty)
    task scout-a scout; register_check; echo off > "$LAB/drv/check-mode"
    arm_fg arm0-baseline FM_CHECK_INTERVAL=1; drain arm0
    settle FM_CHECK_INTERVAL=1
    echo loud > "$LAB/drv/check-mode"
    for r in 1 2 3 4 5 6; do
      sleep 2
      printf 'blocked [key=k%s]: round %s\n' "$r" "$r" >> "$ST/scout-a.status"
      log "round $r: fleet appended a status line; a due check also reports every round"
      arm_fg "arm$r" FM_CHECK_INTERVAL=1; drain "arm$r" >/dev/null
    done
    ;;
  lockgrace)
    task scout-a scout
    arm_fg arm1-baseline; drain arm1
    settle
    mkdir -p "$LAB/drv/shim"; real_mv=$(command -v mv)
    cat > "$LAB/drv/shim/mv" <<SH
#!/usr/bin/env bash
dest=\${!#}
"$real_mv" "\$@" || exit
if [ "\$dest" = "$ST/.prelude-progress" ] && [ ! -e "$LAB/drv/injected" ] && [ "\$(cat "\$dest")" = 7 ]; then
  printf 'done: finished beside process-event delivery\n' >> "$ST/scout-a.status"; touch "$LAB/drv/injected"
fi
SH
    chmod +x "$LAB/drv/shim/mv"
    "${CLEAN_ENV[@]}" bash -c '. "$1/bin/fm-wake-lib.sh"; fm_wake_append check procevent:live-lab:1 "check: process-event fixture"' _ "$CODE"
    log "queued one process-event result; arming with a 6s signal grace"
    arm_bg arm2 PATH="$LAB/drv/shim:$PATH" FM_SIGNAL_GRACE=6
    i=0; while [ ! -e "$LAB/drv/injected" ] && [ $i -lt 600 ]; do sleep 0.1; i=$((i+1)); done
    log "status line landed in the step-6..7 race window (injected=$([ -e "$LAB/drv/injected" ] && echo yes || echo no))"
    ( while kill -0 "$ARM_PID" 2>/dev/null; do
        if [ -e "$ST/.wake-queue.lock" ]; then hp=$(cat "$ST/.wake-queue.lock/pid" 2>/dev/null)
          printf '%s holder=%s %s\n' "$(perl -MTime::HiRes=time -e 'printf "%.1f", time')" "$hp" "$(ps -o command= -p "$hp" 2>/dev/null | cut -c1-80)"; fi
        sleep 0.2; done ) > "$LAB/drv/holders" 2>&1 &
    max=0 n=0
    while kill -0 "$ARM_PID" 2>/dev/null && [ $n -lt 400 ]; do
      n=$((n+1)); t0=$(perl -MTime::HiRes=time -e 'printf "%.0f", time*1000')
      "${CLEAN_ENV[@]}" bash -c '. "$1/bin/fm-wake-lib.sh"; fm_wake_append check "queue-writer:$2" "check: concurrent queue writer"' _ "$CODE" "$n"
      t1=$(perl -MTime::HiRes=time -e 'printf "%.0f", time*1000'); w=$((t1-t0)); [ $w -le $max ] || max=$w; [ $w -lt 2000 ] || log "slow append #$n waited ${w}ms ending $(perl -MTime::HiRes=time -e 'printf "%.1f", time')"
      sleep 0.05
    done
    wait "$ARM_PID"; log "arm2 exit=$? output:"; sed 's/^/    /' "$LAB/drv/arm2.out"
    log "concurrent queue writer: $n appends, max lock wait ${max}ms (signal grace = 6000ms)"
    if [ "$max" -ge 2000 ]; then log "lock holder samples:"; sed 's/^/    /' "$LAB/drv/holders"; fi
    awk -F '\t' '$3=="signal"{print "    signal row queued: " $4}' "$ST/.wake-queue"
    arm_fg arm3
    ;;
esac
(cd "$CODE" && "${CLEAN_ENV[@]}" bin/fm-watch-arm.sh --stop >/dev/null 2>&1)
rm -rf "$LAB"
log "lab removed"

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (1h29m50s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

• ⚠️ bin/fm-watch.sh:2133 - The new pre-output signal flush runs while procevent_surface_queued still holds FM_WAKE_QUEUE_LOCK. Sequence: a process-event row is queued, and a status line is appended between the first signal scan and prelude step 7. procevent_surface_queued takes the queue lock at :2102 and calls wake at :2133. watch_before_wake (:2943-2947) runs signal_phase_flush, so signal_phase finds the pending file and sleeps SIGNAL_GRACE (:3226, default 30s). It then runs signal classification (crew-state lookup, bounded no-mistakes call, pane capture), all with the queue lock still held. Before this change the process-event wake printed and released the lock at once. Effects: (1) a concurrent presentation drain gives up after FM_STATUS_PRESENTATION_LOCK_TIMEOUT (default 10s) with 'WAKE DRAIN SKIPPED' (bin/fm-wake-drain.sh:837-843). Under Pi this overlap is real, because the successor is armed before the predecessor's wake is delivered and drained (.pi/extensions/fm-primary-pi-watch.ts:1007-1008). (2) Acknowledgement drains (fm-wake-drain.sh:836) and every other queue writer stall for the whole grace period plus classification: fm-inbox.sh:347, fm-mail.sh:378, fm-wake-grant.sh, fm-session-launch-policy-lib.sh:53, fm-contributions.sh:331. The existing fixture (tests/fm-watch-triage.test.sh:2585-2611) checks only that the lock is released afterwards, not how long it is held. Remedy: in procevent_surface_queued, run signal_phase_flush before the lock is taken at :2102. Also skip the grace sleep when the flush runs under the held queue lock (signal_append=fm_wake_append_locked at :3215-3218), so only the small race window is scanned while locked. No other wake site in the changed code calls wake while holding the queue lock.

🔧 Fix applied.
1 info still open:

• ℹ️ tests/fm-watch-triage.test.sh:2548 - The round-1 F1 fix moved this test off its target path. test_late_signal_marker_failure_before_procevent appends the done line immediately after prelude step 6 commits. The new pre-lock flush at bin/fm-watch.sh:2102 now finds that line before the queue lock is taken. The marker failure then exits through signal_phase_note_queued with FM_WAKE_POST_OUTPUT_ACTION empty. The under-lock failure branch at bin/fm-watch.sh:2933-2936 (which calls procevent_surface_after_output 1 to release the lock) no longer runs in any test. The assertion at tests/fm-watch-triage.test.sh:2595 ('signal fallback retained the process-event queue lock') now passes trivially, because the lock is never taken. The branch is still reachable: a status line can land between the pre-lock flush and the flush inside wake. Remedy: add an under-lock injection shape to this test. Use the same ln hook on .wake-queue.lock that test_procevent_wake_never_lingers_under_the_queue_lock uses at tests/fm-watch-triage.test.sh:2632-2651. Keep the existing assertions. No production code change is needed.

🔧 **Test** - 1 issue found → auto-fixed ✅

• ⚠️ tests/fm-watch-triage.test.sh:2675 - test_procevent_wake_never_lingers_under_the_queue_lock is timing-flaky under heavy host load. It failed 2 of 9 isolated runs ('before-lock queue writer waited 5s on the 6s signal grace') at host load average 43-75 and passed the other 7. Six live lab repeats with lock-holder sampling showed concurrent writers acquiring the lock between the watcher's short critical sections. The watcher never held the lock continuously for the grace period. The worst writer wait was 4.4s and the typical wait was about 1s, so the cause is process-spawn latency under load, not grace-under-lock. The <3s bound with whole-second date measurement is close to that latency on a saturated runner. If CI fails it, rerun on a less loaded runner before suspecting the product. Evidence: lock-test-reruns.log, targeted-regression-tests.log, live-lockgrace-repeat.log.
• Live validation: ✅ go - 8 of 9 scenarios driven live against the product

Scenario	Result	Live	Evidence
Scout writes 'done [at=...]' while no watcher runs (between Main turns); the next armed watcher wakes with 'signal: <status>' and Main's drain shows the done event	✅ pass	live	live-between-target.log (arm2 'signal: .../scout-a.status', drain annotation shows done line); live-between-base.log shows the base emitting 'check: open-loop-ledger' with no signal row
Lanes append keyed 'needs-decision [key=...]' and 'blocked [key=...]' lines between watchers; the next watcher wakes with a signal naming both and Main's drain lists them as OPEN DECISIONS	✅ pass	live	live-keyed-target.log (signal headline for vernant-d2 and vernant-d3, needs-decision row reason); live-keyed-base.log reproduces the ledger-only delivery
Scout writes done while the watcher runs a slow quiet check; the same watcher cycle delivers 'signal:'	✅ pass	live	live-incycle-quiet-target.log (arm2 'signal: .../scout-a.status' with drain annotation for the done line)
Adversarial: done lands while a slow check later reports actionable output; the check headline is kept and the signal row is queued in the same wake, so Main's drain sees the done	✅ pass	live	live-incycle-loud-target.log (queue rows 4 check plus 5 and 6 signal, drain annotates done); live-incycle-loud-base.log shows only the check row, with the signal deferred to the next arm
Adversarial: after Main has drained the late-signal batch, the next watcher emits no duplicate follow-up wake	✅ pass	live	live-incycle-loud-target.log arm3 stays quiet until the driver timeout, with no follow-up signal
Late signal queued beside a process-event wake gets an independent 'signal:' follow-up on the next watcher when not yet drained	✅ pass	live	live-lockgrace-target.log arm3 'signal: .../scout-a.status'
Adversarial chatty fleet: status lines arrive every round and a due check reports every round; neither signals nor maintenance or checks are starved (bounded deferral of 3)	✅ pass	live	live-chatty-target.log (check, signal, signal, signal, check, signal); live-chatty-base.log shows only check wakes for all rounds
F1 guard: a process-event wake with a status line in the race window never holds the wake-queue lock across the 6s signal grace; a concurrent queue writer proceeds and the signal is still delivered	✅ pass	live	live-lockgrace-target.log (max wait 938 ms against a 6000 ms grace); live-lockgrace-repeat.log (6 runs, max 954-4444 ms under load average about 50, holder sampling shows writers interleaving between…
Existing automated regressions for between/in-cycle signals, routing, output failure, and bounded deferral	⏸️ untested	no	The prior payload did not establish a live result for this scenario. Its only evidence was hermetic test-harness runs with fake crew-state (targeted-regression-tests.log: 10 of 11 passed on the first…

• FM_TEST_ONLY=<name> bash tests/fm-watch-triage.test.sh for each of the 11 new regressions (test_done_and_keyed_lines_surface_between_and_during_cycles, test_signal_not_held_behind_a_blocked_check, test_flushed_signals_preserve_main_owned_routing, test_heartbeat_late_signal_retains_fleet_scope, test_late_signal_follow_up_filters_drained_rows, test_late_signal_output_failure_retries, test_late_signal_marker_failure_before_procevent, test_procevent_wake_never_lingers_under_the_queue_lock, test_signal_not_preempted_by_an_overdue_ledger, test_cycle_work_runs_after_bounded_signal_deferrals, test_invalid_prelude_deferral_marker_restores_the_previous_order)
• FM_TEST_ONLY=test_procevent_wake_never_lingers_under_the_queue_lock bash tests/fm-watch-triage.test.sh repeated 8 more times to diagnose flakiness
• Live lab driver (live-driver.sh <code> between) on the target and on a git archive 924f479c base export: real bin/fm-watch-arm.sh plus bin/fm-wake-drain.sh against a bin/fm-lab-home.sh create home
• Live live-driver.sh <code> keyed on target and base: keyed needs-decision and blocked lines between watchers
• Live live-driver.sh <code> incycle-quiet and incycle-loud on target and base: done appended during a registered slow custom check (bin/fm-check-register.sh)
• Live live-driver.sh <code> chatty on target and base: 6 rounds of status lines with a check that reports every round
• Live live-driver.sh <target> lockgrace run 7 times: process-event row plus a status line in the step-6/7 race window, a concurrent fm_wake_append writer measuring lock wait, and lock-holder sampling

🔧 Fix applied.
✅ Re-checked - no issues remain.

• Live validation: ✅ go - 6 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Scout appends 'done [at=...]' while no watcher runs; the next armed watcher emits a signal wake and Main's drain shows the done line	✅ pass	live	r2-live-between-target.log: arm2 'signal: .../scout-a.status'; drain annotation 'done [at=1791588358]: backlog triage finished'
Vernant lanes append keyed needs-decision and keyed blocked lines; Main receives signal wakes and OPEN DECISIONS for both keys	✅ pass	live	r2-live-keyed-target.log: arm2 signal for vernant-d2/d3; drain lists main996-preservation-diagnosis and analytics-gre1948-native-validation
Scout appends done while the watcher runs a slow quiet check; the same watcher emits a signal wake, and the next arm stays quiet (no duplicate)	✅ pass	live	r2-live-incycle-quiet-target.log: arm2 'signal: .../scout-a.status', drain shows done; arm3 emitted nothing before its 90s timeout
Scout appends done while a slow check reports a result; Main gets both the check row and the done signal row in the same drain	✅ pass	live	r2-live-incycle-loud-target.log: drain rows seq 4 check + seq 6 signal with done annotation
Chatty fleet: a due check reports every round and a status line lands every round; signal wakes still reach Main (base never delivered one in 6 rounds)	✅ pass	live	r2-live-chatty-target.log: arm2, arm3, arm4, arm6 emit signal; prior base log live-chatty-base.log emitted only check wakes
Adversarial F1: a status line lands in the race window beside a queued process-event wake; a concurrent real queue writer never waits on the 6s grace, and the signal reaches Main on the next arm	✅ pass	live	r2-live-lockgrace-target.log: 24 appends, max lock wait 627ms vs 6000ms grace; signal row queued; arm3 'signal: .../scout-a.status'
T1: the lock-grace test proves the rule by mechanism; it passes 10/10 under heavy load and fails on the pre-F1 code and on a half fix	⏸️ untested	no	The prior payload did not establish a live result for this scenario: it was recorded with live=false. It is a test-suite determinism property driven through tests/fm-watch-triage.test.sh with a sleep…

• FM_TEST_ONLY=test_procevent_wake_never_lingers_under_the_queue_lock bash tests/fm-watch-triage.test.sh x10 on fab06b96 (load avg 46-71): 10/10 ok
• Same test on pre-F1 product code (git archive 51774c6b + current test file) x3: 3/3 not ok (before-lock slept grace under lock)
• Same test on half fix (flush moved before lock, grace sleep under held lock restored): not ok (under-lock slept grace under lock)
• bash live-driver.sh <worktree> between (real arm + drain, disposable FM_HOME lab)
• bash live-driver.sh <worktree> keyed
• bash live-driver.sh <worktree> incycle-quiet
• bash live-driver.sh <worktree> incycle-loud
• bash live-driver.sh <worktree> chatty
• bash live-driver.sh <worktree> lockgrace (concurrent real queue writer + lock-holder sampling)
• FM_TEST_ONLY targeted runs: test_done_and_keyed_lines_surface_between_and_during_cycles, test_signal_not_held_behind_a_blocked_check, test_flushed_signals_preserve_main_owned_routing, test_heartbeat_late_signal_retains_fleet_scope, test_late_signal_follow_up_filters_drained_rows, test_late_signal_output_failure_retries, test_late_signal_marker_failure_before_procevent, test_signal_not_preempted_by_an_overdue_ledger, test_cycle_work_runs_after_bounded_signal_deferrals, test_invalid_prelude_deferral_marker_restores_the_previous_order

✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅

• ⚠️ linter found issues (exit code 1)

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.