fix: keep watcher arms and reply listeners alive through slow cycles

[kunchenguid]

Intent

A

Context: that is the choice to ship parts 1+2+3 in one PR, as recommended by the two watcher-stall scout reports (data/fm-watcher-stall-s1/report.md and data/fm-watcher-cycle-cost-s1/report.md in the fmdev-f1 home). In the main home, watcher poll cycles take 130-270 s (one about 328 s) against a 300 s heartbeat grace, because every cycle walks all ~2,030 never-pruned pending-reply records with per-record locks, forks, and re-sourcing; under load a cycle passed the grace twice on 2026-09-28 and the Stop-hook auto-arm failed each time. The three parts:

1. One-pass record selection in fm_pending_reply_tick, so the pending-reply walk stops costing per-record forks and stops growing with the record count.
2. An attached arm keeps following a live holder until the lock changes or the 3x-grace stall bound, removing the false auto-arm FAILED and the silent lost-wake path.
3. Treat the mini job worker's preemption exit 76 of a reply long-poll as a closed window, so the per-cycle liveness probe stops killing and relaunching the remote-reply listeners.
   Reproduce first in a real setup with a large pending-reply store, and prove each part with observable behavior (cycle time, attach behavior, listener stability). Not approved, follow-ups only: archiving long-resolved records, and decoupling the contributions poll from the check cadence.

What Changed

• Select pending-reply records needing work in one pass, avoiding per-record locks and forks for settled records.
• Keep an attached arm following a live, identity-matched watcher through a stale beacon until the shared stall bound; report a stalled-holder failure at that bound.
• Treat remote reply-poll preemption as a closed window so the listener retains its claim and polls again, without publishing a caught-up watermark. Add regression coverage and update the related documentation.

Risk Assessment

High (review): the review found no code defect. Its one finding, that the real-setup reproduction and cycle-time proof were absent from the diff, was resolved by the lab evidence in "Reproduction and measurements" below; that evidence stays outside the diff by decision.

Testing

No baseline commands were supplied for this phase. The three targeted end-to-end suites passed: settled records were skipped despite a held lock, an attached arm followed a slow watcher and handled a stalled holder, and a preempted remote-reply listener kept its claim without a false caught-up watermark or relaunch. A separate disposable 2,030-record tick took 0.132–0.141 seconds across three runs. The test fixtures were cleaned up; this measures the pending-reply tick, not a full watcher cycle or a before/after comparison.

• Live validation: ✅ go - 4 of 4 scenarios driven live against the product

Scenario	Result	Live	Evidence
Poll a large settled pending-reply store without blocking on a settled record’s lock, while processing records still needing work	✅ pass	live	bash tests/fm-pending-reply.test.sh; 2030-record-cycle.log
Attach to a slow live watcher and receive its wake rather than failing when its beacon passes grace	✅ pass	live	bash tests/fm-watch-arm.test.sh — slow-holder wake scenario
Stop following a holder at the stall bound and let a retry replace it	✅ pass	live	bash tests/fm-watch-arm.test.sh — stalled-holder replacement scenario
Preempt a remote-reply long-poll without publishing false freshness, losing its listener, or relaunching it on reconcile	✅ pass	live	bash tests/fm-remote-reply.test.sh — preemption and listener-stability scenarios

Evidence: 2,030-record pending-reply tick timings

Source: 2,030-record pending-reply tick timings

cycle=1 records=2030 elapsed=0.132s exit=0 stdout='' stderr=''
cycle=2 records=2030 elapsed=0.141s exit=0 stdout='' stderr=''
cycle=3 records=2030 elapsed=0.133s exit=0 stdout='' stderr=''

Reproduction and measurements

Every part was reproduced on the base code (b5fdf74d, a pristine copy) before the fix and re-run on the fix, in disposable homes on the Mac mini (load 5-8); no live home was touched.

Part 1 - one-pass record selection

Lab: 2,030 resolved records (every 7th with a closed escalation) plus 3 needing work.

Measurement	Before	After
One fm_pending_reply_tick, 2,033 records	115.4 s, 119.1 s	0.82 s first tick (does the 3 records' work), 0.43 s steady
Whole watcher cycle through the real fm-watch-arm.sh + fm-watch.sh, FM_POLL=15, 2,030 records	131 s, 136 s	15 s, 16 s, 15 s, 16 s (the poll sleep)
Tick while a live foreign process holds one settled record's per-record lock	still blocked after 60 s; the record needing work was never processed	0.63 s; that record resolved

Old and new ticks on identical 2,033-record labs with a fixed clock left identical record and status files; the only differences were the wall-clock [at=] stamp on the appended escalation close and a file identity.

Part 2 - attached arm follows a live holder

Real Stop hook -> supervision host -> arm -> watcher under a fake claude session (the scouts' Appendix A harness), grace scaled to 20 s: old attach give-up at 31 s, stall bound 60 s (main: 300 / 311 / 900 s). The second park's first arm attaches to the main-only pass-through's detached successor, as on main.

Case	Before	After
Holder SIGSTOPped 45 s (past grace + confirmation, under the bound), then resumed	attached-cycle-ended beacon_age=31, retry nonzero-exit beacon_age=32 against the live holder, hook exit 2 with firstmate watcher auto-arm FAILED	hook stayed parked, no failure; a later needs-decision reached main as a rewake with its signal: line
Holder wedged (live, blocked on a foreign-held per-record lock)	same FAILED notice at 31/33 s	attached-holder-stalled beacon_age=60; retry evicted the holder (stalled-holder-replaced) and main got an ordinary check: rearm-resurface wake
Silent-loss timing: holder resumed at beacon age 32 s	hook exit 0, then a later needs-decision was lost (watcher_live=no hook_rerun=no queued_rows=2 arms_following=0)	hook stayed parked; the later needs-decision reached main as a rewake

Part 3 - preempted reply long-poll

Real adapter, runner, fm-on.sh, remote entrypoint, and job worker (only ssh faked); a same-home non-preemptible job, the liveness probe's shape, preempts the listener's poll.

	Before	After
Listener after preemption	exited no-result: remote-reply-ios (exit 76), claim released	alive, same claim pid, polled again
Next fm-procevent.sh reconcile	started=1	started=0
A later remote reply	mirrored by a relaunched listener	mirrored by the same listener

Regression tests

Each new test fails on the base code and passes on the fix: test_tick_leaves_settled_records_alone ("the tick blocked on a settled record's foreign-held lock"), test_attached_arm_follows_a_slow_live_holder and test_attached_arm_hands_a_stalled_holder_to_its_replacement (watcher: FAILED - cycle ended without an actionable reason), and the remote-reply preemption cases ("a preempted reply poll did not report a closed window: 76").
A 2,000-record timing assertion is deliberately kept out of the unit suite; the cycle-time proof is the lab measurement above.

Follow-ups and known limitations

• Not approved in this PR, follow-ups only: archiving long-resolved pending-reply records, and decoupling the contributions poll from the X-mode check cadence.
• Known limitation left as is by decision: the silent-loss path now needs the holder to stall for the full stall bound (3 x grace, 900 s on main). A holder that resumes within about a second of the arm's stalled-holder verdict passes the Stop hook's (and the Cursor park's) benign health recheck, so the hook exits 0 and a later wake is lost; the lab reproduced it as queued_rows=2 arms_following=0.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 error

• 🚨 tests/fm-pending-reply.test.sh:1118 - The intent requires: “Reproduce first in a real setup with a large pending-reply store, and prove each part with observable behavior (cycle time, attach behavior, listener stability).” This test builds 600 settled records and checks that the tick finishes within 60 seconds, but the change provides no before/after watcher-cycle timing for the reported ~2,030-record setup. The attach and listener tests provide behavioral evidence; the required real-setup reproduction and cycle-time proof remain absent from this change. Please provide that evidence or ask the author whether this criterion can be deferred.

✅ **Test** - passed

✅ No issues found.

• Live validation: ✅ go - 4 of 4 scenarios driven live against the product

Scenario	Result	Live	Evidence
Poll a large settled pending-reply store without blocking on a settled record’s lock, while processing records still needing work	✅ pass	live	bash tests/fm-pending-reply.test.sh; 2030-record-cycle.log
Attach to a slow live watcher and receive its wake rather than failing when its beacon passes grace	✅ pass	live	bash tests/fm-watch-arm.test.sh — slow-holder wake scenario
Stop following a holder at the stall bound and let a retry replace it	✅ pass	live	bash tests/fm-watch-arm.test.sh — stalled-holder replacement scenario
Preempt a remote-reply long-poll without publishing false freshness, losing its listener, or relaunching it on reconcile	✅ pass	live	bash tests/fm-remote-reply.test.sh — preemption and listener-stability scenarios

• bash tests/fm-pending-reply.test.sh
• bash tests/fm-watch-arm.test.sh
• bash tests/fm-remote-reply.test.sh
• Ran three fm_pending_reply_tick cycles against a disposable 2,030-record store and measured elapsed time.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Improves watcher and reply listener resilience during slow cycles.

The PR appears safe to merge from the reviewed behavior, but the requested real-setup watcher-cycle timing evidence should be supplied or explicitly deferred.

_{Reviews (1) · Last reviewed commit: "no-mistakes(document): Clarify watcher g..."}