fix: resurface durable work after watcher re-arm

[dnth]

Intent

Port the upstream supervision-recovery tranche from kunchenguid/firstmate into this fork (dnth/firstmate) as ONE coherent, OMP-aware PR: PRs kunchenguid#2065 -> kunchenguid#2212 -> kunchenguid#2733, which form a dependency chain and are landed in that order as three commits.

Why this work exists: when the watcher goes down and later re-arms, this fork can silently lose queued wakes and the resurfacing of still-open decisions - a critical supervision-continuity gap that surfaces as the "WATCHER DOWN" state at session start. The tranche guarantees durable recovery after re-arm, makes the recovery acknowledgement livelock-safe, and bounds recovery announcements so the successor supervision cycle is never lost.

What each PR must deliver, adapted to this fork's OMP-primary, multi-backend wake core:

• fix: resurface durable supervision work after re-arm kunchenguid/firstmate#2065 (CRITICAL): after accepted watcher downtime and a re-arm, guarantee BOTH a durable wake-queue drain AND resurfacing of still-open decisions. This fork adds OMP as a primary adapter that must preserve that recovery contract.
• fix(bin): prevent watcher recovery acknowledgement livelock kunchenguid/firstmate#2212 (CRITICAL, correctness follow-up to fix: resurface durable supervision work after re-arm kunchenguid/firstmate#2065): make the recovery acknowledgement generation-stable so concurrent republication cannot continually invalidate it - a post-compaction primary must not be able to die and re-arm in a livelock without ever retiring the recovery episode.
• fix: bound recovery announcements and preserve supervision kunchenguid/firstmate#2733 (HIGH, follow-up to fix: resurface durable supervision work after re-arm kunchenguid/firstmate#2065/fix(bin): prevent watcher recovery acknowledgement livelock kunchenguid/firstmate#2212): bound repeated recovery announcements, removing the unbounded ~52s announcement loop, and stop the recovery path from silently losing its successor supervision cycle. Upstream targets the Pi/OpenCode adapters; this fork must also provide the equivalent for its OMP primary adapter.

Porting approach the user chose: port ADAPTED, not cherry-picked - this fork adapts upstream, so patch IDs differ. Preserve the fork's OMP primary-extension watcher-continuity model, the durable wake-queue and open-decisions machinery, and every experimental backend (herdr, zellij, orca, cmux) plus the Pi, OpenCode, and OMP adapters.

Constraints the user set: this is the wake/supervision core the whole fleet depends on, so blast radius is CRITICAL. No existing safety invariant may be weakened - fail-closed behavior on malformed input, single-flight arming, never broadly killing watchers, and the OMP extension-owned continuity contract must all survive unchanged, and the OMP primary adapter's watcher-continuity semantics must be preserved exactly. Delivery is no-mistakes with yolo off, held for captain merge. Because this edits firstmate's own tracked material, the firstmate-coding-guidelines apply: one full sentence per line in tracked Markdown, plain dash never an em dash, shellcheck-clean bin scripts via bin/fm-lint.sh, colocated tests extending existing tests/ scripts rather than a new runner, and no agent name as a commit co-author.

Deliberate decisions and tradeoffs made while doing this work, which a reviewer reading only the diff would not know:

1. Upstream patched its Pi extension directly. This fork had already extracted the watcher lifecycle into the shared bin/fm-primary-watch-core.ts used by BOTH the Pi and OMP adapters, so the handling-delivery handshake, the confirm-with-retry logic, and the typed-failure classification were placed in that core exactly once. That is how OMP inherits the contract instead of receiving a duplicated copy, and it is why .omp/extensions/fm-primary-omp.ts itself is unchanged.

2. Intentional fix beyond the upstream diff: bin/fm-watch.sh's EXIT trap performs a recovery-marker transition, and a signal can fire that trap while an interrupted frame is still inside a marker critical section. Waiting there for a lock the same process already holds can never be satisfied, so the watcher spun forever, ignored every later signal, and pinned state/.wake-queue.lock with it - wedging the whole home rather than one cycle. This was reproduced deterministically, traced to fm_recovery_marker_arm_check's release being interrupted by watcher_cleanup, and fixed with fm_lock_held_by_self plus _fm_recovery_lock_acquire/_fm_recovery_lock_release. On reentry the transition proceeds under the exclusion it already holds and deliberately leaves the lock exactly as the interrupted frame left it, because the exiting process abandons it and fm_lock_try_acquire reclaims a dead owner's lock as usual. Reentry is scoped to self-ownership only, so exclusion between processes is deliberately unchanged; tests/fm-wake-queue.test.sh pins both halves, including that a live foreign holder is still not treated as reentry.

3. Deliberate adaptation: fm_wake_append's fork-only bounded-attempt mode keeps its whole-call bound. Recovery publication takes the same attempt budget and reports exhaustion as 3 rather than blocking, so bin/fm-send.sh's delivered-no-turn path cannot hang. Upstream had no bounded mode to preserve.

4. Deliberate exclusions from the upstream diffs, because the corresponding fork surfaces do not exist: upstream's --reemit hunks in bin/fm-session-start.sh and tests/fm-session-start.test.sh (this fork has no --reemit path); upstream fix(bin): prevent watcher recovery acknowledgement livelock kunchenguid/firstmate#2212's inactive-outcome receipt stage in the acknowledgement path (it comes from upstream feat(bin): reconcile inactive terminal crew outcomes kunchenguid/firstmate#2167, which this fork has not ported); upstream's prime_status_seen helper in tests/wake-helpers.sh (it depends on wake-lib signature owners this fork does not have, so only recovery_marker_generation and ack_drain_err were taken); and upstream's fm-wake-drain-unread-status and fm-inactive-reconcile entries in bin/fm-test-run.sh's watcher-wake-lock family (neither test exists here).

5. Deliberate test-fixture adaptations to this fork's real contracts, not test weakening. In tests/fm-watch-arm.test.sh the remote parent-reply fixture carries its correlation in the note rather than a second bracket token, because bin/fm-procevent-remote-reply.sh's line_valid accepts exactly one bracket token before the colon and separately requires a corr= token; this keeps the line both ingestible and foldable as a keyed decision. In the same file status_signature was aligned to bin/fm-watch.sh's stat_sig (inode:size:nanosecond mtime); upstream's coarser size:mtime signature never suppressed this fork's signal scan, so the fixture watcher raced an unrelated signal wake into the case under test. Also in that file, the re-arm liveness check became a bounded wait instead of a fixed sub-second sample, because this fork's watcher runs PR-check migration, pending-reply, and process-event work before its first poll, so a healthy re-arm was being read as a stayed-live failure; wait_for_exit still stops the child on timeout, so the never-surfacing path still leaves nothing behind.

6. Every intentional mid-test watcher stop in tests/fm-watch-triage.test.sh now acknowledges its recovery episode through ack_stopped_cycle. That is required by the new contract - without it the next cycle correctly announces check: rearm-resurface instead of the path under test - and is not incidental churn.

7. Added fork-specific OMP coverage the user explicitly asked for, in tests/fm-omp-primary.test.sh: OMP confirms the recovery handling handshake only after delivering its wake steer, and a refused handshake is surfaced as exactly one typed wake rather than swallowed. tests/fm-watch-recovery-loop.test.sh's fixture also installs the shared core and the Pi-compatible runtime allowlist, without which this fork's adapter cannot load at all.

8. bin/fm-guard.sh's header comment was corrected because the drain no longer empties the queue; that is the only change to that file and it is deliberate.

9. bin/fm-classify-lib.sh appeared in the task's expected touched surface but needed no change: this fork's incremental open-decisions cursor already satisfies the resurfacing contract, so it was deliberately left alone rather than modified to match the upstream file list.

10. Deliberately NOT changed: tests/fm-watcher-lock.test.sh's HUP-case timing bound. Its 8s wait has little headroom because the watcher can sit in sleep FM_POLL=5 before its trap runs, but that bound predates this branch and the path was measured terminating correctly 10 times out of 10, so widening an unrelated test's timing inside a critical-safety port was rejected.

What Changed

• Persist generation-stable recovery episodes across watcher downtime and re-arm, resurfacing unacknowledged wake rows and still-open decisions before acknowledgement.
• Add bounded successor handoff confirmation for Pi, OMP, and OpenCode so recovery is announced once per generation without losing the next supervision cycle.
• Fail closed on incomplete recovery handling and make recovery-marker locking safe when watcher signal cleanup reenters a transition.

Risk Assessment

🚨 High: The away-mode recovery episode can still be retired after only partial or unverified decision resurfacing, violating the approved fail-closed continuity invariant.

Testing

No baseline commands were supplied; targeted Pi, OpenCode, OMP, durable queue, watcher-lock, decision-routing, re-arm, generation-stability, and bounded-announcement checks all passed, with direct evidence showing one recovery announcement, a live successor, and a durably queued crew event.

Evidence: Bounded recovery and live successor transcript

T2_WATCH_OUTPUT=signal: .../state/crew.status T2_QUEUE_ROW=... signal crew.status ... T1_MESSAGES=1 T1_MARKER=announced:downtime:seed.1.aaa

T2_WATCH_OUTPUT=signal: /tmp/fm-watch-recovery-loop.KJ1Bpe/recovery-gap-successor/state/crew.status 
T2_QUEUE_ROW=1787406408	2	signal	crew.status	signal: /tmp/fm-watch-recovery-loop.KJ1Bpe/recovery-gap-successor/state/crew.status
ok - a resurfacing handling successor stays alive and supervises instead of going blind
T1_MESSAGES=1
T1_LOCK_PID=3929962
T1_MARKER=announced:downtime:seed.1.aaa
ok - unacknowledged recovery is announced at most once per generation and the successor stays alive

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 error

• 🚨 bin/fm-primary-watch-core.ts:390 - Intent requires “OMP confirms the recovery handling handshake only after delivering its wake steer,” but this confirms first and calls sendWake afterward. A failed/blocked steer can therefore leave the episode marked as handling without delivering it. Reverse the ordering and strengthen the OMP behavioral test, which currently claims but does not assert confirmation-after-steer.
• 🚨 bin/fm-primary-watch-core.ts:616 - Intent requires that “the successor supervision cycle is never lost,” but an actionable successor close is discarded while the preceding restoration/follow-up remains active. If successor B finds a real event while A's follow-up blocks, B clears the child reference and this return loses its notification and next re-arm. Queue the close or process it after current delivery; apply the equivalent fix to OpenCode's restorationInFlight guard.
• 🚨 bin/fm-wake-lib.sh:525 - The required confirm-with-retry and typed-failure path can block forever here: fm_recovery_marker_begin_handling waits unboundedly for .watcher-down.lock, while both adapters invoke this command synchronously. A live stopped foreign holder freezes delivery and supervision without a typed failure. Add a bounded acquire at this shared confirmation boundary while preserving foreign-owner exclusion.
• 🚨 bin/fm-supervise-daemon.sh:1354 - The durable no-loss requirement remains reachable in away mode: handle_wake can fail to persist an actionable escalation, but its status is ignored and the code still acknowledges all rows at line 1367. An unwritable/full state directory can therefore delete the only durable wake without buffering or injection. Abort before acknowledgement unless routing persistence succeeds.
• ⚠️ bin/fm-watch-arm.sh:391 - The new internal --handling-delivered mode and conditional recovery-generation started-line suffix are absent from the script's authoritative usage header, which still documents only --restart and the old exact output. Update the header to own both protocol additions.
• ⚠️ bin/fm-wake-drain.sh:29 - This introduces the critical acknowledgement CLI, but the authoritative script header has no usage syntax for it. Add an explicit usage block covering presentation and --ack-through ... --recovery-generation ... modes.
• ⚠️ docs/verification/supervision.md:326 - The new dated maintainer-verification record includes command and output but no tested source/tool version, contrary to the repository's verification-record requirements. Record the tested revision or equivalent version.

🔧 Fix: Bound recovery confirmation and document protocols
2 errors still open:

• 🚨 bin/fm-primary-watch-core.ts:378 - The required retry “against the current generation and successor” repeats the same handoff. Both spawnSync confirmations execute back-to-back, preventing child output/close callbacks from updating owner.child or armRecovery before the second snapshot. A replacement between readiness and confirmation therefore causes two attempts against the stale generation/PID, followed by a fallback. Make the retry refresh the live handoff before its second confirmation; the equivalent OpenCode implementation has the same defect.
• 🚨 bin/fm-supervise-daemon.sh:1350 - Away-mode recovery drops decision-only resurfacing. The daemon parses only five-field queue rows from the drain output, discarding its OPEN DECISIONS section; with an empty queue and a buried still-open decision, it routes only generic check: rearm-resurface and then acknowledges the episode. The injected digest claims it was pre-read but contains no decision, while last-status housekeeping cannot recover a buried decision. Route the owned OPEN DECISIONS output before acknowledgement.

🔧 Fix: Route away-mode decisions before recovery acknowledgement
1 error still open:

• 🚨 bin/fm-supervise-daemon.sh:1365 - The new decision route ignores escalate_add failure and proceeds to generation-bound acknowledgement. If the escalation buffer becomes unwritable or storage is exhausted, the open decision is never persisted or injected, yet the recovery episode is retired at line 1384. This contradicts the required guarantee that still-open decisions cannot be lost after re-arm. Track routing failure and return before acknowledgement, while leaving the separately approved handle_wake behavior unchanged.

🔧 Fix: Retain recovery when away decision routing fails
1 error still open:

• 🚨 bin/fm-supervise-daemon.sh:1370 - The required fail-closed behavior is still violated when the drain's OPEN DECISIONS section is truncated: this branch logs an unterminated section but continues to the generation-bound acknowledgement. Because fm-wake-drain.sh deliberately swallows decision-section output failures, the daemon can route only a prefix, delete its capture files, and then successfully retire the recovery episode. Treat EOF while in_open_decisions=true as a routing failure and return before acknowledgement.

🔧 Fix: Fail closed until away decisions are fully delivered
1 error still open:

• 🚨 bin/fm-supervise-daemon.sh:1396 - The standing invariant requires acknowledgement only after the COMPLETE open-decision content is routed, but completeness is not positively proven. The parser accepts outside as valid when a capture is truncated before its header, and it treats OPEN DECISIONS: N more omitted (byte cap) as ordinary routed content. Either path reaches --ack-through while decisions were never injected. Make the drain emit an explicit complete frame even for an empty set, reject omitted/incomplete frames, and acknowledge only a positively completed machine-readable presentation.

✅ **Test** - passed

✅ No issues found.

• FM_TEST_EVIDENCE=1 bash tests/fm-watch-recovery-loop.test.sh
• bash tests/fm-omp-primary.test.sh
• bash tests/fm-wake-queue.test.sh
• bash tests/fm-watch-arm.test.sh
• bash tests/fm-wake-daemon-lifecycle-e2e.test.sh
• bash tests/fm-watcher-lock.test.sh
• bash tests/fm-pi-watch-extension.test.sh (initially exposed the fixture race, then passed after the test-only synchronization fix)

⚠️ **Document** - 1 error

• 🚨 bin/fm-primary-watch-core.ts:390 - The intent requires OMP handling confirmation after wake-steer delivery, but the shared core confirms first and sends afterward; executable changes are outside this documentation phase.

🔧 **Lint** - 1 issue found → auto-fixed ✅

• ⚠️ linter found issues (exit code 1)

🔧 Fix: Scope fixture environments to satisfy ShellCheck
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.