Skip to content

fix(bin): harden supervision liveness (autoarm reclaim, silent-idle backstop, resolve-key) - #11

Merged
landonbrice merged 1 commit into
mainfrom
fm/fm-supervision-liveness
Sep 23, 2026
Merged

landonbrice merged 1 commit into
mainfrom
fm/fm-supervision-liveness

Conversation

@landonbrice

Copy link
Copy Markdown
Owner

Summary

Ships the three items firstmate scoped for the "Supervision liveness" lane (fm-stop-autoarm-deadlock items 1+2, fm-silent-done-strand, cim-mini-inbox-not-drained's remaining defect).

1. Dead auto-arm owner / bg-spare ancestry regression

Reproduced the reported ledger shape (outcome=arming, owner pid dead) directly against the real hook: on the very next Stop firing, fm_autoarm_claim_open/fm_autoarm_claim_next already reclaim it correctly (RC=2, ledger advances, arm runs) - this path was not broken.

Live evidence surfaced a different, real bug in the same area: Claude Code 2.1.280 (measured 2026-09-22, main home) inserted a second bg-spare pass-through layer below bg-pty-host in its process chain. fm_harness_ancestry_pids (bin/fm-session-lock-lib.sh) hit that bg-spare match as the first harness ancestor and stopped immediately, before ever reaching the outermost interactive claude session - so fm_session_lock_owned_by_self read false, the Stop hook exited 0 at its identity gate every time, and the epoch ledger froze on outcome=arming until a brand-new session happened to relaunch it. Fixed by passing through every daemon run/bg-spare-shaped layer instead of stopping at the first one (matches the intent of the earlier daemon-ancestry fix, PR #5, without regressing it).

For the watcher's successor=none exits: documented in bin/fm-claude-stop-autoarm.sh's header why the Stop hook is the sole re-arm owner by design (Claude arms its successor at the next Stop, unlike Pi/omp/OpenCode which arm before delivering the wake - docs/watcher-continuity.md already owns that contract) and why no second arm owner is added.

Verified: bash tests/fm-session-lock-ancestry.test.sh (10/10 ok, including a new stacked bg-spare-below-bg-pty-host case) and bash tests/fm-claude-stop-autoarm.test.sh (40/40 ok, including a new genuinely-dead-owner-pid regression). Not verified: could not reproduce the original 22h production outage end-to-end (no live Claude Code 2.1.280 daemon available in this worktree); the ancestry fix is verified against the exact process shape reported live.

2. Silent-done detection

Added silent_idle_check to bin/fm-watch.sh: a wall-clock idle-duration backstop, independent of the existing pane-hash-stability wedge-escalation cascade, so it still fires even when that cascade's own per-hash state has gone quiet. Emits stale: <window> (silent-idle ...) once per idle stretch when a window has been idle (not busy) for FM_SILENT_IDLE_SECS (default 7200s / 2h, env-overridable) with no terminal status line (done:/needs-decision:/blocked:/failed:) and no declared paused:/captain-held line. Surface only - never tears down or steers. Documented in docs/architecture.md and docs/configuration.md.

Did not fold in the Bridge snapshot's unrecorded-agents check (live agent in a pool copy with no task record): different data source (process/pool scan vs. this watcher's per-task polling loop) and out of scope for a clean fold; leaving as a follow-up.

Verified: bash tests/fm-watch-triage.test.sh - full suite, 113/113 ok, 0 failures, including two new tests (positive surface + declared-pause suppression). Not verified: real-world 15-day-strand reproduction (inherently long-running); the fix targets the mechanism the incident evidence pointed at.

3. fm-send.sh --resolve-key + --fire-and-forget

Removed the refusal that made --resolve-key and --fire-and-forget mutually exclusive. They're independent by construction throughout the rest of the script: --resolve-key closes a status-log decision (RESOLVE_KEYS-keyed), --fire-and-forget only skips minting a new pending-reply correlation (PENDING_REPLY_CORR is never touched when FIRE_AND_FORGET_ID is set) - every downstream branch already handles the combination correctly. This previously made an escalated pending-reply decision unclosable except by arming another correlation, per bin/fm-pending-reply-lib.sh's own escalation-lifecycle contract, which names fm-send --resolve-key as that decision's operator-facing close.

Verified: bash tests/fm-send-resolve-key.test.sh (21/21 ok, including a new combined-flags test closing a reserved pending-reply-* key while confirming no new correlation record is created), plus fm-send-remote-delivery.test.sh, fm-task-inbox.test.sh, and fm-secondmate-reconcile.test.sh for regression (all green).

Other verification

  • bin/fm-lint.sh: shellcheck-clean on every changed bin/*.sh (0.11.0, pinned). Exit 1 only from actionlint missing for .github/workflows/ linting, unrelated to this PR (no workflow files touched) - not installed in this sandboxed worktree.
  • bin/fm-doc-audience-check.sh: ok (surfaces=100, local_links=398).
  • Out of scope, noted per the brief: fm-codex-hooks-trust-dialog, fm-paused-resurface-too-soon, fm-done-ship-stale-nag untouched.

Test plan

  • tests/fm-session-lock-ancestry.test.sh
  • tests/fm-claude-stop-autoarm.test.sh
  • tests/fm-watch-triage.test.sh (full suite)
  • tests/fm-send-resolve-key.test.sh
  • tests/fm-send-remote-delivery.test.sh, tests/fm-task-inbox.test.sh, tests/fm-secondmate-reconcile.test.sh
  • bin/fm-lint.sh (shellcheck clean on changed bin scripts)
  • bin/fm-doc-audience-check.sh

…backstop, resolve-key

Claude Code 2.1.280 inserted a second bg-spare pass-through layer below
bg-pty-host, so a hook shell's harness-ancestry walk could hit that layer
first and stop before reaching the outermost interactive session, leaving
the Stop-owned auto-arm unable to claim its home and freezing the epoch
ledger on outcome=arming. fm_harness_ancestry_pids now passes through every
daemon-run/bg-spare-shaped layer instead of stopping at the first one.

Add a wall-clock silent-idle backstop to bin/fm-watch.sh, independent of the
existing hash-stability wedge cascade, so a worker that finished only
through chat or an interactive pane menu and never wrote a terminal status
line still surfaces after FM_SILENT_IDLE_SECS (default 2h) instead of
staying silent indefinitely.

Allow bin/fm-send.sh's --resolve-key to combine with --fire-and-forget: the
two are independent (one closes a status-log decision, the other only skips
minting a new pending-reply correlation), and the old refusal made an
escalated pending-reply decision unclosable without arming another
correlation.
@landonbrice
landonbrice merged commit 632a8a0 into main Sep 23, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant