From 73d4944e3e296eaa6aff3ceaed0500bddadf1c80 Mon Sep 17 00:00:00 2001 From: scottgerlach Date: Thu, 1 Oct 2026 16:30:14 -0600 Subject: [PATCH 01/13] fix(bin): stop an away daemon that outlives away mode from taking worker events An away-mode supervise daemon kept running after away mode ended, with state/.afk and its singleton lock both gone. Its watcher child was the home's only watcher, the attended primary's arm attached to it and followed its cycles forever, and the daemon drained and acknowledged every worker event into state/.subsuper-escalations, whose injection is gated off while away mode is off. Done handoffs and decisions never reached the attended firstmate. How the daemon lost its lock: the lock's recorded identity is fm_pid_identity, whose macOS form renders `ps -o lstart` in the local time zone. After the host time zone changes, a live daemon no longer matches it. The return (fm-afk-launch.sh stop) then saw no daemon and cleared state/.afk without a signal, and the next entry (fm-afk-start.sh) treated the live daemon as a reused pid, deleted its lock, and started a second daemon. The first daemon kept running without its lock, and nothing made it exit. The fix: - bin/fm-supervise-daemon.sh exits on its own, before it drains another wake, once state/.afk is gone or once it no longer holds its lock, and it removes only its own pid file and instance record. - bin/fm-afk-daemon-lib.sh (new) owns finding a home's live daemons: a per-instance record with a UTC-rendered identity, the lock, and the parent of this home's verified watcher. - bin/fm-afk-launch.sh stop signals every live daemon those proofs name, including one that lost its lock. - bin/fm-afk-start.sh never deletes the lock of a live process that runs the daemon script. - bin/fm-watch-arm.sh never attaches to a watcher whose parent is a daemon while state/.afk is absent. It stops that daemon and owns a fresh cycle, home-scoped and without pkill. The repeated "check: procevent lavish ... 14" escalation had the same cause: fm-procevent.sh reconcile republishes an unhandled result on every watcher cycle until `handled` runs, and only the attended firstmate runs it. The lifecycle regression covers it, because the daemon no longer drains wakes after away mode ends. Harness reach: the takeover lives in bin/fm-watch-arm.sh, so it covers the Claude Stop auto-arm, the supervision host, the Cursor turn-end guard, and the Pi and omp extensions. The Codex checkpoint (bin/fm-watch-checkpoint.sh) runs fm-watch.sh directly, so a daemon from before this fix on a Codex primary makes the checkpoint report that a watcher already runs outside it instead of taking over. A daemon that runs this code exits on its own on every harness. Separate findings, reported here and not fixed: - A healthy away daemon (state/.afk present) also buffers the same unhandled process-event result again on every wake; escalate_add dedups only unknown wakes. A 40-second isolated run buffered 76 lines for 12 distinct results. - fm_pid_identity is time-zone-sensitive for every caller on macOS (watcher lock, procevent claims, wake grants, fm_afk_daemon_owns_supervision). Pinning TZ=UTC0 in its ps form is a one-line fix, but it invalidates every identity recorded before the upgrade, so it needs its own migration. --- .agents/skills/afk/SKILL.md | 2 +- bin/fm-afk-daemon-lib.sh | 249 +++++++++++++++++++++ bin/fm-afk-launch.sh | 54 ++--- bin/fm-afk-start.sh | 45 +--- bin/fm-supervise-daemon.sh | 86 +++++-- bin/fm-watch-arm.sh | 46 ++++ docs/architecture.md | 2 + docs/scripts.md | 1 + tests/fm-afk-launch.test.sh | 135 +++++++++++ tests/fm-wake-daemon-lifecycle-e2e.test.sh | 105 +++++++++ tests/fm-watch-arm.test.sh | 128 +++++++++++ 11 files changed, 765 insertions(+), 88 deletions(-) create mode 100755 bin/fm-afk-daemon-lib.sh diff --git a/.agents/skills/afk/SKILL.md b/.agents/skills/afk/SKILL.md index 68b7bf1b7cd..3d278cc019b 100644 --- a/.agents/skills/afk/SKILL.md +++ b/.agents/skills/afk/SKILL.md @@ -43,7 +43,7 @@ Hold-for-return is the default and the only reach profile this release records: - **Every other harness** (codex, opencode, omp, and cursor on a home that does not run the supervision host, and kimi): run `bin/fm-afk-launch.sh start`. It is the single owner of the daemon terminal: it creates a NON-VISIBLE tracked terminal for the current backend and passes the captain pane in as `FM_SUPERVISOR_TARGET` so the daemon injects into the captain, not its own new pane (docs/herdr-backend.md "Away-mode supervisor support"). Both daemon paths require the record `enter` wrote and share `bin/fm-afk-start.sh` as the daemon entry. - The daemon is **presence-gated**: it injects escalations only while `state/.afk` exists, and stays quiet otherwise. + The daemon is **presence-gated**: it injects escalations only while `state/.afk` exists, and exits on its own once that flag is gone, so it never takes the attended session's wakes. 3. **Announce, then read back after entry.** Relay the announcement in spirit: hold-for-return only, no phone channel, your instructions are recorded and the away session will carry them out where it can, anything it is unsure of, or that needs you, waits for your return, and destructive, irreversible, and security-sensitive actions are never pre-authorizable whatever the words say. Then give your own plain-sentence restatement of the words in `AGENTS.md` section 9 language - what you read them as asking for, sentence by sentence, never a numbered field list - beside the expected return, the spend cap, and the one-sentence reach announcement. diff --git a/bin/fm-afk-daemon-lib.sh b/bin/fm-afk-daemon-lib.sh new file mode 100755 index 00000000000..8a9618598f8 --- /dev/null +++ b/bin/fm-afk-daemon-lib.sh @@ -0,0 +1,249 @@ +#!/usr/bin/env bash +# fm-afk-daemon-lib.sh - the single owner of finding a home's live away-mode +# supervise daemons (bin/fm-supervise-daemon.sh): its singleton lock, the +# instance record every daemon leaves, and the parent link of the watcher a +# daemon runs. Sourced by the daemon, bin/fm-afk-start.sh, bin/fm-afk-launch.sh, +# and bin/fm-watch-arm.sh, each after bin/fm-wake-lib.sh, whose fm_pid_alive, +# fm_pid_identity, and fm_watcher_lock_matches_pid it calls. Only +# fm_afk_daemon_stop signals anything. +# +# The lock alone cannot find every live daemon. Its recorded identity is +# fm_pid_identity, whose ps form renders the start time in the local time zone, +# so once the host's time zone changes a live daemon no longer matches it: the +# return then saw no daemon to stop, and the next entry took the live daemon's +# lock and started a second one beside it. The record below stores the same +# identity rendered under TZ=UTC0, which a time-zone change cannot move. +# +# Proofs that a pid is a live daemon of this home, strongest first: +# record state/.supervise-daemon.instances/, written by the daemon once +# it holds its lock and removed by its own shutdown. It holds +# pid=, state= (the physical state dir), and identity= lines; the +# pid must be alive with that UTC identity, under this state dir. +# lock the lock names a live pid whose recorded identity still matches, +# or that recorded no identity and runs the daemon script. +# watcher this home's identity-verified watcher's live parent process runs +# the daemon script, which also names a daemon with no record. +# command the lock names a live pid that runs the daemon script although its +# recorded identity no longer matches. That keeps the lock and stops a +# second daemon from starting; it is never enough to signal the pid. + +FM_AFK_DAEMON_SCRIPT=fm-supervise-daemon.sh + +fm_afk_daemon_lock_path() { # + printf '%s/.supervise-daemon.lock\n' "$1" +} + +fm_afk_daemon_record_dir() { # + printf '%s/.supervise-daemon.instances\n' "$1" +} + +# The identity a record stores: fm_pid_identity with the time zone pinned. +fm_afk_daemon_identity() { # + TZ=UTC0 fm_pid_identity "$1" +} + +_fm_afk_daemon_physical_dir() { # + (CDPATH='' cd -- "$1" 2>/dev/null && pwd -P) +} + +_fm_afk_daemon_record_field() { # + sed -n "s/^$2=//p" "$1" 2>/dev/null | head -n 1 +} + +# Print 's command line, from a Linux-compatible proc root when present. +fm_afk_daemon_pid_command() { # + local pid=$1 proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc} + case "$pid" in ''|*[!0-9]*) return 1 ;; esac + if [ -r "$proc_root/$pid/cmdline" ]; then + tr '\0' ' ' < "$proc_root/$pid/cmdline" 2>/dev/null + return + fi + COLUMNS=10000 LC_ALL=C ps -p "$pid" -o command= 2>/dev/null +} + +# Print 's parent pid. +fm_afk_daemon_pid_parent() { # + local pid=$1 proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc} stat_line parent + local -a fields + case "$pid" in ''|*[!0-9]*) return 1 ;; esac + if [ -r "$proc_root/$pid/stat" ]; then + stat_line=$(cat "$proc_root/$pid/stat" 2>/dev/null) || return 1 + # After the final comm delimiter, index 1 is proc stat field 4 (ppid). + read -r -a fields <<< "${stat_line##*)}" + parent=${fields[1]:-} + else + parent=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d '[:space:]') + fi + case "$parent" in ''|*[!0-9]*) return 1 ;; esac + printf '%s\n' "$parent" +} + +# True when is alive and its command line runs the daemon script. +fm_afk_daemon_pid_runs_daemon() { # + local command + fm_pid_alive "$1" || return 1 + command=$(fm_afk_daemon_pid_command "$1") || return 1 + case " $command " in + *"/$FM_AFK_DAEMON_SCRIPT "*|*" $FM_AFK_DAEMON_SCRIPT "*) return 0 ;; + esac + return 1 +} + +# Record a daemon instance; the daemon calls this for itself once it holds +# its lock. +fm_afk_daemon_record_write() { # + local state=$1 pid=$2 dir identity physical pending + case "$pid" in ''|*[!0-9]*) return 1 ;; esac + identity=$(fm_afk_daemon_identity "$pid") || return 1 + [ -n "$identity" ] || return 1 + physical=$(_fm_afk_daemon_physical_dir "$state") || return 1 + dir=$(fm_afk_daemon_record_dir "$state") + mkdir -p "$dir" || return 1 + pending=$(mktemp "$dir/.pending.XXXXXX") || return 1 + if ! printf 'pid=%s\nstate=%s\nidentity=%s\n' "$pid" "$physical" "$identity" > "$pending" \ + || ! mv -f "$pending" "$dir/$pid"; then + rm -f "$pending" + return 1 + fi +} + +fm_afk_daemon_record_remove() { # + rm -f "$(fm_afk_daemon_record_dir "$1")/$2" 2>/dev/null || true +} + +# True when 's record for proves a live daemon of this home. +fm_afk_daemon_record_live() { # + local state=$1 pid=$2 file physical recorded current + case "$pid" in ''|*[!0-9]*) return 1 ;; esac + file="$(fm_afk_daemon_record_dir "$state")/$pid" + [ -f "$file" ] || return 1 + [ "$(_fm_afk_daemon_record_field "$file" pid)" = "$pid" ] || return 1 + physical=$(_fm_afk_daemon_physical_dir "$state") || return 1 + [ "$(_fm_afk_daemon_record_field "$file" state)" = "$physical" ] || return 1 + recorded=$(_fm_afk_daemon_record_field "$file" identity) + [ -n "$recorded" ] || return 1 + fm_pid_alive "$pid" || return 1 + current=$(fm_afk_daemon_identity "$pid") || return 1 + [ "$current" = "$recorded" ] +} + +# Drop every record that no longer proves a live daemon. +fm_afk_daemon_records_prune() { # + local state=$1 file + for file in "$(fm_afk_daemon_record_dir "$state")"/*; do + [ -f "$file" ] || continue + fm_afk_daemon_record_live "$state" "${file##*/}" || rm -f "$file" 2>/dev/null || true + done +} + +# Print the lock's owner directory: the symlink target, or a legacy plain dir. +fm_afk_daemon_lock_owner() { # + local lock owner + lock=$(fm_afk_daemon_lock_path "$1") + if [ -L "$lock" ]; then + owner=$(readlink "$lock" 2>/dev/null) || return 1 + [ -n "$owner" ] || return 1 + case "$owner" in + /*) printf '%s\n' "$owner" ;; + *) printf '%s/%s\n' "$(dirname "$lock")" "$owner" ;; + esac + return 0 + fi + [ -d "$lock" ] || return 1 + printf '%s\n' "$lock" +} + +fm_afk_daemon_lock_pid() { # + local owner + owner=$(fm_afk_daemon_lock_owner "$1") || return 1 + cat "$owner/pid" 2>/dev/null || true +} + +# The lock proof (header): strong enough to signal the lock's pid. +fm_afk_daemon_lock_holder_proven() { # + local state=$1 owner pid identity current + owner=$(fm_afk_daemon_lock_owner "$state") || return 1 + pid=$(cat "$owner/pid" 2>/dev/null || true) + fm_pid_alive "$pid" || return 1 + fm_afk_daemon_record_live "$state" "$pid" && return 0 + identity=$(cat "$owner/pid-identity" 2>/dev/null || true) + if [ -z "$identity" ]; then + fm_afk_daemon_pid_runs_daemon "$pid" + return + fi + current=$(fm_pid_identity "$pid") || return 1 + [ "$current" = "$identity" ] +} + +# The lock or command proof (header): a live daemon holds the lock, so it must +# be kept and no second daemon may start. +fm_afk_daemon_lock_holder_live() { # + local pid + fm_afk_daemon_lock_holder_proven "$1" && return 0 + pid=$(fm_afk_daemon_lock_pid "$1") || return 1 + fm_afk_daemon_pid_runs_daemon "$pid" +} + +# The watcher proof (header): print the pid of the daemon whose child is this +# home's live watcher. +fm_afk_daemon_watcher_owner() { # + local state=$1 watch_path=$2 home=$3 watcher parent + watcher=$(cat "$state/.watch.lock/pid" 2>/dev/null || true) + fm_pid_alive "$watcher" || return 1 + fm_watcher_lock_matches_pid "$state" "$watch_path" "$watcher" "$home" || return 1 + parent=$(fm_afk_daemon_pid_parent "$watcher") || return 1 + case "$parent" in 0|1) return 1 ;; esac + fm_afk_daemon_pid_runs_daemon "$parent" || return 1 + printf '%s\n' "$parent" +} + +# Print each live daemon of that a record, lock, or watcher proof names, +# once. The watcher proof needs and . +fm_afk_daemon_live_pids() { # [ ] + local state=$1 watch_path=${2:-} home=${3:-} file pid found=' ' + for file in "$(fm_afk_daemon_record_dir "$state")"/*; do + [ -f "$file" ] || continue + pid=${file##*/} + fm_afk_daemon_record_live "$state" "$pid" && found="$found$pid " + done + if fm_afk_daemon_lock_holder_proven "$state"; then + pid=$(fm_afk_daemon_lock_pid "$state") + case "$found" in *" $pid "*) ;; *) found="$found$pid " ;; esac + fi + if [ -n "$watch_path" ] && pid=$(fm_afk_daemon_watcher_owner "$state" "$watch_path" "$home"); then + case "$found" in *" $pid "*) ;; *) found="$found$pid " ;; esac + fi + for pid in $found; do + printf '%s\n' "$pid" + done +} + +# SIGTERM each and wait up to ten seconds for all of them to exit. A pid +# that still runs under the identity it had before the signal is printed, and +# then the call fails; a pid that exited or was reused counts as stopped. +fm_afk_daemon_stop() { # ... + local pid i alive status=0 + local -a pids=() identities=() + for pid in "$@"; do + fm_pid_alive "$pid" || continue + pids+=("$pid") + identities+=("$(fm_afk_daemon_identity "$pid" 2>/dev/null || true)") + kill -TERM "$pid" 2>/dev/null || true + done + [ "${#pids[@]}" -gt 0 ] || return 0 + for _ in $(seq 1 40); do + alive=0 + for pid in "${pids[@]}"; do + fm_pid_alive "$pid" && alive=1 + done + [ "$alive" -eq 1 ] || break + sleep 0.25 + done + for i in "${!pids[@]}"; do + fm_pid_alive "${pids[$i]}" || continue + [ "$(fm_afk_daemon_identity "${pids[$i]}" 2>/dev/null || true)" = "${identities[$i]}" ] || continue + printf '%s\n' "${pids[$i]}" + status=1 + done + return "$status" +} diff --git a/bin/fm-afk-launch.sh b/bin/fm-afk-launch.sh index 23bab8d4da8..eff23758310 100755 --- a/bin/fm-afk-launch.sh +++ b/bin/fm-afk-launch.sh @@ -82,9 +82,11 @@ # fm-afk-launch.sh start-native # Prepare lifecycle state for a harness-native # background job and record that no terminal exists. -# fm-afk-launch.sh stop Correct-ordered exit: SIGTERM the daemon so its -# cleanup flushes WHILE state/.afk is still present, -# wait for it, close a recorded non-native terminal +# fm-afk-launch.sh stop Correct-ordered exit: SIGTERM every live daemon of +# this home, including one that lost its lock +# (bin/fm-afk-daemon-lib.sh), so its cleanup flushes +# WHILE state/.afk is still present, wait for each, +# close a recorded non-native terminal # by exact id, clear state/.afk, then archive the # record last. A Pi or native entry that never # launched a daemon reports that none was running. @@ -813,41 +815,31 @@ fm_afk_launch_start_native() { } fm_afk_launch_stop() { - local pid pid_identity current_identity result=0 read_result archived closed_daemon_terminal=0 + local pid survivors result=0 read_result archived closed_daemon_terminal=0 + local -a daemon_pids=() fm_afk_launch_record_read read_result=$? if [ "$read_result" -eq 2 ]; then fm_afk_launch_log "malformed daemon terminal record; refusing to stop away mode" return 1 fi - # (1) SIGTERM the daemon so its cleanup trap flushes buffered escalations - # WHILE state/.afk is still present (the exit-ordering fix: clearing .afk - # first would make that flush a no-op via inject_msg's presence gate). - pid="" - pid_identity="" - if daemon_lock_held_by_live_daemon; then - pid=$(daemon_lock_pid 2>/dev/null) || return 1 - pid_identity=$(fm_pid_identity "$pid" 2>/dev/null) || return 1 + # (1) SIGTERM every live daemon of this home so each cleanup trap flushes + # buffered escalations WHILE state/.afk is still present (the exit-ordering + # fix: clearing .afk first would make that flush a no-op via inject_msg's + # presence gate). bin/fm-afk-daemon-lib.sh owns which proofs name a daemon, + # including one that lost its lock. + while IFS= read -r pid; do + [ -n "$pid" ] && daemon_pids+=("$pid") + done </dev/null; then - fm_afk_launch_log "failed to signal away-mode daemon pid=$pid" - result=1 - fi - for _ in $(seq 1 40); do - fm_pid_alive "$pid" || break - sleep 0.25 - done - fi - if [ -n "$pid" ] && fm_pid_alive "$pid"; then - current_identity=$(fm_pid_identity "$pid" 2>/dev/null) || { - fm_afk_launch_log "could not confirm away-mode daemon exit; preserving lifecycle state" - return 1 - } - if [ "$current_identity" = "$pid_identity" ]; then - fm_afk_launch_log "away-mode daemon did not exit after SIGTERM; preserving lifecycle state" - return 1 - fi + fm_afk_daemon_records_prune "$FM_AFK_LAUNCH_STATE" + if daemon_lock_held_by_live_daemon; then + fm_afk_launch_log "a live away daemon pid=$(daemon_lock_pid) still holds the lock, but nothing proves it is this home's, so it was not signalled; the attended arm stops it once it runs this home's watcher" fi # (2) Close the daemon's own terminal by exact id. A native/none record or # an absent record means no terminal existed for this entry (Pi never diff --git a/bin/fm-afk-start.sh b/bin/fm-afk-start.sh index 3ddafc5107f..6a1862881fa 100755 --- a/bin/fm-afk-start.sh +++ b/bin/fm-afk-start.sh @@ -41,6 +41,8 @@ FM_AFK_DAEMON="$FM_AFK_START_DIR/fm-supervise-daemon.sh" # shellcheck source=bin/fm-wake-lib.sh . "$FM_AFK_START_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-afk-daemon-lib.sh +. "$FM_AFK_START_DIR/fm-afk-daemon-lib.sh" fm_afk_start_usage() { sed -n '2,14p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' @@ -67,48 +69,14 @@ fm_afk_clear_stale_artifacts() { # "$state/.subsuper-unknown-acked" 2>/dev/null } -daemon_lock_owner() { - local owner - if [ -L "$FM_AFK_LOCK" ]; then - owner=$(readlink "$FM_AFK_LOCK" 2>/dev/null) || return 1 - [ -n "$owner" ] || return 1 - case "$owner" in - /*) printf '%s\n' "$owner" ;; - *) printf '%s/%s\n' "$(dirname "$FM_AFK_LOCK")" "$owner" ;; - esac - return 0 - fi - [ -d "$FM_AFK_LOCK" ] || return 1 - printf '%s\n' "$FM_AFK_LOCK" -} - -daemon_pid_matches() { - local pid=$1 owner=$2 identity current command - identity=$(cat "$owner/pid-identity" 2>/dev/null || true) - if [ -n "$identity" ]; then - current=$(fm_pid_identity "$pid") || return 1 - [ "$current" = "$identity" ] - return - fi - command=$(ps -p "$pid" -o command= 2>/dev/null || true) - case "$command" in - *"$FM_AFK_DAEMON"*|*"fm-supervise-daemon.sh"*) return 0 ;; - esac - return 1 -} - daemon_lock_pid() { - local owner - owner=$(daemon_lock_owner) || return 1 - cat "$owner/pid" 2>/dev/null || true + fm_afk_daemon_lock_pid "$FM_AFK_STATE" } +# A live daemon holds the lock (bin/fm-afk-daemon-lib.sh's lock or command +# proof): keep the lock and start no second daemon. daemon_lock_held_by_live_daemon() { - local owner pid - owner=$(daemon_lock_owner) || return 1 - pid=$(cat "$owner/pid" 2>/dev/null || true) - fm_pid_alive "$pid" || return 1 - daemon_pid_matches "$pid" "$owner" + fm_afk_daemon_lock_holder_live "$FM_AFK_STATE" } fm_afk_flag_write() { # [mode] @@ -164,6 +132,7 @@ fm_afk_start_main() { return 0 fi + # A live lock pid that is not a daemon was reused, so the lock is stale. if fm_pid_alive "$pid" && [ -n "$pid" ]; then fm_lock_remove_path "$FM_AFK_LOCK" 2>/dev/null || true fi diff --git a/bin/fm-supervise-daemon.sh b/bin/fm-supervise-daemon.sh index a2a7664f4fb..0a6ab95b198 100755 --- a/bin/fm-supervise-daemon.sh +++ b/bin/fm-supervise-daemon.sh @@ -14,14 +14,16 @@ # or omits anything it names a state/.subsuper-digests/ file holding every # buffered event verbatim. # -# PRESENCE-GATING (the /afk contract). The daemon is the away-mode engine: it -# injects ONLY when the durable away-mode flag state/.afk is present. Invoking -# the /afk skill sets that flag and starts this daemon; any real (unmarked) -# user message clears it and firstmate resumes full responsiveness. -# When afk is off, normal fm-watch.sh always-on triage is the active mechanism. -# Any buffered daemon escalations that remain while afk is off survive in -# state/.subsuper-escalations and are flushed on the next "while you were out" -# catch-up or when afk is re-entered. +# PRESENCE-GATING (the /afk contract). The daemon is the away-mode engine and +# lives only while the durable away-mode flag state/.afk is present: it injects +# only then. Invoking the /afk skill sets that flag and starts this daemon; any +# real (unmarked) user message clears it and firstmate resumes full +# responsiveness. Once the flag is gone, or once this daemon no longer holds its +# singleton lock, it exits on its own before it drains another wake, so the +# attended firstmate's own watcher receives every wake again; a daemon left +# running would drain and acknowledge each one into a buffer nothing delivers. +# Escalations it could not deliver survive in state/.subsuper-escalations as +# evidence for the return catch-up. # # IN-BAND OPERATIONAL INPUT. bin/fm-operational-input.sh constructs every # current daemon injection as the typed away-supervisor kind after the stable @@ -155,7 +157,9 @@ # FM_LOG_MAX_BYTES / FM_LOG_KEEP_LINES / FM_CRASH_* log + crash guards # FM_STATE_OVERRIDE alternate state dir (testing) # Logs each wake to state/.supervise-daemon.log (size-capped). Single -# instance via portable lock on state/.supervise-daemon.lock. Trapped +# instance via portable lock on state/.supervise-daemon.lock, plus an +# instance record that lets a return stop it without that lock +# (bin/fm-afk-daemon-lib.sh). Trapped # SIGTERM/SIGINT shut down within ~1s, flush escalations, release the # lock. A crashing fm-watch.sh is logged and restarted, never killing # the daemon; a tight crash-restart spin is detected and backed off. @@ -202,6 +206,11 @@ FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" # shellcheck source=bin/fm-busy-lib.sh . "$FM_DAEMON_DIR/fm-busy-lib.sh" +# The instance record this daemon leaves so a return can stop it without its +# lock (fm_afk_daemon_record_write and fm_afk_daemon_record_remove). +# shellcheck source=bin/fm-afk-daemon-lib.sh +. "$FM_DAEMON_DIR/fm-afk-daemon-lib.sh" + # --- tunables --------------------------------------------------------------- # Supervisor backends this daemon knows how to inject into today. zellij, orca, # and cmux are real backends elsewhere in firstmate (bin/fm-backend.sh) but this @@ -1420,8 +1429,9 @@ inject_msg() { # [state] local msg=$1 state target backend retries sleep_s verdict composer encoded bytes errf err='' body state="${2:-$(_state_root)}" # (1) Presence-gate: inject ONLY when afk is active. When afk is off, the - # daemon self-handles and stays quiet; firstmate drives the normal always-on - # watcher triage. Escalations buffer and survive for the next catch-up flush. + # daemon is about to exit (PRESENCE-GATING above) and firstmate drives the + # normal always-on watcher triage; escalations stay buffered for the return + # catch-up. INJECT_LAST_FAILURE= INJECT_SUBMIT_ATTEMPTED=0 afk_active "$state" || { INJECT_LAST_FAILURE="deferred: afk inactive"; log "inject $INJECT_LAST_FAILURE"; return 1; } @@ -1775,6 +1785,7 @@ fm_super_main() { fi exit 1 fi + local LOCK_OWNER=${FM_LOCK_OWNER_DIR:-} echo "$$" > "$PIDFILE" # The recorded identity is what proves this daemon still owns supervision after # its watcher child exits (fm_afk_daemon_owns_supervision, read by the turn-end @@ -1785,6 +1796,34 @@ fm_super_main() { rm -f "$LOCK/pid-identity" 2>/dev/null || true log "warn: could not record this daemon's process identity; the turn-end guard cannot recognize away-mode supervision" fi + if ! fm_afk_daemon_record_write "$STATE" "$$"; then + log "warn: could not record this daemon instance; a return that finds no lock cannot stop it, so it relies on its own exit when away mode ends" + fi + + # Release only what this instance owns: the lock and pid file may already + # name a successor daemon. + release_instance() { + fm_lock_release "$LOCK" 2>/dev/null || true + [ "$(cat "$PIDFILE" 2>/dev/null || true)" != "$$" ] || rm -f "$PIDFILE" 2>/dev/null || true + fm_afk_daemon_record_remove "$STATE" "$$" + } + + # The daemon exists only for away or quiet mode (state/.afk) and only while it + # holds the singleton lock. Without either it would keep draining and + # acknowledging every wake into a buffer that nothing delivers, so it exits. + holds_lock() { + [ "$(cat "$LOCK/pid" 2>/dev/null || true)" = "$$" ] || return 1 + [ -z "$LOCK_OWNER" ] || fm_lock_points_to_owner "$LOCK" "$LOCK_OWNER" + } + exit_reason() { + if ! afk_active "$STATE"; then + printf 'away and quiet mode are no longer active' + elif ! holds_lock; then + printf 'this daemon no longer holds %s' "$LOCK" + else + return 1 + fi + } # --- auto-discover the supervisor BACKEND (tmux vs herdr) first ----------- # Priority: FM_SUPERVISOR_BACKEND override > $TMUX_PANE (tmux) > $HERDR_ENV=1 @@ -1817,8 +1856,7 @@ fm_super_main() { if ! fm_backend_list_contains "$FM_SUPERVISOR_SUPPORTED_BACKENDS" "$BACKEND"; then echo "error: away-mode daemon does not support supervisor backend '$BACKEND' yet (supported: $FM_SUPERVISOR_SUPPORTED_BACKENDS); set FM_SUPERVISOR_BACKEND=tmux|herdr and FM_SUPERVISOR_TARGET to run firstmate's own pane under a supported backend" >&2 log "startup failed: unsupported supervisor backend '$BACKEND' (source=$backend_source)" - fm_lock_release "$LOCK" 2>/dev/null || true - rm -f "$PIDFILE" 2>/dev/null || true + release_instance exit 1 fi @@ -1855,8 +1893,7 @@ fm_super_main() { if ! fm_backend_target_exists "$BACKEND" "$TARGET"; then echo "error: supervisor target '$TARGET' does not resolve to a $BACKEND pane; set FM_SUPERVISOR_TARGET" >&2 log "startup failed: target '$TARGET' not found (backend=$BACKEND)" - fm_lock_release "$LOCK" 2>/dev/null || true - rm -f "$PIDFILE" 2>/dev/null || true + release_instance exit 1 fi @@ -1878,8 +1915,7 @@ fm_super_main() { if [ -n "${CUR_TMP:-}" ]; then rm -f "$CUR_TMP" 2>/dev/null || true fi - fm_lock_release "$LOCK" 2>/dev/null || true - rm -f "$PIDFILE" 2>/dev/null || true + release_instance log "daemon shutting down" exit 0 } @@ -1911,8 +1947,16 @@ fm_super_main() { WATCHER_PID=$! } - local rc reason + local rc reason why while true; do + # --- presence-gated exit ------------------------------------------------- + # Checked before anything below can drain or acknowledge a wake, so the + # attended firstmate receives every wake once away mode has ended. + if why=$(exit_reason); then + log "exiting: $why" + cleanup + fi + # --- pane-gone guard (preserved) --------------------------------------- # With the #29 watcher's enqueue-before-suppress, a wake is no longer # swallowed by running the watcher with no injection target. We still back @@ -1957,6 +2001,12 @@ fm_super_main() { sleep "${HOUSEKEEPING_TICK:-$HOUSEKEEPING_TICK_DEFAULT}" continue fi + if why=$(exit_reason); then + # The child is reaped; never signal its pid again. + WATCHER_PID="" + log "exiting before handling wake '$reason' (it stays queued): $why" + cleanup + fi log "wake: $reason" if ! handle_durable_wakes "$reason" "$STATE"; then log "durable wake handling was not acknowledged; restarting for recovery" diff --git a/bin/fm-watch-arm.sh b/bin/fm-watch-arm.sh index 3a996a0320e..e6a16a4a17f 100755 --- a/bin/fm-watch-arm.sh +++ b/bin/fm-watch-arm.sh @@ -35,6 +35,9 @@ # watcher: FAILED - attached watcher pid= stalled (beacon s at or past hard bound s) # - the followed holder is alive but its beacon # reached the stall bound +# watcher: FAILED - leftover away daemon pid= did not stop, so it still owns this home's watcher +# watcher: FAILED - a leftover away daemon pid= still runs this home's watcher after a takeover +# - the takeover below could not end that daemon # It NEVER reports started/attached/healthy off a stale beacon or a dead/reused pid: a # stale-beacon or dead-pid holder either self-heals (the fresh child steals the # dead lock per the singleton self-eviction/steal path and is confirmed) or this @@ -53,6 +56,13 @@ # so the failure is loud. A live cycle already present means re-arm attaches - do # not start a second watcher. # +# The one exception is a watcher whose live parent is an away daemon while +# state/.afk is absent: that daemon outlived away mode and would take every +# wake, so the arm never attaches to it. It stops that daemon (which stops its +# own watcher), then stops this home's watcher if one remains, then re-runs +# itself in place to own a fresh cycle (take_over_from_leftover_daemon; +# bin/fm-afk-daemon-lib.sh proves the ownership). +# # Every observed watcher cycle appends one tab-separated lifecycle record to # state/.watch-cycle-exits.log. The arm layer owns that bounded ledger; it records # arm/watcher identities, timestamps, exit/signal classification, beacon age, @@ -121,6 +131,8 @@ if [ "${FM_GATE_REFUSE_BYPASS:-}" != 1 ]; then fi # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-afk-daemon-lib.sh +. "$SCRIPT_DIR/fm-afk-daemon-lib.sh" WATCH="$SCRIPT_DIR/fm-watch.sh" WATCH_LOCK="$STATE/.watch.lock" @@ -317,6 +329,36 @@ report_attached() { echo "watcher: attached pid=$HEALTHY_PID (beacon ${age}s)" } +# An away daemon owns triage only while state/.afk exists. A healthy watcher +# whose parent is a daemon while that flag is absent belongs to a daemon that +# outlived away mode: following it would hand every wake to that daemon, which +# drains and acknowledges it into a buffer nothing delivers. So before attaching +# to such a holder, stop that daemon (it stops its own watcher) and this home's +# watcher, then re-run this arm in place to own a fresh cycle. Only the daemon +# that bin/fm-afk-daemon-lib.sh proves runs this home's watcher is signalled. +take_over_from_leftover_daemon() { # + local holder=$1 daemon survivors + [ ! -e "$STATE/.afk" ] || return 0 + daemon=$(fm_afk_daemon_watcher_owner "$STATE" "$WATCH" "$FM_HOME") || return 0 + [ "$(cat "$WATCH_LOCK/pid" 2>/dev/null || true)" = "$holder" ] || return 0 + [ "$cycle_active" -eq 1 ] || cycle_begin "$holder" leftover-daemon "$HEALTHY_IDENTITY" + if [ "${FM_WATCH_ARM_TOOK_OVER:-}" = 1 ]; then + cycle_log_append none none leftover-daemon-returned "daemon:$daemon" + echo "watcher: FAILED - a leftover away daemon pid=$daemon still runs this home's watcher after a takeover" + exit 1 + fi + if ! survivors=$(fm_afk_daemon_stop "$daemon"); then + cycle_log_append none none leftover-daemon-survived "daemon:$survivors" + echo "watcher: FAILED - leftover away daemon pid=$daemon did not stop, so it still owns this home's watcher" + exit 1 + fi + stop_home_watcher || exit 1 + cycle_log_append none none leftover-daemon-stopped "daemon:$daemon" + [ -z "${child_out:-}" ] || rm -f "$child_out" 2>/dev/null || true + trap - HUP TERM INT + FM_WATCH_ARM_TOOK_OVER=1 exec "$SCRIPT_DIR/fm-watch-arm.sh" +} + # Give a successor the same bounded confirmation window used for a fresh child. # Adapter-owned continuations normally win immediately, but the bound avoids a # false failure when process-close delivery and lock publication cross briefly. @@ -397,6 +439,7 @@ attach_and_wait() { while :; do if healthy_watcher; then if [ "$HEALTHY_PID" != "$attached_pid" ] || [ "$HEALTHY_IDENTITY" != "$cycle_watcher_identity" ]; then + take_over_from_leftover_daemon "$HEALTHY_PID" cycle_log_append unknown unknown lock-replaced "attached:$HEALTHY_PID" attached_pid=$HEALTHY_PID cycle_begin "$attached_pid" attached "$HEALTHY_IDENTITY" @@ -416,6 +459,7 @@ attach_and_wait() { return 1 fi if wait_for_healthy_successor; then + take_over_from_leftover_daemon "$HEALTHY_PID" cycle_log_append unknown unknown attached-cycle-ended "attached:$HEALTHY_PID" attached_pid=$HEALTHY_PID cycle_begin "$attached_pid" attached "$HEALTHY_IDENTITY" @@ -614,6 +658,7 @@ fi # then, not as an immediate empty wake. (--restart skips this: it just stopped # this home's watcher and wants a fresh one.) if [ "$mode" = arm ] && healthy_watcher; then + take_over_from_leftover_daemon "$HEALTHY_PID" cycle_mark_predecessor_successor "attached:$HEALTHY_PID" cycle_begin "$HEALTHY_PID" attached "$HEALTHY_IDENTITY" report_attached @@ -696,6 +741,7 @@ owned_child_finished() { if [ "$rc" -eq 0 ]; then if wait_for_healthy_successor; then + take_over_from_leftover_daemon "$HEALTHY_PID" cycle_log_append "$rc" "$signal" unexpected-clean-exit "attached:$HEALTHY_PID" print_watch_output "$child_out" rm -f "$child_out" 2>/dev/null || true diff --git a/docs/architecture.md b/docs/architecture.md index 4b2b6f9cbfe..bc19adccd32 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -196,6 +196,8 @@ While the away record exists neither supervisor rechecks an item held for the ca On Pi and pi-signed the away daemon is no longer launched: the ordinary supervision session continues under the record with main parked, so the supervision branch takes every actionable wake, captain outcomes accumulate for the return brief, and main's standing authority relocates to the branch through the guarded scripts, each keeping its own gate ([`pi-supervision-branch.md`](pi-supervision-branch.md#postures)); a wake the branch cannot take and a watcher failure still reach main. On a non-Pi home that runs the [supervision host](supervision-host.md), the host runs the away session instead of the daemon. A presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) still extends walk-away supervision on the remaining harnesses: the `/afk` skill starts it through the tracked foreground helper `bin/fm-afk-start.sh` once the record exists, after which the watcher reverts to daemon-managed one-shot mode and the daemon self-handles routine wakes in bash. +The daemon lives only as long as `state/.afk` and its singleton lock: it exits on its own, before it drains another wake, once either is gone, because a daemon left running drains and acknowledges every wake into a buffer that nothing delivers while attended. +`bin/fm-afk-daemon-lib.sh` owns finding a home's live daemons, including one whose lock is gone or whose lock identity drifted after a time-zone change, for the return's shutdown and for the arm, which stops a leftover daemon that still runs the watcher while `state/.afk` is absent instead of attaching to its cycle. The watcher and daemon share `bin/fm-classify-lib.sh` for captain-relevant status verbs, declared-wait vocabulary (a `paused:` external wait and a verified `captain-held` transfer alike, through one combined predicate), and status-scan primitives. Terminal verbs remain captain-relevant, while a nonterminal progress verb cannot become terminal merely because its prose contains a legacy free-text token such as `merged`; bare legacy free-text lines remain compatible. The shared latest-event read takes the most recent line that leads with a recognized verb, a legacy token, or an unrecognized status prefix, so a bad declaration stays visible as itself while continuation prose and trailing blank lines after a multi-line record cannot hide a declared wait. diff --git a/docs/scripts.md b/docs/scripts.md index 8d656ec203e..065e3f35e81 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -97,6 +97,7 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md | `fm-afk-start.sh` | Run the common sourceable away-mode daemon entry in the foreground | | `fm-afk-launch.sh` | Own away/quiet entry (same-turn record write, then read-back), exit, rollback, and any backend terminal lifecycle | | `fm-afk-return.sh` | Own deterministic return shutdown, the return brief, catch-up evidence, and the firstmate-actionable blocker gate | +| `fm-afk-daemon-lib.sh` | Find a home's live away-mode daemons, including one without its lock, through its instance record, its lock, or the watcher it runs | | `fm-supervisor-target-lib.sh` | Resolve the shared supervisor target and backend for the daemon and launcher | | `fm-supervise-daemon.sh` | Presence-gated away-mode sub-supervisor: self-handle routine wakes, guard injection by the detected primary harness, escalate batched digests, alert on failed delivery | | `fm-crew-state.sh` | Print one deterministic current-state line for a crew | diff --git a/tests/fm-afk-launch.test.sh b/tests/fm-afk-launch.test.sh index 4cf07950437..630ee91e8fd 100755 --- a/tests/fm-afk-launch.test.sh +++ b/tests/fm-afk-launch.test.sh @@ -1525,6 +1525,138 @@ unit_stop_confirms_daemon_exit() { rm -rf "$st" } +# A stand-in away daemon: a live process whose command line runs a script named +# fm-supervise-daemon.sh, so a lock identity that no longer matches cannot be +# mistaken for a reused pid. It exits on TERM, as the real daemon does, and +# stops itself after a bound so an escaped copy cannot outlive the suite. +make_daemon_standin() { # + mkdir -p "$1" + cat > "$1/fm-supervise-daemon.sh" <<'SH' +#!/usr/bin/env bash +trap 'exit 0' TERM +end=$(( $(date +%s) + 120 )) +while [ "$(date +%s)" -lt "$end" ]; do sleep 0.2; done +SH + chmod +x "$1/fm-supervise-daemon.sh" + printf '%s\n' "$1/fm-supervise-daemon.sh" +} + +# Start a stand-in and wait until its exec chain settles, so an identity the +# fixture records matches the one the stand-in keeps; the real daemon records +# its own identity once it runs. Sets STANDIN_PID in this shell. +STANDIN_PID= +start_daemon_standin() { #