Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ A lock-refused session must not spawn, steer, merge, drain the wake queue, repai
When the lock could not be acquired, the worktree-tangle check uses read-only advisory wording without a checkout repair command.
The five MUTATING sweeps - non-executing legacy PR-check migration, fleet sync, the local secondmate fast-forward sweep, the secondmate liveness sweep, and X-mode artifact writes - run only when this session actually holds the lock from step 1.
The secondmate liveness sweep deterministically guarantees every registered secondmate is actually running: it probes each live secondmate's endpoint for a real agent process (not just pane presence) and respawns only on a confident dead reading, reported as `SECONDMATE_LIVENESS:` lines (`bin/fm-bootstrap.sh`; `bin/fm-backend.sh`'s `fm_backend_agent_alive`).
3. **Wake queue** - when locked, drains the durable wake queue and prints the records prominently as this turn's first work queue, exactly as `bin/fm-wake-drain.sh` did before; a lapsed watcher chain still surfaces here via the same guard banner.
3. **Wake queue** - when locked, drains the durable wake queue and prints the records prominently as this turn's first work queue, exactly as `bin/fm-wake-drain.sh` did before; a lapsed watcher chain still surfaces here via the same guard alarm.
When the lock could not be acquired, the queue is left untouched because another session owns it, and the guard's tangle/watcher-liveness alarms still print in read-only advisory mode without drain, supervision repair, or checkout repair commands.
4. **Context digest** - the full contents of `data/projects.md`, `data/secondmates.md`, `data/captain.md`, and `data/learnings.md`, each clearly delimited.
A file that does not exist prints an explicit `ABSENT` marker, never confused with an empty-but-present file: absence is meaningful (`captain.md` absent means use this template's defaults, `projects.md` absent means rebuild it from the clones under `projects/`, etc.).
Expand Down
169 changes: 137 additions & 32 deletions bin/fm-guard.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,14 @@
# liveness beacon (state/.last-watcher-beat, touched every poll cycle) is
# missing or older than FM_GUARD_GRACE seconds, prints a loud, clearly delimited
# banner so the agent cannot skim past it in the tool output of whatever it was
# doing - the one channel every harness has. Normal wake handling (watcher
# briefly down between a wake and the next supervision resume) stays inside the
# grace window and stays silent. Always exits 0: the guard warns, it never
# blocks.
# doing - the one channel every harness has. The full banner is emitted once per
# distinct staleness episode in this FM_HOME (keyed to beacon mtime or absence);
# later guarded commands in the same episode print a one-line reminder instead.
# Episode state lives only under state/.guard-watcher-stale-banner (volatile,
# bounded). Independent alarms (queued wakes, worktree tangle) are never
# suppressed by that dedup. Normal wake handling (watcher briefly down between a
# wake and the next supervision resume) stays inside the grace window and stays
# silent. Always exits 0: the guard warns, it never blocks.
set -u

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Expand All @@ -26,13 +30,89 @@ READ_ONLY=${FM_GUARD_READ_ONLY:-0}
case "$READ_ONLY" in 1|true|TRUE|yes|YES) READ_ONLY=1 ;; *) READ_ONLY=0 ;; esac
CONTINUE_LINE=${FM_GUARD_CONTINUE_LINE:-This is a supervision warning only; the guarded operation WILL still run.}

# Volatile, home-scoped episode marker: one line = the current stale-episode key.
# Cleared when the home leaves the unhealthy state so a later episode re-arms.
STALE_BANNER_MARKER="$STATE/.guard-watcher-stale-banner"

# shellcheck source=bin/fm-wake-lib.sh
. "$SCRIPT_DIR/fm-wake-lib.sh"
# shellcheck source=bin/fm-tangle-lib.sh
. "$SCRIPT_DIR/fm-tangle-lib.sh"
# shellcheck source=bin/fm-supervision-lib.sh
. "$SCRIPT_DIR/fm-supervision-lib.sh"

# Deterministic episode key from beacon state: same continuous stale beacon
# (or continuous absence) shares a key; a recovered-then-restale beacon gets a
# new mtime and therefore a new episode.
fm_guard_stale_episode_key() {
local state=$1 beat m
beat="$state/.last-watcher-beat"
if [ -e "$beat" ]; then
m=$(fm_sup_stat_mtime "$beat")
printf 'beat:%s\n' "${m:-unknown}"
else
printf 'beat:absent\n'
fi
}

# Claim the full banner for this episode. Exit 0 = print full banner (this call
# owns the first announcement). Exit 1 = same episode already announced (print
# reminder). The shared wake lock helper owns the race-safety mechanics; the
# re-check under the lock makes concurrent claims idempotent.
fm_guard_claim_stale_banner() {
local state=$1 key=$2
local marker="$state/.guard-watcher-stale-banner"
local lock="$state/.guard-watcher-stale-banner.lock"
local seen i

seen=$(cat "$marker" 2>/dev/null || true)
# Strip a single trailing newline so key comparison is line-content based.
seen=${seen%$'\n'}
if [ "$seen" = "$key" ]; then
return 1
fi

i=0
while [ "$i" -lt 50 ]; do
if fm_lock_try_acquire "$lock"; then
seen=$(cat "$marker" 2>/dev/null || true)
seen=${seen%$'\n'}
if [ "$seen" = "$key" ]; then
fm_lock_release "$lock" 2>/dev/null || true
return 1
fi
# Bounded write: one line, no growth across episodes (overwrite).
printf '%s\n' "$key" > "$marker" || true
fm_lock_release "$lock" 2>/dev/null || true
return 0
fi
seen=$(cat "$marker" 2>/dev/null || true)
seen=${seen%$'\n'}
if [ "$seen" = "$key" ]; then
return 1
fi
# Brief yield; 0.02s is fine on macOS/Linux sleep, fall back to 1s.
sleep 0.02 2>/dev/null || sleep 1
i=$((i + 1))
done
# Contended past the spin budget: stay loud rather than dropping the alarm.
return 0
}

fm_guard_stale_banner_seen() {
local state=$1 key=$2
local marker="$state/.guard-watcher-stale-banner"
local seen

seen=$(cat "$marker" 2>/dev/null || true)
seen=${seen%$'\n'}
[ "$seen" = "$key" ]
}

fm_guard_clear_stale_banner() {
rm -f "$STALE_BANNER_MARKER" 2>/dev/null || true
}

# Worktree-tangle alarm, checked FIRST and independent of in-flight tasks: the
# firstmate PRIMARY checkout (FM_ROOT) must stay on its default branch. If a
# crewmate's branch/commits landed here instead of in its own isolated worktree,
Expand Down Expand Up @@ -68,43 +148,68 @@ fm_supervision_status "$STATE" "$GRACE"
in_flight=$FM_SUP_IN_FLIGHT
watcher_fresh=$FM_SUP_WATCHER_FRESH
beacon_desc=$FM_SUP_BEACON_DESC
[ "$in_flight" -eq 0 ] && exit 0
if [ "$in_flight" -eq 0 ]; then
# Leave the unhealthy state (no work riding on the watcher): clear so a later
# in-flight + stale combination is a fresh episode even if the beacon is still
# absent with the same key string.
[ "$READ_ONLY" -eq 1 ] || fm_guard_clear_stale_banner
exit 0
fi

[ -s "$FM_WAKE_QUEUE" ] && queue_pending=true

# No fresh watcher with tasks in flight is the dangerous state: emit a prominent,
# bordered banner FIRST so it reads as an alarm, not a buried stderr line.
# bordered banner FIRST so it reads as an alarm, not a buried stderr line. Later
# calls in the same episode get a one-line reminder only.
if [ "$watcher_fresh" = false ]; then
afk=0
[ -e "$STATE/.afk" ] && afk=1
queue_arg=0
"$queue_pending" && queue_arg=1
x_mode=0
[ -f "$CONFIG/x-mode.env" ] && x_mode=1
fix=$("$SCRIPT_DIR/fm-supervision-instructions.sh" \
--read-only "$READ_ONLY" \
--afk "$afk" \
--x-mode "$x_mode" \
--queue-pending "$queue_arg" \
--repair-line 2>/dev/null || printf '%s\n' 'Resume supervision according to the session-start operating block.')
rule='━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━'
{
printf '●%s\n' "$rule"
printf '● WATCHER DOWN - SUPERVISION IS OFF\n'
printf '● %s task(s) in flight, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$in_flight" "$beacon_desc" "$GRACE"
if [ "$READ_ONLY" -eq 1 ]; then
printf '● This read-only session should report the lapse, not repair it.\n'
else
printf '● Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.\n'
fi
printf '● %s\n' "$CONTINUE_LINE"
printf '● %s\n' "$fix"
printf '●%s\n' "$rule"
} >&2
episode_key=$(fm_guard_stale_episode_key "$STATE")
episode_key=${episode_key%$'\n'}
print_full_banner=0
if [ "$READ_ONLY" -eq 1 ]; then
fm_guard_stale_banner_seen "$STATE" "$episode_key" || print_full_banner=1
elif fm_guard_claim_stale_banner "$STATE" "$episode_key"; then
print_full_banner=1
fi
if [ "$print_full_banner" -eq 1 ]; then
afk=0
[ -e "$STATE/.afk" ] && afk=1
queue_arg=0
"$queue_pending" && queue_arg=1
x_mode=0
[ -f "$CONFIG/x-mode.env" ] && x_mode=1
fix=$("$SCRIPT_DIR/fm-supervision-instructions.sh" \
--read-only "$READ_ONLY" \
--afk "$afk" \
--x-mode "$x_mode" \
--queue-pending "$queue_arg" \
--repair-line 2>/dev/null || printf '%s\n' 'Resume supervision according to the session-start operating block.')
rule='━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━'
{
printf '●%s\n' "$rule"
printf '● WATCHER DOWN - SUPERVISION IS OFF\n'
printf '● %s task(s) in flight, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$in_flight" "$beacon_desc" "$GRACE"
if [ "$READ_ONLY" -eq 1 ]; then
printf '● This read-only session should report the lapse, not repair it.\n'
else
printf '● Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.\n'
fi
printf '● %s\n' "$CONTINUE_LINE"
printf '● %s\n' "$fix"
printf '●%s\n' "$rule"
} >&2
else
printf 'WARNING: watcher still down (same stale episode; last beat: %s, grace %ss) - full banner already printed this episode.\n' \
"$beacon_desc" "$GRACE" >&2
fi
else
# Healthy again while work is still in flight: end the episode so a later
# restale re-prints the full banner.
[ "$READ_ONLY" -eq 1 ] || fm_guard_clear_stale_banner
fi

# Queued wakes are an independent hazard; warn whenever they are pending, even if
# a watcher is alive. Kept after the banner so the no-watcher alarm reads first.
# Dedup of the watcher-down banner never suppresses this warning.
if "$queue_pending"; then
if [ "$READ_ONLY" -eq 1 ]; then
echo "WARNING: queued wakes pending - left untouched for the session holding the fleet lock." >&2
Expand Down
4 changes: 2 additions & 2 deletions bin/fm-session-start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -276,8 +276,8 @@ fi
# --- 3. wake-drain -------------------------------------------------------
# Drained records are this turn's first work queue (AGENTS.md section 8); the
# drain also runs fm-guard.sh internally on the locked path, so the
# tangle/watcher-liveness banners land right here too, ahead of the bulk
# digest below. The read-only path never touches the queue (another session
# tangle/watcher-liveness alarms land right here too, ahead of the bulk digest
# below. The read-only path never touches the queue (another session
# may be actively draining it) but still runs fm-guard.sh directly with
# non-mutating advisory text, so the same alarms surface without repair
# commands.
Expand Down
2 changes: 1 addition & 1 deletion bin/fm-wake-drain.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ DRAIN_LOCK_HELD=false
# every wake-handling and recovery turn, so assert watcher liveness here too. A
# lapsed supervision chain then surfaces on a plain drain-and-handle turn, not
# only when a guarded supervision script (fm-peek/fm-send/...) happens to run.
# Reuse fm-guard.sh's existing graced, beacon-based banner (FM_GUARD_GRACE) - do
# Reuse fm-guard.sh's existing graced, beacon-based alarm (FM_GUARD_GRACE) - do
# not duplicate the beacon math. Because the watcher touches its beacon every
# poll cycle, a normal fire leaves a recent beacon well inside grace and stays
# silent; only a genuine stale-beyond-grace lapse with work in flight warns. Call
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ On `attached` it stays live until that existing cycle ends so background-notify
Its `--restart` mode signals only the watcher recorded in the current home's `state/.watch.lock`, so restarting one home cannot kill sibling secondmate watchers.
A pull-based guard (`bin/fm-guard.sh`) warns through supervision tool output if the primary checkout is tangled, or if tasks are in flight and that watcher stops running or queued wakes are waiting to be drained.
The drain script calls that guard after emptying the queue, which avoids repeating the queued-wakes warning for records it just consumed while still warning on stale watcher liveness.
It leads with prominent bordered banners for the tangle and no-watcher cases so they cannot be skimmed past.
It leads with a prominent bordered tangle banner, while `bin/fm-guard.sh` owns the stale-watcher banner/reminder policy so repeated guarded commands stay noisy without reprinting the full watcher-down banner in the same episode.
On every verified primary harness, tracked hook integration gives the primary session a push-based backstop: when work is in flight and no identity-matched watcher lock with a fresh beacon is live, direct Stop hooks block and passive turn-end hooks force one bounded follow-up.
The guard covers the main primary and genuinely marked secondmate homes, exempts child crewmate/scout worktrees, is loop-safe per harness, and is documented in [turnend-guard.md](turnend-guard.md).

Expand Down
Loading