Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion bin/fm-push-transition-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,44 @@ _hb_surfaced_path() {
printf '%s/.hb-surfaced-%s' "$STATE" "$(printf '%s' "$1" | tr ':/.' '___')"
}

# Record a captain-relevant status after its durable wake has been enqueued.
_fm_surface_digest() {
if command -v md5 >/dev/null 2>&1; then md5 -q; else md5sum | cut -d ' ' -f1; fi
}

open_decision_id() { # <task>
local task=$1 open
[ -n "$task" ] || return 0
open=$(status_open_decisions "$STATE/$task.status")
[ -n "$open" ] || return 0
printf '%s' "$open" | _fm_surface_digest
}

_stale_decision_marker_path() { # <task>
local task=$1 target key
target=$(fm_backend_target_of_meta "$STATE/$task.meta")
[ -n "$target" ] || return 0
key=$(printf '%s' "$target" | tr ':/.' '___')
printf '%s/.stale-decision-%s' "$STATE" "$key"
}

_reconcile_surfaced_open_decision() { # <task>
local task=$1 marker decision
marker=$(_stale_decision_marker_path "$task")
[ -n "$marker" ] || return 0
decision=$(open_decision_id "$task")
if [ -n "$decision" ]; then
printf '%s' "$decision" > "$marker"
else
rm -f "$marker"
fi
}

# Record a surfaced status after its durable wake has been enqueued.
mark_surfaced() { # <status-file>
local f=$1 task last
task=$(basename "$f"); task="${task%.status}"
case "$f" in *.status) ;; *) return 0 ;; esac
_reconcile_surfaced_open_decision "$task"
last=$(last_status_line "$f")
[ -n "$last" ] || return 0
status_is_captain_relevant "$last" || return 0
Expand Down
77 changes: 56 additions & 21 deletions bin/fm-watch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,16 @@
# Firstmate watcher.
# Classifies supervision wakes in bash. In normal mode it absorbs benign wakes
# and keeps blocking; it queues and exits only for actionable wakes.
# The no-verb signal and stale path is absorb-only-when-provably-working: a wake
# is absorbed only when the crew shows POSITIVE evidence it is still working (an
# actively-running no-mistakes step, or a backend busy signal), and surfaced
# otherwise, so a crew that finishes (or stops and waits) without a current
# working signal is never silently swallowed. A declared external-wait pause is
# the separate idle absorb case and re-surfaces only on its long bounded cadence,
# although its initial no-verb status signal still surfaces in normal mode.
# The no-verb signal and first-sighting stale paths are
# absorb-only-when-provably-working: a wake is absorbed only when the crew shows
# POSITIVE evidence it is still working (an actively-running no-mistakes step,
# or a backend busy signal), and surfaced otherwise, so a crew that finishes
# without a current working signal is never silently swallowed. A declared
# external-wait pause is the separate idle absorb case and re-surfaces only on
# its long bounded cadence. A repaint-only repeat of an already-surfaced keyed
# open-decision set is also absorbed, but a confidently dead parked agent still
# enters the wedge timer. The initial no-verb status signal still surfaces in
# normal mode.
# While state/.afk exists, the daemon owns triage and this watcher queues and exits
# on every wake. Printed reason lines:
# signal: <file>... status/turn-end signals, surfaced when a listed status
Expand All @@ -20,8 +23,11 @@
# line, since the crew's own log gets no new entry once
# firstmate hands it to a no-mistakes validation. A declared
# external-wait pause is absorbed instead with its own long
# re-surface cadence, never as a wedge. Only when neither
# absorb class applies does the log's last line decide:
# re-surface cadence, never as a wedge. An unchanged keyed
# open-decision set is deduped across pane repaint, unless
# the backend confidently reports the parked agent dead and
# routes it through the wedge timer. Only when none of these
# absorb classes applies does the log's last line decide:
# terminal (captain-relevant) or non-terminal (no verb),
# both surfaced at once. A provably-working stale past the
# wedge threshold also surfaces, with an "escalation N"
Expand Down Expand Up @@ -119,20 +125,22 @@ SIGNAL_GRACE=${FM_SIGNAL_GRACE:-30} # seconds to linger after a signal so trai
# than wake firstmate's LLM for each, this watcher classifies every wake in bash
# and ABSORBS the benign majority - it advances the suppression marker, logs to a
# debug log, and keeps blocking WITHOUT enqueuing or exiting. The no-verb signal
# / stale path is absorb-only-when-provably-working: such a wake is absorbed ONLY
# while the crew shows positive evidence it is still working (an actively-running
# no-mistakes step, or a busy pane, via crew_is_provably_working over
# fm-crew-state.sh); a crew that stopped its turn with no running pipeline and no
# / first-sighting stale path is absorb-only-when-provably-working: such a wake
# is absorbed ONLY while the crew shows positive evidence it is still working
# (an actively-running no-mistakes step, or a busy pane, via
# crew_is_provably_working over fm-crew-state.sh); a crew that stopped its turn
# with no running pipeline and no
# busy pane is SURFACED, so a finish reported only through interactive pane menus
# (no done: status) is never swallowed. An ACTIONABLE wake (a captain-relevant
# signal, a no-verb signal whose crew is not provably working, any check, a stale
# pane whose crew is not provably working, a provably-working stale past the
# threshold, or anything unknown) is written to the durable queue and exits, which
# is what wakes the LLM through the background-task completion. The same classifier
# signal, a no-verb signal whose crew is not provably working, any check, a
# first-sighting stale pane whose crew is not provably working, a
# provably-working stale past the threshold, or anything unknown) is written to
# the durable queue and exits, which is what wakes the LLM through the
# background-task completion. The same classifier
# (fm-classify-lib.sh) backs the away-mode daemon; while state/.afk exists the
# daemon owns triage, so this watcher reverts to one-shot (enqueue + exit on every
# wake) and never double-triages - and never runs the costly provably-working read.
STALE_ESCALATE_SECS=${FM_STALE_ESCALATE_SECS:-240} # idle secs before a provably-working stale escalates as a possible wedge
STALE_ESCALATE_SECS=${FM_STALE_ESCALATE_SECS:-240} # idle secs before a provably-working stale or dead parked-decision repeat escalates as a possible wedge
# A busy pane is unconditional proof of liveness with no built-in duration bound,
# so a hung foreground call can remain hidden even while its rendered busy
# footer changes every poll. BUSY_TURN_MAX_SECS bounds how long any busy pane
Expand Down Expand Up @@ -173,7 +181,7 @@ _event_cap_fails=0
afk_present() { [ -e "$STATE/.afk" ]; }

hash_pane() {
if command -v md5 >/dev/null 2>&1; then md5 -q; else md5sum | cut -d' ' -f1; fi
_fm_surface_digest
}

# window_is_busy: 0 (busy) iff the task's harness is PROVABLY working, through
Expand Down Expand Up @@ -406,6 +414,16 @@ pause_state_class() { # <window> <task>
printf '%s' "$class"
}

# 0 when a parked crew has stopped responding: its recorded endpoint no longer
# carries a live agent. Only the confident `dead` verdict counts (an ambiguous,
# unreadable, or unverified backend read stays absorbed), so suppressing the
# repeat surface above never costs wedge detection for a parked crew that dies.
parked_agent_is_dead() { # <window>
local win=$1 state
state=$(fm_backend_agent_alive "$(window_backend "$win")" "$win" 2>/dev/null) || return 1
[ "$state" = dead ]
}

surface_nonterminal_stale() { # <window> <hash>
local win=$1 h=$2 key task last
key=$(printf '%s' "$win" | tr ':/.' '___')
Expand Down Expand Up @@ -544,7 +562,7 @@ mark_all_captain_relevant_surfaced() {
local f task last
while IFS=$(printf '\t') read -r f task last; do
[ -n "$f" ] || continue
printf '%s' "$last" > "$(_hb_surfaced_path "$task")"
mark_surfaced "$f"
done < <(scan_captain_relevant_statuses "$STATE")
}

Expand Down Expand Up @@ -918,7 +936,24 @@ EOF
# line. On a NEW hash, give an active run/busy pane (the same
# authoritative source fm-crew-state.sh itself already prioritizes
# over the log) a chance to override before trusting the log.
if [ "$(cat "$sf" 2>/dev/null || true)" != "$h" ]; then
#
# Key this one-shot on the complete open-decision set, not volatile
# pane bytes. mark_surfaced reconciles this marker for every surfaced
# status, while a confidently dead parked agent still advances the
# shared wedge timer below. docs/architecture.md owns the full wake
# contract.
did=$(open_decision_id "$task")
dsf="$STATE/.stale-decision-$key"
[ -n "$did" ] || rm -f "$dsf"
if [ -n "$did" ] && [ "$(cat "$dsf" 2>/dev/null || true)" = "$did" ]; then
printf '%s' "$h" > "$sf"
if parked_agent_is_dead "$w"; then
wedge_timer_check "$w" "$ssf" "stale (parked on an open decision, agent gone)" "$ewf"
else
rm -f "$ssf" "$ewf"
triage_log "absorbed stale (open decision already surfaced): $w"
fi
elif [ "$(cat "$sf" 2>/dev/null || true)" != "$h" ]; then
if crew_is_provably_working "$(window_to_task "$w" "$STATE")"; then
printf '%s' "$h" > "$sf"
date +%s > "$ssf"
Expand Down
6 changes: 5 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ firstmate's always-loaded operating contract and routing index for conditional p
## Event-driven supervision

A zero-token bash watcher (`bin/fm-watch.sh`) sleeps on the fleet, classifies detected wakes in bash, and wakes the first mate only when something is actionable.
Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or an X-mode mention, stale panes whose crew is not provably working whether their status log looks terminal or non-terminal, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS`, declared external waits that remain paused past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits.
Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or an X-mode mention, first-sighting stale panes whose crew is not provably working, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS`, declared external waits that remain paused past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits.
Normal mode separately absorbs repaint-only repeats of an already-surfaced keyed open-decision set unless the backend confidently reports that the parked agent is dead.
Repeated provably-working stale escalations on the same unchanged pane add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker.
A busy pane is otherwise exempt from staleness, but only until its latest `state/<id>.turn-ended` marker reaches `FM_BUSY_TURN_MAX_SECS`, or its `state/<id>.meta` spawn record reaches that age before any turn completes; past that bound it is routed through the same wedge escalation, with the identical reason, escalation count, and `demand-deep-inspection` marker, for inspection only - never an automatic interrupt, signal, or restart.
Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so a missed process exit can be recovered by draining the queue.
Expand All @@ -23,6 +24,9 @@ For an ordinary crew that has stopped, the normal-mode watcher first surfaces on
Live or inconclusive liveness remains fail-open at that initial surface, and the secondmate idle-endpoint exemption is unchanged.
Its initial normal-mode status signal still surfaces through the no-verb path, while away mode self-handles that routine signal and owns the later recheck.
Fresh stale panes use the same current-state read before trusting the status log, so an active run or a proven busy worker outranks an old captain-relevant status-log line left behind before validation.
When that captain-relevant status includes a still-open keyed decision, the one-shot suppressor is keyed on a digest of the complete open-decision set returned by `bin/fm-classify-lib.sh`'s `status_open_decisions` fold rather than on the pane hash, because an idle harness pane repaints on its own and a hash-keyed suppressor re-surfaced an unchanged already-escalated waiting state on every repaint.
While the backend reports the agent live or returns inconclusive liveness, an unchanged open-decision set therefore surfaces only once despite pane repaint, while a fresh actionable status signal, a changed decision key or set, and resolution still surface normally.
A parked crew whose backend confidently reports its agent dead still escalates through the same wedge timer.
No-change heartbeats are also benign.
Absorbed wakes advance their suppression markers, log to `state/.watch-triage.log`, and keep the watcher blocking without a queue record or LLM turn.
After each drain, `fm-wake-drain.sh` runs the same liveness guard as the supervision scripts, so a lapsed watcher chain surfaces even on a turn that only drains and handles queued wakes.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -470,7 +470,7 @@ FM_WATCHER_STALE_GRACE=300 # defaults to FM_GUARD_GRACE; seconds a live watche
FM_SIGNAL_GRACE=30 # seconds to coalesce nearby status and turn-end signals into one wake
FM_CAPTAIN_RE='done:|needs-decision:|blocked:|failed:|PR ready|checks green|ready in branch|merged' # captain-relevant status regex; nonterminal progress verbs remain excluded even when their prose matches
FM_CLASSIFY_PAUSED_VERB=paused # leading status verb for a declared external wait; excluded from FM_CAPTAIN_RE and distinct from blocked
FM_STALE_ESCALATE_SECS=240 # idle seconds before a provably-working stale pane escalates; stale panes whose crew is not provably working surface immediately unless they declare the pause verb
FM_STALE_ESCALATE_SECS=240 # idle seconds before a provably-working stale pane or a confidently dead parked-decision repeat escalates; other first-sighting stale states surface immediately unless they declare the pause verb
FM_BUSY_TURN_MAX_SECS=3600 # maximum age of a busy pane's latest state/<id>.turn-ended marker, or its state/<id>.meta spawn record before any turn completes, before the same wedge escalation used for a provably-working non-busy stale takes over; inspection-only, never an automatic interrupt or restart
FM_PAUSE_RESURFACE_SECS=3600 # seconds before an idle declared external wait re-surfaces for a recheck in the watcher or away-mode daemon
FM_WEDGE_DEMAND_INSPECT_COUNT=3 # consecutive provably-working stale escalations on the same unchanged pane before demand-deep-inspection is added
Expand Down
Loading
Loading