Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ projects/ cloned repos; gitignored; read-only except under hard rule
state/ volatile runtime signals; gitignored
<id>.status appended by crewmates: "<state>: <note>" wake-event lines, not current-state truth
<id>.turn-ended task completion signal; harness-adapters routes each producer contract to its authoritative implementation
<id>.run-step last observed no-mistakes run step; private cache owned by bin/fm-crew-state.sh and removed by teardown
<id>.agy-trust agy trust cleanup marker; exact lifecycle lives in bin/fm-agy-trust-lib.sh
<id>.grok-turnend-token firstmate-owned grok hook registry token for the task; removed by teardown
<id>.kimi-turnend-token firstmate-owned Kimi hook registry token for the task; removed by teardown
Expand Down Expand Up @@ -326,7 +327,7 @@ Send the same worker one exact decision naming the decision key, step, action, a
Require the matching `resolved` event, forbid `--yes`, and require the worker to process every synchronous return until completion or a genuinely new escalation.
Resume fleet supervision immediately after the decision lands.

Judge validation by the current-code-matched run step through `bin/fm-crew-state.sh`, not by shell liveness or the last status event.
Judge validation by the reconciled run-step verdict from `bin/fm-crew-state.sh`, not by shell liveness or the last status event.
Running, fixing, or CI states remain working; parked approval or fix-review states require the worker to follow the active gate help; passed or checks-passed is done; failed or cancelled is failed.
A worker hand-editing, committing, aborting, or restarting during an active validation run duplicates pipeline ownership outside the supersession sequence above; steer it back to the gate response flow.
The worker reports the PR when CI first becomes green rather than waiting for merge monitoring to finish.
Expand Down
6 changes: 4 additions & 2 deletions bin/fm-classify-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ signal_reason_is_actionable() { # <file> ...
# Classify WHY an idle/stale crew MIGHT be safely absorbed instead of surfaced,
# from bin/fm-crew-state.sh's one authoritative current-state line
# ("state: <s> · source: <src> · <detail>"). Prints exactly one token:
# working - an actively-running no-mistakes step (running/fixing/ci) or a busy
# working - a current no-mistakes step, its bounded degraded replay, or a busy
# pane; the crew is legitimately mid-work on a static-looking pane
# (e.g. waiting on CI);
# paused - the crew's authoritative current state is a declared external-wait
Expand All @@ -340,7 +340,9 @@ crew_absorb_class() { # <id>
if [ "$state" = paused ]; then printf 'paused'; return; fi
if [ "$state" = working ]; then
src=${line#*source: }; src=${src%% *}
case "$src" in run-step|pane) printf 'working'; return ;; esac
# A working run-step-degraded verdict is bounded positive evidence whose
# replay and safety rules are owned by fm-crew-state.sh.
case "$src" in run-step|run-step-degraded|pane) printf 'working'; return ;; esac
fi
printf 'none'
}
Expand Down
393 changes: 310 additions & 83 deletions bin/fm-crew-state.sh

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion bin/fm-fleet-snapshot.sh
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,10 @@ task_json_lines() {
# multiplex many concerns onto one stream, so activity on one concern must
# never clear another concern's keyed decision. A parked/blocked state, or a
# non-authoritative status-log/none read on a still-live task, keeps the fold's
# open decision surfacing.
# open decision surfacing. `run-step-degraded` is deliberately absent from the
# live-activity sources: it is a remembered step the reader could not
# re-confirm, which is enough to keep a crew provably working for wedge triage
# but never enough to clear a captain decision.
open_decisions_tsv=$(status_open_decisions "$status_log")
if [ "$kind" != secondmate ] && \
{ { { [ "$current_source" = run-step ] || [ "$current_source" = pane ]; } \
Expand Down
4 changes: 3 additions & 1 deletion bin/fm-teardown.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1676,6 +1676,7 @@ cleanup_firstmate_home_children() {
retire_busy_state "$sub_state" "$child_id" "$child_busy_gen" || return 1
rm -f "$sub_state/$child_id.status" "$sub_state/$child_id.turn-ended" \
"$sub_state/$child_id.meta" "$sub_state/$child_id.pi-ext.ts" \
"$sub_state/$child_id.run-step" \
"$sub_state/$child_id.grok-turnend-token" "$sub_state/$child_id.kimi-turnend-token"
done
}
Expand Down Expand Up @@ -1934,7 +1935,8 @@ fi
remove_pr_poll_artifacts "$STATE" "$ID" || exit 1
retire_busy_state "$STATE" "$ID" "$BUSY_GEN" || exit 1
rm -f "$STATE/$ID.status" "$STATE/$ID.turn-ended" "$STATE/$ID.meta" \
"$STATE/$ID.pi-ext.ts" "$STATE/$ID.grok-turnend-token" \
"$STATE/$ID.pi-ext.ts" "$STATE/$ID.run-step" \
"$STATE/$ID.grok-turnend-token" \
"$STATE/$ID.kimi-turnend-token"
if [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && [ "$MODE" != local-only ]; then
"$FM_ROOT/bin/fm-fleet-sync.sh" "$PROJ" || true
Expand Down
28 changes: 15 additions & 13 deletions bin/fm-watch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,23 @@
# and keeps blocking; it queues and exits only for actionable wakes.
# The no-verb signal and first-sighting stale paths are
# absorb-only-when-provably-working: a wake is absorbed only when the crew shows
# POSITIVE evidence it is still working (an actively-running no-mistakes step,
# or a backend busy signal), and surfaced otherwise, so a crew that finishes
# without a current working signal is never silently swallowed. A declared
# external-wait pause is the separate idle absorb case and re-surfaces only on
# its long bounded cadence. A repaint-only repeat of an already-surfaced keyed
# open-decision set is also absorbed, but a confidently dead parked agent still
# enters the wedge timer. The initial no-verb status signal still surfaces in
# normal mode.
# POSITIVE evidence it is still working (a current or bounded-degraded
# no-mistakes run step, or a backend busy signal), and surfaced otherwise, so a
# crew that finishes (or stops and waits) without a current working signal is
# never silently swallowed. A declared external-wait pause is the separate idle
# absorb case and re-surfaces only on its long bounded cadence. A repaint-only
# repeat of an already-surfaced keyed open-decision set is also absorbed, but a
# confidently dead parked agent still enters the wedge timer. The initial no-verb
# status signal still surfaces in normal mode.
# While state/.afk exists, the daemon owns triage and this watcher queues and exits
# on every wake. Printed reason lines:
# signal: <file>... status/turn-end signals, surfaced when a listed status
# has a captain-relevant verb OR a no-verb signal's crew
# is not provably working, unless afk is active
# stale: <window> a provably-working stale is ALWAYS absorbed (with a wedge
# timer) regardless of what the status log says - an active
# run-step or busy pane outranks even a captain-relevant log
# current or bounded-degraded run step, or a busy pane,
# outranks even a captain-relevant log
# line, since the crew's own log gets no new entry once
# firstmate hands it to a no-mistakes validation. A declared
# external-wait pause is absorbed instead with its own long
Expand Down Expand Up @@ -132,7 +133,7 @@ SIGNAL_GRACE=${FM_SIGNAL_GRACE:-30} # seconds to linger after a signal so trai
# debug log, and keeps blocking WITHOUT enqueuing or exiting. The no-verb signal
# / first-sighting stale path is absorb-only-when-provably-working: such a wake
# is absorbed ONLY while the crew shows positive evidence it is still working
# (an actively-running no-mistakes step, or a busy pane, via
# (a current or bounded-degraded no-mistakes run step, or a busy pane, via
# crew_is_provably_working over fm-crew-state.sh); a crew that stopped its turn
# with no running pipeline and no
# busy pane is SURFACED, so a finish reported only through interactive pane menus
Expand Down Expand Up @@ -1085,9 +1086,10 @@ EOF
# poll. Root cause of the 2026-07 herdr false-surface incidents: a
# validating crew was surfaced as stale every few minutes despite an
# actively-running pipeline, purely because of this stale leftover
# line. On a NEW hash, give an active run/busy pane (the same
# authoritative source fm-crew-state.sh itself already prioritizes
# over the log) a chance to override before trusting the log.
# line. On a NEW hash, give a working current or bounded-degraded
# run/busy pane (the same authoritative source fm-crew-state.sh itself
# already prioritizes over the log) a chance to override before
# trusting the log.
#
# Key this one-shot on the complete open-decision set, not volatile
# pane bytes. mark_surfaced reconciles this marker for every surfaced
Expand Down
11 changes: 7 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ When a canonical validated PR poll returns exactly `merged`, the watcher appends
The receipt makes retirement safely retryable across restarts: fixed-path recovery revalidates the same evidence, removes the runnable check first, removes its registration and data sidecars, removes the receipt last, and preserves task metadata including `pr=` and `pr_head=`.
A concurrent replacement remains armed, every non-merged or invalid observation remains unchanged, and retirement never performs task or persistent-secondmate cleanup.
`bin/fm-pr-lib.sh` owns the receipt format and strict identity mechanics, while `bin/fm-watch.sh` owns queue-before-retirement ordering.
No-verb wakes, such as `working:` notes and bare turn-ended signals, are benign only when `bin/fm-crew-state.sh` reports positive evidence that the crew is still working: an actively running no-mistakes step attributed to that crew's current code, or an exact busy verdict from the semantic busy-state contract.
No-verb wakes, such as `working:` notes and bare turn-ended signals, are benign only when `bin/fm-crew-state.sh` reports positive evidence that the crew is still working: an attributed active no-mistakes step, its recent age-bounded degraded replay after a lookup failure, or an exact busy verdict from the semantic busy-state contract.
A crew that declares `paused:` for a known external wait is separately absorbed while idle and re-surfaced only on the longer pause cadence, rather than being treated as a possible wedge.
For an ordinary crew that has stopped, the normal-mode watcher first surfaces one stale wake, then applies that same cadence to an unchanged `paused:` or durable `captain-held` endpoint only when the backend confidently reports its agent dead.
Live or inconclusive liveness remains fail-open at that initial surface, and the secondmate idle-endpoint exemption is unchanged.
Expand All @@ -33,11 +33,14 @@ After each drain, `fm-wake-drain.sh` runs the same liveness guard as the supervi
Routine watcher polling, supervision no-ops, elapsed waiting time, and absorbed benign wakes stay silent.
A declared external wait trades that silence for one bounded recheck per pause window, so a forgotten pause cannot remain invisible indefinitely.
Crew status files are append-only wake-event logs, not current-state fields.
`bin/fm-crew-state.sh <id>` is the cheap current-state read for an actionable heartbeat review: it attributes a no-mistakes run, active or terminal, only when it matches the crew's branch and current code identity, then keeps that run-step authoritative even if the pane has closed.
The script header owns the exact run-head ancestry rules.
`bin/fm-crew-state.sh <id>` is the cheap current-state read for an actionable heartbeat review: it attributes a no-mistakes run to the crew's branch while reconciling the run head with pipeline-owned commit behavior, then keeps that run-step authoritative even if the pane has closed.
The script header owns the exact run-head relationship, current-versus-historical source, lookup-failure, cache, and source-precedence rules.
Architecturally, an in-progress pipeline may advance the crew tip or report a head unavailable in that worktree without losing attribution, while the narrowed exceptions continue to reject stale historical evidence.
A run lookup that cannot complete remains distinct from a confirmed absence and may replay that crew's recent observed step under source `run-step-degraded` only within a finite window after endpoint-liveness and exact-busy checks, preserving wedge detection when the crew stops or the outage persists.
During no-mistakes' `ci` monitor phase, it also reads the ci step log tail because `axi status` reports both "still waiting on checks" and "checks green, waiting on merge" as `ci,running`.
The most recent recognized ci log marker wins, so checks-green monitoring reports done while a later re-arm, failed-check, or issue marker returns the crew to working.
Only when no matching run exists does it consult semantic busy state; exact busy reports working, exact idle permits fallback to a status-log event whose verb maps to a recognized run-state, and unknown or a dead pane stays unknown instead of trusting a stale log.
When no run is attributed, an exact busy verdict reports working before any degraded or fallback source.
After a failed lookup, a usable degraded record then outranks idle or unavailable harness evidence and the status log; without that record, or after a completed lookup, exact idle permits fallback to a status-log event whose verb maps to a recognized run-state, while unknown or a dead pane stays unknown instead of trusting a stale log.
Decision-only events such as `resolved` never become current state or leak their prose into the current-state detail.
In that status-log fallback, a declared external wait reports the distinct `paused` state with its reason.
The semantic branch reports working only on an exact busy verdict and names the source that produced it; an unknown verdict never becomes working, never permits the status-log fallback, and never becomes a silent idle.
Expand Down
3 changes: 2 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,7 +516,8 @@ FM_PROCEVENT_MAX_OUTPUT_BYTES=1048576 # bound on one captured process-to-event
FM_PROCEVENT_CLAIM_ROOT= # machine-wide source claim root; default $XDG_STATE_HOME/firstmate/procevent-claims
FM_CODEX_WATCH_CHECKPOINT=180 # seconds per foreground watcher checkpoint in Codex primary supervision
FM_CREW_STATE_NM_TIMEOUT=10 # seconds allowed per no-mistakes query inside fm-crew-state.sh
FM_CREW_STATE_RUNS_LIMIT=200 # recent no-mistakes run rows scanned when axi status cannot be attributed to the current code
FM_CREW_STATE_RUNS_LIMIT=200 # recent no-mistakes run rows scanned when axi status cannot be attributed to the worktree
FM_CREW_STATE_DEGRADED_MAX_AGE=900 # seconds a recorded run-step may stand in as the degraded answer while the no-mistakes lookup cannot complete; 0 disables the degrade
FM_CREW_STATE_BIN=bin/fm-crew-state.sh # test override for the current-state reader used by working/paused watcher triage
FMX_PAIRING_TOKEN= # X mode pairing token; .env opt-in authorizes replies and eligible lifecycle actions
FMX_RELAY_URL=https://myfirstmate.io # optional X relay override, mainly for local relay development
Expand Down
Loading
Loading