Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .agents/skills/bootstrap-diagnostics/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,5 +72,7 @@ When any diagnostic needs captain attention, report the plain consequence and re
An unsafe-outbox variant requires path and file-type inspection before any retry.
- `NUDGE_SECONDMATES: secondmate <id>: send failed: <reason>` - secondmate convergence changed a running home's loaded instructions or inherited config, but the deterministic `fm-send.sh fm-<id>` re-read nudge failed.
Inspect the reason, keep the pending marker under `state/.secondmate-nudge-pending/` intact, and rerun session start after the endpoint or metadata issue is fixed so bootstrap can retry the exact same marked send on the same local or remote route.
- `NUDGE_SECONDMATES: secondmate <id>: deferred: ...` - the same nudge was not sent because that secondmate is waiting on its own open decision or blocker, and automatic nudges never wake a waiting worker.
Keep the pending marker; when you answer that decision, add the re-read instruction to the answer so the secondmate does not resume on stale instructions.
- `FMX: X mode on ...` / `FMX: X mode off ...` - bootstrap confirmed or removed the local Relay poll artifacts (`docs/configuration.md` "Relay (.env)"); the emitted line still carries Relay's former `X mode` wording.
Only when a running watcher needs the cadence transition applied immediately, restart the home-scoped watcher through the emitted harness supervision protocol; bootstrap deliberately never restarts the watcher itself.
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,7 @@ A persistent secondmate is recorded in the secondmate registry and runtime state
Steer a worker with ordinary text through fail-closed `fm-send`: the message becomes a durable record in the task's steering inbox (multi-line text is legal, local and remote alike) and the worker's terminal receives only a constant doorbell line, with the watcher re-ringing an unacknowledged local message and escalating a stuck one (`bin/fm-task-inbox-lib.sh`; `bin/fm-send.sh` owns the typed-plane carve-outs).
A remote secondmate steer rides the same durable-inbox model through the remote transport; after an unconfirmed delivery, only the exact `FM_PENDING_REPLY_EXISTING_CORR=<id>` resend command printed by `fm-send` is safe because it preserves the request body for remote enqueue deduplication (`bin/fm-send.sh` header).
When a steer answers an open keyed decision or blocker, pass `fm-send`'s `--resolve-key` so the answer itself closes that decision record at answer time, identically for local and remote workers (contract: `bin/fm-send.sh` header).
While a worker's decision or blocker is open, send it only the answer: a holding note or a nudge wakes a worker whose wait should cost nothing.
`fm-send` is the data plane for text the worker should read; never use its key or text paths for interrupt, exit, or other lifecycle control, because routing-marked lifecycle text becomes chat the worker reasons about instead of executing.
Drive a worker's lifecycle through `bin/fm-control.sh <task-id> interrupt|exit|relaunch`, which owns the per-runtime mechanics, verifies each action, and never tears down or discards anything ([`docs/agent-control.md`](docs/agent-control.md)).
A secondmate's routed reply returns through status or a document pointer, not by firstmate peeking into its chat.
Expand Down
36 changes: 27 additions & 9 deletions bin/fm-bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -395,8 +395,22 @@ secondmate_sync() {
fm_secondmate_nudge_write "$STATE" "$id" "$home" "$commit" "$instr" "$message" "$remote"
}

# fm-send exits 4 with a "deferred:" line while the mate waits on its own open
# decision; the retained marker retries the nudge at a later session start.
# The exit status is what classifies the result, not the shape of the output:
# anything the send prints ahead of that line must not read as a failure.
secondmate_nudge_unsent() { # <id> <fm-send-output> <fm-send-status>
local detail
if [ "${3:-1}" -eq 4 ]; then
detail=$(printf '%s\n' "$2" | grep -m1 '^deferred:') || detail=$(first_line "$2")
echo "NUDGE_SECONDMATES: secondmate $1: $detail"
else
echo "NUDGE_SECONDMATES: secondmate $1: send failed: $(first_line "$2")"
fi
}

secondmate_send_nudge() {
local id=$1 home=$2 commit=$3 instr=$4 selector marker out
local id=$1 home=$2 commit=$3 instr=$4 selector marker out send_rc
selector="fm-$id"
marker=$(secondmate_nudge_marker_path "$id") || {
echo "NUDGE_SECONDMATES: secondmate $id: send failed: unsafe id"
Expand All @@ -406,11 +420,12 @@ secondmate_sync() {
echo "NUDGE_SECONDMATES: secondmate $id: send failed: cannot record retry marker"
return 0
fi
if out=$(FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" "$SCRIPT_DIR/fm-send.sh" "$selector" "$SECOND_MATE_NUDGE_MESSAGE" 2>&1); then
out=$(FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" "$SCRIPT_DIR/fm-send.sh" "$selector" --automatic "$SECOND_MATE_NUDGE_MESSAGE" 2>&1) && send_rc=0 || send_rc=$?
if [ "$send_rc" -eq 0 ]; then
rm -f "$marker"
echo "BOOTSTRAP_INFO: nudged $selector with '$SECOND_MATE_NUDGE_MESSAGE'"
else
echo "NUDGE_SECONDMATES: secondmate $id: send failed: $(first_line "$out")"
secondmate_nudge_unsent "$id" "$out" "$send_rc"
fi
}

Expand All @@ -420,7 +435,7 @@ secondmate_sync() {
}

secondmate_retry_pending_nudges() {
local marker id selector home commit message remote expected_marker meta meta_home home_real head out
local marker id selector home commit message remote expected_marker meta meta_home home_real head out send_rc
[ -d "$SECOND_MATE_NUDGE_PENDING_DIR" ] || return 0
for marker in "$SECOND_MATE_NUDGE_PENDING_DIR"/*.pending; do
[ -f "$marker" ] || continue
Expand Down Expand Up @@ -479,11 +494,12 @@ secondmate_sync() {
echo "NUDGE_SECONDMATES: secondmate $id: send failed: retry target is not at recorded instruction commit"
continue
}
if out=$(FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" "$SCRIPT_DIR/fm-send.sh" "$selector" "$SECOND_MATE_NUDGE_MESSAGE" 2>&1); then
out=$(FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" "$SCRIPT_DIR/fm-send.sh" "$selector" --automatic "$SECOND_MATE_NUDGE_MESSAGE" 2>&1) && send_rc=0 || send_rc=$?
if [ "$send_rc" -eq 0 ]; then
rm -f "$marker"
echo "BOOTSTRAP_INFO: nudged $selector with '$SECOND_MATE_NUDGE_MESSAGE'"
else
echo "NUDGE_SECONDMATES: secondmate $id: send failed: $(first_line "$out")"
secondmate_nudge_unsent "$id" "$out" "$send_rc"
fi
done
}
Expand Down Expand Up @@ -623,12 +639,14 @@ secondmate_sync() {
fi
[ "$remote_pending" -eq 0 ] || nudge_needed=1
if [ "$converged" -eq 1 ] && [ "$nudge_needed" -eq 1 ]; then
if out=$(FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" \
"$SCRIPT_DIR/fm-send.sh" "fm-$id" "$REMOTE_SECOND_MATE_NUDGE_MESSAGE" 2>&1); then
local send_rc=0
out=$(FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" \
"$SCRIPT_DIR/fm-send.sh" "fm-$id" --automatic "$REMOTE_SECOND_MATE_NUDGE_MESSAGE" 2>&1) && send_rc=0 || send_rc=$?
if [ "$send_rc" -eq 0 ]; then
rm -f "$remote_marker"
[ "${FM_BOOTSTRAP_VERBOSE_FACTS:-0}" != 1 ] || echo "BOOTSTRAP_INFO: nudged remote fm-$id after convergence"
else
echo "NUDGE_SECONDMATES: secondmate $id: send failed: $(first_line "$out")"
secondmate_nudge_unsent "$id" "$out" "$send_rc"
fi
elif [ "$converged" -eq 1 ]; then
rm -f "$remote_marker"
Expand Down
28 changes: 25 additions & 3 deletions bin/fm-brief.sh
Original file line number Diff line number Diff line change
Expand Up @@ -208,16 +208,34 @@ INBOX_DIR=$(shell_quote "$STATE/$ID.inbox")
# The receive-and-ack half of the steering-inbox contract, included in every
# scaffold kind. The record format, doorbell line, and re-ring ladder are
# owned by bin/fm-task-inbox-lib.sh; the doorbell itself is self-describing,
# so this section is reinforcement for the natural-checkpoint habit, not the
# only carrier of the instruction.
# so this section is reinforcement, not the only carrier of the instruction.
# Every waiting record rings, so the worker never lists the inbox unprompted.
IFS= read -r -d '' INBOX_SECTION <<EOF || true
# Firstmate instruction inbox
Firstmate steers you through durable message files in $INBOX_DIR.
When a terminal message says an instruction is waiting there - and at any natural checkpoint when you are unsure - list $INBOX_DIR/*.msg, read and act on each message in numeric order, then acknowledge each handled message by moving it: \`mv $INBOX_DIR/NNN.msg $INBOX_DIR/handled/\`.
When a terminal message says an instruction is waiting there, list $INBOX_DIR/*.msg, read and act on each message in numeric order, then acknowledge each handled message by moving it: \`mv $INBOX_DIR/NNN.msg $INBOX_DIR/handled/\`.
The move IS the acknowledgement: without it firstmate rings again and eventually treats you as stuck. An empty or absent inbox needs no action.
Every waiting instruction rings, so never list the inbox on your own.
EOF
INBOX_SECTION=${INBOX_SECTION%$'\n'}

# How a crewmate or scout waits. Every model turn resends the whole context, so
# a wait must cost no turns: a decision wait ends the turn, and an external
# wait sleeps in one bounded blocking shell command sized to the harness.
IFS= read -r -d '' WAIT_SECTION <<'EOF' || true
# Waiting
Every turn you take resends your whole context, so a wait must cost no turns.
After you append `needs-decision:` or `blocked:`, end your turn at once: do not check the inbox, the status file, or anything else, because the answer arrives as a terminal message that starts your next turn.
Wait on anything external - a pipeline gate, PR checks, a heavy-test slot - with ONE blocking shell command that returns when the state changes: `no-mistakes axi run` or `respond` with `--wait`, `gh pr checks <pr> --watch`, or `until <condition>; do sleep 30; done` for anything else.
Never spend turns on `sleep` followed by a status check, and never background a command in order to poll it.
In Claude Code that `until` loop in a single Bash call is the sanctioned foreground wait: when the harness refuses a sleep-then-check command and points you at backgrounding instead, reissue the wait as the loop rather than accepting the background.
Bound that command by what your harness lets one command run: in Pi pass the bash tool a `timeout` of at most 2700 seconds, because Pi sets none by default; in Claude Code pass the Bash tool its maximum `timeout` of 600000 ms, because its default is 2 minutes; in Codex keep waiting on a still-running command with empty `write_stdin` polls of up to 300000 ms; elsewhere assume 10 minutes.
Give any `--wait` a duration a little under that bound.
When the bound passes with nothing changed, run the same blocking command again, with no status check in between.
A wait your shell can watch this way is not a `paused:` wait: stay in the command instead of declaring one.
EOF
WAIT_SECTION=${WAIT_SECTION%$'\n'}

if [ "$KIND" = secondmate ]; then
SECONDMATE_PROJECTS=""
idx=1
Expand Down Expand Up @@ -419,6 +437,8 @@ The report is the only thing that survives, so anything worth keeping must be in
the daemon accepts \`respond\` immediately and runs the round in the background, so a killed or
timed-out call was only waiting for a read while the run kept working.

$WAIT_SECTION

$INBOX_SECTION

# Definition of done
Expand Down Expand Up @@ -515,6 +535,8 @@ $ASK_USER_BLOCK
the daemon accepts \`respond\` immediately and runs the round in the background, so a killed or
timed-out call was only waiting for a read while the run kept working.

$WAIT_SECTION

$INBOX_SECTION

# Project memory
Expand Down
14 changes: 14 additions & 0 deletions bin/fm-classify-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -640,6 +640,20 @@ status_open_decisions() { # <status-file>
printf '%s' "$open"
}

# The subset of status_open_decisions the task raised about its own work: a
# reserved-namespace key is raised by a supervisor library about the task (a
# pending-reply escalation), so the task is not waiting on it. Automatic senders
# consult this set and leave a task alone while it is non-empty.
status_own_open_decisions() { # <status-file>
local line prefix
status_open_decisions "$1" | while IFS= read -r line || [ -n "$line" ]; do
for prefix in ${FM_CLASSIFY_RESERVED_KEY_PREFIXES:-$FM_CLASSIFY_RESERVED_KEY_PREFIXES_DEFAULT}; do
case "$line" in "$prefix"*) continue 2 ;; esac
done
printf '%s\n' "$line"
done
}

# 0 when <key> has a record in a folded "<key>\t<verb>\t<note>" open set.
_fm_open_set_has() { # <open-set> <key>
case "$1" in
Expand Down
24 changes: 17 additions & 7 deletions bin/fm-config-inherit-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -873,17 +873,23 @@ fm_config_reread_publish_stage() {
}

fm_config_reread_send_failure() {
local id=$1 instruction_path=$2 pending_path=$3 detail=$4
local id=$1 instruction_path=$2 pending_path=$3 detail=$4 rc=${5:-1}
if ! fm_config_reread_mark_pending "$instruction_path" "$pending_path"; then
detail="$detail; could not record retry marker"
fi
printf 'CONFIG_REREAD: secondmate %s: send failed: %s\n' "$id" "$detail"
# fm-send's exit status classifies the result; 4 is the mate waiting on its
# own open decision. Anything it printed ahead of that line is not a failure.
if [ "$rc" -eq 4 ]; then
printf 'CONFIG_REREAD: secondmate %s: %s\n' "$id" "$detail"
else
printf 'CONFIG_REREAD: secondmate %s: send failed: %s\n' "$id" "$detail"
fi
return 1
}

# fm_config_reread_send_pointer <id> <instruction-path>
fm_config_reread_send_pointer() {
local id=$1 instruction_path=$2 pending_path selector out rc send_bin message pending_pointer
local id=$1 instruction_path=$2 pending_path selector out rc detail send_bin message pending_pointer
pending_path="$instruction_path.pending"
if [ ! -f "$instruction_path" ] || [ -L "$instruction_path" ]; then
printf 'CONFIG_REREAD: secondmate %s: send failed: pending instruction file is missing\n' "$id"
Expand All @@ -909,14 +915,18 @@ fm_config_reread_send_pointer() {
FM_ROOT_OVERRIDE="${FM_ROOT_OVERRIDE:-}" \
FM_STATE_OVERRIDE="${FM_STATE_OVERRIDE:-}" \
FM_SEND_SETTLE="${FM_SEND_SETTLE:-0}" \
"$send_bin" "$selector" "$message" 2>&1) && rc=0 || rc=$?
"$send_bin" "$selector" --automatic "$message" 2>&1) && rc=0 || rc=$?
if [ "$rc" -eq 0 ]; then
rm -f "$pending_path"
return 0
fi
out=${out%%$'\n'*}
[ -n "$out" ] || out="fm-send exited $rc"
fm_config_reread_send_failure "$id" "$instruction_path" "$pending_path" "$out"
if [ "$rc" -eq 4 ]; then
detail=$(printf '%s\n' "$out" | grep -m1 '^deferred:') || detail=${out%%$'\n'*}
else
detail=${out%%$'\n'*}
fi
[ -n "$detail" ] || detail="fm-send exited $rc"
fm_config_reread_send_failure "$id" "$instruction_path" "$pending_path" "$detail" "$rc"
return 1
}

Expand Down
8 changes: 6 additions & 2 deletions bin/fm-config-push.sh
Original file line number Diff line number Diff line change
Expand Up @@ -152,10 +152,14 @@ while IFS='|' read -r id home _window meta; do
if printf '%s\n' "$remote_out" | grep -Eq '^(pushed|removed):'; then remote_nudge=1; fi
[ "$remote_pending" -eq 0 ] || remote_nudge=1
if [ "$remote_nudge" -eq 1 ]; then
if FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" \
"$SCRIPT_DIR/fm-send.sh" "fm-$id" "$FM_REMOTE_SECOND_MATE_NUDGE_MESSAGE" >/dev/null 2>&1; then
send_rc=0
FM_HOME="$FM_HOME" FM_ROOT_OVERRIDE="$FM_ROOT" FM_STATE_OVERRIDE="$STATE" \
"$SCRIPT_DIR/fm-send.sh" "fm-$id" --automatic "$FM_REMOTE_SECOND_MATE_NUDGE_MESSAGE" >/dev/null 2>&1 || send_rc=$?
if [ "$send_rc" -eq 0 ]; then
rm -f -- "$remote_marker"
echo " config-reread: sent"
elif [ "$send_rc" -eq 4 ]; then
echo " config-reread: deferred while the mate waits on its open decision; retry retained"
else
echo " config-reread: send failed; retry retained"
errors=1
Expand Down
3 changes: 3 additions & 0 deletions bin/fm-pending-reply-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -946,6 +946,9 @@ fm_pending_reply_send_recovery() { # <state-dir> <corr_id>
task_id=$(fm_pending_reply_get "$rec" task_id)
# A remote mate's report may exist and simply not have been mirrored yet.
fm_pending_reply_missing_report_is_evidence "$state" "$task_id" "$completed" || return 1
# A mate waiting on its own open decision or blocker is never poked: the
# recovery stays unattempted until firstmate's deliberate answer lands.
[ -z "$(status_own_open_decisions "$state/$task_id.status")" ] || return 1
parent_home=$(fm_pending_reply_get "$rec" parent_home)
msg=$(fm_pending_reply_recovery_message "$rec")
sender_pid=${BASHPID:-$$}
Expand Down
10 changes: 8 additions & 2 deletions bin/fm-secondmate-reconcile.sh
Original file line number Diff line number Diff line change
Expand Up @@ -384,7 +384,7 @@ cmd_process_requests() {
"$SCRIPT_DIR/fm-secondmate-reconcile.sh" notify --snapshot "$claimed" \
> "$output" 2>&1 || rc=$?
if [ "$rc" -eq 0 ] \
&& ! grep -Eq '^(skipped|failed|sent-unrecorded):' "$output" 2>/dev/null; then
&& ! grep -Eq '^(skipped|deferred|failed|sent-unrecorded):' "$output" 2>/dev/null; then
if rm -f -- "$claimed"; then
processed=$((processed + 1))
else
Expand Down Expand Up @@ -524,8 +524,14 @@ cmd_notify() {
send_rc=0
FM_TASK_INBOX_LOCK_WAIT_SECS=0 FM_SEND_EXPECTED_SPAWN_GEN="$sampled_spawn_gen" \
FM_SEND_EXPECTED_REMOTE_HOST="$expected_remote_host" \
"$SCRIPT_DIR/fm-send.sh" "$id" --fire-and-forget "$did" \
"$SCRIPT_DIR/fm-send.sh" "$id" --automatic --fire-and-forget "$did" \
"$(reconcile_text)" >/dev/null 2>&1 || send_rc=$?
# exit 4: the mate is waiting on its own open decision, so nothing was sent
# and the request stays for a later pass without starting the cooldown.
if [ "$send_rc" -eq 4 ]; then
printf 'deferred: %s %s\n' "$id" "$kind"
continue
fi
# exit 3 is "typed but unconfirmed": the mate may already hold the ask, so
# record the nudge rather than risk asking twice.
if [ "$send_rc" -ne 0 ] && [ "$send_rc" -ne 3 ]; then
Expand Down
Loading
Loading