Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions .agents/skills/stuck-crewmate-recovery/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,9 @@ Do not sweep another home's endpoints or infer ownership from a matching window

Before relaunch, prove that no live agent still owns the recorded task and that the existing worktree remains available.
Preserve its uncommitted changes and commits, keep the same task identity, and resume or relaunch the recorded harness in that existing worktree with the same brief plus a concise progress note.
A HERDR endpoint that is not merely idle but destroyed - a pane or workspace removed in Herdr churn - is recovered by that same relaunch, which creates one fresh endpoint in the existing worktree and rebinds the task's record to it; nothing special is needed, and the worktree is untouched ([`docs/agent-control.md`](../../../docs/agent-control.md) "Reclaiming a task whose endpoint is gone").
That relaunch proves the endpoint is destroyed before it rebinds, so a Herdr server that was merely stopped is adopted back rather than duplicated.
On tmux there is no reclaim: a task record carries no socket identity for its endpoint, so a `missing` window cannot be told apart from one on a tmux server this seat cannot address, and both `exit` and `relaunch` refuse.
Do not work around either refusal by respawning - it means a live agent may still hold that worktree.
An endpoint that is not merely idle but destroyed - a Herdr pane or workspace removed in churn, a tmux window or server that no longer exists - is recovered by that same `bin/fm-control.sh <id> relaunch`, which creates one fresh endpoint in the existing worktree and rebinds the task's record to it; nothing special is needed, the worktree is untouched, and a parked validation run is left exactly as it was. The Herdr reclaim works on every platform, while the tmux one reads `/proc` and so is Linux-only - on macOS a tmux task whose endpoint was destroyed refuses instead, and the refusal says so ([`docs/agent-control.md`](../../../docs/agent-control.md) "Reclaiming a task whose endpoint is gone").
That relaunch proves the endpoint is gone before it rebinds, so a Herdr server that was merely stopped is adopted back and an agent still working in the worktree refuses rather than being duplicated.
Do not work around a refusal by respawning or by editing the task record - it means a live agent may still hold that worktree, and the refusal names what to do next.
That reclaim is the owning home's operation only, and a secondmate is the one exception: recover it through `bin/fm-spawn.sh <id> --secondmate` as above.
Do not use a fresh generic spawn while the recorded worktree is unaccounted for, because allocating another worktree can split one task across two copies.
If the worktree or ownership cannot be reconciled safely, leave all state intact and report the task failed or blocked with the conflicting evidence.
Expand Down
34 changes: 27 additions & 7 deletions bin/backends/tmux.sh
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,24 @@ fm_backend_tmux_container_ensure() {
}

# fm_backend_tmux_create_task: create the task's window in <proj-abs>,
# refusing an existing <window-name> in <session>. Mirrors fm-spawn.sh's
# duplicate-check-then-new-window sequence, including the exact error text
# (session:window, matching how fm-spawn.sh composed its own $T). Prints the
# created window's stable window id on stdout for the caller to target.
# refusing an existing <window-name> in <session>. Prints the created window's
# stable window id on stdout for the caller to target.
#
# The refusal never becomes a close-and-replace, unlike the Herdr adapter's
# husk path: a herdr tab is addressed inside the session the record names,
# while `fm-<id>` carries no home identity and every seat outside tmux
# resolves to the one session `firstmate`, so two homes holding equal task ids
# collide here. Closing the window this seat did not record would destroy
# another home's deliberately preserved endpoint - the very dead-end the
# reclaim exists to remove - so whose window it is stays the operator's call.
#
# Because that refusal is the only thing standing between a reclaim and a
# foreign window, absence of the name has to be READ, never inferred from an
# empty stream: fm_backend_tmux_window_inventory's three-way status separates
# "this session has no such window" from "the read failed", and only the first
# licenses the create. tmux permits duplicate window names, and the record
# persists the NAME form, so a second `fm-<id>` would leave every later
# name-targeted read and write resolving to whichever one tmux matches first.
#
# Robustness (fm-spawn tmux window handling under a non-default captain config):
# - Capture a STABLE window id with -P -F '#{window_id}', and let tmux append
Expand All @@ -95,9 +109,15 @@ fm_backend_tmux_container_ensure() {
# The returned window id lets callers target the window even if its name is ever
# lost, so worktree discovery cannot fall back to the active client's window.
fm_backend_tmux_create_task() { # <session> <window-name> <proj-abs> -> prints window id
local ses=$1 wname=$2 proj_abs=$3 wid
if tmux list-windows -t "$ses" -F '#{window_name}' | grep -qx "$wname"; then
echo "error: window $ses:$wname already exists" >&2
local ses=$1 wname=$2 proj_abs=$3 wid windows inventory_status
windows=$(fm_backend_tmux_window_inventory "$ses")
inventory_status=$?
if [ "$inventory_status" -eq 1 ]; then
echo "error: session $ses's windows could not be read, so whether $ses:$wname already exists is unknown; refusing rather than risk a second window under one name - retry once tmux answers again" >&2
return 1
fi
if [ "$inventory_status" -eq 0 ] && printf '%s\n' "$windows" | grep -qxF -- "$wname"; then
echo "error: window $ses:$wname already exists; read it with bin/fm-peek.sh $ses:$wname to see what holds it. No control-plane command closes a window this home's records do not name, so a leftover or another home's window has to be closed where it runs before this task can open its own" >&2
return 1
fi
wid=$(tmux new-window -dP -F '#{window_id}' -t "$ses:" -n "$wname" -c "$proj_abs") || return 1
Expand Down
72 changes: 72 additions & 0 deletions bin/fm-agent-process-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,75 @@ fm_agent_process_classify() { # <name> <argv0> <args> [pid] -> agent|shell|othe
printf 'other'
fi
}

# fm_agent_process_worktree_scan: whether a verified harness process is still
# working in <worktree> anywhere on this host, read from the kernel's own
# per-process working directory rather than from any terminal endpoint. It
# answers the one question a lost endpoint leaves open - would launching a
# replacement join an agent that is still running on this task's local copy? -
# without trusting a runtime backend that could not see the endpoint.
#
# Only processes whose /proc entry this user owns are read: a worker is always
# launched as that user, and another user's working directory is not readable
# anyway. A non-dumpable process is skipped along with them, because Linux
# gives its /proc entry to root - no verified harness is non-dumpable, so none
# is skipped that way. Each process is classified by fm_agent_process_classify,
# so this answers from the same name vocabulary as every liveness probe.
#
# Prints "<verdict>\t<detail>", always exactly one TAB:
# none - every readable process was read and none is an agent there
# agent - "<pid> <name>" of an agent process working in <worktree>
# unreadable - the reason the process table could not be read; the caller
# must treat this as "an agent may be there"
# The only reader is /proc, whose per-process `cwd` link is the kernel's own
# answer for one exact process. A host without /proc answers `unreadable`, so
# no tmux endpoint is proven absent there: every other way to ask - lsof's
# file-set records, a ps listing - reports a path it may have failed to
# resolve without saying so for the one process that matters, and reading that
# silence as "no agent" is what launches a duplicate onto a live worktree.
# A process that exits mid-scan, and a zombie that holds no working directory,
# are skipped.
fm_agent_process_worktree_scan() { # <worktree>
local wt=${1-} wt_real dir pid cwd name argv0 args stat rest
if [ -z "$wt" ] || ! wt_real=$(cd "$wt" 2>/dev/null && pwd -P); then
printf 'unreadable\tthe worktree %s cannot be resolved' "'$wt'"
return 0
fi
if [ ! -r /proc/self/cmdline ] || [ ! -L /proc/self/cwd ]; then
printf 'unreadable\tthe absence proof reads /proc and this host has none, so reclaiming a tmux task is unavailable here. The only route left is a seat that still addresses the recorded tmux server, where the window classifies directly and no absence proof is needed - and once that server is gone there is no such seat, and no route'
return 0
fi
for dir in /proc/[0-9]*; do
[ -O "$dir" ] || continue
pid=${dir#/proc/}
if cwd=$(readlink "$dir/cwd" 2>/dev/null); then
case "$cwd" in
"$wt_real"|"$wt_real"/*) ;;
*) continue ;;
esac
else
# A process that exited mid-scan, or a zombie, has no working directory
# left. The refusal below covers only the race that reaches here: the
# link vanishing from a /proc directory this user still owns, mid-exec
# or mid-exit.
[ -d "$dir" ] || continue
stat=$(cat "$dir/stat" 2>/dev/null) || continue
rest=${stat##*) }
case "${rest%% *}" in
Z|X) continue ;;
esac
cwd=
fi
name=$(cat "$dir/comm" 2>/dev/null) || continue
argv0=$(tr '\0' '\n' < "$dir/cmdline" 2>/dev/null | head -n 1) || argv0=
args=$(tr '\0' ' ' < "$dir/cmdline" 2>/dev/null) || args=
[ "$(fm_agent_process_classify "$name" "$argv0" "$args" "$pid")" = agent ] || continue
if [ -z "$cwd" ]; then
printf 'unreadable\tthe working directory of agent process %s (%s) could not be read' "$pid" "${name:-$argv0}"
return 0
fi
printf 'agent\t%s %s' "$pid" "${name:-$argv0}"
return 0
done
printf 'none\t'
}
85 changes: 63 additions & 22 deletions bin/fm-control-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# This file owns three capability tables plus their pure artifact-path tables,
# and ONE named exception to that purity - fm_control_endpoint_absence_verdict,
# the single owner of the per-backend endpoint-absence proof, which does run
# backend reads. Everything else has no side effects, runs no backend command,
# backend reads and a process-table scan. Everything else has no side effects, runs no backend command,
# and reads no state, so sourcing this file is still free and the tables can be
# read by a test as a pure contract:
#
Expand Down Expand Up @@ -302,8 +302,9 @@ fm_control_backend_state_verified() { # <backend>
}

# fm_control_endpoint_absence_verdict: the ONE owner of the per-backend proof
# that an endpoint reading `missing` is actually GONE rather than merely
# unreachable from this seat. Call it only for a `missing` raw state.
# that an endpoint reading `missing` holds no agent this home could duplicate,
# so the task may be reclaimed rather than stranded. Call it only for a
# `missing` raw state.
#
# Prints "<verdict>\t<reason>" - always exactly one TAB, so a caller splits
# unambiguously with ${raw%%$'\t'*} and ${raw#*$'\t'}. The reason is empty
Expand All @@ -313,7 +314,17 @@ fm_control_backend_state_verified() { # <backend>
# assignment made here could never reach them.
#
# The verdicts:
# gone - absence is PROVEN. There is no endpoint and therefore no agent.
# gone - absence is PROVEN, in the terms the backend can prove, and
# there is nothing here to adopt. Herdr proves the ENDPOINT: it
# re-read the recorded pane through the recorded session's own
# socket and the pane is not there. tmux proves only the AGENT:
# a record's `session:window` carries no socket, so the same
# address can name a live window on another server, and what the
# scan establishes is that no agent process works in the recorded
# worktree. A tmux window may therefore still survive elsewhere -
# on a server this seat cannot address, or under a session name
# the record no longer matches, even on this seat's own server -
# and it holds no agent either way.
# dead - the endpoint is there after all and holds no agent.
# alive - the endpoint is there and an agent is running in it.
# unproven - neither could be established; the caller must refuse.
Expand All @@ -323,31 +334,41 @@ fm_control_backend_state_verified() { # <backend>
# endpoint can still hold a live agent on the task's worktree, so every caller
# that would act on absence - `exit` claiming the agent stopped, `relaunch`
# re-creating the endpoint - must come through here rather than trusting the
# raw verdict.
#
# Whether absence is provable AT ALL is a property of the backend, not of the
# reading:
# herdr CAN prove it. Every read goes through fm_backend_herdr_cli, which
# passes `--session <session>`, so the recheck starts and reads the session
# the RECORD names, through that session's own socket. The answer is about
# the task's endpoint and nothing else.
# tmux CANNOT. `list-windows -a` describes only the server the CURRENT
# process addresses (its TMUX_TMPDIR/socket), and a task's record does not
# carry the endpoint's socket identity - so a different but running server
# would answer "not anywhere" about a window it was never able to see.
# There is no read available here that closes that gap, so tmux always
# returns `unproven` and both verbs refuse. tmux is left exactly as
# deadlocked as it was before this change - no worse - but deliberately.
# raw verdict. How that is proven is a property of the backend:
# herdr proves the ENDPOINT absent. Every read goes through
# fm_backend_herdr_cli, which passes `--session <session>`, so the recheck
# starts and reads the session the RECORD names, through that session's
# own socket. The answer is about the task's endpoint and nothing else.
# tmux proves the AGENT absent. `list-windows` describes only the server the
# CURRENT process addresses, and a task record carries no socket identity,
# so no tmux read can tell a destroyed window from one on a server this
# seat cannot address. What a duplicate launch would actually collide with
# is an agent process working in the recorded worktree, and the kernel
# knows every process's working directory regardless of which terminal
# server holds it (fm_agent_process_worktree_scan). No agent process there
# is the proof; one found there, or a process table that cannot be read,
# refuses. A window that outlived its agent on a server this
# seat cannot address is left alone - it holds no agent, and the reclaim
# opens its replacement on the server this seat does address.
#
# Both control-plane callers share this one implementation so the proof cannot
# drift into two answers for the same endpoint.
fm_control_endpoint_absence_verdict() { # <backend> <target>
local backend=${1-} target=${2-}
fm_control_endpoint_absence_verdict() { # <backend> <target> <worktree>
local backend=${1-} target=${2-} worktree=${3-} scan
fm_backend_source "$backend" \
|| { printf 'unproven\tbackend %s could not be loaded to prove anything about that endpoint' "'$backend'"; return 0; }
case "$backend" in
tmux)
printf 'unproven\ttmux absence cannot be proven from a task record: the record does not carry the endpoint'"'"'s socket identity, and a server-wide window inventory only describes the tmux server this process addresses, so a window absent from it may still be alive on another'
scan=$(fm_agent_process_worktree_scan "$worktree")
case "${scan%%$'\t'*}" in
none) printf 'gone\t' ;;
agent)
printf 'unproven\tagent process %s is still working in the recorded worktree %s, so its window may be on a tmux server this seat cannot address; stop that process where it runs, or address its tmux server from this seat, then retry' "${scan#*$'\t'}" "$worktree"
;;
*)
printf 'unproven\tno tmux read can prove that window absent, and the process scan that would prove no agent is still working in %s failed: %s' "$worktree" "${scan#*$'\t'}"
;;
esac
;;
herdr)
# Start the RECORDED session's server (only the server - nothing is
Expand All @@ -366,6 +387,26 @@ fm_control_endpoint_absence_verdict() { # <backend> <target>
esac
}

# fm_control_unclassified_next_step: the one sentence every lifecycle refusal
# of an endpoint that is not positively classified ends with, so `exit`,
# `relaunch`, and the launch owner never point an operator at each other for a
# state none of them acts on. `alive`, `dead`, and a proven-gone `missing` are
# all actionable, and <verb> is the one the caller was actually asked to
# perform: a refused `exit` names `exit`, because sending a stop request off to
# the verb that REPLACES the agent is the same cross-pointing this sentence
# exists to end.
fm_control_unclassified_next_step() { # <task-id> <state> <verb-with-its-flags>
local id=${1-} state=${2-} verb=${3-}
case "$state" in
ambiguous)
printf 'no lifecycle command acts on it until it reads alive, dead, or gone: inspect it with bin/fm-peek.sh %s - a foreground process that is not a verified agent reads ambiguous - and once it is back at a shell or its agent, retry bin/fm-control.sh %s %s' "$id" "$id" "$verb"
;;
*)
printf 'no lifecycle command acts on it until it reads alive, dead, or gone: the read itself failed, so retry bin/fm-control.sh %s %s once bin/fm-peek.sh %s can read the endpoint again' "$id" "$verb" "$id"
;;
esac
}

# The per-task wiring artifacts a harness leaves behind, so a relaunch that
# changes harness (or re-arms the same one with a fresh busy generation) can
# clear the previous incarnation's wiring instead of leaving a stale hook
Expand Down
Loading