Skip to content
This repository was archived by the owner on Aug 25, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,8 @@ Project worktrees start at detached HEAD on a clean default branch; ship briefs
After spawning, peek the pane to confirm the crewmate is processing the brief and handle any trust dialog with `harness-adapters`.
Add the task to `data/backlog.md` under In flight.

Herdr crew dispatch uses the dedicated `firstmate` session and never targets the captain-owned default/CAPTAIN workspace; the full endpoint-identity and teardown contract lives in [`docs/worker-isolation.md`](docs/worker-isolation.md).

### Supervise

Covered by section 8.
Expand Down
141 changes: 117 additions & 24 deletions bin/backends/herdr.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# per task inside that workspace.
# Target resolution stays parallel to the tmux adapter in both layouts.
#
# Target string shape: "<herdr-session>:<pane-id>", e.g. "default:w1:p2" (the
# Target string shape: "<herdr-session>:<pane-id>", e.g. "firstmate:ws1:p2" (the
# pane id itself contains a colon; the session is always the FIRST field, the
# remainder is the whole pane id - fm_backend_herdr_parse_target splits on the
# first colon only). This is the value stored in a herdr task's meta window=
Expand All @@ -24,7 +24,8 @@
#
# Authoritative task recovery uses labels and exact persisted endpoint ids.
# Live teardown also proves the task's current declared process identity before
# issuing a bound close for that exact pane.
# issuing a bound close, or the isolated-session legacy close fallback, for that
# exact pane.
#
# Requires: herdr (CLI + socket), jq (JSON parsing). Bootstrap detects these
# through fm_backend_required_tools only when herdr is the resolved backend;
Expand Down Expand Up @@ -67,6 +68,7 @@ FM_BACKEND_HERDR_MIN_PROTOCOL=14
# (14): the adapter's spawn/capture/send primitives work on 14, only the push
# subscriber needs 16.
FM_BACKEND_HERDR_MIN_EVENTS_PROTOCOL=16
FM_BACKEND_HERDR_DEDICATED_SESSION=firstmate
# Per-pane escalation dedupe marker prefix, under the state dir. One marker per
# window (keyed like the watcher's own .stale-<key>): set when a ->blocked edge
# is enqueued, cleared on any working edge, so exactly one wake fires per
Expand Down Expand Up @@ -240,15 +242,25 @@ fm_backend_herdr_bound_tab_close_capable() {
'workspace_id:string,tab_id:string,pane_id:string'
}

fm_backend_herdr_legacy_close_capable() {
local schema
schema=$(herdr api schema --json 2>/dev/null) || return 1
printf '%s' "$schema" | jq -e '
[.schemas.request.oneOf[]?.properties.method.const] as $methods
| ($methods | index("pane.close") != null)
and ($methods | index("tab.close") != null)
' >/dev/null 2>&1
}

# fm_backend_herdr_version_check: refuse loudly on a missing/incompatible
# herdr client or missing bound-mutation capabilities. Verified locally: v0.7.1,
# herdr client or missing mutation capabilities. Verified locally: v0.7.1,
# protocol 14 (herdr status --json's .client.protocol; client info is
# session-independent, unlike .server). Live task teardown requires
# pane.close_bound(expected_pid, expected_start_time); workspace/task-tab
# reconciliation requires tab.close_bound(workspace_id, tab_id, pane_id).
# session-independent, unlike .server). The isolated firstmate session may
# fall back to identity-checked pane.close/tab.close on hosts that lack the
# newer bound methods; every other session still requires both bound methods.
fm_backend_herdr_version_check() {
fm_backend_herdr_tool_check || return 1
local status protocol version
local status protocol version session pane_bound tab_bound
status=$(herdr status --json 2>/dev/null) || { echo "error: 'herdr status --json' failed; is herdr installed correctly?" >&2; return 1; }
protocol=$(printf '%s' "$status" | jq -r '.client.protocol // empty' 2>/dev/null)
version=$(printf '%s' "$status" | jq -r '.client.version // empty' 2>/dev/null)
Expand All @@ -262,14 +274,33 @@ fm_backend_herdr_version_check() {
echo "error: herdr protocol $protocol (version ${version:-unknown}) is older than the verified minimum $FM_BACKEND_HERDR_MIN_PROTOCOL; update herdr (herdr update) before using backend=herdr" >&2
return 1
fi
if ! fm_backend_herdr_bound_close_capable; then
session=$(fm_backend_herdr_spawn_session) || return 1
if fm_backend_herdr_bound_close_capable; then
pane_bound=1
else
pane_bound=0
fi
if fm_backend_herdr_bound_tab_close_capable; then
tab_bound=1
else
tab_bound=0
fi
if [ "$pane_bound" != 1 ] \
&& [ "$session" != "$FM_BACKEND_HERDR_DEDICATED_SESSION" ]; then
echo "error: herdr provider lacks atomic pane.close_bound(expected_pid); refusing a backend that cannot safely finish live task teardown" >&2
return 1
fi
if ! fm_backend_herdr_bound_tab_close_capable; then
if [ "$tab_bound" != 1 ] \
&& [ "$session" != "$FM_BACKEND_HERDR_DEDICATED_SESSION" ]; then
echo "error: herdr provider lacks atomic tab.close_bound(workspace_id,tab_id,pane_id); refusing a backend that cannot safely reconcile task-tab creation" >&2
return 1
fi
if [ "$session" = "$FM_BACKEND_HERDR_DEDICATED_SESSION" ] \
&& { [ "$pane_bound" != 1 ] || [ "$tab_bound" != 1 ]; } \
&& ! fm_backend_herdr_legacy_close_capable; then
echo "error: isolated Herdr session lacks pane.close/tab.close; refusing unbound cleanup" >&2
return 1
fi
return 0
}

Expand All @@ -280,14 +311,17 @@ fm_backend_herdr_server_available() { # <session>
[ "$running" = true ]
}

# fm_backend_herdr_session: resolve which named herdr session this normal
# spawn/op uses. HERDR_SESSION mirrors tmux's $TMUX ambient-selection for
# adapter workspace/tab/pane operations: an operator (or firstmate's own
# isolated test harness) sets it explicitly; absent means herdr's own
# "default" session. Do not use HERDR_SESSION alone for destructive test
# cleanup; tests/herdr-test-safety.sh documents and guards that path.
fm_backend_herdr_spawn_session() {
printf '%s' "$FM_BACKEND_HERDR_DEDICATED_SESSION"
}

fm_backend_herdr_session() {
printf '%s' "${HERDR_SESSION:-default}"
local session=${HERDR_SESSION:-$FM_BACKEND_HERDR_DEDICATED_SESSION}
if [ "$session" = default ]; then
echo "error: normal Herdr crew dispatch cannot target the captain-owned default session; use '$FM_BACKEND_HERDR_DEDICATED_SESSION'" >&2
return 1
fi
printf '%s' "$session"
}

fm_backend_herdr_provider_close_bound() {
Expand All @@ -302,11 +336,22 @@ fm_backend_herdr_provider_close_bound() {

fm_backend_herdr_provider_close_tab_bound() {
local session=${1:-} workspace_id=${2:-} tab_id=${3:-} pane_id=${4:-}
local expected_pid=${5:-} expected_start=${6:-}
local socket helper=${FM_BACKEND_HERDR_BOUND_CLOSE_HELPER:-$FM_BACKEND_HERDR_ROOT/bin/backends/herdr-pane-close-bound.py}
[ -n "$session" ] && [ -n "$workspace_id" ] && [ -n "$tab_id" ] && [ -n "$pane_id" ] || return 1
socket=$(fm_backend_herdr_socket_path "$session") || return 1
[ -x "$helper" ] || return 1
"$helper" "$socket" --tab "$workspace_id" "$tab_id" "$pane_id" >/dev/null 2>&1
if fm_backend_herdr_bound_tab_close_capable; then
"$helper" "$socket" --tab "$workspace_id" "$tab_id" "$pane_id" >/dev/null 2>&1
return $?
fi
[ "$session" = "$FM_BACKEND_HERDR_DEDICATED_SESSION" ] || return 1
[ -n "$expected_pid" ] && [ -n "$expected_start" ] || return 1
fm_backend_herdr_pane_identity_matches \
"$session" "$pane_id" "$expected_pid" "$expected_start" || return 1
fm_backend_herdr_tab_pane_identity_matches \
"$session" "$workspace_id" "$tab_id" "$pane_id" || return 1
fm_backend_herdr_cli "$session" tab close "$tab_id" >/dev/null 2>&1
}

fm_backend_herdr_server_ensure() { # <session>
Expand Down Expand Up @@ -604,7 +649,7 @@ fm_backend_herdr_workspace_ensure() { # <session> <cwd>
fm_backend_herdr_container_ensure() { # <cwd-for-a-fresh-workspace>
local cwd=${1:-$PWD} session label
fm_backend_herdr_version_check || return 1
session=$(fm_backend_herdr_session)
session=$(fm_backend_herdr_spawn_session)
fm_backend_herdr_server_ensure "$session" || return 1
fm_backend_herdr_workspace_ensure "$session" "$cwd" >/dev/null || { label=$(fm_backend_herdr_workspace_label); echo "error: failed to ensure herdr workspace '$label' in session '$session'" >&2; return 1; }
if [ -z "$FM_BACKEND_HERDR_WS_ID" ]; then
Expand Down Expand Up @@ -1839,10 +1884,58 @@ fm_backend_herdr_create_task() { # <container> <label> <cwd> [seeded-default-ta
printf '%s %s' "$tab_id" "$pane_id"
}

fm_backend_herdr_pane_identity_matches() { # <session> <pane> <pid> <start>
local session=$1 pane_id=$2 expected_pid=$3 expected_start=$4 info current_start
[ -n "$expected_pid" ] && [ -n "$expected_start" ] || return 1
info=$(fm_backend_herdr_cli "$session" pane process-info --pane "$pane_id" 2>/dev/null) || return 1
printf '%s' "$info" | jq -e --arg pane "$pane_id" --arg pid "$expected_pid" '
.result.process_info.pane_id == $pane
and ([.result.process_info.foreground_processes[]? |
select((.pid | tostring) == $pid)] | length) == 1
' >/dev/null 2>&1 || return 1
current_start=$(fm_backend_herdr_proc_start_time "$expected_pid") || return 1
[ "$current_start" = "$expected_start" ]
}

fm_backend_herdr_proc_start_time() {
local pid=$1 stat rest start
case "$pid" in
''|*[!0-9]*) return 1 ;;
esac
if [ -r "/proc/$pid/stat" ]; then
stat=$(cat "/proc/$pid/stat" 2>/dev/null) || return 1
rest=$(printf '%s\n' "$stat" | sed -E 's/^[0-9]+ \(.*\) //')
start=$(printf '%s\n' "$rest" | awk '{print $20}')
else
start=$(ps -o lstart= -p "$pid" 2>/dev/null | sed 's/^[[:space:]]*//')
fi
[ -n "$start" ] || return 1
printf '%s' "$start"
}

fm_backend_herdr_provider_close_safe() { # <session> <pane> <pid> <start>
local session=$1 pane_id=$2 expected_pid=$3 expected_start=$4
if fm_backend_herdr_bound_close_capable; then
fm_backend_herdr_provider_close_bound \
"$session" "$pane_id" "$expected_pid" "$expected_start"
return $?
fi
# Legacy pane.close is safe only in the isolated worker session. In
# particular, never turn missing pane.close_bound into a close of default's
# CAPTAIN workspace.
if [ "$session" != "$FM_BACKEND_HERDR_DEDICATED_SESSION" ]; then
echo "error: Herdr unbound pane.close is forbidden outside dedicated session '$FM_BACKEND_HERDR_DEDICATED_SESSION'" >&2
return 1
fi
fm_backend_herdr_pane_identity_matches \
"$session" "$pane_id" "$expected_pid" "$expected_start" || return 1
fm_backend_herdr_cli "$session" pane close "$pane_id" >/dev/null 2>&1
}

# fm_backend_herdr_kill: close the task's exact pane only with its expected
# process pid/start-time identity, then prove it disappeared. A bound close
# prevents pane/PID drift. Verified: closing a tab's only pane closes the tab
# too, so a separate tab close is unnecessary for normal teardown.
# process pid/start-time identity, then prove it disappeared. A bound close is
# preferred; the isolated firstmate session may use legacy pane.close only
# after independently proving that identity on the recorded pane.
fm_backend_herdr_kill() { # <target> [pid] [start-time]
local target=$1 expected_pid=${2:-} expected_start=${3:-} state
fm_backend_herdr_target_ready "$target" || {
Expand All @@ -1857,10 +1950,10 @@ fm_backend_herdr_kill() { # <target> [pid] [start-time]
echo "error: Herdr teardown target '$target' lacks bound process identity" >&2
return 1
fi
fm_backend_herdr_provider_close_bound \
fm_backend_herdr_provider_close_safe \
"$FM_BACKEND_HERDR_SESSION" "$FM_BACKEND_HERDR_PANE" "$expected_pid" \
"$expected_start" || {
echo "error: Herdr bound pane.close_bound failed for target '$target'" >&2
echo "error: Herdr safe pane close failed identity verification or mutation for target '$target'" >&2
return 1
}
;;
Expand All @@ -1870,7 +1963,7 @@ fm_backend_herdr_kill() { # <target> [pid] [start-time]
;;
esac
if [ "$(fm_backend_herdr_pane_agent_state "$FM_BACKEND_HERDR_SESSION" "$FM_BACKEND_HERDR_PANE")" != dead ]; then
echo "error: Herdr bound pane.close_bound did not close target '$target'" >&2
echo "error: Herdr pane close did not close target '$target'" >&2
return 1
fi
}
2 changes: 1 addition & 1 deletion bin/fm-spawn.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1749,7 +1749,7 @@ case "$BACKEND" in
if [ "$KIND" = secondmate ]; then
HERDR_LABEL_HOME=$PROJ_ABS
fi
HERDR_SES=$(fm_backend_herdr_session)
HERDR_SES=$(fm_backend_herdr_spawn_session)
HERDR_LABEL_LOCK="$STATE/.herdr-label.lock"
if ! fm_lock_acquire_wait "$HERDR_LABEL_LOCK"; then
echo "error: timed out waiting for another Herdr spawn to finish reserving its display label" >&2
Expand Down
11 changes: 6 additions & 5 deletions docs/worker-isolation.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Per-provider process id availability:
| Provider | Per-pane process id | Consequence |
|---|---|---|
| tmux | `#{pane_pid}`, a real shell pid | Process cwd is readable without the marker, but task isolation still requires complete worker identity. |
| herdr | none; the pane API exposes `foreground_cwd` only | Authoritative cwd reading requires the declaration marker; task teardown uses a separately proven process identity with bound close. |
| herdr | no shell PID through `fm_agent_backend_shell_pid`; the adapter's `pane.process-info` exposes a foreground process id for teardown | Authoritative cwd reading requires the declaration marker; teardown separately proves the foreground PID and `/proc` start time before closing. |
| zellij | none exposed at all | Same. |
| cmux | none on the control socket | Same. |
| orca | none on the terminal endpoint | Same. |
Expand All @@ -80,11 +80,12 @@ A provider with no process id is not a failure of the library.
It means a task that also lacks the declaration marker has only a hint, which is reported as `unknown` rather than promoted to evidence.

Herdr has a separate identity contract for task presentation endpoints.
`bin/fm-spawn.sh` records the exact Herdr session, workspace, tab, and pane ids in task metadata.
Select the backend with `config/backend` or `FM_BACKEND`; putting `herdr` in `config/backend` is sufficient and does not require an `FM_BACKEND=tmux` override.
New `backend=herdr` crew spawns default to the dedicated `firstmate` session, never Herdr's default captain session, and `bin/fm-spawn.sh` records the exact session, workspace, tab, and pane ids in task metadata.
Before live teardown, `bin/fm-teardown.sh` revalidates those ids and the task label, then proves the current declaration-backed process pid and start time.
`bin/backends/herdr.sh` keeps its fixed spawn-time command path atomic with Herdr's `pane.run` primitive and closes a live pane only through `pane.close_bound` with that expected process identity.
`bin/backends/herdr.sh` keeps its fixed spawn-time command path atomic with Herdr's `pane.run` primitive. When `pane.close_bound` exists it is preferred; on hosts without it, only the isolated `firstmate` session may use legacy `pane.close`, and only after `pane.process-info`'s PID and the live `/proc` start time match the recorded identity. Captain-owned `default`/`CAPTAIN` targets refuse before mutation.
The production launch path still sends its launch text and Enter separately; production launch atomicity is outside this focused proof.
The adapter refuses at preflight when the required bound capabilities are absent; failed workspace or task-tab creation is reconciled only from exact provider identities, and unresolved cleanup is surfaced as uncertainty rather than retried blindly.
If any required identity is missing or mismatched, teardown refuses and leaves the recorded pane in place. The adapter refuses at preflight when a non-isolated session lacks the required bound capabilities; failed workspace or task-tab creation is reconciled only from exact provider identities, using the same isolated-session boundary, and cleanup is recorded as uncertain rather than closing an unproven endpoint.

A tmux target is resolved to its stable window id by exact enumeration before any pane is read.
`display-message` given a window name it cannot find silently answers for the *active client's* window instead, so a task whose window name was lost or auto-renamed would hand back firstmate's own pane, whose working directory is the primary checkout - a healthy worker reported as collapsed.
Expand Down Expand Up @@ -205,7 +206,7 @@ A sweep that cries wolf on a normal fleet is worse than no sweep, because the `b

`tests/fm-worker-isolation.test.sh` covers all four mechanisms: the declaration's exact bytes and its refusals, the launch declaration for every verified harness, each consuming refusal, the process-cwd method of record against a deliberately lying pane path, the provider matrix, the stable-window-id resolution and its refusal to answer from a lost window name, all three slot-conflict forms plus the clean-disposal case, teardown retiring a contested lease under `--force`, the contested-then-released and still-blocked stamp sequences, and the sweep outcomes including a healthy secondmate staying silent, unproven evidence blocking, stale endpoints staying quiet, and live foreign-owner processes still blocking.
`tests/fm-slot-occupant-proof.test.sh` owns focused pooled-slot endpoint proof: exact endpoint selection, PID reuse, foreign and undeclared occupants, closed-endpoint census uncertainty, cross-home metadata, and disposal only after a complete empty census.
`tests/fm-backend-herdr-presentation-e2e.test.sh` owns the focused Herdr presentation proof: read-only installed-host preflight, exact workspace/tab/pane identity, atomic adapter-command submission, bound process teardown, precise failure diagnostics, and provider-mutation reconciliation.
`tests/fm-backend-herdr-presentation-e2e.test.sh` owns the focused Herdr presentation proof: read-only installed-host preflight, dedicated-session routing, exact workspace/tab/pane identity, atomic adapter-command submission, PID/start-time teardown matching, captain-session refusal, precise failure diagnostics, and provider-mutation reconciliation.

## Maintaining this file

Expand Down
Loading