From c76eed3f8d0c6818e1ad2520e94fe84620339e4b Mon Sep 17 00:00:00 2001 From: Luis Gonzalez Date: Wed, 30 Sep 2026 14:34:31 -0600 Subject: [PATCH 1/3] fix(bin): keep a seeded secondmate's reused pool slot from being returned A "-" home seed durably leases a Treehouse pool slot but never published Firstmate's slot-owner claim, so completed tasks that used the slot before still name what is now the secondmate's home. Once only one stale record remained and its old claim read as its own, or the claim was absent, its teardown returned the seeded home to the pool. The reassigned-slot shortcut also let any different claim skip the duplicate-record scan on such a home, even when its ownership could not be proved or the old endpoint was live. - fm-wake-lib: read a slot's durable lease from the pool state, recognize persistent-home evidence, positively prove a committed local secondmate home (pool identity, lease holder, identity marker, local parent binding, exact registry route), and transfer a claim atomically while keeping the replaced one at .fm-slot-owner.prior. - fm-home-seed: publish the claim under the Treehouse project lock when a seed leases its home, drop only that claim on rollback, and add claim-slot to re-publish it for an already-seeded home after re-proving ownership. - fm-teardown: never return, reset, or reap a slot showing a persistent home while the claim still makes the ordinary task its owner, even with --force; once the claim names the proved owner, a completed scout whose exact endpoint is dead or missing may take the existing no-slot-touch cleanup while other stale records or the owner's own record name the home, and any other secondmate record still refuses. Forced parent teardown refuses a child slot that is a persistent home. - Tests cover the diagnostic cases, proof mismatches, live and unreadable endpoints, completion gates, interrupted reconciliation, seed claim and rollback, and real Herdr endpoint classification in an isolated lab. --- .../skills/secondmate-provisioning/SKILL.md | 3 + bin/fm-home-seed.sh | 138 +++++- bin/fm-teardown.sh | 125 ++++- bin/fm-test-isolation-proof.sh | 1 + bin/fm-test-run.sh | 1 + bin/fm-wake-lib.sh | 174 ++++++- docs/architecture.md | 4 +- docs/configuration.md | 1 + docs/herdr-backend.md | 1 + docs/verification/runtime-backends.md | 19 + tests/fm-secondmate-safety.test.sh | 173 +++++++ tests/fm-teardown-endpoint-safety.test.sh | 441 ++++++++++++++++++ ...teardown-persistent-slot-herdr-e2e.test.sh | 264 +++++++++++ tests/secondmate-helpers.sh | 12 +- 14 files changed, 1345 insertions(+), 12 deletions(-) create mode 100755 tests/fm-teardown-persistent-slot-herdr-e2e.test.sh diff --git a/.agents/skills/secondmate-provisioning/SKILL.md b/.agents/skills/secondmate-provisioning/SKILL.md index f105b3253d5..642d7ae2474 100644 --- a/.agents/skills/secondmate-provisioning/SKILL.md +++ b/.agents/skills/secondmate-provisioning/SKILL.md @@ -159,6 +159,9 @@ Run `bin/fm-home-seed.sh validate` when checking registry integrity; its header Seeding is transactional. If validation, cloning, no-mistakes initialization, or registry update fails, generated briefs, new homes, new project clones, and registry edits are rolled back. +A leased home also carries its Treehouse slot's owner claim. +When teardown of an older task whose record names a seeded home refuses while that claim does not yet name the secondmate, run `bin/fm-home-seed.sh claim-slot ` from the registering home, then re-run the teardown; its header owns the proof, and the older records, claim, and home are never deleted, hidden, or hand-edited to get past the refusal. + Secondmate project lists may include `no-mistakes` and `direct-PR` projects only. `local-only` projects stay with the main firstmate. For `no-mistakes` projects, seeding initializes only projects newly cloned into a secondmate home and refuses to mutate a preexisting clone that is not already initialized. diff --git a/bin/fm-home-seed.sh b/bin/fm-home-seed.sh index 3eb2286d969..83a4b052e89 100755 --- a/bin/fm-home-seed.sh +++ b/bin/fm-home-seed.sh @@ -22,6 +22,13 @@ # generated briefs, new homes, new project clones, and registry edits are # rolled back. Treehouse-acquired homes are returned only when the rollback # target is safe; a failed return warns because the lease may still be held. +# A "-" seed holds the firstmate repo's Treehouse project lock +# (bin/fm-wake-lib.sh's fm_treehouse_project_lock_path) from before the +# lease until the leased home, once validated and proved leased to in +# the pool state, carries the slot-owner claim naming and this home; +# a contended lock refuses before leasing. Rollback re-takes that lock +# (waiting up to 30 seconds) to return the home and drop that claim, and +# warns and leaves the lease held when it cannot. # Set FM_SECONDMATE_CHARTER='' to seed from inline charter text # when no filled charter brief exists. Set FM_SECONDMATE_SCOPE='' # to override the registry routing scope. Otherwise the registry summary @@ -30,6 +37,18 @@ # Refuse records that operational consumers cannot parse, unavailable or # unsafe registry files when present, non-absolute or unresolvable homes, # duplicate ids or homes, and nested or overlapping homes. +# fm-home-seed.sh claim-slot +# Publish, for a local secondmate this home already registers, the +# slot-owner claim a "-" seed now writes, so a home seeded before seeding +# did is reconciled rather than left naming whichever task used the slot +# before it. Under this home's registry lock and then the firstmate repo's +# Treehouse project lock, it refuses without writing unless +# fm_treehouse_secondmate_slot_proof proves the registered home is a pool +# slot of this code root leased to , marked , and bound to this +# home as its local parent, and unless the current claim is readable. +# The replaced claim is kept at .prior; a claim already naming +# and this home is reported unchanged, so a retry after an interruption +# converges. It never returns, resets, or reseeds the slot. set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -53,6 +72,7 @@ SUB_HOME_PARENT_MARKER=".fm-secondmate-parent" usage() { echo "usage: fm-home-seed.sh {...|--no-projects}" >&2 echo " fm-home-seed.sh validate" >&2 + echo " fm-home-seed.sh claim-slot " >&2 } validate_registry_home_text() { @@ -519,6 +539,9 @@ SEED_ROLLBACK_ACTIVE=0 SEED_COMMITTED=0 SEED_REGISTRY_LOCK= SEED_REGISTRY_LOCK_HELD=0 +SEED_TREEHOUSE_LOCK= +SEED_TREEHOUSE_LOCK_HELD=0 +SEED_ID= seed_registry_lock_release() { if [ "$SEED_REGISTRY_LOCK_HELD" -eq 1 ]; then @@ -527,10 +550,48 @@ seed_registry_lock_release() { fi } +seed_treehouse_lock_acquire() { + SEED_TREEHOUSE_LOCK=$(fm_treehouse_project_lock_path "$FM_ROOT") || { + echo "error: cannot resolve the Treehouse project lock for $FM_ROOT; nothing was changed" >&2 + return 1 + } + fm_lock_try_acquire "$SEED_TREEHOUSE_LOCK" || { + echo "error: another Treehouse slot allocation or return is in progress for $FM_ROOT; nothing was changed - retry" >&2 + return 1 + } + SEED_TREEHOUSE_LOCK_HELD=1 +} + +seed_treehouse_lock_release() { + if [ "$SEED_TREEHOUSE_LOCK_HELD" -eq 1 ]; then + fm_lock_release "$SEED_TREEHOUSE_LOCK" + SEED_TREEHOUSE_LOCK_HELD=0 + fi +} + seed_exit_cleanup() { seed_rollback + seed_treehouse_lock_release seed_registry_lock_release } + +# A leased home that is a pool slot of this code root carries the slot-owner +# claim naming this secondmate and home (bin/fm-wake-lib.sh owns the claim), +# published under the project lock only once the pool state records the lease +# this seed just took. A leased home outside such a pool has no claim to carry. +seed_claim_leased_slot() { # + local id=$1 home=$2 holder + fm_treehouse_pool_slot "$FM_ROOT" "$home" || return 0 + holder=$(fm_treehouse_slot_lease_holder "$home") || holder= + if [ "$holder" != "$id" ]; then + echo "error: treehouse leased $home, but its pool state does not record a durable lease held by $id; refusing to seed a home whose ownership cannot be proved" >&2 + return 1 + fi + fm_treehouse_slot_owner_transfer "$home" "$id" "$(resolved_path "$FM_HOME")" || { + echo "error: could not publish the slot-owner claim for $id beside $home; inspect that slot's .fm-slot-owner" >&2 + return 1 + } +} SEED_HOME= SEED_HOME_ACQUIRED=0 SEED_HOME_CREATED=0 @@ -591,16 +652,30 @@ seed_rollback_target() { } seed_return_treehouse_home() { - local home=$1 abs_home + local home=$1 abs_home marker='' claimed=0 abs_home=$(seed_rollback_target "$home" "treehouse-acquired home") || return 0 if ! command -v treehouse >/dev/null 2>&1; then echo "warning: failed to return treehouse-acquired home $abs_home during seed rollback; treehouse command not found" >&2 return 0 fi + if [ "$SEED_TREEHOUSE_LOCK_HELD" -ne 1 ]; then + if [ -z "$SEED_TREEHOUSE_LOCK" ] || ! fm_lock_acquire_wait_max "$SEED_TREEHOUSE_LOCK" 30; then + echo "warning: failed to return treehouse-acquired home $abs_home during seed rollback; the Treehouse project lock stayed held elsewhere, so the lease is still held" >&2 + return 0 + fi + SEED_TREEHOUSE_LOCK_HELD=1 + fi + # Read before the return, which may remove the checkout the path names; the + # claim beside it is this seed's only while it names this secondmate. + fm_treehouse_slot_owner_state "$abs_home" "$SEED_ID" + if [ "$FM_TREEHOUSE_SLOT_OWNER" = mine ]; then + marker=$(fm_treehouse_slot_owner_marker "$abs_home") && claimed=1 + fi ( cd "$FM_ROOT" && treehouse return --force "$abs_home" >/dev/null ) || { echo "warning: failed to return treehouse-acquired home $abs_home during seed rollback; lease may still be held" >&2 return 0 } + [ "$claimed" -eq 0 ] || rm -f -- "$marker" 2>/dev/null || true } seed_remove_created_home() { @@ -875,7 +950,9 @@ seed_home() { cp "$REG" "$SEED_BACKUP_DIR/parent-secondmates.md" fi + SEED_ID=$id if [ "$requested_home" = "-" ]; then + seed_treehouse_lock_acquire || return 1 SEED_HOME_ACQUIRED=1 home=$(acquire_treehouse_home "$id") SEED_HOME="$home" @@ -894,6 +971,10 @@ seed_home() { validate_operational_dirs "$home" || return 1 validate_seed_leaf_files "$home" || return 1 validate_existing_parent_binding "$home" || return 1 + if [ "$SEED_TREEHOUSE_LOCK_HELD" -eq 1 ]; then + seed_claim_leased_slot "$id" "$home" || return 1 + seed_treehouse_lock_release + fi if [ "$no_projects" -eq 1 ]; then refuse_populated_projectless_home "$home" || return 1 if [ -f "$SEED_PARENT_BRIEF" ]; then @@ -987,11 +1068,66 @@ seed_home() { printf 'home=%s\n' "$home" } +claim_slot_exit_cleanup() { + seed_treehouse_lock_release + seed_registry_lock_release +} + +# The header's claim-slot contract: re-publish a registered local secondmate's +# slot-owner claim once its persistent ownership is proved again. +claim_slot() { # + local id=$1 home parent marker + case "$id" in + ''|*[!A-Za-z0-9._-]*) echo "error: invalid secondmate id: $id" >&2; return 1 ;; + esac + [ -d "$STATE" ] || { echo "error: this home has no state directory: $STATE" >&2; return 1; } + SEED_REGISTRY_LOCK=$(secondmate_registry_lock_path "$STATE") + fm_lock_acquire_wait "$SEED_REGISTRY_LOCK" || return 1 + SEED_REGISTRY_LOCK_HELD=1 + trap claim_slot_exit_cleanup EXIT + validate_registry + if ! secondmate_registry_validate_bindings "$REG" resolved_path "$id"; then + echo "REFUSED: $SECONDMATE_REGISTRY_ERROR; nothing was changed" >&2 + return 1 + fi + if [ "$SECONDMATE_REGISTRY_MATCH_REMOTE" -ne 0 ]; then + echo "REFUSED: secondmate $id is a remote route; its home has no local Treehouse slot to claim; nothing was changed" >&2 + return 1 + fi + home=$SECONDMATE_REGISTRY_MATCH_HOME + if ! fm_treehouse_pool_slot "$FM_ROOT" "$home"; then + echo "REFUSED: secondmate $id's registered home $home is not a Treehouse pool slot of $FM_ROOT, so it has no slot-owner claim to reconcile; nothing was changed" >&2 + return 1 + fi + seed_treehouse_lock_acquire || return 1 + if ! fm_treehouse_secondmate_slot_proof "$FM_ROOT" "$home"; then + echo "REFUSED: cannot prove $home is secondmate $id's persistent home: $FM_TREEHOUSE_SECONDMATE_SLOT_ERROR; nothing was changed" >&2 + return 1 + fi + parent=$(resolved_path "$FM_HOME") + if [ "$FM_TREEHOUSE_SECONDMATE_SLOT_ID" != "$id" ] \ + || [ "$FM_TREEHOUSE_SECONDMATE_SLOT_PARENT" != "$parent" ]; then + echo "REFUSED: $home is proved to be secondmate $FM_TREEHOUSE_SECONDMATE_SLOT_ID's home under parent $FM_TREEHOUSE_SECONDMATE_SLOT_PARENT, not $id's under this home ($parent); nothing was changed" >&2 + return 1 + fi + marker=$(fm_treehouse_slot_owner_marker "$home") + if ! fm_treehouse_slot_owner_transfer "$home" "$id" "$parent"; then + echo "REFUSED: the slot-owner claim at $marker cannot be read, or its prior claim cannot be kept at $marker.prior; the claim was not replaced - inspect both paths" >&2 + return 1 + fi + printf 'slot-owner claim %s: %s names task=%s home=%s\n' \ + "$FM_TREEHOUSE_SLOT_OWNER_TRANSFER" "$marker" "$id" "$parent" +} + case "${1:-}" in validate) [ $# -eq 1 ] || { usage; exit 1; } validate_registry ;; + claim-slot) + [ $# -eq 2 ] || { usage; exit 1; } + claim_slot "$2" + ;; -h|--help|'') usage exit 0 diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index 24ed4644c76..ba5aa719d60 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -99,9 +99,10 @@ # this home or any locally registered Firstmate home may name the same live path # in its worktree= or home=. One live path with two task records is the reuse # collision itself, whichever record is stale. The one exception is a slot whose -# owner claim (below) names another task: this teardown is then records-only and -# touches nothing under the slot, so the scan is skipped rather than stranding -# the stale record and, with it, the claimant's own teardown. +# owner claim (below) names another task and that is not a persistent secondmate +# home (below): this teardown is then records-only and touches nothing under the +# slot, so the scan is skipped rather than stranding the stale record and, with +# it, the claimant's own teardown. # That scan alone cannot prove THIS record is the current owner, because the task # that took the slot next may leave no record it can reach - its own worker may # have exited and its record been cleaned up, or it may live in a home this @@ -126,6 +127,22 @@ # absent claim - a slot taken before claims existed, or already returned - keeps # exactly the record-scan protection it had before, because refusing it would # strand every task in flight across that change on no evidence at all. +# A persistent secondmate home seeded into a reused slot is the one owner a +# claim alone cannot settle: a home seeded before bin/fm-home-seed.sh published +# its own claim still carries the claim of whichever task used the slot before, +# and completed scouts' stale records still name it. So a slot showing a +# durable Treehouse lease or a secondmate identity marker or parent binding is +# never returned, reset, or reaped by an ordinary task: while the claim +# would still make this task its owner, teardown refuses, even with --force, +# and names bin/fm-home-seed.sh claim-slot when the home's ownership is +# otherwise proved. Once the claim names the persistent owner that +# bin/fm-wake-lib.sh's fm_treehouse_secondmate_slot_proof proves, the +# reassigned-slot cleanup above applies; a completed scout (no --force) whose +# exact recorded endpoint the backend's recovery-grade classifier reads dead or +# missing may take it even while other ordinary records, or the owner's own +# record, name the home. Any other record naming it, a live or unreadable +# endpoint, and the report, decision-inventory, incarnation, backlog, and +# public-follow-up gates still refuse exactly as before. # Why Treehouse's own state cannot answer this for crewmate slots, and why the # claim file sits on top of it, is owned by bin/fm-wake-lib.sh's slot-owner # claim comment. @@ -2356,16 +2373,30 @@ collect_local_firstmate_states() { done } +# With [owner-id] [owner-home], the slot is that secondmate's proved persistent +# home and the caller touches nothing under it (teardown_persistent_slot_gate): +# another ordinary record naming the slot, or the owner's own record in its +# registering home, is then reported and left for its own cleanup rather than +# refused. Any other secondmate record naming the slot still refuses. require_exclusive_worktree_slot_record() { local record_meta=$1 record_id=$2 record_state=$3 worktree=$4 + local owner_id=${5:-} owner_home=${6:-} owner_state= local slot state_dir other other_id field other_path other_slot slot=$(canonical_existing_dir "$worktree") || return 0 + if [ -n "$owner_id" ]; then + owner_state=$(canonical_existing_dir "$owner_home/state") || owner_state= + fi # A slot whose owner claim names another task was reassigned, so this record's # teardown is records-only and touches nothing under it; another record naming # the slot is then no hazard, and refusing would strand this stale record and - # block the claimant's own teardown behind it. - fm_treehouse_slot_owner_state "$slot" "$record_id" - [ "$FM_TREEHOUSE_SLOT_OWNER" != other ] || return 0 + # block the claimant's own teardown behind it. A persistent secondmate home is + # the exception: its claim alone never excuses another record, so the scan + # below still runs, narrowed only by the owner teardown_persistent_slot_gate + # proved. + if ! fm_treehouse_slot_persistent_evidence "$slot"; then + fm_treehouse_slot_owner_state "$slot" "$record_id" + [ "$FM_TREEHOUSE_SLOT_OWNER" != other ] || return 0 + fi collect_local_firstmate_states "$record_state" || return 1 for state_dir in "${TREEHOUSE_OWNER_STATES[@]}"; do for other in "$state_dir"/*.meta; do @@ -2381,6 +2412,13 @@ require_exclusive_worktree_slot_record() { [ -n "$other_path" ] || continue other_slot=$(canonical_existing_dir "$other_path") || continue [ "$other_slot" = "$slot" ] || continue + if [ -n "$owner_id" ] \ + && { [ "$(fm_meta_get "$other" kind)" != secondmate ] \ + || { [ "$other_id" = "$owner_id" ] && [ -n "$owner_state" ] \ + && [ "$(canonical_existing_dir "$state_dir" || true)" = "$owner_state" ]; }; }; then + echo "warning: task $other_id's recorded $field also names $slot, which is secondmate $owner_id's proved persistent home; this cleanup touches nothing under it, so $other_id's record is left for its own cleanup." >&2 + continue 2 + fi echo "REFUSED: task $record_id's recorded worktree $slot is also task $other_id's recorded $field." >&2 echo "Returning that pool slot would kill $other_id's processes and reset its copy, so nothing was changed - not even with --force." >&2 echo "Reconcile whichever record is wrong (bin/fm-crew-state.sh $record_id; bin/fm-crew-state.sh $other_id), then re-run teardown." >&2 @@ -2393,7 +2431,65 @@ require_exclusive_worktree_slot_record() { require_exclusive_task_worktree_slot() { local slot slot=$(teardown_live_slot_path) || return 0 - require_exclusive_worktree_slot_record "$META" "$ID" "$STATE" "$slot" + if [ "$TEARDOWN_SLOT_PERSISTENT_NO_TOUCH" = 1 ]; then + require_exclusive_worktree_slot_record "$META" "$ID" "$STATE" "$slot" \ + "$TEARDOWN_SLOT_PERSISTENT_OWNER" "$TEARDOWN_SLOT_PERSISTENT_PARENT" + else + require_exclusive_worktree_slot_record "$META" "$ID" "$STATE" "$slot" + fi +} + +# Persistent secondmate homes (see the script header). A pool slot showing a +# durable lease or a secondmate marker is a home no ordinary task may return, +# reset, or reap, so a claim that would still make this task its owner (its own +# or none) refuses, even with --force. Only a positive +# fm_treehouse_secondmate_slot_proof whose persistent owner the claim already +# names records that owner; a completed scout (no --force) whose exact recorded +# endpoint is then conclusively dead or missing may take the existing +# no-slot-touch cleanup even while another record names the slot. Every other +# shape keeps the ordinary exclusivity and ownership checks unchanged. +TEARDOWN_SLOT_PERSISTENT_OWNER= +TEARDOWN_SLOT_PERSISTENT_PARENT= +TEARDOWN_SLOT_PERSISTENT_NO_TOUCH=0 +teardown_persistent_slot_gate() { + local slot claim_home endpoint + slot=$(teardown_live_slot_path) || return 0 + fm_treehouse_slot_persistent_evidence "$slot" || return 0 + fm_treehouse_slot_owner_state "$slot" "$ID" + if fm_treehouse_secondmate_slot_proof "$PROJ" "$slot"; then + claim_home=$(canonical_existing_dir "$FM_TREEHOUSE_SLOT_OWNER_HOME") || claim_home= + if [ "$FM_TREEHOUSE_SLOT_OWNER" = other ] \ + && [ "$FM_TREEHOUSE_SLOT_OWNER_ID" = "$FM_TREEHOUSE_SECONDMATE_SLOT_ID" ] \ + && [ "$claim_home" = "$FM_TREEHOUSE_SECONDMATE_SLOT_PARENT" ]; then + TEARDOWN_SLOT_PERSISTENT_OWNER=$FM_TREEHOUSE_SECONDMATE_SLOT_ID + TEARDOWN_SLOT_PERSISTENT_PARENT=$FM_TREEHOUSE_SECONDMATE_SLOT_PARENT + [ "$KIND" = scout ] && [ "$FORCE" != --force ] || return 0 + if [ "$TEARDOWN_WINDOWLESS" = 1 ]; then + endpoint=missing + else + endpoint=$(fm_backend_agent_state "$BACKEND" "$T") + fi + case "$endpoint" in + dead|missing) TEARDOWN_SLOT_PERSISTENT_NO_TOUCH=1 ;; + esac + return 0 + fi + case "$FM_TREEHOUSE_SLOT_OWNER" in + mine|absent) + echo "REFUSED: task $ID's recorded worktree $slot is secondmate $FM_TREEHOUSE_SECONDMATE_SLOT_ID's leased persistent home, but that slot's owner claim still names ${FM_TREEHOUSE_SLOT_OWNER_ID:-no task}; returning or resetting it would discard that home, so nothing was changed - not even with --force." >&2 + echo "Reconcile the claim from its registering home (FM_HOME=$FM_TREEHOUSE_SECONDMATE_SLOT_PARENT bin/fm-home-seed.sh claim-slot $FM_TREEHOUSE_SECONDMATE_SLOT_ID), then re-run teardown." >&2 + return 1 + ;; + esac + return 0 + fi + case "$FM_TREEHOUSE_SLOT_OWNER" in + mine|absent) + echo "REFUSED: task $ID's recorded worktree $slot shows a persistent secondmate home (a durable Treehouse lease or a secondmate marker), but its ownership cannot be proved: $FM_TREEHOUSE_SECONDMATE_SLOT_ERROR. Returning or resetting it could discard that home, so nothing was changed - not even with --force." >&2 + echo "Reconcile the home's lease, identity marker, parent binding, and registry route, then re-run teardown." >&2 + return 1 + ;; + esac } # Positive slot ownership, read from the claim the task that took the slot wrote @@ -2986,7 +3082,15 @@ preflight_descendant_treehouse_slots() { owner_rc=0 require_owned_worktree_slot_record "$task_id" "$worktree" || owner_rc=$? case "$owner_rc" in - 0|"$TEARDOWN_SLOT_REASSIGNED_RC") ;; + 0) + # A child that would take its slot back must not take a persistent + # secondmate home with it (teardown_persistent_slot_gate). + if fm_treehouse_slot_persistent_evidence "$(canonical_existing_dir "$worktree")"; then + echo "REFUSED: child $task_id's recorded worktree $worktree shows a persistent secondmate home (a durable Treehouse lease or a secondmate marker) while its slot-owner claim names no other owner; forced teardown changed nothing - reconcile that home's claim with bin/fm-home-seed.sh claim-slot from its registering home first." >&2 + return 1 + fi + ;; + "$TEARDOWN_SLOT_REASSIGNED_RC") ;; *) return 1 ;; esac done @@ -3332,8 +3436,13 @@ remove_secondmate_registry_entry() { return "$rc" } +teardown_persistent_slot_gate || exit 1 require_exclusive_task_worktree_slot || exit 1 require_owned_task_worktree_slot || exit 1 +if [ "$TEARDOWN_SLOT_PERSISTENT_NO_TOUCH" = 1 ] && teardown_owns_worktree; then + echo "REFUSED: task $ID's recorded worktree $WT changed owner while teardown held its locks; nothing was changed" >&2 + exit 1 +fi validate_pr_poll_cleanup "$STATE" "$ID" || exit 1 diff --git a/bin/fm-test-isolation-proof.sh b/bin/fm-test-isolation-proof.sh index 64ff6894737..60a1f37b7ec 100755 --- a/bin/fm-test-isolation-proof.sh +++ b/bin/fm-test-isolation-proof.sh @@ -139,6 +139,7 @@ exclusion_reason() { fm-backend-herdr-presentation-e2e.test.sh|fm-backend-herdr-prune-safety-e2e.test.sh|\ fm-backend-herdr-respawn-idem-e2e.test.sh|fm-backend-herdr-smoke.test.sh|\ fm-backend-herdr-agent-exit-shell-e2e.test.sh|\ + fm-teardown-persistent-slot-herdr-e2e.test.sh|\ fm-backend-herdr-workspace-per-home-e2e.test.sh|fm-herdr-session-cleanup-e2e.test.sh) printf '%s\n' 'real Herdr-gated; Herdr lane is a later phase' ;; diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index dfff544fbdf..9074ee5ea9b 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -326,6 +326,7 @@ family_for_basename() { fm-backend-herdr-focus-flash-e2e.test.sh|\ fm-backend-herdr-stale-active-tab-e2e.test.sh|\ fm-backend-herdr-agent-exit-shell-e2e.test.sh|\ + fm-teardown-persistent-slot-herdr-e2e.test.sh|\ fm-herdr-attached-viewer-live-e2e.test.sh|fm-herdr-session-cleanup-e2e.test.sh|\ fm-backend-herdr-smoke.test.sh|fm-backend-herdr-workspace-per-home-e2e.test.sh|\ fm-control-herdr-smoke.test.sh) diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 60a9d289090..d38b4b043c5 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1458,11 +1458,17 @@ fm_treehouse_pool_slot() { # # bin/fm-spawn.sh under the same project lock that allocates the slot and # released by bin/fm-teardown.sh when the slot goes back to the pool. Moving # crewmate spawns onto the durable lease is separate follow-up work. +# A leased secondmate home carries the same claim, naming the secondmate and +# its registering home: bin/fm-home-seed.sh publishes it under that lock when +# it leases the home, and its claim-slot command re-publishes it for a home +# seeded before seeding did, both through fm_treehouse_slot_owner_transfer +# after the lease or the full fm_treehouse_secondmate_slot_proof holds. # # The claim lives at //.fm-slot-owner - a sibling of the repo # checkout rather than a file inside it - so claiming a slot can never dirty the # copy teardown's landed-work checks inspect, and a returned slot carries no -# untracked leftover from it. +# untracked leftover from it. A transfer keeps the replaced claim's exact bytes +# beside it at .fm-slot-owner.prior as recovery evidence. fm_treehouse_slot_owner_marker() { # local worktree=$1 slot slot=$(CDPATH='' cd -- "$worktree" 2>/dev/null && pwd -P) || return 1 @@ -1539,6 +1545,172 @@ fm_treehouse_slot_owner_release() { # rm -f "$marker" 2>/dev/null || true } +# Hand a pool slot's claim to of , keeping the claim it replaces +# as recovery evidence at .prior. The caller holds the slot's project +# lock. Idempotent: a claim already naming exactly that task and home is left +# alone, and a retry after an interruption rewrites the same prior bytes before +# publishing. An unreadable claim, or a prior path that is not a plain file, +# refuses without writing. Sets FM_TREEHOUSE_SLOT_OWNER_TRANSFER to unchanged, +# claimed (no previous claim), or replaced. +fm_treehouse_slot_owner_transfer() { # + local worktree=$1 id=$2 home=$3 marker prior tmp + FM_TREEHOUSE_SLOT_OWNER_TRANSFER= + [ -n "$id" ] && [ -n "$home" ] || return 1 + marker=$(fm_treehouse_slot_owner_marker "$worktree") || return 1 + fm_treehouse_slot_owner_state "$worktree" "$id" + case "$FM_TREEHOUSE_SLOT_OWNER" in + mine) + if [ "$FM_TREEHOUSE_SLOT_OWNER_HOME" = "$home" ]; then + # shellcheck disable=SC2034 # Output global, read by the sourcing caller. + FM_TREEHOUSE_SLOT_OWNER_TRANSFER=unchanged + return 0 + fi + ;; + other) ;; + absent) + fm_treehouse_slot_owner_claim "$worktree" "$id" "$home" || return 1 + # shellcheck disable=SC2034 # Output global, read by the sourcing caller. + FM_TREEHOUSE_SLOT_OWNER_TRANSFER=claimed + return 0 + ;; + *) return 1 ;; + esac + prior="$marker.prior" + if { [ -e "$prior" ] || [ -L "$prior" ]; } \ + && { [ ! -f "$prior" ] || [ -L "$prior" ]; }; then + return 1 + fi + tmp="$prior.tmp.${BASHPID:-$$}" + rm -f "$tmp" || return 1 + cat -- "$marker" > "$tmp" 2>/dev/null || { rm -f "$tmp"; return 1; } + mv -f "$tmp" "$prior" 2>/dev/null || { rm -f "$tmp"; return 1; } + fm_treehouse_slot_owner_claim "$worktree" "$id" "$home" || return 1 + # shellcheck disable=SC2034 # Output global, read by the sourcing caller. + FM_TREEHOUSE_SLOT_OWNER_TRANSFER=replaced +} + +# Read a pool slot's durable Treehouse lease from its pool's +# treehouse-state.json. Prints the holder and returns 0 when exactly one entry +# resolves to and holds a lease; returns 1 when that one entry holds +# none (a crewmate slot's live process lease is not a durable lease); returns 2 +# when the state cannot answer - unreadable or malformed state, no entry or +# several, a non-boolean leased flag, or a leased entry without a plain holder. +fm_treehouse_slot_lease_holder() { # + local slot state + slot=$(CDPATH='' cd -- "$1" 2>/dev/null && pwd -P) || return 2 + state="$(dirname "$(dirname "$slot")")/treehouse-state.json" + [ -f "$state" ] && [ ! -L "$state" ] || return 2 + # shellcheck disable=SC2016 # Perl, not the shell, expands its variables. + LC_ALL=C perl -MJSON::PP -MCwd=realpath -e ' + my ($state, $slot) = @ARGV; + open(my $fh, "<:raw", $state) or exit 2; + my $text = do { local $/; <$fh> }; + close $fh; + my $doc = eval { JSON::PP->new->utf8->decode($text) }; + exit 2 unless ref $doc eq "HASH" && ref $doc->{worktrees} eq "ARRAY"; + my @hits; + for my $entry (@{$doc->{worktrees}}) { + next unless ref $entry eq "HASH" && defined $entry->{path} && !ref $entry->{path}; + my $path = realpath($entry->{path}); + push @hits, $entry if defined $path && $path eq $slot; + } + exit 2 unless @hits == 1; + my $leased = $hits[0]{leased}; + exit 1 unless defined $leased; + exit 2 unless JSON::PP::is_bool($leased); + exit 1 unless $leased; + my $holder = $hits[0]{lease_holder}; + exit 2 unless defined $holder && !ref $holder && $holder =~ /\A[A-Za-z0-9._-]+\z/; + print "$holder\n"; + exit 0; + ' "$state" "$slot" +} + +# True when a pool slot shows a persistent secondmate home that no ordinary task +# may ever return: a durable Treehouse lease is held on it, or it carries a +# secondmate identity marker or parent binding whatever its lease reads. +fm_treehouse_slot_persistent_evidence() { # + local slot=$1 + fm_treehouse_slot_lease_holder "$slot" >/dev/null && return 0 + [ -e "$slot/.fm-secondmate-home" ] || [ -L "$slot/.fm-secondmate-home" ] \ + || [ -e "$slot/.fm-secondmate-parent" ] || [ -L "$slot/.fm-secondmate-parent" ] +} + +# Positive proof that a Treehouse pool slot is a committed local secondmate +# home, persistently reassigned away from whatever task used it before. Every +# element must hold and agree: the slot belongs to 's pool (the same +# Git common directory), its pool state records exactly one durable lease with +# a readable holder, the slot's .fm-secondmate-home names that holder, its +# .fm-secondmate-parent is a valid local binding to a home in this machine's +# Firstmate tree, and that home's data/secondmates.md validates and routes that +# id, locally, to exactly this slot. Anything missing, unreadable, or +# contradictory fails. On success FM_TREEHOUSE_SECONDMATE_SLOT_ID is the +# secondmate id and FM_TREEHOUSE_SECONDMATE_SLOT_PARENT the canonical +# registering home; on failure FM_TREEHOUSE_SECONDMATE_SLOT_ERROR names the +# first element that did not hold. +# shellcheck disable=SC2034 # FM_TREEHOUSE_SECONDMATE_SLOT_* are output globals. +fm_treehouse_secondmate_slot_proof() { # + local project=$1 worktree=$2 slot holder marker_id parent slot_root home_root reg + FM_TREEHOUSE_SECONDMATE_SLOT_ID= + FM_TREEHOUSE_SECONDMATE_SLOT_PARENT= + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR= + if ! fm_treehouse_pool_slot "$project" "$worktree"; then + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$worktree is not a Treehouse pool slot of $project" + return 1 + fi + slot=$(CDPATH='' cd -- "$worktree" 2>/dev/null && pwd -P) || { + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$worktree cannot be resolved" + return 1 + } + holder=$(fm_treehouse_slot_lease_holder "$slot") || { + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="no readable durable Treehouse lease is recorded for $slot" + return 1 + } + if [ ! -f "$slot/.fm-secondmate-home" ] || [ -L "$slot/.fm-secondmate-home" ]; then + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$slot has no plain secondmate identity marker" + return 1 + fi + marker_id=$(cat -- "$slot/.fm-secondmate-home" 2>/dev/null) || marker_id= + if [ "$marker_id" != "$holder" ]; then + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$slot's secondmate identity marker names '${marker_id}', but its durable lease is held by '$holder'" + return 1 + fi + if ! command -v fm_secondmate_parent_record_parse >/dev/null 2>&1; then + # shellcheck source=bin/fm-secondmate-parent-lib.sh + . "$FM_WAKE_LIB_DIR/fm-secondmate-parent-lib.sh" + fi + if ! fm_secondmate_parent_record_parse "$slot/.fm-secondmate-parent" \ + || [ "$FM_SECONDMATE_PARENT_ROUTE" != local ]; then + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$slot has no valid local parent binding" + return 1 + fi + parent=$(CDPATH='' cd -- "$FM_SECONDMATE_PARENT_HOME" 2>/dev/null && pwd -P) || { + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$slot's parent binding names an unavailable home: $FM_SECONDMATE_PARENT_HOME" + return 1 + } + slot_root=$(fm_firstmate_root_home "$slot") || slot_root= + home_root=$(fm_firstmate_root_home "$FM_HOME") || home_root= + if [ -z "$slot_root" ] || [ "$slot_root" != "$home_root" ]; then + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$slot's parent binding to $parent is not in this Firstmate home's local tree" + return 1 + fi + if ! command -v secondmate_registry_validate_bindings >/dev/null 2>&1; then + # shellcheck source=bin/fm-secondmate-registry-lib.sh + . "$FM_WAKE_LIB_DIR/fm-secondmate-registry-lib.sh" + fi + reg="$parent/data/secondmates.md" + if ! secondmate_registry_validate_bindings "$reg" secondmate_registry_path_key "$holder" "$slot"; then + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="the parent binding's registry does not route $holder to $slot: $SECONDMATE_REGISTRY_ERROR" + return 1 + fi + if [ "$SECONDMATE_REGISTRY_MATCH_REMOTE" != 0 ]; then + FM_TREEHOUSE_SECONDMATE_SLOT_ERROR="$holder is registered in $reg as a remote route, not a local home" + return 1 + fi + FM_TREEHOUSE_SECONDMATE_SLOT_ID=$holder + FM_TREEHOUSE_SECONDMATE_SLOT_PARENT=$parent +} + fm_failure_episode_reset() { local state=$1 mode=${2:-acquire} lock current pid acquired=0 path lock="$state/.turnend-claude-blocks.lock" diff --git a/docs/architecture.md b/docs/architecture.md index 4b2b6f9cbfe..6cd5d21f20c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -413,9 +413,11 @@ A later merged poll consumes only that matching persisted value; with no match i Teardown is fail-closed for ship worktrees: dirty worktrees refuse, and committed work must be landed before the worktree is returned. A pool worktree is only returned after teardown passes the slot-ownership proof: a contradictory task record or a supported live endpoint refuses without touching either task, and no discard authority relaxes that. A slot's own owner claim, written by the spawn that takes it under the allocation lock and owned by [`bin/fm-wake-lib.sh`](../bin/fm-wake-lib.sh), covers a slot reassigned to another task, including one that left no record the scan could reach: a claim naming a different task releases nothing, even alongside that task's contradictory record - teardown warns, names the claimant, and finishes only the task's own cleanup - because Treehouse's own live process lease cannot answer ownership once the worker's exit releases it. +A leased secondmate home carries the same claim, published by [`bin/fm-home-seed.sh`](../bin/fm-home-seed.sh) under that lock when it leases the home and re-published by its `claim-slot` command, after a positive lease, identity, parent-binding, and registry proof, for a home seeded before it did. +A slot showing a durable lease or a secondmate marker is never returned by an ordinary task, and there a different claim alone never excuses another record naming the slot: only that proof lets a completed scout whose exact endpoint is dead or missing finish its own cleanup while other stale records still name the home. Allocation and return serialize on one project lock per machine-local Firstmate tree: every home reachable through local parent links shares that lock, and a home seeded from another machine anchors its own, because a lock taken on this filesystem is neither held nor observable across that boundary. Before the worktree is returned, teardown concludes the task's own no-mistakes run when it is parked at a gate, including a run whose head the task copy cannot resolve - the shared runs-ledger continuation proof is the only recognition for that case, so cleanup never orphans a parked run the pipeline advanced past the submitted head. -[`bin/fm-teardown.sh`](../bin/fm-teardown.sh)'s header owns the landed-work proofs, slot-ownership proof, endpoint-close refusal, PR-discovery fallback, pre-teardown run conclusion, and stale-lock recovery procedure; [`tests/fm-teardown-endpoint-safety.test.sh`](../tests/fm-teardown-endpoint-safety.test.sh) and [`tests/fm-secondmate-safety.test.sh`](../tests/fm-secondmate-safety.test.sh) pin the slot-collision boundary. +[`bin/fm-teardown.sh`](../bin/fm-teardown.sh)'s header owns the landed-work proofs, slot-ownership proof, endpoint-close refusal, PR-discovery fallback, pre-teardown run conclusion, and stale-lock recovery procedure; [`tests/fm-teardown-endpoint-safety.test.sh`](../tests/fm-teardown-endpoint-safety.test.sh) and [`tests/fm-secondmate-safety.test.sh`](../tests/fm-secondmate-safety.test.sh) pin the slot-collision boundary, and [`tests/fm-teardown-persistent-slot-herdr-e2e.test.sh`](../tests/fm-teardown-persistent-slot-herdr-e2e.test.sh) pins its real-Herdr endpoint classification. ## Optional Relay diff --git a/docs/configuration.md b/docs/configuration.md index 965ddbce5a9..984fbe1c43e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -699,6 +699,7 @@ A project-less seed requires no existing project clones or `data/projects.md` en A preexisting project-bearing charter is also refused until it is re-scaffolded with `--no-projects` or removed. The lease is held under the secondmate id until explicit retirement or seed rollback returns it, so normal restarts do not free or recycle the home. +The leased pool slot also carries the owner claim naming the secondmate; `fm-home-seed.sh claim-slot ` re-publishes it, after re-proving ownership, for a home seeded before seeding did. Teardown of a leased home fails closed if `treehouse return` cannot release the lease; plain-clone homes with no treehouse pool slot are removed directly. ### Project modes and backlog handoff diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index c28afacd8be..ab125dc0f79 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -845,6 +845,7 @@ tests/fm-backend-herdr-workspace-per-home-e2e.test.sh tests/fm-backend-herdr-launcher-workspace-e2e.test.sh tests/fm-backend-herdr-presentation-e2e.test.sh tests/fm-backend-herdr-agent-exit-shell-e2e.test.sh +tests/fm-teardown-persistent-slot-herdr-e2e.test.sh tests/fm-herdr-pi-stale-registration-live-e2e.test.sh tests/fm-backend-herdr-eventwait-smoke.test.sh tests/fm-control-herdr-smoke.test.sh diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index bef0095bb91..038e047a29d 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -1681,6 +1681,25 @@ ok - real herdr: a drifted agent-free shell returns to its worktree and reuses t `tests/fm-control-relaunch.test.sh` drives a tmux stub and proves that tmux retains its prior refusal without sending `cd` or any other input to the pane. The Herdr refusal when a shell accepts the command but does not move is not exercised in this change. +### Persistent-home scout cleanup + +Measured 2026-09-30 on macOS aarch64 against Herdr 0.9.1 in an isolated `fm-lab-` session, with no agent launched. +Teardown lets a completed scout whose record still names a proved persistent secondmate home skip the duplicate-record refusal only when `fm_backend_agent_state` reads its exact recorded pane `dead` or `missing`. +A process whose argv0 is `claude` stands in for a harness, and `herdr pane report-agent` registers it. +Refresh with: + +```sh +tests/fm-teardown-persistent-slot-herdr-e2e.test.sh +``` + +Observed 2026-09-30: + +```text +evidence: herdr 0.9.1 endpoint states: shell-only=dead closed=missing registered-claude=alive unregistered-claude=unreadable +ok - real Herdr: a live or unreadable old endpoint keeps the duplicate-record refusal on a reconciled persistent home +ok - real Herdr: completed scouts whose endpoints are dead or missing finish without touching the reconciled persistent home +``` + ### Stale agent registration Measured 2026-09-10 on macOS aarch64 against Herdr 0.9.0 (protocol 22) and Pi 0.85.1 in an isolated `fm-lab-` session (upstream issue #4115, duplicates #3639, #3487, #2908, #3545). diff --git a/tests/fm-secondmate-safety.test.sh b/tests/fm-secondmate-safety.test.sh index 38fabff2091..b115808b941 100755 --- a/tests/fm-secondmate-safety.test.sh +++ b/tests/fm-secondmate-safety.test.sh @@ -363,6 +363,119 @@ test_home_seed_warns_when_acquired_home_return_fails() { pass "home seed rollback warns when treehouse-acquired return fails" } +# A leased home that is a real Treehouse pool slot of the code root: the slot's +# previous task left its claim and a completed scout record behind. +make_claim_seed_fixture() { # -> prints " " + local name=$1 root pool slot home + root="$TMP_ROOT/$name-root" + pool="$TMP_ROOT/$name-pool" + home="$TMP_ROOT/$name-home" + mkdir -p "$root/bin" "$pool/1" "$home/data" "$home/state" "$home/projects" + printf '# fixture firstmate\n' > "$root/AGENTS.md" + printf 'fixture\n' > "$root/bin/README" + git -C "$root" init -q -b main + git -C "$root" add AGENTS.md bin/README + git -C "$root" -c user.name=test -c user.email=test@example.invalid commit -qm fixture + git -C "$root" worktree add -q --detach "$pool/1/firstmate" + slot=$(cd "$pool/1/firstmate" && pwd -P) + home=$(cd "$home" && pwd -P) + printf '{"worktrees":[{"name":"1","path":"%s"}]}\n' "$slot" > "$pool/treehouse-state.json" + printf 'task=old-scout\nhome=%s\n' "$home" > "$pool/1/.fm-slot-owner" + mkdir -p "$home/data/old-scout" + printf 'Complete fixture report.\n' > "$home/data/old-scout/report.md" + fm_write_meta "$home/state/old-scout.meta" \ + "window=firstmate:fm-old-scout" "endpoint_task_id=old-scout" \ + "worktree=$slot" "project=$root" "kind=scout" \ + "decisions_reviewed=1" "decision_keys=" + FM_HOME="$home" FM_SECONDMATE_CHARTER='claim fixture scope' FM_SECONDMATE_SCOPE='claim fixture scope' \ + "$ROOT/bin/fm-brief.sh" mate --secondmate --no-projects >/dev/null \ + || fail "could not scaffold the claim fixture's charter brief" + printf '%s %s %s %s\n' "$root" "$pool" "$slot" "$home" +} + +# Every file in a fixture pool except the slot checkout's own Git pointer. +claim_pool_fingerprint() { # + ( + cd "$1" || exit 1 + find . -path ./1/firstmate/.git -prune -o -type f -print | LC_ALL=C sort | while IFS= read -r path; do + printf '%s %s\n' "$(cksum < "$path")" "$path" + done + ) +} + +test_home_seed_claims_its_leased_pool_slot() { + local root pool slot home fakebin log out snapshot + read -r root pool slot home </dev/null || fail "seed did not report the leased slot" + [ "$(cat "$pool/1/.fm-slot-owner")" = "$(printf 'task=mate\nhome=%s' "$home")" ] \ + || fail "seed did not claim its leased pool slot: $(cat "$pool/1/.fm-slot-owner")" + [ "$(cat "$pool/1/.fm-slot-owner.prior")" = "$(printf 'task=old-scout\nhome=%s' "$home")" ] \ + || fail "seed did not keep the replaced claim as recovery evidence" + + # The real operator path continues: the old scout's completed record is + # cleaned up without touching the seeded home, with no reconciliation step. + snapshot=$(claim_pool_fingerprint "$pool") + : > "$log" + PATH="$fakebin:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" FM_FAKE_TMUX_LOG="$log" \ + "$ROOT/bin/fm-teardown.sh" old-scout > "$TMP_ROOT/claim-seed.out" 2> "$TMP_ROOT/claim-seed.err" \ + || fail "the old scout on a freshly seeded home did not finish: $(cat "$TMP_ROOT/claim-seed.err")" + assert_absent "$home/state/old-scout.meta" "the old scout's record was not removed" + [ "$(claim_pool_fingerprint "$pool")" = "$snapshot" ] \ + || fail "cleaning up the old scout changed the freshly seeded home, its lease, or its claim" + ! grep -F 'treehouse return' "$log" >/dev/null \ + || fail "cleaning up the old scout returned the freshly seeded home: $(cat "$log")" + pass "home seeding claims its leased pool slot under the project lock, so earlier scouts clean up without touching the home" +} + +test_home_seed_rollback_releases_its_slot_claim() { + local root pool slot home fakebin log err + read -r root pool slot home <> "$home/data/mate/brief.md" + if PATH="$fakebin:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$root" \ + FM_FAKE_TREEHOUSE_HOME="$slot" FM_FAKE_TREEHOUSE_STATE="$pool/treehouse-state.json" \ + FM_FAKE_TMUX_LOG="$log" "$ROOT/bin/fm-home-seed.sh" mate - --no-projects >/dev/null 2>"$err"; then + fail "seed succeeded with a placeholder charter" + fi + grep -F "treehouse return --force $slot" "$log" >/dev/null \ + || fail "the failed seed did not return its leased slot" + assert_absent "$pool/1/.fm-slot-owner" "the failed seed left its claim on the returned slot" + [ "$(cat "$pool/1/.fm-slot-owner.prior")" = "$(printf 'task=old-scout\nhome=%s' "$home")" ] \ + || fail "the failed seed lost the replaced claim's evidence" + + # Treehouse leased the slot but its pool state does not show the lease: the + # seed refuses to claim a home whose ownership it cannot prove, and returns it. + : > "$log" + printf 'task=old-scout\nhome=%s\n' "$home" > "$pool/1/.fm-slot-owner" + cp "$TMP_ROOT/claim-rollback-brief.md" "$home/data/mate/brief.md" + if PATH="$fakebin:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$root" \ + FM_FAKE_TREEHOUSE_HOME="$slot" FM_FAKE_TMUX_LOG="$log" \ + "$ROOT/bin/fm-home-seed.sh" mate - --no-projects >/dev/null 2>"$err"; then + fail "seed claimed a pool slot whose pool state records no lease" + fi + grep -F 'does not record a durable lease held by mate' "$err" >/dev/null \ + || fail "the unproved-lease refusal was not explained: $(cat "$err")" + [ "$(cat "$pool/1/.fm-slot-owner")" = "$(printf 'task=old-scout\nhome=%s' "$home")" ] \ + || fail "the unproved-lease refusal rewrote the slot's claim" + pass "home seed rollback returns its slot and drops only its own claim, and an unrecorded lease refuses the claim" +} + test_home_seed_does_not_return_unsafe_acquired_home() { local home descendant fakebin log err home="$TMP_ROOT/dash-active-home" @@ -2058,6 +2171,63 @@ EOF pass "forced secondmate teardown refuses duplicated descendant pool slots" } +# A forced retirement discards its own children's work, never another +# secondmate's persistent home that a child's stale record still names. +test_secondmate_force_teardown_refuses_child_slot_that_is_a_persistent_home() { + local home subhome childproj childwt fakebin log err rc + home="$TMP_ROOT/force-persistent-slot-home" + subhome="$TMP_ROOT/force-persistent-slot-subhome" + childproj="$subhome/projects/alpha" + childwt="$TMP_ROOT/force-persistent-slot-pool/1/alpha" + err="$TMP_ROOT/force-persistent-slot.err" + mkdir -p "$home/state" "$home/data" "$subhome/state" "$(dirname "$childwt")" + fm_git_worktree "$childproj" "$childwt" persistent-child + childwt=$(cd "$childwt" && pwd -P) + printf '{"worktrees":[{"name":"1","path":"%s","leased":true,"lease_holder":"other-mate"}]}\n' "$childwt" \ + > "$TMP_ROOT/force-persistent-slot-pool/treehouse-state.json" + printf 'other-mate\n' > "$childwt/.fm-secondmate-home" + printf 'domain\n' > "$subhome/.fm-secondmate-home" + cat > "$home/state/domain.meta" < "$home/data/secondmates.md" + cat > "$subhome/state/stale-child.meta" < "$TMP_ROOT/force-persistent-slot-pool/1/.fm-slot-owner" + fakebin=$(make_fake_tmux "$TMP_ROOT/force-persistent-slot-fake") + log="$TMP_ROOT/force-persistent-slot-fake/tmux.log" + + set +e + PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" \ + FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/force-persistent-slot-fake/pane.txt" \ + "$ROOT/bin/fm-teardown.sh" domain --force >/dev/null 2>"$err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "forced secondmate teardown returned a child slot that is a persistent home" + [ -f "$childwt/.fm-secondmate-home" ] || fail "forced secondmate teardown removed the persistent home in a child slot" + [ -e "$subhome/state/stale-child.meta" ] || fail "forced secondmate teardown removed the stale child record" + grep -F 'kill-window' "$log" >/dev/null && fail "forced secondmate teardown killed a child before refusing its persistent-home slot" + grep -F 'treehouse return' "$log" >/dev/null && fail "forced secondmate teardown returned a persistent-home slot" + grep -F 'persistent secondmate home' "$err" >/dev/null \ + || fail "forced secondmate teardown did not explain the persistent-home refusal: $(cat "$err")" + pass "forced secondmate teardown refuses a child slot that is another secondmate's persistent home" +} + test_secondmate_force_teardown_preserves_child_on_unproven_lock() { local home subhome childproj childwt fakebin log err rc lock home="$TMP_ROOT/force-lock-home" @@ -3035,6 +3205,8 @@ test_home_seed_uses_treehouse_acquired_home test_home_seed_returns_treehouse_acquired_home_on_assignment_failure test_home_seed_warns_when_acquired_home_return_fails test_home_seed_does_not_return_unsafe_acquired_home +test_home_seed_claims_its_leased_pool_slot +test_home_seed_rollback_releases_its_slot_claim test_home_seed_rolls_back_failed_clone test_home_seed_refuses_missing_filled_charter test_home_seed_refuses_placeholder_charter @@ -3081,6 +3253,7 @@ test_secondmate_teardown_refuses_failed_leased_home_return test_secondmate_teardown_removes_plain_clone_home_without_treehouse_return test_secondmate_force_teardown_discards_child_work test_secondmate_force_teardown_refuses_duplicated_child_slot +test_secondmate_force_teardown_refuses_child_slot_that_is_a_persistent_home test_secondmate_force_teardown_preserves_child_on_unproven_lock test_secondmate_force_teardown_allows_non_state_operational_dir_symlinks_inside_home test_secondmate_force_teardown_refuses_operational_dir_symlink_outside_home diff --git a/tests/fm-teardown-endpoint-safety.test.sh b/tests/fm-teardown-endpoint-safety.test.sh index 7ee608d2024..3fcddef517c 100755 --- a/tests/fm-teardown-endpoint-safety.test.sh +++ b/tests/fm-teardown-endpoint-safety.test.sh @@ -1420,6 +1420,442 @@ test_already_gone_endpoint_still_completes_without_a_refusal() { pass "fm-teardown: an already-exited endpoint, and a server that is already gone, still complete cleanup silently" } +# --- A persistent secondmate home seeded into a reused pool slot ------------- +# +# Seeding a secondmate durably leases a pool slot that completed scouts used +# before, so their stale records - and, for a home seeded before seeding +# published its own claim, the slot's old owner claim - still name what is now +# the secondmate's home. Returning or resetting that slot would discard the +# seeded home. These cases drive the real bin/fm-home-seed.sh claim-slot and +# bin/fm-teardown.sh interfaces: the claim is reconciled only after the +# persistent owner is proved, a completed scout whose endpoint is gone then +# finishes its own cleanup without touching the home, and every unproved, +# contradictory, live, or incomplete shape refuses before any mutation. + +PERSISTENT_MATE=harbor + +# Lays the committed seed of a local secondmate over the case's pool slot: a +# durable Treehouse lease, the identity marker, the local parent binding, the +# home's operational files and project clone, and the registering route. +seed_slot_as_persistent_home() { # [lease-holder] [registered-home] [parent-home] + local dir=$1 slot home holder registered parent + slot=$(cd "$dir/pool/1/project" && pwd -P) + home=$(cd "$dir/home" && pwd -P) + holder=${2:-$PERSISTENT_MATE} + registered=${3:-$slot} + parent=${4:-$home} + printf '{"worktrees":[{"name":"1","path":"%s","leased":true,"lease_holder":"%s"}]}\n' \ + "$slot" "$holder" > "$dir/pool/treehouse-state.json" + printf '%s\n' "$PERSISTENT_MATE" > "$slot/.fm-secondmate-home" + printf 'schema=fm-secondmate-parent.v1\nroute=local\nparent_home=%s\n' "$parent" \ + > "$slot/.fm-secondmate-parent" + mkdir -p "$slot/data" "$slot/state" "$slot/config" "$slot/projects" + printf 'Persistent fixture charter\n' > "$slot/data/charter.md" + printf 'saved review\n' > "$slot/data/review.md" + [ -d "$slot/projects/harbor/.git" ] || fm_git_init_commit "$slot/projects/harbor" >/dev/null + printf -- '- %s - Persistent fixture domain (home: %s; scope: persistent work; projects: harbor; added 2026-09-29)\n' \ + "$PERSISTENT_MATE" "$registered" > "$dir/home/data/secondmates.md" +} + +# A completed scout record naming the seeded slot: a report, a captain-call +# inventory that verifies, and an endpoint the fake tmux reports as gone. +write_persistent_scout() { # [decision-keys] + local dir=$1 id=$2 keys=${3:-} + mkdir -p "$dir/home/data/$id" + printf 'Complete fixture report with no unresolved choices.\n' > "$dir/home/data/$id/report.md" + fm_write_meta "$dir/home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" \ + "worktree=$dir/worktree" "project=$dir/project" "kind=scout" \ + "decisions_reviewed=1" "decision_keys=$keys" + printf 'done [at=123]: fixture report complete\n' > "$dir/home/state/$id.status" +} + +make_persistent_case() { # ... + local dir claimant=$2 + dir=$(make_case "$1") + shift 2 + mark_case_as_treehouse_pool "$dir" + seed_slot_as_persistent_home "$dir" + while [ "$#" -gt 0 ]; do + write_persistent_scout "$dir" "$1" + shift + done + claim_pool_slot "$dir" "$claimant" "$(cd "$dir/home" && pwd -P)" + printf '%s\n' "$dir" +} + +# Every byte of the pool - the seeded home, its clone, its lease, the claim and +# any retained prior claim - plus both checkouts' commits. +persistent_home_fingerprint() { # + ( + cd "$1/pool" || exit 1 + find . \( -type f -o -type l \) -print | LC_ALL=C sort | while IFS= read -r path; do + printf '%s %s\n' "$(cksum < "$path")" "$path" + done + git -C 1/project rev-parse HEAD + git -C 1/project/projects/harbor rev-parse HEAD + ) +} + +run_persistent_teardown() { # [teardown flags...] + local dir=$1 id=$2 + shift 2 + FM_HOME="$dir/home" FM_ROOT_OVERRIDE="$ROOT" FM_RUNTIME_LOG="$dir/runtime.log" \ + PATH="$dir/fakebin:$PATH" "$TEARDOWN" "$id" "$@" > "$dir/stdout" 2> "$dir/stderr" +} + +run_claim_slot() { # [mate] + local dir=$1 mate=${2:-$PERSISTENT_MATE} + FM_HOME="$dir/home" FM_ROOT_OVERRIDE="$dir/project" \ + "$ROOT/bin/fm-home-seed.sh" claim-slot "$mate" > "$dir/claim.out" 2> "$dir/claim.err" +} + +assert_persistent_teardown_refused() { # [teardown flags...] + local dir=$1 id=$2 before=$3 description=$4 meta_before rc=0 + shift 4 + meta_before=$(cksum < "$dir/home/state/$id.meta") + : > "$dir/runtime.log" + run_persistent_teardown "$dir" "$id" "$@" || rc=$? + [ "$rc" -ne 0 ] || fail "$description: teardown succeeded: $(cat "$dir/stdout")" + [ "$(cksum < "$dir/home/state/$id.meta")" = "$meta_before" ] \ + || fail "$description: the task record changed before the refusal" + [ "$(persistent_home_fingerprint "$dir")" = "$before" ] \ + || fail "$description: the seeded home, its clone, lease, or claim changed" + ! grep -Eq 'treehouse|kill-|send-keys' "$dir/runtime.log" \ + || fail "$description: a runtime close or pool command ran before the refusal: $(cat "$dir/runtime.log")" +} + +assert_claim_slot_refused() { # + local dir=$1 before=$2 description=$3 expected=$4 rc=0 + run_claim_slot "$dir" || rc=$? + [ "$rc" -ne 0 ] || fail "$description: claim-slot succeeded: $(cat "$dir/claim.out")" + [ "$(persistent_home_fingerprint "$dir")" = "$before" ] \ + || fail "$description: claim-slot changed the seeded home, its lease, or its claim" + assert_contains "$(cat "$dir/claim.err")" "$expected" "$description: claim-slot diagnostic" +} + +test_seeded_home_named_by_stale_scout_records_is_never_returned() { + local dir before id + + # The live shape before reconciliation: both completed scouts still name the + # seeded home and the claim still names the first. Neither order may proceed. + for id in old-scout-a old-scout-b; do + dir=$(make_persistent_case "stale-claim-$id" old-scout-a old-scout-a old-scout-b) + before=$(persistent_home_fingerprint "$dir") + assert_persistent_teardown_refused "$dir" "$id" "$before" "stale claim, two records, cleanup of $id" + assert_present "$dir/home/state/old-scout-a.meta" "stale claim: old-scout-a's record was removed" + assert_present "$dir/home/state/old-scout-b.meta" "stale claim: old-scout-b's record was removed" + done + + # The last stale record alone: its old claim reads as its own, which used to + # take the pool-return path against the seeded home. The durable lease is + # proof the slot is a persistent home, so the refusal holds even with --force + # and names the reconciliation to run. + for id in unforced --force; do + dir=$(make_persistent_case "stale-claim-sole$id" old-scout-a old-scout-a) + before=$(persistent_home_fingerprint "$dir") + if [ "$id" = --force ]; then + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "stale claim, sole record, --force" --force + else + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "stale claim, sole record" + fi + assert_contains "$(cat "$dir/stderr")" "claim-slot $PERSISTENT_MATE" \ + "stale claim, sole record: the refusal should name the claim reconciliation" + done + + pass "fm-teardown: a seeded persistent home named by stale scout records and an old claim is never returned, reset, or reaped" +} + +test_reconciled_seeded_home_lets_completed_dead_scouts_finish() { + local dir before home unrelated_before mate_before + + dir=$(make_persistent_case reconciled old-scout-a old-scout-a old-scout-b) + home=$(cd "$dir/home" && pwd -P) + # An unrelated task with its own endpoint must be left entirely alone. + fm_write_meta "$dir/home/state/unrelated.meta" \ + "window=firstmate:fm-unrelated" "endpoint_task_id=unrelated" \ + "worktree=$dir/unrelated-worktree" "project=$dir/project" "kind=ship" + unrelated_before=$(cksum < "$dir/home/state/unrelated.meta") + + run_claim_slot "$dir" || fail "claim-slot refused a proved persistent home: $(cat "$dir/claim.err")" + [ "$(cat "$dir/pool/1/.fm-slot-owner")" = "$(printf 'task=%s\nhome=%s' "$PERSISTENT_MATE" "$home")" ] \ + || fail "claim-slot did not publish the secondmate's claim: $(cat "$dir/pool/1/.fm-slot-owner")" + [ "$(cat "$dir/pool/1/.fm-slot-owner.prior")" = "$(printf 'task=old-scout-a\nhome=%s' "$home")" ] \ + || fail "claim-slot did not retain the replaced claim as recovery evidence" + before=$(persistent_home_fingerprint "$dir") + + : > "$dir/runtime.log" + run_persistent_teardown "$dir" old-scout-a \ + || fail "a completed dead scout on a reconciled persistent home did not finish: $(cat "$dir/stderr")" + assert_absent "$dir/home/state/old-scout-a.meta" "old-scout-a's own record was not removed" + assert_present "$dir/home/state/old-scout-b.meta" "old-scout-a's cleanup removed the other stale record" + assert_contains "$(cat "$dir/stdout")" "left to task $PERSISTENT_MATE" \ + "the completion line should name the persistent owner the slot was left to" + assert_contains "$(cat "$dir/stderr")" "old-scout-b" \ + "the cleanup should name the other stale record it left alone" + [ "$(persistent_home_fingerprint "$dir")" = "$before" ] \ + || fail "cleanup of old-scout-a changed the seeded home, its clone, lease, or claim" + ! grep -Fq 'treehouse' "$dir/runtime.log" \ + || fail "cleanup of old-scout-a ran a pool operation: $(cat "$dir/runtime.log")" + ! grep -Eq 'kill-[a-z]* <-t> <[^>]*(fm-unrelated|fm-old-scout-b)' "$dir/runtime.log" \ + || fail "cleanup of old-scout-a closed an unrelated endpoint: $(cat "$dir/runtime.log")" + [ "$(cksum < "$dir/home/state/unrelated.meta")" = "$unrelated_before" ] \ + || fail "cleanup of old-scout-a changed an unrelated task record" + + : > "$dir/runtime.log" + run_persistent_teardown "$dir" old-scout-b \ + || fail "the second completed dead scout did not finish: $(cat "$dir/stderr")" + assert_absent "$dir/home/state/old-scout-b.meta" "old-scout-b's own record was not removed" + [ "$(persistent_home_fingerprint "$dir")" = "$before" ] \ + || fail "cleanup of old-scout-b changed the seeded home, its clone, lease, or claim" + ! grep -Fq 'treehouse' "$dir/runtime.log" \ + || fail "cleanup of old-scout-b ran a pool operation: $(cat "$dir/runtime.log")" + + # Launching the secondmate first adds its own record naming the home. That + # record is the proved owner itself, so a stale scout still finishes and the + # secondmate's record and endpoint are untouched. + dir=$(make_persistent_case reconciled-launched old-scout-a old-scout-a) + run_claim_slot "$dir" || fail "claim-slot refused before launch: $(cat "$dir/claim.err")" + fm_write_meta "$dir/home/state/$PERSISTENT_MATE.meta" \ + "window=firstmate:fm-$PERSISTENT_MATE" "endpoint_task_id=$PERSISTENT_MATE" \ + "kind=secondmate" "home=$dir/worktree" "worktree=$dir/worktree" + mate_before=$(cksum < "$dir/home/state/$PERSISTENT_MATE.meta") + before=$(persistent_home_fingerprint "$dir") + : > "$dir/runtime.log" + run_persistent_teardown "$dir" old-scout-a \ + || fail "a stale scout beside the launched secondmate's record did not finish: $(cat "$dir/stderr")" + [ "$(cksum < "$dir/home/state/$PERSISTENT_MATE.meta")" = "$mate_before" ] \ + || fail "cleanup changed the launched secondmate's record" + ! grep -Eq "kill-[a-z]* <-t> <[^>]*fm-$PERSISTENT_MATE" "$dir/runtime.log" \ + || fail "cleanup closed the launched secondmate's endpoint: $(cat "$dir/runtime.log")" + [ "$(persistent_home_fingerprint "$dir")" = "$before" ] \ + || fail "cleanup beside the launched secondmate changed its home" + + # A second secondmate record naming the same home contradicts the proof. + dir=$(make_persistent_case reconciled-contradicted old-scout-a old-scout-a) + run_claim_slot "$dir" || fail "claim-slot refused before the contradiction: $(cat "$dir/claim.err")" + fm_write_meta "$dir/home/state/other-mate.meta" \ + "window=firstmate:fm-other-mate" "endpoint_task_id=other-mate" \ + "kind=secondmate" "home=$dir/worktree" "worktree=$dir/worktree" + before=$(persistent_home_fingerprint "$dir") + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "another secondmate record naming the home" + assert_contains "$(cat "$dir/stderr")" "other-mate" \ + "the contradiction refusal should name the other secondmate record" + + pass "fm-teardown: after a proved claim reconciliation, completed scouts with dead endpoints finish without touching the seeded home" +} + +test_persistent_home_reconciliation_refuses_unproved_ownership() { + local dir before elsewhere + + # Each shape breaks one element of the proof. claim-slot refuses without + # writing, and the stale scout's cleanup still refuses without touching. + dir=$(make_persistent_case unproved-lease-holder old-scout-a old-scout-a old-scout-b) + seed_slot_as_persistent_home "$dir" other-mate + before=$(persistent_home_fingerprint "$dir") + assert_claim_slot_refused "$dir" "$before" "mismatched lease holder" "lease" + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "mismatched lease holder" + + # A claim already naming the secondmate is a different owner, but on a + # persistent home that alone must not skip the duplicate-record scan the + # failed proof leaves in force. + dir=$(make_persistent_case unproved-lease-reconciled-claim "$PERSISTENT_MATE" old-scout-a old-scout-b) + seed_slot_as_persistent_home "$dir" other-mate + before=$(persistent_home_fingerprint "$dir") + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "mismatched lease holder under a secondmate claim" + assert_contains "$(cat "$dir/stderr")" "is also task old-scout-b's recorded worktree" \ + "mismatched lease holder under a secondmate claim: the duplicate-record refusal should stand" + + dir=$(make_persistent_case unproved-no-lease old-scout-a old-scout-a) + printf '{"worktrees":[{"name":"1","path":"%s"}]}\n' "$(cd "$dir/pool/1/project" && pwd -P)" \ + > "$dir/pool/treehouse-state.json" + before=$(persistent_home_fingerprint "$dir") + assert_claim_slot_refused "$dir" "$before" "no durable lease" "lease" + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "identity marker without a durable lease" + + dir=$(make_persistent_case unproved-pool-state old-scout-a old-scout-a) + printf 'not json\n' > "$dir/pool/treehouse-state.json" + before=$(persistent_home_fingerprint "$dir") + assert_claim_slot_refused "$dir" "$before" "unreadable pool state" "lease" + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "unreadable pool state" + + dir=$(make_persistent_case unproved-registry old-scout-a old-scout-a old-scout-b) + elsewhere="$dir/elsewhere-home" + mkdir -p "$elsewhere/state" "$elsewhere/data" + seed_slot_as_persistent_home "$dir" "$PERSISTENT_MATE" "$(cd "$elsewhere" && pwd -P)" + before=$(persistent_home_fingerprint "$dir") + assert_claim_slot_refused "$dir" "$before" "registry route naming another home" "registered" + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "registry route naming another home" + + dir=$(make_persistent_case unproved-parent old-scout-a old-scout-a old-scout-b) + elsewhere="$dir/elsewhere-parent" + mkdir -p "$elsewhere/state" "$elsewhere/data" + seed_slot_as_persistent_home "$dir" "$PERSISTENT_MATE" "" "$(cd "$elsewhere" && pwd -P)" + before=$(persistent_home_fingerprint "$dir") + assert_claim_slot_refused "$dir" "$before" "parent binding naming another home" "parent" + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "parent binding naming another home" + + dir=$(make_persistent_case unproved-identity old-scout-a old-scout-a) + printf 'other-mate\n' > "$dir/pool/1/project/.fm-secondmate-home" + before=$(persistent_home_fingerprint "$dir") + assert_claim_slot_refused "$dir" "$before" "identity marker naming another secondmate" "identity" + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "identity marker naming another secondmate" + + dir=$(make_persistent_case unproved-unsafe-claim old-scout-a old-scout-a) + rm -f "$dir/pool/1/.fm-slot-owner" + mkdir "$dir/pool/1/.fm-slot-owner" + before=$(persistent_home_fingerprint "$dir") + assert_claim_slot_refused "$dir" "$before" "unsafe claim" "claim" + [ -d "$dir/pool/1/.fm-slot-owner" ] || fail "unsafe claim: claim-slot replaced a directory claim" + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "unsafe claim" + + pass "fm-teardown: a mismatched, missing, unreadable, or contradictory persistent-home proof refuses reconciliation and cleanup without mutation" +} + +test_persistent_home_cleanup_keeps_completion_and_endpoint_gates() { + local dir before rc mode + + # Absent report and a failing decision inventory keep refusing: reconciling + # the home never completes a scout on its behalf. + dir=$(make_persistent_case gate-report old-scout-a old-scout-a old-scout-b) + run_claim_slot "$dir" || fail "claim-slot refused: $(cat "$dir/claim.err")" + rm -f "$dir/home/data/old-scout-a/report.md" + before=$(persistent_home_fingerprint "$dir") + assert_persistent_teardown_refused "$dir" old-scout-a "$before" "absent scout report" + assert_contains "$(cat "$dir/stderr")" "has no report" "absent report: the refusal should name the report" + + dir=$(make_persistent_case gate-decisions old-scout-a old-scout-a old-scout-b) + run_claim_slot "$dir" || fail "claim-slot refused: $(cat "$dir/claim.err")" + write_persistent_scout "$dir" old-scout-b fixture-unheld-decision + before=$(persistent_home_fingerprint "$dir") + assert_persistent_teardown_refused "$dir" old-scout-b "$before" "failing captain-call inventory" + assert_contains "$(cat "$dir/stderr")" "captain-call completion gate" \ + "failing inventory: the refusal should name the completion gate" + + # A live or unreadable old endpoint cannot use the reconciled path: the + # duplicate-record refusal stands until the endpoint is conclusively gone. + # The scripted tmux reports each window listed in live-windows as running a + # claude process; the real-backend death evidence is pinned separately by + # tests/fm-teardown-persistent-slot-herdr-e2e.test.sh. + dir=$(make_persistent_case gate-endpoint old-scout-a old-scout-a old-scout-b) + run_claim_slot "$dir" || fail "claim-slot refused: $(cat "$dir/claim.err")" + printf 'fm-old-scout-a\nfm-unrelated\n' > "$dir/live-windows" + cat > "$dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +printf 'tmux' >> "${FM_RUNTIME_LOG:?}" +printf ' <%s>' "$@" >> "${FM_RUNTIME_LOG:?}" +printf '\n' >> "${FM_RUNTIME_LOG:?}" +target= +prev= +for arg in "$@"; do + [ "$prev" != -t ] || target=$arg + prev=$arg +done +window=${target##*:} +window=${window#=} +case "${1:-}" in + list-windows) + if [ -n "${FM_TEST_UNREADABLE_LIST:-}" ]; then + echo "lost server" >&2 + exit 1 + fi + cat "${FM_TEST_LIVE_WINDOWS:?}" + ;; + display-message) + case "$*" in + *'#{pane_current_command}'*) + ! grep -Fqx -- "$window" "$FM_TEST_LIVE_WINDOWS" || echo claude + ;; + esac + ;; + kill-window) + grep -Fvx -- "$window" "$FM_TEST_LIVE_WINDOWS" > "$FM_TEST_LIVE_WINDOWS.next" || true + mv "$FM_TEST_LIVE_WINDOWS.next" "$FM_TEST_LIVE_WINDOWS" + ;; +esac +exit 0 +SH + chmod +x "$dir/fakebin/tmux" + before=$(persistent_home_fingerprint "$dir") + for mode in live unreadable; do + : > "$dir/runtime.log" + rc=0 + if [ "$mode" = unreadable ]; then + FM_TEST_UNREADABLE_LIST=1 FM_TEST_LIVE_WINDOWS="$dir/live-windows" \ + run_persistent_teardown "$dir" old-scout-a || rc=$? + else + FM_TEST_LIVE_WINDOWS="$dir/live-windows" run_persistent_teardown "$dir" old-scout-a || rc=$? + fi + [ "$rc" -ne 0 ] || fail "$mode old endpoint: teardown finished on a reconciled home" + assert_present "$dir/home/state/old-scout-a.meta" "$mode old endpoint: the record was removed" + [ "$(persistent_home_fingerprint "$dir")" = "$before" ] \ + || fail "$mode old endpoint: the seeded home changed" + ! grep -Eq 'treehouse|kill-' "$dir/runtime.log" \ + || fail "$mode old endpoint: a pool or close operation ran: $(cat "$dir/runtime.log")" + grep -Fqx fm-old-scout-a "$dir/live-windows" || fail "$mode old endpoint: the live endpoint was closed" + assert_contains "$(cat "$dir/stderr")" "is also task old-scout-b's recorded worktree" \ + "$mode old endpoint: the duplicate-record refusal should stand and name the other record" + done + + # The interrupted cleanup retried once the endpoint is really gone finishes. + printf 'fm-unrelated\n' > "$dir/live-windows" + : > "$dir/runtime.log" + FM_TEST_LIVE_WINDOWS="$dir/live-windows" run_persistent_teardown "$dir" old-scout-a \ + || fail "the retried cleanup after the endpoint died did not finish: $(cat "$dir/stderr")" + assert_absent "$dir/home/state/old-scout-a.meta" "the retried cleanup left the record" + [ "$(persistent_home_fingerprint "$dir")" = "$before" ] \ + || fail "the retried cleanup changed the seeded home" + ! grep -Fq 'treehouse' "$dir/runtime.log" \ + || fail "the retried cleanup ran a pool operation: $(cat "$dir/runtime.log")" + grep -Fqx fm-unrelated "$dir/live-windows" || fail "the retried cleanup closed an unrelated endpoint" + + pass "fm-teardown: reconciliation keeps the report, decision-inventory, and live or unreadable endpoint refusals, and a retry after endpoint death finishes" +} + +test_persistent_home_claim_reconciliation_is_idempotent_and_recoverable() { + local dir home claim prior lock_project lock_registry holder + + dir=$(make_persistent_case claim-idempotent old-scout-a old-scout-a) + home=$(cd "$dir/home" && pwd -P) + run_claim_slot "$dir" || fail "first claim-slot refused: $(cat "$dir/claim.err")" + claim=$(cksum < "$dir/pool/1/.fm-slot-owner") + prior=$(cksum < "$dir/pool/1/.fm-slot-owner.prior") + run_claim_slot "$dir" || fail "repeated claim-slot refused: $(cat "$dir/claim.err")" + assert_contains "$(cat "$dir/claim.out")" "unchanged" "a repeated claim-slot should report the claim unchanged" + [ "$(cksum < "$dir/pool/1/.fm-slot-owner")" = "$claim" ] || fail "a repeated claim-slot rewrote the claim" + [ "$(cksum < "$dir/pool/1/.fm-slot-owner.prior")" = "$prior" ] \ + || fail "a repeated claim-slot overwrote the retained prior claim" + + # A claim-slot killed after retaining the prior claim but before publishing + # the new one: its temp file and both locks are left behind by a dead process. + dir=$(make_persistent_case claim-interrupted old-scout-a old-scout-a) + home=$(cd "$dir/home" && pwd -P) + cp "$dir/pool/1/.fm-slot-owner" "$dir/pool/1/.fm-slot-owner.prior" + printf 'task=%s\nhome=%s\n' "$PERSISTENT_MATE" "$home" > "$dir/pool/1/.fm-slot-owner.tmp.4242" + lock_project=$(FM_HOME="$dir/home" bash -c '. "$1"; fm_treehouse_project_lock_path "$2"' _ \ + "$ROOT/bin/fm-wake-lib.sh" "$dir/project") || fail "could not resolve the project lock" + lock_registry="$dir/home/state/.secondmate-registry.lock" + # shellcheck disable=SC2016 # Expanded by the inner shells, not this one. + FM_HOME="$dir/home" bash -c ' + bash -c ". \"\$1\"; fm_lock_try_acquire \"\$2\" && fm_lock_try_acquire \"\$3\" && kill -KILL \"\$BASHPID\"" _ "$@" + exit 0 + ' _ "$ROOT/bin/fm-wake-lib.sh" "$lock_project" "$lock_registry" 2>/dev/null + [ -e "$lock_project" ] && [ -e "$lock_registry" ] || fail "could not stage the abandoned locks" + holder=$(cat "$lock_project/pid" 2>/dev/null || true) + ! kill -0 "$holder" 2>/dev/null || fail "the staged lock holder is still alive" + run_claim_slot "$dir" || fail "claim-slot did not recover an interrupted reconciliation: $(cat "$dir/claim.err")" + [ "$(cat "$dir/pool/1/.fm-slot-owner")" = "$(printf 'task=%s\nhome=%s' "$PERSISTENT_MATE" "$home")" ] \ + || fail "the recovered claim-slot did not publish the secondmate's claim" + [ "$(cat "$dir/pool/1/.fm-slot-owner.prior")" = "$(printf 'task=old-scout-a\nhome=%s' "$home")" ] \ + || fail "the recovered claim-slot lost the replaced claim's evidence" + run_persistent_teardown "$dir" old-scout-a \ + || fail "the stale scout did not finish after a recovered reconciliation: $(cat "$dir/stderr")" + ! grep -Fq 'treehouse' "$dir/runtime.log" \ + || fail "cleanup after a recovered reconciliation ran a pool operation: $(cat "$dir/runtime.log")" + + pass "fm-home-seed claim-slot: a repeated reconciliation is a no-op and an interrupted one converges without losing prior-claim evidence" +} + test_invalid_endpoint_records_refuse_before_mutation test_control_lock_contention_refuses_before_mutation test_non_pool_teardown_ignores_task_set_lock @@ -1442,6 +1878,11 @@ test_sole_slot_record_still_tears_down test_reassigned_pool_slot_finishes_own_cleanup_without_touching_the_slot test_stale_record_on_claimed_slot_retires_then_claimant_tears_down test_own_and_absent_slot_claims_still_tear_down +test_seeded_home_named_by_stale_scout_records_is_never_returned +test_reconciled_seeded_home_lets_completed_dead_scouts_finish +test_persistent_home_reconciliation_refuses_unproved_ownership +test_persistent_home_cleanup_keeps_completion_and_endpoint_gates +test_persistent_home_claim_reconciliation_is_idempotent_and_recoverable test_recorded_endpoint_that_changed_directory_still_tears_down test_project_lock_anchors_at_the_local_root_across_home_layouts test_remote_seeded_home_returns_its_uncontested_slot diff --git a/tests/fm-teardown-persistent-slot-herdr-e2e.test.sh b/tests/fm-teardown-persistent-slot-herdr-e2e.test.sh new file mode 100755 index 00000000000..4ee1cdcb039 --- /dev/null +++ b/tests/fm-teardown-persistent-slot-herdr-e2e.test.sh @@ -0,0 +1,264 @@ +#!/usr/bin/env bash +# Real-Herdr regression for cleaning up completed scouts whose records still +# name a persistent secondmate home seeded into their reused pool slot. +# bin/fm-teardown.sh lets such a scout skip the duplicate-record refusal only +# when the recovery-grade classifier reads its exact recorded endpoint dead or +# missing, so the endpoints here are real panes in an isolated named lab +# session: a shell-only pane (dead), a closed pane (missing), a registered +# agent over a claude-named process (alive), and that process unregistered +# (unreadable). The test drives the real bin/fm-home-seed.sh claim-slot and +# bin/fm-teardown.sh. Every adapter call goes through the guarded lab helper, +# and Treehouse is a logging stub that refuses, so a pool operation against the +# seeded home fails the test instead of running. No agent is launched and no +# model tokens are spent. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=tests/herdr-test-safety.sh +. "$ROOT/tests/herdr-test-safety.sh" + +herdr_forget_inherited_pane +fm_live_gate default-on FM_TEARDOWN_PERSISTENT_SLOT_HERDR_E2E herdr jq + +HERDR_LAB_HELPER=${HERDR_LAB_HELPER:-$ROOT/bin/fm-herdr-lab.sh} +[ -x "$HERDR_LAB_HELPER" ] || { echo "skip: live: Herdr lab helper not executable at $HERDR_LAB_HELPER"; exit 0; } + +HERDR_ORIGINAL_PATH=$PATH +REAL_HERDR=$(command -v herdr) +TMP_ROOT=$(fm_test_tmproot fm-teardown-persistent-slot-herdr-e2e) +FAKEBIN="$TMP_ROOT/fakebin" +RUNTIME_LOG="$TMP_ROOT/runtime.log" +mkdir -p "$FAKEBIN" +: > "$RUNTIME_LOG" + +HERDR_LAB_SESSION=$("$HERDR_LAB_HELPER" name fm-persistent-slot) +export HERDR_LAB_HELPER HERDR_LAB_SESSION HERDR_ORIGINAL_PATH REAL_HERDR RUNTIME_LOG + +cleanup() { + local status=$? + env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" teardown "$HERDR_LAB_SESSION" || status=1 + fm_test_cleanup + exit "$status" +} +trap cleanup EXIT +"$HERDR_LAB_HELPER" provision "$HERDR_LAB_SESSION" || fail "could not provision the isolated Herdr lab" + +# Log every production-adapter call, strip its already-validated trailing lab +# session flag, and send it through the lab helper, which alone appends the +# real session flag. The adapter's session-independent version read cannot pass +# the helper's leading-option guard, so only that read goes straight to the +# real binary with the same explicit lab session. Any other session refuses. +cat > "$FAKEBIN/herdr" <<'SH' +#!/usr/bin/env bash +set -u +printf 'herdr' >> "$RUNTIME_LOG" +printf ' <%s>' "$@" >> "$RUNTIME_LOG" +printf '\n' >> "$RUNTIME_LOG" +args=("$@") +last=$((${#args[@]} - 1)) +flag=$((last - 1)) +if [ "${#args[@]}" -ge 2 ] \ + && [ "${args[$flag]}" = --session ] \ + && [ "${args[$last]}" = "$HERDR_LAB_SESSION" ]; then + unset "args[$last]" "args[$flag]" +fi +set -- "${args[@]}" +for arg in "$@"; do + case "$arg" in --session|--session=*) exit 9 ;; esac +done +if [ "${1:-}" = --version ]; then + exec env PATH="$HERDR_ORIGINAL_PATH" "$REAL_HERDR" "$@" --session "$HERDR_LAB_SESSION" +fi +exec env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" "$@" +SH +cat > "$FAKEBIN/treehouse" <<'SH' +#!/usr/bin/env bash +printf 'treehouse' >> "$RUNTIME_LOG" +printf ' <%s>' "$@" >> "$RUNTIME_LOG" +printf '\n' >> "$RUNTIME_LOG" +exit 92 +SH +chmod +x "$FAKEBIN/herdr" "$FAKEBIN/treehouse" + +lab() { env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" "$@"; } + +endpoint_state() { # + PATH="$FAKEBIN:$HERDR_ORIGINAL_PATH" bash -c ' + set -u + . "$1/bin/fm-backend.sh" + fm_backend_agent_state herdr "$2:$3" + ' _ "$ROOT" "$HERDR_LAB_SESSION" "$1" +} + +pane_present() { # + lab pane get "$1" >/dev/null 2>&1 +} + +# --- The seeded home: a Treehouse pool slot leased to the secondmate --------- +MATE=harbor +PROJECT="$TMP_ROOT/project" +HOME_DIR="$TMP_ROOT/home" +POOL="$TMP_ROOT/pool" +fm_git_init_commit "$PROJECT" >/dev/null +mkdir -p "$HOME_DIR/data" "$HOME_DIR/state" "$HOME_DIR/config" "$POOL/1" +git -C "$PROJECT" worktree add -q --detach "$POOL/1/project" +SLOT=$(cd "$POOL/1/project" && pwd -P) +HOME_DIR=$(cd "$HOME_DIR" && pwd -P) +printf '{"worktrees":[{"name":"1","path":"%s","leased":true,"lease_holder":"%s"}]}\n' \ + "$SLOT" "$MATE" > "$POOL/treehouse-state.json" +printf '%s\n' "$MATE" > "$SLOT/.fm-secondmate-home" +printf 'schema=fm-secondmate-parent.v1\nroute=local\nparent_home=%s\n' "$HOME_DIR" > "$SLOT/.fm-secondmate-parent" +mkdir -p "$SLOT/data" "$SLOT/state" "$SLOT/config" "$SLOT/projects" +printf 'saved review\n' > "$SLOT/data/review.md" +fm_git_init_commit "$SLOT/projects/$MATE" >/dev/null +printf -- '- %s - Persistent fixture domain (home: %s; scope: persistent work; projects: %s; added 2026-09-29)\n' \ + "$MATE" "$SLOT" "$MATE" > "$HOME_DIR/data/secondmates.md" +# The slot's claim still names the scout that used it before the seed. +printf 'task=old-scout-a\nhome=%s\n' "$HOME_DIR" > "$POOL/1/.fm-slot-owner" + +home_fingerprint() { + ( + cd "$POOL" || exit 1 + find . \( -type f -o -type l \) -print | LC_ALL=C sort | while IFS= read -r path; do + printf '%s %s\n' "$(cksum < "$path")" "$path" + done + git -C 1/project rev-parse HEAD + git -C "1/project/projects/$MATE" rev-parse HEAD + ) +} + +# --- Four completed scouts, each bound to a real lab pane -------------------- +CREATE=$(lab workspace create --cwd "$TMP_ROOT" --label persistent-slot --no-focus) \ + || fail "could not create the lab workspace" +WS=$(printf '%s' "$CREATE" | jq -er '.result.workspace.workspace_id') || fail "no workspace id" +CONTROL_PANE=$(printf '%s' "$CREATE" | jq -er '.result.root_pane.pane_id') || fail "no control pane id" + +new_scout_pane() { # -> " " + local out + out=$(lab tab create --workspace "$WS" --cwd "$TMP_ROOT" --label "fm-$1" --no-focus) \ + || fail "could not create the lab tab for $1" + printf '%s %s\n' \ + "$(printf '%s' "$out" | jq -er '.result.tab.tab_id')" \ + "$(printf '%s' "$out" | jq -er '.result.root_pane.pane_id')" +} + +write_scout() { # + mkdir -p "$HOME_DIR/data/$1" + printf 'Complete fixture report with no unresolved choices.\n' > "$HOME_DIR/data/$1/report.md" + fm_write_meta "$HOME_DIR/state/$1.meta" \ + "window=$HERDR_LAB_SESSION:$3" "endpoint_task_id=$1" \ + "worktree=$SLOT" "project=$PROJECT" "kind=scout" "backend=herdr" \ + "herdr_session=$HERDR_LAB_SESSION" "herdr_workspace_id=$WS" \ + "herdr_tab_id=$2" "herdr_pane_id=$3" \ + "decisions_reviewed=1" "decision_keys=" +} + +read -r TAB_A PANE_A </dev/null || fail "could not close old-scout-b's pane" +lab pane run "$PANE_C" "(exec -a claude sleep 600)" >/dev/null || fail "could not start old-scout-c's process" +lab pane run "$PANE_D" "(exec -a claude sleep 600)" >/dev/null || fail "could not start old-scout-d's process" +for _ in $(seq 1 50); do + lab pane process-info --pane "$PANE_C" 2>/dev/null | grep -q '"argv0":"claude"' \ + && lab pane process-info --pane "$PANE_D" 2>/dev/null | grep -q '"argv0":"claude"' && break + sleep 0.2 +done +lab pane report-agent --source fm-test --agent claude --state idle "$PANE_C" >/dev/null \ + || fail "could not register old-scout-c's agent" +# Herdr detects the claude-named process on its own; classify only once it has, +# so a slow detection cannot read as an agent-free pane. +for _ in $(seq 1 100); do + lab agent get "$PANE_C" 2>/dev/null | jq -e '.result.agent' >/dev/null 2>&1 \ + && lab agent get "$PANE_D" 2>/dev/null | jq -e '.result.agent' >/dev/null 2>&1 && break + sleep 0.2 +done +lab agent get "$PANE_D" 2>/dev/null | jq -e '.result.agent' >/dev/null 2>&1 \ + || fail "Herdr never detected old-scout-d's claude-named process" + +run_teardown() { # + : > "$RUNTIME_LOG" + env FM_HOME="$HOME_DIR" FM_ROOT_OVERRIDE="$ROOT" PATH="$FAKEBIN:$HERDR_ORIGINAL_PATH" \ + "$ROOT/bin/fm-teardown.sh" "$1" > "$TMP_ROOT/$1.out" 2> "$TMP_ROOT/$1.err" +} + +assert_refused_untouched() { # + local id=$1 pane=$2 description=$3 before_meta + before_meta=$(cksum < "$HOME_DIR/state/$id.meta") + if run_teardown "$id"; then + fail "$description: teardown finished: $(cat "$TMP_ROOT/$id.out")" + fi + [ "$(cksum < "$HOME_DIR/state/$id.meta")" = "$before_meta" ] || fail "$description: the record changed" + [ "$(home_fingerprint)" = "$BEFORE" ] || fail "$description: the seeded home, lease, or claim changed" + ! grep -Eq 'treehouse|pane> "$TMP_ROOT/claim.out" 2> "$TMP_ROOT/claim.err" \ + || fail "claim-slot refused a proved persistent home: $(cat "$TMP_ROOT/claim.err")" +BEFORE=$(home_fingerprint) + +# The real classifier verdicts this cleanup path depends on. +STATE_A=$(endpoint_state "$PANE_A") +STATE_B=$(endpoint_state "$PANE_B") +STATE_C=$(endpoint_state "$PANE_C") +STATE_D=$(endpoint_state "$PANE_D") +printf 'evidence: %s endpoint states: shell-only=%s closed=%s registered-claude=%s unregistered-claude=%s\n' \ + "$("$FAKEBIN/herdr" --version --session "$HERDR_LAB_SESSION" 2>/dev/null | head -1)" "$STATE_A" "$STATE_B" "$STATE_C" "$STATE_D" +[ "$STATE_A" = dead ] || fail "a shell-only pane classified '$STATE_A', want dead" +[ "$STATE_B" = missing ] || fail "a closed pane classified '$STATE_B', want missing" +[ "$STATE_C" = alive ] || fail "a registered claude process classified '$STATE_C', want alive" +[ "$STATE_D" = unreadable ] || fail "an unregistered claude process classified '$STATE_D', want unreadable" + +assert_refused_untouched old-scout-c "$PANE_C" "live old endpoint" +assert_contains "$(cat "$TMP_ROOT/old-scout-c.err")" "is also task" \ + "a live old endpoint should keep the duplicate-record refusal" +assert_refused_untouched old-scout-d "$PANE_D" "unreadable old endpoint" +assert_contains "$(cat "$TMP_ROOT/old-scout-d.err")" "is also task" \ + "an unreadable old endpoint should keep the duplicate-record refusal" +pass "real Herdr: a live or unreadable old endpoint keeps the duplicate-record refusal on a reconciled persistent home" + +run_teardown old-scout-a || fail "the dead-endpoint scout did not finish: $(cat "$TMP_ROOT/old-scout-a.err")" +assert_absent "$HOME_DIR/state/old-scout-a.meta" "the dead-endpoint scout's record remained" +pane_present "$PANE_A" && fail "the dead-endpoint scout's own pane was not closed" +[ "$(home_fingerprint)" = "$BEFORE" ] || fail "the dead-endpoint cleanup changed the seeded home, lease, or claim" +! grep -Fq treehouse "$RUNTIME_LOG" || fail "the dead-endpoint cleanup ran a pool operation: $(cat "$RUNTIME_LOG")" + +run_teardown old-scout-b || fail "the missing-endpoint scout did not finish: $(cat "$TMP_ROOT/old-scout-b.err")" +assert_absent "$HOME_DIR/state/old-scout-b.meta" "the missing-endpoint scout's record remained" +[ "$(home_fingerprint)" = "$BEFORE" ] || fail "the missing-endpoint cleanup changed the seeded home, lease, or claim" +! grep -Fq treehouse "$RUNTIME_LOG" || fail "the missing-endpoint cleanup ran a pool operation: $(cat "$RUNTIME_LOG")" + +for pane in "$CONTROL_PANE" "$PANE_C" "$PANE_D"; do + pane_present "$pane" || fail "an unrelated or refused endpoint $pane was closed" +done +assert_present "$HOME_DIR/state/old-scout-c.meta" "the live scout's record was removed" +assert_present "$HOME_DIR/state/old-scout-d.meta" "the unreadable scout's record was removed" +pass "real Herdr: completed scouts whose endpoints are dead or missing finish without touching the reconciled persistent home" diff --git a/tests/secondmate-helpers.sh b/tests/secondmate-helpers.sh index 6ab1d955aa8..7c1d4e91f2a 100644 --- a/tests/secondmate-helpers.sh +++ b/tests/secondmate-helpers.sh @@ -14,7 +14,10 @@ # FM_FAKE_TMUX_WINDOW, capture-pane echoes FM_FAKE_TMUX_CAPTURE) plus a fake # treehouse (durable lease of FM_FAKE_TREEHOUSE_HOME, recording the lease holder # to FM_FAKE_TREEHOUSE_LEASE_FILE; `return` removes the target and lease unless -# FM_FAKE_TREEHOUSE_RETURN_FAIL is set). Echoes the fakebin dir. +# FM_FAKE_TREEHOUSE_RETURN_FAIL is set). With FM_FAKE_TREEHOUSE_STATE, the home +# is instead a real pool slot: `get` records the lease in that pool state file +# and `return` clears it, leaving the slot's checkout in place as Treehouse +# does. Echoes the fakebin dir. make_fake_tmux() { local dir=$1 fakebin capture fakebin=$(fm_fakebin "$dir") @@ -106,6 +109,9 @@ case "${1:-}" in if [ -n "${FM_FAKE_TREEHOUSE_HOME:-}" ]; then mkdir -p "$FM_FAKE_TREEHOUSE_HOME" [ -n "${FM_FAKE_TREEHOUSE_LEASE_FILE:-}" ] && printf '%s\n' "$holder" > "$FM_FAKE_TREEHOUSE_LEASE_FILE" + [ -z "${FM_FAKE_TREEHOUSE_STATE:-}" ] \ + || printf '{"worktrees":[{"name":"1","path":"%s","leased":true,"lease_holder":"%s"}]}\n' \ + "$FM_FAKE_TREEHOUSE_HOME" "$holder" > "$FM_FAKE_TREEHOUSE_STATE" printf 'leased worktree for %s\n' "${holder:-unknown}" >&2 printf '%s\n' "$FM_FAKE_TREEHOUSE_HOME" fi @@ -123,6 +129,10 @@ case "${1:-}" in done [ -z "${FM_FAKE_TREEHOUSE_RETURN_FAIL:-}" ] || exit 17 [ -n "${FM_FAKE_TREEHOUSE_LEASE_FILE:-}" ] && rm -f "$FM_FAKE_TREEHOUSE_LEASE_FILE" + if [ -n "${FM_FAKE_TREEHOUSE_STATE:-}" ]; then + printf '{"worktrees":[{"name":"1","path":"%s"}]}\n' "$target" > "$FM_FAKE_TREEHOUSE_STATE" + exit 0 + fi [ -n "$target" ] && rm -rf -- "$target" exit 0 ;; From 727f69232a8c5b0d9e6a2b0e39827fce4cf37c98 Mon Sep 17 00:00:00 2001 From: Luis Gonzalez Date: Wed, 30 Sep 2026 16:10:42 -0600 Subject: [PATCH 2/3] test(herdr): pin the unreadable persistent-slot endpoint's status The real-Herdr persistent-slot cleanup test relied on Herdr's own screen detection leaving an unreported claude-named pane at the undetermined `unknown` status. Herdr settles that pane on `idle` about 4 seconds after detection, so a slow run classified it `alive` instead of `unreadable` and failed; a 6-second delay before classification reproduced it every time. Report that pane's status as `unknown` explicitly, which makes the reporter the status authority, and wait for that status before classifying. The classifier path under test and every assertion are unchanged; the fixed test passes with 6- and 15-second delays and in 10 of 10 sequential runs. --- docs/verification/runtime-backends.md | 5 ++-- ...teardown-persistent-slot-herdr-e2e.test.sh | 23 +++++++++++-------- 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 038e047a29d..514a2a2512b 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -1685,7 +1685,8 @@ The Herdr refusal when a shell accepts the command but does not move is not exer Measured 2026-09-30 on macOS aarch64 against Herdr 0.9.1 in an isolated `fm-lab-` session, with no agent launched. Teardown lets a completed scout whose record still names a proved persistent secondmate home skip the duplicate-record refusal only when `fm_backend_agent_state` reads its exact recorded pane `dead` or `missing`. -A process whose argv0 is `claude` stands in for a harness, and `herdr pane report-agent` registers it. +A process whose argv0 is `claude` stands in for a harness, and `herdr pane report-agent` registers it with `idle` for the live case and `unknown` for the unreadable case. +Without a report, Herdr's own screen detection records such a pane `unknown` and settles it on `idle` about 4 seconds later, while a reported `unknown` held for the full 30-second sample, so only the report makes the unreadable case deterministic. Refresh with: ```sh @@ -1695,7 +1696,7 @@ tests/fm-teardown-persistent-slot-herdr-e2e.test.sh Observed 2026-09-30: ```text -evidence: herdr 0.9.1 endpoint states: shell-only=dead closed=missing registered-claude=alive unregistered-claude=unreadable +evidence: herdr 0.9.1 endpoint states: shell-only=dead closed=missing registered-claude=alive unknown-status-claude=unreadable ok - real Herdr: a live or unreadable old endpoint keeps the duplicate-record refusal on a reconciled persistent home ok - real Herdr: completed scouts whose endpoints are dead or missing finish without touching the reconciled persistent home ``` diff --git a/tests/fm-teardown-persistent-slot-herdr-e2e.test.sh b/tests/fm-teardown-persistent-slot-herdr-e2e.test.sh index 4ee1cdcb039..6d7abbd57d8 100755 --- a/tests/fm-teardown-persistent-slot-herdr-e2e.test.sh +++ b/tests/fm-teardown-persistent-slot-herdr-e2e.test.sh @@ -5,8 +5,8 @@ # when the recovery-grade classifier reads its exact recorded endpoint dead or # missing, so the endpoints here are real panes in an isolated named lab # session: a shell-only pane (dead), a closed pane (missing), a registered -# agent over a claude-named process (alive), and that process unregistered -# (unreadable). The test drives the real bin/fm-home-seed.sh claim-slot and +# agent over a claude-named process (alive), and that process reported with +# the undetermined `unknown` status (unreadable). The test drives the real bin/fm-home-seed.sh claim-slot and # bin/fm-teardown.sh. Every adapter call goes through the guarded lab helper, # and Treehouse is a logging stub that refuses, so a pool operation against the # seeded home fails the test instead of running. No agent is launched and no @@ -173,7 +173,7 @@ write_scout old-scout-d "$TAB_D" "$PANE_D" # old-scout-a: a shell-only pane. old-scout-b: its pane is closed. # old-scout-c: a claude-named process under a registered agent. -# old-scout-d: the same process with no registration. +# old-scout-d: the same process whose agent status is undetermined. lab pane close "$PANE_B" >/dev/null || fail "could not close old-scout-b's pane" lab pane run "$PANE_C" "(exec -a claude sleep 600)" >/dev/null || fail "could not start old-scout-c's process" lab pane run "$PANE_D" "(exec -a claude sleep 600)" >/dev/null || fail "could not start old-scout-d's process" @@ -184,15 +184,18 @@ for _ in $(seq 1 50); do done lab pane report-agent --source fm-test --agent claude --state idle "$PANE_C" >/dev/null \ || fail "could not register old-scout-c's agent" -# Herdr detects the claude-named process on its own; classify only once it has, -# so a slow detection cannot read as an agent-free pane. +# Herdr's own screen detection reads an unreported claude-named pane `unknown` +# for only a few seconds before settling on `idle`, so old-scout-d's status is +# pinned by an explicit report, which makes the reporter the status authority. +lab pane report-agent --source fm-test --agent claude --state unknown "$PANE_D" >/dev/null \ + || fail "could not report old-scout-d's undetermined status" for _ in $(seq 1 100); do lab agent get "$PANE_C" 2>/dev/null | jq -e '.result.agent' >/dev/null 2>&1 \ - && lab agent get "$PANE_D" 2>/dev/null | jq -e '.result.agent' >/dev/null 2>&1 && break + && lab agent get "$PANE_D" 2>/dev/null | jq -e '.result.agent.agent_status == "unknown"' >/dev/null 2>&1 && break sleep 0.2 done -lab agent get "$PANE_D" 2>/dev/null | jq -e '.result.agent' >/dev/null 2>&1 \ - || fail "Herdr never detected old-scout-d's claude-named process" +lab agent get "$PANE_D" 2>/dev/null | jq -e '.result.agent.agent_status == "unknown"' >/dev/null 2>&1 \ + || fail "Herdr never recorded old-scout-d's undetermined status" run_teardown() { # : > "$RUNTIME_LOG" @@ -230,12 +233,12 @@ STATE_A=$(endpoint_state "$PANE_A") STATE_B=$(endpoint_state "$PANE_B") STATE_C=$(endpoint_state "$PANE_C") STATE_D=$(endpoint_state "$PANE_D") -printf 'evidence: %s endpoint states: shell-only=%s closed=%s registered-claude=%s unregistered-claude=%s\n' \ +printf 'evidence: %s endpoint states: shell-only=%s closed=%s registered-claude=%s unknown-status-claude=%s\n' \ "$("$FAKEBIN/herdr" --version --session "$HERDR_LAB_SESSION" 2>/dev/null | head -1)" "$STATE_A" "$STATE_B" "$STATE_C" "$STATE_D" [ "$STATE_A" = dead ] || fail "a shell-only pane classified '$STATE_A', want dead" [ "$STATE_B" = missing ] || fail "a closed pane classified '$STATE_B', want missing" [ "$STATE_C" = alive ] || fail "a registered claude process classified '$STATE_C', want alive" -[ "$STATE_D" = unreadable ] || fail "an unregistered claude process classified '$STATE_D', want unreadable" +[ "$STATE_D" = unreadable ] || fail "a claude process with an undetermined status classified '$STATE_D', want unreadable" assert_refused_untouched old-scout-c "$PANE_C" "live old endpoint" assert_contains "$(cat "$TMP_ROOT/old-scout-c.err")" "is also task" \ From 1e71c998afd4ec213e1fbe2833f1ff24502e4789 Mon Sep 17 00:00:00 2001 From: Luis Gonzalez Date: Wed, 30 Sep 2026 18:40:44 -0600 Subject: [PATCH 3/3] no-mistakes(review): Ignore retired secondmate markers on slots the pool records unleased --- bin/fm-teardown.sh | 9 ++-- bin/fm-wake-lib.sh | 15 ++++-- docs/architecture.md | 2 +- tests/fm-secondmate-safety.test.sh | 14 +++++- tests/fm-teardown-endpoint-safety.test.sh | 59 ++++++++++++++++++++++- 5 files changed, 89 insertions(+), 10 deletions(-) diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index ba5aa719d60..d5d747f1642 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -131,8 +131,10 @@ # claim alone cannot settle: a home seeded before bin/fm-home-seed.sh published # its own claim still carries the claim of whichever task used the slot before, # and completed scouts' stale records still name it. So a slot showing a -# durable Treehouse lease or a secondmate identity marker or parent binding is -# never returned, reset, or reaped by an ordinary task: while the claim +# durable Treehouse lease, or a secondmate identity marker or parent binding +# while its pool state cannot answer, is never returned, reset, or reaped by an +# ordinary task; a slot its pool state records unleased is an ordinary slot +# whatever markers a retired secondmate left in it. While the claim # would still make this task its owner, teardown refuses, even with --force, # and names bin/fm-home-seed.sh claim-slot when the home's ownership is # otherwise proved. Once the claim names the persistent owner that @@ -2440,7 +2442,8 @@ require_exclusive_task_worktree_slot() { } # Persistent secondmate homes (see the script header). A pool slot showing a -# durable lease or a secondmate marker is a home no ordinary task may return, +# durable lease, or a secondmate marker while its pool state cannot answer +# (fm_treehouse_slot_persistent_evidence), is a home no ordinary task may return, # reset, or reap, so a claim that would still make this task its owner (its own # or none) refuses, even with --force. Only a positive # fm_treehouse_secondmate_slot_proof whose persistent owner the claim already diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index d38b4b043c5..f3ba0745ac0 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1627,11 +1627,18 @@ fm_treehouse_slot_lease_holder() { # } # True when a pool slot shows a persistent secondmate home that no ordinary task -# may ever return: a durable Treehouse lease is held on it, or it carries a -# secondmate identity marker or parent binding whatever its lease reads. +# may ever return: a durable Treehouse lease is held on it, or its pool state +# cannot answer and it carries a secondmate identity marker or parent binding. +# A slot its pool state records unleased is an ordinary returned slot: Treehouse +# keeps gitignored files across a return, so a retired secondmate's markers +# outlive the home they named and prove nothing about the task now using it. fm_treehouse_slot_persistent_evidence() { # - local slot=$1 - fm_treehouse_slot_lease_holder "$slot" >/dev/null && return 0 + local slot=$1 rc=0 + fm_treehouse_slot_lease_holder "$slot" >/dev/null || rc=$? + case "$rc" in + 0) return 0 ;; + 1) return 1 ;; + esac [ -e "$slot/.fm-secondmate-home" ] || [ -L "$slot/.fm-secondmate-home" ] \ || [ -e "$slot/.fm-secondmate-parent" ] || [ -L "$slot/.fm-secondmate-parent" ] } diff --git a/docs/architecture.md b/docs/architecture.md index 6cd5d21f20c..b24ec377479 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -414,7 +414,7 @@ Teardown is fail-closed for ship worktrees: dirty worktrees refuse, and committe A pool worktree is only returned after teardown passes the slot-ownership proof: a contradictory task record or a supported live endpoint refuses without touching either task, and no discard authority relaxes that. A slot's own owner claim, written by the spawn that takes it under the allocation lock and owned by [`bin/fm-wake-lib.sh`](../bin/fm-wake-lib.sh), covers a slot reassigned to another task, including one that left no record the scan could reach: a claim naming a different task releases nothing, even alongside that task's contradictory record - teardown warns, names the claimant, and finishes only the task's own cleanup - because Treehouse's own live process lease cannot answer ownership once the worker's exit releases it. A leased secondmate home carries the same claim, published by [`bin/fm-home-seed.sh`](../bin/fm-home-seed.sh) under that lock when it leases the home and re-published by its `claim-slot` command, after a positive lease, identity, parent-binding, and registry proof, for a home seeded before it did. -A slot showing a durable lease or a secondmate marker is never returned by an ordinary task, and there a different claim alone never excuses another record naming the slot: only that proof lets a completed scout whose exact endpoint is dead or missing finish its own cleanup while other stale records still name the home. +A slot showing a durable lease, or a secondmate marker while its pool state cannot be read, is never returned by an ordinary task - one the pool records unleased is ordinary whatever markers a retired secondmate left in it - and there a different claim alone never excuses another record naming the slot: only that proof lets a completed scout whose exact endpoint is dead or missing finish its own cleanup while other stale records still name the home. Allocation and return serialize on one project lock per machine-local Firstmate tree: every home reachable through local parent links shares that lock, and a home seeded from another machine anchors its own, because a lock taken on this filesystem is neither held nor observable across that boundary. Before the worktree is returned, teardown concludes the task's own no-mistakes run when it is parked at a gate, including a run whose head the task copy cannot resolve - the shared runs-ledger continuation proof is the only recognition for that case, so cleanup never orphans a parked run the pipeline advanced past the submitted head. [`bin/fm-teardown.sh`](../bin/fm-teardown.sh)'s header owns the landed-work proofs, slot-ownership proof, endpoint-close refusal, PR-discovery fallback, pre-teardown run conclusion, and stale-lock recovery procedure; [`tests/fm-teardown-endpoint-safety.test.sh`](../tests/fm-teardown-endpoint-safety.test.sh) and [`tests/fm-secondmate-safety.test.sh`](../tests/fm-secondmate-safety.test.sh) pin the slot-collision boundary, and [`tests/fm-teardown-persistent-slot-herdr-e2e.test.sh`](../tests/fm-teardown-persistent-slot-herdr-e2e.test.sh) pins its real-Herdr endpoint classification. diff --git a/tests/fm-secondmate-safety.test.sh b/tests/fm-secondmate-safety.test.sh index b115808b941..07918958b10 100755 --- a/tests/fm-secondmate-safety.test.sh +++ b/tests/fm-secondmate-safety.test.sh @@ -2225,7 +2225,19 @@ EOF grep -F 'treehouse return' "$log" >/dev/null && fail "forced secondmate teardown returned a persistent-home slot" grep -F 'persistent secondmate home' "$err" >/dev/null \ || fail "forced secondmate teardown did not explain the persistent-home refusal: $(cat "$err")" - pass "forced secondmate teardown refuses a child slot that is another secondmate's persistent home" + + # Once that secondmate retires, the pool records the slot unleased while its + # gitignored marker stays behind; the child's slot is then ordinary again. + printf '{"worktrees":[{"name":"1","path":"%s"}]}\n' "$childwt" \ + > "$TMP_ROOT/force-persistent-slot-pool/treehouse-state.json" + PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" \ + FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/force-persistent-slot-fake/pane.txt" \ + "$ROOT/bin/fm-teardown.sh" domain --force >/dev/null 2>"$err" \ + || fail "forced secondmate teardown refused a child slot holding only a retired secondmate's marker: $(cat "$err")" + grep -F "treehouse return --force $childwt" "$log" >/dev/null \ + || fail "forced secondmate teardown did not return the child's reused slot: $(cat "$log")" + [ ! -e "$subhome/state/stale-child.meta" ] || fail "forced secondmate teardown left the child record" + pass "forced secondmate teardown refuses a child slot that is another secondmate's persistent home, and returns it once that home is retired" } test_secondmate_force_teardown_preserves_child_on_unproven_lock() { diff --git a/tests/fm-teardown-endpoint-safety.test.sh b/tests/fm-teardown-endpoint-safety.test.sh index 3fcddef517c..9f805b15835 100755 --- a/tests/fm-teardown-endpoint-safety.test.sh +++ b/tests/fm-teardown-endpoint-safety.test.sh @@ -1671,7 +1671,6 @@ test_persistent_home_reconciliation_refuses_unproved_ownership() { > "$dir/pool/treehouse-state.json" before=$(persistent_home_fingerprint "$dir") assert_claim_slot_refused "$dir" "$before" "no durable lease" "lease" - assert_persistent_teardown_refused "$dir" old-scout-a "$before" "identity marker without a durable lease" dir=$(make_persistent_case unproved-pool-state old-scout-a old-scout-a) printf 'not json\n' > "$dir/pool/treehouse-state.json" @@ -1856,6 +1855,63 @@ test_persistent_home_claim_reconciliation_is_idempotent_and_recoverable() { pass "fm-home-seed claim-slot: a repeated reconciliation is a no-op and an interrupted one converges without losing prior-claim evidence" } +# A retired secondmate's slot goes back to the pool unleased, but Treehouse +# keeps gitignored files across a return, so its identity marker and parent +# binding stay in the slot. The next ordinary task the pool hands it to must +# still tear down and return it; the same markers keep refusing while the pool +# records a durable lease or cannot answer. +make_retired_home_slot_case() { # + local dir slot home id=$2 + dir=$(make_case "$1") + mark_case_as_treehouse_pool "$dir" + slot=$(cd "$dir/pool/1/project" && pwd -P) + home=$(cd "$dir/home" && pwd -P) + printf '%s\n' "$PERSISTENT_MATE" > "$slot/.fm-secondmate-home" + printf 'schema=fm-secondmate-parent.v1\nroute=local\nparent_home=%s\n' "$home" \ + > "$slot/.fm-secondmate-parent" + fm_git_init_commit "$slot/projects/harbor" >/dev/null + write_persistent_scout "$dir" "$id" + claim_pool_slot "$dir" "$id" "$home" + printf '%s\n' "$dir" +} + +test_retired_secondmate_markers_do_not_hold_a_reused_slot() { + local dir mode before + + for mode in unforced --force; do + dir=$(make_retired_home_slot_case "retired-home-reused$mode" next-crew) + : > "$dir/runtime.log" + if [ "$mode" = --force ]; then + run_persistent_teardown "$dir" next-crew --force \ + || fail "forced teardown of a crewmate in a retired secondmate's slot refused: $(cat "$dir/stderr")" + else + run_persistent_teardown "$dir" next-crew \ + || fail "teardown of a crewmate in a retired secondmate's slot refused: $(cat "$dir/stderr")" + fi + assert_absent "$dir/home/state/next-crew.meta" "$mode reused retired slot: the task record was left" + assert_absent "$dir/pool/1/.fm-slot-owner" "$mode reused retired slot: the spent claim was left" + grep -Fq "treehouse " "$dir/runtime.log" \ + || fail "$mode reused retired slot: the pool slot was not returned: $(cat "$dir/runtime.log")" + done + + dir=$(make_retired_home_slot_case retired-home-still-leased next-crew) + printf '{"worktrees":[{"name":"1","path":"%s","leased":true,"lease_holder":"%s"}]}\n' \ + "$(cd "$dir/pool/1/project" && pwd -P)" "$PERSISTENT_MATE" > "$dir/pool/treehouse-state.json" + before=$(persistent_home_fingerprint "$dir") + assert_persistent_teardown_refused "$dir" next-crew "$before" "markers on a still-leased slot" --force + assert_contains "$(cat "$dir/stderr")" "persistent secondmate home" \ + "markers on a still-leased slot: the refusal should name the persistent home" + + dir=$(make_retired_home_slot_case retired-home-unreadable-pool next-crew) + printf 'not json\n' > "$dir/pool/treehouse-state.json" + before=$(persistent_home_fingerprint "$dir") + assert_persistent_teardown_refused "$dir" next-crew "$before" "markers with an unreadable pool state" --force + assert_contains "$(cat "$dir/stderr")" "persistent secondmate home" \ + "markers with an unreadable pool state: the refusal should name the persistent home" + + pass "fm-teardown: a retired secondmate's leftover markers do not hold a reused unleased slot, while a leased or unreadable pool still refuses" +} + test_invalid_endpoint_records_refuse_before_mutation test_control_lock_contention_refuses_before_mutation test_non_pool_teardown_ignores_task_set_lock @@ -1883,6 +1939,7 @@ test_reconciled_seeded_home_lets_completed_dead_scouts_finish test_persistent_home_reconciliation_refuses_unproved_ownership test_persistent_home_cleanup_keeps_completion_and_endpoint_gates test_persistent_home_claim_reconciliation_is_idempotent_and_recoverable +test_retired_secondmate_markers_do_not_hold_a_reused_slot test_recorded_endpoint_that_changed_directory_still_tears_down test_project_lock_anchors_at_the_local_root_across_home_layouts test_remote_seeded_home_returns_its_uncontested_slot