feat(spawn): bootstrap project task environments before worker launch

[bramdokman]

Intent

Close the gap between a fresh Git worktree and a trustworthy execution environment for Firstmate tasks. Add a project-declared, toolchain-agnostic prerequisite bootstrap that runs after isolated worktree acquisition and before worker launch, uses a lockfile/toolchain/readiness fingerprint so a matching preserved environment skips cheaply, reruns when stale, and fails loudly without launching the worker or publishing success when a prerequisite fails. Do not add any pre-spawn git fetch, do not modify projects/, and do not hardcode Prisma, pnpm, or any specific project's toolchain into Firstmate. Cover the behavior with a biting regression that failed on the pre-change code; the exact pre-fix result was: 'not ok - spawn continued after its project-declared bootstrap failed'. In the PR body, state that no project declaration is added by this change and that each project remains responsible for declaring complete fingerprint inputs and prerequisite commands; Firstmate validates protocol execution, not project-specific correctness.

What Changed

• New bin/fm-task-bootstrap.sh runs a project's tracked, executable .firstmate/bootstrap declaration through Firstmate-owned fingerprint, run, and optional timeout modes. It hashes the declaration's own bytes together with the project's declared fingerprint text, stores the last successful value in the worktree-specific Git directory (so it survives Treehouse's reset and git clean -fd), skips run on a match, reruns when stale, and recomputes the fingerprint after a successful run. Projects with no declaration exit 0 with no output. The declaration is refused if it is untracked, not a regular executable file, or reached through a symlinked .firstmate; every mode runs with stdin from /dev/null and under a wall-clock bound (default 900s, project-overridable) enforced by timeout, gtimeout, or an inline Node executor that signals the whole process group with SIGTERM then SIGKILL — there is no unbounded fallback, and a host with none of the three refuses to run the declaration.
• bin/fm-spawn.sh invokes the bootstrap for every non-secondmate spawn after the isolated-worktree proof and before worker launch, exiting nonzero without launching the worker or publishing task metadata when it fails. To make that ordinary failure safe, tmux/zellij/cmux spawns gained an abort path (ENDPOINT_ABORT_CLEANUP) that kills the endpoint this process created and returns the leased Treehouse worktree — but only the acquisition validate_spawn_worktree positively accepted, never an unvalidated pane reading. The cleanup is disarmed once state/<id>.meta is published and teardown owns the release.
• Added tests/fm-task-bootstrap.test.sh (11 regressions covering the declaration contract, fingerprint skip/rerun lifecycle, bound enforcement, and fail-closed spawn) and routed fm-task-bootstrap.{sh,test.sh} into the backend-dispatch family in bin/fm-test-run.sh. Docs updated in docs/configuration.md (new "Project task bootstrap" section), docs/architecture.md, and docs/scripts.md; docs/herdr-backend.md records under Active limits that a failed Herdr spawn still leaves its task container and leased worktree behind, unlike the other backends.

No project bootstrap declaration is added by this change — no projects/ path is touched, no .firstmate/bootstrap is created, and no pre-spawn git fetch is introduced. Firstmate carries no Prisma, pnpm, or other toolchain-specific branches: each project remains responsible for declaring complete fingerprint inputs and correct prerequisite commands. Firstmate validates protocol execution — that the declaration is reviewable, noninteractive, bounded, and fails closed — not project-specific correctness. An incomplete project fingerprint yields a stale skip that only that project can fix.

Risk Assessment

✅ Low: Both round-2 findings are durably closed at the right boundary — the bound now has no unbounded path on any host and its regression fails rather than skips where no runtime exists, and the Treehouse return is armed solely from the validated acquisition — leaving an opt-in, tracked-only, symlink-guarded, noninteractive, fail-closed addition with behavioral coverage for every corrected invariant.

Testing

Ran the targeted tests/fm-task-bootstrap.test.sh suite (11 checks, all passing) plus the four adjacent spawn and teardown suites touched by the new endpoint abort-cleanup path, and proved the regression bites by reverting only bin/fm-spawn.sh to the base commit, which reproduced the stated pre-fix failure not ok - spawn continued after its project-declared bootstrap failed verbatim before I restored the change. For product-level evidence I wrote and ran a CLI demo that drives the real fm-spawn.sh against a fictional widgetc/deps.lock project: it captures a failed prerequisite refusing worker launch with no state/<id>.meta published and both the tmux endpoint and leased treehouse worktree released, a successful bootstrap followed by an actual worker launch, and the fingerprint lifecycle skipping cheaply on an unchanged environment while rerunning on a changed lockfile, a changed toolchain version, and a missing generated artifact — four prerequisite runs across six spawns. I also confirmed the intent's forbidden constraints in the diff (no projects/ changes, no declaration added, no pre-spawn git fetch, no project-toolchain hardcoding). This is a Bash CLI orchestration change with no rendered UI surface, so a command transcript is the artifact an end user would actually experience; no screenshot applies. Everything passed and the working tree was left clean with no transient artifacts.

Evidence: End-to-end fm-spawn task bootstrap transcript (fail-closed + fingerprint lifecycle)

=== Scene A: a declared prerequisite fails === $ fm-spawn.sh fix-widget-a1 acme-widgets task bootstrap: environment is stale; running .../wt-broken/.firstmate/bootstrap (bounded at 900s) [acme] installing declared dependencies from deps.lock ... cp: cannot stat 'deps.lock': No such file or directory task bootstrap: task bootstrap failed: .../wt-broken/.firstmate/bootstrap run error: task bootstrap failed for fix-widget-a1; refusing to launch worker exit status: 1 $ ls fm-home/state/fix-widget-a1.meta # was the worker's task published? ls: cannot access .../fix-widget-a1.meta: No such file or directory -> no task metadata, so no worker was launched and no success published $ cat calls.log # what the aborted spawn released tmux send-keys -t firstmate:fm-fix-widget-a1 treehouse get Enter tmux kill-window -t =firstmate:=fm-fix-widget-a1 treehouse return --force .../wt-broken $ grep send-keys calls.log | grep -v 'treehouse get' # harness launch keystrokes (no matches) -> only the worktree acquisition was sent; the codex worker itself was never launched === Scene B: fingerprint lifecycle on a preserved worktree === $ fm-spawn.sh add-widget-b1 acme-widgets # fresh worktree: nothing is prepared yet task bootstrap: environment is stale; running .../wt-ok/.firstmate/bootstrap (bounded at 900s) [acme] installing declared dependencies from deps.lock ... [acme] generating the widget client ... [acme] environment ready task bootstrap: prerequisites complete; recorded environment fingerprint spawned add-widget-b1 harness=codex kind=ship mode=no-mistakes yolo=off window=firstmate:fm-add-widget-b1 worktree=.../wt-ok $ ls fm-home/state/add-widget-b1.meta # worker task published? add-widget-b1.meta -> worker launched after prerequisites completed $ ls wt-ok/vendor wt-ok/build # what the project's prerequisites produced .../wt-ok/build: generated-client .../wt-ok/vendor: installed.lock $ fm-spawn.sh add-widget-b2 acme-widgets # same environment, nothing changed task bootstrap: environment fingerprint matches; skipping .../wt-ok/.firstmate/bootstrap $ echo "widget-core 1.5.0" >> wt-ok/deps.lock # lockfile moves $ fm-spawn.sh add-widget-b3 acme-widgets # lockfile changed task bootstrap: environment is stale; running ... (bounded at 900s) task bootstrap: prerequisites complete; recorded environment fingerprint $ echo widgetc-4.0 > wt-ok/toolchain.version # toolchain moves $ fm-spawn.sh add-widget-b4 acme-widgets # toolchain version changed task bootstrap: environment is stale; running ... (bounded at 900s) task bootstrap: prerequisites complete; recorded environment fingerprint $ rm wt-ok/build/generated-client # generated output goes missing $ fm-spawn.sh add-widget-b5 acme-widgets # readiness check fails task bootstrap: environment is stale; running ... (bounded at 900s) task bootstrap: prerequisites complete; recorded environment fingerprint $ fm-spawn.sh add-widget-b6 acme-widgets # environment is ready again task bootstrap: environment fingerprint matches; skipping .../wt-ok/.firstmate/bootstrap $ wc -l run.log # how many times the project's prerequisites actually ran 4 prerequisite runs across 6 spawns on the same preserved worktree

$ cat acme-widgets/.firstmate/bootstrap   # the project declares, Firstmate executes
    #!/usr/bin/env bash
    # Project-owned declaration. Firstmate knows none of these commands or inputs.
    set -eu
    case "${1:-}" in
      fingerprint)
        # Every input that can stale this environment: lockfile, toolchain version,
        # and a cheap readiness check on generated output.
        printf 'deps.lock=%s\n' "$(git hash-object deps.lock)"
        printf 'widgetc=%s\n' "$(cat toolchain.version)"
        if [ -f build/generated-client ]; then
          printf 'generated-client=%s\n' "$(git hash-object build/generated-client)"
        else
          printf 'generated-client=absent\n'
        fi
        ;;
      run)
        echo "  [acme] installing declared dependencies from deps.lock ..."
        mkdir -p vendor build
        cp deps.lock vendor/installed.lock
        echo "  [acme] generating the widget client ..."
        cat deps.lock toolchain.version > build/generated-client
        printf 'ran\n' >> "$DEMO_RUN_LOG"
        echo "  [acme] environment ready"
        ;;
      *) exit 2 ;;
    esac

�[0m========== Scene A: a declared prerequisite fails ==========

$ fm-spawn.sh fix-widget-a1 acme-widgets
    fatal: could not open 'deps.lock' for reading: No such file or directory
    task bootstrap: environment is stale; running /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-broken/.firstmate/bootstrap (bounded at 900s)
      [acme] installing declared dependencies from deps.lock ...
    cp: cannot stat 'deps.lock': No such file or directory
    task bootstrap: task bootstrap failed: /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-broken/.firstmate/bootstrap run
    error: task bootstrap failed for fix-widget-a1; refusing to launch worker

    exit status: 1

$ ls fm-home/state/fix-widget-a1.meta   # was the worker's task published?
    ls: cannot access .../fix-widget-a1.meta: No such file or directory
    -> no task metadata, so no worker was launched and no success published

$ cat calls.log   # what the aborted spawn released
    tmux send-keys -t firstmate:fm-fix-widget-a1 treehouse get Enter
    tmux kill-window -t =firstmate:=fm-fix-widget-a1
    treehouse return --force /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-broken

$ grep send-keys calls.log | grep -v 'treehouse get'   # harness launch keystrokes
    (no matches)   -> only the worktree acquisition was sent; the codex
                      worker itself was never launched

�[0m========== Scene B: fingerprint lifecycle on a preserved worktree ==========

$ fm-spawn.sh add-widget-b1 acme-widgets   # fresh worktree: nothing is prepared yet
    task bootstrap: environment is stale; running /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/.firstmate/bootstrap (bounded at 900s)
      [acme] installing declared dependencies from deps.lock ...
      [acme] generating the widget client ...
      [acme] environment ready
    task bootstrap: prerequisites complete; recorded environment fingerprint
    spawned add-widget-b1 harness=codex kind=ship mode=no-mistakes yolo=off window=firstmate:fm-add-widget-b1 worktree=/tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok

$ ls fm-home/state/add-widget-b1.meta   # worker task published?
    add-widget-b1.meta   -> worker launched after prerequisites completed

$ ls wt-ok/vendor wt-ok/build   # what the project's prerequisites produced
    /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/build:
    generated-client
    
    /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/vendor:
    installed.lock

$ fm-spawn.sh add-widget-b2 acme-widgets   # same environment, nothing changed
    task bootstrap: environment fingerprint matches; skipping /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/.firstmate/bootstrap

$ echo "widget-core 1.5.0" >> wt-ok/deps.lock   # lockfile moves

$ fm-spawn.sh add-widget-b3 acme-widgets   # lockfile changed
    task bootstrap: environment is stale; running /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/.firstmate/bootstrap (bounded at 900s)
      [acme] installing declared dependencies from deps.lock ...
      [acme] generating the widget client ...
      [acme] environment ready
    task bootstrap: prerequisites complete; recorded environment fingerprint

$ echo widgetc-4.0 > wt-ok/toolchain.version   # toolchain moves

$ fm-spawn.sh add-widget-b4 acme-widgets   # toolchain version changed
    task bootstrap: environment is stale; running /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/.firstmate/bootstrap (bounded at 900s)
      [acme] installing declared dependencies from deps.lock ...
      [acme] generating the widget client ...
      [acme] environment ready
    task bootstrap: prerequisites complete; recorded environment fingerprint

$ rm wt-ok/build/generated-client   # generated output goes missing

$ fm-spawn.sh add-widget-b5 acme-widgets   # readiness check fails
    task bootstrap: environment is stale; running /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/.firstmate/bootstrap (bounded at 900s)
      [acme] installing declared dependencies from deps.lock ...
      [acme] generating the widget client ...
      [acme] environment ready
    task bootstrap: prerequisites complete; recorded environment fingerprint

$ fm-spawn.sh add-widget-b6 acme-widgets   # environment is ready again
    task bootstrap: environment fingerprint matches; skipping /tmp/no-mistakes-evidence/01KZ5BSSW40FVRAYQJQTNE6BZ0/demo-run/wt-ok/.firstmate/bootstrap

$ wc -l run.log   # how many times the project's prerequisites actually ran
    4 prerequisite runs across 6 spawns on the same preserved worktree

�[0m========== Done ==========

Evidence: Regression bites on pre-change code (bin/fm-spawn.sh reverted to base cbd5d67)

$ git checkout cbd5d67 -- bin/fm-spawn.sh $ bash tests/fm-task-bootstrap.test.sh not ok - spawn continued after its project-declared bootstrap failed exit=1 $ git checkout HEAD -- bin/fm-spawn.sh # change restored, worktree clean

not ok - spawn continued after its project-declared bootstrap failed

Evidence: Targeted task-bootstrap suite on the target commit

$ bash tests/fm-task-bootstrap.test.sh ok - a failed project bootstrap loudly stops spawn before worker launch ok - stale fingerprints run bootstrap while matching fingerprints skip it ok - a failed prerequisite stays loud and cannot publish a success marker ok - only a tracked declaration counts as opt-in ok - a symlinked .firstmate parent cannot smuggle in a declaration ok - every declared mode runs with stdin from /dev/null ok - every available bounded runner stops a hanging run and kills its process group ok - a host with no bounded runtime refuses the declaration instead of running it unbounded ok - only timeout, gtimeout, or node may be pinned as the bounded runner ok - declarations without a timeout mode run under the documented 900s default ok - an unvalidated pane path is never returned to the treehouse pool # all task-bootstrap tests passed

ok - a failed project bootstrap loudly stops spawn before worker launch
ok - stale fingerprints run bootstrap while matching fingerprints skip it
ok - a failed prerequisite stays loud and cannot publish a success marker
ok - only a tracked declaration counts as opt-in
ok - a symlinked .firstmate parent cannot smuggle in a declaration
ok - every declared mode runs with stdin from /dev/null
ok - every available bounded runner stops a hanging run and kills its process group
ok - a host with no bounded runtime refuses the declaration instead of running it unbounded
ok - only timeout, gtimeout, or node may be pinned as the bounded runner
ok - declarations without a timeout mode run under the documented 900s default
ok - an unvalidated pane path is never returned to the treehouse pool
# all task-bootstrap tests passed

Evidence: Reproducible end-user demo script used to produce the transcript

#!/usr/bin/env bash
# End-user demo of the project-declared task bootstrap, driven through the real
# bin/fm-spawn.sh with a fake tmux/treehouse runtime backend.
#
# Scene A: a project whose declared prerequisite fails -> spawn refuses to launch
#          the worker, publishes no task metadata, and releases what it acquired.
# Scene B: the same project with a working declaration -> first spawn prepares the
#          environment, a second spawn on the preserved environment skips cheaply,
#          and a changed lockfile / changed toolchain / missing generated output
#          each make it rerun.
set -u

FM_SRC=${FM_SRC:?FM_SRC unset}
DEMO=${DEMO:?DEMO unset}
SPAWN="$FM_SRC/bin/fm-spawn.sh"

rule() { printf '\n\033[0m========== %s ==========\n' "$*"; }
step() { printf '\n$ %s\n' "$*"; }

git_id=(-c user.name='Demo' -c user.email='demo@example.invalid' -c commit.gpgsign=false)

fakebin="$DEMO/fakebin"
mkdir -p "$fakebin"
cat > "$fakebin/tmux" <<'SH'
#!/usr/bin/env bash
set -u
case "$*" in
  *"#{pane_current_path}"*) printf '%s\n' "${FM_FAKE_PANE_PATH:?}"; exit 0 ;;
esac
case "${1:-}" in
  display-message) printf 'firstmate\n'; exit 0 ;;
  list-windows) exit 0 ;;
  kill-window|send-keys)
    [ -z "${FM_FAKE_CALL_LOG:-}" ] || printf 'tmux %s\n' "$*" >> "$FM_FAKE_CALL_LOG"
    exit 0 ;;
esac
exit 0
SH
cat > "$fakebin/treehouse" <<'SH'
#!/usr/bin/env bash
set -u
[ -z "${FM_FAKE_CALL_LOG:-}" ] || printf 'treehouse %s\n' "$*" >> "$FM_FAKE_CALL_LOG"
exit 0
SH
chmod +x "$fakebin/tmux" "$fakebin/treehouse"

home="$DEMO/fm-home"
mkdir -p "$home/projects" "$home/state" "$home/config" "$home/data"
printf 'codex\n' > "$home/config/crew-harness"
touch "$home/state/.last-watcher-beat"

# --- a project that is nobody's specific toolchain --------------------------
project="$DEMO/acme-widgets"
mkdir -p "$project"
git -C "$project" init -q
mkdir -p "$project/.firstmate"
cat > "$project/.firstmate/bootstrap" <<'SH'
#!/usr/bin/env bash
# Project-owned declaration. Firstmate knows none of these commands or inputs.
set -eu
case "${1:-}" in
  fingerprint)
    # Every input that can stale this environment: lockfile, toolchain version,
    # and a cheap readiness check on generated output.
    printf 'deps.lock=%s\n' "$(git hash-object deps.lock)"
    printf 'widgetc=%s\n' "$(cat toolchain.version)"
    if [ -f build/generated-client ]; then
      printf 'generated-client=%s\n' "$(git hash-object build/generated-client)"
    else
      printf 'generated-client=absent\n'
    fi
    ;;
  run)
    echo "  [acme] installing declared dependencies from deps.lock ..."
    mkdir -p vendor build
    cp deps.lock vendor/installed.lock
    echo "  [acme] generating the widget client ..."
    cat deps.lock toolchain.version > build/generated-client
    printf 'ran\n' >> "$DEMO_RUN_LOG"
    echo "  [acme] environment ready"
    ;;
  *) exit 2 ;;
esac
SH
chmod +x "$project/.firstmate/bootstrap"
printf 'widget-core 1.4.0\nwidget-ui 2.0.1\n' > "$project/deps.lock"
printf 'widgetc-3.11\n' > "$project/toolchain.version"
printf 'vendor/\nbuild/\n' > "$project/.gitignore"
git -C "$project" add -A
git -C "$project" "${git_id[@]}" commit -qm 'declare task prerequisites'

step "cat acme-widgets/.firstmate/bootstrap   # the project declares, Firstmate executes"
sed 's/^/    /' "$project/.firstmate/bootstrap"

run_spawn() {  # <id> <worktree> <call-log>
  local id=$1 wt=$2 log=$3
  mkdir -p "$home/data/$id"
  printf 'demo task brief\n' > "$home/data/$id/brief.md"
  FM_ROOT_OVERRIDE='' FM_HOME="$home" FM_GATE_REFUSE_BYPASS=1 \
    FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \
    FM_PROJECTS_OVERRIDE="$home/projects" FM_CONFIG_OVERRIDE="$home/config" \
    FM_SPAWN_NO_GUARD=1 TMUX='fake,1,0' FM_FAKE_PANE_PATH="$wt" \
    FM_FAKE_CALL_LOG="$log" DEMO_RUN_LOG="$DEMO/run.log" \
    PATH="$fakebin:$PATH" "$SPAWN" "$id" "$project" 2>&1 | sed 's/^/    /'
  return "${PIPESTATUS[0]}"
}

# ============================================================================
rule "Scene A: a declared prerequisite fails"
# ============================================================================
wt_a="$DEMO/wt-broken"
git -C "$project" worktree add --quiet -b demo-broken "$wt_a"
# The project's dependency source is unreachable for this task.
rm "$wt_a/deps.lock"
log_a="$DEMO/calls-a.log"; : > "$log_a"

step "fm-spawn.sh fix-widget-a1 acme-widgets"
run_spawn fix-widget-a1 "$wt_a" "$log_a"; status=$?
printf '\n    exit status: %s\n' "$status"

step "ls fm-home/state/fix-widget-a1.meta   # was the worker's task published?"
if [ -e "$home/state/fix-widget-a1.meta" ]; then
  printf '    PUBLISHED (bad)\n'
else
  printf '    ls: cannot access .../fix-widget-a1.meta: No such file or directory\n'
  printf '    -> no task metadata, so no worker was launched and no success published\n'
fi

step "cat calls.log   # what the aborted spawn released"
sed 's/^/    /' "$log_a"

step "grep send-keys calls.log | grep -v 'treehouse get'   # harness launch keystrokes"
if grep 'send-keys' "$log_a" | grep -v 'treehouse get' | sed 's/^/    /' | grep .; then
  printf '    ^ a harness was launched (bad)\n'
else
  printf '    (no matches)   -> only the worktree acquisition was sent; the codex\n'
  printf '                      worker itself was never launched\n'
fi

# ============================================================================
rule "Scene B: fingerprint lifecycle on a preserved worktree"
# ============================================================================
wt_b="$DEMO/wt-ok"
git -C "$project" worktree add --quiet -b demo-ok "$wt_b"
log_b="$DEMO/calls-b.log"; : > "$log_b"
: > "$DEMO/run.log"

step "fm-spawn.sh add-widget-b1 acme-widgets   # fresh worktree: nothing is prepared yet"
run_spawn add-widget-b1 "$wt_b" "$log_b" | grep -E 'task bootstrap|\[acme\]|spawned|error' || true

step "ls fm-home/state/add-widget-b1.meta   # worker task published?"
[ -e "$home/state/add-widget-b1.meta" ] \
  && printf '    add-widget-b1.meta   -> worker launched after prerequisites completed\n' \
  || printf '    MISSING (bad)\n'

step "ls wt-ok/vendor wt-ok/build   # what the project's prerequisites produced"
ls "$wt_b/vendor" "$wt_b/build" 2>&1 | sed 's/^/    /'

reuse() {  # <label> <id>
  local label=$1 id=$2
  step "fm-spawn.sh $id acme-widgets   # $label"
  rm -f "$home/state/$id.meta"
  run_spawn "$id" "$wt_b" "$log_b" | grep -E 'task bootstrap|\[acme\]' || true
}

reuse "same environment, nothing changed" add-widget-b2

printf '\n$ echo "widget-core 1.5.0" >> wt-ok/deps.lock   # lockfile moves\n'
printf 'widget-core 1.5.0\n' >> "$wt_b/deps.lock"
reuse "lockfile changed" add-widget-b3

printf '\n$ echo widgetc-4.0 > wt-ok/toolchain.version   # toolchain moves\n'
printf 'widgetc-4.0\n' > "$wt_b/toolchain.version"
reuse "toolchain version changed" add-widget-b4

printf '\n$ rm wt-ok/build/generated-client   # generated output goes missing\n'
rm -f "$wt_b/build/generated-client"
reuse "readiness check fails" add-widget-b5

reuse "environment is ready again" add-widget-b6

step "wc -l run.log   # how many times the project's prerequisites actually ran"
printf '    %s prerequisite runs across 6 spawns on the same preserved worktree\n' \
  "$(wc -l < "$DEMO/run.log" | tr -d '[:space:]')"

rule "Done"

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 4 issues found → auto-fixed (2) ✅

• ⚠️ bin/fm-task-bootstrap.sh:103 - The project-declared run is executed with no timeout and with fm-spawn's stdin (the primary session's tty) inherited. A prerequisite that stalls on the network, or one that prompts for input, blocks fm-spawn indefinitely: the only other unbounded step in this region is bounded (the for _ in $(seq 1 60) settle loop at bin/fm-spawn.sh:1322 gives up after 60s), and in batch dispatch (bin/fm-spawn.sh:408-410) each id=repo pair is re-exec'd serially, so one hung bootstrap stalls every remaining spawn in the batch. The intent requires the bootstrap to 'fail loudly'; a silent hang is neither loud nor a failure. Consider a bounded wrapper (configurable, e.g. FM_TASK_BOOTSTRAP_TIMEOUT) and running the declaration with stdin redirected from /dev/null so a prompt fails instead of blocking.
• ⚠️ bin/fm-spawn.sh:1351 - When the bootstrap fails on the tmux, zellij, or cmux backends, spawn exits at :1353 after the task window/pane was created and treehouse get already leased a worktree, but before state/$ID.meta is written at :1658. fm-teardown refuses without that file (bin/fm-teardown.sh:146 'error: no meta for task'), so the window and the leased worktree are orphaned with no supported cleanup, and a retry of the same id creates a second window. The orca path is covered (ORCA_ABORT_CLEANUP is set at :1195 and consumed by the EXIT trap) and herdr is covered (HERDR_PROJECTION_ABORT_CLEANUP is only cleared at :1700), so only these three backends leak. The gap pre-dates this change (the settle-timeout exit at :1341 has it too), but a failed project install is a routine outcome unlike the isolation guard, so it now fires on ordinary failures.
• ℹ️ bin/fm-task-bootstrap.sh:63 - [ ! -L "$declaration" ] only tests the final path component, so the deliberate 'declaration must not be a symbolic link' guard is bypassed when .firstmate itself is a symlink: $worktree_real/.firstmate/bootstrap then resolves and executes a file outside the worktree, whose contents never appear in the repo's diff. Resolving the declaration with pwd -P on its directory and requiring the result to stay under $worktree_real closes it consistently with the guard already written.
• ℹ️ bin/fm-task-bootstrap.sh:59 - The script header (:5) and docs/configuration.md both define opt-in as 'tracking' an executable .firstmate/bootstrap, but the checks at :63-68 only require a non-symlink executable regular file. Any executable that lands at that path in a preserved/pooled worktree and survives Treehouse's git clean -fd (i.e. an ignored path) is executed by the primary session before the next worker launches, with no reviewed-content guarantee. A git -C "$worktree_real" ls-files --error-unmatch .firstmate/bootstrap check would make the implementation match the documented contract.

🔧 Fix: bound and isolate task bootstrap, release aborted spawns
2 warnings still open:

• ⚠️ bin/fm-task-bootstrap.sh:103 - The timeout ladder is timeout -> gtimeout -> run unbounded with a stderr warning. macOS ships neither GNU timeout nor gtimeout (coreutils is not in the base system) and README.md advertises platform-macOS | Linux, so on a stock macOS primary session the wall-clock bound this commit exists to add simply does not apply and the authorized failure is still fully reachable: a stalled project run blocks fm-spawn forever and, via the serialized batch re-exec at bin/fm-spawn.sh:408-410, every remaining pair. Three places assert the bound unconditionally while it is absent: docs/configuration.md:201 ("every mode runs with stdin from /dev/null and under a wall-clock bound"), the header at bin/fm-task-bootstrap.sh:28, and the runtime line at :184 which still prints "bounded at 900s". The regression at tests/fm-task-bootstrap.test.sh:281-283 pass-skips when no timeout command exists, so CI stays green on exactly the platform where the invariant is missing. The earliest supported boundary already exists in this repo: bin/fm-bearings-snapshot.sh:192-202 uses the same ladder with a perl -e 'alarm ...' fallback that forks, setpgrps, sends TERM then KILL to the group, and exits 124 - matching this script's 124 contract, requiring no watchdog beyond what timeout itself is, and available on stock macOS. Adopting it also closes the secondary gap that neither the timeout nor gtimeout branch passes --kill-after, so today a declaration that ignores SIGTERM still hangs past its bound.
• ⚠️ bin/fm-spawn.sh:326 - The new abort-path guard is [ -n "${WT:-}" ] && [ "$WT" != "$PROJ_ABS" ] && [ -d "$WT" ], but its comment claims it returns "only a worktree this spawn actually leased". It does not: the settle loop at :1355-1370 assigns WT="$p" as soon as two consecutive pane reads agree on any path whose physical form differs from PROJ_ABS_REAL, and validate_spawn_worktree then exits 1 for a path that is not a git worktree root. Concrete path: treehouse get does not take effect and the pane's shell settles in $HOME (or any non-repo directory) -> the loop accepts it -> validate_spawn_worktree exits 1 -> the EXIT trap runs ( cd "$PROJ_ABS" && treehouse return --force "$HOME" ), handing an arbitrary user directory to a pool manager with --force, with all output discarded so only a generic warning surfaces. The comparison is also logical where the rest of the file is deliberately physical: PROJ_ABS_REAL exists at :869 and the comment at :1341-1345 explains that a symlinked project prefix defeats the string form. Gate the return on the worktree having passed validate_spawn_worktree (set a flag immediately after :1375), or re-derive git -C "$WT" rev-parse --show-toplevel in the trap and require it to resolve to $WT and to differ from $PROJ_ABS_REAL.

🔧 Fix: enforce bootstrap bound everywhere, return only validated worktrees
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• bash tests/fm-task-bootstrap.test.sh — all 11 targeted bootstrap/spawn regressions pass on the target commit
• Biting-regression proof: git checkout cbd5d67 -- bin/fm-spawn.sh && bash tests/fm-task-bootstrap.test.sh reproduced the stated pre-fix result verbatim, then git checkout HEAD -- bin/fm-spawn.sh restored the change (worktree verified clean)
• End-to-end manual demo demo-task-bootstrap.sh driving the real bin/fm-spawn.sh with a fake tmux/treehouse backend: fail-closed spawn, worker launch after successful bootstrap, and the full fingerprint skip/rerun lifecycle over 6 spawns on one preserved worktree
• bash tests/fm-spawn-batch.test.sh, bash tests/fm-spawn-worktree-settle.test.sh, bash tests/fm-spawn-dispatch-profile.test.sh, bash tests/fm-teardown-endpoint-safety.test.sh — spawn/teardown suites adjacent to the new endpoint abort-cleanup path
• bin/fm-task-bootstrap.sh --help renders the declaration contract the docs point users to
• Opt-out path: bin/fm-task-bootstrap.sh <repo-with-no-declaration> exits 0 with no output
• Forbidden-constraint checks over git diff cbd5d67..bd454ab: no projects/ paths, no .firstmate/bootstrap declaration added, no added git fetch, no prisma/pnpm/yarn/cargo/lockfile tokens in changed bin/ code

⚠️ **Document** - 1 info

• ℹ️ bin/fm-spawn.sh:1231 - Documented (not fixed) asymmetry: a Herdr spawn that fails before publishing task metadata — now reachable via a failed project task bootstrap — cleans up only projected panes and never returns the leased Treehouse worktree, unlike tmux/zellij/cmux (ENDPOINT_ABORT_CLEANUP) and Orca. I narrowed the docs claim and recorded the gap under Herdr's Active limits; closing the gap itself is a code change outside this documentation phase.

🔧 **Lint** - 1 issue found → auto-fixed ✅

• ⚠️ linter found issues (exit code 1)

🔧 Fix: use env for bootstrap runner override in test
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

[coderabbitai]

Warning

Review limit reached

@bramdokman, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 56 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c21a27c8-acf7-42e4-b599-cf5e9eb36dfa

📥 Commits

Reviewing files that changed from the base of the PR and between cbd5d67 and 7b10702.

📒 Files selected for processing (8)

• bin/fm-spawn.sh
• bin/fm-task-bootstrap.sh
• bin/fm-test-run.sh
• docs/architecture.md
• docs/configuration.md
• docs/herdr-backend.md
• docs/scripts.md
• tests/fm-task-bootstrap.test.sh

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Captain, this PR adds a project-opt-in, toolchain-agnostic “task bootstrap” step that runs after fm-spawn.sh has proven an isolated worktree and before any worker is launched, ensuring prerequisite failures stop spawns cleanly and deterministically (with a fingerprint-based skip for preserved environments).

Changes:

• Introduces bin/fm-task-bootstrap.sh to execute a tracked .firstmate/bootstrap declaration in bounded, noninteractive modes (fingerprint, run, optional timeout), persisting a per-worktree success fingerprint in the worktree’s Git dir.
• Updates bin/fm-spawn.sh to run the task bootstrap before worker launch and to abort-cleanup tmux/zellij/cmux endpoints + return only a validated leased Treehouse worktree when spawn fails pre-meta.
• Adds a dedicated regression suite tests/fm-task-bootstrap.test.sh, wires it into bin/fm-test-run.sh, and documents the protocol and backend implications.

Reviewed changes

Copilot reviewed 5 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
bin/fm-task-bootstrap.sh	New bounded, noninteractive runner for project-declared prerequisites with fingerprint skip/rerun and strict declaration validation.
bin/fm-spawn.sh	Runs task bootstrap before worker launch; adds abort cleanup for pre-meta failures and returns only validated leased worktrees.
tests/fm-task-bootstrap.test.sh	Regression coverage for declaration contract, bounds enforcement, fingerprint lifecycle, and fail-closed spawn behavior.
bin/fm-test-run.sh	Routes the new test and bootstrap script into the backend-dispatch test family selection.
docs/configuration.md	Documents .firstmate/bootstrap protocol, safety constraints, bound enforcement, and backend cleanup behavior.
docs/architecture.md	Notes the bootstrap step as part of the spawn lifecycle after isolated-worktree proof.
docs/scripts.md	Adds fm-task-bootstrap.sh to the scripts catalog.
docs/herdr-backend.md	Records the documented limitation that failed pre-meta Herdr spawns may leave residue requiring manual cleanup.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.