fix: contain runaway test and lab process trees

bin/fm-brief.sh

@@ -596,6 +596,12 @@ IFS= read -r -d '' SHARED_INFRA_RULE <<'EOF' || true
 EOF
 SHARED_INFRA_RULE=${SHARED_INFRA_RULE%$'\n'}
 
+# One shared string keeps the ship and scout process-budget rule identical.
+# The kernel counts the user's whole process table, so one runaway scratch tree
+# starves every lane; bin/fm-proc-budget.sh owns the budget, this is the one
+# line that reaches ad-hoc scripts no runner wraps.
+SHARED_PROC_BUDGET_RULE="9. Run any shim, lab, or measurement script through \`$(shell_quote "$FM_ROOT/bin/fm-proc-budget.sh") -- <command...>\`: a runaway process tree then stops at its own limit instead of starving every lane of fork."
+
 if [ -n "$BASE_BRANCH" ]; then
   SETUP_BASE="You are in a disposable git worktree of $REPO, at a detached HEAD on a clean copy of its base branch.
 Base branch: $BASE_BRANCH"
@@ -661,6 +667,7 @@ $CREWMATE_PAUSE_INSTRUCTIONS
    Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved [at=<epoch>]: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
 $SHARED_INFRA_RULE
 $PRIVATE_SERVICE_RULE
+$SHARED_PROC_BUDGET_RULE
 
 $WAIT_BLOCK$INBOX_SECTION
 
@@ -742,6 +749,7 @@ $ASK_USER_BLOCK
    Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append \`resolved [at=<epoch>]: {how it cleared}\` yourself (same \`[key=<slug>]\` if you opened it with one) as you resume.
 $SHARED_INFRA_RULE
 $PRIVATE_SERVICE_RULE
+$SHARED_PROC_BUDGET_RULE
 
 $WAIT_BLOCK$INBOX_SECTION
 

bin/fm-claude-stop-autoarm.sh

@@ -72,26 +72,21 @@
 #     line, which is actionable here like a wake line; its rewake banner
 #     carries every "supervision-host:" line the host printed, in order, while
 #     its wake lines keep the arm's eight-line cap. A "supervision-host stood
-#     down:" close exits 0 silently, and a host that died without a close is
-#     retried instead of being judged by the healthy-watcher predicate
-#     (docs/supervision-host.md). On a home that opted out nothing below
-#     changes.
+#     down:" close exits 0 silently; docs/supervision-host.md owns host-close
+#     retry and failed-hand-back classification. On a home that opted out
+#     nothing below changes.
 #   - Translation: while supervision is still needed and AFK remains inactive,
 #     an actionable arm close (signal:/stale:/check:/heartbeat) prints one
 #     rewake banner to stderr and exits 2, which wakes Claude even while idle
 #     ("Stop hook feedback"). The irrevocable commit point is the EXIT STATUS:
 #     the harness delivers the collected stderr only on exit 2, so an owned
 #     terminal commit decides the exit. Markerless outcomes commit with the
 #     ledger write; the failure notice additionally requires its marker write.
-#     A refused generation exits 0 silently even after printing. A close that
-#     reports no actionable reason is benign when a live identity-matched
-#     watcher still has a fresh beacon.
-#   - Failure handling: a typed failure is rechecked against the same live,
-#     fresh watcher predicate and retried a bounded number of times in this
-#     hook. Only an exhausted failure with no verified watcher emits one
-#     last-resort notice per failure episode; later consecutive failures still
-#     exit 2 to guarantee the next Stop-owned retry without repeating notice,
-#     until the synchronous guard has consumed its attended fail-open.
+#     A refused generation exits 0 silently even after printing.
+#   - Failure handling: docs/watcher-continuity.md "Claude arm failures" owns
+#     direct-arm close classification and retries; docs/supervision-host.md owns
+#     host closes. docs/turnend-guard.md owns failure-episode notices and the
+#     attended fail-open boundary.
 #
 # The epoch ledger state/.claude-autoarm-epoch records the latest claim
 # generation and outcome, and binds rewake outcomes to the session-lock pid and
@@ -449,7 +444,7 @@ while [ "$attempt" -lt "$AUTOARM_ATTEMPTS" ]; do
     # A host that died without a close may have left its cycle running with
     # no owner to deliver the close; retrying lets the next host stop what it
     # left and own a fresh cycle, which the healthy-watcher predicate cannot.
-    if [ "$HOST_RC" -gt 128 ] || [ -z "$OUT" ] || [ ! -s "$OUT" ]; then
+    if [ "$HOST_RC" -gt 128 ]; then
       [ "$attempt" -lt "$AUTOARM_ATTEMPTS" ] || break
       [ -z "$OUT" ] || rm -f "$OUT" 2>/dev/null || true
       OUT=

bin/fm-herdr-lab.sh

@@ -27,6 +27,9 @@
 # destructive call.
 # Provision records the running default session as a fleet-state tripwire and
 # teardown requires that record to be identical afterward.
+# Provision's server and the viewer launcher start through bin/fm-proc-budget.sh;
+# its header owns the inherited process-budget contract. Status, stop, and
+# teardown client calls are not wrapped, so budget setup cannot block cleanup.
 # The viewer command attaches or detaches one real foreground Herdr client on
 # an owned lab session over a fixed 40-row by 120-column pty;
 # bin/fm-herdr-lab-viewer.py owns the pty mechanics.
@@ -302,7 +305,7 @@ fm_herdr_lab_viewer_start() { # <session>
     trap 'trap - INT TERM; [ -z "${launcher_pid:-}" ] || fm_herdr_lab_cancel_viewer_launcher "$launcher_pid"; exit 130' INT
     trap 'trap - INT TERM; [ -z "${launcher_pid:-}" ] || fm_herdr_lab_cancel_viewer_launcher "$launcher_pid"; exit 143' TERM
   fi
-  nohup python3 "$launcher" "$name" "$record" >"$log" 2>&1 &
+  nohup "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-proc-budget.sh" -- python3 "$launcher" "$name" "$record" >"$log" 2>&1 &
   launcher_pid=$!
 
   waited=0
@@ -434,7 +437,7 @@ fm_herdr_lab_provision() { # <session>
   else
     fm_herdr_lab_prepare "$name" || return 1
   fi
-  fm_herdr_lab_raw "$name" server >/dev/null 2>&1 &
+  HERDR_SESSION="$name" "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-proc-budget.sh" -- herdr server --session "$name" >/dev/null 2>&1 &
   server_pid=$!
   attempt=0
   max_attempts=300

bin/fm-live-lab.sh

@@ -39,6 +39,9 @@
 #                    tmux-dir, with no user tmux config (its plugins never run
 #                    in a lab), started from an empty environment so no inherited
 #                    TMUX, Herdr, or Pi marker reaches a lab process.
+#                    The server starts through bin/fm-proc-budget.sh (whose
+#                    header owns the budget contract), so every pane inherits
+#                    its process limit.
 #                    TREEHOUSE_ROOT points into <lab-root>, so a worker's pool
 #                    never lands in ~/.treehouse, and DISABLE_AUTOUPDATER=1
 #                    keeps Claude Code from replacing the shared binary under
@@ -176,7 +179,7 @@ lab_run() {  # [NAME=VALUE...] <command...>: run in the lab's clean environment
   local -a base=()
   local line
   while IFS= read -r line; do base+=("$line"); done < <(lab_env_base)
-  env -i "${base[@]}" "$@"
+  "$SCRIPT_DIR/fm-proc-budget.sh" -- env -i "${base[@]}" "$@"
 }
 
 # window_id <name>: the tmux id of the lab window with exactly this name, or

bin/fm-proc-budget.sh

@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# fm-proc-budget.sh - run a command tree under a per-tree process budget.
+#
+# Usage:
+#   fm-proc-budget.sh [<extra>] -- <command...>   run <command...> with a budget
+#
+# The kernel compares RLIMIT_NPROC with the user's WHOLE count, not with one
+# process tree: processes on Darwin, tasks (threads included) on Linux.
+# A runaway tree therefore starves every other process of the same user until
+# the cap is full, and a bash 3.2 script exits 128 at its first failed fork.
+# This wrapper reads the current count and sets the limit (soft and hard) to
+# count + <extra>, then execs the command, so only the tree below it carries the
+# lowered limit and the rest of the user keeps its headroom. <extra> is a
+# positive decimal integer without leading zeros; omit it for 1500.
+#
+# The budget only tightens. A limit the command already inherits that is lower
+# than count + <extra> stays as it is, and the hard limit is lowered with the
+# soft one so the tree cannot raise it again; a nested wrapper therefore never
+# widens an outer budget.
+#
+# Only commands started through the wrapper are protected. Route every shim,
+# lab, and measurement script through it. The headers of bin/fm-test-run.sh,
+# bin/fm-live-lab.sh, and bin/fm-herdr-lab.sh own their launch integration.
+#
+# The budget is headroom over the count at the moment the command starts, so a
+# tree that outlives a large rise in the rest of the user's processes can meet
+# the limit without having grown.
+#
+# Exit status: the command's own, 2 for a malformed call, and 125 when no budget
+# can be set (an unreadable process count, or a limit the host refuses). The
+# command never runs unbudgeted behind the caller's back in those two cases.
+#
+# tests/fm-proc-budget.test.sh is the regression entry point for a leashed
+# recursive shim and an unbudgeted probe that must keep forking alongside it.
+set -u
+
+fm_proc_budget_error() {
+  echo "fm-proc-budget: $*" >&2
+}
+
+fm_proc_budget_usage() {
+  fm_proc_budget_error "usage: fm-proc-budget.sh [<extra>] -- <command...>"
+}
+
+fm_proc_budget_count() {
+  local uid count
+  uid=$(id -u) || return 1
+  set -o pipefail
+  case "$(uname -s)" in
+    Linux) count=$(ps -L -U "$uid" -o lwp= | wc -l) || return 1 ;;
+    *) count=$(ps -U "$uid" -o pid= | wc -l) || return 1 ;;
+  esac
+  count=$(printf '%s' "$count" | tr -d '[:space:]')
+  case "$count" in
+    ''|*[!0-9]*|0) return 1 ;;
+  esac
+  printf '%s\n' "$count"
+}
+
+# fm_proc_budget_apply <extra>: lower this process's limit to count + <extra>
+# unless the inherited limit is already lower. Prints nothing on success.
+fm_proc_budget_apply() {
+  local extra=$1 count current hard limit
+  count=$(fm_proc_budget_count) || {
+    fm_proc_budget_error "cannot read the process count, so no budget can be set"
+    return 125
+  }
+  limit=$((count + extra))
+  current=$(ulimit -S -u) || {
+    fm_proc_budget_error "cannot read the current process limit"
+    return 125
+  }
+  hard=$(ulimit -H -u) || {
+    fm_proc_budget_error "cannot read the hard process limit"
+    return 125
+  }
+  case "$current" in
+    ''|*[!0-9]*) ;;  # unlimited: the budget is the only bound
+    *) [ "$limit" -lt "$current" ] || limit=$current ;;
+  esac
+  case "$hard" in
+    ''|*[!0-9]*) ;;
+    *) [ "$limit" -lt "$hard" ] || limit=$hard ;;
+  esac
+  ulimit -u "$limit" 2>/dev/null || {
+    fm_proc_budget_error "this host refused a process limit of $limit (count $count + $extra)"
+    return 125
+  }
+}
+
+extra=1500
+if [ "${1:-}" != -- ] && [ "$#" -gt 0 ]; then
+  extra=$1
+  shift
+fi
+if [ "${1:-}" != -- ]; then
+  fm_proc_budget_usage
+  exit 2
+fi
+shift
+[ "$#" -gt 0 ] || { fm_proc_budget_usage; exit 2; }
+case "$extra" in
+  ''|*[!0-9]*|0|0[0-9]*)
+    fm_proc_budget_error "extra must be a positive decimal integer, got '$extra'"
+    exit 2
+    ;;
+esac
+
+fm_proc_budget_apply "$extra" || exit 125
+exec "$@"

bin/fm-test-run.sh

@@ -129,11 +129,17 @@
 # An unnested runner takes one pass per executed script from the host-wide pool
 # (bin/fm-cpu-pass.sh; docs/cpu-pass-pool.md owns the protocol), outside its
 # per-script bound, so participating test bursts across worktrees take turns.
-# A runner already inside a pass (FM_CPU_PASS_HELD set) runs directly with at
-# most that many concurrent scripts, reporting a reduced --jobs on stderr.
+# A runner already inside a pass (FM_CPU_PASS_HELD set) takes no additional pass
+# and runs at most that many concurrent scripts, reporting a reduced --jobs on stderr.
 # The marker must be a nonnegative decimal integer or execution exits 125;
-# 0 denotes degraded work and imposes no budget.
-# Without python3 or the pool tool beside it, scripts run directly.
+# 0 denotes degraded work with no CPU-pass concurrency limit.
+# Without python3 or the pool tool beside it, scripts run without a CPU pass.
+#
+# When the sibling bin/fm-proc-budget.sh is present and executable, every
+# executed script runs through it; its header owns the process-budget contract.
+# The budget is taken when the script starts, after any pass wait. If the budget
+# cannot be set, that script fails with wrapper exit 125 rather than running
+# unbudgeted. Without the executable wrapper, scripts run without this budget.
 # With a usable pool, --jobs above its size still starts that many workers,
 # but only pool-size scripts run at once.
 #
@@ -849,6 +855,7 @@ tests/fm-pr-check-security.test.sh 300675
 tests/fm-pr-reviewers.test.sh 273
 tests/fm-pr-state-live-e2e.test.sh 47
 tests/fm-pr-state.test.sh 531
+tests/fm-proc-budget.test.sh 8000
 tests/fm-procevent-quota.test.sh 2459
 tests/fm-procevent-when.test.sh 25674
 tests/fm-procevent.test.sh 370820
@@ -2467,6 +2474,10 @@ elif ! command -v python3 >/dev/null 2>&1; then
   CPU_PASS_ACTIVE=0
   log "running without CPU passes: python3 not found"
 fi
+PROC_BUDGET_ACTIVE=1
+if [ ! -x "$ROOT/bin/fm-proc-budget.sh" ]; then
+  PROC_BUDGET_ACTIVE=0
+fi
 if [ "$JOBS" -gt 1 ] || [ "$MODE" = changed ] || [ "$MODE" = scripts ]; then
   SELECTION_DESC="${SELECTION_DESC};jobs=$JOBS"
 fi
@@ -2647,6 +2658,11 @@ run_script_bounded() {  # <script> <out> <stream> <id>
   else
     cmd=(bash "$script")
   fi
+  if [ "$PROC_BUDGET_ACTIVE" -eq 1 ]; then
+    # Innermost, so the budget is taken when the script starts, not before a
+    # pass wait or after another script has already grown the process count.
+    cmd=("$ROOT/bin/fm-proc-budget.sh" -- "${cmd[@]}")
+  fi
   if [ "$PER_SCRIPT_TIMEOUT_SECS" -gt 0 ]; then
     # The bound runs inside the pass holder so the pass wait stays outside it.
     # shellcheck disable=SC2016

docs/architecture.md

@@ -185,7 +185,7 @@ The arm layer records one bounded lifecycle row per observed cycle in `state/.wa
 Pi, omp, and OpenCode verify session-lock ownership and launch one singleton successor from their child-close handlers before delivering an actionable wake prompt, with bounded exponential retry for failed restoration.
 Pi additionally retains an established predecessor across ordinary same-process session shutdown until the replacement generation commits its tracked arm, and its active-versus-handoff generation marker prevents an absent replacement extension from satisfying the fresh-beacon handoff tolerance.
 Claude's `bin/fm-claude-stop-autoarm.sh` hook fires on every Stop and, when the home is eligible and still needs supervision, claims one home-scoped cycle, foregrounds the arm wrapper (or the [supervision host](supervision-host.md), which runs by default on Claude), and translates actionable closes into exit-2 rewakes.
-It suppresses failed-looking closes when the same identity-matched watcher is healthy, retries genuine failures within a bound, and coordinates exhausted failure episodes with the Claude turn-end guard as documented in [`turnend-guard.md`](turnend-guard.md).
+Close classification and retries are owned by [`watcher-continuity.md`](watcher-continuity.md#claude-arm-failures) for direct arms and [`supervision-host.md`](supervision-host.md#failure-direction) for hosts; [`turnend-guard.md`](turnend-guard.md#failure-progression-and-block-budget) owns failure-episode coordination.
 [`watcher-continuity.md`](watcher-continuity.md) owns Claude's residual active-turn coverage and watcher-status command-gating boundary.
 Cursor's `bin/fm-turnend-guard-cursor.sh` hook is the same between-turns shape in one synchronous step: it parks the awaited `stop` hook on the arm wrapper and translates an actionable close into one `followup_message`, with a generation baton that makes an older park still running after the next `stop` claim stand down instead of leaking a stale duplicate wake.
 The existing turn-end guard remains the final backstop for every harness-engine protocol, with pi-signed sharing Pi's protocol, omp's blocking `session_stop` hook compelling one continuation per turn, the `--claude` mode cooperating with the auto-arm claim, and Cursor's `--cursor` mode rendering a block as one bounded follow-up because its `stop` step cannot be blocked.

docs/cpu-pass-pool.md

@@ -72,8 +72,8 @@ Phase 1 completes only when the Vernant participant lands as a separate Vernant-
 
 `bin/fm-load-report.sh` records host load and reads pipeline agent durations; its [engine header](../bin/fm-load-report.py) owns the sample format and command behavior, while this recipe owns the measurement window and acceptance criteria.
 
-1. Only once Phase 1 is complete, note the time with `date +%s` and start one recorder per host: `nohup bin/fm-load-report.sh watch --interval 60 >/dev/null 2>&1 &`.
-2. After 24 to 48 hours of normal fleet work, run `bin/fm-load-report.sh report --since <that epoch>`.
+1. Only once Phase 1 is complete, note the time with `date +%s` and start one recorder per host: `nohup bin/fm-proc-budget.sh -- bin/fm-load-report.sh watch --interval 60 >/dev/null 2>&1 &`.
+2. After 24 to 48 hours of normal fleet work, run `bin/fm-proc-budget.sh -- bin/fm-load-report.sh report --since <that epoch>`.
 3. Read its two verdicts: `load_within_2x_cpus` (1-minute load p95 at or under twice the CPU count) and `converged_within_2_fix_rounds` (the fixed first ten pipeline runs that reached review and were created after the recorded cutover epoch, ordered by creation time then run id, completed successfully with at most two review-fix rounds and none hit a timeout).
    Failed and cancelled cohort members count as not converged.
    The convergence verdict is pending (`null` in JSON, `pending` in text) until ten review-reaching runs exist and every pending or running run that sorts at or before the tenth member is terminal, including runs that have not reached review yet.

docs/scripts.md

@@ -55,6 +55,7 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md
 | `fm-test-isolation-proof.sh` | Concurrent isolation harness and portable candidate set owner |
 | `fm-cpu-pass.sh`         | Reserve an exact positive worker count, no larger than the host CPU count, while a CPU-heavy burst runs ([protocol](cpu-pass-pool.md)) |
 | `fm-cpu-pass.py`         | The engine behind `fm-cpu-pass.sh`                                                   |
+| [`fm-proc-budget.sh`](../bin/fm-proc-budget.sh) | Bound a command tree's process growth; its header owns the budget contract |
 | `fm-load-report.sh`      | Record host load and judge it with pipeline agent durations ([recipe](cpu-pass-pool.md#judging-the-pool)) |
 | `fm-load-report.py`      | The engine behind `fm-load-report.sh`                                                |
 | `fm-ensure-agents-md.sh` | Manually initialize project agent-memory files (see the helper's header and help) |

docs/supervision-host.md

@@ -270,6 +270,8 @@ So it never stops the owner's host or watcher or releases its leases.
 ### A host that dies without a close
 
 The host's owner retries it.
+Claude's Stop hook retries a non-actionable host close as a crash only when its exit status is above 128, not merely because its output is empty.
+An explicit failed hand-back instead reaches the failure-notification path immediately, even if no readiness line was printed or its successor watcher remains healthy.
 Grok's model and Codex's checkpoint see it as a failed cycle and start the next one.
 Before it arms, the next host does two things: