From ac0811c4c820d82009d7a3b8f09555cbf38340c8 Mon Sep 17 00:00:00 2001 From: Tiago Date: Wed, 7 Oct 2026 11:32:21 -0300 Subject: [PATCH 01/12] feat(bin): defer spawns beyond a declared per-project capacity, opt-in (#5343) * refactor(bin): share the local Firstmate home walk from the wake library Teardown's walk over the root home and its registered local secondmate homes moves into bin/fm-wake-lib.sh as fm_local_firstmate_state_dirs, next to fm_firstmate_root_home, so a second consumer can count task records across this machine's homes without a copy. Teardown keeps its exact refusal wording through a thin wrapper. * feat(bin): defer spawns beyond a project's declared machine capacity A project whose machine-local resource only serves a few workers at once had no way to tell Firstmate so: every queued item was launched, and the surplus workers spent full-context turns retrying the resource. config/project-capacity in the root home now declares how many workers each named project admits at once on this machine. bin/fm-spawn.sh counts the ship and scout records on the same project origin across the root and its local secondmate homes, skipping ones whose ready PR is recorded, while holding the shared project lock through publication. A spawn with every place held exits 75 before any brief render, endpoint, worktree, record, or backlog move, so the item stays queued; batches report it as deferred. Undeclared projects keep today's uncapped dispatch, and an unreadable declaration refuses rather than guessing the limit. Refs #4237 * no-mistakes(review): Document that capacity matches the clone directory name * no-mistakes(document): Rewrap stale fm-wake-lib root-home doc comment * no-mistakes(review): Dedupe local state dirs by identity to avoid double-counting * no-mistakes(document): Rewrap fm_local_firstmate_state_dirs error doc comment * no-mistakes(ci): I fixed all four Greptile findings. All 14 tests in tests/fm-project-capacity.test.sh pass, and shellcheck at warning level is clean on the changed files. Each new test failed against the old code and passes now. - **ci-1 (spaced names):** a declaration line must give a name its capacity whenever the name is a valid clone directory name. `fm_project_capacity_lookup` now trims each line, skips blank lines and lines whose first non-blank character is `#`, and takes the last field as the capacity. Everything before that field is the name, so it may contain spaces. The old error cases still refuse: a single field is rejected, and trailing text leaves a last field that is not an integer. The library header and docs/configuration.md now say a name starting with `#` cannot be declared. New test `test_spaced_project_name_is_declared` declares `my heavy project 1` next to an indented comment line and gets a deferral. - **ci-2 (unreadable records):** the holder count must never silently leave out a holder. `fm_project_capacity_occupants` now refuses when a local home's state directory exists but cannot be read or listed, or when a `.meta` file cannot be read. The error names the path, and `fm-spawn.sh` shows it in its existing refusal message. New test `test_unreadable_holders_refuse_admission` covers an unreadable record in the root home and an unreadable state directory in a registered local secondmate home, then checks that the spawn is admitted once both are readable. The test is skipped when run as root. - **ci-3 (Orca lock):** any spawn that can become a holder for a capped project must take that project's lock. The lookup now also reports whether the declaration caps any project at all, and an Orca spawn takes the per-origin lock whenever it does. This covers every capped same-origin clone. It also covers some cases where no same-origin clone is capped, because a spawn cannot find clones under other directory names without searching for them. With no declaration file, Orca still skips the lock. The comments in the library and in the `fm-spawn.sh` header are updated. The Orca test now clones the origin as `project-2`, which has no declaration, and checks that its Orca spawn refuses while the lock is held and publishes no record. - **ci-4 (worktrees):** `assert_nothing_created` now also compares the project's `git worktree list` from before and after a deferred spawn. Both tests that call it take that snapshot first. Files changed: bin/fm-project-capacity-lib.sh, bin/fm-spawn.sh, docs/configuration.md, tests/fm-project-capacity.test.sh * fix(bin): declare capacity for a project name that begins with # A clone directory whose name begins with # was skipped as a comment, so that project stayed uncapped. A line is a declaration when the # is written against the rest of the name and the line ends with a capacity; a # followed by whitespace stays a comment. * no-mistakes(document): Rewrap project-capacity library header comment * no-mistakes(ci): Lint 2 fails because this PR's code pushes ShellCheck past its memory cap. ShellCheck ran out of memory analyzing bin/fm-teardown.sh in CI (reason=memory, rc=251, peak about 8.39 GB). On current main the same file passes at about 7.29 GB. **Cause:** the new `fm_local_firstmate_state_dirs` function in bin/fm-wake-lib.sh had a conditional `. fm-secondmate-registry-lib.sh` with a `# shellcheck source=` directive inside the function. ShellCheck followed that source again, inside a function scope, wherever fm-wake-lib.sh is sourced, and bin/fm-teardown.sh is the heaviest root that sources it. Measured locally with `shellcheck --norc --external-sources bin/fm-teardown.sh`: - current main (fd325b1b): 7.29 GB - main merged with this PR: 7.86 GB - the same merge without the in-function source: 7.27 GB **Rule this restores:** this change must not make any lint root heavier than it is on main. That function holds the only new nested source in the change. **Fix:** I removed the in-function source, which no caller needs. Both callers already load the registry library at top level before calling the function: - bin/fm-teardown.sh sources it directly. - bin/fm-spawn.sh, the only user of bin/fm-project-capacity-lib.sh, gets it through bin/fm-ff-lib.sh. I also documented the requirement in the function's comment and in the "Requires" note in bin/fm-project-capacity-lib.sh. No behaviour changes. **Verification:** - ShellCheck on head: bin/fm-teardown.sh peaks at 7.12 GB and bin/fm-spawn.sh at 6.68 GB, both with rc=0. bin/fm-wake-lib.sh and bin/fm-project-capacity-lib.sh lint clean. - tests/fm-project-capacity.test.sh, tests/fm-teardown.test.sh (102 ok) and tests/fm-teardown-endpoint-safety.test.sh all pass. Files changed: bin/fm-wake-lib.sh, bin/fm-project-capacity-lib.sh * fix(bin): release the Herdr session lock when reclaim finishes A concurrent resume in another home waits five seconds for that lock. Reclaim is the last presentation change on the recovery path, so holding the lock through the launch tail made the waiter time out. The contributions arm check also freezes its one-second clock, the same way the budget tests do, because an unfrozen clock can tick past before the first forge read. * no-mistakes(review): Keep Herdr session lock through launch handoff after reclaim * no-mistakes(review): Skip the spawning task's own record in capacity count * no-mistakes(review): Restore release test comment above its test * docs: scope PR-ready re-evaluation to a declared project capacity A ready pull request frees a place only when that project declares capacity, so the always-loaded backlog contract should re-evaluate on that handoff only in that case. --- .../skills/operational-home-layout/SKILL.md | 1 + AGENTS.md | 4 +- bin/fm-pr-check.sh | 2 + bin/fm-project-capacity-lib.sh | 229 +++++++ bin/fm-spawn.sh | 76 ++- bin/fm-teardown.sh | 47 +- bin/fm-test-run.sh | 1 + bin/fm-wake-lib.sh | 75 +- docs/configuration.md | 23 + tests/fm-contributions.test.sh | 3 + tests/fm-project-capacity.test.sh | 642 ++++++++++++++++++ 11 files changed, 1043 insertions(+), 60 deletions(-) create mode 100644 bin/fm-project-capacity-lib.sh create mode 100755 tests/fm-project-capacity.test.sh diff --git a/.agents/skills/operational-home-layout/SKILL.md b/.agents/skills/operational-home-layout/SKILL.md index 2786265f63b..49c728d93af 100644 --- a/.agents/skills/operational-home-layout/SKILL.md +++ b/.agents/skills/operational-home-layout/SKILL.md @@ -24,6 +24,7 @@ config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "de config/claude-permission-mode optional one-token permission posture for every Claude worker launch: absent or "bypass" keeps --dangerously-skip-permissions, "auto" launches with --permission-mode auto; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Claude permission mode" config/claude-account config/pi-account optional per-home worker account pin for Claude and Pi launches; LOCAL, gitignored, not inherited; absent keeps today's ambient account; present refuses a launch unless the pinned account resolves and is signed in (section 4 owns the refusal rule); see docs/configuration.md "Worker account pin" config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes +config/project-capacity optional per-machine count of workers each named project admits at once, read from the root home by every local home; LOCAL, gitignored; see docs/configuration.md "Project capacity" config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = the configured tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), herdr has its own required CI lane (docs/herdr-backend.md), while zellij, orca, and cmux remain experimental with no dedicated real-backend CI lane (docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning diff --git a/AGENTS.md b/AGENTS.md index fc3fbb67cc7..90be0c1d908 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,6 @@ Tracked files hold shared instructions and tooling; `data/` holds durable privat Load `operational-home-layout` when locating, interpreting, or changing Firstmate home, config, data, state, project, or generated runtime paths. - A `state/.status` line is a wake event, not current-state truth; `bin/fm-crew-state.sh` owns current-state reconciliation. Treat `data/captain.md` as the domain-local record of captain preferences, optional `data/captain-shared.md` as the main-authoritative shared captain-preference file for secondmate inheritance, and `data/learnings.md` as curated home-local knowledge, regardless of harness memory. @@ -195,6 +194,7 @@ An unregistered project or absent registry resolves to `no-mistakes` with yolo o Record the resulting mode, `yolo` merge posture, and the one-line reason for any deviation in the backlog item note. Treat file or subsystem overlap as a risk signal rather than an automatic reason to wait, and dispatch isolated work immediately with no concurrency cap when each change can be independently implemented and validated and the selected delivery path can reconcile ordinary rebases or conflicts. +A project's declared machine capacity (`config/project-capacity`) still bounds that dispatch: a spawn beyond it exits 75 without launching, and its item stays queued rather than blocked. Serialize only for a true semantic dependency, shared mutable external state, incompatible concurrent migration, or another concrete condition that makes independent progress or reconciliation unsafe; same-file editing alone is insufficient, and genuine blockers remain durable. Write the task-specific brief under section 11 before spawning. Fill the task subsections according to section 11. @@ -366,7 +366,7 @@ A decision is simply a task held for the captain: create the task with `bin/fm-t When a main-side thread such as a pending captain decision or relay reminder is worth durable tracking, file it as its own work item and hold it through that wrapper. Captain calls discovered by investigations or visual reviews follow `captain-hold-lifecycle`, which owns their completion gate and recorded-answer rules. When the automatic transition gate applies, dispatch and completion move the item themselves - `bin/fm-spawn.sh` and `bin/fm-teardown.sh` own those transitions and refuse rather than report success without them - so what remains yours is filing the item before dispatch, recording decisions, and keeping notes current; `docs/configuration.md` owns gate applicability and the manual-backend exception. -Re-evaluate queued work after every teardown and heartbeat, dispatching items only when dependencies and time gates have cleared. +Re-evaluate queued work after every teardown and heartbeat, and also after a recorded PR-ready handoff when `config/project-capacity` caps that project, dispatching items only when dependencies, time gates, and project capacity have cleared. `.tasks.toml`, `docs/configuration.md`, and current `tasks-axi --help` own the backlog schema, compatibility, retention, and routine command syntax. Use compatible `tasks-axi` when the configured backend selects it, always through `bin/fm-tasks-axi.sh` so the call reaches this home's backlog from any directory, and the documented manual path otherwise; keep only the configured recent Done entries. diff --git a/bin/fm-pr-check.sh b/bin/fm-pr-check.sh index 4091bcce5be..977225c4357 100755 --- a/bin/fm-pr-check.sh +++ b/bin/fm-pr-check.sh @@ -17,6 +17,8 @@ # draft state does not refuse, matching how the head read below is optional. # bin/fm-pr-merge.sh records through this script with FM_PR_CHECK_MERGE=1 and # skips this refusal, because its own merge-time draft refusal is authoritative. +# The recorded pr= also frees the task's place in a declared project capacity +# (bin/fm-project-capacity-lib.sh). # Usage: fm-pr-check.sh set -eu diff --git a/bin/fm-project-capacity-lib.sh b/bin/fm-project-capacity-lib.sh new file mode 100644 index 00000000000..a184c3d28a9 --- /dev/null +++ b/bin/fm-project-capacity-lib.sh @@ -0,0 +1,229 @@ +#!/usr/bin/env bash +# fm-project-capacity-lib.sh - how many workers a project admits at once on this +# machine, and whether a fresh worker spawn still fits. +# +# A project can depend on a machine-local resource that only a few workers can +# use at the same time: a heavy test suite, a local editor stack, a device. +# Firstmate cannot see which part of a worker's life touches that resource, so +# the captain declares how many workers the project admits on this machine, and +# bin/fm-spawn.sh defers a fresh worker beyond that number instead of launching +# it only to spend full-context turns retrying the resource. A deferred task +# keeps its queued backlog item and is dispatched again when a place frees. +# Without a declaration nothing changes and dispatch stays uncapped +# (AGENTS.md section 7). +# +# This file is the single owner of the declaration format, of what holds a +# place, and of the admission verdict. docs/configuration.md "Project capacity" +# is the operator reference, and bin/fm-spawn.sh owns where the check runs. +# +# Declaration: config/project-capacity in the local root Firstmate home (the home +# bin/fm-wake-lib.sh's fm_firstmate_root_home resolves), so every home on this +# machine reads the same number for the same machine's resources. One line per +# project: +# +# is the project's registered name, which is the basename of its +# clone directory, and is a positive integer of at most six digits. +# The capacity is the last whitespace-separated field, so the name before it may +# contain spaces. Blank lines are ignored. A line is a comment when it is `#`, +# when `#` is followed by whitespace, or when it starts with `#` and its last +# field is not an integer. A project whose name begins with `#` is declared by +# writing that `#` immediately against the rest of the name and ending the line +# with the capacity, for example `#repo 2`. A name that is `#`, or that begins +# with `#` followed by a space, is the same spelling as a comment and cannot be +# declared. Any other shape, a project named twice, or an unreadable file makes +# the whole declaration unreadable, and bin/fm-spawn.sh then refuses every fresh +# ship or scout spawn from this machine's homes rather than guessing which limit +# was meant. +# +# Occupancy: a place is held by every task record, in any local Firstmate home on +# this machine (fm_local_firstmate_state_dirs in bin/fm-wake-lib.sh), that +# - is not a secondmate, which is a persistent home rather than a worker, +# - names the same project identity, meaning its project resolves to the same +# shared project lock path (fm_treehouse_project_lock_path), which is keyed +# by the project's resolved origin, so workers in any clone of that origin +# are counted, and +# - has no recorded PR handoff: the pr= line bin/fm-pr-check.sh records when a +# worker's PR is ready, after which the worker waits on review or merge and +# no longer uses local resources. +# A place therefore frees when a PR-based ship records its ready PR, or when any +# task is cleaned up and its record removed. A local-only ship and a scout have +# no recorded handoff and hold their place until cleanup. A worker that is +# steered back into work after its PR handoff is not counted again. +# The declaration is matched by the spawning clone's directory name, so clones +# of one origin share the cap only when they use that same directory name. A +# clone of that origin under a different directory name finds no declaration +# and is not capped, though its workers still count as holders for a +# same-origin clone that is capped. +# A record whose project directory no longer exists cannot be matched and holds +# no place. Remote homes are never walked, because their workers run on another +# machine. +# +# Race safety: bin/fm-spawn.sh evaluates admission while holding the shared +# project lock and keeps holding it until the new task record is published, so +# two concurrent spawns for one project can never both publish from the same +# count. A spawn on an uncapped project can still publish a holder for a capped +# same-origin clone, so every backend takes that lock whenever the declaration +# caps any project; Orca, which otherwise never takes it, is included. Freeing a +# place needs no lock, because removing a record or adding pr= only ever lowers +# the count. +# +# Requires bin/fm-wake-lib.sh (root home, local homes, project lock path), +# bin/fm-secondmate-registry-lib.sh (which the local-homes walk reads), and +# bin/fm-backend.sh (fm_meta_get) to be sourced first. No side effects on source. + +# Exit status of a spawn deferred because the project is at capacity: the +# sysexits "temporary failure" code, so a caller can tell a deferral that leaves +# the task queued from an ordinary failure. +# shellcheck disable=SC2034 # read by bin/fm-spawn.sh after sourcing. +FM_PROJECT_CAPACITY_DEFER_EXIT=75 + +# The config directory holding this machine's declaration: the spawning home's +# own when that home is the local root (so an override of it +# applies), otherwise the root home's config/. +fm_project_capacity_config_dir() { # + local home=$1 config=$2 root home_real + root=$(fm_firstmate_root_home "$home") || return 1 + home_real=$(CDPATH='' cd -- "$home" 2>/dev/null && pwd -P) || return 1 + if [ "$root" = "$home_real" ]; then + printf '%s\n' "$config" + else + printf '%s/config\n' "$root" + fi +} + +# Read the declared capacity for one project. +# Sets FM_PROJECT_CAPACITY_FILE to the declaration path, FM_PROJECT_CAPACITY +# to the project's capacity, or to empty when the project declares none, and +# FM_PROJECT_CAPACITY_ANY to 1 when the declaration caps any project at all. +# Returns 1 with FM_PROJECT_CAPACITY_ERROR when the declaration is unreadable. +fm_project_capacity_lookup() { # + local name=$2 line lineno=0 pname pcap seen='|' + FM_PROJECT_CAPACITY_FILE="$1/project-capacity" + FM_PROJECT_CAPACITY= + FM_PROJECT_CAPACITY_ANY= + FM_PROJECT_CAPACITY_ERROR= + if [ ! -e "$FM_PROJECT_CAPACITY_FILE" ] && [ ! -L "$FM_PROJECT_CAPACITY_FILE" ]; then + return 0 + fi + if [ ! -f "$FM_PROJECT_CAPACITY_FILE" ] || [ ! -r "$FM_PROJECT_CAPACITY_FILE" ]; then + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE is not a readable regular file" + return 1 + fi + while IFS= read -r line || [ -n "$line" ]; do + lineno=$((lineno + 1)) + line=${line%$'\r'} + line=${line#"${line%%[![:space:]]*}"} + line=${line%"${line##*[![:space:]]}"} + # '#' followed by whitespace is always a comment, including one that ends + # with a number. A line that begins with '#' glued to the rest of a name is + # a declaration only when its last field is an integer; any other such line + # stays a comment, so a note does not refuse every spawn. + case "$line" in + '' | '#' | '#'[[:space:]]*) continue ;; + '#'*) + case "${line##*[[:space:]]}" in + *[!0-9]*) continue ;; + esac + ;; + esac + # The capacity is the last field, so the name before it may hold spaces. + pcap=${line##*[[:space:]]} + pname=${line%"$pcap"} + pname=${pname%"${pname##*[![:space:]]}"} + if [ -z "$pname" ]; then + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno is not ' '" + FM_PROJECT_CAPACITY= + return 1 + fi + case "$pcap" in + '' | *[!0-9]* | 0*) + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno gives $pname a capacity that is not a positive integer" + FM_PROJECT_CAPACITY= + return 1 + ;; + esac + if [ "${#pcap}" -gt 6 ]; then + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno gives $pname a capacity longer than six digits" + FM_PROJECT_CAPACITY= + return 1 + fi + case "$seen" in + *"|$pname|"*) + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno names $pname a second time" + FM_PROJECT_CAPACITY= + return 1 + ;; + esac + seen="$seen$pname|" + [ "$pname" != "$name" ] || FM_PROJECT_CAPACITY=$pcap + done < "$FM_PROJECT_CAPACITY_FILE" + [ "$seen" = '|' ] || FM_PROJECT_CAPACITY_ANY=1 + return 0 +} + +# Count the task records holding a place in one project's capacity. +# is fm_treehouse_project_lock_path for the project being +# admitted, and is that project's own directory, which matches +# without recomputing its identity. The local homes come from +# fm_local_firstmate_state_dirs . is the task being +# admitted; its own record in is the one this spawn replaces, so +# it is not counted. +# Sets FM_PROJECT_CAPACITY_OCCUPANTS to the count and +# FM_PROJECT_CAPACITY_OCCUPANT_IDS to a comma-separated list of the holders, +# each outside qualified with its home. Returns 1 with +# FM_PROJECT_CAPACITY_ERROR when the local homes cannot be enumerated, or when +# a state directory or task record in them cannot be read, since skipping it +# could undercount the holders. +fm_project_capacity_occupants() { # + local want=$1 own=$2 first=$3 self=$4 state meta kind project lock id label i + local -a cache_dirs cache_locks + FM_PROJECT_CAPACITY_OCCUPANTS=0 + FM_PROJECT_CAPACITY_OCCUPANT_IDS= + FM_PROJECT_CAPACITY_ERROR= + fm_local_firstmate_state_dirs "$first" || { + FM_PROJECT_CAPACITY_ERROR=$FM_LOCAL_FIRSTMATE_ERROR + return 1 + } + cache_dirs=("$own") + cache_locks=("$want") + for state in "${FM_LOCAL_FIRSTMATE_STATES[@]}"; do + if [ -e "$state" ] && { [ ! -d "$state" ] || [ ! -r "$state" ] || [ ! -x "$state" ]; }; then + FM_PROJECT_CAPACITY_ERROR="local Firstmate state directory $state cannot be read" + return 1 + fi + for meta in "$state"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + [ "$meta" != "$first/$self.meta" ] || continue + [ -r "$meta" ] || { + FM_PROJECT_CAPACITY_ERROR="task record $meta cannot be read" + return 1 + } + kind=$(fm_meta_get "$meta" kind) + [ "$kind" != secondmate ] || continue + [ -z "$(fm_meta_get "$meta" pr)" ] || continue + project=$(fm_meta_get "$meta" project) + [ -n "$project" ] || continue + lock= + i=0 + while [ "$i" -lt "${#cache_dirs[@]}" ]; do + if [ "${cache_dirs[$i]}" = "$project" ]; then + lock=${cache_locks[$i]} + break + fi + i=$((i + 1)) + done + if [ "$i" -ge "${#cache_dirs[@]}" ]; then + lock=$(fm_treehouse_project_lock_path "$project" 2>/dev/null) || lock= + cache_dirs+=("$project") + cache_locks+=("$lock") + fi + [ -n "$lock" ] && [ "$lock" = "$want" ] || continue + id=$(basename "$meta" .meta) + label=$id + [ "$state" = "$first" ] || label="$id in $(dirname "$state")" + FM_PROJECT_CAPACITY_OCCUPANTS=$((FM_PROJECT_CAPACITY_OCCUPANTS + 1)) + FM_PROJECT_CAPACITY_OCCUPANT_IDS="${FM_PROJECT_CAPACITY_OCCUPANT_IDS:+$FM_PROJECT_CAPACITY_OCCUPANT_IDS, }$label" + done + done + return 0 +} diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index d2feb0b65fd..445d9e13ca0 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -182,6 +182,20 @@ # fm_firstmate_root_home resolves, so a home seeded from another machine anchors # that lock itself rather than failing to resolve one; # contention refuses rather than waits. +# Project capacity: when this machine declares how many workers a project +# admits at once (config/project-capacity; bin/fm-project-capacity-lib.sh owns +# the declaration, what holds a place, and the race argument), a fresh ship or +# scout spawn counts the places already held while holding that same +# project-identity lock - taken on every backend whenever the declaration caps +# any project, Orca included, because an uncapped clone's worker still holds a +# place for a capped clone of the same origin - and holds it through metadata +# publication. A spawn that finds +# every place held prints one `deferred:` line and exits 75 before any brief +# render, endpoint, worktree, record, or backlog move exists, so the task stays +# exactly as queued as it was; an unreadable declaration refuses with exit 1. +# A batch reports such a pair as `batch: DEFERRED` and exits 75 when nothing +# else failed. A relaunch and a --secondmate spawn are never counted against +# capacity. # With no harness arg, a crewmate/scout spawn resolves the CREW harness only when # config/crew-dispatch.json is absent. When that file exists, crewmate/scout # spawns require an explicit harness so firstmate cannot silently skip dispatch @@ -665,6 +679,8 @@ fm_backlog_directory_present "$STATE" "state directory" || { . "$SCRIPT_DIR/fm-remote-readiness-lib.sh" # shellcheck source=bin/fm-timeout-lib.sh . "$SCRIPT_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-project-capacity-lib.sh +. "$SCRIPT_DIR/fm-project-capacity-lib.sh" # shellcheck source=bin/fm-worker-account-lib.sh . "$SCRIPT_DIR/fm-worker-account-lib.sh" # Fail closed before any fleet mutation: a no-mistakes gate agent must never spawn @@ -1554,16 +1570,17 @@ if [ "${#POS[@]}" -gt 0 ] && [ "${POS[0]}" != "$idpart" ] && case "$idpart" in * echo "error: batch dispatch does not support --secondmate; spawn each secondmate explicitly" >&2 rc=2 continue - elif [ "$KIND" = scout ]; then - if FM_SPAWN_NO_GUARD=1 "$FM_ROOT/bin/fm-spawn.sh" "${pair%%=*}" "${pair#*=}" "${shared_args[@]+"${shared_args[@]}"}" --scout; then :; else - echo "batch: FAILED to spawn ${pair%%=*} (${pair#*=})" >&2 - rc=1 - fi - else - if FM_SPAWN_NO_GUARD=1 "$FM_ROOT/bin/fm-spawn.sh" "${pair%%=*}" "${pair#*=}" "${shared_args[@]+"${shared_args[@]}"}"; then :; else - echo "batch: FAILED to spawn ${pair%%=*} (${pair#*=})" >&2 - rc=1 - fi + fi + pair_args=("${pair%%=*}" "${pair#*=}" "${shared_args[@]+"${shared_args[@]}"}") + [ "$KIND" != scout ] || pair_args+=(--scout) + pair_rc=0 + FM_SPAWN_NO_GUARD=1 "$FM_ROOT/bin/fm-spawn.sh" "${pair_args[@]}" || pair_rc=$? + if [ "$pair_rc" -eq "$FM_PROJECT_CAPACITY_DEFER_EXIT" ]; then + echo "batch: DEFERRED ${pair%%=*} (${pair#*=}) - its project is at capacity, so it stays queued" >&2 + [ "$rc" -ne 0 ] || rc=$FM_PROJECT_CAPACITY_DEFER_EXIT + elif [ "$pair_rc" -ne 0 ]; then + echo "batch: FAILED to spawn ${pair%%=*} (${pair#*=})" >&2 + rc=1 fi done exit "$rc" @@ -3099,17 +3116,52 @@ else WT="" BRIEF="$DATA/$ID/brief.md" fi -if [ "$RELAUNCH" -eq 0 ] && [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then +# Project capacity admission (bin/fm-project-capacity-lib.sh owns the +# declaration, what holds a place, and why this is race-safe). A fresh worker +# for a project whose declared capacity is already held is deferred here, before +# any brief render, endpoint, worktree, record, or backlog move exists, so the +# deferral leaves the task exactly as queued as it was. A relaunch replaces a +# worker that already holds a place, and a secondmate is not a worker. +SPAWN_PROJECT_CAPACITY= +SPAWN_PROJECT_CAPACITY_ANY= +if [ "$RELAUNCH" -eq 0 ] && [ "$KIND" != secondmate ]; then + SPAWN_CAPACITY_CONFIG=$(fm_project_capacity_config_dir "$FM_HOME" "$CONFIG") || { + echo "error: could not resolve the root Firstmate home that declares project capacity for $PROJ_ABS" >&2 + exit 1 + } + if ! fm_project_capacity_lookup "$SPAWN_CAPACITY_CONFIG" "$(basename "$PROJ_ABS")"; then + echo "error: spawn refused: the project capacity declaration is unreadable ($FM_PROJECT_CAPACITY_ERROR); fix it so the captain's worker limits are known (docs/configuration.md \"Project capacity\")" >&2 + exit 1 + fi + SPAWN_PROJECT_CAPACITY=$FM_PROJECT_CAPACITY + SPAWN_PROJECT_CAPACITY_ANY=$FM_PROJECT_CAPACITY_ANY +fi +if [ "$RELAUNCH" -eq 0 ] && [ "$KIND" != secondmate ] && + { [ "$BACKEND" != orca ] || [ -n "$SPAWN_PROJECT_CAPACITY_ANY" ]; }; then SPAWN_TREEHOUSE_PROJECT_LOCK=$(fm_treehouse_project_lock_path "$PROJ_ABS") || { echo "error: could not resolve the shared Treehouse project lock for $PROJ_ABS" >&2 exit 1 } if ! fm_lock_try_acquire "$SPAWN_TREEHOUSE_PROJECT_LOCK"; then - echo "error: another Treehouse slot allocation or return is in progress for $PROJ_ABS; refusing to race it" >&2 + if [ "$BACKEND" = orca ]; then + echo "error: another spawn or cleanup holds the shared project lock for $PROJ_ABS; refusing to race its capacity admission" >&2 + else + echo "error: another Treehouse slot allocation or return is in progress for $PROJ_ABS; refusing to race it" >&2 + fi exit 1 fi SPAWN_TREEHOUSE_PROJECT_LOCK_HELD=1 fi +if [ -n "$SPAWN_PROJECT_CAPACITY" ]; then + if ! fm_project_capacity_occupants "$SPAWN_TREEHOUSE_PROJECT_LOCK" "$PROJ_ABS" "$STATE" "$ID"; then + echo "error: spawn refused: project $(basename "$PROJ_ABS") declares a capacity of $SPAWN_PROJECT_CAPACITY, but this machine's task records cannot all be read to count it ($FM_PROJECT_CAPACITY_ERROR)" >&2 + exit 1 + fi + if [ "$FM_PROJECT_CAPACITY_OCCUPANTS" -ge "$SPAWN_PROJECT_CAPACITY" ]; then + echo "deferred: project $(basename "$PROJ_ABS") admits $SPAWN_PROJECT_CAPACITY worker(s) at once on this machine ($FM_PROJECT_CAPACITY_FILE) and $FM_PROJECT_CAPACITY_OCCUPANTS already hold a place ($FM_PROJECT_CAPACITY_OCCUPANT_IDS); task $ID was not launched and its backlog item stays queued - dispatch it again once one of them records its ready PR or is cleaned up" >&2 + exit "$FM_PROJECT_CAPACITY_DEFER_EXIT" + fi +fi [ -f "$BRIEF" ] || { echo "error: task $ID has no brief at inaccessible data path $BRIEF" >&2 exit 1 diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index 1a0f6c14e78..30b2d1b4507 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -2337,50 +2337,15 @@ teardown_live_slot_path() { canonical_existing_dir "$WT" } +# Every local Firstmate state directory whose records can name a pool slot this +# task's slot might also be; bin/fm-wake-lib.sh's fm_local_firstmate_state_dirs +# owns the walk and what it refuses. collect_local_firstmate_states() { - local record_state=$1 root home reg line child known existing i=0 - local -a homes - TREEHOUSE_OWNER_STATES=("$record_state") - root=$(fm_firstmate_root_home "$FM_HOME") || { - echo "REFUSED: cannot resolve the root Firstmate home; nothing was changed" >&2 + fm_local_firstmate_state_dirs "$1" || { + echo "REFUSED: $FM_LOCAL_FIRSTMATE_ERROR; nothing was changed" >&2 return 1 } - homes=("$root") - while [ "$i" -lt "${#homes[@]}" ]; do - home=${homes[$i]} - i=$((i + 1)) - known=0 - for existing in "${TREEHOUSE_OWNER_STATES[@]}"; do - [ "$existing" != "$home/state" ] || known=1 - done - [ "$known" = 1 ] || TREEHOUSE_OWNER_STATES+=("$home/state") - reg="$home/data/secondmates.md" - [ ! -e "$reg" ] && [ ! -L "$reg" ] && continue - [ -f "$reg" ] && [ ! -L "$reg" ] || { - echo "REFUSED: local Firstmate registry is unsafe at $reg; nothing was changed" >&2 - return 1 - } - while IFS= read -r line || [ -n "$line" ]; do - case "$line" in - "- "*) - secondmate_registry_parse_line "$line" || { - echo "REFUSED: malformed local Firstmate registry entry in $reg; nothing was changed" >&2 - return 1 - } - [ "$SECONDMATE_REGISTRY_REMOTE" -eq 0 ] || continue - child=$(canonical_existing_dir "$SECONDMATE_REGISTRY_HOME") || { - echo "REFUSED: registered local Firstmate home is unavailable: $SECONDMATE_REGISTRY_HOME; nothing was changed" >&2 - return 1 - } - known=0 - for existing in "${homes[@]}"; do - [ "$existing" != "$child" ] || known=1 - done - [ "$known" = 1 ] || homes+=("$child") - ;; - esac - done < "$reg" - done + TREEHOUSE_OWNER_STATES=("${FM_LOCAL_FIRSTMATE_STATES[@]}") } require_exclusive_worktree_slot_record() { diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 0676f6dc517..340d57b48cf 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -393,6 +393,7 @@ family_for_basename() { fm-trace-context-spawn.test.sh|fm-spawn-worktree-settle.test.sh|\ fm-spawn-compact-adviser-disable.test.sh|\ fm-spawn-compact-adviser-disable-remote.test.sh|\ + fm-project-capacity.test.sh|\ fm-teardown-endpoint-safety.test.sh) printf '%s\n' backend-dispatch ;; diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index ddc708aadc8..c4d51a24731 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1432,10 +1432,10 @@ fm_task_set_lock_path() { # # the walk at the current home, which is the correct answer rather than an # error: the parent lives on another machine, so its filesystem can neither hold # nor be observed by a lock taken here, and a remote-seeded home is itself the -# top of the local tree that bin/fm-teardown.sh's collect_local_firstmate_states -# enumerates (that walk already skips remote registry entries for the same -# reason). Refusing a remote binding instead made every operation anchored here -# fail closed inside a remote secondmate home and its local descendants. +# top of the local tree that fm_local_firstmate_state_dirs below enumerates +# (that walk already skips remote registry entries for the same reason). +# Refusing a remote binding instead made every operation anchored here fail +# closed inside a remote secondmate home and its local descendants. # # Everything else still fails closed: an unreadable or malformed binding, an # unreachable local parent, a cycle, and a chain deeper than the bound. @@ -1464,7 +1464,72 @@ fm_firstmate_root_home() { printf '%s\n' "$home" } -# The one lock serializing Treehouse slot allocation and return for a project. +# Every Firstmate state directory on THIS machine whose task records can share a +# machine-local resource with : itself, then the local +# root home and each local secondmate home registered below it, walked through +# every data/secondmates.md breadth-first. Remote registry entries are skipped, +# because their workers run on another machine. +# +# Sets FM_LOCAL_FIRSTMATE_STATES to that list, first and without +# duplicates however each directory is spelled. Returns 1 with +# FM_LOCAL_FIRSTMATE_ERROR naming what could not be proved - an unresolvable +# root, an unsafe or malformed registry, or an unavailable registered local +# home - so a caller refuses rather than treating an unreadable home as one +# with no tasks. Requires bin/fm-secondmate-registry-lib.sh to be sourced first. +# shellcheck disable=SC2034 # FM_LOCAL_FIRSTMATE_ERROR is read by callers. +fm_local_firstmate_state_dirs() { # + local first=$1 root home reg line child known existing i=0 + local -a homes + FM_LOCAL_FIRSTMATE_STATES=("$first") + FM_LOCAL_FIRSTMATE_ERROR= + root=$(fm_firstmate_root_home "$FM_HOME") || { + FM_LOCAL_FIRSTMATE_ERROR="cannot resolve the root Firstmate home" + return 1 + } + homes=("$root") + while [ "$i" -lt "${#homes[@]}" ]; do + home=${homes[$i]} + i=$((i + 1)) + known=0 + for existing in "${FM_LOCAL_FIRSTMATE_STATES[@]}"; do + if [ "$existing" = "$home/state" ] || [ "$existing" -ef "$home/state" ]; then + known=1 + fi + done + [ "$known" = 1 ] || FM_LOCAL_FIRSTMATE_STATES+=("$home/state") + reg="$home/data/secondmates.md" + [ ! -e "$reg" ] && [ ! -L "$reg" ] && continue + [ -f "$reg" ] && [ ! -L "$reg" ] || { + FM_LOCAL_FIRSTMATE_ERROR="local Firstmate registry is unsafe at $reg" + return 1 + } + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + "- "*) + secondmate_registry_parse_line "$line" || { + FM_LOCAL_FIRSTMATE_ERROR="malformed local Firstmate registry entry in $reg" + return 1 + } + [ "$SECONDMATE_REGISTRY_REMOTE" -eq 0 ] || continue + child=$([ -d "$SECONDMATE_REGISTRY_HOME" ] && + CDPATH='' cd -- "$SECONDMATE_REGISTRY_HOME" 2>/dev/null && pwd -P) || { + FM_LOCAL_FIRSTMATE_ERROR="registered local Firstmate home is unavailable: $SECONDMATE_REGISTRY_HOME" + return 1 + } + known=0 + for existing in "${homes[@]}"; do + [ "$existing" != "$child" ] || known=1 + done + [ "$known" = 1 ] || homes+=("$child") + ;; + esac + done < "$reg" + done +} + +# The one lock serializing Treehouse slot allocation and return for a project, +# and project capacity admission (bin/fm-project-capacity-lib.sh), which a +# fresh spawn evaluates under it on every backend. # # It is anchored in the local root home's state directory so that every home on # this machine that can reach the same pool - the root, and each secondmate home diff --git a/docs/configuration.md b/docs/configuration.md index 760c0195672..0eecb1af049 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1061,6 +1061,29 @@ When stripping is enabled, the hooks directory is read-only, so a hook manager r The flag is a home-wide attribution choice, so it is inherited into secondmate homes under the [`secondmate-provisioning`](../.agents/skills/secondmate-provisioning/SKILL.md) inherited-local-material contract and a secondmate's own workers keep AI trailers too. Per-machine Cursor `cli-config.json` attribution-off is not this contract: it does not travel with Firstmate, defaults back to on when unset, and only feeds the CLI's request to the server, so it suppresses the trailer rather than preventing it. +## Project capacity (config/project-capacity) + +The optional local, gitignored `config/project-capacity` tells Firstmate how many workers a project can run at once on this machine, for a project whose machine-local resource - a heavy test suite, a local editor stack, a device - only serves a few workers at a time. +Without it, dispatch stays uncapped as `AGENTS.md` section 7 describes, and a surplus worker is launched only to spend full-context turns waiting for the resource. +The file lives in the machine's root Firstmate home, so every local secondmate home reads the same limit, and it holds one line per project: + +```text +# heavy suite serves two workers +my-project 2 +``` + +The name is the project's registered name, which is its clone directory name and may contain spaces, and the number, the last field on the line, is a positive integer. +A line that is only `#`, or that begins with `#` followed by whitespace, is a comment, as is a `#` line whose last field is not an integer. +A project name may begin with `#` when that `#` is written immediately against the rest of the name and the line ends with the project's capacity. +A name that is `#`, or that begins with `#` and a space, cannot be declared, because that line is a comment. +A place is held by every ship or scout on that project in the root home or any local secondmate home registered under it, including one working in a separate clone of the same origin, until its ready PR is recorded or it is cleaned up; a local-only ship or a scout holds its place until cleanup. +The declaration is matched by the spawning clone's directory name, so clones of the same origin share the cap only when they use that same directory name. +A clone of that origin under a different directory name finds no declaration and is not capped, though its workers are still counted as holders for a same-origin clone that is capped. +When every place is held, `bin/fm-spawn.sh` launches nothing, creates no record, leaves the backlog item queued, prints one `deferred:` line naming the holders, and exits 75, so Firstmate dispatches the item again once a place frees. +A malformed or unreadable file refuses every fresh ship or scout spawn until it is fixed, rather than guessing the intended limit, and so does a local home's state directory or task record that cannot be read while counting a capped project's holders. +Firstmate cannot see which part of a worker's life uses the resource, so the number bounds whole workers from launch to handoff, and the tightest resource every worker needs should decide it. +[`bin/fm-project-capacity-lib.sh`](../bin/fm-project-capacity-lib.sh) owns the file format, what holds a place, and why concurrent spawns cannot both take the last one. + ## Crew dispatch profiles (config/crew-dispatch.json) `config/crew-dispatch.json` is an optional local, gitignored file containing natural-language rules that firstmate reads before dispatching a crewmate or scout. diff --git a/tests/fm-contributions.test.sh b/tests/fm-contributions.test.sh index 47ccb09854b..3c71ee9132e 100755 --- a/tests/fm-contributions.test.sh +++ b/tests/fm-contributions.test.sh @@ -1024,6 +1024,9 @@ test_arm_plumbs_a_configured_budget_into_the_check_shim() { # before the first forge call, so nothing is ever observed. /bin/date +%s > "$home/forge/clock" printf 'hang\n' > "$home/forge/fault" + # Freeze the clock. An unfrozen one-second budget can tick past before the + # first forge call, so the generated check never writes forge/calls. + /bin/date +%s > "$home/forge/clock" if [ "$mode" = configured ]; then with_home "$home" env FM_CONTRIBUTIONS_BUDGET=1 "$ROOT/bin/fm-contributions.sh" arm >/dev/null \ || fail 'arm with a configured budget failed' diff --git a/tests/fm-project-capacity.test.sh b/tests/fm-project-capacity.test.sh new file mode 100755 index 00000000000..bc64b8283b1 --- /dev/null +++ b/tests/fm-project-capacity.test.sh @@ -0,0 +1,642 @@ +#!/usr/bin/env bash +# Behavior tests for project capacity admission: a project that declares how +# many workers it admits at once on this machine never gets a fresh worker +# launched beyond that number (bin/fm-project-capacity-lib.sh owns the +# contract; bin/fm-spawn.sh runs the check). +# +# Every case drives the real bin/fm-spawn.sh against a real project clone with +# an origin, fake tmux and treehouse binaries that log every call, and a real +# markdown backlog when tasks-axi is installed. A deferred spawn is judged by +# what it left behind - no task record, no rendered launch brief, no endpoint, +# no worktree allocation, and a backlog item still queued - never by wording +# alone. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +unset TASKS_AXI_BACKEND || : + +SPAWN="$ROOT/bin/fm-spawn.sh" +TEARDOWN="$ROOT/bin/fm-teardown.sh" +TMP_ROOT=$(fm_test_tmproot fm-project-capacity) +DEFER_EXIT=75 +HAVE_TASKS_AXI=0 +command -v tasks-axi >/dev/null 2>&1 && HAVE_TASKS_AXI=1 + +# --- fixture ---------------------------------------------------------------- + +write_brief() { # + mkdir -p "$1/data/$2" + cat > "$1/data/$2/brief.md" < [task-id...] + local home=$1 id + shift + mkdir -p "$home/state" "$home/config" "$home/data" "$home/projects" + touch "$home/state/.last-watcher-beat" + printf '%s\n' codex > "$home/config/crew-harness" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + printf '%s\n' '# Backlog' '' '## In flight' '' '## Queued' '' '## Done' \ + > "$home/data/backlog.md" + cat > "$home/.tasks.toml" <<'EOF' +backend = "markdown" + +[markdown] +path = "data/backlog.md" +EOF + fi + for id in "$@"; do + write_brief "$home" "$id" + add_item "$home" "$id" + done +} + +# A case: one home, a project clone with an origin, and fake tmux/treehouse +# that record every call so a deferral can be proved to have created nothing. +make_case() { # [task-id...] + local name=$1 case_dir fakebin + shift + case_dir="$TMP_ROOT/$name" + mkdir -p "$case_dir" + fakebin=$(fm_fakebin "$case_dir") + : > "$case_dir/calls.log" + cat > "$fakebin/tmux" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$FM_FAKE_CALL_LOG" +case "$*" in + *"#{pane_current_path}"*) + # A spawn that is told to hold waits here, after its capacity check and + # while it still holds the project lock, until the test releases it. + if [ -n "${FM_FAKE_HOLD:-}" ]; then + : > "$FM_FAKE_HOLD.reached" + i=0 + while [ ! -f "$FM_FAKE_HOLD.release" ]; do + i=$((i + 1)) + [ "$i" -lt 400 ] || exit 1 + sleep 0.05 + done + fi + printf '%s\n' "${FM_FAKE_PANE_PATH:-}" + exit 0 + ;; +esac +case "${1:-}" in display-message) printf 'firstmate\n' ;; esac +exit 0 +SH + cat > "$fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +printf 'treehouse %s\n' "$*" >> "$FM_FAKE_CALL_LOG" +exit 0 +SH + chmod +x "$fakebin/tmux" "$fakebin/treehouse" + fm_fake_exit0 "$fakebin" gh gh-axi no-mistakes + fm_git_init_commit "$case_dir/project" + fm_git_add_origin "$case_dir/project" "$case_dir/project.origin.git" + fm_git_init_commit "$case_dir/other-project" + fm_git_add_origin "$case_dir/other-project" "$case_dir/other-project.origin.git" + make_home "$case_dir/home" "$@" + printf '%s\n' "$case_dir" +} + +add_item() { # + [ "$HAVE_TASKS_AXI" = 1 ] || return 0 + tasks-axi add "$2" "item for $2" --kind ship --file "$1/data/backlog.md" >/dev/null +} + +row_state() { # + tasks-axi show "$2" --file "$1/data/backlog.md" 2>/dev/null | + sed -n 's/^ state: *//p' | head -1 +} + +declare_capacity() { # ... + local home=$1 + shift + printf '%s\n' "$@" > "$home/config/project-capacity" +} + +# A live worker already on a project: the record shape bin/fm-spawn.sh +# publishes, with its backlog item In flight so cleanup can close it. +write_live() { # [extra-line...] + local home=$1 id=$2 project=$3 + shift 3 + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" \ + "endpoint_task_id=$id" \ + "worktree=$home/absent-worktree-$id" \ + "project=$project" \ + "harness=codex" \ + "kind=ship" \ + "mode=no-mistakes" \ + "yolo=off" \ + "spawn_gen=s-$id" \ + "$@" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + add_item "$home" "$id" + tasks-axi start "$id" --file "$home/data/backlog.md" >/dev/null + fi +} + +# One isolated worktree per spawn, so every admitted launch has its own copy. +new_worktree() { # + git -C "$1/project" worktree add --quiet -b "wt-$2" "$1/wt-$2" + printf '%s\n' "$1/wt-$2" +} + +run_spawn() { # + local case_dir=$1 home=$2 pane=$3 + shift 3 + FM_ROOT_OVERRIDE='' FM_HOME="$home" \ + FM_STATE_OVERRIDE='' FM_DATA_OVERRIDE='' FM_PROJECTS_OVERRIDE='' FM_CONFIG_OVERRIDE='' \ + FM_SPAWN_NO_GUARD=1 TMUX="fake,1,0" FM_BACKEND=tmux \ + FM_FAKE_PANE_PATH="$pane" FM_FAKE_CALL_LOG="$case_dir/calls.log" \ + PATH="$case_dir/fakebin:$PATH" \ + "$SPAWN" "$@" 2>&1 +} + +spawn_ship() { # [pane-path] + local case_dir=$1 id=$2 pane=${3:-} + [ -n "$pane" ] || pane=$(new_worktree "$case_dir" "$id") + run_spawn "$case_dir" "$case_dir/home" "$pane" "$id" "$case_dir/project" --mode no-mistakes --yolo off +} + +# Everything a deferred spawn must not have created for ; +# is worktree_list taken before the spawn. +assert_nothing_created() { # + local case_dir=$1 home=$2 id=$3 before=$4 worktrees=$5 after + assert_absent "$home/state/$id.meta" "a deferred spawn published a task record for $id" + assert_absent "$home/data/$id/launch-brief.md" "a deferred spawn rendered a launch brief for $id" + after=$(call_count "$case_dir") + [ "$after" -eq "$before" ] || + fail "a deferred spawn touched the terminal or worktree pool for $id: $(tail -n +"$((before + 1))" "$case_dir/calls.log")" + assert_equals "$worktrees" "$(worktree_list "$case_dir")" "a deferred spawn left a git worktree for $id" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + [ "$(row_state "$home" "$id")" = queued ] || + fail "a deferred spawn moved $id's backlog item: $(row_state "$home" "$id")" + fi +} + +call_count() { wc -l < "$1/calls.log" | tr -d ' '; } + +worktree_list() { git -C "$1/project" worktree list --porcelain; } + +# --- cases ------------------------------------------------------------------ + +# The reported incident's shape before any declaration: the project is already +# busy, and nothing caps a further launch. Absent a declaration that stays true. +test_undeclared_capacity_keeps_dispatch_uncapped() { + local case_dir home out rc=0 + case_dir=$(make_case undeclared task-c) + home="$case_dir/home" + write_live "$home" live-a "$case_dir/project" + write_live "$home" live-b "$case_dir/project" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "an undeclared project refused a spawn: $out" + assert_contains "$out" "spawned task-c" "an undeclared project did not launch the worker" + assert_present "$home/state/task-c.meta" "an undeclared project's spawn published no record" + pass "a project with no declared capacity keeps today's uncapped dispatch" +} + +test_available_capacity_admits_the_worker() { + local case_dir home out rc=0 + case_dir=$(make_case available task-c) + home="$case_dir/home" + declare_capacity "$home" "# heavy suite serves two workers" "project 2" "other-project 1" + write_live "$home" live-a "$case_dir/project" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "a spawn with a free place was refused: $out" + assert_contains "$out" "spawned task-c" "a spawn with a free place did not launch" + assert_not_contains "$out" "deferred:" "a spawn with a free place reported a deferral" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + [ "$(row_state "$home" task-c)" = in_flight ] || fail "an admitted spawn did not move its item In flight" + fi + pass "a spawn is admitted while its project still has a free place" +} + +test_exhausted_capacity_defers_without_leaving_anything_behind() { + local case_dir home out rc=0 before worktrees + case_dir=$(make_case exhausted task-c) + home="$case_dir/home" + declare_capacity "$home" "project 2" + write_live "$home" live-a "$case_dir/project" + write_live "$home" live-b "$case_dir/project" + before=$(call_count "$case_dir") + worktrees=$(worktree_list "$case_dir") + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a spawn beyond capacity was not deferred: $out" + assert_contains "$out" "deferred: project project admits 2 worker(s) at once on this machine ($home/config/project-capacity) and 2 already hold a place (live-a, live-b)" \ + "the deferral did not name the capacity and its holders" + assert_contains "$out" "task task-c was not launched and its backlog item stays queued" \ + "the deferral did not say the task stays queued" + assert_nothing_created "$case_dir" "$home" task-c "$before" "$worktrees" + pass "a spawn beyond capacity is deferred before any record, brief, endpoint, worktree, or backlog move exists" +} + +# The capacity is the last field, so a project whose clone directory name holds +# spaces can be declared. An indented '#' line is still a comment. +test_spaced_project_name_is_declared() { + local case_dir home spaced out rc=0 + case_dir=$(make_case spaced task-c) + home="$case_dir/home" + spaced="$case_dir/my heavy project" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$spaced" + declare_capacity "$home" " # my heavy project 9" "my heavy project 1" "project 5" + write_live "$home" live-a "$spaced" + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-c "$spaced" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a project whose name holds spaces was not capped by its declaration: $out" + assert_contains "$out" "deferred: project my heavy project admits 1 worker(s) at once" \ + "the deferral did not use the spaced project's declared capacity" + assert_absent "$home/state/task-c.meta" "the deferred spaced-name spawn published a record" + pass "a project name with spaces is declared by taking the capacity from the last field" +} + +# A clone directory may be named with a leading '#'. That name is declared when +# the '#' is written against the rest of the name and the line ends with the +# capacity. A '#' followed by whitespace stays a comment even when the line +# ends with a number, and a '#' note that is not a capacity stays a comment. +test_hash_prefixed_project_name_is_declared() { + local case_dir home hashed spaced out rc=0 wt + case_dir=$(make_case hash-name task-c task-d) + home="$case_dir/home" + hashed="$case_dir/#hash-project" + spaced="$case_dir/# serves" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$hashed" + git clone -q "$(git -C "$case_dir/other-project" remote get-url origin)" "$spaced" + declare_capacity "$home" "# serves 2" "#not-a-capacity" "#hash-project 1" + write_live "$home" live-a "$hashed" + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-c "$hashed" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a project whose name begins with # was not capped by its declaration: $out" + assert_contains "$out" "deferred: project #hash-project admits 1 worker(s) at once" \ + "the deferral did not use the hash-prefixed project's declared capacity" + assert_absent "$home/state/task-c.meta" "the deferred hash-prefixed spawn published a record" + + write_live "$home" live-b "$spaced" + write_live "$home" live-c "$spaced" + git -C "$spaced" worktree add --quiet -b wt-d "$case_dir/wt-d" + wt="$case_dir/wt-d" + rc=0 + out=$(run_spawn "$case_dir" "$home" "$wt" task-d "$spaced" --mode no-mistakes --yolo off) || rc=$? + expect_code 0 "$rc" "a '#' comment that ends with a number was read as a capacity: $out" + assert_contains "$out" "spawned task-d" "a project whose declaration line is a comment did not stay uncapped" + pass "a project name beginning with # is declared, and a # comment stays a comment" +} + +# A home reached through a symlink is still one home: its workers hold one +# place each, not one per spelling of its state directory. +test_symlinked_home_counts_each_worker_once() { + local case_dir home out rc=0 + case_dir=$(make_case symlinked-home task-c) + home="$case_dir/home" + ln -s "$home" "$case_dir/home-link" + declare_capacity "$home" "project 2" + write_live "$home" live-a "$case_dir/project" + out=$(run_spawn "$case_dir" "$case_dir/home-link" "$(new_worktree "$case_dir" task-c)" \ + task-c "$case_dir/project" --mode no-mistakes --yolo off) || rc=$? + expect_code 0 "$rc" "a spawn through a symlinked home counted its own worker twice: $out" + assert_contains "$out" "spawned task-c" "a spawn through a symlinked home did not launch" + pass "a home reached through a symlink counts each of its workers once" +} + +# A fresh spawn that restarts an existing task id replaces that task's own +# record, so the record does not hold a place against it. Any other task on the +# project still sees that record as a holder. +test_restart_does_not_count_its_own_record() { + local case_dir home out rc=0 + case_dir=$(make_case restart task-c task-d) + home="$case_dir/home" + declare_capacity "$home" "project 1" + fm_write_meta "$home/state/task-c.meta" \ + "window=firstmate:fm-task-c" \ + "project=$case_dir/project" \ + "kind=ship" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + assert_not_contains "$out" "deferred:" "a restart was deferred by its own record" + [ "$rc" -ne "$DEFER_EXIT" ] || fail "a restart exited with the deferral code: $out" + rc=0 + out=$(spawn_ship "$case_dir" task-d "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "another task ignored the restarted task's place: $out" + assert_contains "$out" "1 already hold a place (task-c)" "another task did not count the restarted task" + pass "a restart of a task id does not count that task's own record" +} + +# A place frees when a worker records its ready PR and when a task is cleaned +# up; each release admits exactly one more worker. +test_release_frees_a_place() { + local case_dir home out rc=0 + case_dir=$(make_case release task-c task-d) + home="$case_dir/home" + declare_capacity "$home" "project 2" + write_live "$home" live-a "$case_dir/project" + write_live "$home" live-b "$case_dir/project" + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "the full project admitted a worker: $out" + + # PR handoff: the line bin/fm-pr-check.sh records for a ready PR. + printf 'pr=%s\n' "https://github.com/o/r/pull/7" >> "$home/state/live-a.meta" + rc=0 + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "a recorded PR handoff did not free a place: $out" + rc=0 + out=$(spawn_ship "$case_dir" task-d "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "one freed place admitted two workers: $out" + assert_contains "$out" "2 already hold a place (live-b, task-c)" "the new worker did not take the freed place" + + # Cleanup: the real teardown removes the record, which frees its place. + rc=0 + out=$(FM_ROOT_OVERRIDE='' FM_HOME="$home" FM_STATE_OVERRIDE='' FM_DATA_OVERRIDE='' FM_CONFIG_OVERRIDE='' \ + FM_FAKE_CALL_LOG="$case_dir/calls.log" PATH="$case_dir/fakebin:$PATH" \ + "$TEARDOWN" live-b 2>&1) || rc=$? + expect_code 0 "$rc" "cleanup of a live worker failed: $out" + assert_absent "$home/state/live-b.meta" "cleanup left the worker's record" + rc=0 + out=$(spawn_ship "$case_dir" task-d) || rc=$? + expect_code 0 "$rc" "cleanup did not free a place: $out" + pass "a recorded PR handoff or a cleanup each frees exactly one place" +} + +# Only workers on the same project identity hold places: a secondmate record, a +# record for another project, and a scout on another project are ignored, while +# a scout and a legacy record without kind= on this project count. +test_occupancy_counts_only_this_projects_workers() { + local case_dir home out rc=0 + case_dir=$(make_case occupancy task-c) + home="$case_dir/home" + declare_capacity "$home" "project 3" + write_live "$home" scout-a "$case_dir/project" + sed -i.bak 's/^kind=ship$/kind=scout/' "$home/state/scout-a.meta" && rm -f "$home/state/scout-a.meta.bak" + fm_write_meta "$home/state/legacy-b.meta" "window=firstmate:fm-legacy-b" "project=$case_dir/project" + write_live "$home" other-c "$case_dir/other-project" + fm_write_meta "$home/state/mate-d.meta" "window=remote:mate-d" "project=$case_dir/project" "kind=secondmate" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "records outside this project's workers took a place: $out" + rc=0 + write_brief "$home" task-e + add_item "$home" task-e + out=$(spawn_ship "$case_dir" task-e "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "the third worker on the project was not counted: $out" + assert_contains "$out" "3 already hold a place (legacy-b, scout-a, task-c)" \ + "occupancy counted the wrong records" + pass "only this project's ship and scout records hold places" +} + +# Capacity belongs to the machine: a local secondmate home's workers on a +# separate clone of the same origin count, the declaration comes from the root +# home even when the secondmate spawns, and a remote home's workers never count. +test_capacity_is_shared_by_every_local_home() { + local case_dir root mate out rc=0 mate_project + case_dir=$(make_case machine) + root="$case_dir/home" + mate="$case_dir/mate" + make_home "$mate" task-m + mate_project="$mate/projects/project" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$mate_project" + printf '%s\n' schema=fm-secondmate-parent.v1 route=local "parent_home=$root" > "$mate/.fm-secondmate-parent" + printf -- '- mate - a local mate (home: %s; scope: project work; projects: project; added 2026-09-01)\n' "$mate" \ + > "$root/data/secondmates.md" + printf -- '- far - a remote mate (host: far.example; root: /srv/fm; home: %s/remote-home; scope: other; projects: project; added 2026-09-01)\n' "$case_dir" \ + >> "$root/data/secondmates.md" + mkdir -p "$case_dir/remote-home/state" + fm_write_meta "$case_dir/remote-home/state/far-x.meta" "window=w" "project=$case_dir/project" "kind=ship" + declare_capacity "$root" "project 1" + write_live "$root" live-a "$case_dir/project" + git -C "$mate_project" worktree add --quiet -b wt-m "$case_dir/wt-m" + out=$(run_spawn "$case_dir" "$mate" "$case_dir/wt-m" task-m "$mate_project" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a local secondmate launched past the machine's capacity: $out" + assert_contains "$out" "admits 1 worker(s) at once on this machine ($root/config/project-capacity) and 1 already hold a place (live-a in $root)" \ + "the secondmate did not count the root home's worker against the root's declaration" + assert_absent "$mate/state/task-m.meta" "the deferred secondmate spawn published a record" + printf 'pr=%s\n' "https://github.com/o/r/pull/8" >> "$root/state/live-a.meta" + rc=0 + out=$(run_spawn "$case_dir" "$mate" "$case_dir/wt-m" task-m "$mate_project" --mode no-mistakes --yolo off) || rc=$? + expect_code 0 "$rc" "the remote home's worker, or a handed-off one, took the machine's place: $out" + pass "every local home shares one declared capacity per project origin, and remote homes do not count" +} + +# A local home's state directory or task record that cannot be read could hide a +# holder, so admission refuses instead of counting without it. +test_unreadable_holders_refuse_admission() { + local case_dir root mate out rc=0 + if [ "$(id -u)" = 0 ]; then + printf 'ok - skipped the unreadable-holder case (root reads files regardless of mode)\n' + return 0 + fi + case_dir=$(make_case unreadable-holders task-c) + root="$case_dir/home" + mate="$case_dir/mate" + make_home "$mate" + printf '%s\n' schema=fm-secondmate-parent.v1 route=local "parent_home=$root" > "$mate/.fm-secondmate-parent" + printf -- '- mate - a local mate (home: %s; scope: project work; projects: project; added 2026-09-01)\n' "$mate" \ + > "$root/data/secondmates.md" + declare_capacity "$root" "project 2" + + write_live "$root" live-a "$case_dir/project" + chmod 000 "$root/state/live-a.meta" + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + chmod 600 "$root/state/live-a.meta" + expect_code 1 "$rc" "an unreadable task record did not refuse admission: $out" + assert_contains "$out" "task record $root/state/live-a.meta cannot be read" \ + "the refusal did not name the unreadable task record" + assert_absent "$root/state/task-c.meta" "a spawn published past an unreadable task record" + + chmod 000 "$mate/state" + rc=0 + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + chmod 755 "$mate/state" + expect_code 1 "$rc" "an unreadable local state directory did not refuse admission: $out" + assert_contains "$out" "local Firstmate state directory $mate/state cannot be read" \ + "the refusal did not name the unreadable state directory" + assert_absent "$root/state/task-c.meta" "a spawn published past an unreadable state directory" + + rc=0 + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "a readable machine did not admit the worker: $out" + pass "an unreadable state directory or task record refuses admission rather than undercounting" +} + +# Two spawns racing for the last place: the one holding the project lock +# publishes, the other cannot publish while it waits and is deferred afterwards. +test_concurrent_spawns_cannot_both_take_the_last_place() { + local case_dir home out rc=0 hold i wt + case_dir=$(make_case concurrent task-a task-b) + home="$case_dir/home" + declare_capacity "$home" "project 1" + hold="$case_dir/hold" + wt=$(new_worktree "$case_dir" task-a) + FM_FAKE_HOLD="$hold" spawn_ship "$case_dir" task-a "$wt" > "$case_dir/a.out" 2>&1 & + i=0 + while [ ! -f "$hold.reached" ]; do + i=$((i + 1)) + [ "$i" -lt 400 ] || fail "the first spawn never reached its launch: $(cat "$case_dir/a.out")" + sleep 0.05 + done + out=$(spawn_ship "$case_dir" task-b "$case_dir/unused") || rc=$? + [ "$rc" -ne 0 ] || fail "a concurrent spawn launched while the last place was being taken: $out" + assert_absent "$home/state/task-b.meta" "a concurrent spawn published past the last place" + : > "$hold.release" + wait || true + assert_contains "$(cat "$case_dir/a.out")" "spawned task-a" "the lock-holding spawn did not finish" + rc=0 + out=$(spawn_ship "$case_dir" task-b "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "the retried spawn was not deferred once the place was taken: $out" + assert_absent "$home/state/task-b.meta" "the retried spawn published past capacity" + pass "concurrent spawns can never both take the last place" +} + +# A spawn that fails after admission removes nothing it did not create and +# leaves no record, so it holds no place afterwards. +test_failed_spawn_after_admission_holds_no_place() { + local case_dir home out rc=0 + case_dir=$(make_case failed task-a task-b) + home="$case_dir/home" + declare_capacity "$home" "project 1" + printf '%s\n' '# Task' "## Captain's intent" '{TASK}' '' '## Firstmate spec' 'x' > "$home/data/task-a/brief.md" + out=$(spawn_ship "$case_dir" task-a "$case_dir/unused") || rc=$? + [ "$rc" -ne 0 ] && [ "$rc" -ne "$DEFER_EXIT" ] || fail "an invalid brief did not fail after admission (exit $rc): $out" + assert_contains "$out" "still contains {TASK}" "the spawn did not fail where expected" + assert_absent "$home/state/task-a.meta" "the failed spawn left a record" + rc=0 + out=$(spawn_ship "$case_dir" task-b) || rc=$? + expect_code 0 "$rc" "a failed spawn kept holding the only place: $out" + pass "a spawn that fails after admission leaves no record and holds no place" +} + +test_unreadable_declaration_refuses_every_spawn() { + local case_dir home out rc label body before worktrees + case_dir=$(make_case unreadable task-c) + home="$case_dir/home" + while IFS='|' read -r label body; do + [ -n "$label" ] || continue + printf '%b' "$body" > "$home/config/project-capacity" + before=$(call_count "$case_dir") + worktrees=$(worktree_list "$case_dir") + rc=0 + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code 1 "$rc" "$label: an unreadable declaration did not refuse: $out" + assert_contains "$out" "the project capacity declaration is unreadable" "$label: the refusal did not name the declaration" + assert_nothing_created "$case_dir" "$home" task-c "$before" "$worktrees" + done <<'ROWS' +missing capacity|project\n +zero capacity|project 0\n +non-numeric capacity|other-project two\n +trailing text|project 2 # suite\n +named twice|project 2\nproject 3\n +ROWS + rm -f "$home/config/project-capacity" + mkdir "$home/config/project-capacity" + rc=0 + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code 1 "$rc" "a declaration that is not a file did not refuse: $out" + pass "an unreadable declaration refuses every fresh spawn rather than guessing the limit" +} + +test_batch_reports_a_deferred_pair() { + local case_dir home out rc=0 + case_dir=$(make_case batch task-c) + home="$case_dir/home" + declare_capacity "$home" "project 1" + write_live "$home" live-a "$case_dir/project" + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" "task-c=$case_dir/project" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a batch whose only pair was deferred did not exit with the deferral status: $out" + assert_contains "$out" "batch: DEFERRED task-c ($case_dir/project) - its project is at capacity, so it stays queued" \ + "the batch did not report the deferral" + assert_not_contains "$out" "batch: FAILED" "the batch reported a deferral as a failure" + pass "a batch reports a capacity deferral as deferred, not failed" +} + +# Orca owns its own worktrees and never takes the Treehouse allocation lock, so +# a declared capacity is what makes an Orca spawn take the shared project lock, +# even from an uncapped clone of a capped origin whose worker would still hold a +# place: it refuses while another holder has it, and defers at capacity before +# asking Orca for anything but its runtime status. +test_orca_spawn_is_admitted_under_the_shared_project_lock() { + local case_dir home out out2 rc=0 rc2 holder i + command -v node >/dev/null 2>&1 || { + printf 'ok - skipped the Orca capacity case (node, which the Orca status check needs, is not installed)\n' + return 0 + } + case_dir=$(make_case orca task-o) + home="$case_dir/home" + cat > "$case_dir/fakebin/orca" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = status ]; then + printf '{"ok":true,"result":{"runtime":{"reachable":true,"state":"ready"}}}\n' + exit 0 +fi +printf 'orca %s\n' "$*" >> "$FM_FAKE_CALL_LOG" +exit 1 +SH + chmod +x "$case_dir/fakebin/orca" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$case_dir/project-2" + declare_capacity "$home" "project 1" + + # shellcheck disable=SC2016 # expanded by the holder's own shell + FM_HOME="$home" FM_STATE_OVERRIDE='' bash -c ' + . "$1/bin/fm-wake-lib.sh" + lock=$(fm_treehouse_project_lock_path "$2") || exit 1 + fm_lock_try_acquire "$lock" || exit 1 + : > "$3.held" + while [ ! -f "$3.release" ]; do sleep 0.05; done + fm_lock_release "$lock" + ' _ "$ROOT" "$case_dir/project" "$case_dir/holder" & + holder=$! + i=0 + while [ ! -f "$case_dir/holder.held" ]; do + i=$((i + 1)) + [ "$i" -lt 200 ] || fail "the lock holder never took the project lock" + sleep 0.05 + done + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-o "$case_dir/project" \ + --backend orca --mode no-mistakes --yolo off) || rc=$? + rc2=0 + out2=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-o "$case_dir/project-2" \ + --backend orca --mode no-mistakes --yolo off) || rc2=$? + : > "$case_dir/holder.release" + wait "$holder" || true + expect_code 1 "$rc" "an Orca spawn ignored a held project lock: $out" + assert_contains "$out" "another spawn or cleanup holds the shared project lock for $case_dir/project; refusing to race its capacity admission" \ + "the Orca spawn did not refuse on the shared project lock" + expect_code 1 "$rc2" "an Orca spawn from an uncapped same-origin clone ignored the held project lock: $out2" + assert_contains "$out2" "another spawn or cleanup holds the shared project lock for $case_dir/project-2" \ + "the uncapped same-origin clone's Orca spawn did not refuse on the shared project lock" + assert_absent "$home/state/task-o.meta" "the uncapped clone's Orca spawn published a record while the lock was held" + assert_no_grep "orca " "$case_dir/calls.log" "the Orca spawn asked Orca for more than its runtime status" + + write_live "$home" live-a "$case_dir/project" + rc=0 + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-o "$case_dir/project" \ + --backend orca --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "an Orca spawn beyond capacity was not deferred: $out" + assert_absent "$home/state/task-o.meta" "the deferred Orca spawn published a record" + assert_no_grep "orca " "$case_dir/calls.log" "the deferred Orca spawn created an Orca worktree" + pass "an Orca spawn takes the shared project lock whenever a same-origin clone is capped and defers before creating anything" +} + +test_undeclared_capacity_keeps_dispatch_uncapped +test_available_capacity_admits_the_worker +test_exhausted_capacity_defers_without_leaving_anything_behind +test_spaced_project_name_is_declared +test_hash_prefixed_project_name_is_declared +test_symlinked_home_counts_each_worker_once +test_release_frees_a_place +test_restart_does_not_count_its_own_record +test_occupancy_counts_only_this_projects_workers +test_capacity_is_shared_by_every_local_home +test_unreadable_holders_refuse_admission +test_concurrent_spawns_cannot_both_take_the_last_place +test_failed_spawn_after_admission_holds_no_place +test_unreadable_declaration_refuses_every_spawn +test_batch_reports_a_deferred_pair +test_orca_spawn_is_admitted_under_the_shared_project_lock From 228b27d203ca7a89c8ef05bebb617b416148ab07 Mon Sep 17 00:00:00 2001 From: yairtech <39274208+falkoro@users.noreply.github.com> Date: Thu, 8 Oct 2026 00:22:19 +0200 Subject: [PATCH 02/12] fix(bin): name the Lavish read message count by the same label as its section (#6785) fm-procevent-lavish.sh read labels tag=message rows SESSION-ENDING MESSAGE only when session_ended is true and CAPTAIN MESSAGE otherwise, but the count line always said session_ending_message_count. Several composer messages on a still-open board were therefore counted as session-ending. The count line now follows the same session_ended switch: session_ending_message_count once the session ended, captain_message_count otherwise. Message rows stay out of the annotation count, per triage. Fixes #6743 --- bin/fm-procevent-lavish.sh | 13 ++++++++----- tests/fm-procevent.test.sh | 28 ++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 5 deletions(-) diff --git a/bin/fm-procevent-lavish.sh b/bin/fm-procevent-lavish.sh index 4442c73fe3c..1ec8511ba67 100755 --- a/bin/fm-procevent-lavish.sh +++ b/bin/fm-procevent-lavish.sh @@ -21,8 +21,10 @@ # It is read-only over the capture: it does not arm, poll, or change # what Lavish delivered. The freeform message (tag=message) is its # own labeled field, printed first and distinct from per-element -# annotations; it is labeled SESSION-ENDING MESSAGE only when the -# session ended. Declared and presented item counts, +# annotations; it is labeled SESSION-ENDING MESSAGE, and counted as +# session_ending_message_count, only when the session ended, and is +# otherwise CAPTAIN MESSAGE and captain_message_count. Declared and +# presented item counts, # plus a completeness verdict, follow before all annotations so a # partial read is obvious. Each annotation retains its element uid, # selector, tag, and text. A non-choice freeform comment (`prompt`) @@ -783,9 +785,9 @@ cmd_read() { return if !@lines || (@lines == 1 && $lines[0] eq ""); print "| $_\n" for @lines; } + my $ended = $session_ended =~ /^(?:true|True|TRUE)$/; if (@messages) { - my $message_label = $session_ended =~ /^(?:true|True|TRUE)$/ - ? "SESSION-ENDING MESSAGE" : "CAPTAIN MESSAGE"; + my $message_label = $ended ? "SESSION-ENDING MESSAGE" : "CAPTAIN MESSAGE"; print "$message_label\n"; for my $i (0 .. $#messages) { print "$message_label PART ", ($i + 1), " of ", scalar(@messages), "\n" if @messages > 1; @@ -806,7 +808,8 @@ cmd_read() { print "lifecycle: $lifecycle\n"; print "session_ended: ", (length $session_ended ? $session_ended : "(unset)"), "\n"; print "annotation_count: ", scalar(@annotations), "\n"; - print "session_ending_message_count: ", scalar(@messages), "\n"; + my $message_count_key = $ended ? "session_ending_message_count" : "captain_message_count"; + print "$message_count_key: ", scalar(@messages), "\n"; print "\n"; if (@annotations) { print "ANNOTATIONS\n"; diff --git a/tests/fm-procevent.test.sh b/tests/fm-procevent.test.sh index 5ecd4db4512..46fb4fa2159 100755 --- a/tests/fm-procevent.test.sh +++ b/tests/fm-procevent.test.sh @@ -3144,6 +3144,34 @@ assert_contains "$out" "CAPTAIN MESSAGE" "an open-session message was mislabeled assert_not_contains "$out" "SESSION-ENDING MESSAGE" "an open-session message was labeled as session-ending" assert_contains "$out" "| captain is still reviewing" "an open-session message was dropped" pass "read distinguishes a live captain message from a session-ending message" +# Composer sends arrive as several tag=message rows beside real annotations. +# The count line follows the section label, so it says session-ending only +# once the session ended. +for ended in no yes; do + { + printf 'session:\n file: /review.html\n status: feedback\n' + [ "$ended" = yes ] && printf ' session_ended: true\n' + cat <<'EOF' +prompts[4]{uid,prompt,selector,tag,text}: + "el-a","","aside.sidebar",note,"Sidebar note" + "","first comment","",message,"Freeform message" + "","second comment","",message,"Freeform message" + "","third comment","",message,"Freeform message" +EOF + } > "$READ" + out=$(read_out) || fail "read failed on several messages with an annotation (ended=$ended)" + if [ "$ended" = yes ]; then + label="SESSION-ENDING MESSAGE" count=session_ending_message_count other=captain_message_count + else + label="CAPTAIN MESSAGE" count=captain_message_count other=session_ending_message_count + fi + assert_contains "$out" "$label PART 3 of 3" "a message part was dropped or mislabeled (ended=$ended)" + assert_contains "$out" "| third comment" "a message body was dropped (ended=$ended)" + assert_contains "$out" "$count: 3" "the message count did not follow the section label (ended=$ended)" + assert_not_contains "$out" "$other" "the message count used the other label (ended=$ended)" + assert_contains "$out" "annotation_count: 1" "a real annotation was miscounted beside messages (ended=$ended)" +done +pass "read names the message count with the same label as the message section" out=$ending_out assert_contains "$out" '| "question": "sample-forged-call",' \ "commas in an unquoted freeform message shifted its fields" From 8fa253827f5509ff7f0a5fb1816a650dea4e6539 Mon Sep 17 00:00:00 2001 From: Asser AboElkhair Date: Wed, 7 Oct 2026 16:23:21 -0600 Subject: [PATCH 03/12] fix(herdr): make the presentation lock namespace per OS account (#6780) The Herdr presentation lock namespace was the fixed machine-global /tmp/firstmate-herdr-presentation, so on a host where two OS users run Firstmate on Herdr the first account to create it owned it and every teardown from the other account was refused with no way to clear it. Suffix the namespace with the account uid. The owner-uid and mode-700 checks are unchanged, so a foreign-owned or wrong-mode name at this account's path is still refused and never adopted, chowned, or removed. Fixes #4716. --- bin/backends/herdr.sh | 18 ++++- docs/herdr-backend.md | 4 +- tests/fm-backend-herdr.test.sh | 4 +- tests/fm-teardown.test.sh | 141 +++++++++++++++++++++++++++++++++ 4 files changed, 161 insertions(+), 6 deletions(-) diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index 15ae10a5800..b17522a4a69 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -773,14 +773,26 @@ fm_backend_herdr_projection_workspace_label() { # printf '└ %s · p:%s' "$(fm_backend_herdr_projection_concise_task_label "$1")" "$2" } -# fm_backend_herdr_presentation_session_lock_path: one machine-private lock +# fm_backend_herdr_presentation_session_lock_path: one account-private lock # path per live named Herdr session/socket, shared across every Firstmate home -# that uses that session. +# of this OS account that uses that session. # The path is never under any one home's state/ and secondmates never write the # primary home. Returns non-zero when the named session's socket cannot be # resolved unambiguously. +# The namespace directory is suffixed with this account's uid, so another OS +# account on the same host can never create it first by ordinary use and lock +# this account out; a deliberately pre-created name still fails the ownership +# and mode checks below and is refused, never adopted, chowned, or removed. +# The uid rather than $XDG_RUNTIME_DIR names it because that variable can differ +# or be absent between login contexts of one account, which would split one +# session's lock across processes. fm_backend_herdr_presentation_lock_namespace() { - printf '%s' '/tmp/firstmate-herdr-presentation' + local uid + uid=$(id -u 2>/dev/null) || return 1 + case "$uid" in + ''|*[!0-9]*) return 1 ;; + esac + printf '/tmp/firstmate-herdr-presentation-%s' "$uid" } fm_backend_herdr_presentation_lock_namespace_mode() { diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index ac111a097a6..0190f1d3d03 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -282,7 +282,7 @@ Projected children are placed in one contiguous block immediately after their ow - The protocol. - The socket. - `python3`. -- The machine-private per-session lock. +- The account-private per-session lock. Existing legacy child labels may extend an already adjacent block read-only but are never renamed or migrated. A foreign, ambiguous, detached, or manually interleaved child makes ordering skip with a warning rather than rewriting the layout. @@ -363,6 +363,8 @@ Once the exact pane is confirmed gone, teardown retires the task's own journal w Recovery is deliberately conservative and presentation-only. An existing journal suppresses another projected create. Before any recovery mutation, Firstmate holds both the task spawn lock and the named-session presentation lock. +That presentation lock lives in a namespace private to the OS account, so another account on the same host running its own Firstmate on Herdr cannot block this account's spawn, recovery, or teardown. +A namespace at this account's name that another account owns, or that is not mode 700, is still refused and is never adopted, chowned, or removed. A same-identity version 2 binding may replace one exact agent-free restart husk in place. A husk is a restored same-labeled tab with a missing pane or no registered agent, as [Restart and liveness behavior](#restart-and-liveness-behavior) describes. diff --git a/tests/fm-backend-herdr.test.sh b/tests/fm-backend-herdr.test.sh index 01d0a7015f8..c113f0a3846 100755 --- a/tests/fm-backend-herdr.test.sh +++ b/tests/fm-backend-herdr.test.sh @@ -3390,8 +3390,8 @@ test_presentation_session_lock_path_is_shared_across_homes() { || fail "session lock path resolution failed for home B" [ "$path_a" = "$path_b" ] || fail "same session/socket must resolve one shared lock path" case "$path_a" in - /tmp/firstmate-herdr-presentation/order-*.lock) ;; - *) fail "session lock path must use the shared machine namespace: $path_a" ;; + "/tmp/firstmate-herdr-presentation-$(id -u)"/order-*.lock) ;; + *) fail "session lock path must use this account's namespace: $path_a" ;; esac case "$path_a" in */state/*) fail "session lock path must not live under a home state directory: $path_a" ;; diff --git a/tests/fm-teardown.test.sh b/tests/fm-teardown.test.sh index 3497c74725b..f880cf92b2e 100755 --- a/tests/fm-teardown.test.sh +++ b/tests/fm-teardown.test.sh @@ -2561,6 +2561,146 @@ test_herdr_flat_teardown_preflight_refuses_before_changes() { pass "herdr flat teardown preflight refuses before every destructive change" } +# The Herdr presentation-lock namespace is named per OS account. These cases +# act as a fixture account uid (via an `id -u` shim) so they never touch the +# real account's namespace or the old shared /tmp/firstmate-herdr-presentation, +# and every directory the fixture account resolves is really owned by the +# running account, i.e. by another uid from the fixture account's view. +# FM_FAKE_NS_STAT names one path whose owner and mode the `stat` shim reports +# instead; the adapter reads ownership through a PATH `stat` only on its non- +# Darwin branch, so the arms that need it run only there. +herdr_lock_ns_fake_uid() { + printf '%s' "$((3000000000 + $$ % 1000000))" +} + +configure_herdr_lock_ns_shims() { # + local case_dir=$1 real_id real_stat + real_id=$(command -v id); real_stat=$(command -v stat) + cat > "$case_dir/fakebin/id" < "$case_dir/fakebin/stat" < + if [ -e "$1" ] || [ -L "$1" ]; then + # A fixed, known path: ls is the portable way to read mode and numeric owner. + # shellcheck disable=SC2012 + ls -ldn "$1" 2>/dev/null | awk '{print $1, $3, $4}' + else + printf 'absent' + fi +} + +run_herdr_lock_ns_teardown() { # [stat-spec] + local case_dir=$1 fake_uid=$2 stat_spec=${3:-} rc=0 + FM_FAKE_ACCOUNT_UID="$fake_uid" FM_FAKE_NS_STAT="$stat_spec" \ + FM_FAKE_HERDR_LOG="$case_dir/herdr.log" FM_FAKE_HERDR_CLOSED="$case_dir/closed" \ + FM_BACKEND_HERDR_IDLE_SHELL_PROOF_POLLS=1 \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" || rc=$? + return "$rc" +} + +new_herdr_lock_ns_case() { # + local case_dir + case_dir=$(make_case "$1") + write_meta "$case_dir" local-only ship + configure_flat_herdr_teardown_case "$case_dir" + configure_herdr_lock_ns_shims "$case_dir" + : > "$case_dir/herdr.log" + : > "$case_dir/state/task-x1.status" + printf '%s' "$case_dir" +} + +assert_herdr_lock_ns_refused() { #