diff --git a/.agents/skills/agent-skill-trigger-index/SKILL.md b/.agents/skills/agent-skill-trigger-index/SKILL.md index 6e70321cb3f..6a0348ab879 100644 --- a/.agents/skills/agent-skill-trigger-index/SKILL.md +++ b/.agents/skills/agent-skill-trigger-index/SKILL.md @@ -10,9 +10,14 @@ metadata: These skills are not captain-invocable; load them only at their precise triggers. +- `operational-home-layout` - load when locating, interpreting, or changing Firstmate home, config, data, state, project, or generated runtime paths. +- `session-start-recovery` - load when the session-start digest reports unfinished checks, actionable diagnostics, recovery inputs, or output requiring interpretation. - `bootstrap-diagnostics` - load whenever the session-start digest's bootstrap or network-checks section prints an actionable diagnostic line (`MISSING:`, `MISSING_MANUAL:`, `PRESENTATION_UNAVAILABLE:`, `BACKEND_INVALID:`, `NEEDS_GH_AUTH`, `TANGLE:`, `STARTUP_MEMORY_BUDGET:`, `CREW_DISPATCH: invalid`, `FLEET_SYNC:`, `NETWORK_CHECKS:`, `HOME_SUMMARY:`, `BACKLOG_RECONCILE:`, `SECONDMATE_SYNC:`, `SECONDMATE_LIVENESS:`, `SECONDMATE_HANDOFF:`, `NUDGE_SECONDMATES:`, or `FMX:`), or when `BOOTSTRAP_INFO:` says an interrupted backlog cleanup may have left an endpoint or local copy; silence and other `BOOTSTRAP_INFO:` facts need no load. - `diagnostic-reasoning` - load before scoping a reported bug and before acting on a diagnostic report. - `ask-user-authority` - load before deciding any ask-user finding. +- `validation-supervision` - load when a ship starts or already has an active no-mistakes validation run, including a mid-run requirement change or finding, and before deciding or answering any ask-user finding. +- `ship-landing` - load when a ship reports a PR or ready branch, when deciding or monitoring landing, and before task cleanup. +- `scout-completion` - load when a scout reports completion, presents a visual artifact for iteration, or is being considered for promotion to implementation. - `quota-array-dispatch` - load before choosing among a matched crew-dispatch profile array from current quota-axi default TOON. - `harness-adapters` - load before spawning or recovering a crewmate or secondmate, handling a trust dialog, sending a harness-specific skill invocation, interrupting or exiting an agent, resuming an exited agent, or verifying a new harness adapter. - `firstmate-orca` - load before switching to Orca, spawning or supervising Orca-backed work, smoke-testing Orca backend behavior, debugging Orca task state, or reconciling Orca-backed task metadata. @@ -21,6 +26,7 @@ These skills are not captain-invocable; load them only at their precise triggers - `stuck-crewmate-recovery` - load when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, after a stale wake, looping pane, repeated confusion, an answered-by-brief question, an unresponsive crewmate, or a failed steer, and whenever a live worker reports its no-mistakes pipeline dead, unreachable, or timed out. - `secondmate-provisioning` - load before creating, seeding, validating, launching, handing backlog to, recovering, pushing inherited local material into, or retiring a secondmate home, and before editing `data/secondmates.md`. - `captain-hold-lifecycle` - load before treating an investigation or visual review as complete, before ending a visual review that exposed a captain decision, when recording or routing the captain's answer, and on any `RECORD DIVERGENCE` line from the wake drain. +- `away-quiet-supervision` - load whenever /afk or /quiet is invoked, an away or quiet record exists, or a marked away-supervisor message arrives. - `process-event-sources` - load before arming a long-polling source, before registering a deterministic condition->action watch (do X as soon as Y is true), on any `procevent ` check wake, and on any `process-event source stranded` or `process-event source failed to start` check wake. Never run a registered source's blocking command yourself in a conversational turn. - `fmx-respond` - load on an `x-mention ` `check:` wake to handle the mention, on an `x-mode-error ...` `check:` wake to report the Relay configuration blocker, on a `public-followup ...` `check:` wake or a startup-surfaced public commitment, and on any milestone or terminal wake for a Relay-linked task before posting its completion follow-up; relevant only when Relay is on. diff --git a/.agents/skills/operational-home-layout/SKILL.md b/.agents/skills/operational-home-layout/SKILL.md index 2786265f63b..49c728d93af 100644 --- a/.agents/skills/operational-home-layout/SKILL.md +++ b/.agents/skills/operational-home-layout/SKILL.md @@ -24,6 +24,7 @@ config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "de config/claude-permission-mode optional one-token permission posture for every Claude worker launch: absent or "bypass" keeps --dangerously-skip-permissions, "auto" launches with --permission-mode auto; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Claude permission mode" config/claude-account config/pi-account optional per-home worker account pin for Claude and Pi launches; LOCAL, gitignored, not inherited; absent keeps today's ambient account; present refuses a launch unless the pinned account resolves and is signed in (section 4 owns the refusal rule); see docs/configuration.md "Worker account pin" config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes +config/project-capacity optional per-machine count of workers each named project admits at once, read from the root home by every local home; LOCAL, gitignored; see docs/configuration.md "Project capacity" config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = the configured tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), herdr has its own required CI lane (docs/herdr-backend.md), while zellij, orca, and cmux remain experimental with no dedicated real-backend CI lane (docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning diff --git a/.opencode/plugins/fm-primary-watch-arm.js b/.opencode/plugins/fm-primary-watch-arm.js index a846736308b..2836288b5d8 100644 --- a/.opencode/plugins/fm-primary-watch-arm.js +++ b/.opencode/plugins/fm-primary-watch-arm.js @@ -1,5 +1,5 @@ import { spawn, spawnSync } from "node:child_process"; -import { existsSync, readFileSync, readdirSync, realpathSync } from "node:fs"; +import { existsSync, readFileSync, realpathSync } from "node:fs"; import { resolve } from "node:path"; import { encodeFirstmateOperationalInput } from "./lib/fm-operational-input.js"; @@ -117,14 +117,32 @@ async function isPrimaryRoot(root, home) { return gitDir.stdout.trim() === commonDir.stdout.trim(); } +// bin/fm-supervision-lib.sh's fm_supervision_needed is the single owner of the +// arm condition set (the turn-end guard decides with the same shared +// predicate), so this plugin can never disagree with the guard again. Away +// mode stays a local decline: its daemon owns supervision. X-mode homes arm +// before their relay poll is registered in the state directory. function shouldArm(paths) { if (existsSync(`${paths.state}/.afk`)) return false; if (existsSync(`${paths.config}/x-mode.env`)) return true; - try { - return readdirSync(paths.state).some((name) => name.endsWith(".meta")); - } catch { - return false; - } + return supervisionNeeded(paths); +} + +// fm_supervision_needed exits 0 exactly when the shared predicate +// says the home needs supervision; exit 0 means arm here. +function supervisionNeeded(paths) { + const result = spawnSync( + "bash", + [ + "-c", + '. "$1/bin/fm-supervision-lib.sh" && fm_supervision_needed "$2"', + "fm-primary-watch-arm", + paths.root, + paths.state, + ], + { stdio: "ignore" }, + ); + return result.status === 0; } async function sessionOwnsLock(paths) { diff --git a/AGENTS.md b/AGENTS.md index 9a86dd21e97..4a2d2c8138d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,6 @@ Tracked files hold shared instructions and tooling; `data/` holds durable privat Load `operational-home-layout` when locating, interpreting, or changing Firstmate home, config, data, state, project, or generated runtime paths. - A `state/.status` line is a wake event, not current-state truth; `bin/fm-crew-state.sh` owns current-state reconciliation. Treat `data/captain.md` as the domain-local record of captain preferences, optional `data/captain-shared.md` as the main-authoritative shared captain-preference file for secondmate inheritance, and `data/learnings.md` as curated home-local knowledge, regardless of harness memory. @@ -196,6 +195,7 @@ An unregistered project or absent registry resolves to `no-mistakes` with yolo o Record the resulting mode, `yolo` merge posture, and the one-line reason for any deviation in the backlog item note. Treat file or subsystem overlap as a risk signal rather than an automatic reason to wait, and dispatch isolated work immediately with no concurrency cap when each change can be independently implemented and validated and the selected delivery path can reconcile ordinary rebases or conflicts. +A project's declared machine capacity (`config/project-capacity`) still bounds that dispatch: a spawn beyond it exits 75 without launching, and its item stays queued rather than blocked. Serialize only for a true semantic dependency, shared mutable external state, incompatible concurrent migration, or another concrete condition that makes independent progress or reconciliation unsafe; same-file editing alone is insufficient, and genuine blockers remain durable. Write the task-specific brief under section 11 before spawning. Fill the task subsections according to section 11. @@ -367,7 +367,7 @@ A decision is simply a task held for the captain: create the task with `bin/fm-t When a main-side thread such as a pending captain decision or relay reminder is worth durable tracking, file it as its own work item and hold it through that wrapper. Captain calls discovered by investigations or visual reviews follow `captain-hold-lifecycle`, which owns their completion gate and recorded-answer rules. When the automatic transition gate applies, dispatch and completion move the item themselves - `bin/fm-spawn.sh` and `bin/fm-teardown.sh` own those transitions and refuse rather than report success without them - so what remains yours is filing the item before dispatch, recording decisions, and keeping notes current; `docs/configuration.md` owns gate applicability and the manual-backend exception. -Re-evaluate queued work after every teardown and heartbeat, dispatching items only when dependencies and time gates have cleared. +Re-evaluate queued work after every teardown and heartbeat, and also after a recorded PR-ready handoff when `config/project-capacity` caps that project, dispatching items only when dependencies, time gates, and project capacity have cleared. `.tasks.toml`, `docs/configuration.md`, and current `tasks-axi --help` own the backlog schema, compatibility, retention, and routine command syntax. Use compatible `tasks-axi` when the configured backend selects it, always through `bin/fm-tasks-axi.sh` so the call reaches this home's backlog from any directory, and the documented manual path otherwise; keep only the configured recent Done entries. diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index 15ae10a5800..b17522a4a69 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -773,14 +773,26 @@ fm_backend_herdr_projection_workspace_label() { # printf '└ %s · p:%s' "$(fm_backend_herdr_projection_concise_task_label "$1")" "$2" } -# fm_backend_herdr_presentation_session_lock_path: one machine-private lock +# fm_backend_herdr_presentation_session_lock_path: one account-private lock # path per live named Herdr session/socket, shared across every Firstmate home -# that uses that session. +# of this OS account that uses that session. # The path is never under any one home's state/ and secondmates never write the # primary home. Returns non-zero when the named session's socket cannot be # resolved unambiguously. +# The namespace directory is suffixed with this account's uid, so another OS +# account on the same host can never create it first by ordinary use and lock +# this account out; a deliberately pre-created name still fails the ownership +# and mode checks below and is refused, never adopted, chowned, or removed. +# The uid rather than $XDG_RUNTIME_DIR names it because that variable can differ +# or be absent between login contexts of one account, which would split one +# session's lock across processes. fm_backend_herdr_presentation_lock_namespace() { - printf '%s' '/tmp/firstmate-herdr-presentation' + local uid + uid=$(id -u 2>/dev/null) || return 1 + case "$uid" in + ''|*[!0-9]*) return 1 ;; + esac + printf '/tmp/firstmate-herdr-presentation-%s' "$uid" } fm_backend_herdr_presentation_lock_namespace_mode() { diff --git a/bin/fm-pending-reply-lib.sh b/bin/fm-pending-reply-lib.sh index dc5b96a96f7..ce189b4cb50 100755 --- a/bin/fm-pending-reply-lib.sh +++ b/bin/fm-pending-reply-lib.sh @@ -5,8 +5,9 @@ # to a secondmate, this library records a durable parent-owned pending-reply # expectation BEFORE delivery, embeds a privacy-safe correlation id in the # outbound message, and later resolves that expectation only from a correlated -# parent status line or status-pointed document - never from transport success, -# chat content, or unrelated status activity. +# line in the asked task's own parent status log, or a document it points to - +# never from transport success, chat content, unrelated status activity, or +# another task's line that echoes or quotes the token. # # Safety property (captain direction 2026-07-22): a secondmate agent may ignore # the marker and answer only in its visible conversation. The parent must notice @@ -194,14 +195,12 @@ fm_pending_reply_extract_corr() { # printf '%s' "$text" | grep -oE "$FM_PENDING_REPLY_CORR_RE" 2>/dev/null | head -1 | cut -d= -f2- | tr 'A-F' 'a-f' || true } -# 0 if carries the exact correlation token for . +# 0 if carries the exact correlation token for , as a whole +# word: xcorr= or corr=ff is a different token, not this one. fm_pending_reply_text_has_corr() { # - local text=$1 corr=$2 token - token=$(fm_pending_reply_corr_token "$corr") - case "$text" in - *"$token"*) return 0 ;; - esac - return 1 + local text=$1 corr=$2 re + re="(^|[^[:alnum:]_])$(fm_pending_reply_corr_token "$corr")([^[:alnum:]_]|\$)" + [[ $text =~ $re ]] } # Sanitize a short request summary: single line, bounded, no control chars. @@ -689,7 +688,13 @@ _fm_pending_reply_try_resolve_locked() { # [status-file-o case "$delivery_state" in attempted|confirmed) ;; *) return 1 ;; esac unconfirmed=1 fi - status_file=${status_override:-$(fm_pending_reply_get "$rec" parent_status)} + status_file=$(fm_pending_reply_get "$rec" parent_status) + # Only the asked task's own status log answers its request: another mate's + # line echoing or quoting this corr= token must leave the request open. + if [ -n "$status_override" ]; then + [ "$status_override" -ef "$status_file" ] || return 1 + status_file=$status_override + fi if [ -z "$status_override" ] && [ "$unconfirmed" = 0 ]; then signature=$(fm_pending_reply_file_signature "$status_file") previous=$(fm_pending_reply_get "$rec" parent_status_scan_signature) diff --git a/bin/fm-pr-check.sh b/bin/fm-pr-check.sh index b7c7469a2c5..722dcb763f9 100755 --- a/bin/fm-pr-check.sh +++ b/bin/fm-pr-check.sh @@ -17,6 +17,8 @@ # draft state does not refuse, matching how the head read below is optional. # bin/fm-pr-merge.sh records through this script with FM_PR_CHECK_MERGE=1 and # skips this refusal, because its own merge-time draft refusal is authoritative. +# The recorded pr= also frees the task's place in a declared project capacity +# (bin/fm-project-capacity-lib.sh). # # --team-review records whether the task's already-recorded PR is out with the # project's human reviewers, as team_review= bound to that URL by diff --git a/bin/fm-procevent-lavish.sh b/bin/fm-procevent-lavish.sh index 4442c73fe3c..1ec8511ba67 100755 --- a/bin/fm-procevent-lavish.sh +++ b/bin/fm-procevent-lavish.sh @@ -21,8 +21,10 @@ # It is read-only over the capture: it does not arm, poll, or change # what Lavish delivered. The freeform message (tag=message) is its # own labeled field, printed first and distinct from per-element -# annotations; it is labeled SESSION-ENDING MESSAGE only when the -# session ended. Declared and presented item counts, +# annotations; it is labeled SESSION-ENDING MESSAGE, and counted as +# session_ending_message_count, only when the session ended, and is +# otherwise CAPTAIN MESSAGE and captain_message_count. Declared and +# presented item counts, # plus a completeness verdict, follow before all annotations so a # partial read is obvious. Each annotation retains its element uid, # selector, tag, and text. A non-choice freeform comment (`prompt`) @@ -783,9 +785,9 @@ cmd_read() { return if !@lines || (@lines == 1 && $lines[0] eq ""); print "| $_\n" for @lines; } + my $ended = $session_ended =~ /^(?:true|True|TRUE)$/; if (@messages) { - my $message_label = $session_ended =~ /^(?:true|True|TRUE)$/ - ? "SESSION-ENDING MESSAGE" : "CAPTAIN MESSAGE"; + my $message_label = $ended ? "SESSION-ENDING MESSAGE" : "CAPTAIN MESSAGE"; print "$message_label\n"; for my $i (0 .. $#messages) { print "$message_label PART ", ($i + 1), " of ", scalar(@messages), "\n" if @messages > 1; @@ -806,7 +808,8 @@ cmd_read() { print "lifecycle: $lifecycle\n"; print "session_ended: ", (length $session_ended ? $session_ended : "(unset)"), "\n"; print "annotation_count: ", scalar(@annotations), "\n"; - print "session_ending_message_count: ", scalar(@messages), "\n"; + my $message_count_key = $ended ? "session_ending_message_count" : "captain_message_count"; + print "$message_count_key: ", scalar(@messages), "\n"; print "\n"; if (@annotations) { print "ANNOTATIONS\n"; diff --git a/bin/fm-project-capacity-lib.sh b/bin/fm-project-capacity-lib.sh new file mode 100644 index 00000000000..a184c3d28a9 --- /dev/null +++ b/bin/fm-project-capacity-lib.sh @@ -0,0 +1,229 @@ +#!/usr/bin/env bash +# fm-project-capacity-lib.sh - how many workers a project admits at once on this +# machine, and whether a fresh worker spawn still fits. +# +# A project can depend on a machine-local resource that only a few workers can +# use at the same time: a heavy test suite, a local editor stack, a device. +# Firstmate cannot see which part of a worker's life touches that resource, so +# the captain declares how many workers the project admits on this machine, and +# bin/fm-spawn.sh defers a fresh worker beyond that number instead of launching +# it only to spend full-context turns retrying the resource. A deferred task +# keeps its queued backlog item and is dispatched again when a place frees. +# Without a declaration nothing changes and dispatch stays uncapped +# (AGENTS.md section 7). +# +# This file is the single owner of the declaration format, of what holds a +# place, and of the admission verdict. docs/configuration.md "Project capacity" +# is the operator reference, and bin/fm-spawn.sh owns where the check runs. +# +# Declaration: config/project-capacity in the local root Firstmate home (the home +# bin/fm-wake-lib.sh's fm_firstmate_root_home resolves), so every home on this +# machine reads the same number for the same machine's resources. One line per +# project: +# +# is the project's registered name, which is the basename of its +# clone directory, and is a positive integer of at most six digits. +# The capacity is the last whitespace-separated field, so the name before it may +# contain spaces. Blank lines are ignored. A line is a comment when it is `#`, +# when `#` is followed by whitespace, or when it starts with `#` and its last +# field is not an integer. A project whose name begins with `#` is declared by +# writing that `#` immediately against the rest of the name and ending the line +# with the capacity, for example `#repo 2`. A name that is `#`, or that begins +# with `#` followed by a space, is the same spelling as a comment and cannot be +# declared. Any other shape, a project named twice, or an unreadable file makes +# the whole declaration unreadable, and bin/fm-spawn.sh then refuses every fresh +# ship or scout spawn from this machine's homes rather than guessing which limit +# was meant. +# +# Occupancy: a place is held by every task record, in any local Firstmate home on +# this machine (fm_local_firstmate_state_dirs in bin/fm-wake-lib.sh), that +# - is not a secondmate, which is a persistent home rather than a worker, +# - names the same project identity, meaning its project resolves to the same +# shared project lock path (fm_treehouse_project_lock_path), which is keyed +# by the project's resolved origin, so workers in any clone of that origin +# are counted, and +# - has no recorded PR handoff: the pr= line bin/fm-pr-check.sh records when a +# worker's PR is ready, after which the worker waits on review or merge and +# no longer uses local resources. +# A place therefore frees when a PR-based ship records its ready PR, or when any +# task is cleaned up and its record removed. A local-only ship and a scout have +# no recorded handoff and hold their place until cleanup. A worker that is +# steered back into work after its PR handoff is not counted again. +# The declaration is matched by the spawning clone's directory name, so clones +# of one origin share the cap only when they use that same directory name. A +# clone of that origin under a different directory name finds no declaration +# and is not capped, though its workers still count as holders for a +# same-origin clone that is capped. +# A record whose project directory no longer exists cannot be matched and holds +# no place. Remote homes are never walked, because their workers run on another +# machine. +# +# Race safety: bin/fm-spawn.sh evaluates admission while holding the shared +# project lock and keeps holding it until the new task record is published, so +# two concurrent spawns for one project can never both publish from the same +# count. A spawn on an uncapped project can still publish a holder for a capped +# same-origin clone, so every backend takes that lock whenever the declaration +# caps any project; Orca, which otherwise never takes it, is included. Freeing a +# place needs no lock, because removing a record or adding pr= only ever lowers +# the count. +# +# Requires bin/fm-wake-lib.sh (root home, local homes, project lock path), +# bin/fm-secondmate-registry-lib.sh (which the local-homes walk reads), and +# bin/fm-backend.sh (fm_meta_get) to be sourced first. No side effects on source. + +# Exit status of a spawn deferred because the project is at capacity: the +# sysexits "temporary failure" code, so a caller can tell a deferral that leaves +# the task queued from an ordinary failure. +# shellcheck disable=SC2034 # read by bin/fm-spawn.sh after sourcing. +FM_PROJECT_CAPACITY_DEFER_EXIT=75 + +# The config directory holding this machine's declaration: the spawning home's +# own when that home is the local root (so an override of it +# applies), otherwise the root home's config/. +fm_project_capacity_config_dir() { # + local home=$1 config=$2 root home_real + root=$(fm_firstmate_root_home "$home") || return 1 + home_real=$(CDPATH='' cd -- "$home" 2>/dev/null && pwd -P) || return 1 + if [ "$root" = "$home_real" ]; then + printf '%s\n' "$config" + else + printf '%s/config\n' "$root" + fi +} + +# Read the declared capacity for one project. +# Sets FM_PROJECT_CAPACITY_FILE to the declaration path, FM_PROJECT_CAPACITY +# to the project's capacity, or to empty when the project declares none, and +# FM_PROJECT_CAPACITY_ANY to 1 when the declaration caps any project at all. +# Returns 1 with FM_PROJECT_CAPACITY_ERROR when the declaration is unreadable. +fm_project_capacity_lookup() { # + local name=$2 line lineno=0 pname pcap seen='|' + FM_PROJECT_CAPACITY_FILE="$1/project-capacity" + FM_PROJECT_CAPACITY= + FM_PROJECT_CAPACITY_ANY= + FM_PROJECT_CAPACITY_ERROR= + if [ ! -e "$FM_PROJECT_CAPACITY_FILE" ] && [ ! -L "$FM_PROJECT_CAPACITY_FILE" ]; then + return 0 + fi + if [ ! -f "$FM_PROJECT_CAPACITY_FILE" ] || [ ! -r "$FM_PROJECT_CAPACITY_FILE" ]; then + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE is not a readable regular file" + return 1 + fi + while IFS= read -r line || [ -n "$line" ]; do + lineno=$((lineno + 1)) + line=${line%$'\r'} + line=${line#"${line%%[![:space:]]*}"} + line=${line%"${line##*[![:space:]]}"} + # '#' followed by whitespace is always a comment, including one that ends + # with a number. A line that begins with '#' glued to the rest of a name is + # a declaration only when its last field is an integer; any other such line + # stays a comment, so a note does not refuse every spawn. + case "$line" in + '' | '#' | '#'[[:space:]]*) continue ;; + '#'*) + case "${line##*[[:space:]]}" in + *[!0-9]*) continue ;; + esac + ;; + esac + # The capacity is the last field, so the name before it may hold spaces. + pcap=${line##*[[:space:]]} + pname=${line%"$pcap"} + pname=${pname%"${pname##*[![:space:]]}"} + if [ -z "$pname" ]; then + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno is not ' '" + FM_PROJECT_CAPACITY= + return 1 + fi + case "$pcap" in + '' | *[!0-9]* | 0*) + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno gives $pname a capacity that is not a positive integer" + FM_PROJECT_CAPACITY= + return 1 + ;; + esac + if [ "${#pcap}" -gt 6 ]; then + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno gives $pname a capacity longer than six digits" + FM_PROJECT_CAPACITY= + return 1 + fi + case "$seen" in + *"|$pname|"*) + FM_PROJECT_CAPACITY_ERROR="$FM_PROJECT_CAPACITY_FILE line $lineno names $pname a second time" + FM_PROJECT_CAPACITY= + return 1 + ;; + esac + seen="$seen$pname|" + [ "$pname" != "$name" ] || FM_PROJECT_CAPACITY=$pcap + done < "$FM_PROJECT_CAPACITY_FILE" + [ "$seen" = '|' ] || FM_PROJECT_CAPACITY_ANY=1 + return 0 +} + +# Count the task records holding a place in one project's capacity. +# is fm_treehouse_project_lock_path for the project being +# admitted, and is that project's own directory, which matches +# without recomputing its identity. The local homes come from +# fm_local_firstmate_state_dirs . is the task being +# admitted; its own record in is the one this spawn replaces, so +# it is not counted. +# Sets FM_PROJECT_CAPACITY_OCCUPANTS to the count and +# FM_PROJECT_CAPACITY_OCCUPANT_IDS to a comma-separated list of the holders, +# each outside qualified with its home. Returns 1 with +# FM_PROJECT_CAPACITY_ERROR when the local homes cannot be enumerated, or when +# a state directory or task record in them cannot be read, since skipping it +# could undercount the holders. +fm_project_capacity_occupants() { # + local want=$1 own=$2 first=$3 self=$4 state meta kind project lock id label i + local -a cache_dirs cache_locks + FM_PROJECT_CAPACITY_OCCUPANTS=0 + FM_PROJECT_CAPACITY_OCCUPANT_IDS= + FM_PROJECT_CAPACITY_ERROR= + fm_local_firstmate_state_dirs "$first" || { + FM_PROJECT_CAPACITY_ERROR=$FM_LOCAL_FIRSTMATE_ERROR + return 1 + } + cache_dirs=("$own") + cache_locks=("$want") + for state in "${FM_LOCAL_FIRSTMATE_STATES[@]}"; do + if [ -e "$state" ] && { [ ! -d "$state" ] || [ ! -r "$state" ] || [ ! -x "$state" ]; }; then + FM_PROJECT_CAPACITY_ERROR="local Firstmate state directory $state cannot be read" + return 1 + fi + for meta in "$state"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + [ "$meta" != "$first/$self.meta" ] || continue + [ -r "$meta" ] || { + FM_PROJECT_CAPACITY_ERROR="task record $meta cannot be read" + return 1 + } + kind=$(fm_meta_get "$meta" kind) + [ "$kind" != secondmate ] || continue + [ -z "$(fm_meta_get "$meta" pr)" ] || continue + project=$(fm_meta_get "$meta" project) + [ -n "$project" ] || continue + lock= + i=0 + while [ "$i" -lt "${#cache_dirs[@]}" ]; do + if [ "${cache_dirs[$i]}" = "$project" ]; then + lock=${cache_locks[$i]} + break + fi + i=$((i + 1)) + done + if [ "$i" -ge "${#cache_dirs[@]}" ]; then + lock=$(fm_treehouse_project_lock_path "$project" 2>/dev/null) || lock= + cache_dirs+=("$project") + cache_locks+=("$lock") + fi + [ -n "$lock" ] && [ "$lock" = "$want" ] || continue + id=$(basename "$meta" .meta) + label=$id + [ "$state" = "$first" ] || label="$id in $(dirname "$state")" + FM_PROJECT_CAPACITY_OCCUPANTS=$((FM_PROJECT_CAPACITY_OCCUPANTS + 1)) + FM_PROJECT_CAPACITY_OCCUPANT_IDS="${FM_PROJECT_CAPACITY_OCCUPANT_IDS:+$FM_PROJECT_CAPACITY_OCCUPANT_IDS, }$label" + done + done + return 0 +} diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index d75cf7936f5..6f9a958b85e 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -188,6 +188,20 @@ # fm_firstmate_root_home resolves, so a home seeded from another machine anchors # that lock itself rather than failing to resolve one; # contention refuses rather than waits. +# Project capacity: when this machine declares how many workers a project +# admits at once (config/project-capacity; bin/fm-project-capacity-lib.sh owns +# the declaration, what holds a place, and the race argument), a fresh ship or +# scout spawn counts the places already held while holding that same +# project-identity lock - taken on every backend whenever the declaration caps +# any project, Orca included, because an uncapped clone's worker still holds a +# place for a capped clone of the same origin - and holds it through metadata +# publication. A spawn that finds +# every place held prints one `deferred:` line and exits 75 before any brief +# render, endpoint, worktree, record, or backlog move exists, so the task stays +# exactly as queued as it was; an unreadable declaration refuses with exit 1. +# A batch reports such a pair as `batch: DEFERRED` and exits 75 when nothing +# else failed. A relaunch and a --secondmate spawn are never counted against +# capacity. # With no harness arg, a crewmate/scout spawn resolves the CREW harness only when # config/crew-dispatch.json is absent. When that file exists, crewmate/scout # spawns require an explicit harness so firstmate cannot silently skip dispatch @@ -671,6 +685,8 @@ fm_backlog_directory_present "$STATE" "state directory" || { . "$SCRIPT_DIR/fm-remote-readiness-lib.sh" # shellcheck source=bin/fm-timeout-lib.sh . "$SCRIPT_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-project-capacity-lib.sh +. "$SCRIPT_DIR/fm-project-capacity-lib.sh" # shellcheck source=bin/fm-worker-account-lib.sh . "$SCRIPT_DIR/fm-worker-account-lib.sh" # Fail closed before any fleet mutation: a no-mistakes gate agent must never spawn @@ -1587,16 +1603,17 @@ if [ "${#POS[@]}" -gt 0 ] && [ "${POS[0]}" != "$idpart" ] && case "$idpart" in * echo "error: batch dispatch does not support --secondmate; spawn each secondmate explicitly" >&2 rc=2 continue - elif [ "$KIND" = scout ]; then - if FM_SPAWN_NO_GUARD=1 "$FM_ROOT/bin/fm-spawn.sh" "${pair%%=*}" "${pair#*=}" "${shared_args[@]+"${shared_args[@]}"}" --scout; then :; else - echo "batch: FAILED to spawn ${pair%%=*} (${pair#*=})" >&2 - rc=1 - fi - else - if FM_SPAWN_NO_GUARD=1 "$FM_ROOT/bin/fm-spawn.sh" "${pair%%=*}" "${pair#*=}" "${shared_args[@]+"${shared_args[@]}"}"; then :; else - echo "batch: FAILED to spawn ${pair%%=*} (${pair#*=})" >&2 - rc=1 - fi + fi + pair_args=("${pair%%=*}" "${pair#*=}" "${shared_args[@]+"${shared_args[@]}"}") + [ "$KIND" != scout ] || pair_args+=(--scout) + pair_rc=0 + FM_SPAWN_NO_GUARD=1 "$FM_ROOT/bin/fm-spawn.sh" "${pair_args[@]}" || pair_rc=$? + if [ "$pair_rc" -eq "$FM_PROJECT_CAPACITY_DEFER_EXIT" ]; then + echo "batch: DEFERRED ${pair%%=*} (${pair#*=}) - its project is at capacity, so it stays queued" >&2 + [ "$rc" -ne 0 ] || rc=$FM_PROJECT_CAPACITY_DEFER_EXIT + elif [ "$pair_rc" -ne 0 ]; then + echo "batch: FAILED to spawn ${pair%%=*} (${pair#*=})" >&2 + rc=1 fi done exit "$rc" @@ -3133,17 +3150,52 @@ else WT="" BRIEF="$DATA/$ID/brief.md" fi -if [ "$RELAUNCH" -eq 0 ] && [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then +# Project capacity admission (bin/fm-project-capacity-lib.sh owns the +# declaration, what holds a place, and why this is race-safe). A fresh worker +# for a project whose declared capacity is already held is deferred here, before +# any brief render, endpoint, worktree, record, or backlog move exists, so the +# deferral leaves the task exactly as queued as it was. A relaunch replaces a +# worker that already holds a place, and a secondmate is not a worker. +SPAWN_PROJECT_CAPACITY= +SPAWN_PROJECT_CAPACITY_ANY= +if [ "$RELAUNCH" -eq 0 ] && [ "$KIND" != secondmate ]; then + SPAWN_CAPACITY_CONFIG=$(fm_project_capacity_config_dir "$FM_HOME" "$CONFIG") || { + echo "error: could not resolve the root Firstmate home that declares project capacity for $PROJ_ABS" >&2 + exit 1 + } + if ! fm_project_capacity_lookup "$SPAWN_CAPACITY_CONFIG" "$(basename "$PROJ_ABS")"; then + echo "error: spawn refused: the project capacity declaration is unreadable ($FM_PROJECT_CAPACITY_ERROR); fix it so the captain's worker limits are known (docs/configuration.md \"Project capacity\")" >&2 + exit 1 + fi + SPAWN_PROJECT_CAPACITY=$FM_PROJECT_CAPACITY + SPAWN_PROJECT_CAPACITY_ANY=$FM_PROJECT_CAPACITY_ANY +fi +if [ "$RELAUNCH" -eq 0 ] && [ "$KIND" != secondmate ] && + { [ "$BACKEND" != orca ] || [ -n "$SPAWN_PROJECT_CAPACITY_ANY" ]; }; then SPAWN_TREEHOUSE_PROJECT_LOCK=$(fm_treehouse_project_lock_path "$PROJ_ABS") || { echo "error: could not resolve the shared Treehouse project lock for $PROJ_ABS" >&2 exit 1 } if ! fm_lock_try_acquire "$SPAWN_TREEHOUSE_PROJECT_LOCK"; then - echo "error: another Treehouse slot allocation or return is in progress for $PROJ_ABS; refusing to race it" >&2 + if [ "$BACKEND" = orca ]; then + echo "error: another spawn or cleanup holds the shared project lock for $PROJ_ABS; refusing to race its capacity admission" >&2 + else + echo "error: another Treehouse slot allocation or return is in progress for $PROJ_ABS; refusing to race it" >&2 + fi exit 1 fi SPAWN_TREEHOUSE_PROJECT_LOCK_HELD=1 fi +if [ -n "$SPAWN_PROJECT_CAPACITY" ]; then + if ! fm_project_capacity_occupants "$SPAWN_TREEHOUSE_PROJECT_LOCK" "$PROJ_ABS" "$STATE" "$ID"; then + echo "error: spawn refused: project $(basename "$PROJ_ABS") declares a capacity of $SPAWN_PROJECT_CAPACITY, but this machine's task records cannot all be read to count it ($FM_PROJECT_CAPACITY_ERROR)" >&2 + exit 1 + fi + if [ "$FM_PROJECT_CAPACITY_OCCUPANTS" -ge "$SPAWN_PROJECT_CAPACITY" ]; then + echo "deferred: project $(basename "$PROJ_ABS") admits $SPAWN_PROJECT_CAPACITY worker(s) at once on this machine ($FM_PROJECT_CAPACITY_FILE) and $FM_PROJECT_CAPACITY_OCCUPANTS already hold a place ($FM_PROJECT_CAPACITY_OCCUPANT_IDS); task $ID was not launched and its backlog item stays queued - dispatch it again once one of them records its ready PR or is cleaned up" >&2 + exit "$FM_PROJECT_CAPACITY_DEFER_EXIT" + fi +fi [ -f "$BRIEF" ] || { echo "error: task $ID has no brief at inaccessible data path $BRIEF" >&2 exit 1 diff --git a/bin/fm-supervision-host.sh b/bin/fm-supervision-host.sh index 26555b69633..4c065009198 100755 --- a/bin/fm-supervision-host.sh +++ b/bin/fm-supervision-host.sh @@ -464,12 +464,17 @@ start_arm() { # [--restart]; sets the started pi local predecessor=$1 out pid shift out=$(mktemp "$STATE/.supervision-host-arm.XXXXXX") || return 1 + # An arm left for main outlives this host, and Claude tears the hook's + # process group down after the exit-2 rewake, so it gets a group of its own + # (the shape start_handling_successor in bin/fm-claude-stop-autoarm.sh uses). + [ "${ARM_OWN_GROUP:-0}" -ne 1 ] || set -m 2>/dev/null || true if [ -n "$predecessor" ]; then - FM_WATCH_PREDECESSOR_ARM_PID=$predecessor FM_GUARD_GRACE="$GRACE" "$SCRIPT_DIR/fm-watch-arm.sh" "$@" >"$out" 2>&1 & + FM_WATCH_PREDECESSOR_ARM_PID=$predecessor FM_GUARD_GRACE="$GRACE" "$SCRIPT_DIR/fm-watch-arm.sh" "$@" >"$out" 2>&1 "$out" 2>&1 & + FM_GUARD_GRACE="$GRACE" "$SCRIPT_DIR/fm-watch-arm.sh" "$@" >"$out" 2>&1 /dev/null || true record_process arm "$pid" STARTED_ARM_PID=$pid STARTED_ARM_OUT=$out @@ -703,7 +708,7 @@ detach_successor() { # downtime (autoarm_commit in bin/fm-claude-stop-autoarm.sh). A failed start # returns 1; the caller still prints the close unchanged. leave_successor_for_main() { - if ! start_successor "$CLOSED_ARM_PID"; then + if ! ARM_OWN_GROUP=1 start_successor "$CLOSED_ARM_PID"; then log_line "pass-through successor-unverified $(printf '%s\n' "$REASON" | head -n 1)" return 1 fi @@ -1116,7 +1121,7 @@ while :; do # A turn that could outlive the boundary would outlive the hook registration. turn_crosses_boundary && boundary_exit - if ! start_successor "$CLOSED_ARM_PID"; then + if ! ARM_OWN_GROUP=1 start_successor "$CLOSED_ARM_PID"; then exit_to_main "the successor watcher cycle could not be verified before handling; this wake is yours" fi if [ -n "$SUCCESSOR_GENERATION" ]; then diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index e611d33bfde..1e9952ff599 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -2339,50 +2339,15 @@ teardown_live_slot_path() { canonical_existing_dir "$WT" } +# Every local Firstmate state directory whose records can name a pool slot this +# task's slot might also be; bin/fm-wake-lib.sh's fm_local_firstmate_state_dirs +# owns the walk and what it refuses. collect_local_firstmate_states() { - local record_state=$1 root home reg line child known existing i=0 - local -a homes - TREEHOUSE_OWNER_STATES=("$record_state") - root=$(fm_firstmate_root_home "$FM_HOME") || { - echo "REFUSED: cannot resolve the root Firstmate home; nothing was changed" >&2 + fm_local_firstmate_state_dirs "$1" || { + echo "REFUSED: $FM_LOCAL_FIRSTMATE_ERROR; nothing was changed" >&2 return 1 } - homes=("$root") - while [ "$i" -lt "${#homes[@]}" ]; do - home=${homes[$i]} - i=$((i + 1)) - known=0 - for existing in "${TREEHOUSE_OWNER_STATES[@]}"; do - [ "$existing" != "$home/state" ] || known=1 - done - [ "$known" = 1 ] || TREEHOUSE_OWNER_STATES+=("$home/state") - reg="$home/data/secondmates.md" - [ ! -e "$reg" ] && [ ! -L "$reg" ] && continue - [ -f "$reg" ] && [ ! -L "$reg" ] || { - echo "REFUSED: local Firstmate registry is unsafe at $reg; nothing was changed" >&2 - return 1 - } - while IFS= read -r line || [ -n "$line" ]; do - case "$line" in - "- "*) - secondmate_registry_parse_line "$line" || { - echo "REFUSED: malformed local Firstmate registry entry in $reg; nothing was changed" >&2 - return 1 - } - [ "$SECONDMATE_REGISTRY_REMOTE" -eq 0 ] || continue - child=$(canonical_existing_dir "$SECONDMATE_REGISTRY_HOME") || { - echo "REFUSED: registered local Firstmate home is unavailable: $SECONDMATE_REGISTRY_HOME; nothing was changed" >&2 - return 1 - } - known=0 - for existing in "${homes[@]}"; do - [ "$existing" != "$child" ] || known=1 - done - [ "$known" = 1 ] || homes+=("$child") - ;; - esac - done < "$reg" - done + TREEHOUSE_OWNER_STATES=("${FM_LOCAL_FIRSTMATE_STATES[@]}") } require_exclusive_worktree_slot_record() { diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 2edc34947a5..348a7d3af08 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -393,6 +393,7 @@ family_for_basename() { fm-trace-context-spawn.test.sh|fm-spawn-worktree-settle.test.sh|\ fm-spawn-compact-adviser-disable.test.sh|\ fm-spawn-compact-adviser-disable-remote.test.sh|\ + fm-project-capacity.test.sh|\ fm-teardown-endpoint-safety.test.sh) printf '%s\n' backend-dispatch ;; diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index ddc708aadc8..c4d51a24731 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1432,10 +1432,10 @@ fm_task_set_lock_path() { # # the walk at the current home, which is the correct answer rather than an # error: the parent lives on another machine, so its filesystem can neither hold # nor be observed by a lock taken here, and a remote-seeded home is itself the -# top of the local tree that bin/fm-teardown.sh's collect_local_firstmate_states -# enumerates (that walk already skips remote registry entries for the same -# reason). Refusing a remote binding instead made every operation anchored here -# fail closed inside a remote secondmate home and its local descendants. +# top of the local tree that fm_local_firstmate_state_dirs below enumerates +# (that walk already skips remote registry entries for the same reason). +# Refusing a remote binding instead made every operation anchored here fail +# closed inside a remote secondmate home and its local descendants. # # Everything else still fails closed: an unreadable or malformed binding, an # unreachable local parent, a cycle, and a chain deeper than the bound. @@ -1464,7 +1464,72 @@ fm_firstmate_root_home() { printf '%s\n' "$home" } -# The one lock serializing Treehouse slot allocation and return for a project. +# Every Firstmate state directory on THIS machine whose task records can share a +# machine-local resource with : itself, then the local +# root home and each local secondmate home registered below it, walked through +# every data/secondmates.md breadth-first. Remote registry entries are skipped, +# because their workers run on another machine. +# +# Sets FM_LOCAL_FIRSTMATE_STATES to that list, first and without +# duplicates however each directory is spelled. Returns 1 with +# FM_LOCAL_FIRSTMATE_ERROR naming what could not be proved - an unresolvable +# root, an unsafe or malformed registry, or an unavailable registered local +# home - so a caller refuses rather than treating an unreadable home as one +# with no tasks. Requires bin/fm-secondmate-registry-lib.sh to be sourced first. +# shellcheck disable=SC2034 # FM_LOCAL_FIRSTMATE_ERROR is read by callers. +fm_local_firstmate_state_dirs() { # + local first=$1 root home reg line child known existing i=0 + local -a homes + FM_LOCAL_FIRSTMATE_STATES=("$first") + FM_LOCAL_FIRSTMATE_ERROR= + root=$(fm_firstmate_root_home "$FM_HOME") || { + FM_LOCAL_FIRSTMATE_ERROR="cannot resolve the root Firstmate home" + return 1 + } + homes=("$root") + while [ "$i" -lt "${#homes[@]}" ]; do + home=${homes[$i]} + i=$((i + 1)) + known=0 + for existing in "${FM_LOCAL_FIRSTMATE_STATES[@]}"; do + if [ "$existing" = "$home/state" ] || [ "$existing" -ef "$home/state" ]; then + known=1 + fi + done + [ "$known" = 1 ] || FM_LOCAL_FIRSTMATE_STATES+=("$home/state") + reg="$home/data/secondmates.md" + [ ! -e "$reg" ] && [ ! -L "$reg" ] && continue + [ -f "$reg" ] && [ ! -L "$reg" ] || { + FM_LOCAL_FIRSTMATE_ERROR="local Firstmate registry is unsafe at $reg" + return 1 + } + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + "- "*) + secondmate_registry_parse_line "$line" || { + FM_LOCAL_FIRSTMATE_ERROR="malformed local Firstmate registry entry in $reg" + return 1 + } + [ "$SECONDMATE_REGISTRY_REMOTE" -eq 0 ] || continue + child=$([ -d "$SECONDMATE_REGISTRY_HOME" ] && + CDPATH='' cd -- "$SECONDMATE_REGISTRY_HOME" 2>/dev/null && pwd -P) || { + FM_LOCAL_FIRSTMATE_ERROR="registered local Firstmate home is unavailable: $SECONDMATE_REGISTRY_HOME" + return 1 + } + known=0 + for existing in "${homes[@]}"; do + [ "$existing" != "$child" ] || known=1 + done + [ "$known" = 1 ] || homes+=("$child") + ;; + esac + done < "$reg" + done +} + +# The one lock serializing Treehouse slot allocation and return for a project, +# and project capacity admission (bin/fm-project-capacity-lib.sh), which a +# fresh spawn evaluates under it on every backend. # # It is anchored in the local root home's state directory so that every home on # this machine that can reach the same pool - the root, and each secondmate home diff --git a/docs/configuration.md b/docs/configuration.md index 760c0195672..0eecb1af049 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1061,6 +1061,29 @@ When stripping is enabled, the hooks directory is read-only, so a hook manager r The flag is a home-wide attribution choice, so it is inherited into secondmate homes under the [`secondmate-provisioning`](../.agents/skills/secondmate-provisioning/SKILL.md) inherited-local-material contract and a secondmate's own workers keep AI trailers too. Per-machine Cursor `cli-config.json` attribution-off is not this contract: it does not travel with Firstmate, defaults back to on when unset, and only feeds the CLI's request to the server, so it suppresses the trailer rather than preventing it. +## Project capacity (config/project-capacity) + +The optional local, gitignored `config/project-capacity` tells Firstmate how many workers a project can run at once on this machine, for a project whose machine-local resource - a heavy test suite, a local editor stack, a device - only serves a few workers at a time. +Without it, dispatch stays uncapped as `AGENTS.md` section 7 describes, and a surplus worker is launched only to spend full-context turns waiting for the resource. +The file lives in the machine's root Firstmate home, so every local secondmate home reads the same limit, and it holds one line per project: + +```text +# heavy suite serves two workers +my-project 2 +``` + +The name is the project's registered name, which is its clone directory name and may contain spaces, and the number, the last field on the line, is a positive integer. +A line that is only `#`, or that begins with `#` followed by whitespace, is a comment, as is a `#` line whose last field is not an integer. +A project name may begin with `#` when that `#` is written immediately against the rest of the name and the line ends with the project's capacity. +A name that is `#`, or that begins with `#` and a space, cannot be declared, because that line is a comment. +A place is held by every ship or scout on that project in the root home or any local secondmate home registered under it, including one working in a separate clone of the same origin, until its ready PR is recorded or it is cleaned up; a local-only ship or a scout holds its place until cleanup. +The declaration is matched by the spawning clone's directory name, so clones of the same origin share the cap only when they use that same directory name. +A clone of that origin under a different directory name finds no declaration and is not capped, though its workers are still counted as holders for a same-origin clone that is capped. +When every place is held, `bin/fm-spawn.sh` launches nothing, creates no record, leaves the backlog item queued, prints one `deferred:` line naming the holders, and exits 75, so Firstmate dispatches the item again once a place frees. +A malformed or unreadable file refuses every fresh ship or scout spawn until it is fixed, rather than guessing the intended limit, and so does a local home's state directory or task record that cannot be read while counting a capped project's holders. +Firstmate cannot see which part of a worker's life uses the resource, so the number bounds whole workers from launch to handoff, and the tightest resource every worker needs should decide it. +[`bin/fm-project-capacity-lib.sh`](../bin/fm-project-capacity-lib.sh) owns the file format, what holds a place, and why concurrent spawns cannot both take the last one. + ## Crew dispatch profiles (config/crew-dispatch.json) `config/crew-dispatch.json` is an optional local, gitignored file containing natural-language rules that firstmate reads before dispatching a crewmate or scout. diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index ac111a097a6..0190f1d3d03 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -282,7 +282,7 @@ Projected children are placed in one contiguous block immediately after their ow - The protocol. - The socket. - `python3`. -- The machine-private per-session lock. +- The account-private per-session lock. Existing legacy child labels may extend an already adjacent block read-only but are never renamed or migrated. A foreign, ambiguous, detached, or manually interleaved child makes ordering skip with a warning rather than rewriting the layout. @@ -363,6 +363,8 @@ Once the exact pane is confirmed gone, teardown retires the task's own journal w Recovery is deliberately conservative and presentation-only. An existing journal suppresses another projected create. Before any recovery mutation, Firstmate holds both the task spawn lock and the named-session presentation lock. +That presentation lock lives in a namespace private to the OS account, so another account on the same host running its own Firstmate on Herdr cannot block this account's spawn, recovery, or teardown. +A namespace at this account's name that another account owns, or that is not mode 700, is still refused and is never adopted, chowned, or removed. A same-identity version 2 binding may replace one exact agent-free restart husk in place. A husk is a restored same-labeled tab with a missing pane or no registered agent, as [Restart and liveness behavior](#restart-and-liveness-behavior) describes. diff --git a/docs/remote-secondmates.md b/docs/remote-secondmates.md index 980fbceaf8c..ee60a26903b 100644 --- a/docs/remote-secondmates.md +++ b/docs/remote-secondmates.md @@ -574,7 +574,9 @@ An SSH exit status of 255 while fetching a referenced document leaves the delta The process-event runner applies each captured delta through this adapter as soon as it is captured. So a mirrored reply reaches the primary status channel without depending on the wake handler running the adapter itself. -A mirrored line that carries a correlation token settles its pending-reply record and closes that request's own open escalation decision. +A mirrored line that carries a correlation token settles the pending-reply record only when the mate it came from is the task that request was sent to. +Another mate echoing the token leaves it open. +A settled record also closes that request's own open escalation decision. A remote reply reaches the primary only through this asynchronous mirror. Because of that, the primary treats a missing correlated report as a missed report only once the mirror has been read through the end of the remote log after that turn ended. diff --git a/docs/supervision-host.md b/docs/supervision-host.md index 1f693b69935..5a138d1fa7f 100644 --- a/docs/supervision-host.md +++ b/docs/supervision-host.md @@ -108,7 +108,8 @@ Only an away record is away: no record, or the record daemon-backed quiet mode w The host asks the Pi branch's offer rule (`branchOfferForWake`, through `bin/fm-branch-dispatch.mjs offer`) whether the branch may take the close. So a close reaches main off Pi exactly when it would on Pi: a check trigger, a decision-owned signal or stale trigger, and a scan that is unsafe or holds nothing for the branch stay main's. On that main-only pass-through the host starts the successor watcher cycle and leaves it running, then prints the close unchanged. -It leaves the watcher's recovery marker reading downtime, confirming no handling handoff, because the re-arm owner delivers a close to main only while that marker reads downtime. +That successor, like one a turn hands back at its start (see [Away](#away)), runs in a process group of its own, so the harness tearing down the hook's group after the rewake does not stop it. +The host leaves the watcher's recovery marker reading downtime, confirming no handling handoff, because the re-arm owner delivers a close to main only while that marker reads downtime. The session's next park without `--restart` requests a take-over to restore a single host-owned arm; the [host header](../bin/fm-supervision-host.sh) owns successor persistence and cleanup, and the [arm header](../bin/fm-watch-arm.sh) owns take-over eligibility and fallback. OpenCode and omp still launch the host with `--restart`, which takes precedence over recorded take-over and lacks its acknowledgement-preserving handover; changing that first-cycle path remains a follow-up. The host-off Claude Stop hook's detached handling successor is also unchanged; see [Claude handling successor](watcher-continuity.md#claude-handling-successor). diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 7fe1ba7df17..febc2fd770c 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -1656,7 +1656,7 @@ ok - real herdr: a stale registration no longer blocks relaunch, and the endpoin ok - real herdr: an agent that does not stop fails closed instead of being reported as stopped ``` -The registry read through `herdr pane report-agent` is the same source `fm_backend_herdr_agent_state` classifies, and since 2026-09-10 that registration counts as an agent only while `pane process-info` shows a harness process behind it, so the guard backs the registration with a real process named like a harness (a symlink to `sleep`) and then stops that process, with no real harness launched. +The registry read through `herdr pane report-agent` is the same source `fm_backend_herdr_agent_state` classifies, and since 2026-09-10 that registration counts as an agent only while `pane process-info` shows a harness process behind it, so the guard backs the registration with a real process named like a harness (using `fm_agent_standin` from [`tests/lib.sh`](../../tests/lib.sh)) and then stops that process, with no real harness launched. That command is the guard that refreshes this record; run it after every Herdr upgrade rather than trusting the version above. For Pi on Herdr 0.9.0, `herdr agent get` reflects whether the agent process remains live; its registration does not persist merely because the pane and parent shell do. diff --git a/docs/verification/supervision.md b/docs/verification/supervision.md index 0e4f6eacaea..11bf2525fb9 100644 --- a/docs/verification/supervision.md +++ b/docs/verification/supervision.md @@ -715,7 +715,17 @@ ok - cursor 2026.09.23-86fc751: the tracked registrations mirrored the captain p ok - host mirror live: 2 harness(es) proved their writers ``` -The run above exercised these payload fields: +On 2026-10-07, the Claude path passed again on Linux 7.0.0-34-generic x86_64 with Claude Code 2.1.293 (`haiku`) and a managed policy that displayed `auto mode on` at its idle composer. +The guard ran in its private tmux socket and required both an idle verdict from the Claude-scoped rendered busy-state check and an empty composer before submitting its prompt. + +```text +$ FM_HOST_MIRROR_LIVE_E2E=1 FM_HOST_MIRROR_LIVE_HARNESSES=claude bash tests/fm-host-mirror-live-e2e.test.sh +ok - claude 2.1.293 (Claude Code): a turn the harness started itself was not mirrored as the captain's words +ok - claude 2.1.293 (Claude Code): the tracked registrations mirrored the captain prompt and main reply +ok - host mirror live: 1 harness(es) proved their writers +``` + +The runs above exercised these payload fields: | Primary | Captain text | Main text | | --- | --- | --- | diff --git a/tests/fm-backend-herdr.test.sh b/tests/fm-backend-herdr.test.sh index 01d0a7015f8..8b1945db714 100755 --- a/tests/fm-backend-herdr.test.sh +++ b/tests/fm-backend-herdr.test.sh @@ -686,12 +686,16 @@ test_exhausted_settle_window_keeps_a_non_shell_foreground_live() { } test_registered_agent_with_an_agent_descendant_outside_the_foreground_stays_alive() { - local lab sleep_bin shell_pid out shell_verdict + local lab standin sleep_bin shell_pid out shell_verdict sleep_bin=$(command -v sleep) || fail "sleep not found" lab="$TMP_ROOT/stale-reg-descendant-bin"; mkdir -p "$lab" - # A symlink to a real long-running binary so the kernel records `pi` as the - # executable identity (a copied platform binary fails code signing on macOS). - ln -sf "$sleep_bin" "$lab/pi" + # A symlink to a real long-running stand-in so the kernel records `pi` as the + # executable identity (tests/lib.sh fm_agent_standin owns why not host sleep). + standin=$(fm_agent_standin "$TMP_ROOT/standin") || { + echo "skip: no long-running stand-in survives a rename, so the agent-named descendant case cannot run" + return 0 + } + ln -sf "$standin" "$lab/pi" # A real shell whose child is that agent-named process, while the canned # foreground view shows only the shell (a suspended or backgrounded agent). sh -c "'$lab/pi' 300; :" & @@ -715,13 +719,16 @@ test_registered_agent_with_an_agent_descendant_outside_the_foreground_stays_aliv } test_agent_descendant_under_a_spaced_install_path_stays_alive() { - local lab sleep_bin shell_pid out - sleep_bin=$(command -v sleep) || fail "sleep not found" + local lab standin shell_pid out + standin=$(fm_agent_standin "$TMP_ROOT/standin") || { + echo "skip: no long-running stand-in survives a rename, so the spaced-path descendant case cannot run" + return 0 + } # The executable path the process table reports contains a space (the macOS # `/Library/Application Support/...` shape), so a field-split read of the # process table sees only a fragment of the name. lab="$TMP_ROOT/stale-reg-spaced-bin/Application Support/Some Dir"; mkdir -p "$lab" - ln -sf "$sleep_bin" "$lab/pi" + ln -sf "$standin" "$lab/pi" sh -c "'$lab/pi' 300; :" & shell_pid=$! sleep 0.3 @@ -3390,8 +3397,8 @@ test_presentation_session_lock_path_is_shared_across_homes() { || fail "session lock path resolution failed for home B" [ "$path_a" = "$path_b" ] || fail "same session/socket must resolve one shared lock path" case "$path_a" in - /tmp/firstmate-herdr-presentation/order-*.lock) ;; - *) fail "session lock path must use the shared machine namespace: $path_a" ;; + "/tmp/firstmate-herdr-presentation-$(id -u)"/order-*.lock) ;; + *) fail "session lock path must use this account's namespace: $path_a" ;; esac case "$path_a" in */state/*) fail "session lock path must not live under a home state directory: $path_a" ;; diff --git a/tests/fm-contributions.test.sh b/tests/fm-contributions.test.sh index 585ea888cac..64ae6e794ba 100755 --- a/tests/fm-contributions.test.sh +++ b/tests/fm-contributions.test.sh @@ -1024,6 +1024,9 @@ test_arm_plumbs_a_configured_budget_into_the_check_shim() { # before the first forge call, so nothing is ever observed. /bin/date +%s > "$home/forge/clock" printf 'hang\n' > "$home/forge/fault" + # Freeze the clock. An unfrozen one-second budget can tick past before the + # first forge call, so the generated check never writes forge/calls. + /bin/date +%s > "$home/forge/clock" if [ "$mode" = configured ]; then with_home "$home" env FM_CONTRIBUTIONS_BUDGET=1 "$ROOT/bin/fm-contributions.sh" arm >/dev/null \ || fail 'arm with a configured budget failed' diff --git a/tests/fm-control-herdr-smoke.test.sh b/tests/fm-control-herdr-smoke.test.sh index 10fbeb1f54b..fdd8704a703 100755 --- a/tests/fm-control-herdr-smoke.test.sh +++ b/tests/fm-control-herdr-smoke.test.sh @@ -21,9 +21,10 @@ # when herdr or jq is missing. set -u -ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" -fail() { printf 'not ok - %s\n' "$1" >&2; cleanup_all; exit 1; } +fail() { printf 'not ok - %s\n' "$1" >&2; exit 1; } pass() { printf 'ok - %s\n' "$1"; } command -v herdr >/dev/null 2>&1 || { echo "skip: herdr not found"; exit 0; } @@ -33,15 +34,49 @@ command -v jq >/dev/null 2>&1 || { echo "skip: jq not found (required by the her . "$ROOT/tests/herdr-test-safety.sh" herdr_forget_inherited_pane +# The agent-named process below is a symlink to fm_agent_standin's stand-in; +# decide it before any lab session exists so an impossible case skips cleanly. +STANDIN_DIR=$(fm_test_tmproot fm-control-herdr-standin) || { + printf 'not ok - %s\n' "could not create the stand-in directory" >&2 + exit 1 +} +STANDIN_BIN=$(fm_agent_standin "$STANDIN_DIR") || { + echo "skip: no long-running stand-in binary survives a rename (multicall coreutils, no C compiler)" + exit 0 +} + SESSION="fm-lab-control-smoke-$$" export HERDR_SESSION="$SESSION" SCRATCH= +LAB_PREPARED=0 cleanup_all() { - [ -n "$SCRATCH" ] && rm -rf "$SCRATCH" - herdr_safe_stop_and_delete "$SESSION" + local status=$? cleanup_status=0 + trap - EXIT + if [ -n "$SCRATCH" ]; then + if [ -d "$SCRATCH/home/state/hsmoke.git-hooks" ]; then + chmod u+w "$SCRATCH/home/state/hsmoke.git-hooks" || { + echo "not ok - could not restore write permission on test git hooks" >&2 + cleanup_status=1 + } + fi + rm -rf "$SCRATCH" || { + echo "not ok - could not remove Herdr smoke scratch tree: $SCRATCH" >&2 + cleanup_status=1 + } + fi + if [ "$LAB_PREPARED" = 1 ]; then + herdr_safe_stop_and_delete "$SESSION" || { + echo "not ok - could not tear down Herdr smoke lab session: $SESSION" >&2 + cleanup_status=1 + } + fi + fm_test_cleanup + [ "$cleanup_status" = 0 ] || status=1 + exit "$status" } trap cleanup_all EXIT fm_herdr_lab_prepare "$SESSION" || fail "could not prepare isolated Herdr lab session" +LAB_PREPARED=1 SCRATCH=$(mktemp -d "${TMPDIR:-/tmp}/fm-control-herdr.XXXXXX") SCRATCH=$(cd "$SCRATCH" && pwd) @@ -207,13 +242,12 @@ pass "real herdr: interrupt refuses when herdr's own agent registry reports no a # A registration alone no longer proves an agent (issue #4115): the adapter # verifies the pane's processes through the real `pane process-info` view. So # the registered agent is backed by a real agent-named foreground process - a -# symlink to a long-running system binary named `claude`, the same construction -# tests/fm-tmux-agent-liveness.test.sh uses (a copied platform binary fails code -# signing on macOS arm64; the symlink name is what the kernel records as argv[0]). +# symlink named `claude` to the fm_agent_standin stand-in decided above, the same +# construction tests/fm-tmux-agent-liveness.test.sh uses (the symlink name is +# what the kernel records as argv[0]; tests/lib.sh owns why it is never a copy). AGENT_BIN="$SCRATCH/agentbin" mkdir -p "$AGENT_BIN" -SLEEP_BIN=$(command -v sleep) || fail "sleep not found" -ln -s "$SLEEP_BIN" "$AGENT_BIN/claude" +ln -s "$STANDIN_BIN" "$AGENT_BIN/claude" printf -v AGENT_Q '%q' "$AGENT_BIN/claude" wait_process_state() { # @@ -309,7 +343,7 @@ awk -F= '$1 == "harness" {$0="harness=claude"} {print}' "$HOME_DIR/state/hsmoke. mv "$HOME_DIR/state/hsmoke.meta.tmp" "$HOME_DIR/state/hsmoke.meta" pass "real herdr: a stale registration no longer blocks relaunch, and the endpoint and local copy survive" -# Last: the foreground process is a plain `sleep`, so the pane never draws any +# Last: the foreground process is the sleeping stand-in, so the pane never draws any # recognized composer chrome. exit's composer-empty guard (bin/fm-control.sh) # therefore refuses before ever typing the exit command, rather than typing it # into a live agent that ignores it and reporting a stop that did not happen. @@ -321,9 +355,16 @@ if OUT=$(run_control hsmoke exit 2>&1); then fail "exit should fail closed when the agent's composer is not proven empty: $OUT" fi case "$OUT" in - *"not proven empty"*) : ;; - *) fail "the exit failure should say the composer is not proven empty, got: $OUT" ;; + *"composer visibly holds pending text; refusing to type the /exit exit command"*|*"not proven empty; refusing to type the /exit exit command"*) : ;; + *) fail "the exit failure should refuse to type the /exit command for an unproven or pending composer, got: $OUT" ;; +esac +[ "$(fm_backend_agent_state herdr "$SESSION:$PANE_ID")" = alive ] \ + || fail "the exit refusal did not preserve the fake foreground agent" +SCREEN_AFTER=$(fm_backend_herdr_visible_capture "$SESSION:$PANE_ID") \ + || fail "could not read the pane after the exit refusal" +case "$SCREEN_AFTER" in + *'/exit'*) fail "the refused /exit command appeared on the agent pane" ;; esac -pass "real herdr: an agent behind an unproven composer fails closed instead of typing an exit command into it" +pass "real herdr: exit refuses an unproven or pending composer without typing an exit command" fm_backend_herdr_kill "$SESSION:$PANE_ID" 2>/dev/null || true diff --git a/tests/fm-host-mirror-live-e2e.test.sh b/tests/fm-host-mirror-live-e2e.test.sh index 9b040adef76..62602b23259 100755 --- a/tests/fm-host-mirror-live-e2e.test.sh +++ b/tests/fm-host-mirror-live-e2e.test.sh @@ -20,6 +20,8 @@ set -u # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=bin/fm-tmux-lib.sh +. "$ROOT/bin/fm-tmux-lib.sh" fm_live_gate opt-in FM_HOST_MIRROR_LIVE_E2E jq tmux @@ -85,6 +87,17 @@ check() { # # harness, so the lock holder is the harness that fires the hooks. LOCKED_EXEC='printf "%s\n" "$$" > state/.lock; exec "$@"' +# These bare tmux launches have no native agent-idle event. Require tmux's +# Claude-scoped busy read and the shared composer classifier to agree on idle. +claude_ready_state() { # -> diagnostic state; success when ready + local socket=$1 target=$2 busy composer + tmux() { command tmux -L "$socket" "$@"; } + busy=$(fm_pane_busy_state "$target" claude) + composer=$(fm_tmux_composer_state "$target") + printf 'busy=%s composer=%s' "$busy" "$composer" + [ "$busy" = idle ] && [ "$composer" = empty ] +} + # Claude runs interactively with one extra Stop hook that rewakes the session # once, as the supervision host's own handback does, so the guard also proves # that a harness-started turn is never mirrored as the captain's words. @@ -104,13 +117,16 @@ SH chmod +x "$root/rewake-once.sh" jq '.hooks.Stop += [{hooks: [{type: "command", command: "\"$CLAUDE_PROJECT_DIR\"/rewake-once.sh", asyncRewake: true, timeout: 60}]}]' \ "$root/.claude/settings.json" > "$root/.claude/settings.json.tmp" && mv "$root/.claude/settings.json.tmp" "$root/.claude/settings.json" - REWAKE_WANTED=mirror-rewake-ok run_interactive claude claude --model haiku --dangerously-skip-permissions + # Keep the idle composer empty for the shared classifier. Claude's rotating + # prompt suggestion is ordinary visible composer text on this tmux surface. + CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false REWAKE_WANTED=mirror-rewake-ok \ + run_interactive claude claude --model haiku --dangerously-skip-permissions } # An interactive session in a private tmux server: answer a trust prompt when # one appears, type the prompt, and wait for the mirror. run_interactive() { # [arguments...] - local harness=$1 command=$2 root version i screen + local harness=$1 command=$2 root version i screen readiness=unknown last_seen=none shift 2 version=$("$command" --version 2>/dev/null | head -n 1) root="$LAB/$harness" @@ -120,6 +136,8 @@ run_interactive() { # [arguments...] i=0 while [ "$i" -lt 60 ]; do screen=$(tmux -L "$SOCKET-$harness" capture-pane -p -t "$harness" 2>/dev/null) + last_seen=$(printf '%s\n' "$screen" | grep -v '^[[:space:]]*$' | tail -n 6) + [ -n "$last_seen" ] || last_seen='empty screen' # A key sent to a dialog is followed by a pause long enough for the # harness to redraw, so the same dialog is never answered twice. case "$screen" in @@ -127,13 +145,17 @@ run_interactive() { # [arguments...] *'Yes, I trust this folder'*|*'Trust all and continue'*) tmux -L "$SOCKET-$harness" send-keys -t "$harness" Down; sleep 0.5; tmux -L "$SOCKET-$harness" send-keys -t "$harness" Enter; sleep 3 ;; *'1. Yes, continue'*) tmux -L "$SOCKET-$harness" send-keys -t "$harness" Enter; sleep 3 ;; - *'bypass permissions on'*) break ;; *'Do you trust the contents of this directory'*) tmux -L "$SOCKET-$harness" send-keys -t "$harness" y; sleep 3 ;; *'Plan, search, build'*) break ;; + *) + if [ "$harness" = claude ]; then + readiness=$(claude_ready_state "$SOCKET-$harness" "$harness") && break + fi ;; esac sleep 1 i=$((i + 1)) done + [ "$i" -lt 60 ] || fail "$harness $version: never reached an idle empty composer (last readiness: $readiness; last screen: $last_seen)" sleep 3 tmux -L "$SOCKET-$harness" send-keys -t "$harness" -l "$PROMPT" sleep 1 @@ -160,7 +182,7 @@ run_interactive() { # [arguments...] for harness in $HARNESSES; do case "$harness" in claude) bin=$harness ;; - cursor) bin=cursor-agent ;; + cursor) bin='cursor-agent' ;; *) fail "unknown harness in FM_HOST_MIRROR_LIVE_HARNESSES: $harness" ;; esac if ! command -v "$bin" >/dev/null 2>&1; then diff --git a/tests/fm-pending-reply.test.sh b/tests/fm-pending-reply.test.sh index 697ec00796f..7410b2d8432 100755 --- a/tests/fm-pending-reply.test.sh +++ b/tests/fm-pending-reply.test.sh @@ -13,6 +13,8 @@ # 4. Second missed turn escalates once and remains durable # 5. Transport success cannot masquerade as reply success # 6. Unrelated events and stale correlation ids cannot resolve a request +# - including another mate's line that echoes the request's token, or a +# token embedded in a longer word # 7. Restart/compaction preserves the expectation and exact parent destination # 8. Wrong-home reports are detected but do not silently acknowledge # 9. Direct unmarked captain input creates no expectation @@ -1067,6 +1069,37 @@ test_unrelated_and_stale_corr_cannot_resolve() { pass "unrelated events and stale correlation ids cannot resolve" } +test_another_mates_echo_cannot_resolve() { + local home state corr_a corr_b + home=$(setup_parent two-mates) + state="$home/state" + # shellcheck disable=SC2031 + export FM_PENDING_REPLY_NOW=6100 + corr_a=$(fm_pending_reply_create "$home" "$state" "alpha" "need alpha's answer") + corr_b=$(fm_pending_reply_create "$home" "$state" "beta" "need beta's answer") + fm_pending_reply_mark_delivered "$state" "$corr_a" + fm_pending_reply_mark_delivered "$state" "$corr_b" + # Beta's reply echoes alpha's token; remote reply ingest hands every corr= in + # beta's payload over together with beta's own status log. + printf 'done [corr=%s]: answered, and alpha still owes corr=%s\n' "$corr_b" "$corr_a" \ + > "$state/beta.status" + if fm_pending_reply_try_resolve "$state" "$corr_a" "$state/beta.status"; then + fail "another mate's line echoing the token must not resolve the request" + fi + [ "$(phase_of "$state" "$corr_a")" = awaiting_report ] || fail "alpha's request must stay open" + fm_pending_reply_try_resolve "$state" "$corr_b" "$state/beta.status" \ + || fail "beta's own correlated line should resolve beta's request" + printf 'done corr=%sff: a longer token is a different token\n' "$corr_a" > "$state/alpha.status" + printf 'done xcorr=%s: so is a prefixed one\n' "$corr_a" >> "$state/alpha.status" + if fm_pending_reply_try_resolve "$state" "$corr_a"; then + fail "a token embedded in a longer word must not resolve" + fi + printf 'done [corr=%s]: alpha answered\n' "$corr_a" >> "$state/alpha.status" + fm_pending_reply_try_resolve "$state" "$corr_a" "$state/alpha.status" \ + || fail "alpha's own correlated line should resolve alpha's request" + pass "another mate's echoed token cannot resolve a request" +} + test_restart_preserves_expectation_and_parent_destination() { local home state corr rec parent_status parent_home home=$(setup_parent restart) @@ -2093,6 +2126,7 @@ test_undelivered_records_are_scan_immutable test_delivery_confirmation_fallback_reconciles test_delivery_confirmation_serializes_with_reconciliation test_unrelated_and_stale_corr_cannot_resolve +test_another_mates_echo_cannot_resolve test_restart_preserves_expectation_and_parent_destination test_wrong_home_detected_not_acknowledged test_unmarked_captain_input_creates_no_expectation diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index 1d810c59926..29bd8365d6d 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -4145,6 +4145,7 @@ test_opencode_primary_watch_plugin_uses_effective_state_home() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4244,6 +4245,7 @@ test_opencode_primary_watch_plugin_requires_session_lock() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4353,6 +4355,7 @@ test_opencode_primary_watch_plugin_rearms_after_wake() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4449,6 +4452,7 @@ test_opencode_primary_watch_plugin_runs_the_supervision_host() { # [away|quiet] mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" if [ "$kind" = quiet ]; then # Quiet mode's record is a present captain (bin/fm-afk-contract.sh AWAY OR @@ -4540,6 +4544,7 @@ test_opencode_pre_ready_actionable_close_preserves_its_successor() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4620,6 +4625,7 @@ test_opencode_hung_successor_falls_back_to_typed_wake() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4691,6 +4697,7 @@ test_opencode_unretired_successor_falls_back_without_retry() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4768,6 +4775,7 @@ test_opencode_late_unretired_close_resumes_supervision() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4863,6 +4871,7 @@ test_opencode_empty_close_retries_instead_of_disappearing() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4922,6 +4931,7 @@ test_opencode_established_empty_close_honors_retry_limit() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -4976,6 +4986,7 @@ test_opencode_actionable_close_rechecks_session_lock() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -5042,6 +5053,7 @@ test_opencode_watch_arm_coordinates_with_turnend_guard() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash @@ -5115,6 +5127,7 @@ test_opencode_healthy_arm_output_does_not_suppress_guard() { mkdir -p "$repo/bin" "$home/state" "$home/config" git init -q "$repo" : > "$repo/AGENTS.md" + cp "$ROOT/bin/fm-supervision-lib.sh" "$repo/bin/fm-supervision-lib.sh" : > "$home/state/task.meta" cat > "$repo/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash diff --git a/tests/fm-procevent.test.sh b/tests/fm-procevent.test.sh index 5ecd4db4512..46fb4fa2159 100755 --- a/tests/fm-procevent.test.sh +++ b/tests/fm-procevent.test.sh @@ -3144,6 +3144,34 @@ assert_contains "$out" "CAPTAIN MESSAGE" "an open-session message was mislabeled assert_not_contains "$out" "SESSION-ENDING MESSAGE" "an open-session message was labeled as session-ending" assert_contains "$out" "| captain is still reviewing" "an open-session message was dropped" pass "read distinguishes a live captain message from a session-ending message" +# Composer sends arrive as several tag=message rows beside real annotations. +# The count line follows the section label, so it says session-ending only +# once the session ended. +for ended in no yes; do + { + printf 'session:\n file: /review.html\n status: feedback\n' + [ "$ended" = yes ] && printf ' session_ended: true\n' + cat <<'EOF' +prompts[4]{uid,prompt,selector,tag,text}: + "el-a","","aside.sidebar",note,"Sidebar note" + "","first comment","",message,"Freeform message" + "","second comment","",message,"Freeform message" + "","third comment","",message,"Freeform message" +EOF + } > "$READ" + out=$(read_out) || fail "read failed on several messages with an annotation (ended=$ended)" + if [ "$ended" = yes ]; then + label="SESSION-ENDING MESSAGE" count=session_ending_message_count other=captain_message_count + else + label="CAPTAIN MESSAGE" count=captain_message_count other=session_ending_message_count + fi + assert_contains "$out" "$label PART 3 of 3" "a message part was dropped or mislabeled (ended=$ended)" + assert_contains "$out" "| third comment" "a message body was dropped (ended=$ended)" + assert_contains "$out" "$count: 3" "the message count did not follow the section label (ended=$ended)" + assert_not_contains "$out" "$other" "the message count used the other label (ended=$ended)" + assert_contains "$out" "annotation_count: 1" "a real annotation was miscounted beside messages (ended=$ended)" +done +pass "read names the message count with the same label as the message section" out=$ending_out assert_contains "$out" '| "question": "sample-forged-call",' \ "commas in an unquoted freeform message shifted its fields" diff --git a/tests/fm-project-capacity.test.sh b/tests/fm-project-capacity.test.sh new file mode 100755 index 00000000000..bc64b8283b1 --- /dev/null +++ b/tests/fm-project-capacity.test.sh @@ -0,0 +1,642 @@ +#!/usr/bin/env bash +# Behavior tests for project capacity admission: a project that declares how +# many workers it admits at once on this machine never gets a fresh worker +# launched beyond that number (bin/fm-project-capacity-lib.sh owns the +# contract; bin/fm-spawn.sh runs the check). +# +# Every case drives the real bin/fm-spawn.sh against a real project clone with +# an origin, fake tmux and treehouse binaries that log every call, and a real +# markdown backlog when tasks-axi is installed. A deferred spawn is judged by +# what it left behind - no task record, no rendered launch brief, no endpoint, +# no worktree allocation, and a backlog item still queued - never by wording +# alone. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +unset TASKS_AXI_BACKEND || : + +SPAWN="$ROOT/bin/fm-spawn.sh" +TEARDOWN="$ROOT/bin/fm-teardown.sh" +TMP_ROOT=$(fm_test_tmproot fm-project-capacity) +DEFER_EXIT=75 +HAVE_TASKS_AXI=0 +command -v tasks-axi >/dev/null 2>&1 && HAVE_TASKS_AXI=1 + +# --- fixture ---------------------------------------------------------------- + +write_brief() { # + mkdir -p "$1/data/$2" + cat > "$1/data/$2/brief.md" < [task-id...] + local home=$1 id + shift + mkdir -p "$home/state" "$home/config" "$home/data" "$home/projects" + touch "$home/state/.last-watcher-beat" + printf '%s\n' codex > "$home/config/crew-harness" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + printf '%s\n' '# Backlog' '' '## In flight' '' '## Queued' '' '## Done' \ + > "$home/data/backlog.md" + cat > "$home/.tasks.toml" <<'EOF' +backend = "markdown" + +[markdown] +path = "data/backlog.md" +EOF + fi + for id in "$@"; do + write_brief "$home" "$id" + add_item "$home" "$id" + done +} + +# A case: one home, a project clone with an origin, and fake tmux/treehouse +# that record every call so a deferral can be proved to have created nothing. +make_case() { # [task-id...] + local name=$1 case_dir fakebin + shift + case_dir="$TMP_ROOT/$name" + mkdir -p "$case_dir" + fakebin=$(fm_fakebin "$case_dir") + : > "$case_dir/calls.log" + cat > "$fakebin/tmux" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$FM_FAKE_CALL_LOG" +case "$*" in + *"#{pane_current_path}"*) + # A spawn that is told to hold waits here, after its capacity check and + # while it still holds the project lock, until the test releases it. + if [ -n "${FM_FAKE_HOLD:-}" ]; then + : > "$FM_FAKE_HOLD.reached" + i=0 + while [ ! -f "$FM_FAKE_HOLD.release" ]; do + i=$((i + 1)) + [ "$i" -lt 400 ] || exit 1 + sleep 0.05 + done + fi + printf '%s\n' "${FM_FAKE_PANE_PATH:-}" + exit 0 + ;; +esac +case "${1:-}" in display-message) printf 'firstmate\n' ;; esac +exit 0 +SH + cat > "$fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +printf 'treehouse %s\n' "$*" >> "$FM_FAKE_CALL_LOG" +exit 0 +SH + chmod +x "$fakebin/tmux" "$fakebin/treehouse" + fm_fake_exit0 "$fakebin" gh gh-axi no-mistakes + fm_git_init_commit "$case_dir/project" + fm_git_add_origin "$case_dir/project" "$case_dir/project.origin.git" + fm_git_init_commit "$case_dir/other-project" + fm_git_add_origin "$case_dir/other-project" "$case_dir/other-project.origin.git" + make_home "$case_dir/home" "$@" + printf '%s\n' "$case_dir" +} + +add_item() { # + [ "$HAVE_TASKS_AXI" = 1 ] || return 0 + tasks-axi add "$2" "item for $2" --kind ship --file "$1/data/backlog.md" >/dev/null +} + +row_state() { # + tasks-axi show "$2" --file "$1/data/backlog.md" 2>/dev/null | + sed -n 's/^ state: *//p' | head -1 +} + +declare_capacity() { # ... + local home=$1 + shift + printf '%s\n' "$@" > "$home/config/project-capacity" +} + +# A live worker already on a project: the record shape bin/fm-spawn.sh +# publishes, with its backlog item In flight so cleanup can close it. +write_live() { # [extra-line...] + local home=$1 id=$2 project=$3 + shift 3 + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" \ + "endpoint_task_id=$id" \ + "worktree=$home/absent-worktree-$id" \ + "project=$project" \ + "harness=codex" \ + "kind=ship" \ + "mode=no-mistakes" \ + "yolo=off" \ + "spawn_gen=s-$id" \ + "$@" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + add_item "$home" "$id" + tasks-axi start "$id" --file "$home/data/backlog.md" >/dev/null + fi +} + +# One isolated worktree per spawn, so every admitted launch has its own copy. +new_worktree() { # + git -C "$1/project" worktree add --quiet -b "wt-$2" "$1/wt-$2" + printf '%s\n' "$1/wt-$2" +} + +run_spawn() { # + local case_dir=$1 home=$2 pane=$3 + shift 3 + FM_ROOT_OVERRIDE='' FM_HOME="$home" \ + FM_STATE_OVERRIDE='' FM_DATA_OVERRIDE='' FM_PROJECTS_OVERRIDE='' FM_CONFIG_OVERRIDE='' \ + FM_SPAWN_NO_GUARD=1 TMUX="fake,1,0" FM_BACKEND=tmux \ + FM_FAKE_PANE_PATH="$pane" FM_FAKE_CALL_LOG="$case_dir/calls.log" \ + PATH="$case_dir/fakebin:$PATH" \ + "$SPAWN" "$@" 2>&1 +} + +spawn_ship() { # [pane-path] + local case_dir=$1 id=$2 pane=${3:-} + [ -n "$pane" ] || pane=$(new_worktree "$case_dir" "$id") + run_spawn "$case_dir" "$case_dir/home" "$pane" "$id" "$case_dir/project" --mode no-mistakes --yolo off +} + +# Everything a deferred spawn must not have created for ; +# is worktree_list taken before the spawn. +assert_nothing_created() { # + local case_dir=$1 home=$2 id=$3 before=$4 worktrees=$5 after + assert_absent "$home/state/$id.meta" "a deferred spawn published a task record for $id" + assert_absent "$home/data/$id/launch-brief.md" "a deferred spawn rendered a launch brief for $id" + after=$(call_count "$case_dir") + [ "$after" -eq "$before" ] || + fail "a deferred spawn touched the terminal or worktree pool for $id: $(tail -n +"$((before + 1))" "$case_dir/calls.log")" + assert_equals "$worktrees" "$(worktree_list "$case_dir")" "a deferred spawn left a git worktree for $id" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + [ "$(row_state "$home" "$id")" = queued ] || + fail "a deferred spawn moved $id's backlog item: $(row_state "$home" "$id")" + fi +} + +call_count() { wc -l < "$1/calls.log" | tr -d ' '; } + +worktree_list() { git -C "$1/project" worktree list --porcelain; } + +# --- cases ------------------------------------------------------------------ + +# The reported incident's shape before any declaration: the project is already +# busy, and nothing caps a further launch. Absent a declaration that stays true. +test_undeclared_capacity_keeps_dispatch_uncapped() { + local case_dir home out rc=0 + case_dir=$(make_case undeclared task-c) + home="$case_dir/home" + write_live "$home" live-a "$case_dir/project" + write_live "$home" live-b "$case_dir/project" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "an undeclared project refused a spawn: $out" + assert_contains "$out" "spawned task-c" "an undeclared project did not launch the worker" + assert_present "$home/state/task-c.meta" "an undeclared project's spawn published no record" + pass "a project with no declared capacity keeps today's uncapped dispatch" +} + +test_available_capacity_admits_the_worker() { + local case_dir home out rc=0 + case_dir=$(make_case available task-c) + home="$case_dir/home" + declare_capacity "$home" "# heavy suite serves two workers" "project 2" "other-project 1" + write_live "$home" live-a "$case_dir/project" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "a spawn with a free place was refused: $out" + assert_contains "$out" "spawned task-c" "a spawn with a free place did not launch" + assert_not_contains "$out" "deferred:" "a spawn with a free place reported a deferral" + if [ "$HAVE_TASKS_AXI" = 1 ]; then + [ "$(row_state "$home" task-c)" = in_flight ] || fail "an admitted spawn did not move its item In flight" + fi + pass "a spawn is admitted while its project still has a free place" +} + +test_exhausted_capacity_defers_without_leaving_anything_behind() { + local case_dir home out rc=0 before worktrees + case_dir=$(make_case exhausted task-c) + home="$case_dir/home" + declare_capacity "$home" "project 2" + write_live "$home" live-a "$case_dir/project" + write_live "$home" live-b "$case_dir/project" + before=$(call_count "$case_dir") + worktrees=$(worktree_list "$case_dir") + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a spawn beyond capacity was not deferred: $out" + assert_contains "$out" "deferred: project project admits 2 worker(s) at once on this machine ($home/config/project-capacity) and 2 already hold a place (live-a, live-b)" \ + "the deferral did not name the capacity and its holders" + assert_contains "$out" "task task-c was not launched and its backlog item stays queued" \ + "the deferral did not say the task stays queued" + assert_nothing_created "$case_dir" "$home" task-c "$before" "$worktrees" + pass "a spawn beyond capacity is deferred before any record, brief, endpoint, worktree, or backlog move exists" +} + +# The capacity is the last field, so a project whose clone directory name holds +# spaces can be declared. An indented '#' line is still a comment. +test_spaced_project_name_is_declared() { + local case_dir home spaced out rc=0 + case_dir=$(make_case spaced task-c) + home="$case_dir/home" + spaced="$case_dir/my heavy project" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$spaced" + declare_capacity "$home" " # my heavy project 9" "my heavy project 1" "project 5" + write_live "$home" live-a "$spaced" + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-c "$spaced" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a project whose name holds spaces was not capped by its declaration: $out" + assert_contains "$out" "deferred: project my heavy project admits 1 worker(s) at once" \ + "the deferral did not use the spaced project's declared capacity" + assert_absent "$home/state/task-c.meta" "the deferred spaced-name spawn published a record" + pass "a project name with spaces is declared by taking the capacity from the last field" +} + +# A clone directory may be named with a leading '#'. That name is declared when +# the '#' is written against the rest of the name and the line ends with the +# capacity. A '#' followed by whitespace stays a comment even when the line +# ends with a number, and a '#' note that is not a capacity stays a comment. +test_hash_prefixed_project_name_is_declared() { + local case_dir home hashed spaced out rc=0 wt + case_dir=$(make_case hash-name task-c task-d) + home="$case_dir/home" + hashed="$case_dir/#hash-project" + spaced="$case_dir/# serves" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$hashed" + git clone -q "$(git -C "$case_dir/other-project" remote get-url origin)" "$spaced" + declare_capacity "$home" "# serves 2" "#not-a-capacity" "#hash-project 1" + write_live "$home" live-a "$hashed" + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-c "$hashed" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a project whose name begins with # was not capped by its declaration: $out" + assert_contains "$out" "deferred: project #hash-project admits 1 worker(s) at once" \ + "the deferral did not use the hash-prefixed project's declared capacity" + assert_absent "$home/state/task-c.meta" "the deferred hash-prefixed spawn published a record" + + write_live "$home" live-b "$spaced" + write_live "$home" live-c "$spaced" + git -C "$spaced" worktree add --quiet -b wt-d "$case_dir/wt-d" + wt="$case_dir/wt-d" + rc=0 + out=$(run_spawn "$case_dir" "$home" "$wt" task-d "$spaced" --mode no-mistakes --yolo off) || rc=$? + expect_code 0 "$rc" "a '#' comment that ends with a number was read as a capacity: $out" + assert_contains "$out" "spawned task-d" "a project whose declaration line is a comment did not stay uncapped" + pass "a project name beginning with # is declared, and a # comment stays a comment" +} + +# A home reached through a symlink is still one home: its workers hold one +# place each, not one per spelling of its state directory. +test_symlinked_home_counts_each_worker_once() { + local case_dir home out rc=0 + case_dir=$(make_case symlinked-home task-c) + home="$case_dir/home" + ln -s "$home" "$case_dir/home-link" + declare_capacity "$home" "project 2" + write_live "$home" live-a "$case_dir/project" + out=$(run_spawn "$case_dir" "$case_dir/home-link" "$(new_worktree "$case_dir" task-c)" \ + task-c "$case_dir/project" --mode no-mistakes --yolo off) || rc=$? + expect_code 0 "$rc" "a spawn through a symlinked home counted its own worker twice: $out" + assert_contains "$out" "spawned task-c" "a spawn through a symlinked home did not launch" + pass "a home reached through a symlink counts each of its workers once" +} + +# A fresh spawn that restarts an existing task id replaces that task's own +# record, so the record does not hold a place against it. Any other task on the +# project still sees that record as a holder. +test_restart_does_not_count_its_own_record() { + local case_dir home out rc=0 + case_dir=$(make_case restart task-c task-d) + home="$case_dir/home" + declare_capacity "$home" "project 1" + fm_write_meta "$home/state/task-c.meta" \ + "window=firstmate:fm-task-c" \ + "project=$case_dir/project" \ + "kind=ship" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + assert_not_contains "$out" "deferred:" "a restart was deferred by its own record" + [ "$rc" -ne "$DEFER_EXIT" ] || fail "a restart exited with the deferral code: $out" + rc=0 + out=$(spawn_ship "$case_dir" task-d "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "another task ignored the restarted task's place: $out" + assert_contains "$out" "1 already hold a place (task-c)" "another task did not count the restarted task" + pass "a restart of a task id does not count that task's own record" +} + +# A place frees when a worker records its ready PR and when a task is cleaned +# up; each release admits exactly one more worker. +test_release_frees_a_place() { + local case_dir home out rc=0 + case_dir=$(make_case release task-c task-d) + home="$case_dir/home" + declare_capacity "$home" "project 2" + write_live "$home" live-a "$case_dir/project" + write_live "$home" live-b "$case_dir/project" + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "the full project admitted a worker: $out" + + # PR handoff: the line bin/fm-pr-check.sh records for a ready PR. + printf 'pr=%s\n' "https://github.com/o/r/pull/7" >> "$home/state/live-a.meta" + rc=0 + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "a recorded PR handoff did not free a place: $out" + rc=0 + out=$(spawn_ship "$case_dir" task-d "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "one freed place admitted two workers: $out" + assert_contains "$out" "2 already hold a place (live-b, task-c)" "the new worker did not take the freed place" + + # Cleanup: the real teardown removes the record, which frees its place. + rc=0 + out=$(FM_ROOT_OVERRIDE='' FM_HOME="$home" FM_STATE_OVERRIDE='' FM_DATA_OVERRIDE='' FM_CONFIG_OVERRIDE='' \ + FM_FAKE_CALL_LOG="$case_dir/calls.log" PATH="$case_dir/fakebin:$PATH" \ + "$TEARDOWN" live-b 2>&1) || rc=$? + expect_code 0 "$rc" "cleanup of a live worker failed: $out" + assert_absent "$home/state/live-b.meta" "cleanup left the worker's record" + rc=0 + out=$(spawn_ship "$case_dir" task-d) || rc=$? + expect_code 0 "$rc" "cleanup did not free a place: $out" + pass "a recorded PR handoff or a cleanup each frees exactly one place" +} + +# Only workers on the same project identity hold places: a secondmate record, a +# record for another project, and a scout on another project are ignored, while +# a scout and a legacy record without kind= on this project count. +test_occupancy_counts_only_this_projects_workers() { + local case_dir home out rc=0 + case_dir=$(make_case occupancy task-c) + home="$case_dir/home" + declare_capacity "$home" "project 3" + write_live "$home" scout-a "$case_dir/project" + sed -i.bak 's/^kind=ship$/kind=scout/' "$home/state/scout-a.meta" && rm -f "$home/state/scout-a.meta.bak" + fm_write_meta "$home/state/legacy-b.meta" "window=firstmate:fm-legacy-b" "project=$case_dir/project" + write_live "$home" other-c "$case_dir/other-project" + fm_write_meta "$home/state/mate-d.meta" "window=remote:mate-d" "project=$case_dir/project" "kind=secondmate" + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "records outside this project's workers took a place: $out" + rc=0 + write_brief "$home" task-e + add_item "$home" task-e + out=$(spawn_ship "$case_dir" task-e "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "the third worker on the project was not counted: $out" + assert_contains "$out" "3 already hold a place (legacy-b, scout-a, task-c)" \ + "occupancy counted the wrong records" + pass "only this project's ship and scout records hold places" +} + +# Capacity belongs to the machine: a local secondmate home's workers on a +# separate clone of the same origin count, the declaration comes from the root +# home even when the secondmate spawns, and a remote home's workers never count. +test_capacity_is_shared_by_every_local_home() { + local case_dir root mate out rc=0 mate_project + case_dir=$(make_case machine) + root="$case_dir/home" + mate="$case_dir/mate" + make_home "$mate" task-m + mate_project="$mate/projects/project" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$mate_project" + printf '%s\n' schema=fm-secondmate-parent.v1 route=local "parent_home=$root" > "$mate/.fm-secondmate-parent" + printf -- '- mate - a local mate (home: %s; scope: project work; projects: project; added 2026-09-01)\n' "$mate" \ + > "$root/data/secondmates.md" + printf -- '- far - a remote mate (host: far.example; root: /srv/fm; home: %s/remote-home; scope: other; projects: project; added 2026-09-01)\n' "$case_dir" \ + >> "$root/data/secondmates.md" + mkdir -p "$case_dir/remote-home/state" + fm_write_meta "$case_dir/remote-home/state/far-x.meta" "window=w" "project=$case_dir/project" "kind=ship" + declare_capacity "$root" "project 1" + write_live "$root" live-a "$case_dir/project" + git -C "$mate_project" worktree add --quiet -b wt-m "$case_dir/wt-m" + out=$(run_spawn "$case_dir" "$mate" "$case_dir/wt-m" task-m "$mate_project" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a local secondmate launched past the machine's capacity: $out" + assert_contains "$out" "admits 1 worker(s) at once on this machine ($root/config/project-capacity) and 1 already hold a place (live-a in $root)" \ + "the secondmate did not count the root home's worker against the root's declaration" + assert_absent "$mate/state/task-m.meta" "the deferred secondmate spawn published a record" + printf 'pr=%s\n' "https://github.com/o/r/pull/8" >> "$root/state/live-a.meta" + rc=0 + out=$(run_spawn "$case_dir" "$mate" "$case_dir/wt-m" task-m "$mate_project" --mode no-mistakes --yolo off) || rc=$? + expect_code 0 "$rc" "the remote home's worker, or a handed-off one, took the machine's place: $out" + pass "every local home shares one declared capacity per project origin, and remote homes do not count" +} + +# A local home's state directory or task record that cannot be read could hide a +# holder, so admission refuses instead of counting without it. +test_unreadable_holders_refuse_admission() { + local case_dir root mate out rc=0 + if [ "$(id -u)" = 0 ]; then + printf 'ok - skipped the unreadable-holder case (root reads files regardless of mode)\n' + return 0 + fi + case_dir=$(make_case unreadable-holders task-c) + root="$case_dir/home" + mate="$case_dir/mate" + make_home "$mate" + printf '%s\n' schema=fm-secondmate-parent.v1 route=local "parent_home=$root" > "$mate/.fm-secondmate-parent" + printf -- '- mate - a local mate (home: %s; scope: project work; projects: project; added 2026-09-01)\n' "$mate" \ + > "$root/data/secondmates.md" + declare_capacity "$root" "project 2" + + write_live "$root" live-a "$case_dir/project" + chmod 000 "$root/state/live-a.meta" + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + chmod 600 "$root/state/live-a.meta" + expect_code 1 "$rc" "an unreadable task record did not refuse admission: $out" + assert_contains "$out" "task record $root/state/live-a.meta cannot be read" \ + "the refusal did not name the unreadable task record" + assert_absent "$root/state/task-c.meta" "a spawn published past an unreadable task record" + + chmod 000 "$mate/state" + rc=0 + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + chmod 755 "$mate/state" + expect_code 1 "$rc" "an unreadable local state directory did not refuse admission: $out" + assert_contains "$out" "local Firstmate state directory $mate/state cannot be read" \ + "the refusal did not name the unreadable state directory" + assert_absent "$root/state/task-c.meta" "a spawn published past an unreadable state directory" + + rc=0 + out=$(spawn_ship "$case_dir" task-c) || rc=$? + expect_code 0 "$rc" "a readable machine did not admit the worker: $out" + pass "an unreadable state directory or task record refuses admission rather than undercounting" +} + +# Two spawns racing for the last place: the one holding the project lock +# publishes, the other cannot publish while it waits and is deferred afterwards. +test_concurrent_spawns_cannot_both_take_the_last_place() { + local case_dir home out rc=0 hold i wt + case_dir=$(make_case concurrent task-a task-b) + home="$case_dir/home" + declare_capacity "$home" "project 1" + hold="$case_dir/hold" + wt=$(new_worktree "$case_dir" task-a) + FM_FAKE_HOLD="$hold" spawn_ship "$case_dir" task-a "$wt" > "$case_dir/a.out" 2>&1 & + i=0 + while [ ! -f "$hold.reached" ]; do + i=$((i + 1)) + [ "$i" -lt 400 ] || fail "the first spawn never reached its launch: $(cat "$case_dir/a.out")" + sleep 0.05 + done + out=$(spawn_ship "$case_dir" task-b "$case_dir/unused") || rc=$? + [ "$rc" -ne 0 ] || fail "a concurrent spawn launched while the last place was being taken: $out" + assert_absent "$home/state/task-b.meta" "a concurrent spawn published past the last place" + : > "$hold.release" + wait || true + assert_contains "$(cat "$case_dir/a.out")" "spawned task-a" "the lock-holding spawn did not finish" + rc=0 + out=$(spawn_ship "$case_dir" task-b "$case_dir/unused") || rc=$? + expect_code "$DEFER_EXIT" "$rc" "the retried spawn was not deferred once the place was taken: $out" + assert_absent "$home/state/task-b.meta" "the retried spawn published past capacity" + pass "concurrent spawns can never both take the last place" +} + +# A spawn that fails after admission removes nothing it did not create and +# leaves no record, so it holds no place afterwards. +test_failed_spawn_after_admission_holds_no_place() { + local case_dir home out rc=0 + case_dir=$(make_case failed task-a task-b) + home="$case_dir/home" + declare_capacity "$home" "project 1" + printf '%s\n' '# Task' "## Captain's intent" '{TASK}' '' '## Firstmate spec' 'x' > "$home/data/task-a/brief.md" + out=$(spawn_ship "$case_dir" task-a "$case_dir/unused") || rc=$? + [ "$rc" -ne 0 ] && [ "$rc" -ne "$DEFER_EXIT" ] || fail "an invalid brief did not fail after admission (exit $rc): $out" + assert_contains "$out" "still contains {TASK}" "the spawn did not fail where expected" + assert_absent "$home/state/task-a.meta" "the failed spawn left a record" + rc=0 + out=$(spawn_ship "$case_dir" task-b) || rc=$? + expect_code 0 "$rc" "a failed spawn kept holding the only place: $out" + pass "a spawn that fails after admission leaves no record and holds no place" +} + +test_unreadable_declaration_refuses_every_spawn() { + local case_dir home out rc label body before worktrees + case_dir=$(make_case unreadable task-c) + home="$case_dir/home" + while IFS='|' read -r label body; do + [ -n "$label" ] || continue + printf '%b' "$body" > "$home/config/project-capacity" + before=$(call_count "$case_dir") + worktrees=$(worktree_list "$case_dir") + rc=0 + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code 1 "$rc" "$label: an unreadable declaration did not refuse: $out" + assert_contains "$out" "the project capacity declaration is unreadable" "$label: the refusal did not name the declaration" + assert_nothing_created "$case_dir" "$home" task-c "$before" "$worktrees" + done <<'ROWS' +missing capacity|project\n +zero capacity|project 0\n +non-numeric capacity|other-project two\n +trailing text|project 2 # suite\n +named twice|project 2\nproject 3\n +ROWS + rm -f "$home/config/project-capacity" + mkdir "$home/config/project-capacity" + rc=0 + out=$(spawn_ship "$case_dir" task-c "$case_dir/unused") || rc=$? + expect_code 1 "$rc" "a declaration that is not a file did not refuse: $out" + pass "an unreadable declaration refuses every fresh spawn rather than guessing the limit" +} + +test_batch_reports_a_deferred_pair() { + local case_dir home out rc=0 + case_dir=$(make_case batch task-c) + home="$case_dir/home" + declare_capacity "$home" "project 1" + write_live "$home" live-a "$case_dir/project" + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" "task-c=$case_dir/project" --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "a batch whose only pair was deferred did not exit with the deferral status: $out" + assert_contains "$out" "batch: DEFERRED task-c ($case_dir/project) - its project is at capacity, so it stays queued" \ + "the batch did not report the deferral" + assert_not_contains "$out" "batch: FAILED" "the batch reported a deferral as a failure" + pass "a batch reports a capacity deferral as deferred, not failed" +} + +# Orca owns its own worktrees and never takes the Treehouse allocation lock, so +# a declared capacity is what makes an Orca spawn take the shared project lock, +# even from an uncapped clone of a capped origin whose worker would still hold a +# place: it refuses while another holder has it, and defers at capacity before +# asking Orca for anything but its runtime status. +test_orca_spawn_is_admitted_under_the_shared_project_lock() { + local case_dir home out out2 rc=0 rc2 holder i + command -v node >/dev/null 2>&1 || { + printf 'ok - skipped the Orca capacity case (node, which the Orca status check needs, is not installed)\n' + return 0 + } + case_dir=$(make_case orca task-o) + home="$case_dir/home" + cat > "$case_dir/fakebin/orca" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = status ]; then + printf '{"ok":true,"result":{"runtime":{"reachable":true,"state":"ready"}}}\n' + exit 0 +fi +printf 'orca %s\n' "$*" >> "$FM_FAKE_CALL_LOG" +exit 1 +SH + chmod +x "$case_dir/fakebin/orca" + git clone -q "$(git -C "$case_dir/project" remote get-url origin)" "$case_dir/project-2" + declare_capacity "$home" "project 1" + + # shellcheck disable=SC2016 # expanded by the holder's own shell + FM_HOME="$home" FM_STATE_OVERRIDE='' bash -c ' + . "$1/bin/fm-wake-lib.sh" + lock=$(fm_treehouse_project_lock_path "$2") || exit 1 + fm_lock_try_acquire "$lock" || exit 1 + : > "$3.held" + while [ ! -f "$3.release" ]; do sleep 0.05; done + fm_lock_release "$lock" + ' _ "$ROOT" "$case_dir/project" "$case_dir/holder" & + holder=$! + i=0 + while [ ! -f "$case_dir/holder.held" ]; do + i=$((i + 1)) + [ "$i" -lt 200 ] || fail "the lock holder never took the project lock" + sleep 0.05 + done + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-o "$case_dir/project" \ + --backend orca --mode no-mistakes --yolo off) || rc=$? + rc2=0 + out2=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-o "$case_dir/project-2" \ + --backend orca --mode no-mistakes --yolo off) || rc2=$? + : > "$case_dir/holder.release" + wait "$holder" || true + expect_code 1 "$rc" "an Orca spawn ignored a held project lock: $out" + assert_contains "$out" "another spawn or cleanup holds the shared project lock for $case_dir/project; refusing to race its capacity admission" \ + "the Orca spawn did not refuse on the shared project lock" + expect_code 1 "$rc2" "an Orca spawn from an uncapped same-origin clone ignored the held project lock: $out2" + assert_contains "$out2" "another spawn or cleanup holds the shared project lock for $case_dir/project-2" \ + "the uncapped same-origin clone's Orca spawn did not refuse on the shared project lock" + assert_absent "$home/state/task-o.meta" "the uncapped clone's Orca spawn published a record while the lock was held" + assert_no_grep "orca " "$case_dir/calls.log" "the Orca spawn asked Orca for more than its runtime status" + + write_live "$home" live-a "$case_dir/project" + rc=0 + out=$(run_spawn "$case_dir" "$home" "$case_dir/unused" task-o "$case_dir/project" \ + --backend orca --mode no-mistakes --yolo off) || rc=$? + expect_code "$DEFER_EXIT" "$rc" "an Orca spawn beyond capacity was not deferred: $out" + assert_absent "$home/state/task-o.meta" "the deferred Orca spawn published a record" + assert_no_grep "orca " "$case_dir/calls.log" "the deferred Orca spawn created an Orca worktree" + pass "an Orca spawn takes the shared project lock whenever a same-origin clone is capped and defers before creating anything" +} + +test_undeclared_capacity_keeps_dispatch_uncapped +test_available_capacity_admits_the_worker +test_exhausted_capacity_defers_without_leaving_anything_behind +test_spaced_project_name_is_declared +test_hash_prefixed_project_name_is_declared +test_symlinked_home_counts_each_worker_once +test_release_frees_a_place +test_restart_does_not_count_its_own_record +test_occupancy_counts_only_this_projects_workers +test_capacity_is_shared_by_every_local_home +test_unreadable_holders_refuse_admission +test_concurrent_spawns_cannot_both_take_the_last_place +test_failed_spawn_after_admission_holds_no_place +test_unreadable_declaration_refuses_every_spawn +test_batch_reports_a_deferred_pair +test_orca_spawn_is_admitted_under_the_shared_project_lock diff --git a/tests/fm-supervision-host-attended-live-e2e.test.sh b/tests/fm-supervision-host-attended-live-e2e.test.sh index 2c4afbc2f20..aaff7c7c16e 100755 --- a/tests/fm-supervision-host-attended-live-e2e.test.sh +++ b/tests/fm-supervision-host-attended-live-e2e.test.sh @@ -37,6 +37,8 @@ set -u # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=bin/fm-tmux-lib.sh +. "$ROOT/bin/fm-tmux-lib.sh" fm_live_gate opt-in FM_SUPERVISION_HOST_ATTENDED_LIVE_E2E claude tmux jq node perl git @@ -242,11 +244,22 @@ choose() { #