From ceb0932c3385ec51329478c376acc85556df65a1 Mon Sep 17 00:00:00 2001 From: 420tombombadil Date: Fri, 31 Jul 2026 23:02:30 -0600 Subject: [PATCH 01/14] fix: reconcile retained supervision state --- .agents/skills/bearings/SKILL.md | 20 +- .../skills/decision-hold-lifecycle/SKILL.md | 17 +- .agents/skills/firstmate-codexapp/SKILL.md | 3 +- .../skills/stuck-crewmate-recovery/SKILL.md | 6 +- AGENTS.md | 9 +- bin/fm-arm-command-policy.mjs | 32 +- bin/fm-backend.sh | 27 + bin/fm-bearings-report.sh | 49 ++ bin/fm-brief.sh | 101 +++- bin/fm-classify-lib.sh | 34 +- bin/fm-crew-state.sh | 82 ++- bin/fm-custody-lib.sh | 77 +++ bin/fm-decision-hold.sh | 498 +++++++++++++++++- bin/fm-model-capacity-hold-lib.sh | 33 ++ bin/fm-model-capacity-hold.sh | 150 ++++++ bin/fm-process-progress.sh | 74 +++ bin/fm-record-reconcile.sh | 157 ++++++ bin/fm-send.sh | 18 +- bin/fm-session-start.sh | 29 +- bin/fm-spawn.sh | 3 + bin/fm-supervise-daemon.sh | 11 +- bin/fm-test-run.sh | 24 +- bin/fm-wake-drain.sh | 12 +- bin/fm-watch-arm.sh | 36 ++ bin/fm-watch.sh | 167 +++++- docs/architecture.md | 19 +- docs/arm-pretool-check.md | 4 + docs/scripts.md | 10 +- docs/watcher-continuity.md | 3 + tests/fm-arm-pretool-check.test.sh | 5 + tests/fm-backend.test.sh | 21 +- tests/fm-bearings-report.test.sh | 59 +++ tests/fm-brief.test.sh | 71 ++- tests/fm-crew-state.test.sh | 57 +- tests/fm-daemon.test.sh | 24 + tests/fm-decision-hold-lifecycle.test.sh | 228 +++++++- tests/fm-model-capacity-hold.test.sh | 88 ++++ tests/fm-record-reconcile.test.sh | 85 +++ tests/fm-send-strict.test.sh | 30 +- tests/fm-session-start.test.sh | 2 + tests/fm-wake-queue.test.sh | 57 +- tests/fm-watch-arm-ownership.test.sh | 57 ++ tests/fm-watch-triage.test.sh | 144 +++++ 43 files changed, 2528 insertions(+), 105 deletions(-) create mode 100755 bin/fm-bearings-report.sh create mode 100644 bin/fm-custody-lib.sh create mode 100644 bin/fm-model-capacity-hold-lib.sh create mode 100755 bin/fm-model-capacity-hold.sh create mode 100755 bin/fm-process-progress.sh create mode 100755 bin/fm-record-reconcile.sh create mode 100755 tests/fm-bearings-report.test.sh create mode 100755 tests/fm-model-capacity-hold.test.sh create mode 100755 tests/fm-record-reconcile.test.sh create mode 100755 tests/fm-watch-arm-ownership.test.sh diff --git a/.agents/skills/bearings/SKILL.md b/.agents/skills/bearings/SKILL.md index 42990edd04f..645452a7c93 100644 --- a/.agents/skills/bearings/SKILL.md +++ b/.agents/skills/bearings/SKILL.md @@ -14,13 +14,14 @@ metadata: Generate a complete current snapshot from the fleet's current state, so the captain can resume in one read after a break, a night, or a context reset. Plain `/bearings` returns only the concise four-section chat digest. Only `/bearings file` writes the dated markdown report artifact and then returns the concise four-section chat digest linked to that report. -This skill is operationally read-only in both modes. -It never tears down a task, merges a PR, dispatches new work, steers a worker, answers a decision, cleans up work, mutates backlog or task state, or writes any file except the single dated report in explicit file mode. +This skill is operationally read-only in plain mode and custody-preserving in file mode. +It never tears down a task, merges a PR, dispatches new work, steers a worker, answers a decision, cleans up work, or mutates backlog or task state. +Explicit file mode may write only the dated report plus the version and receipt for a prior same-day report through `bin/fm-bearings-report.sh`. ## Invocation modes - Plain `/bearings` gathers a fresh bounded snapshot and renders the four-section chat digest without creating, deleting, reading, or replacing `data/status-report-.md`. -- `/bearings file` gathers a fresh bounded snapshot, replaces today's `data/status-report-.md` from scratch, and renders the four-section chat digest with a link or path to that report. +- `/bearings file` gathers a fresh bounded snapshot, versions any prior same-day report before replacement, and renders the four-section chat digest with a link or path to that report. - Treat `file` only as an explicit invocation option in the slash command. - Do not treat natural-language requests such as "write a report", "save this", "persist it", or "make a file" as file mode unless the invocation explicitly includes the standalone `file` option. - When the captain asks to include PRs, pass the snapshot command's live-PR opt-in. @@ -49,12 +50,14 @@ It never tears down a task, merges a PR, dispatches new work, steers a worker, a The chat response uses the four complete sections in the chat-response contract below, in the same order, each always present. Plain mode stops here and writes no report artifact. -3. **In explicit file mode only, compose and replace the detailed report file.** +3. **In explicit file mode only, compose and custody-preserve the detailed report file.** The report uses the same four complete sections as the chat, in the same order, and adds the detail the chat omits. Never read an earlier `data/status-report-*.md` to decide what to omit, include, describe as changed, or call current. - Write the full report to `data/status-report-.md` using today's date. - If today's file already exists, delete it first, then create a new file from scratch. - This is the only write allowed by the skill. + Compose the full report in a temporary draft outside `data/status-report-*.md`, then install it with `bin/fm-bearings-report.sh `. + The installer writes `data/status-report-.md` using today's date. + If today's file already exists, the installer first copies its exact bytes to a create-exclusive unique path under `data/status-report-versions/` and writes a pre-transition receipt containing its digest, byte count, source ref, custody class, and timestamp. + Never delete, truncate, or directly overwrite the dated report, and never hand-write the custody receipt. + The version and receipt preserve evidence only; never read them to decide what belongs in the new current report. The detailed report includes: - **Title** - `# Bearings - ` (use "Morning status" only when the captain specifically asks for a morning brief), followed by two or three sentences framing where things stand. - **Captain's Call** - every open decision summarized with its options from the structured decision record, plus each PR ready to merge and each needed credential or login, every PR with the full `https://...` URL, never a bare `#number`. @@ -103,5 +106,6 @@ Rules that keep the contract unambiguous: ## Supervision discipline This skill changes no fleet state. -Do not tear down a task, merge a PR, dispatch queued work, steer a worker, answer a queued decision, clean up work, or mutate any `state/` or `data/` file other than the single report file in explicit file mode. +Do not tear down a task, merge a PR, dispatch queued work, steer a worker, answer a queued decision, clean up work, or mutate task state. +In file mode, the dated report and its prior-version custody artifacts are the only permitted `data/` writes. If the state you read suggests an action - a PR ready to merge, a queued item whose gate has arrived, or a needs-decision finding - name it in its section and leave the action to the normal lifecycle and configured authority rather than taking it from inside this skill. diff --git a/.agents/skills/decision-hold-lifecycle/SKILL.md b/.agents/skills/decision-hold-lifecycle/SKILL.md index 5db5690ebc9..0d49f041348 100644 --- a/.agents/skills/decision-hold-lifecycle/SKILL.md +++ b/.agents/skills/decision-hold-lifecycle/SKILL.md @@ -18,10 +18,18 @@ Every unresolved decision that belongs to the captain and is discovered while pr The agent performs the semantic inventory because scripts must not infer decisions from report prose, visual-review artifacts, terminal output, or chat. Give each distinct unresolved decision a stable privacy-safe key, register it through `bin/fm-decision-hold.sh hold`, and use the same key on retry so registration is idempotent while different decisions retain different durable identities. After inventorying the whole report and review surface, run `bin/fm-decision-hold.sh complete` with every unresolved key, or with `--none` only when the reviewed surface contains no unresolved captain decision. +A live originating task may be cleaned up after `complete` retains its report-bound task and dispatch carrier, including for `--none`, binds every unresolved hold to that exact dispatch, and proves each hold reappears in Bearings. +Cleanup does not require the captain to answer in the same session, and it never closes the hold. +Do not hand-write a backlog row, archive row, decision object, or receipt to satisfy this gate; only the script-owned lifecycle is authoritative. A completed investigation and an ended visual review use this same owner and completion command; a visual tool, including Lavish, never owns a parallel completion policy. Run the command in the originating work's authoritative `FM_HOME`; main-home work creates main-home holds, and secondmate-owned work creates holds in that secondmate home's backlog rather than copying them into the main backlog. Do not close a hold merely because the originating investigation completed, its report was archived, its visual review ended, or its task was torn down. The hold remains the authoritative Captain's Call item until the captain's answer is durably recorded, dependent work is created in the same backlog and blocked by that hold, and `bin/fm-decision-hold.sh resolve` routes the answer by clearing those dependency edges before closing the hold. +Resolution retains a digest-bound decision object and a task-and-dispatch-bound receipt. +For a historical open hold whose exact endpoint dispatch binding survives, re-run `complete` with the full recorded inventory to reconstruct the cleanup receipt from current authoritative state. +If that binding is absent, or if a historical resolved row lacks its script-owned cleanup receipt or canonical decision object, preserve the origin metadata and hold row and keep cleanup refused because no safe automatic migration is shipped. +An exact `resolve` retry may finish a missing resolution receipt only when the script-owned cleanup receipt and canonical decision object both survive. +Never substitute force or discard for the missing historical authority. Resolved findings, recommendations that need no captain choice, and prose that merely sounds decision-like do not create holds. Bearings reads the resulting structured state and must never compensate by scraping historical reports, visual-review artifacts, terminal output, chat, or other prose. @@ -31,10 +39,11 @@ Bearings reads the resulting structured state and must never compensate by scrap 2. Inventory only genuine unresolved choices that require the captain. 3. For each choice, choose a stable key and use the script's `hold` command with a concise title, reason, and repository. 4. Run the script's `complete` command with the full unresolved-key inventory for that review pass. -5. Relay the choices to the captain as decisions from Bearings' Captain's Call section under `AGENTS.md` section 9; do not use the word hold in captain chat. -6. After the captain decides, record dependent work with normal tasks-axi commands and block it by the hold identity. -7. Put the captain's exact durable decision in a file and use the script's `resolve` command with every routed task. -8. Confirm Bearings no longer shows the closed hold and that routed work remains in structured backlog state. +5. Before cleanup, let the script's read-only `verify` command confirm the task identity, exact dispatch, nonzero object digests, and a fresh Bearings appearance; unresolved or indeterminate evidence refuses and follows the historical remedy above without force or discard. +6. Relay the choices to the captain as decisions from Bearings' Captain's Call section under `AGENTS.md` section 9; do not use the word hold in captain chat. +7. After the captain decides, record dependent work with normal tasks-axi commands and block it by the hold identity. +8. Put the captain's exact durable decision in a file and use the script's `resolve` command with every routed task. +9. Confirm `verify-resolution` accepts the trusted receipt, Bearings no longer shows the closed hold, and routed work remains in structured backlog state. `bin/fm-decision-hold.sh --help` owns command syntax, identity construction, completion attestation, retry behavior, and close ordering. `docs/decision-hold-lifecycle.md` records the mechanism and regression evidence without restating this policy. diff --git a/.agents/skills/firstmate-codexapp/SKILL.md b/.agents/skills/firstmate-codexapp/SKILL.md index 6428439639a..cf03a14050c 100644 --- a/.agents/skills/firstmate-codexapp/SKILL.md +++ b/.agents/skills/firstmate-codexapp/SKILL.md @@ -61,8 +61,9 @@ For a Firstmate-managed task, include an explicit status instruction: ```text Append supervisor-visible status lines to /state/.status. -Use only these prefixes for status changes: working:, needs-decision:, blocked:, paused:, done:, failed:. +Use only these prefixes for status changes: working:, needs-decision:, blocked:, paused:, awaiting-captain:, done:, failed:. Use paused: only for a deliberate known external wait that should be rechecked later, never for a blocker that needs firstmate to act. +Use awaiting-captain [key=]: only after work is complete and an unbounded captain answer is required, and close it only with resolved [key=]: after the captain answers. Before doing substantive work, append "working: Codex Desktop thread started". ``` diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md index d97ebee3025..a47c8a14c88 100644 --- a/.agents/skills/stuck-crewmate-recovery/SKILL.md +++ b/.agents/skills/stuck-crewmate-recovery/SKILL.md @@ -23,7 +23,10 @@ Load `secondmate-provisioning` instead for `kind=secondmate` recovery. Treat the digest's endpoint result as a presence signal, not proof that the task's work or validation run is gone. Read the targeted current state with `bin/fm-crew-state.sh ` before deciding to relaunch. -A no-mistakes run matched to the crew's branch and current code remains authoritative when the endpoint is dead: handle a terminal or parked run through the normal lifecycle, and keep supervising an active run instead of creating a duplicate worker. +A pane and a detached pipeline worker are independent supervision subjects: interrupting or exiting the pane affects only the interactive agent, not a headless native agent or its process tree. +A no-mistakes run matched to the crew's branch and current code remains authoritative when the endpoint is dead: handle a terminal or parked run through the normal lifecycle, and keep supervising an active headless worker instead of recording the task stopped or creating a duplicate worker. +When the recorded pipeline step is `running` but its bound native-agent PID is dead or suspended, report that contradiction instead of recording the run as live; use the pipeline's supported abort and custody flow only after its process ownership is reconciled. +If the run record cannot be bound to a native-agent identity, report that limit as indeterminate rather than inferring live or dead from the pane. When no authoritative run accounts for the task, inspect only its recorded backend and worktree inventory. Use `treehouse status` for treehouse-backed tmux, herdr, zellij, or cmux tasks, and use the recorded `orca_worktree_id=` and `terminal=` for Orca tasks. @@ -42,6 +45,7 @@ Escalate in order: 2. If the crewmate is waiting on a question its brief already answers, answer in one line via `FM_HOME= bin/fm-send.sh` from an active firstmate session unless `FM_HOME` is already set to the active firstmate home. 3. If the crewmate is confused or looping, interrupt with the adapter's interrupt key, then redirect with one corrective line. For example, for a single-Escape adapter: `FM_HOME= bin/fm-send.sh --key Escape`. + Re-read `bin/fm-crew-state.sh ` immediately afterward: the interrupt does not stop a detached validation worker, and a still-live headless worker keeps the task working. 4. If the crewmate is genuinely wedged after redirection, exit the agent with the adapter's exit command and relaunch with the same brief plus a `progress so far` note appended to it. Genuine wedging means looping, unresponsive, repeating the same obstacle, or truly dead. A low context reading is not wedging; modern harnesses auto-compact and keep going. diff --git a/AGENTS.md b/AGENTS.md index f0af22e686f..50bb4c2c603 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -321,12 +321,16 @@ Apart from that single supported abort, do not hand-edit, commit, restart, or st Once ownership is settled, validate exactly once against that final head so no obsolete or intermediate head is ever treated as authoritative. An ask-user finding returns as `needs-decision`; firstmate decides only when the configured authority permits, otherwise escalates to the captain. +When completed producer work is retained solely for an answer only the captain can give, record `awaiting-captain [key=]: ` after the question has surfaced. +This state is an unbounded human wait, not a bounded external pause and not a blocker firstmate can clear; only a matching `resolved [key=]: ` closes it. +It never authorizes teardown of unlanded work and never outranks evidence that the worker or its headless pipeline resumed. Send the same worker one exact decision naming the decision key, step, action, affected finding IDs, instructions where needed, and exact response command. Require the matching `resolved` event, forbid `--yes`, and require the worker to process every synchronous return until completion or a genuinely new escalation. Resume fleet supervision immediately after the decision lands. Judge validation by the current-code-matched run step through `bin/fm-crew-state.sh`, not by shell liveness or the last status event. Running, fixing, or CI states remain working; parked approval or fix-review states require the worker to follow the active gate help; passed or checks-passed is done; failed or cancelled is failed. +Pane lifecycle and headless pipeline lifecycle are independent: an interrupt or exit affects only the pane, so a bound live native worker keeps the task working, while a dead or suspended bound native worker contradicts a stale `running` ledger and must be reconciled before the task is called live. A worker hand-editing, committing, aborting, or restarting during an active validation run duplicates pipeline ownership outside the supersession sequence above; steer it back to the gate response flow. The worker reports the PR when CI first becomes green rather than waiting for merge monitoring to finish. @@ -369,11 +373,12 @@ Session start is the only exception because its one-shot digest already drained Treat any `OPEN DECISIONS` section from the drain as actionable reconciliation input even when no wake record was queued. A status line is a wake event, not current state; use `bin/fm-crew-state.sh` when current state matters, especially before re-escalating an old decision, blocker, or pause. A declared `paused:` event means a bounded external wait expected to clear on its own, while `blocked:` means firstmate action is needed. +An `awaiting-captain:` event means completed work is retained for an unbounded captain answer; after its first wake, supervision retires it from stale escalation only while the terminal event, clean worktree, and recorded HEAD remain unchanged, and only the captain's keyed answer clears it. Handle actionable wakes as follows: 1. For `signal:`, read the listed event lines first, then reconcile current state only where action depends on it. -2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. +2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. Never infer that a detached pipeline stopped from a pane interrupt or infer that a dead bound native worker is live from a `running` row. 3. For `check:`, act on the named poll result, including merges, X-mode events, and process-to-event source results. 4. For `heartbeat:`, review the whole fleet from the structured fleet view, reconcile suspicious tasks and PR state, update the backlog, and never report an unchanged fleet as progress. @@ -419,7 +424,7 @@ When evidence uses an internal label, rewrite it before sending: - worktree, checkout, primary checkout, or local-main -> local copy, isolated copy, or local branch, only if the location matters. - teardown -> cleanup. - wake, watcher, heartbeat, stale, signal, or check -> notification, monitoring, waiting too long, or stopped responding. -- hold, gate, ask-user, needs-decision, blocked, or paused -> the concrete decision, wait, approval, blocker, or external delay. +- hold, gate, ask-user, needs-decision, blocked, paused, or awaiting-captain -> the concrete decision, wait, approval, blocker, or external delay. - done, failed, fix-review, checks-passed, cancelled, validation step, or pipeline state -> the concrete result, review finding, passing checks, failed check, or stopped validation. - brief -> instructions. - crewmate -> worker, only when naming the helper matters. diff --git a/bin/fm-arm-command-policy.mjs b/bin/fm-arm-command-policy.mjs index 846965fa9a5..a1f30577755 100755 --- a/bin/fm-arm-command-policy.mjs +++ b/bin/fm-arm-command-policy.mjs @@ -619,31 +619,47 @@ function shellInvocation(position) { const name = basename(position.command.value); if (!["sh", "bash", "zsh"].includes(name)) return null; const words = position.words; + let noexec = false; for (let i = position.index + 1; i < words.length; i += 1) { const option = words[i]; if (/^-[A-Za-z]*c[A-Za-z]*$/.test(option.value)) { + if (option.value.slice(1).includes("n")) noexec = true; let payloadIndex = i + 1; if (words[payloadIndex]?.value === "--") payloadIndex += 1; - return { kind: "command", payload: words[payloadIndex] || null }; + return { kind: "command", payload: words[payloadIndex] || null, noexec }; } if (/^[-+]O$/.test(option.value)) { i += 1; continue; } - if (option.value === "--" || /^[-+]/.test(option.value)) continue; - return { kind: "script", payload: option }; + if (option.value === "--noexec") { + noexec = true; + continue; + } + if (option.value === "--") { + const payload = words[i + 1] || null; + return payload ? { kind: "script", payload, noexec } : { kind: "stdin", payload: null, noexec }; + } + if (/^[-+][A-Za-z]+$/.test(option.value)) { + if (option.value.slice(1).includes("n")) noexec = option.value.startsWith("-"); + continue; + } + if (/^[-+]/.test(option.value)) continue; + return { kind: "script", payload: option, noexec }; } - return { kind: "stdin", payload: null }; + return { kind: "stdin", payload: null, noexec }; } function shellHeredocPayloads(tokens, position) { - if (shellInvocation(position)?.kind !== "stdin") return []; + const shell = shellInvocation(position); + if (shell?.kind !== "stdin" || shell.noexec) return []; const heredocs = tokens.filter((token) => token.type === "redir" && token.fd === 0 && typeof token.heredoc === "string"); return heredocs.length === 0 ? [] : [heredocs.at(-1).heredoc]; } function shellHereStringPayloads(tokens, position) { - if (shellInvocation(position)?.kind !== "stdin") return []; + const shell = shellInvocation(position); + if (shell?.kind !== "stdin" || shell.noexec) return []; const payloads = []; for (let i = 0; i < tokens.length; i += 1) { const token = tokens[i]; @@ -787,8 +803,8 @@ function analyzeProgram(command, context, depth = 0) { } const shell = shellInvocation(position); - const shellPayload = shell?.kind === "command" ? shell.payload : null; - const shellScript = shell?.kind === "script" ? shell.payload : null; + const shellPayload = shell?.kind === "command" && !shell.noexec ? shell.payload : null; + const shellScript = shell?.kind === "script" && !shell.noexec ? shell.payload : null; const sourceScript = sourcedScript(position); const literalEvalPayload = evalPayload(position); const heredocPayloads = shellHeredocPayloads(tokens, position); diff --git a/bin/fm-backend.sh b/bin/fm-backend.sh index e505b99f757..7830114fc48 100644 --- a/bin/fm-backend.sh +++ b/bin/fm-backend.sh @@ -818,6 +818,33 @@ fm_backend_composer_state() { # -> empty|pending|pending-unp esac } +# fm_backend_process_root_pid: return the shell/process root whose descendant +# closure belongs to one pane. Only tmux and herdr expose a verified PID binding; +# other backends fail without guessing so progress sampling stays supplemental. +fm_backend_process_root_pid() { # + local backend=$1 target=$2 session pane info + case "$backend" in + tmux) + tmux display-message -p -t "$target" '#{pane_pid}' 2>/dev/null + ;; + herdr) + fm_backend_source herdr || return 1 + session=${target%%:*} + pane=${target#*:} + [ -n "$session" ] && [ -n "$pane" ] && [ "$pane" != "$target" ] || return 1 + info=$(fm_backend_herdr_cli "$session" pane process-info --pane "$pane" 2>/dev/null) || return 1 + printf '%s' "$info" | jq -er --arg pane "$pane" ' + .result.process_info + | select(.pane_id == $pane) + | .shell_pid + | select(type == "number" and . > 1) + | floor + ' 2>/dev/null + ;; + *) return 1 ;; + esac +} + # fm_backend_target_exists: cheap, READ-ONLY existence check - does the # recorded TARGET endpoint still exist on BACKEND? Never starts a server or # session: for herdr this deliberately queries the pane directly instead of diff --git a/bin/fm-bearings-report.sh b/bin/fm-bearings-report.sh new file mode 100755 index 00000000000..0f8c974964d --- /dev/null +++ b/bin/fm-bearings-report.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# Install a complete Bearings draft as today's dated report. If a same-day +# report exists, preserve its exact bytes at a unique create-exclusive path and +# write a digest-bound custody receipt before replacing it. +# +# Usage: fm-bearings-report.sh +# +# FM_BEARINGS_REPORT_DATE may pin YYYY-MM-DD for deterministic tests. The draft +# remains untouched. The command prints the installed report path. +set -eu + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" + +# shellcheck source=bin/fm-custody-lib.sh +. "$SCRIPT_DIR/fm-custody-lib.sh" + +fail() { + printf 'fm-bearings-report: %s\n' "$*" >&2 + exit 1 +} + +[ "$#" -eq 1 ] || fail "usage: fm-bearings-report.sh " +draft=$1 +[ -f "$draft" ] && [ ! -L "$draft" ] || fail "draft must be a regular non-symlink file: $draft" +report_date=${FM_BEARINGS_REPORT_DATE:-$(date '+%Y-%m-%d')} +case "$report_date" in + ????-??-??) ;; + *) fail "report date must be YYYY-MM-DD: $report_date" ;; +esac +mkdir -p "$DATA" +target="$DATA/status-report-$report_date.md" +versions="$DATA/status-report-versions" +source_ref=$(git -C "$FM_ROOT" rev-parse HEAD 2>/dev/null || printf 'unversioned-firstmate-root') + +if [ -e "$target" ]; then + [ -f "$target" ] && [ ! -L "$target" ] || fail "existing report is not a regular non-symlink file: $target" + fm_custody_preserve "$target" "$versions" "$source_ref" bearings-prior-snapshot >/dev/null \ + || fail "could not preserve the prior report before replacement" +fi + +tmp="$DATA/.status-report-$report_date.${BASHPID:-$$}.tmp" +trap 'rm -f "$tmp"' EXIT INT TERM +(umask 077; cp "$draft" "$tmp") || fail "could not stage the complete report" +mv "$tmp" "$target" || fail "could not install the complete report" +trap - EXIT INT TERM +printf '%s\n' "$target" diff --git a/bin/fm-brief.sh b/bin/fm-brief.sh index 79f835e342d..4d4446ea7ae 100755 --- a/bin/fm-brief.sh +++ b/bin/fm-brief.sh @@ -6,8 +6,8 @@ # description, acceptance criteria, and context, and may adjust other sections # when the task genuinely deviates (e.g. working an existing external PR instead # of shipping a new one). -# Usage: fm-brief.sh --mode [--herdr-lab] -# fm-brief.sh --scout [--herdr-lab] +# Usage: fm-brief.sh --mode [--herdr-lab] [--spec-forging] +# fm-brief.sh --scout [--herdr-lab] [--spec-forging] # fm-brief.sh --secondmate {...|--no-projects} # --scout writes the scout contract instead: the deliverable is a report at # data//report.md (no branch, no push, no PR) and the worktree is scratch. @@ -27,6 +27,15 @@ # The flag must be explicit because {TASK} is filled after scaffolding and the # caller-supplied repo string cannot reliably identify this repo. Briefs made # without it carry a loud declaration so an omitted contract cannot be silent. +# --spec-forging is mandatory when the task will forge requirements, design, +# or tasks from a signed product contract. It requires all eight Faber +# preflight data through these environment variables: +# FM_SPEC_FORGING_PRD FM_SPEC_FORGING_AGENTS FM_SPEC_FORGING_REPO +# FM_SPEC_FORGING_FILES FM_SPEC_FORGING_SLICE FM_SPEC_FORGING_VISUAL_GATE +# FM_SPEC_FORGING_SURFACE FM_SPEC_FORGING_VALIDATION_GATE +# Missing, whitespace-only, or multiline data fail before a task directory is +# created. The generated gate returns producer scope changes to an independently +# commissioned architect and never lets the producer appoint itself. # For ship tasks, --mode is REQUIRED and shapes the definition of done. Firstmate # resolves it per task at intake (AGENTS.md section 7); data/projects.md holds the # captain's standing posture as context, and this script never reads it: @@ -48,7 +57,9 @@ # Every scaffold's status protocol distinguishes the configured # declared-external-wait verb (FM_CLASSIFY_PAUSED_VERB, default "paused") from # "blocked:": pause for a known external wait expected to clear on its own, -# blocked when firstmate must act. +# blocked when firstmate must act. The separate fixed `awaiting-captain:` verb is +# an unbounded captain decision after the producer work is complete; it remains +# open until an explicit keyed resolution records the captain's answer. # Ship tasks include a project-memory section so durable project-intrinsic # learnings can be committed to AGENTS.md through the project's delivery path; # it carries the AGENTS.md authoring bar (widely useful knowledge only, pointers @@ -103,6 +114,7 @@ else fi KIND=ship HERDR_LAB=0 +SPEC_FORGING=0 NO_PROJECTS=0 MODE= MODE_SET=0 @@ -124,6 +136,7 @@ for a in "$@"; do --scout) KIND=scout ;; --secondmate) KIND=secondmate ;; --herdr-lab) HERDR_LAB=1 ;; + --spec-forging) SPEC_FORGING=1 ;; --no-projects) NO_PROJECTS=1 ;; --mode) want_value=mode ;; --mode=*) MODE=${a#--mode=}; MODE_SET=1 ;; @@ -161,11 +174,44 @@ if [ "$KIND" = secondmate ] && [ "$HERDR_LAB" -eq 1 ]; then exit 1 fi +if [ "$KIND" = secondmate ] && [ "$SPEC_FORGING" -eq 1 ]; then + echo "error: --spec-forging applies only to crewmate ship or scout briefs" >&2 + exit 1 +fi + if [ "$NO_PROJECTS" -eq 1 ] && [ "$KIND" != secondmate ]; then echo "error: --no-projects applies only to --secondmate charters" >&2 exit 1 fi +require_spec_forging_datum() { + local variable=$1 datum=$2 value=${!1:-} + case "$value" in + *[![:space:]]*) : ;; + *) + echo "error: --spec-forging missing $datum ($variable)" >&2 + return 1 + ;; + esac + case "$value" in + *$'\n'*|*$'\r'*) + echo "error: --spec-forging $datum must be one line ($variable)" >&2 + return 1 + ;; + esac +} + +if [ "$SPEC_FORGING" -eq 1 ]; then + require_spec_forging_datum FM_SPEC_FORGING_PRD "exact PRD path and signature evidence" || exit 1 + require_spec_forging_datum FM_SPEC_FORGING_AGENTS "target repo AGENTS.md path" || exit 1 + require_spec_forging_datum FM_SPEC_FORGING_REPO "repo, root, verification ref, and push ref" || exit 1 + require_spec_forging_datum FM_SPEC_FORGING_FILES "exact spec directory and files to produce" || exit 1 + require_spec_forging_datum FM_SPEC_FORGING_SLICE "slice and out-of-scope" || exit 1 + require_spec_forging_datum FM_SPEC_FORGING_VISUAL_GATE "visual-gate state" || exit 1 + require_spec_forging_datum FM_SPEC_FORGING_SURFACE "forging surface" || exit 1 + require_spec_forging_datum FM_SPEC_FORGING_VALIDATION_GATE "validation-gate mechanism" || exit 1 +fi + BRIEF="$DATA/$ID/brief.md" [ -e "$BRIEF" ] && { echo "error: $BRIEF already exists" >&2; exit 1; } mkdir -p "$DATA/$ID" @@ -236,14 +282,15 @@ A message with NO marker is the captain typing directly into your pane: treat it Handle routine work yourself. Report only true captain-relevant outcomes or a declared external wait by appending one line: \`echo "{state}: {one short line}" >> $STATUS_FILE\` -States: working, needs-decision, blocked, $PAUSED_VERB, done, failed. +States: working, needs-decision, blocked, $PAUSED_VERB, awaiting-captain, done, failed. Use \`$PAUSED_VERB: {why}\` (distinct from \`blocked:\`) only when your domain is deliberately idling on a known external wait you expect to clear on its own; use \`blocked:\` when you are stuck and need firstmate to act. +Use \`awaiting-captain [key=]: {exact question}\` only after the producer work is complete and the exact question requires an unbounded captain answer; it surfaces once, stays quiet while the terminal status, clean tree, and HEAD remain unchanged, and closes only with a matching \`resolved [key=]: {captain answer}\` event. Use this only for material phase changes, a captain decision, a real blocker, a failure, or work ready for review. This is also how you return the answer to a marked from-firstmate request above. A marked request requires one correlated answer after the work; it does not require a separate receipt or start acknowledgement. Never append \`working:\` merely to acknowledge receipt or announce that a marked request has started. When a routed-work phase has a supervisor-actionable material change worth reporting under the rule above, give that reported phase a stable key. -If its first reportable event is \`working [key=]: {material phase}\`, use the same key on its later \`$PAUSED_VERB\`, \`done\`, \`failed\`, \`needs-decision\`, or \`blocked\` event so the earlier working phase is superseded. +If its first reportable event is \`working [key=]: {material phase}\`, use the same key on its later \`$PAUSED_VERB\`, \`awaiting-captain\`, \`done\`, \`failed\`, \`needs-decision\`, or \`blocked\` event so the earlier working phase is superseded. When a keyed phase ends without another reportable state, append \`resolved [key=]: {why it is no longer active}\`. When a decision you escalated is answered or a blocker clears and your domain resumes, append \`resolved: {how it was decided or unblocked}\` (keyed with \`[key=]\` if you opened it with one) so it is durably closed instead of resurfacing behind later unrelated events. Routine internal supervision, heartbeats, retries, and crewmate churn stay inside your own home and must not touch that status file. @@ -265,6 +312,34 @@ fi REPO=${POS[1]} +if [ "$SPEC_FORGING" -eq 1 ]; then +IFS= read -r -d '' SPEC_FORGING_SECTION < "$BRIEF" <> $STATUS_FILE\` - States: working, needs-decision, blocked, $PAUSED_VERB, done, failed. + States: working, needs-decision, blocked, $PAUSED_VERB, awaiting-captain, done, failed. Each append wakes firstmate, so report sparingly: only phase changes a supervisor would act on and the needs-decision/blocked/paused/done/failed states. No step-by-step FYI progress lines; firstmate reads your pane for that. @@ -326,6 +407,7 @@ The report is the only thing that survives, so anything worth keeping must be in known external wait you expect to clear on its own (an upstream release, a rate-limit reset): firstmate then leaves your idle pane alone and rechecks it on a long cadence instead of treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. + Use \`awaiting-captain [key=]: {exact question}\` only after the producer work is complete and the exact question requires an unbounded captain answer; it stays retired while the terminal status, clean tree, and HEAD remain unchanged and closes only after the captain answers through a matching \`resolved [key=]: {captain answer}\` event. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs to a human (product choices, destructive actions), append \`needs-decision: {summary of options}\` and stop. Firstmate will reply with the decision. @@ -413,7 +495,7 @@ You are a crewmate: an autonomous worker agent managed by firstmate. Work on you # Task {TASK} -$HERDR_SECTION +$BRIEF_CONTRACTS # Setup You are in a disposable git worktree of $REPO, at a detached HEAD on a clean default branch. @@ -430,7 +512,7 @@ $RULE1 3. Use gh-axi for GitHub operations and chrome-devtools-axi for browser operations. 4. Report status by appending one line: \`echo "{state}: {one short line}" >> $STATUS_FILE\` - States: working, needs-decision, blocked, $PAUSED_VERB, done, failed. + States: working, needs-decision, blocked, $PAUSED_VERB, awaiting-captain, done, failed. Each append wakes firstmate, so report sparingly: only phase changes a supervisor would act on (setup done, bug reproduced, fix implemented, validation passed) and the needs-decision/blocked/paused/done/failed states. No step-by-step FYI progress lines; @@ -441,6 +523,7 @@ $RULE1 known external wait you expect to clear on its own (an upstream release, a rate-limit reset, a scheduled window): firstmate then leaves your idle pane alone and rechecks it on a long cadence instead of treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. + Use \`awaiting-captain [key=]: {exact question}\` only after the producer work is complete and the exact question requires an unbounded captain answer; it stays retired while the terminal status, clean tree, and HEAD remain unchanged and closes only after the captain answers through a matching \`resolved [key=]: {captain answer}\` event. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs above the implementation worker (product choices, destructive actions, ask-user findings), append \`needs-decision: {summary of options}\` and stop. Firstmate will apply the configured authority and reply with the decision. diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh index 8ad7e6813b7..e6324b1b28e 100755 --- a/bin/fm-classify-lib.sh +++ b/bin/fm-classify-lib.sh @@ -43,6 +43,7 @@ FM_CREW_STATE_BIN="${FM_CREW_STATE_BIN:-$_FM_CLASSIFY_LIB_DIR/fm-crew-state.sh}" # merely because its prose contains one of those tokens (for example # "working: rebased onto merged #76"). FM_CLASSIFY_CAPTAIN_RE_DEFAULT='done:|needs-decision:|blocked:|failed:|PR ready|checks green|ready in branch|merged' +FM_CLASSIFY_AWAITING_CAPTAIN_VERB_DEFAULT='awaiting-captain' # The deliberate-external-wait verb. A crew (or firstmate steering it) appends # paused: @@ -88,11 +89,21 @@ status_is_terminal_verb() { [ -n "$line" ] || return 1 verb=$(status_line_verb "$line") case "$verb" in - done|needs-decision|blocked|failed) return 0 ;; + done|needs-decision|blocked|failed|"${FM_CLASSIFY_AWAITING_CAPTAIN_VERB:-$FM_CLASSIFY_AWAITING_CAPTAIN_VERB_DEFAULT}") return 0 ;; *) return 1 ;; esac } +# 0 only for a completed producer report. Unlike blocked, failed, or +# needs-decision, a surfaced done event requires no repeated stale escalation. +# Its worktree and metadata may still need retention for an independent gate or +# unlanded commits; callers must not treat this predicate as cleanup authority. +status_is_done() { # + local line=$1 + [ -n "$line" ] || return 1 + [ "$(status_line_verb "$line")" = 'done' ] +} + # 0 if the given (last) status line matches a captain-relevant verb. # Verb-aware by default: terminal verbs always match; nonterminal progress verbs # (working, resolved, captain-held) and paused never match from free-text prose; @@ -110,7 +121,7 @@ status_is_captain_relevant() { esac if [ -z "${FM_CAPTAIN_RE+x}" ]; then case "$verb" in - done|needs-decision|blocked|failed) return 0 ;; + done|needs-decision|blocked|failed|"${FM_CLASSIFY_AWAITING_CAPTAIN_VERB:-$FM_CLASSIFY_AWAITING_CAPTAIN_VERB_DEFAULT}") return 0 ;; esac fi printf '%s' "$line" | grep -qiE "${FM_CAPTAIN_RE:-$FM_CLASSIFY_CAPTAIN_RE_DEFAULT}" @@ -127,6 +138,16 @@ status_is_paused() { # [ "$verb" = "${FM_CLASSIFY_PAUSED_VERB:-$FM_CLASSIFY_PAUSED_VERB_DEFAULT}" ] } +# A durable, unbounded wait for a captain answer. Unlike paused, it is +# captain-relevant once and never implies an external condition will clear on +# its own. The keyed decision fold below keeps it open until resolved. +status_is_awaiting_captain() { # + local line=$1 verb + [ -n "$line" ] || return 1 + verb=$(status_line_verb "$line") + [ "$verb" = "${FM_CLASSIFY_AWAITING_CAPTAIN_VERB:-$FM_CLASSIFY_AWAITING_CAPTAIN_VERB_DEFAULT}" ] +} + # 0 if a status line declares either an external-wait pause or a verified # captain-held transfer. # Both declarations can intentionally leave an exited crew's endpoint idle, so @@ -224,7 +245,7 @@ status_open_decisions() { # verb=$(status_line_verb "$line") key=$(_fm_decision_key "$line") || continue case "$verb" in - needs-decision|blocked) + needs-decision|blocked|"${FM_CLASSIFY_AWAITING_CAPTAIN_VERB:-$FM_CLASSIFY_AWAITING_CAPTAIN_VERB_DEFAULT}") note=$(status_line_note "$line") open=$(_fm_decision_drop "$open" "$key") [ -n "$open" ] && open="${open}"$'\n' @@ -263,6 +284,11 @@ EOF return 0 } +status_awaiting_captain_decisions() { # + status_open_decisions "$1" | awk -F '\t' -v verb="${FM_CLASSIFY_AWAITING_CAPTAIN_VERB:-$FM_CLASSIFY_AWAITING_CAPTAIN_VERB_DEFAULT}" \ + '$2 == verb { print }' +} + # Fold material routed-work phases in the same keyed event stream. # A working or declared-pause event opens or replaces one phase for its key. # A later done, failed, needs-decision, blocked, or resolved event carrying that @@ -289,7 +315,7 @@ _fm_status_open_activities_stream() { [ -n "$open" ] && open="${open}"$'\n' open="${open}${key}"$'\t'"${verb}"$'\t'"${note}"$'\n' ;; - done|failed|needs-decision|blocked|"$resolve"|"$held") + done|failed|needs-decision|blocked|"${FM_CLASSIFY_AWAITING_CAPTAIN_VERB:-$FM_CLASSIFY_AWAITING_CAPTAIN_VERB_DEFAULT}"|"$resolve"|"$held") open=$(_fm_decision_drop "$open" "$key") [ -n "$open" ] && open="${open}"$'\n' ;; diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index 2cb290373cb..c6143f1d95e 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -16,7 +16,7 @@ # fixed mapping logic, no heuristics and no LLM. Output is one stable, parseable, # token-tight line firstmate can read every heartbeat: # -# state: · source: · +# state: · source: · # # Logic, in order: # 1. Resolve worktree + backend target + kind from state/.meta. @@ -35,6 +35,11 @@ # checks" from "checks green, waiting on merge" (see nm_ci_checks_state) - # a ci-step log-tail check overrides working -> done once checks read # green, so a green PR is never silently read as still-validating. +# For an active full run whose step log names its native worker PID, the +# process is checked independently of the pane: live remains working, +# stopped/suspended or absent never inherits the stale running row, and an +# unrecognized reused PID becomes unknown. Pane interruption cannot stop or +# prove the state of this detached worker. # 3. Reconcile the status log: if its last line says needs-decision/blocked but # the run-step shows the run moved on, the log is deterministically stale and # is flagged superseded. A genuinely parked run plus a needs-decision log @@ -121,6 +126,10 @@ log_last_line() { # reports `paused` distinctly, so a supervisor reading this sees a declared pause # and its reason rather than a wedge-suspect idle. map_log_state() { # + if status_is_awaiting_captain "$1"; then + echo awaiting-captain + return + fi if status_is_paused "$1"; then echo paused return @@ -304,6 +313,53 @@ nm_ci_checks_state() { *) printf 'unknown' ;; esac } + +# Resolve native no-mistakes step-worker health when the current step log exposes +# a start PID. Absence of such an identity is an explicit unsupported shape and +# leaves the run-record mapping unchanged; a named PID is never allowed to do so +# when the process is dead, suspended, or has been reused by an unrelated command. +NM_WORKER_HEALTH=unavailable +NM_WORKER_PID= +nm_headless_worker_health() { + local run_id step log started pid related ps_out stat command + NM_WORKER_HEALTH=unavailable + NM_WORKER_PID= + run_id=$(strip_quotes "$(nm_field id)") + [ -n "$run_id" ] || return 0 + step=$(printf '%s\n' "$RUN_OUT" \ + | awk -F, '$2 ~ /^[[:space:]]*"?(running|fixing)"?[[:space:]]*$/ { gsub(/^[[:space:]]+|[[:space:]]+$/, "", $1); found=$1 } END { print found }') + [ -n "$step" ] || step=$(strip_quotes "$(nm_field status)") + [ -n "$step" ] || return 0 + log=$(nm_run axi logs --step "$step" --run "$run_id") + [ -n "$log" ] || return 0 + started=$(printf '%s\n' "$log" \ + | grep -Ei '(codex|claude|opencode|grok|kimi|pi|agent).*started.*pid[=: ]+[0-9]+' \ + | tail -1) + [ -n "$started" ] || return 0 + pid=$(printf '%s\n' "$started" | sed -nE 's/.*pid[=: ]+([0-9]+).*/\1/p') + case "$pid" in ''|*[!0-9]*) return 0 ;; esac + NM_WORKER_PID=$pid + related=$(printf '%s\n' "$log" \ + | grep -Ei "((started|exited|stopped).*(pid[=: ]+)?$pid)|((pid[=: ]+)?$pid.*(started|exited|stopped))" \ + | tail -1) + case "$related" in + *exited*|*stopped*) NM_WORKER_HEALTH=dead; return 0 ;; + esac + ps_out=$(ps -p "$pid" -o stat= -o command= 2>/dev/null) || { + NM_WORKER_HEALTH=dead + return 0 + } + stat=$(printf '%s\n' "$ps_out" | awk 'NR == 1 { print $1 }') + command=${ps_out#*"$stat"} + if ! printf '%s\n' "$command" | grep -Eiq 'codex|claude|opencode|grok|kimi|(^|[ /])pi([ /]|$)|agent'; then + NM_WORKER_HEALTH=unknown + return 0 + fi + case "$stat" in + *T*) NM_WORKER_HEALTH=suspended ;; + *) NM_WORKER_HEALTH=live ;; + esac +} # Coarse fallback for cross-branch attribution. `no-mistakes axi status` (bare) # reports the active-or-most-recent run for the CURRENT branch when one # exists, else falls back to some other branch's run purely as informational @@ -513,6 +569,30 @@ if [ "$HAVE_RUN" = 1 ]; then fi fi + if [ "$RUN_STATE" = working ] && [ "$RUN_SOURCE" = full ]; then + nm_headless_worker_health + case "$NM_WORKER_HEALTH" in + live) + RUN_DETAIL="$RUN_DETAIL${SEP}headless pipeline worker live pid=$NM_WORKER_PID; pane state is independent" + ;; + dead) + RUN_STATE=blocked + RUN_DETAIL="run record still active${SEP}headless pipeline worker dead pid=$NM_WORKER_PID" + ;; + suspended) + RUN_STATE=blocked + RUN_DETAIL="run record still active${SEP}headless pipeline worker suspended pid=$NM_WORKER_PID" + ;; + unknown) + RUN_STATE=unknown + RUN_DETAIL="run record still active${SEP}headless worker pid=$NM_WORKER_PID has unrecognized identity" + ;; + *) + RUN_DETAIL="$RUN_DETAIL${SEP}headless worker identity unavailable; run record only" + ;; + esac + fi + # Reconcile the status log. A needs-decision/blocked log line that the run-step # has moved past (anything but a genuinely parked run) is deterministically # stale: the gate resolved and the run resumed or finished. diff --git a/bin/fm-custody-lib.sh b/bin/fm-custody-lib.sh new file mode 100644 index 00000000000..e4db9755a1c --- /dev/null +++ b/bin/fm-custody-lib.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# Create an immutable, create-exclusive copy of an ephemeral artifact and a +# machine-verifiable receipt before its caller performs a destructive transition. +# +# Source this file, then call: +# fm_custody_preserve +# +# The function prints "". It never removes or +# changes the source. Every output path is newly created with shell noclobber; +# retries choose another suffix and never replace prior evidence. + +fm_custody_sha256() { + if command -v shasum >/dev/null 2>&1; then + shasum -a 256 "$1" | awk '{print $1}' + elif command -v sha256sum >/dev/null 2>&1; then + sha256sum "$1" | awk '{print $1}' + else + printf 'fm-custody: shasum or sha256sum is required\n' >&2 + return 1 + fi +} + +fm_custody_preserve() { # + local source=$1 archive_dir=$2 source_ref=$3 custody_class=$4 + local timestamp stamp base attempt archived receipt digest bytes + [ -f "$source" ] && [ ! -L "$source" ] || { + printf 'fm-custody: source must be a regular non-symlink file: %s\n' "$source" >&2 + return 1 + } + case "$source_ref$custody_class" in + *$'\n'*|*$'\r'*) + printf 'fm-custody: source ref and custody class must be one line\n' >&2 + return 1 + ;; + esac + [ -n "$source_ref" ] && [ -n "$custody_class" ] || { + printf 'fm-custody: source ref and custody class are required\n' >&2 + return 1 + } + mkdir -p "$archive_dir" || return 1 + timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ') || return 1 + stamp=$(date -u '+%Y%m%dT%H%M%SZ') || return 1 + base=$(basename "$source") + attempt=0 + while [ "$attempt" -lt 100 ]; do + archived="$archive_dir/$base.$stamp.${BASHPID:-$$}.$attempt" + receipt="$archived.receipt" + if (set -C; umask 077; printf '%s' '' > "$archived") 2>/dev/null; then + break + fi + attempt=$((attempt + 1)) + done + [ "$attempt" -lt 100 ] || { + printf 'fm-custody: could not allocate a create-exclusive archive path\n' >&2 + return 1 + } + if ! cp "$source" "$archived"; then + printf 'fm-custody: could not copy source to %s\n' "$archived" >&2 + return 1 + fi + digest=$(fm_custody_sha256 "$archived") || return 1 + bytes=$(wc -c < "$archived" | tr -d '[:space:]') || return 1 + if ! (set -C; umask 077; { + printf 'schema=fm-custody.v1\n' + printf 'timestamp_utc=%s\n' "$timestamp" + printf 'source_path=%s\n' "$source" + printf 'source_ref=%s\n' "$source_ref" + printf 'custody_class=%s\n' "$custody_class" + printf 'archived_path=%s\n' "$archived" + printf 'sha256=%s\n' "$digest" + printf 'bytes=%s\n' "$bytes" + } > "$receipt") 2>/dev/null; then + printf 'fm-custody: could not create receipt %s\n' "$receipt" >&2 + return 1 + fi + printf '%s\t%s\n' "$archived" "$receipt" +} diff --git a/bin/fm-decision-hold.sh b/bin/fm-decision-hold.sh index aeb140a296a..4f76a930d08 100755 --- a/bin/fm-decision-hold.sh +++ b/bin/fm-decision-hold.sh @@ -22,6 +22,7 @@ # --title --reason <reason> [--repo <repo>] # fm-decision-hold.sh complete <origin-id> (--none | <decision-key>...) # fm-decision-hold.sh verify <origin-id> +# fm-decision-hold.sh verify-resolution <origin-id> <decision-key> # fm-decision-hold.sh resolve <origin-id> <decision-key> \ # --decision-file <path> --routed-to <task-id> [--routed-to <task-id>...] # @@ -30,13 +31,29 @@ # no unresolved captain decision. Later review passes may add keys; a live task's # metadata inventory is unioned idempotently. A post-teardown visual review can # complete against the surviving report and holds without recreating task state. -# `verify` is read-only and is called by scout teardown so teardown cannot erase a -# source before this gate has succeeded. +# `verify` is read-only and is called by scout teardown. An unresolved hold is +# cleanup-authoritative only after `complete` records a receipt under +# data/decision-hold-receipts/ that binds origin id, endpoint dispatch id, the +# exact backlog object digest, and a Bearings snapshot in which the hold appears. +# Every live completion, including `--none`, also retains a report-bound origin +# receipt so a later post-teardown visual-review pass keeps the authenticated +# task and dispatch association needed to create and verify hold receipts. +# This permits cleanup without requiring the captain to answer in the same session. # # `resolve` requires every --routed-to task to exist and to be blocked by the hold. # It writes the captain decision and routed identities into the hold body, clears # those dependency edges, and only then marks the hold Done. A failure before the # final step leaves the captain hold open. +# A resolved hold additionally retains the captain decision in +# <hold-id>.decision.md and records <hold-id>.resolved beside it. Verification +# requires one live-or-archived row, a nonzero decision digest, exact origin and +# dispatch links, the canonical existing decision object, every routed task, and +# matching object digests. Historical resolved rows without this receipt refuse; +# there is no implicit or hand-written-row migration. An open historical hold may +# reconstruct its cleanup receipt only by repeating `complete` while its exact +# endpoint dispatch binding survives. Binding-less or already-resolved rows whose +# script-owned objects are absent remain preserved and refused; no safe automatic +# migration, force, or discard is supplied by this command. set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -44,6 +61,7 @@ FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +RECEIPT_DIR="$DATA/decision-hold-receipts" # shellcheck source=bin/fm-classify-lib.sh # shellcheck disable=SC1091 @@ -90,6 +108,38 @@ sha256_text() { # <text> fi } +sha256_file() { # <path> + if command -v shasum >/dev/null 2>&1; then + shasum -a 256 "$1" | awk '{print $1}' + elif command -v sha256sum >/dev/null 2>&1; then + sha256sum "$1" | awk '{print $1}' + else + fail "shasum or sha256sum is required" + fi +} + +digest_is_nonzero_sha256() { # <digest> + printf '%s\n' "$1" | grep -Eq '^[0-9a-f]{64}$' || return 1 + [ "$1" != 0000000000000000000000000000000000000000000000000000000000000000 ] +} + +regular_nonsymlink_file() { # <path> + [ -f "$1" ] && [ ! -L "$1" ] +} + +write_atomic_file() { # <path> (content on stdin) + local path=$1 dir tmp + dir=${path%/*} + mkdir -p "$dir" || fail "could not create receipt directory: $dir" + tmp=$(mktemp "$dir/.receipt.tmp.XXXXXX") || fail "could not allocate receipt temporary file" + chmod 600 "$tmp" || { rm -f "$tmp"; fail "could not protect receipt temporary file"; } + if ! cat > "$tmp"; then + rm -f "$tmp" + fail "could not write receipt temporary file" + fi + mv "$tmp" "$path" || { rm -f "$tmp"; fail "could not publish receipt: $path"; } +} + hold_id() { # <origin-id> <decision-key> validate_slug origin-id "$1" validate_slug decision-key "$2" @@ -110,6 +160,70 @@ task_show() { # <id> tasks_axi show "$1" --full 2>/dev/null } +task_row_count() { # <id> + local id=$1 prefix file + prefix="- [x] $id - " + { + for file in "$DATA/backlog.md" "$DATA/done-archive.md"; do + [ -f "$file" ] || continue + awk -v open="- [ ] $id - " -v done="$prefix" ' + index($0, open) == 1 || index($0, done) == 1 { count++ } + END { print count + 0 } + ' "$file" + done + } | awk '{ total += $1 } END { print total + 0 }' +} + +task_row_source() { # <id> + local id=$1 file count + [ "$(task_row_count "$id")" -eq 1 ] || return 1 + for file in "$DATA/backlog.md" "$DATA/done-archive.md"; do + [ -f "$file" ] || continue + count=$(awk -v open="- [ ] $id - " -v done="- [x] $id - " ' + index($0, open) == 1 || index($0, done) == 1 { count++ } + END { print count + 0 } + ' "$file") || return 1 + if [ "$count" -eq 1 ]; then + printf '%s\n' "$file" + return 0 + fi + done + return 1 +} + +archived_task_show() { # <id> + local id=$1 archive="$DATA/done-archive.md" prefix + [ -f "$archive" ] || return 1 + prefix="- [x] $id - " + { + printf '## In flight\n\n## Queued\n\n## Done\n' + awk -v prefix="$prefix" ' + /^- \[[ x]\] / { + if (capture) exit + if (index($0, prefix) == 1) { + capture = 1 + print + } + next + } + capture { + if ($0 ~ /^## /) exit + print + } + ' "$archive" + } | tasks_axi show "$id" --full --file /dev/stdin 2>/dev/null +} + +task_show_durable() { # <id> + local id=$1 source + source=$(task_row_source "$id") || return 1 + if [ "$source" = "$DATA/backlog.md" ]; then + task_show "$id" + else + archived_task_show "$id" + fi +} + show_field() { # <show-output> <field> local output=$1 field=$2 printf '%s\n' "$output" | sed -n "s/^ $field: //p" | head -1 @@ -140,6 +254,21 @@ meta_value() { # <meta> <key> grep "^$2=" "$1" 2>/dev/null | tail -1 | cut -d= -f2- || true } +meta_exact_value() { # <meta> <key> + local meta=$1 key=$2 count + count=$(grep -c "^$key=" "$meta" 2>/dev/null || true) + [ "$count" -eq 1 ] || return 1 + sed -n "s/^$key=//p" "$meta" +} + +receipt_value() { # <receipt> <key> + local receipt=$1 key=$2 count + regular_nonsymlink_file "$receipt" || return 1 + count=$(grep -c "^$key=" "$receipt" 2>/dev/null || true) + [ "$count" -eq 1 ] || return 1 + sed -n "s/^$key=//p" "$receipt" +} + origin_open_decisions() { # <origin-id> local origin=$1 meta="$STATE/$1.meta" status_file="$STATE/$1.status" open kind last verb open=$(status_open_decisions "$status_file") @@ -172,7 +301,7 @@ verify_hold_active() { # <hold-id> verify_hold_resolved() { # <hold-id> local id=$1 show state kind body - show=$(task_show "$id") || return 1 + show=$(task_show_durable "$id") || return 1 state=$(show_field "$show" state) kind=$(show_field "$show" kind) body=$(show_field "$show" body) @@ -184,9 +313,10 @@ verify_hold_resolved() { # <hold-id> return 1 } -verify_hold_durable() { # <hold-id> - local id=$1 show state held kind hold_kind body - show=$(task_show "$id") || fail "captain decision $id is absent from $FM_HOME/data/backlog.md" +verify_hold_durable() { # <origin-id> <hold-id> + local origin=$1 id=$2 show state held kind hold_kind body + show=$(task_show_durable "$id") \ + || fail "captain decision $id is absent, duplicated, or indeterminate in the live backlog and archive" state=$(show_field "$show" state) held=$(show_field "$show" held) kind=$(show_field "$show" kind) @@ -197,12 +327,257 @@ verify_hold_durable() { # <hold-id> fi if [ "$state" = "done" ] && [ "$kind" = captain ]; then case "$body" in - *"Resolution recorded by fm-decision-hold."*"Routed work:"*) return 0 ;; + *"Resolution recorded by fm-decision-hold."*"Routed work:"*) + verify_resolution_receipt "$origin" "$id" + return 0 + ;; esac fi fail "captain decision $id is neither actively held nor durably resolved" } +resolution_body_fields() { # <hold-id> <body>; sets RESOLUTION_DIGEST RESOLUTION_ROUTES + local id=$1 body=$2 fields prefix + prefix='"Resolution recorded by fm-decision-hold.\nDecision digest: ' + case "$body" in + "$prefix"*) fields=${body#"$prefix"} ;; + *) fail "captain hold $id has no authoritative resolution body" ;; + esac + case "$fields" in + *'\nRouted identities: '*'\n\nCaptain decision:'*'\n\nRouted work:'*) : ;; + *) fail "captain hold $id has a malformed resolution body" ;; + esac + RESOLUTION_DIGEST=${fields%%\\n*} + fields=${fields#*\\nRouted identities: } + RESOLUTION_ROUTES=${fields%%\\n*} + digest_is_nonzero_sha256 "$RESOLUTION_DIGEST" \ + || fail "captain hold $id must carry a nonzero sha256 decision digest" + [ -n "$RESOLUTION_ROUTES" ] || fail "captain hold $id has no routed task identities" +} + +cleanup_receipt_path() { # <hold-id> + printf '%s/%s.hold\n' "$RECEIPT_DIR" "$1" +} + +resolution_receipt_path() { # <hold-id> + printf '%s/%s.resolved\n' "$RECEIPT_DIR" "$1" +} + +decision_object_path() { # <hold-id> + printf '%s/%s.decision.md\n' "$RECEIPT_DIR" "$1" +} + +origin_receipt_path() { # <origin-id> + printf '%s/%s.origin\n' "$RECEIPT_DIR" "$1" +} + +write_origin_receipt() { # <origin-id> <dispatch-id> + local origin=$1 dispatch=$2 report="$DATA/$1/report.md" report_sha receipt + regular_nonsymlink_file "$report" || fail "origin $origin has no safe report object to retain its dispatch binding" + report_sha=$(sha256_file "$report") + digest_is_nonzero_sha256 "$report_sha" || fail "origin $origin produced an invalid report digest" + receipt=$(origin_receipt_path "$origin") + { + printf 'schema=fm-decision-hold-origin.v1\n' + printf 'origin_id=%s\n' "$origin" + printf 'dispatch_id=%s\n' "$dispatch" + printf 'origin_path=%s\n' "$report" + printf 'origin_sha256=%s\n' "$report_sha" + } | write_atomic_file "$receipt" +} + +verify_origin_receipt() { # <origin-id> [expected-dispatch-id] + local origin=$1 expected=${2:-} receipt schema recorded_origin recorded_dispatch origin_path origin_sha + receipt=$(origin_receipt_path "$origin") + regular_nonsymlink_file "$receipt" || fail "origin $origin has no trusted dispatch carrier" + schema=$(receipt_value "$receipt" schema) || fail "origin $origin has a malformed dispatch carrier schema" + recorded_origin=$(receipt_value "$receipt" origin_id) || fail "origin $origin dispatch carrier has no task identity" + recorded_dispatch=$(receipt_value "$receipt" dispatch_id) || fail "origin $origin dispatch carrier has no dispatch identity" + origin_path=$(receipt_value "$receipt" origin_path) || fail "origin $origin dispatch carrier has no source path" + origin_sha=$(receipt_value "$receipt" origin_sha256) || fail "origin $origin dispatch carrier has no source digest" + [ "$schema" = fm-decision-hold-origin.v1 ] || fail "origin $origin has an unsupported dispatch carrier" + [ "$recorded_origin" = "$origin" ] || fail "origin $origin dispatch carrier links a different task" + validate_slug dispatch-id "$recorded_dispatch" + [ "$recorded_dispatch" = "$origin" ] || fail "origin $origin dispatch carrier links a different endpoint dispatch: $recorded_dispatch" + [ -z "$expected" ] || [ "$recorded_dispatch" = "$expected" ] \ + || fail "origin $origin dispatch carrier disagrees with cleanup dispatch $expected" + [ "$origin_path" = "$DATA/$origin/report.md" ] || fail "origin $origin dispatch carrier names an unauthorized source path" + regular_nonsymlink_file "$origin_path" || fail "origin $origin dispatch source path does not exist safely" + digest_is_nonzero_sha256 "$origin_sha" || fail "origin $origin dispatch source digest must be nonzero sha256" + [ "$(sha256_file "$origin_path")" = "$origin_sha" ] || fail "origin $origin dispatch source digest no longer matches" + printf '%s\n' "$recorded_dispatch" +} + +completion_dispatch() { # <origin-id> <meta-path> <has-meta> + local origin=$1 meta=$2 has_meta=$3 dispatch + if [ "$has_meta" = 1 ]; then + dispatch=$(meta_exact_value "$meta" endpoint_task_id) \ + || fail "origin $origin has no unique endpoint dispatch binding; preserve origin metadata and holds because no safe automatic migration is shipped" + [ "$dispatch" = "$origin" ] || fail "origin $origin metadata links a different endpoint dispatch: $dispatch" + write_origin_receipt "$origin" "$dispatch" + verify_origin_receipt "$origin" "$dispatch" >/dev/null + else + dispatch=$(verify_origin_receipt "$origin") || return 1 + fi + printf '%s\n' "$dispatch" +} + +fail_missing_cleanup_receipt() { # <origin-id> <hold-id> + local origin=$1 id=$2 + if verify_hold_resolved "$id"; then + fail "historical or hand-written resolved row $id has no trusted cleanup receipt; preserve the origin and row: no safe automatic migration is shipped" + fi + fail "captain hold $id has no trusted cleanup receipt; re-run complete $origin with its full recorded decision inventory only while the hold remains open and the exact dispatch binding survives; otherwise preserve origin metadata and holds because no safe automatic migration is shipped" +} + +verify_cleanup_receipt_base() { # <origin-id> <dispatch-id> <hold-id> + local origin=$1 dispatch=$2 id=$3 receipt schema phase recorded_origin recorded_dispatch recorded_hold object_path object_sha bearings_sha + receipt=$(cleanup_receipt_path "$id") + regular_nonsymlink_file "$receipt" \ + || fail_missing_cleanup_receipt "$origin" "$id" + schema=$(receipt_value "$receipt" schema) || fail "captain hold $id has a malformed cleanup receipt schema" + phase=$(receipt_value "$receipt" phase) || fail "captain hold $id has a malformed cleanup receipt phase" + recorded_origin=$(receipt_value "$receipt" origin_id) || fail "captain hold $id has no cleanup task identity" + recorded_dispatch=$(receipt_value "$receipt" dispatch_id) || fail "captain hold $id has no cleanup dispatch identity" + recorded_hold=$(receipt_value "$receipt" hold_id) || fail "captain hold $id has no cleanup hold identity" + object_path=$(receipt_value "$receipt" object_path) || fail "captain hold $id has no cleanup object path" + object_sha=$(receipt_value "$receipt" object_sha256) || fail "captain hold $id has no cleanup object digest" + bearings_sha=$(receipt_value "$receipt" bearings_sha256) || fail "captain hold $id has no Bearings evidence digest" + [ "$schema" = fm-decision-hold-receipt.v1 ] || fail "captain hold $id has an unsupported cleanup receipt" + [ "$phase" = hold ] || fail "captain hold $id cleanup receipt has the wrong phase" + [ "$recorded_origin" = "$origin" ] || fail "captain hold $id cleanup receipt links a different task" + [ "$recorded_dispatch" = "$dispatch" ] || fail "captain hold $id cleanup receipt links a different dispatch" + [ "$recorded_hold" = "$id" ] || fail "captain hold $id cleanup receipt links a different hold" + verify_origin_receipt "$origin" "$dispatch" >/dev/null + [ "$object_path" = "$DATA/backlog.md" ] || fail "captain hold $id cleanup receipt names an unauthorized object path" + regular_nonsymlink_file "$object_path" || fail "captain hold $id cleanup object path does not exist safely" + digest_is_nonzero_sha256 "$object_sha" || fail "captain hold $id cleanup object digest must be nonzero sha256" + digest_is_nonzero_sha256 "$bearings_sha" || fail "captain hold $id Bearings evidence digest must be nonzero sha256" +} + +bearings_snapshot() { + local bearings=${FM_DECISION_HOLD_BEARINGS:-$SCRIPT_DIR/fm-bearings-snapshot.sh} + [ -x "$bearings" ] || fail "Bearings snapshot executable is unavailable: $bearings" + command -v jq >/dev/null 2>&1 || fail "jq is required for Bearings receipt verification" + FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_DATA_OVERRIDE="$DATA" \ + "$bearings" --json --all-decisions +} + +bearings_has_hold() { # <json> <hold-id> + printf '%s\n' "$1" | jq -e --arg id "$2" \ + '.schema == "fm-bearings.v1" and (.decisions_open | any(.id == $id and .verb == "captain-hold"))' \ + >/dev/null 2>&1 +} + +verify_cleanup_receipt() { # <origin-id> <dispatch-id> <hold-id> + local origin=$1 dispatch=$2 id=$3 receipt show recorded_sha fresh + verify_cleanup_receipt_base "$origin" "$dispatch" "$id" + receipt=$(cleanup_receipt_path "$id") + show=$(task_show_durable "$id") || fail "captain hold $id cleanup object is absent, duplicated, or indeterminate" + recorded_sha=$(receipt_value "$receipt" object_sha256) || fail "captain hold $id cleanup object digest is unavailable" + [ "$(sha256_text "$show")" = "$recorded_sha" ] \ + || fail "captain hold $id cleanup object digest no longer matches" + fresh=$(bearings_snapshot) || fail "could not obtain fresh Bearings evidence for $id" + bearings_has_hold "$fresh" "$id" || fail "captain hold $id does not reappear in Bearings" +} + +write_cleanup_receipt() { # <origin-id> <dispatch-id> <hold-id> <bearings-json> + local origin=$1 dispatch=$2 id=$3 bearings=$4 show object_sha bearings_sha receipt + show=$(task_show_durable "$id") || fail "captain hold $id is not one durable backlog object" + object_sha=$(sha256_text "$show") + bearings_sha=$(sha256_text "$bearings") + digest_is_nonzero_sha256 "$object_sha" || fail "captain hold $id produced an invalid object digest" + digest_is_nonzero_sha256 "$bearings_sha" || fail "captain hold $id produced invalid Bearings evidence" + receipt=$(cleanup_receipt_path "$id") + { + printf 'schema=fm-decision-hold-receipt.v1\n' + printf 'phase=hold\n' + printf 'origin_id=%s\n' "$origin" + printf 'dispatch_id=%s\n' "$dispatch" + printf 'hold_id=%s\n' "$id" + printf 'object_path=%s\n' "$DATA/backlog.md" + printf 'object_sha256=%s\n' "$object_sha" + printf 'bearings_sha256=%s\n' "$bearings_sha" + } | write_atomic_file "$receipt" +} + +verify_resolution_receipt() { # <origin-id> <hold-id> + local origin=$1 id=$2 show state kind body receipt cleanup dispatch schema phase recorded_origin recorded_dispatch recorded_hold decision_path decision_sha routed_ids object_path object_sha source routed task + show=$(task_show_durable "$id") \ + || fail "captain decision $id is absent, duplicated, or indeterminate in the live backlog and archive" + state=$(show_field "$show" state) + kind=$(show_field "$show" kind) + body=$(show_field "$show" body) + [ "$state" = "done" ] || fail "captain hold $id is unresolved" + [ "$kind" = captain ] || fail "backlog item $id is not kind captain" + resolution_body_fields "$id" "$body" + + cleanup=$(cleanup_receipt_path "$id") + regular_nonsymlink_file "$cleanup" \ + || fail_missing_cleanup_receipt "$origin" "$id" + dispatch=$(receipt_value "$cleanup" dispatch_id) || fail "captain hold $id cleanup receipt has no dispatch link" + verify_cleanup_receipt_base "$origin" "$dispatch" "$id" + + receipt=$(resolution_receipt_path "$id") + regular_nonsymlink_file "$receipt" \ + || fail "historical or hand-written resolved row $id has no trusted resolution receipt; repeat the exact resolve only when the script-owned cleanup receipt and canonical decision object survive; otherwise preserve the origin and row because no safe automatic migration is shipped" + schema=$(receipt_value "$receipt" schema) || fail "captain hold $id has a malformed resolution receipt schema" + phase=$(receipt_value "$receipt" phase) || fail "captain hold $id has a malformed resolution receipt phase" + recorded_origin=$(receipt_value "$receipt" origin_id) || fail "captain hold $id resolution receipt has no task link" + recorded_dispatch=$(receipt_value "$receipt" dispatch_id) || fail "captain hold $id resolution receipt has no dispatch link" + recorded_hold=$(receipt_value "$receipt" hold_id) || fail "captain hold $id resolution receipt has no hold link" + decision_path=$(receipt_value "$receipt" decision_path) || fail "captain hold $id resolution receipt has no decision object path" + decision_sha=$(receipt_value "$receipt" decision_sha256) || fail "captain hold $id resolution receipt has no decision digest" + routed_ids=$(receipt_value "$receipt" routed_ids) || fail "captain hold $id resolution receipt has no routed task links" + object_path=$(receipt_value "$receipt" object_path) || fail "captain hold $id resolution receipt has no row object path" + object_sha=$(receipt_value "$receipt" object_sha256) || fail "captain hold $id resolution receipt has no row object digest" + [ "$schema" = fm-decision-hold-receipt.v1 ] || fail "captain hold $id has an unsupported resolution receipt" + [ "$phase" = resolved ] || fail "captain hold $id resolution receipt has the wrong phase" + [ "$recorded_origin" = "$origin" ] || fail "captain hold $id resolution receipt links a different task" + [ "$recorded_dispatch" = "$dispatch" ] || fail "captain hold $id resolution receipt links a different dispatch" + [ "$recorded_hold" = "$id" ] || fail "captain hold $id resolution receipt links a different hold" + [ "$decision_path" = "$(decision_object_path "$id")" ] \ + || fail "captain hold $id resolution receipt names an unauthorized decision object" + regular_nonsymlink_file "$decision_path" || fail "captain hold $id decision object path does not exist safely" + digest_is_nonzero_sha256 "$decision_sha" || fail "captain hold $id resolution receipt decision digest must be nonzero sha256" + [ "$(sha256_file "$decision_path")" = "$decision_sha" ] || fail "captain hold $id decision object digest does not match" + [ "$decision_sha" = "$RESOLUTION_DIGEST" ] || fail "captain hold $id row and decision object digests differ" + [ "$routed_ids" = "$RESOLUTION_ROUTES" ] || fail "captain hold $id row and receipt routed tasks differ" + source=$(task_row_source "$id") || fail "captain hold $id row object is no longer unique" + [ "$object_path" = "$source" ] || fail "captain hold $id resolution receipt names the wrong row object path" + regular_nonsymlink_file "$object_path" || fail "captain hold $id row object path does not exist safely" + digest_is_nonzero_sha256 "$object_sha" || fail "captain hold $id row object digest must be nonzero sha256" + [ "$(sha256_text "$show")" = "$object_sha" ] || fail "captain hold $id row object digest does not match" + routed=$(printf '%s\n' "$routed_ids" | tr ',' ' ') + for task in $routed; do + validate_slug routed-task "$task" + task_show_durable "$task" >/dev/null \ + || fail "captain hold $id routed task $task is absent, duplicated, or indeterminate" + done +} + +write_resolution_receipt() { # <origin-id> <dispatch-id> <hold-id> <decision-path> <decision-sha> <routed-csv> + local origin=$1 dispatch=$2 id=$3 decision_path=$4 decision_sha=$5 routed_csv=$6 show source object_sha receipt + show=$(task_show_durable "$id") || fail "captain hold $id resolved row is absent, duplicated, or indeterminate" + source=$(task_row_source "$id") || fail "captain hold $id resolved row has no unique object path" + object_sha=$(sha256_text "$show") + digest_is_nonzero_sha256 "$decision_sha" || fail "captain hold $id decision digest must be nonzero sha256" + digest_is_nonzero_sha256 "$object_sha" || fail "captain hold $id row digest must be nonzero sha256" + receipt=$(resolution_receipt_path "$id") + { + printf 'schema=fm-decision-hold-receipt.v1\n' + printf 'phase=resolved\n' + printf 'origin_id=%s\n' "$origin" + printf 'dispatch_id=%s\n' "$dispatch" + printf 'hold_id=%s\n' "$id" + printf 'decision_path=%s\n' "$decision_path" + printf 'decision_sha256=%s\n' "$decision_sha" + printf 'routed_ids=%s\n' "$routed_csv" + printf 'object_path=%s\n' "$source" + printf 'object_sha256=%s\n' "$object_sha" + } | write_atomic_file "$receipt" +} + verify_resolution_identity() { local id=$1 hold_body=$2 decision_digest=$3 routed_csv=$4 resolution_prefix resolution_fields recorded_digest recorded_routes resolution_prefix='"Resolution recorded by fm-decision-hold.\nDecision digest: ' @@ -249,7 +624,7 @@ command_hold() { require_tasks_axi origin_exists_here "$origin" || fail "origin $origin is not owned by the active home $FM_HOME" id=$(hold_id "$origin" "$key") - if show=$(task_show "$id"); then + if show=$(task_show_durable "$id"); then state=$(show_field "$show" state) kind=$(show_field "$show" kind) existing_title=$(show_field "$show" title) @@ -275,7 +650,7 @@ command_hold() { } command_complete() { - local origin=${1:-} meta previous='' supplied='' keys='' key status_file open raw_open key_seen=0 has_meta=0 + local origin=${1:-} meta previous='' supplied='' keys='' key status_file open raw_open key_seen=0 has_meta=0 dispatch='' bearings='' id [ "$#" -ge 2 ] || { usage >&2; exit 2; } validate_slug origin-id "$origin" shift @@ -297,15 +672,6 @@ command_complete() { previous=$(meta_value "$meta" decision_keys) fi keys=$(sorted_key_union "$previous" "$supplied") - if [ -n "$keys" ]; then - while IFS= read -r key; do - [ -n "$key" ] || continue - verify_hold_durable "$(hold_id "$origin" "$key")" - done <<EOF -$(printf '%s\n' "$keys" | tr ',' '\n') -EOF - fi - status_file="$STATE/$origin.status" raw_open=$(status_open_decisions "$status_file") open=$(origin_open_decisions "$origin") @@ -317,6 +683,29 @@ EOF $open EOF + dispatch=$(completion_dispatch "$origin" "$meta" "$has_meta") || return 1 + if [ -n "$keys" ]; then + while IFS= read -r key; do + [ -n "$key" ] || continue + verify_hold_durable "$origin" "$(hold_id "$origin" "$key")" + done <<EOF +$(printf '%s\n' "$keys" | tr ',' '\n') +EOF + + bearings=$(bearings_snapshot) || fail "could not obtain Bearings evidence for $origin" + while IFS= read -r key; do + [ -n "$key" ] || continue + id=$(hold_id "$origin" "$key") + if ! verify_hold_resolved "$id"; then + bearings_has_hold "$bearings" "$id" \ + || fail "captain hold $id does not reappear in Bearings" + write_cleanup_receipt "$origin" "$dispatch" "$id" "$bearings" + fi + done <<EOF +$(printf '%s\n' "$keys" | tr ',' '\n') +EOF + fi + if [ "$has_meta" = 1 ]; then if [ "$(meta_value "$meta" decisions_reviewed)" != 1 ] || [ "$previous" != "$keys" ]; then printf 'decisions_reviewed=1\ndecision_keys=%s\n' "$keys" >> "$meta" @@ -338,7 +727,7 @@ EOF } command_verify() { - local origin=${1:-} meta reviewed keys key open + local origin=${1:-} meta reviewed keys key open dispatch id [ "$#" -eq 1 ] || { usage >&2; exit 2; } validate_slug origin-id "$origin" meta="$STATE/$origin.meta" @@ -347,28 +736,52 @@ command_verify() { reviewed=$(meta_value "$meta" decisions_reviewed) [ "$reviewed" = 1 ] || fail "origin $origin has no completed unresolved-decision inventory" keys=$(meta_value "$meta" decision_keys) + open=$(origin_open_decisions "$origin") + if [ -n "$keys" ] || [ -n "$open" ]; then + dispatch=$(meta_exact_value "$meta" endpoint_task_id) \ + || fail "origin $origin has no unique endpoint dispatch binding; preserve origin metadata and holds because no safe automatic migration is shipped" + [ "$dispatch" = "$origin" ] || fail "origin $origin metadata links a different endpoint dispatch: $dispatch" + fi if [ -n "$keys" ]; then while IFS= read -r key; do [ -n "$key" ] || continue - verify_hold_durable "$(hold_id "$origin" "$key")" + id=$(hold_id "$origin" "$key") + if verify_hold_resolved "$id"; then + verify_resolution_receipt "$origin" "$id" + else + verify_hold_active "$id" + verify_cleanup_receipt "$origin" "$dispatch" "$id" + fi done <<EOF $(printf '%s\n' "$keys" | tr ',' '\n') EOF fi - open=$(origin_open_decisions "$origin") while IFS=$'\t' read -r key _verb _summary; do [ -n "$key" ] || continue list_has_key "$keys" "$key" \ || fail "open structured decision $origin/$key is outside the reviewed inventory" - verify_hold_durable "$(hold_id "$origin" "$key")" + id=$(hold_id "$origin" "$key") + verify_hold_active "$id" + verify_cleanup_receipt "$origin" "$dispatch" "$id" done <<EOF $open EOF printf 'verified: %s unresolved-decision inventory\n' "$origin" } +command_verify_resolution() { + local origin=${1:-} key=${2:-} id + [ "$#" -eq 2 ] || { usage >&2; exit 2; } + validate_slug origin-id "$origin" + validate_slug decision-key "$key" + require_tasks_axi + id=$(hold_id "$origin" "$key") + verify_resolution_receipt "$origin" "$id" + printf 'verified: %s trusted resolution receipt\n' "$id" +} + command_resolve() { - local origin=${1:-} key=${2:-} decision_file='' id='' decision='' decision_digest='' body='' routed='' routed_csv='' dep show blocked state hold_show hold_body resolution_recorded=0 + local origin=${1:-} key=${2:-} decision_file='' id='' decision='' decision_digest='' body='' routed='' routed_csv='' dep show blocked state hold_show hold_body resolution_recorded=0 cleanup dispatch decision_object [ "$#" -ge 2 ] || { usage >&2; exit 2; } shift 2 while [ "$#" -gt 0 ]; do @@ -394,21 +807,53 @@ command_resolve() { require_tasks_axi id=$(hold_id "$origin" "$key") if verify_hold_resolved "$id"; then - hold_show=$(task_show "$id") + hold_show=$(task_show_durable "$id") \ + || fail "captain hold $id disappeared after its durable resolution was found" hold_body=$(show_field "$hold_show" body) verify_resolution_identity "$id" "$hold_body" "$decision_digest" "$routed_csv" + cleanup=$(cleanup_receipt_path "$id") + regular_nonsymlink_file "$cleanup" || fail_missing_cleanup_receipt "$origin" "$id" + dispatch=$(receipt_value "$cleanup" dispatch_id) \ + || fail "historical resolved row $id has no trustworthy dispatch link; preserve the origin and row because no safe automatic migration is shipped" + verify_cleanup_receipt_base "$origin" "$dispatch" "$id" + decision_object=$(decision_object_path "$id") + regular_nonsymlink_file "$decision_object" \ + || fail "historical resolved row $id has no canonical decision object; preserve the origin and row because no safe automatic migration is shipped" + [ "$(sha256_file "$decision_object")" = "$decision_digest" ] \ + || fail "captain hold $id canonical decision object does not match the requested decision" + if [ ! -f "$(resolution_receipt_path "$id")" ]; then + write_resolution_receipt "$origin" "$dispatch" "$id" "$decision_object" "$decision_digest" "$routed_csv" + fi + verify_resolution_receipt "$origin" "$id" printf 'resolved: %s\n' "$id" return 0 fi verify_hold_active "$id" hold_show=$(task_show "$id") hold_body=$(show_field "$hold_show" body) + cleanup=$(cleanup_receipt_path "$id") + regular_nonsymlink_file "$cleanup" || fail_missing_cleanup_receipt "$origin" "$id" + dispatch=$(receipt_value "$cleanup" dispatch_id) \ + || fail "captain hold $id has no trustworthy cleanup dispatch link; preserve origin metadata and holds because no safe automatic migration is shipped" + verify_cleanup_receipt_base "$origin" "$dispatch" "$id" case "$hold_body" in *"Resolution recorded by fm-decision-hold."*) verify_resolution_identity "$id" "$hold_body" "$decision_digest" "$routed_csv" resolution_recorded=1 ;; esac + decision_object=$(decision_object_path "$id") + if [ "$resolution_recorded" = 1 ]; then + regular_nonsymlink_file "$decision_object" \ + || fail "captain hold $id partial resolution lost its decision object" + [ "$(sha256_file "$decision_object")" = "$decision_digest" ] \ + || fail "captain hold $id partial resolution decision object drifted" + else + verify_cleanup_receipt "$origin" "$dispatch" "$id" + printf '%s' "$decision" | write_atomic_file "$decision_object" + [ "$(sha256_file "$decision_object")" = "$decision_digest" ] \ + || fail "captain hold $id decision object digest did not stabilize" + fi for dep in $routed; do show=$(task_show "$dep") || fail "routed task $dep does not exist in the active home" @@ -449,7 +894,9 @@ command_resolve() { esac done tasks_axi "done" "$id" >/dev/null || fail "could not close resolved captain hold $id" - verify_hold_resolved "$id" || fail "captain hold $id did not retain its durable resolution record" + verify_hold_resolved "$id" || fail "captain hold $id did not retain its durable resolution row" + write_resolution_receipt "$origin" "$dispatch" "$id" "$decision_object" "$decision_digest" "$routed_csv" + verify_resolution_receipt "$origin" "$id" printf 'resolved: %s -> %s\n' "$id" "$routed" } @@ -458,6 +905,7 @@ case "${1:-}" in hold) shift; command_hold "$@" ;; complete) shift; command_complete "$@" ;; verify) shift; command_verify "$@" ;; + verify-resolution) shift; command_verify_resolution "$@" ;; resolve) shift; command_resolve "$@" ;; -h|--help) usage ;; *) usage >&2; exit 2 ;; diff --git a/bin/fm-model-capacity-hold-lib.sh b/bin/fm-model-capacity-hold-lib.sh new file mode 100644 index 00000000000..7555b699163 --- /dev/null +++ b/bin/fm-model-capacity-hold-lib.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Read-only enforcement for the one home-wide registered model-capacity hold. +# Firstmate-controlled paths that can launch or prompt a model call this before +# mutation. Interrupt and cleanup-only paths remain available. + +fm_model_capacity_hold_path() { + printf '%s/.model-capacity-hold\n' "${STATE:-${FM_STATE_OVERRIDE:-$FM_HOME/state}}" +} + +fm_model_capacity_hold_value() { # <path> <key> + local path=$1 key=$2 count + [ -f "$path" ] && [ ! -L "$path" ] || return 1 + count=$(grep -c "^$key=" "$path" 2>/dev/null || true) + [ "$count" -eq 1 ] || return 1 + sed -n "s/^$key=//p" "$path" +} + +fm_model_capacity_hold_refuse() { # <operation> + local operation=$1 marker schema id reason dispatch + marker=$(fm_model_capacity_hold_path) + [ -e "$marker" ] || return 0 + schema=$(fm_model_capacity_hold_value "$marker" schema) || schema=invalid + id=$(fm_model_capacity_hold_value "$marker" hold_id) || id=unknown + reason=$(fm_model_capacity_hold_value "$marker" reason) || reason='malformed hold record' + dispatch=$(fm_model_capacity_hold_value "$marker" dispatch_ref) || dispatch=unknown + if [ "$schema" != fm-model-capacity-hold.v1 ]; then + printf 'error: model-capacity hold record is malformed; refusing %s until %s is reconciled\n' "$operation" "$marker" >&2 + return 1 + fi + printf 'error: model-capacity hold %s is active; refusing %s (reason=%s; dispatch_ref=%s)\n' \ + "$id" "$operation" "$reason" "$dispatch" >&2 + return 1 +} diff --git a/bin/fm-model-capacity-hold.sh b/bin/fm-model-capacity-hold.sh new file mode 100755 index 00000000000..3988670c0ef --- /dev/null +++ b/bin/fm-model-capacity-hold.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# Register, inspect, or release the home-wide hold that blocks Firstmate-owned +# model-consuming execution paths while leaving interrupts and cleanup available. +# +# Usage: +# fm-model-capacity-hold.sh register <hold-id> --reason <one-line> --dispatch-ref <one-line> +# fm-model-capacity-hold.sh status +# fm-model-capacity-hold.sh release <hold-id> --authority-file <path> +# +# Registration and release receipts live under data/model-capacity-holds/. A +# release requires a regular authority object and records its SHA-256 before the +# active marker is retired. The current primary turn and native same-session +# continuations, direct TUI input, provider CLIs, lower-level backend sends, +# already-running agents, and direct no-mistakes commands are outside this +# command's control; docs/architecture.md owns the complete boundary inventory. +set -eu + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +MARKER="$STATE/.model-capacity-hold" +RECEIPTS="$DATA/model-capacity-holds" + +# shellcheck source=bin/fm-custody-lib.sh +. "$SCRIPT_DIR/fm-custody-lib.sh" +# shellcheck source=bin/fm-model-capacity-hold-lib.sh +. "$SCRIPT_DIR/fm-model-capacity-hold-lib.sh" + +fail() { + printf 'fm-model-capacity-hold: %s\n' "$*" >&2 + exit 1 +} + +validate_id() { + case "$1" in ''|*[!A-Za-z0-9._-]*) fail "hold id must be a privacy-safe slug" ;; esac +} + +validate_line() { + [ -n "$2" ] || fail "$1 is required" + case "$2" in *$'\n'*|*$'\r'*) fail "$1 must be one line" ;; esac +} + +write_atomic() { # <path> <exclusive:0|1>, content on stdin + local path=$1 exclusive=$2 dir tmp + dir=${path%/*} + mkdir -p "$dir" + tmp=$(mktemp "$dir/.model-hold.tmp.XXXXXX") || fail "could not allocate receipt temporary" + chmod 600 "$tmp" + cat > "$tmp" || { rm -f "$tmp"; fail "could not write temporary"; } + if [ "$exclusive" = 1 ] && [ -e "$path" ]; then + rm -f "$tmp" + fail "refusing to overwrite existing receipt: $path" + fi + if [ "$exclusive" = 1 ]; then + (set -C; : > "$path") 2>/dev/null || { rm -f "$tmp"; fail "receipt appeared concurrently: $path"; } + fi + mv "$tmp" "$path" || { rm -f "$tmp"; fail "could not publish $path"; } +} + +command_register() { + local id reason dispatch timestamp receipt existing_id + id=${1:-} + reason= + dispatch= + [ "$#" -ge 1 ] || fail "register requires a hold id" + shift + while [ "$#" -gt 0 ]; do + case "$1" in + --reason) shift; reason=${1:-} ;; + --dispatch-ref) shift; dispatch=${1:-} ;; + *) fail "unknown register argument: $1" ;; + esac + shift + done + validate_id "$id" + validate_line reason "$reason" + validate_line dispatch-ref "$dispatch" + mkdir -p "$STATE" "$RECEIPTS" + if [ -e "$MARKER" ]; then + existing_id=$(fm_model_capacity_hold_value "$MARKER" hold_id) || fail "active hold marker is malformed" + [ "$existing_id" = "$id" ] || fail "another model-capacity hold is active: $existing_id" + printf 'registered: %s already active\n' "$id" + return 0 + fi + timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ') + receipt="$RECEIPTS/$id.registered" + { + printf 'schema=fm-model-capacity-hold.v1\n' + printf 'hold_id=%s\n' "$id" + printf 'reason=%s\n' "$reason" + printf 'dispatch_ref=%s\n' "$dispatch" + printf 'registered_at=%s\n' "$timestamp" + } | write_atomic "$receipt" 1 + cp "$receipt" "$MARKER.tmp.${BASHPID:-$$}" || fail "could not stage active marker" + mv "$MARKER.tmp.${BASHPID:-$$}" "$MARKER" || fail "could not publish active marker" + printf 'registered: %s\n' "$id" +} + +command_status() { + if [ ! -e "$MARKER" ]; then + printf 'inactive\n' + return 0 + fi + cat "$MARKER" +} + +command_release() { + local id authority active_id digest timestamp registered receipt + id=${1:-} + authority= + [ "$#" -ge 1 ] || fail "release requires a hold id" + shift + while [ "$#" -gt 0 ]; do + case "$1" in + --authority-file) shift; authority=${1:-} ;; + *) fail "unknown release argument: $1" ;; + esac + shift + done + validate_id "$id" + [ -f "$authority" ] && [ ! -L "$authority" ] || fail "authority file must be a regular non-symlink file" + active_id=$(fm_model_capacity_hold_value "$MARKER" hold_id) || fail "no valid active model-capacity hold" + [ "$active_id" = "$id" ] || fail "active hold is $active_id, not $id" + registered="$RECEIPTS/$id.registered" + [ -f "$registered" ] && [ ! -L "$registered" ] || fail "registration receipt is missing" + digest=$(fm_custody_sha256 "$authority") || fail "could not digest release authority" + timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ') + receipt="$RECEIPTS/$id.released" + { + printf 'schema=fm-model-capacity-hold-release.v1\n' + printf 'hold_id=%s\n' "$id" + printf 'registration_sha256=%s\n' "$(fm_custody_sha256 "$registered")" + printf 'authority_path=%s\n' "$authority" + printf 'authority_sha256=%s\n' "$digest" + printf 'released_at=%s\n' "$timestamp" + } | write_atomic "$receipt" 1 + mv "$MARKER" "$RECEIPTS/$id.active-marker-retired" \ + || fail "release receipt exists but active marker could not be retired" + printf 'released: %s\n' "$id" +} + +case "${1:-}" in + register) shift; command_register "$@" ;; + status) shift; [ "$#" -eq 0 ] || fail "status takes no arguments"; command_status ;; + release) shift; command_release "$@" ;; + -h|--help) sed -n '2,/^set -eu/p' "$0" | sed '$d; s/^# \{0,1\}//' ;; + *) fail "usage: fm-model-capacity-hold.sh register|status|release ..." ;; +esac diff --git a/bin/fm-process-progress.sh b/bin/fm-process-progress.sh new file mode 100755 index 00000000000..cda68c49bcc --- /dev/null +++ b/bin/fm-process-progress.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# Sample cumulative CPU for the descendant closure of one recorded backend pane. +# This is progress evidence, not a current-state oracle: callers compare two +# samples with the same root-process identity and use only a positive delta. +# +# Usage: fm-process-progress.sh <backend> <target> +# Output: <root-identity-sha256><TAB><cumulative-centiseconds> +# +# Verified root PID bindings currently exist for tmux and herdr. zellij, Orca, +# and cmux return nonzero rather than guessing. FM_PROCESS_PROGRESS_ROOT_PID is +# a test seam for a pre-bound root. +set -eu + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=bin/fm-backend.sh +. "$SCRIPT_DIR/fm-backend.sh" + +[ "$#" -eq 2 ] || { printf 'usage: fm-process-progress.sh <backend> <target>\n' >&2; exit 2; } +backend=$1 +target=$2 +root=${FM_PROCESS_PROGRESS_ROOT_PID:-} +[ -n "$root" ] || root=$(fm_backend_process_root_pid "$backend" "$target") +case "$root" in ''|*[!0-9]*) exit 1 ;; esac + +identity=$(LC_ALL=C ps -p "$root" -o lstart= -o command= 2>/dev/null) || exit 1 +[ -n "$identity" ] || exit 1 +if command -v shasum >/dev/null 2>&1; then + identity_sha=$(printf '%s' "$root:$identity" | shasum -a 256 | awk '{print $1}') +else + identity_sha=$(printf '%s' "$root:$identity" | sha256sum | awk '{print $1}') +fi + +rows=$(LC_ALL=C ps -axo pid=,ppid=,time=,command= 2>/dev/null) || exit 1 +cpu=$(printf '%s\n' "$rows" | awk -v root="$root" ' + function centis(raw, d, rest, n, a, h, m, s) { + d = 0 + rest = raw + if (index(rest, "-") > 0) { + split(rest, dayparts, "-") + d = dayparts[1] + 0 + rest = dayparts[2] + } + n = split(rest, a, ":") + h = 0; m = 0; s = 0 + if (n == 3) { h = a[1] + 0; m = a[2] + 0; s = a[3] + 0 } + else if (n == 2) { m = a[1] + 0; s = a[2] + 0 } + else { s = a[1] + 0 } + return int((((d * 24 + h) * 60 + m) * 60 + s) * 100 + 0.5) + } + { + pid[NR] = $1 + parent[$1] = $2 + value[$1] = centis($3) + seen[$1] = 1 + } + END { + member[root] = 1 + changed = 1 + while (changed) { + changed = 0 + for (p in seen) { + if (!member[p] && member[parent[p]]) { + member[p] = 1 + changed = 1 + } + } + } + total = 0 + for (p in member) total += value[p] + print total + 0 + } +') +case "$cpu" in ''|*[!0-9]*) exit 1 ;; esac +printf '%s\t%s\n' "$identity_sha" "$cpu" diff --git a/bin/fm-record-reconcile.sh b/bin/fm-record-reconcile.sh new file mode 100755 index 00000000000..d6cbfa0446f --- /dev/null +++ b/bin/fm-record-reconcile.sh @@ -0,0 +1,157 @@ +#!/usr/bin/env bash +# Reconcile terminal producer reports with structured backlog rows, then write a +# durable inventory receipt for every metadata/backlog mismatch without deleting +# any row, metadata, status, report, worktree, or commit that proves what happened. +# +# Usage: fm-record-reconcile.sh +# +# A `done:` producer or a producer declaring the complete-only +# `awaiting-captain:` state while still In flight moves to Done with --no-prune +# and an explicit retained-lifecycle note. Its endpoint metadata and worktree +# remain; teardown separately refuses unlanded work. Scout rows move only after +# the report exists and the decision-hold completion gate verifies. Missing +# metadata and orphan metadata are accounted in the receipt and preserved. +set -eu + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +BACKLOG="$DATA/backlog.md" +RECEIPT_DIR="$DATA/record-reconciliation" + +# shellcheck source=bin/fm-classify-lib.sh +. "$SCRIPT_DIR/fm-classify-lib.sh" +# shellcheck source=bin/fm-tasks-axi-lib.sh +. "$SCRIPT_DIR/fm-tasks-axi-lib.sh" +# shellcheck source=bin/fm-custody-lib.sh +. "$SCRIPT_DIR/fm-custody-lib.sh" + +[ -f "$BACKLOG" ] || exit 0 +events= + +append_event() { # <class> <id> <detail> + events="${events}${1}"$'\t'"${2}"$'\t'"${3}"$'\n' +} + +in_flight_ids() { + awk ' + /^## In flight/ { inside=1; next } + /^## / { inside=0 } + inside && /^- \[ \] / { sub(/^- \[ \] /, ""); sub(/ -.*/, ""); print } + ' "$BACKLOG" +} + +task_state() { # <id> + (cd "$FM_HOME" && tasks-axi show "$1" --full 2>/dev/null) \ + | sed -n 's/^ state: //p' | head -1 +} + +TERMINAL_WORKTREE= +TERMINAL_HEAD= +terminal_tree_is_clean() { # <meta> + local meta=$1 count status + TERMINAL_WORKTREE= + TERMINAL_HEAD= + count=$(grep -c '^worktree=' "$meta" 2>/dev/null || true) + [ "$count" -eq 1 ] || return 1 + TERMINAL_WORKTREE=$(sed -n 's/^worktree=//p' "$meta") + [ -d "$TERMINAL_WORKTREE" ] \ + && git -C "$TERMINAL_WORKTREE" rev-parse --is-inside-work-tree >/dev/null 2>&1 \ + || return 1 + TERMINAL_HEAD=$(git -C "$TERMINAL_WORKTREE" rev-parse HEAD 2>/dev/null) || return 1 + status=$(git -C "$TERMINAL_WORKTREE" status --porcelain --untracked-files=normal 2>/dev/null) \ + || return 1 + [ -z "$status" ] +} + +if fm_tasks_axi_compatible; then + while IFS= read -r id; do + [ -n "$id" ] || continue + meta="$STATE/$id.meta" + [ -f "$meta" ] || continue + kind=$(grep '^kind=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$kind" ] || kind=ship + [ "$kind" != secondmate ] || continue + last=$(last_status_line "$STATE/$id.status") + status_is_done "$last" || status_is_awaiting_captain "$last" || continue + if ! terminal_tree_is_clean "$meta"; then + append_event terminal-unreconciled "$id" 'terminal status retained in flight because worktree identity or clean-tree evidence is unavailable' + continue + fi + if [ "$kind" = scout ]; then + if [ ! -f "$DATA/$id/report.md" ] || [ -L "$DATA/$id/report.md" ]; then + append_event terminal-unreconciled "$id" 'scout report missing or unsafe; row preserved' + continue + fi + if ! FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_DATA_OVERRIDE="$DATA" \ + "$SCRIPT_DIR/fm-decision-hold.sh" verify "$id" >/dev/null 2>&1; then + append_event terminal-unreconciled "$id" 'decision inventory not cleanup-authoritative; row preserved' + continue + fi + fi + (cd "$FM_HOME" && tasks-axi 'done' "$id" --no-prune \ + --note "producer terminal status reconciled at $TERMINAL_HEAD; endpoint metadata and worktree retained pending landing, independent gate, or captain answer") >/dev/null \ + || { append_event terminal-unreconciled "$id" 'tasks-axi transition failed; row preserved'; continue; } + append_event terminal-retained "$id" "moved from In flight to Done at head=$TERMINAL_HEAD with tree=clean; metadata and worktree retained" + done < <(in_flight_ids) +fi + +# Inventory the post-transition state. These are explanations, not cleanup +# authority: every mismatched object remains at its original path. +while IFS= read -r id; do + [ -n "$id" ] || continue + [ -f "$STATE/$id.meta" ] || append_event missing-meta "$id" 'live in-flight row preserved without metadata' +done < <(in_flight_ids) + +for meta in "$STATE"/*.meta; do + [ -e "$meta" ] || continue + id=${meta##*/} + id=${id%.meta} + state=$(task_state "$id" || true) + [ -n "$state" ] || append_event orphan-meta "$id" 'metadata preserved without one live backlog row' +done + +for status_file in "$STATE"/*.status; do + [ -f "$status_file" ] && [ ! -L "$status_file" ] || continue + id=${status_file##*/} + id=${id%.status} + [ -f "$STATE/$id.meta" ] \ + || append_event orphan-status "$id" 'append-only status evidence preserved without endpoint metadata' +done + +in_flight_count=$(in_flight_ids | awk 'NF { count++ } END { print count + 0 }') +metadata_count=0 +for meta in "$STATE"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + metadata_count=$((metadata_count + 1)) +done +metadata_delta=$((metadata_count - in_flight_count)) + +mkdir -p "$RECEIPT_DIR" +timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ') +stamp=$(date -u '+%Y%m%dT%H%M%SZ') +backlog_sha=$(fm_custody_sha256 "$BACKLOG") +attempt=0 +while [ "$attempt" -lt 100 ]; do + receipt="$RECEIPT_DIR/reconcile.$stamp.${BASHPID:-$$}.$attempt.receipt" + if (set -C; umask 077; { + printf 'schema=fm-record-reconciliation.v1\n' + printf 'timestamp_utc=%s\n' "$timestamp" + printf 'backlog_path=%s\n' "$BACKLOG" + printf 'backlog_sha256=%s\n' "$backlog_sha" + printf 'in_flight_count=%s\n' "$in_flight_count" + printf 'metadata_count=%s\n' "$metadata_count" + printf 'metadata_minus_in_flight=%s\n' "$metadata_delta" + printf 'events_begin\n' + printf '%s' "$events" + printf 'events_end\n' + } > "$receipt") 2>/dev/null; then + printf '%s\n' "$receipt" + exit 0 + fi + attempt=$((attempt + 1)) +done +printf 'fm-record-reconcile: could not allocate a create-exclusive receipt\n' >&2 +exit 1 diff --git a/bin/fm-send.sh b/bin/fm-send.sh index 9ffbb913bc8..c93a59e654a 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -10,7 +10,10 @@ # Key support is backend-specific: tmux/herdr support Escape, Enter, and C-c; # Orca currently supports Enter and C-c only, and rejects Escape. # -# Text submission is verified: the line is typed ONCE, then Enter is sent and +# Text submission first requires an affirmatively empty composer. Pending or +# unreadable input fails before typing, because appending to a parked stale order +# can execute the wrong work while a later empty-composer verdict falsely appears +# to confirm the new order. The line is then typed ONCE, then Enter is sent and # retried (Enter only, never retyped) until the target backend confirms a # submit or reports an inconclusive send. If a swallowed Enter is positively # confirmed, fm-send exits NON-ZERO so the caller knows the steer did not land @@ -75,6 +78,8 @@ fi . "$SCRIPT_DIR/fm-marker-lib.sh" # shellcheck source=bin/fm-pending-reply-lib.sh . "$SCRIPT_DIR/fm-pending-reply-lib.sh" +# shellcheck source=bin/fm-model-capacity-hold-lib.sh +. "$SCRIPT_DIR/fm-model-capacity-hold-lib.sh" FM_GUARD_CONTINUE_LINE='This is a supervision warning only; the requested message WILL still be sent.' "$SCRIPT_DIR/fm-guard.sh" || true @@ -260,6 +265,11 @@ fi # error with the attempted resolution attached. if [ "${1:-}" = "--key" ]; then + # A capacity hold refuses the model turn itself, so it gates Enter before the + # local/remote transport split rather than inside either arm. + if [ "${2:-}" = Enter ]; then + fm_model_capacity_hold_refuse "Enter submission to $T" || exit 1 + fi if [ "$TARGET_BACKEND" = remote ]; then if ! "$SCRIPT_DIR/fm-on.sh" "$TARGET_REMOTE_ID" fm-remote-secondmate-control.sh key "$TARGET_REMOTE_ID" "$2" < /dev/null; then echo "error: key '$2' not sent to remote secondmate $TARGET_REMOTE_ID; completion may be unknown" >&2 @@ -272,6 +282,12 @@ if [ "${1:-}" = "--key" ]; then fm_send_record_interrupt "$2" || exit 1 else MESSAGE=$* + fm_model_capacity_hold_refuse "text submission to $T" || exit 1 + composer_state=$(fm_backend_composer_state "$TARGET_BACKEND" "$T" "$EXPECTED_LABEL") + if [ "$composer_state" != empty ]; then + echo "error: text not sent to $T because its composer is not affirmatively empty (verdict=${composer_state:-unknown}; backend=$TARGET_BACKEND; tried $RESOLUTION_TRIED). Preserve the pane and reconcile its pending input before retrying." >&2 + exit 1 + fi if [ "$MARK_FROM_FIRSTMATE" = 1 ]; then # Reuse an existing correlation id for recovery resends; otherwise create a # durable parent expectation before delivery. Transport success never diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index a9bd93e3447..0f290d1cc0d 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -12,9 +12,9 @@ # belong in a script, not in N agent turns. # # COMPOSITION, NOT DUPLICATION: this script calls fm-lock.sh, fm-bootstrap.sh, -# and fm-wake-drain.sh as real subprocesses and prints their real output. It +# fm-record-reconcile.sh, and fm-wake-drain.sh as real subprocesses and prints their real output. It # never re-implements their logic; all sequencing/formatting logic added here -# stays local to this file. Those three scripts remain fully working +# stays local to this file. Those four scripts remain fully working # standalone with unchanged default behavior - other flows (fm-bootstrap.sh # install <tools> after consent, /updatefirstmate, the afk daemon, existing # tests) still call them directly. The one seam this script needed - @@ -34,16 +34,19 @@ # secondmate liveness, pending remote handoff retry, # X-mode artifact writes, fleet sync) also run only when # locked. -# 3. wake-drain - mutates the durable wake queue, so it also only runs +# 3. record reconcile - retires surfaced terminal rows into Done without +# pruning and receipts every metadata/backlog mismatch. +# It only runs while locked. +# 4. wake-drain - mutates the durable wake queue, so it also only runs # when locked. -# 4. context digest - data/projects.md, data/secondmates.md, data/captain.md, +# 5. context digest - data/projects.md, data/secondmates.md, data/captain.md, # data/captain-shared.md, data/learnings.md: read-only, # always safe, always runs. -# 5. fleet digest - a compact data/backlog.md identity/metadata listing, +# 6. fleet digest - a compact data/backlog.md identity/metadata listing, # every state/*.meta, a bounded state/*.status tail, # state/.afk, and a cheap per-task endpoint-liveness read: # read-only, always runs. -# 6. closing reminder - prints the context-specific watcher next step; this +# 7. closing reminder - prints the context-specific watcher next step; this # script points back to the emitted harness supervision # block and deliberately never arms the watcher itself. # @@ -291,11 +294,21 @@ else printf '(silent - all good)\n' fi -# --- 3. wake-drain ------------------------------------------------------- +# --- 3. record reconciliation -------------------------------------------- +subsection "RECORD RECONCILIATION" +if [ "$READ_ONLY" -eq 1 ]; then + printf 'skipped (read-only session) - terminal rows and inventory drift remain untouched.\n' +else + RECONCILE_OUT=$("$SCRIPT_DIR/fm-record-reconcile.sh" 2>&1) + printf '%s\n' "${RECONCILE_OUT:-'(no backlog to reconcile)'}" +fi + +# --- 4. wake-drain ------------------------------------------------------- # Drained records are this turn's first work queue, and the drain's separate # OPEN DECISIONS section remains actionable even when that queue is empty # (AGENTS.md sections 3 and 8). -# The drain also runs fm-guard.sh internally on the locked path, so the +# The +# drain also runs fm-guard.sh internally on the locked path, so the # tangle/watcher-liveness alarms land right here too, ahead of the bulk digest # below. The read-only path never touches the queue because it lacks mutation # authority, and another session may be actively draining it. It still runs diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 088079d0254..aa8d57a6420 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -211,9 +211,12 @@ SUB_HOME_MARKER=".fm-secondmate-home" . "$SCRIPT_DIR/fm-trace-context-lib.sh" # shellcheck source=bin/fm-remote-readiness-lib.sh . "$SCRIPT_DIR/fm-remote-readiness-lib.sh" +# shellcheck source=bin/fm-model-capacity-hold-lib.sh +. "$SCRIPT_DIR/fm-model-capacity-hold-lib.sh" # Fail closed before any fleet mutation: a no-mistakes gate agent must never spawn # a direct report (see bin/fm-gate-refuse-lib.sh). fm_refuse_if_gate_agent +fm_model_capacity_hold_refuse "crewmate or secondmate spawn" || exit 1 # Skip the watcher guard when re-exec'd for one pair of a batch (FM_SPAWN_NO_GUARD is # set by the batch loop below), so the guard runs once for the batch, not once per pair. [ -n "${FM_SPAWN_NO_GUARD:-}" ] || "$FM_ROOT/bin/fm-guard.sh" || true diff --git a/bin/fm-supervise-daemon.sh b/bin/fm-supervise-daemon.sh index 400a8bf5357..f55d1bde7b4 100755 --- a/bin/fm-supervise-daemon.sh +++ b/bin/fm-supervise-daemon.sh @@ -180,6 +180,11 @@ FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" # shellcheck source=bin/fm-busy-lib.sh . "$FM_DAEMON_DIR/fm-busy-lib.sh" +# Durable capacity holds block away-mode model injections while leaving watcher +# bookkeeping and cleanup available. +# shellcheck source=bin/fm-model-capacity-hold-lib.sh +. "$FM_DAEMON_DIR/fm-model-capacity-hold-lib.sh" + # --- tunables --------------------------------------------------------------- # Supervisor backends this daemon knows how to inject into today. zellij, orca, # and cmux are real backends elsewhere in firstmate (bin/fm-backend.sh) but this @@ -1112,12 +1117,16 @@ window_for_task() { # <task-key> [state] # line, or a previous injection's unsent text), defer entirely - injecting # would merge with the human's text. inject_msg() { # <message> [state] - local msg=$1 state target backend retries sleep_s verdict composer encoded + local msg=$1 state target backend retries sleep_s verdict composer encoded hold_error state="${2:-$(_state_root)}" # (1) Presence-gate: inject ONLY when afk is active. When afk is off, the # daemon self-handles and stays quiet; firstmate drives the normal always-on # watcher triage. Escalations buffer and survive for the next catch-up flush. afk_active "$state" || { log "inject deferred: afk inactive"; return 1; } + if ! hold_error=$(STATE="$state" fm_model_capacity_hold_refuse "away-supervisor model injection" 2>&1); then + log "inject deferred: $hold_error" + return 1 + fi # (2) Single-line digest: collapse any embedded newlines so submission via # send-keys + Enter is unambiguous regardless of how the TUI composer treats # them. Then use the canonical typed envelope so downstream consumers retain diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 2c7ff806ee5..9de0bbcd4b4 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -139,7 +139,7 @@ family_for_basename() { fm-crew-state.test.sh|fm-decision-hold-lifecycle.test.sh|\ fm-documentation-audiences.test.sh|fm-ensure-agents-md.test.sh|fm-grok-harness.test.sh|\ fm-kimi-harness.test.sh|fm-herdr-lab.test.sh|fm-lint.test.sh|\ - fm-operational-input.test.sh|fm-pi-primary-types.test.sh|\ + fm-model-capacity-hold.test.sh|fm-operational-input.test.sh|fm-pi-primary-types.test.sh|\ fm-send-popup-settle.test.sh|fm-send-settle.test.sh|\ fm-subagent-pretool-check.test.sh|\ fm-supervision-instructions.test.sh|fm-task-delivery.test.sh|\ @@ -151,7 +151,8 @@ family_for_basename() { fm-daemon.test.sh|fm-guard-stale-banner.test.sh|fm-pi-watch-extension.test.sh|\ fm-session-lock-ancestry.test.sh|\ fm-supervision-events.test.sh|fm-turnend-guard.test.sh|fm-wake-daemon-lifecycle-e2e.test.sh|\ - fm-wake-queue.test.sh|fm-watch-arm.test.sh|fm-watch-checkpoint.test.sh|fm-watch-triage.test.sh|\ + fm-wake-queue.test.sh|fm-watch-arm.test.sh|fm-watch-arm-ownership.test.sh|\ + fm-watch-checkpoint.test.sh|fm-watch-triage.test.sh|\ fm-watcher-lock.test.sh) printf '%s\n' watcher-wake-lock ;; @@ -174,7 +175,7 @@ family_for_basename() { printf '%s\n' secondmate ;; fm-bootstrap.test.sh|fm-fleet-sync.test.sh|fm-gate-refuse.test.sh|fm-gotmp.test.sh|\ - fm-session-start.test.sh|fm-sessionstart-nudge.test.sh|fm-tangle-guard.test.sh|\ + fm-record-reconcile.test.sh|fm-session-start.test.sh|fm-sessionstart-nudge.test.sh|fm-tangle-guard.test.sh|\ fm-update.test.sh) printf '%s\n' session-bootstrap ;; @@ -200,7 +201,7 @@ family_for_basename() { fm-afk-inject-e2e.test.sh|fm-afk-return.test.sh) printf '%s\n' afk ;; - fm-bearings-snapshot.test.sh|fm-fleet-snapshot-view.test.sh) + fm-bearings-report.test.sh|fm-bearings-snapshot.test.sh|fm-fleet-snapshot-view.test.sh) printf '%s\n' snapshot-bearings ;; fm-backend-cmux.test.sh|fm-backend-cmux-smoke.test.sh) @@ -874,6 +875,10 @@ families_for_changed_path() { bin/fm-classify-lib.sh|bin/fm-daemon*|bin/fm-turnend-guard*|bin/fm-guard.sh) printf '%s\n' watcher-wake-lock ;; + bin/fm-process-progress.sh) + printf '%s\n' watcher-wake-lock + printf '%s\n' backend-dispatch + ;; bin/fm-afk*) printf '%s\n' afk printf '%s\n' real-herdr-gated @@ -895,7 +900,7 @@ families_for_changed_path() { ;; bin/fm-session-start.sh|bin/fm-bootstrap.sh|bin/fm-fleet-sync.sh|\ bin/fm-sessionstart-nudge.sh|bin/fm-tangle*|bin/fm-update.sh|\ - bin/fm-gate-refuse*|bin/fm-lock*|bin/fm-quota-axi-lib.sh) + bin/fm-gate-refuse*|bin/fm-lock*|bin/fm-quota-axi-lib.sh|bin/fm-record-reconcile.sh) printf '%s\n' session-bootstrap ;; bin/fm-pr-*|bin/fm-merge-local.sh|bin/fm-teardown.sh|bin/fm-review-diff.sh|\ @@ -914,7 +919,12 @@ families_for_changed_path() { printf '%s\n' backend-dispatch printf '%s\n' pure-contract-unit ;; - bin/fm-bearings-snapshot.sh|bin/fm-fleet-snapshot.sh|bin/fm-fleet-view.sh) + bin/fm-bearings-report.sh|bin/fm-bearings-snapshot.sh|bin/fm-fleet-snapshot.sh|bin/fm-fleet-view.sh) + printf '%s\n' snapshot-bearings + ;; + bin/fm-custody-lib.sh) + printf '%s\n' pure-contract-unit + printf '%s\n' watcher-wake-lock printf '%s\n' snapshot-bearings ;; bin/fm-install-herdr.sh|bin/fm-install-treehouse.sh|bin/fm-herdr-ci-cleanup.sh) @@ -923,7 +933,7 @@ families_for_changed_path() { # lane's contract coverage re-runs. printf '%s\n' real-herdr-gated ;; - bin/fm-lint.sh|bin/fm-install-shellcheck.sh|\ + bin/fm-lint.sh|bin/fm-install-shellcheck.sh|bin/fm-model-capacity-hold*|\ bin/fm-brief.sh|bin/fm-ensure-agents-md.sh|bin/fm-crew-state.sh|\ bin/fm-decision-hold.sh|bin/fm-supervision*|bin/fm-transition-lib.sh|\ bin/fm-tmux-lib.sh|bin/fm-marker-lib.sh|bin/fm-operational-input.sh|bin/fm-tasks-axi-lib.sh|\ diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index 1cb8f4dfb48..d6c2f7672e2 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash -# Atomically drain durable watcher wake records, optionally annotate validated -# signal status keys after raw consumption commits, then assert liveness. +# Atomically drain durable watcher wake records, preserve the consumed queue and +# its digest-bound receipt, optionally annotate validated signal status keys +# after raw consumption commits, then assert liveness. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -8,6 +9,10 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/fm-wake-lib.sh" # shellcheck source=bin/fm-classify-lib.sh . "$SCRIPT_DIR/fm-classify-lib.sh" +# shellcheck source=bin/fm-custody-lib.sh +. "$SCRIPT_DIR/fm-custody-lib.sh" + +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" DRAIN_TMP= DRAIN_LOCK_HELD=false @@ -119,6 +124,9 @@ if [ -n "$RAW_ROWS" ]; then # crash in this micro-gap may replay a wake, and annotations stay outside it. printf '%s\n' "$RAW_ROWS" || exit "$?" fi +source_ref=$(git -C "$FM_ROOT" rev-parse HEAD 2>/dev/null || printf 'unversioned-firstmate-root') +fm_custody_preserve "$DRAIN_TMP" "$DATA/wake-receipts" \ + "$source_ref:$(basename "$FM_WAKE_QUEUE")" consumed-wake-queue >/dev/null || exit 1 rm -f "$DRAIN_TMP" || exit "$?" DRAIN_TMP= fm_lock_release "$FM_WAKE_QUEUE_LOCK" diff --git a/bin/fm-watch-arm.sh b/bin/fm-watch-arm.sh index 81c09098d84..013b18e6637 100755 --- a/bin/fm-watch-arm.sh +++ b/bin/fm-watch-arm.sh @@ -28,6 +28,10 @@ # watcher: started pid=<N> (beacon fresh) - it launched one and confirmed it # watcher: attached pid=<N> (beacon <age>s) - a live+fresh successor holds the lock; # this arm attaches and follows it +# watcher: delegated to Claude auto-arm owner=<N> watcher=<N> +# - another live Stop-hook owner already +# owns this cycle; a manual arm does not +# attach a second lifecycle to it # watcher: FAILED - no live watcher with a fresh beacon - could not confirm one # watcher: FAILED - cycle ended without an actionable reason # - a clean cycle ended with no wake and no @@ -269,6 +273,19 @@ wait_for_healthy_successor() { done } +pid_is_current_ancestor() { # <pid> + local wanted=$1 pid=${BASHPID:-$$} parent i=0 + case "$wanted" in ''|*[!0-9]*) return 1 ;; esac + while [ "$i" -lt 32 ] && [ "$pid" -gt 1 ] 2>/dev/null; do + [ "$pid" = "$wanted" ] && return 0 + parent=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d '[:space:]') || return 1 + case "$parent" in ''|*[!0-9]*) return 1 ;; esac + pid=$parent + i=$((i + 1)) + done + return 1 +} + fail_unexplained_cycle() { echo "watcher: FAILED - cycle ended without an actionable reason" return 1 @@ -372,6 +389,25 @@ print_watch_output() { [ -s "$out" ] && cat "$out" } +# A manual repair and Claude's Stop hook can otherwise attach two arm lifecycles +# to one watcher. The live ledger reproduced that shape directly: one started +# arm and one attached arm named the same watcher, then the attached lifecycle +# failed after the owner closed. When a foreign live auto-arm owner exists, do +# not attach. Verify that its watcher is healthy and return an explicit delegated +# outcome; the Stop-hook descendant itself is allowed through. +AUTOARM_OWNER_LOCK="$STATE/.claude-autoarm.lock" +autoarm_owner=$(cat "$AUTOARM_OWNER_LOCK/pid" 2>/dev/null || true) +if fm_pid_alive "$autoarm_owner" && ! pid_is_current_ancestor "$autoarm_owner"; then + if wait_for_healthy_successor; then + echo "watcher: delegated to Claude auto-arm owner=$autoarm_owner watcher=$HEALTHY_PID" + exit 0 + fi + if fm_pid_alive "$autoarm_owner"; then + echo "watcher: FAILED - Claude auto-arm owner=$autoarm_owner has no live watcher with a fresh beacon" + exit 1 + fi +fi + mode=arm case "${1:-}" in ''|arm|--arm) mode=arm ;; diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 2f150af60d8..2c4743acd41 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -23,7 +23,10 @@ # re-surface cadence, never as a wedge. Only when neither # absorb class applies does the log's last line decide: # terminal (captain-relevant) or non-terminal (no verb), -# both surfaced at once. A provably-working stale past the +# both surfaced at once. Once a done event has been +# surfaced, its retained lane is excluded from future +# stale classification without deleting metadata or the +# worktree. A provably-working stale past the # wedge threshold also surfaces, with an "escalation N" # count in the reason; at FM_WEDGE_DEMAND_INSPECT_COUNT # consecutive escalations on the SAME pane, the reason @@ -82,6 +85,8 @@ mkdir -p "$STATE" . "$SCRIPT_DIR/fm-pending-reply-lib.sh" # shellcheck source=bin/fm-busy-lib.sh . "$SCRIPT_DIR/fm-busy-lib.sh" +# shellcheck source=bin/fm-custody-lib.sh +. "$SCRIPT_DIR/fm-custody-lib.sh" WATCH_LOCK="$STATE/.watch.lock" WATCH_PATH="$SCRIPT_DIR/fm-watch.sh" @@ -266,6 +271,28 @@ recorded_windows() { # pane/hash state resets to genuinely active (see the two rm-on-reset call sites # below). FM_WEDGE_DEMAND_INSPECT_COUNT=${FM_WEDGE_DEMAND_INSPECT_COUNT:-3} +FM_PROCESS_PROGRESS_BIN=${FM_PROCESS_PROGRESS_BIN:-$SCRIPT_DIR/fm-process-progress.sh} + +# Compare cumulative CPU only across the same bound pane-root identity. A +# positive delta is real progress even when pane bytes and sparse status events +# are unchanged; it resets the wedge timer but never promotes current state. +crew_cumulative_progress_advanced() { # <window> + local win=$1 key sample identity cpu previous previous_identity previous_cpu + key=$(printf '%s' "$win" | tr ':/.' '___') + sample=$("$FM_PROCESS_PROGRESS_BIN" "$(window_backend "$win")" "$win" 2>/dev/null) || return 1 + case "$sample" in *$'\t'*) : ;; *) return 1 ;; esac + identity=${sample%%$'\t'*} + cpu=${sample#*$'\t'} + [ -n "$identity" ] || return 1 + case "$cpu" in ''|*[!0-9]*) return 1 ;; esac + previous=$(cat "$STATE/.progress-$key" 2>/dev/null || true) + printf '%s\n' "$sample" > "$STATE/.progress-$key" + previous_identity=${previous%%$'\t'*} + previous_cpu=${previous#*$'\t'} + [ "$previous_identity" = "$identity" ] || return 1 + case "$previous_cpu" in ''|*[!0-9]*) return 1 ;; esac + [ "$cpu" -gt "$previous_cpu" ] +} # Repeat-poll wedge-timer bookkeeping for an already-classified stale hash # absorbed as provably-working - repairs a missing/corrupt timer (self-heals a @@ -277,6 +304,12 @@ FM_WEDGE_DEMAND_INSPECT_COUNT=${FM_WEDGE_DEMAND_INSPECT_COUNT:-3} # line that an active run/busy pane outranked). wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count-file> local win=$1 since_file=$2 label=$3 escalation_file=$4 since age n reason + if crew_cumulative_progress_advanced "$win"; then + date +%s > "$since_file" + rm -f "$escalation_file" + triage_log "absorbed $label (cumulative CPU advanced): $win" + return 0 + fi since=$(cat "$since_file" 2>/dev/null || true) case "$since" in ''|*[!0-9]*) @@ -362,6 +395,93 @@ clear_pause_tracking() { # <window> rm -f "$STATE/.stale-$key" "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" } +retained_marker_field() { # <marker> <field> + sed -n "s/^${2}=//p" "$1" 2>/dev/null | tail -1 +} + +retained_marker_preserve() { # <marker> <task> <window> <transition> + local marker=$1 task=$2 win=$3 transition=$4 + [ -f "$marker" ] || return 0 + fm_custody_preserve "$marker" "$FM_HOME/data/retention-receipts" \ + "task=$task window=$win transition=$transition" 'retained-lane-prior-evidence' >/dev/null +} + +# Verify and record the terminal/clean-tree/unchanged-HEAD triple that permits a +# completed lane to leave stale escalation while its endpoint, worktree, and +# unlanded commits remain retained. Return 0 when the evidence is stable, 1 when +# retirement is unavailable, and 2 after a previously trusted HEAD, worktree, or +# clean-tree condition changed. Every replacement/removal versions the prior +# marker before the transition, so a reconciliation never destroys its evidence. +retained_lane_check() { # <task> <window> <status-evidence> <marker> <reason> + local task=$1 win=$2 evidence=$3 marker=$4 reason=$5 meta worktree head digest tmp count tree_status + local old_head old_worktree + meta="$STATE/$task.meta" + count=$(grep -c '^worktree=' "$meta" 2>/dev/null || true) + if [ "$count" -eq 1 ]; then + worktree=$(sed -n 's/^worktree=//p' "$meta") + else + worktree= + fi + if [ -z "$worktree" ] || [ ! -d "$worktree" ] \ + || ! git -C "$worktree" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + if [ -f "$marker" ]; then + retained_marker_preserve "$marker" "$task" "$win" 'worktree-unavailable' || return 1 + rm -f "$marker" + return 2 + fi + return 1 + fi + head=$(git -C "$worktree" rev-parse HEAD 2>/dev/null) || return 1 + tree_status=$(git -C "$worktree" status --porcelain --untracked-files=normal 2>/dev/null) \ + || return 1 + if [ -n "$tree_status" ]; then + if [ -f "$marker" ]; then + retained_marker_preserve "$marker" "$task" "$win" 'tree-became-dirty' || return 1 + rm -f "$marker" + return 2 + fi + return 1 + fi + digest=$(printf '%s' "$evidence" | { + if command -v shasum >/dev/null 2>&1; then shasum -a 256; else sha256sum; fi + } | awk '{print $1}') || return 1 + tmp=$(mktemp "$STATE/.retained-evidence.XXXXXX") || return 1 + { + printf 'schema=fm-retained-lane.v2\n' + printf 'task=%s\n' "$task" + printf 'window=%s\n' "$win" + printf 'status_digest=%s\n' "$digest" + printf 'worktree=%s\n' "$worktree" + printf 'head=%s\n' "$head" + printf 'tree_state=clean\n' + printf 'retained_reason=%s\n' "$reason" + } > "$tmp" + if [ ! -f "$marker" ]; then + mv "$tmp" "$marker" + return 0 + fi + if cmp -s "$marker" "$tmp"; then + rm -f "$tmp" + return 0 + fi + old_head=$(retained_marker_field "$marker" head) + old_worktree=$(retained_marker_field "$marker" worktree) + retained_marker_preserve "$marker" "$task" "$win" 'eligible-evidence-changed' \ + || { rm -f "$tmp"; return 1; } + mv "$tmp" "$marker" + if [ "$old_head" != "$head" ] || [ "$old_worktree" != "$worktree" ]; then + return 2 + fi + return 0 +} + +clear_answered_captain_wait() { # <task> <window> <marker> + local task=$1 win=$2 marker=$3 + [ -f "$marker" ] || return 0 + retained_marker_preserve "$marker" "$task" "$win" 'captain-answer-recorded' || return 1 + rm -f "$marker" +} + # Reconcile a declared pause or captain-held status with authoritative crew state. # Only a confidently dead ordinary crew may recover paused classification after # fm-crew-state has fallen back to stopped or unknown. @@ -930,6 +1050,51 @@ EOF key=${key//\//_} key=${key//./_} last=$(last_status_line "$STATE/$task.status") + retained_key=${w//:/_} + retained_key=${retained_key//\//_} + retained_key=${retained_key//./_} + awaiting_marker="$STATE/.awaiting-captain-$retained_key" + awaiting_open=$(status_awaiting_captain_decisions "$STATE/$task.status") + if [ -z "$awaiting_open" ]; then + clear_answered_captain_wait "$task" "$w" "$awaiting_marker" || true + elif ! crew_is_provably_working "$task" \ + && { [ -f "$awaiting_marker" ] \ + || { status_is_awaiting_captain "$last" \ + && [ "$(cat "$(_hb_surfaced_path "$task")" 2>/dev/null || true)" = "$last" ]; }; }; then + retained_lane_check "$task" "$w" "$awaiting_open" "$awaiting_marker" \ + 'complete-awaiting-unbounded-captain-decision' + retained_rc=$? + if [ "$retained_rc" -eq 0 ]; then + clear_pause_tracking "$w" + triage_log "retained awaiting-captain lane excluded from stale escalation: $w" + continue + elif [ "$retained_rc" -eq 2 ]; then + reason="stale: $w (retained evidence changed while awaiting captain)" + fm_wake_append stale "$w" "$reason" || exit 1 + wake "$reason" + fi + fi + # A completed report is actionable once, via its signal or first stale + # fallback. After that exact done line is recorded as surfaced, an idle pane + # is expected and cannot become a wedge merely because its clean worktree is + # retained for an independent gate or unlanded commits. A live worker always + # overrides this status-log evidence and remains under wedge supervision. + if status_is_done "$last" \ + && [ "$(cat "$(_hb_surfaced_path "$task")" 2>/dev/null || true)" = "$last" ] \ + && ! crew_is_provably_working "$task"; then + retained_lane_check "$task" "$w" "$last" "$STATE/.terminal-retained-$retained_key" \ + 'completed-awaiting-lifecycle-reconciliation' + retained_rc=$? + if [ "$retained_rc" -eq 0 ]; then + clear_pause_tracking "$w" + triage_log "retained completed lane excluded from stale escalation: $w" + continue + elif [ "$retained_rc" -eq 2 ]; then + reason="stale: $w (retained evidence changed after terminal report)" + fm_wake_append stale "$w" "$reason" || exit 1 + wake "$reason" + fi + fi if ! status_is_paused_or_captain_held "$last" && [ -e "$STATE/.paused-$key" ]; then clear_pause_tracking "$w" fi diff --git a/docs/architecture.md b/docs/architecture.md index 4d249606ce0..de4ae5a56d2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -9,7 +9,7 @@ firstmate's always-loaded operating contract and routing index for conditional p ## Event-driven supervision A zero-token bash watcher (`bin/fm-watch.sh`) sleeps on the fleet, classifies detected wakes in bash, and wakes the first mate only when something is actionable. -Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or an X-mode mention, stale panes whose crew is not provably working whether their status log looks terminal or non-terminal, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS`, declared external waits that remain paused past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits. +Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or an X-mode mention, stale panes whose crew is not provably working, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS`, declared external waits that remain paused past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits. Repeated provably-working stale escalations on the same unchanged pane add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker. A busy pane is otherwise exempt from staleness, but only until its latest `state/<id>.turn-ended` marker reaches `FM_BUSY_TURN_MAX_SECS`, or its `state/<id>.meta` spawn record reaches that age before any turn completes; past that bound it is routed through the same wedge escalation, with the identical reason, escalation count, and `demand-deep-inspection` marker, for inspection only - never an automatic interrupt, signal, or restart. Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so a missed process exit can be recovered by draining the queue. @@ -19,18 +19,22 @@ A concurrent replacement remains armed, every non-merged or invalid observation `bin/fm-pr-lib.sh` owns the receipt format and strict identity mechanics, while `bin/fm-watch.sh` owns queue-before-retirement ordering. No-verb wakes, such as `working:` notes and bare turn-ended signals, are benign only when `bin/fm-crew-state.sh` reports positive evidence that the crew is still working: an actively running no-mistakes step attributed to that crew's current code, or an exact busy verdict from the semantic busy-state contract. A crew that declares `paused:` for a known external wait is separately absorbed while idle and re-surfaced only on the longer pause cadence, rather than being treated as a possible wedge. +A crew that reports `done:` is retired from stale escalation after one mechanical retention check proves the exact terminal event, a clean worktree, and its current HEAD; `awaiting-captain [key=...]` uses the same baseline for an unbounded decision wait. +The baseline is trusted until the terminal event digest, worktree cleanliness, HEAD, or resolved-decision set changes; a change surfaces exactly once for reconciliation while preserving the endpoint metadata, worktree, commits, and prior baseline as custody evidence. For an ordinary crew that has stopped, the normal-mode watcher first surfaces one stale wake, then applies that same cadence to an unchanged `paused:` or durable `captain-held` endpoint only when the backend confidently reports its agent dead. Live or inconclusive liveness remains fail-open at that initial surface, and the secondmate idle-endpoint exemption is unchanged. Its initial normal-mode status signal still surfaces through the no-verb path, while away mode self-handles that routine signal and owns the later recheck. Fresh stale panes use the same current-state read before trusting the status log, so an active run or a proven busy worker outranks an old captain-relevant status-log line left behind before validation. No-change heartbeats are also benign. Absorbed wakes advance their suppression markers, log to `state/.watch-triage.log`, and keep the watcher blocking without a queue record or LLM turn. -After each drain, `fm-wake-drain.sh` runs the same liveness guard as the supervision scripts, so a lapsed watcher chain surfaces even on a turn that only drains and handles queued wakes. +After each drain, `fm-wake-drain.sh` first custody-versions the exact consumed queue bytes and writes a digest, byte count, source reference, custody class, and timestamp receipt, then runs the same liveness guard as the supervision scripts, so a lapsed watcher chain surfaces even on a turn that only drains and handles queued wakes. Routine watcher polling, supervision no-ops, elapsed waiting time, and absorbed benign wakes stay silent. A declared external wait trades that silence for one bounded recheck per pause window, so a forgotten pause cannot remain invisible indefinitely. Crew status files are append-only wake-event logs, not current-state fields. Because of that, a per-wake read of only the latest line can bury an earlier still-open `needs-decision`/`blocked` under later unrelated appends; `fm-wake-drain.sh` prints a separate, fleet-wide OPEN DECISIONS section on every drain (including the empty-queue path session-start relies on), built from `fm-classify-lib.sh`'s `status_open_decisions` fold so the buried decision keeps surfacing until it is explicitly resolved. `bin/fm-crew-state.sh <id>` is the cheap current-state read for an actionable heartbeat review: it attributes a no-mistakes run, active or terminal, only when it matches the crew's branch and current code identity, then keeps that run-step authoritative even if the pane has closed. +For a running step with a recorded native-agent PID in the current step log, that process identity must still be live and unsuspended; a pane interrupt does not stop it, and a dead or suspended bound PID is reported as a pipeline contradiction rather than working. +Older or unsupported run shapes without a verifiable PID remain explicitly indeterminate at the process layer instead of borrowing pane liveness. The script header owns the exact run-head ancestry rules. During no-mistakes' `ci` monitor phase, it also reads the ci step log tail because `axi status` reports both "still waiting on checks" and "checks green, waiting on merge" as `ci,running`. The most recent recognized ci log marker wins, so checks-green monitoring reports done while a later re-arm, failed-check, or issue marker returns the crew to working. @@ -99,7 +103,8 @@ Codex and standalone Kimi classify unknown behind explicit probes until a semant Missing, malformed, stale, untrusted, or unverified semantic state is unknown, never idle, and unknown is never promoted to busy either. Ordinary task-state consumers act only on an exact busy verdict, so an unreadable worker surfaces for a closer look instead of being absorbed as still-working or written off as finished. -Endpoint death is the only process-level override and yields dead; child processes, CPU, process sleep state, and marker modification times are not state signals. +Endpoint death is the only pane-level process override and yields dead when no current-code-matched pipeline run accounts for the task; child process presence, instantaneous CPU, process sleep state, and marker modification times are not state signals. +A positive cumulative-CPU delta across two samples of the same recorded root process and descendant closure is progress evidence only: it resets stale aging but never classifies current state, while zero delta proves nothing. `state/<id>.turn-ended` files remain wake notifications, not current state. Each record is bound to an incarnation token minted when the task's wiring is armed, so an event from a superseded incarnation is rejected rather than applied, and a record left behind by one classifies unknown. @@ -155,6 +160,14 @@ Independently, `fm-spawn.sh`, `fm-send.sh`, and `fm-teardown.sh` source `bin/fm- A normal primary checkout or crewmate worktree has neither signal and remains unaffected. The helper's header owns the exact signal detection, relocated-home limitation, test-harness bypass, and relationship to no-mistakes' HEAD-continuity guard. +## Model-capacity hold boundary + +`bin/fm-model-capacity-hold.sh` registers one durable home-wide capacity hold and retains registration and digest-bound release receipts under `data/model-capacity-holds/`. +While the marker is active, Firstmate-controlled new model work is refused at `fm-spawn.sh`, text and Enter submission at `fm-send.sh`, and away-supervisor injection at `fm-supervise-daemon.sh`. +Interrupt keys, watcher bookkeeping, evidence capture, and cleanup remain available, so a capacity hold does not strand finished work. +The boundary is intentionally honest rather than universal: the primary's current model turn and harness-native same-session watcher or Stop-hook continuations, direct human input in a TUI, provider CLI or API calls, native subagent controls, direct no-mistakes commands, lower-level backend send functions invoked outside the guarded entrypoints, and agents or detached pipeline workers that were already running before registration remain outside Firstmate's enforcement. +A capacity hold therefore prevents new work only through the listed supervised entrypoints; stopping an already-running worker requires its own explicit, evidence-preserving lifecycle action. + ## Two task shapes Ship tasks change projects and ship by project mode (`no-mistakes`, `direct-PR`, or `local-only`); scout tasks leave standalone investigation reports at `data/<id>/report.md` and never push. diff --git a/docs/arm-pretool-check.md b/docs/arm-pretool-check.md index a07084d25f9..8bca04d9509 100644 --- a/docs/arm-pretool-check.md +++ b/docs/arm-pretool-check.md @@ -75,6 +75,10 @@ This covers statically-visible literal words in command position; opaque dynamic `bin/fm-watch.sh` is protected but is not a blessed entry point. A direct `bin/fm-watch.sh` execution - relative, `<code-root>`-anchored, `$VAR`-prefixed, or `~`-prefixed - always denies with `watcher-direct`, whose reason points the caller at `bin/fm-watch-arm.sh` and `bin/fm-watch-checkpoint.sh`. +Shell no-execute syntax checks such as `bash -n bin/fm-watch.sh`, combined short flags containing `n`, and `bash --noexec bin/fm-watch.sh` are read-only data uses and are allowed. +The exception belongs to the semantic classifier, not the byte prefilter: the same path executed as `bash bin/fm-watch.sh` remains a denied direct watcher launch. +The regression matrix covers both halves through every adapter transport, alongside the full dangerous-command deny corpus, so a syntax-validation allowance cannot weaken execution protection. + The same bytes in an argument, comment, assertion, documentation query, Python string, `printf`, or `tmux send-keys` payload are data and do not make the outer command relevant. Literal `sh`, `bash`, or `zsh` `-c` payloads and literal `eval` payloads are recursively classified. diff --git a/docs/scripts.md b/docs/scripts.md index 61ed9a5e094..d0368a0c31c 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -7,7 +7,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | Script | Purpose | | ------------------------ | ------------------------------------------------------------------------------------ | -| `fm-session-start.sh` | Compose lock, bootstrap, and wake drain into the single ordered session-start digest | +| `fm-session-start.sh` | Compose lock, bootstrap, record reconciliation, and wake drain into the single ordered session-start digest | | `fm-sessionstart-nudge.sh` | Print the native session-start hook nudge when the primary has not already run the digest | | `fm-operational-input.sh` | Construct and parse the canonical cross-language operational-input protocol | | `fm-bootstrap.sh` | Detect toolchain and fleet problems, run the locked session-start sweeps, and install approved tools | @@ -15,6 +15,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-fleet-snapshot.sh` | Print the read-only structured fleet snapshot JSON (schema `fm-fleet-snapshot.v1`) | | `fm-fleet-view.sh` | Render the fleet snapshot as a human Markdown view | | `fm-bearings-snapshot.sh` | Project the fleet snapshot to the compact TOON bearings view; local-only unless `--include-prs` | +| `fm-bearings-report.sh` | Install today's Bearings report after custody-versioning any same-day predecessor | | `fm-update.sh` | Fast-forward-only self-update of firstmate and local or remote secondmate homes | | `fm-on.sh` | Execute one tracked Firstmate command in a configured remote secondmate home, using its job worker except for the doctor bootstrap | | `fm-remote-job-lib.sh` | Shared bounded remote job queue, worker readiness, LaunchAgent contract, and filesystem-composed PATH | @@ -23,6 +24,9 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-backlog-handoff.sh` | Validate and delegate queued backlog-item moves into a secondmate home | | `fm-backlog-receive.sh` | Idempotently ingest one confined remote handoff outbox through tasks-axi | | `fm-decision-hold.sh` | Create, verify, complete, and resolve durable captain-held decisions | +| `fm-model-capacity-hold.sh` | Register, inspect, and receipt release of a home-wide model-capacity hold | +| `fm-model-capacity-hold-lib.sh` | Refuse Firstmate-controlled model entrypoints while a registered capacity hold is active | +| `fm-custody-lib.sh` | Create-exclusive evidence copy and digest-bound pre-transition custody receipt | | `fm-brief.sh` | Scaffold ship (explicit `--mode`), scout, secondmate-charter, and Herdr-lab briefs | | `fm-herdr-lab.sh` | Provision and guardedly operate an isolated, never-default Herdr lab session | | `fm-install-herdr.sh` | Install CI's exact-version Herdr pin with official asset URL, SHA-256, and protocol checks | @@ -73,6 +77,8 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-supervise-daemon.sh` | Presence-gated away-mode sub-supervisor: self-handle routine wakes, guard injection by the detected primary harness, escalate batched digests, alert on failed delivery | | `fm-crew-state.sh` | Print one deterministic current-state line for a crew | | `fm-nm-run-lib.sh` | Shared branch-and-code-identity attribution for no-mistakes runs | +| `fm-process-progress.sh` | Sample cumulative CPU for one identity-bound backend process closure as progress-only evidence | +| `fm-record-reconcile.sh` | Reconcile verified terminal rows and receipt metadata/backlog drift without erasure | | `fm-tangle-lib.sh` | Shared default-branch resolution and primary-checkout tangle classification | | `fm-supervision-lib.sh` | Shared in-flight-work-without-fresh-watcher-beacon predicate | | `fm-ff-lib.sh` | Shared guarded fast-forward helper for origin pulls and local secondmate syncs | @@ -81,7 +87,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-tasks-axi-lib.sh` | Shared backlog-backend selector and `tasks-axi` compatibility probe | | `fm-quota-axi-lib.sh` | Shared `quota-axi` compatibility floor for the bootstrap diagnostic | | `fm-vendor-auth-probe.sh`| Run one hard-bounded, non-destructive authentication probe of a named vendor CLI and report the fact | -| `fm-wake-drain.sh` | Atomically drain queued watcher wakes, emit bounded best-effort status-event annotations and a fleet-wide OPEN DECISIONS section, then assert supervision health | +| `fm-wake-drain.sh` | Atomically drain and custody-version queued wakes, emit bounded status annotations and a fleet-wide OPEN DECISIONS section, then assert watcher liveness | | `fm-wake-lib.sh` | Shared durable wake queue, portable locks, and watcher identity/health helpers | | `fm-classify-lib.sh` | Shared wake-classification vocabulary and durable keyed-decision folds and scans | | `fm-send.sh` | Send one verified literal line or supported key through the target's recorded backend | diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 2ae9a6b17bb..abaff5d07fe 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -33,6 +33,8 @@ The durable wake queue preserves actionable events during the residual active-tu The model no longer re-arms after ordinary wakes. No PreToolUse hook denies fleet commands based on watcher status. A genuine auto-arm failure describes the automatic mechanism as broken and never directs a routine manual background arm. +If a manual repair races a live Claude Stop-hook owner, `fm-watch-arm.sh` verifies the owner's fresh watcher and returns a typed delegated outcome instead of attaching a second arm lifecycle to the same cycle. +The Stop-hook descendant itself is recognized through process ancestry and remains the only lifecycle owner. Terminal arm-output classification (`started`, `attached`, or `FAILED`) remains defense in depth for the manual recovery path. Codex retains its bounded foreground checkpoint protocol. Grok retains its tracked background-task notification protocol. @@ -43,6 +45,7 @@ The turn-end guard remains the final backstop rather than the normal continuity ## Arm-layer cycle contract `bin/fm-watch-arm.sh` never returns a clean empty success. +The explicit delegated-to-Claude-owner result is a clean non-empty no-op because another verified owner already carries continuity; it is not an attached arm and never waits on or signals that owner's watcher. An actionable child output returns that reason normally. A zero/empty child return rechecks the home lock and beacon, attaches to a verified healthy successor when one exists, or resolves the close against the watcher's bounded terminal-delivery ledger. An attached arm follows verified identity-matched successors and resolves the same way when that chain ends without one, because it holds no handle on the watcher's stdout and cannot read the reason line itself. diff --git a/tests/fm-arm-pretool-check.test.sh b/tests/fm-arm-pretool-check.test.sh index 5ba750aea09..c03869436e7 100755 --- a/tests/fm-arm-pretool-check.test.sh +++ b/tests/fm-arm-pretool-check.test.sh @@ -63,6 +63,8 @@ matrix_case R16 allow $'# bin/fm-watch-arm.sh &\necho ok' matrix_case R17 allow "printf '%s\\n' 'fm-watch.sh; a && b || c > out' | sed -n '1p'" matrix_case R18 allow "sh -c 'tmux send-keys -t lab \"bin/fm-watch-arm.sh &\" Enter'" matrix_case R19 allow "eval 'printf \"%s\\n\" \"bin/fm-watch-arm.sh &\"'" +matrix_case R20 allow 'bash -n bin/fm-watch.sh' +matrix_case R21 allow 'sed -n '\''1,20p'\'' bin/fm-watch.sh' matrix_case D01 deny 'bin/fm-watch-arm.sh &' matrix_case D02 deny 'nohup bin/fm-watch-arm.sh' @@ -122,6 +124,7 @@ matrix_case D55 deny 'while true; do pkill -f fm-watch; done' matrix_case D56 deny 'for x in 1; do pkill -f fm-watch; done' matrix_case D57 deny 'case x in x) pkill -f fm-watch ;; esac' matrix_case D58 deny 'until false; do kill $(pgrep -f fm-watch); done' +matrix_case D59 deny 'bash bin/fm-watch.sh' matrix_case E01 allow "bin/fm-watch-checkpoint.sh --seconds '180;still-one-arg'" matrix_case E02 allow "bin/fm-watch-checkpoint.sh --label 'fm-watch-arm.sh; literal argument'" @@ -233,6 +236,8 @@ test_direct_policy_contract() { assert_policy direct-watch-not-blessed $'deny\twatcher-direct' 'bin/fm-watch.sh' assert_policy direct-watch-expanded $'deny\twatcher-direct' '$FM_HOME/bin/fm-watch.sh' assert_policy direct-watch-safe-shape $'deny\twatcher-direct' 'cd /tmp; bin/fm-watch.sh' + assert_policy direct-watch-syntax-read allow 'bash -n bin/fm-watch.sh' + assert_policy direct-watch-script-wrapper $'deny\twatcher-nested' 'bash bin/fm-watch.sh' heredoc_data=$'cat <<\'EOF\'\nbin/fm-watch-arm.sh &\nEOF' heredoc_watcher=$'bin/fm-watch-arm.sh <<\'EOF\'\ndata only\nEOF' assert_policy direct-heredoc-data allow "$heredoc_data" diff --git a/tests/fm-backend.test.sh b/tests/fm-backend.test.sh index 1ca3cbbe764..1e8d5292360 100755 --- a/tests/fm-backend.test.sh +++ b/tests/fm-backend.test.sh @@ -689,12 +689,15 @@ strip_send_preflight() { # <log> } test_send_conformance_old_vs_new() { - local old_bin fb log_old log_new home rc_old rc_new filtered_old filtered_new + local old_bin fb log_old log_new home rc_old rc_new filtered_old filtered_new guarded_new expected_cursor expected_capture old_bin=$(build_old_bin send-old) fb=$(make_send_fakebin "$TMP_ROOT/send-fake") home="$TMP_ROOT/send-home"; mkdir -p "$home/state" log_old="$TMP_ROOT/send-old.log"; log_new="$TMP_ROOT/send-new.log" filtered_old="$TMP_ROOT/send-old.filtered.log"; filtered_new="$TMP_ROOT/send-new.filtered.log" + guarded_new="$TMP_ROOT/send-new.guarded.log" + expected_cursor=$'tmux\x1fdisplay-message\x1f-p\x1f-t\x1fsess:win\x1f#{cursor_y}' + expected_capture=$'tmux\x1fcapture-pane\x1f-e\x1f-p\x1f-t\x1fsess:win\x1f-S\x1f0\x1f-E\x1f-' # Case 1: --key path. run_send_case "$old_bin" "$fb" "$log_old" "$home" -- "sess:win" --key Escape @@ -717,7 +720,12 @@ test_send_conformance_old_vs_new() { rc_new=$? expect_code "$rc_old" "$rc_new" "fm-send plain text: old vs new exit code" strip_send_preflight "$log_old" > "$filtered_old" - strip_send_preflight "$log_new" > "$filtered_new" + strip_send_preflight "$log_new" > "$guarded_new" + [ "$(sed -n '1p' "$guarded_new")" = "$expected_cursor" ] \ + || fail "fm-send plain text: strict composer guard did not read cursor position before typing" + [ "$(sed -n '2p' "$guarded_new")" = "$expected_capture" ] \ + || fail "fm-send plain text: strict composer guard did not capture the composer before typing" + sed '1,2d' "$guarded_new" > "$filtered_new" diff -u "$filtered_old" "$filtered_new" > "$TMP_ROOT/send-diff-plain.txt" 2>&1 \ || fail "fm-send plain text: tmux command log differs old vs new"$'\n'"$(cat "$TMP_ROOT/send-diff-plain.txt")" assert_contains "$(cat "$log_new")" $'\x1f''send-keys'$'\x1f''-t'$'\x1f''sess:win'$'\x1f''-l'$'\x1f''hello captain' \ @@ -733,11 +741,16 @@ test_send_conformance_old_vs_new() { rc_new=$? expect_code "$rc_old" "$rc_new" "fm-send /skill: old vs new exit code" strip_send_preflight "$log_old" > "$filtered_old" - strip_send_preflight "$log_new" > "$filtered_new" + strip_send_preflight "$log_new" > "$guarded_new" + [ "$(sed -n '1p' "$guarded_new")" = "$expected_cursor" ] \ + || fail "fm-send /skill: strict composer guard did not read cursor position before typing" + [ "$(sed -n '2p' "$guarded_new")" = "$expected_capture" ] \ + || fail "fm-send /skill: strict composer guard did not capture the composer before typing" + sed '1,2d' "$guarded_new" > "$filtered_new" diff -u "$filtered_old" "$filtered_new" > "$TMP_ROOT/send-diff-slash.txt" 2>&1 \ || fail "fm-send /skill: tmux command log differs old vs new"$'\n'"$(cat "$TMP_ROOT/send-diff-slash.txt")" - pass "fm-send.sh: explicit tmux targets are verified, while --key/plain/slash send command shape stays old-compatible" + pass "fm-send.sh: explicit targets and empty composers are verified before old-compatible key/plain/slash submission" } # --- old vs new: fm-peek.sh -------------------------------------------------- diff --git a/tests/fm-bearings-report.test.sh b/tests/fm-bearings-report.test.sh new file mode 100755 index 00000000000..2ecd5f3dfb8 --- /dev/null +++ b/tests/fm-bearings-report.test.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Behavior tests for create-exclusive Bearings report versioning. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +TMP_ROOT=$(fm_test_tmproot fm-bearings-report) +WRITER="$ROOT/bin/fm-bearings-report.sh" + +test_prior_report_is_versioned_before_replacement() { + local home draft target archived receipt old_digest old_bytes + home="$TMP_ROOT/home" + mkdir -p "$home/data" + target="$home/data/status-report-2026-07-31.md" + draft="$home/draft.md" + printf '# Prior Bearings\nEvidence cited elsewhere.\n' > "$target" + printf '# Current Bearings\nFresh fleet state.\n' > "$draft" + old_digest=$(shasum -a 256 "$target" | awk '{print $1}') + old_bytes=$(wc -c < "$target" | tr -d '[:space:]') + + FM_HOME="$home" FM_BEARINGS_REPORT_DATE=2026-07-31 "$WRITER" "$draft" >/dev/null \ + || fail "Bearings report writer refused a valid replacement" + cmp -s "$target" "$draft" || fail "dated report did not receive the fresh draft" + + archived=$(find "$home/data/status-report-versions" -type f ! -name '*.receipt' | head -1) + receipt=$(find "$home/data/status-report-versions" -type f -name '*.receipt' | head -1) + assert_present "$archived" "prior Bearings report was destroyed without a version" + assert_present "$receipt" "prior Bearings report has no custody receipt" + printf '# Prior Bearings\nEvidence cited elsewhere.\n' > "$home/expected-prior" + cmp -s "$archived" "$home/expected-prior" || fail "versioned report bytes changed" + assert_grep "sha256=$old_digest" "$receipt" "receipt digest does not bind the prior report" + assert_grep "bytes=$old_bytes" "$receipt" "receipt byte count does not bind the prior report" + assert_grep 'source_ref=' "$receipt" "receipt omitted the source ref" + assert_grep 'custody_class=bearings-prior-snapshot' "$receipt" "receipt omitted custody class" + assert_grep 'timestamp_utc=' "$receipt" "receipt omitted its pre-transition timestamp" + assert_grep "archived_path=$archived" "$receipt" "receipt does not link the preserved bytes" + pass "Bearings versions a prior same-day report with a pre-transition custody receipt" +} + +test_repeated_replacements_never_overwrite_versions() { + local home draft count + home="$TMP_ROOT/repeat" + mkdir -p "$home/data" + draft="$home/draft.md" + printf 'version zero\n' > "$home/data/status-report-2026-07-31.md" + printf 'version one\n' > "$draft" + FM_HOME="$home" FM_BEARINGS_REPORT_DATE=2026-07-31 "$WRITER" "$draft" >/dev/null \ + || fail "first replacement failed" + printf 'version two\n' > "$draft" + FM_HOME="$home" FM_BEARINGS_REPORT_DATE=2026-07-31 "$WRITER" "$draft" >/dev/null \ + || fail "second replacement failed" + count=$(find "$home/data/status-report-versions" -type f ! -name '*.receipt' | wc -l | tr -d '[:space:]') + [ "$count" -eq 2 ] || fail "expected two create-exclusive versions, got $count" + pass "repeated Bearings replacements create unique versions" +} + +test_prior_report_is_versioned_before_replacement +test_repeated_replacements_never_overwrite_versions diff --git a/tests/fm-brief.test.sh b/tests/fm-brief.test.sh index 0e139133ad2..2439aad2a63 100755 --- a/tests/fm-brief.test.sh +++ b/tests/fm-brief.test.sh @@ -212,6 +212,8 @@ test_ship_modes_generate_clean_briefs() { assert_grep "{TASK}" "$brief" "$id: brief missing the {TASK} placeholder" assert_grep "mid-task \`working:\` line (including setup complete) is nonterminal" "$brief" \ "$id: brief missing nonterminal working:/setup-complete gate protection" + assert_no_grep "# Spec-forging contract - HARD GATE" "$brief" \ + "$id: ordinary brief rendered the opt-in spec-forging contract" assert_no_grep "EOF" "$brief" "$id: brief leaked a heredoc EOF marker (unterminated heredoc)" done pass "fm-brief.sh: no-mistakes/direct-PR/local-only briefs generate cleanly" @@ -439,6 +441,66 @@ test_herdr_lab_omission_is_loud_for_ship_and_scout() { pass "fm-brief.sh: ship and scout scaffolds make omitted Herdr intent fail-visible" } +test_spec_forging_authority_contract() { + local home id brief kind missing variable err status + home="$TMP_ROOT/spec-forging-authority-home" + mkdir -p "$home/data" + + for kind in ship scout; do + id="brief-spec-forging-$kind" + set -- --mode no-mistakes + [ "$kind" = scout ] && set -- --scout + FM_SPEC_FORGING_PRD='docs/product.md plus Cesar signature receipt' \ + FM_SPEC_FORGING_AGENTS='AGENTS.md' \ + FM_SPEC_FORGING_REPO='coreldh/example at /work/example; verify origin/main; push fork/fm/spec' \ + FM_SPEC_FORGING_FILES='.kiro/specs/example requirements.md design.md tasks.md' \ + FM_SPEC_FORGING_SLICE='slice one; deployment excluded' \ + FM_SPEC_FORGING_VISUAL_GATE='not applicable; no UI' \ + FM_SPEC_FORGING_SURFACE='commissioned Faber worker' \ + FM_SPEC_FORGING_VALIDATION_GATE='independent reviewer receives path and sha' \ + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" example "$@" --spec-forging >/dev/null 2>&1 + brief="$home/data/$id/brief.md" + assert_present "$brief" "$kind spec-forging brief was not scaffolded" + assert_grep "# Spec-forging contract - HARD GATE" "$brief" \ + "$kind spec-forging brief lost its hard gate" + assert_grep "Architecture or scope changes return to Marco Antonio; the forger never authorizes or resolves them." "$brief" \ + "$kind spec-forging brief let the forger self-resolve architecture or scope" + assert_grep "The architect is the Marco Antonio station or role and may be an explicitly commissioned worker." "$brief" \ + "$kind spec-forging brief lost the commissioned-architect definition" + assert_grep "That worker never self-appoints and never signs for Cesar." "$brief" \ + "$kind spec-forging brief let a worker self-appoint or sign for Cesar" + assert_grep "Agrippa is not the architect." "$brief" \ + "$kind spec-forging brief mislabeled Agrippa as architect" + assert_grep "After legitimate authorization, the correction returns to an independent gate." "$brief" \ + "$kind spec-forging brief lost independent re-gating after authorization" + done + + for variable in \ + FM_SPEC_FORGING_PRD FM_SPEC_FORGING_AGENTS FM_SPEC_FORGING_REPO \ + FM_SPEC_FORGING_FILES FM_SPEC_FORGING_SLICE FM_SPEC_FORGING_VISUAL_GATE \ + FM_SPEC_FORGING_SURFACE FM_SPEC_FORGING_VALIDATION_GATE; do + id="brief-spec-missing-${variable#FM_SPEC_FORGING_}" + missing=$(printf '%s' "$variable" | tr '[:upper:]_' '[:lower:]-') + err="$home/$missing.err" + status=0 + env \ + FM_SPEC_FORGING_PRD='prd' FM_SPEC_FORGING_AGENTS='agents' FM_SPEC_FORGING_REPO='repo' \ + FM_SPEC_FORGING_FILES='files' FM_SPEC_FORGING_SLICE='slice' \ + FM_SPEC_FORGING_VISUAL_GATE='visual' FM_SPEC_FORGING_SURFACE='surface' \ + FM_SPEC_FORGING_VALIDATION_GATE='gate' "$variable=" FM_HOME="$home" \ + "$ROOT/bin/fm-brief.sh" "$id" example --mode no-mistakes --spec-forging \ + >/dev/null 2>"$err" || status=$? + expect_code 1 "$status" "missing $variable must refuse spec-forging scaffold" + assert_absent "$home/data/$id" "missing $variable created a partial task directory" + done + + status=0 + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" brief-spec-secondmate --secondmate --no-projects --spec-forging \ + >/dev/null 2>"$home/secondmate.err" || status=$? + expect_code 1 "$status" "secondmate charter must reject --spec-forging" + pass "fm-brief.sh: producer scope authority is fail-closed and never self-appointed" +} + test_secondmate_no_projects_charter() { local home brief status home="$TMP_ROOT/no-projects-home" @@ -506,7 +568,7 @@ test_secondmate_marked_request_reporting_contract() { "secondmate charter did not limit keyed phases to reportable material changes" assert_grep "If its first reportable event is \`working [key=<work-slug>]: {material phase}\`" "$brief" \ "secondmate charter lost keyed working syntax for a reportable material phase" - assert_grep "use the same key on its later \`paused\`, \`done\`, \`failed\`, \`needs-decision\`, or \`blocked\` event" "$brief" \ + assert_grep "use the same key on its later \`paused\`, \`awaiting-captain\`, \`done\`, \`failed\`, \`needs-decision\`, or \`blocked\` event" "$brief" \ "secondmate charter lost same-key closure for a reportable material phase" assert_grep 'resolved [key=<work-slug>]' "$brief" \ "secondmate charter lost resolved closure for a keyed material phase" @@ -521,8 +583,10 @@ test_secondmate_marked_request_reporting_contract() { "secondmate charter lost declared external waits" assert_grep 'a captain decision, a real blocker, a failure, or work ready for review' "$brief" \ "secondmate charter lost decisions, blockers, failures, or ready outcomes" - assert_grep 'States: working, needs-decision, blocked, paused, done, failed.' "$brief" \ + assert_grep 'States: working, needs-decision, blocked, paused, awaiting-captain, done, failed.' "$brief" \ "secondmate charter changed the preserved status vocabulary" + assert_grep 'awaiting-captain [key=<slug>]' "$brief" \ + "secondmate charter omitted the durable unbounded captain-wait vocabulary" pass "fm-brief.sh: marked requests avoid generic acknowledgements and preserve material reporting" } @@ -655,7 +719,7 @@ test_pause_verb_override_renders_all_brief_scaffolds() { ;; esac brief="$home/data/$id/brief.md" - assert_grep "States: working, needs-decision, blocked, awaiting, done, failed." "$brief" \ + assert_grep "States: working, needs-decision, blocked, awaiting, awaiting-captain, done, failed." "$brief" \ "$kind brief did not render the configured pause verb in its states list" # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded. assert_grep 'Use `awaiting: {why}`' "$brief" \ @@ -721,6 +785,7 @@ test_ship_project_memory_wording test_herdr_lab_contract_is_explicit_and_complete test_herdr_lab_contract_quotes_foreign_firstmate_path test_herdr_lab_omission_is_loud_for_ship_and_scout +test_spec_forging_authority_contract test_herdr_lab_contract_applies_to_scouts_but_not_secondmates test_secondmate_no_projects_charter test_secondmate_marked_request_reporting_contract diff --git a/tests/fm-crew-state.test.sh b/tests/fm-crew-state.test.sh index 8f986b6139e..9c7948c434c 100755 --- a/tests/fm-crew-state.test.sh +++ b/tests/fm-crew-state.test.sh @@ -122,7 +122,17 @@ case "${1:-}" in esac exit 0 SH - chmod +x "$fb/no-mistakes" "$fb/tmux" "$fb/herdr" + cat > "$fb/ps" <<'SH' +#!/usr/bin/env bash +set -u +if printf '%s\n' "$*" | grep -F -- '-p 4242' >/dev/null; then + [ -n "${FM_FAKE_WORKER_PS_STATE:-}" ] || exit 1 + printf '%s codex native worker\n' "$FM_FAKE_WORKER_PS_STATE" + exit 0 +fi +exec /bin/ps "$@" +SH + chmod +x "$fb/no-mistakes" "$fb/tmux" "$fb/herdr" "$fb/ps" printf '%s\n' "$fb" } @@ -170,8 +180,10 @@ reset_fakes() { FM_FAKE_HERDR_MISSING=0 FM_FAKE_HERDR_AGENT_STATUS="" FM_FAKE_CI_LOGS="" + FM_FAKE_WORKER_PS_STATE="" export FM_FAKE_AXI_STATUS FM_FAKE_AXI_STATUS_RUN FM_FAKE_RUNS_LIST FM_FAKE_BUSY FM_FAKE_BUSY_TEXT FM_FAKE_TMUX_MISSING export FM_FAKE_HERDR_BUSY FM_FAKE_HERDR_MISSING FM_FAKE_HERDR_AGENT_STATUS FM_FAKE_CI_LOGS + export FM_FAKE_WORKER_PS_STATE } # --- run-object fixtures (TOON, as `no-mistakes axi status` emits) ----------- @@ -358,6 +370,47 @@ test_active_run_is_authoritative() { pass "active run-step is authoritative" } +test_headless_pipeline_worker_liveness_overrides_pane_interruption() { + reset_fakes + local d out + d=$(new_case headless-live) + make_repo_on_branch "$d/wt" fm/headless-live + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/headless-live.meta" "window=fm:fm-headless-live" "worktree=$d/wt" "kind=ship" "harness=codex" + FM_FAKE_AXI_STATUS="$(run_running fm/headless-live)" + FM_FAKE_CI_LOGS='codex started pid=4242' + FM_FAKE_WORKER_PS_STATE='S+' + FM_FAKE_TMUX_MISSING=1 + out=$(run_crew_state "$d" headless-live) + assert_contains "$out" 'state: working' "live detached worker was recorded halted with its pane gone" + assert_contains "$out" 'headless pipeline worker live pid=4242' \ + "run-step did not distinguish the detached worker from its interrupted pane" + pass "live headless pipeline worker remains working independently of pane interruption" +} + +test_dead_or_suspended_pipeline_worker_is_not_recorded_working() { + reset_fakes + local d out + d=$(new_case headless-dead) + make_repo_on_branch "$d/wt" fm/headless-dead + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/headless-dead.meta" "window=fm:fm-headless-dead" "worktree=$d/wt" "kind=ship" "harness=codex" + FM_FAKE_AXI_STATUS="$(run_running fm/headless-dead)" + FM_FAKE_CI_LOGS='codex started pid=4242' + FM_FAKE_WORKER_PS_STATE='' + out=$(run_crew_state "$d" headless-dead) + assert_not_contains "$out" 'state: working' "dead step worker inherited the stale running row" + assert_contains "$out" 'headless pipeline worker dead pid=4242' \ + "dead step worker was not distinguished from the run record" + + FM_FAKE_WORKER_PS_STATE='T' + out=$(run_crew_state "$d" headless-dead) + assert_not_contains "$out" 'state: working' "suspended step worker inherited the stale running row" + assert_contains "$out" 'headless pipeline worker suspended pid=4242' \ + "suspended step worker was not distinguished from a live worker" + pass "dead and suspended headless workers never inherit a stale running verdict" +} + # (b) needs-decision log + a resumed (running/fixing) run = SUPERSEDED test_stale_needs_decision_superseded() { reset_fakes @@ -1310,6 +1363,8 @@ test_missing_run_head_falls_back_to_current_state() { } test_active_run_is_authoritative +test_headless_pipeline_worker_liveness_overrides_pane_interruption +test_dead_or_suspended_pipeline_worker_is_not_recorded_working test_stale_needs_decision_superseded test_stale_blocked_superseded test_genuine_parked_not_superseded diff --git a/tests/fm-daemon.test.sh b/tests/fm-daemon.test.sh index 0cadb5af1f6..c3fed330145 100755 --- a/tests/fm-daemon.test.sh +++ b/tests/fm-daemon.test.sh @@ -1827,6 +1827,29 @@ test_inject_msg_defers_on_unrecognized_composer_state() { pass "inject_msg: unrecognized composer states defer by default" } +test_inject_msg_defers_on_registered_model_capacity_hold() { + local dir state touched + dir=$(make_supercase inject-model-capacity-hold) + state="$dir/state" + touched="$dir/backend-called" + afk_enter "$state" + cat > "$state/.model-capacity-hold" <<'EOF' +schema=fm-model-capacity-hold.v1 +hold_id=reserve-daemon +reason=capacity reserved for another dispatch +dispatch_ref=dispatch reserve-daemon +registered_at=2026-07-31T00:00:00Z +EOF + ( + fm_backend_target_exists() { : > "$touched"; return 0; } + if FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="default:w1:p2" inject_msg "hello" "$state"; then + fail "inject_msg should defer while a registered model-capacity hold is active" + fi + ) || fail "model-capacity-held inject_msg subshell failed" + assert_absent "$touched" "held injection reached the backend before refusing" + pass "inject_msg: registered model-capacity hold preserves the buffer before backend work" +} + test_afk_start_refuses_when_flag_cannot_be_written test_afk_start_ignores_stale_pidfile_without_lock test_afk_start_reclaims_stale_daemon_lock_reused_pid @@ -1926,3 +1949,4 @@ test_inject_msg_herdr_pane_gone_defers test_inject_msg_herdr_submits_through_backend_dispatch test_inject_msg_defers_on_dead_shell_unknown test_inject_msg_defers_on_unrecognized_composer_state +test_inject_msg_defers_on_registered_model_capacity_hold diff --git a/tests/fm-decision-hold-lifecycle.test.sh b/tests/fm-decision-hold-lifecycle.test.sh index 0ef84c4a6f5..f86fc35a937 100755 --- a/tests/fm-decision-hold-lifecycle.test.sh +++ b/tests/fm-decision-hold-lifecycle.test.sh @@ -111,6 +111,7 @@ write_origin_meta() { # <home> <id> [kind] local home=$1 id=$2 kind=${3:-scout} fm_write_meta "$home/state/$id.meta" \ "window=firstmate:fm-$id" \ + "endpoint_task_id=$id" \ "worktree=$home/projects/missing-$id" \ "project=$home/projects/sample" \ "harness=codex" \ @@ -119,7 +120,7 @@ write_origin_meta() { # <home> <id> [kind] } test_structured_holds_survive_teardown_and_route_resolution() { - local home id route_hold access_hold before after json open show + local home id route_hold access_hold before after json open show hold_receipt resolved_receipt decision_object saved_receipt invisible_bearings home=$(make_home durable-lifecycle) id=sample-systems-review mkdir -p "$home/data/$id" @@ -167,6 +168,21 @@ EOF || fail "shared investigation completion gate failed" assert_grep "decisions_reviewed=1" "$home/state/$id.meta" "completion attestation missing" assert_grep "decision_keys=access,route" "$home/state/$id.meta" "decision inventory was not deterministic" + hold_receipt="$home/data/decision-hold-receipts/$route_hold.hold" + assert_present "$hold_receipt" "completion did not create a durable cleanup receipt" + assert_grep "schema=fm-decision-hold-receipt.v1" "$hold_receipt" \ + "cleanup receipt lost its schema" + assert_grep "origin_id=$id" "$hold_receipt" "cleanup receipt lost task identity" + assert_grep "dispatch_id=$id" "$hold_receipt" "cleanup receipt lost dispatch identity" + assert_grep "hold_id=$route_hold" "$hold_receipt" "cleanup receipt lost hold identity" + grep -Eq '^object_sha256=[0-9a-f]{64}$' "$hold_receipt" \ + || fail "cleanup receipt object digest is absent, zero, or malformed" + assert_no_grep "object_sha256=0000000000000000000000000000000000000000000000000000000000000000" "$hold_receipt" \ + "cleanup receipt object digest is all zeroes" + grep -Eq '^bearings_sha256=[0-9a-f]{64}$' "$hold_receipt" \ + || fail "cleanup receipt Bearings digest is absent, zero, or malformed" + assert_no_grep "bearings_sha256=0000000000000000000000000000000000000000000000000000000000000000" "$hold_receipt" \ + "cleanup receipt Bearings digest is all zeroes" open=$(bash -c '. "$1"; status_open_decisions "$2"' _ \ "$ROOT/bin/fm-classify-lib.sh" "$home/state/$id.status") [ -z "$open" ] || fail "captain-held transfer did not close duplicate live status decisions: $open" @@ -181,6 +197,36 @@ EOF and (.gates | any(.id == $route or .id == $access) | not) ' >/dev/null || fail "Bearings did not surface structured captain holds: $json" + saved_receipt=$(cat "$hold_receipt") + sed 's/^dispatch_id=.*/dispatch_id=other-dispatch/' "$hold_receipt" > "$hold_receipt.tmp" + mv "$hold_receipt.tmp" "$hold_receipt" + if run_teardown "$home" "$id" >"$home/unresolved-wrong-dispatch.out" 2>"$home/unresolved-wrong-dispatch.err"; then + fail "unresolved cleanup receipt with a wrong dispatch identity allowed teardown" + fi + assert_present "$home/state/$id.meta" "wrong-dispatch refusal removed scout metadata" + printf '%s\n' "$saved_receipt" > "$hold_receipt" + + sed 's/^object_sha256=.*/object_sha256=0000000000000000000000000000000000000000000000000000000000000000/' \ + "$hold_receipt" > "$hold_receipt.tmp" + mv "$hold_receipt.tmp" "$hold_receipt" + if run_teardown "$home" "$id" >"$home/unresolved-zero-digest.out" 2>"$home/unresolved-zero-digest.err"; then + fail "unresolved cleanup receipt with an all-zero digest allowed teardown" + fi + assert_present "$home/state/$id.meta" "zero-digest refusal removed scout metadata" + printf '%s\n' "$saved_receipt" > "$hold_receipt" + + invisible_bearings="$home/invisible-bearings" + cat > "$invisible_bearings" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' '{"schema":"fm-bearings.v1","decisions_open":[]}' +EOF + chmod +x "$invisible_bearings" + if FM_DECISION_HOLD_BEARINGS="$invisible_bearings" run_teardown "$home" "$id" \ + >"$home/unresolved-invisible.out" 2>"$home/unresolved-invisible.err"; then + fail "unresolved hold absent from fresh Bearings still allowed teardown" + fi + assert_present "$home/state/$id.meta" "Bearings-invisible refusal removed scout metadata" + run_teardown "$home" "$id" >/dev/null 2> "$home/teardown.err" \ || fail "reviewed investigation teardown failed: $(cat "$home/teardown.err")" tasks_in "$home" "done" "$id" --report "data/$id/report.md" --keep 0 >/dev/null \ @@ -248,6 +294,34 @@ EOF run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ --routed-to sample-route-implementation --routed-to sample-route-followup >/dev/null \ || fail "could not resume and complete partial decision routing" + resolved_receipt="$home/data/decision-hold-receipts/$route_hold.resolved" + decision_object="$home/data/decision-hold-receipts/$route_hold.decision.md" + assert_present "$resolved_receipt" "resolved hold did not create its trusted receipt" + assert_present "$decision_object" "resolved hold did not retain the digest-bound decision object" + assert_grep "origin_id=$id" "$resolved_receipt" "resolution receipt lost task identity" + assert_grep "dispatch_id=$id" "$resolved_receipt" "resolution receipt lost dispatch identity" + assert_grep "routed_ids=sample-route-followup,sample-route-implementation" "$resolved_receipt" \ + "resolution receipt lost its routed task identities" + grep -Eq '^decision_sha256=[0-9a-f]{64}$' "$resolved_receipt" \ + || fail "resolution receipt decision digest is absent, zero, or malformed" + assert_no_grep "decision_sha256=0000000000000000000000000000000000000000000000000000000000000000" "$resolved_receipt" \ + "resolution receipt decision digest is all zeroes" + run_decisions "$home" verify-resolution "$id" route >/dev/null \ + || fail "fresh trusted resolution receipt did not verify" + saved_receipt=$(cat "$resolved_receipt") + sed 's/^dispatch_id=.*/dispatch_id=some-other-dispatch/' "$resolved_receipt" > "$resolved_receipt.tmp" + mv "$resolved_receipt.tmp" "$resolved_receipt" + if run_decisions "$home" verify-resolution "$id" route >"$home/wrong-dispatch.out" 2>"$home/wrong-dispatch.err"; then + fail "resolution receipt with the wrong dispatch identity verified" + fi + printf '%s\n' "$saved_receipt" > "$resolved_receipt" + mv "$decision_object" "$decision_object.missing" + if run_decisions "$home" verify-resolution "$id" route >"$home/missing-object.out" 2>"$home/missing-object.err"; then + fail "resolution receipt verified while its decision object path was absent" + fi + mv "$decision_object.missing" "$decision_object" + run_decisions "$home" verify-resolution "$id" route >/dev/null \ + || fail "restored trusted resolution receipt did not verify" run_decisions "$home" resolve "$id" route --decision-file "$home/route-decision.txt" \ --routed-to sample-route-implementation --routed-to sample-route-followup >/dev/null \ || fail "identical resolution retry was not idempotent" @@ -276,6 +350,102 @@ EOF pass "captain holds are idempotent, distinct, teardown-safe, Bearings-visible, and durably routed before close" } +write_done_resolution_row() { # <home> <hold-id> <digest> <routed-id> [body-marker] + local home=$1 hold=$2 digest=$3 routed=$4 marker=${5:-Routed work:} + cat >> "$home/data/backlog.md" <<EOF +- [x] $hold - Historical route choice (repo: sample) (kind: captain) (done 2026-07-31) + Resolution recorded by fm-decision-hold. + Decision digest: $digest + Routed identities: $routed + + Captain decision: + Use the historical route. + + $marker + - $routed +EOF +} + +test_resolution_receipt_attack_constructions_refuse() { + local home origin hold digest case_name duplicate_row + digest=123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0 + for case_name in absent malformed duplicate wrong-origin faithful zero-digest; do + home=$(make_home "receipt-attack-$case_name") + origin="sample-$case_name-review" + hold="$origin-decision-route" + mkdir -p "$home/data/$origin" + write_origin_meta "$home" "$origin" + printf '# Historical review\n' > "$home/data/$origin/report.md" + tasks_in "$home" add routed-task "Routed task" --kind ship --repo sample >/dev/null + case "$case_name" in + absent) : ;; + malformed) write_done_resolution_row "$home" "$hold" "$digest" routed-task "Routed identities only:" ;; + duplicate) + write_done_resolution_row "$home" "$hold" "$digest" routed-task + duplicate_row=$(awk -v prefix="- [x] $hold - " ' + index($0, prefix) == 1 {capture=1} + capture {print} + ' "$home/data/backlog.md") + printf '%s\n' "$duplicate_row" >> "$home/data/done-archive.md" + ;; + wrong-origin) write_done_resolution_row "$home" "different-review-decision-route" "$digest" routed-task ;; + faithful) write_done_resolution_row "$home" "$hold" "$digest" routed-task ;; + zero-digest) write_done_resolution_row "$home" "$hold" \ + 0000000000000000000000000000000000000000000000000000000000000000 routed-task ;; + esac + if run_decisions "$home" verify-resolution "$origin" route \ + >"$home/$case_name.out" 2>"$home/$case_name.err"; then + fail "$case_name hand-written or indeterminate resolution construction verified" + fi + done + assert_grep "trusted cleanup receipt" "$TMP_ROOT/receipt-attack-faithful/faithful.err" \ + "faithful hand-written historical row did not refuse for missing authority receipt" + assert_grep "preserve the origin and row" "$TMP_ROOT/receipt-attack-faithful/faithful.err" \ + "historical resolved-row refusal did not name the non-destructive operator remedy" + assert_grep "no safe automatic migration is shipped" "$TMP_ROOT/receipt-attack-faithful/faithful.err" \ + "historical resolved-row refusal promised an unavailable migration" + assert_grep "nonzero" "$TMP_ROOT/receipt-attack-zero-digest/zero-digest.err" \ + "all-zero historical decision digest did not fail explicitly" + pass "absent, malformed, duplicate, wrong-origin, hand-written, and zero-digest resolution rows refuse" +} + +test_historical_open_inventory_diagnostics_name_safe_remedies() { + local home origin hold receipt meta_tmp + home=$(make_home historical-open-remedy) + origin=sample-historical-open-review + mkdir -p "$home/data/$origin" + write_origin_meta "$home" "$origin" + printf '# Historical open review\n' > "$home/data/$origin/report.md" + hold=$(run_decisions "$home" hold "$origin" route \ + --title "Choose the historical route" --reason "captain route choice pending" --repo sample) \ + || fail "could not register historical-open hold fixture" + run_decisions "$home" complete "$origin" route >/dev/null \ + || fail "could not complete historical-open inventory fixture" + receipt="$home/data/decision-hold-receipts/$hold.hold" + assert_present "$receipt" "historical-open fixture did not establish its cleanup receipt" + rm "$receipt" + + if run_decisions "$home" verify "$origin" >"$home/open.out" 2>"$home/open.err"; then + fail "historical open inventory without a receipt unexpectedly verified" + fi + assert_grep "re-run complete $origin with its full recorded decision inventory" "$home/open.err" \ + "repairable historical open inventory did not name the safe complete retry" + assert_grep "only while the hold remains open and the exact dispatch binding survives" "$home/open.err" \ + "historical open remedy overstated when receipt reconstruction is safe" + + meta_tmp="$home/state/$origin.meta.next" + grep -v '^endpoint_task_id=' "$home/state/$origin.meta" > "$meta_tmp" + mv "$meta_tmp" "$home/state/$origin.meta" + if run_decisions "$home" verify "$origin" >"$home/bindingless.out" 2>"$home/bindingless.err"; then + fail "binding-less historical inventory unexpectedly verified" + fi + assert_grep "preserve origin metadata and holds" "$home/bindingless.err" \ + "binding-less historical refusal did not preserve the only authoritative evidence" + assert_grep "no safe automatic migration is shipped" "$home/bindingless.err" \ + "binding-less historical refusal promised an unavailable migration" + pass "historical open and binding-less refusals name only safe operator remedies" +} + test_scout_teardown_always_requires_inventory_verification() { local home id home=$(make_home unconditional-teardown) @@ -324,7 +494,7 @@ test_origin_slug_validation_precedes_path_construction() { } test_visual_review_uses_shared_completion_owner() { - local home id hold json + local home id hold json origin_receipt hold_receipt saved_origin routed decision home=$(make_home visual-review) id=sample-board-review mkdir -p "$home/data/$id" @@ -334,6 +504,13 @@ test_visual_review_uses_shared_completion_owner() { printf '# Sample board investigation\n\nThe initial findings need no captain choice.\n' > "$home/data/$id/report.md" run_decisions "$home" complete "$id" --none >/dev/null \ || fail "initial investigation could not pass the shared completion owner" + origin_receipt="$home/data/decision-hold-receipts/$id.origin" + assert_present "$origin_receipt" "empty initial inventory did not preserve its dispatch carrier" + assert_grep "origin_id=$id" "$origin_receipt" "dispatch carrier lost its task identity" + assert_grep "dispatch_id=$id" "$origin_receipt" "dispatch carrier lost its endpoint identity" + assert_grep "origin_path=$home/data/$id/report.md" "$origin_receipt" "dispatch carrier lost its exact report path" + grep -Eq '^origin_sha256=[0-9a-f]{64}$' "$origin_receipt" \ + || fail "dispatch carrier source digest is absent, zero, or malformed" run_teardown "$home" "$id" >/dev/null 2> "$home/visual-teardown.err" \ || fail "completed investigation teardown failed: $(cat "$home/visual-teardown.err")" tasks_in "$home" "done" "$id" --report "data/$id/report.md" --keep 0 >/dev/null @@ -343,8 +520,17 @@ test_visual_review_uses_shared_completion_owner() { hold=$(run_decisions "$home" hold "$id" layout \ --title "Choose the sample layout" --reason "captain layout choice pending" --repo sample) \ || fail "post-teardown visual review could not use the shared hold owner" + saved_origin=$(cat "$origin_receipt") + sed 's#^origin_path=.*#origin_path=/tmp/forged-report.md#' "$origin_receipt" > "$origin_receipt.tmp" + mv "$origin_receipt.tmp" "$origin_receipt" + if run_decisions "$home" complete "$id" layout >"$home/forged-origin.out" 2>"$home/forged-origin.err"; then + fail "post-teardown completion accepted a forged report path" + fi + printf '%s\n' "$saved_origin" > "$origin_receipt" run_decisions "$home" complete "$id" layout >/dev/null \ || fail "post-teardown visual review could not use the shared completion owner" + hold_receipt="$home/data/decision-hold-receipts/$hold.hold" + assert_present "$hold_receipt" "post-teardown completion did not create a cleanup receipt" [ "$hold" = "$id-decision-layout" ] || fail "visual review used a separate identity policy" json=$(run_bearings "$home") || fail "Bearings failed after the ended visual review" printf '%s' "$json" | jq -e --arg hold "$hold" ' @@ -352,9 +538,42 @@ test_visual_review_uses_shared_completion_owner() { ' >/dev/null || fail "ended visual review did not leave its durable Captain Call: $json" [ ! -e "$home/data/visual-review-decisions.json" ] \ || fail "visual review created a second decision database" + routed=sample-layout-implementation + tasks_in "$home" add "$routed" "Apply the selected sample layout" --kind ship --repo sample >/dev/null + tasks_in "$home" block "$routed" --by "$hold" >/dev/null + decision="$home/layout-decision.txt" + printf 'Use the compact sample layout.\n' > "$decision" + run_decisions "$home" resolve "$id" layout --decision-file "$decision" --routed-to "$routed" >/dev/null \ + || fail "post-teardown hold could not route its trusted resolution" + run_decisions "$home" verify-resolution "$id" layout >/dev/null \ + || fail "post-teardown resolution receipt did not verify" pass "ended visual review follows the same decision-hold completion owner" } +test_post_teardown_handwritten_resolution_refuses_completion() { + local home origin hold digest + home=$(make_home post-teardown-handwritten) + origin=sample-post-teardown-review + hold="$origin-decision-route" + digest=123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0 + mkdir -p "$home/data/$origin" + tasks_in "$home" add "$origin" "Review the historical route" --kind scout --repo sample --start >/dev/null + write_origin_meta "$home" "$origin" + printf 'done: report complete\n' > "$home/state/$origin.status" + printf '# Historical route review\n\nNo initial captain choice.\n' > "$home/data/$origin/report.md" + run_decisions "$home" complete "$origin" --none >/dev/null + run_teardown "$home" "$origin" >/dev/null 2>"$home/teardown.err" + tasks_in "$home" "done" "$origin" --report "data/$origin/report.md" --keep 0 >/dev/null + tasks_in "$home" add routed-task "Routed task" --kind ship --repo sample >/dev/null + write_done_resolution_row "$home" "$hold" "$digest" routed-task + if run_decisions "$home" complete "$origin" route >"$home/complete.out" 2>"$home/complete.err"; then + fail "post-teardown completion trusted a hand-written resolved row" + fi + assert_grep "trusted cleanup receipt" "$home/complete.err" \ + "post-teardown hand-written row did not refuse for missing script authority" + pass "post-teardown completion rejects hand-written resolved rows" +} + test_none_inventory_and_resolved_prose_do_not_create_holds() { local home id json home=$(make_home no-false-holds) @@ -481,6 +700,8 @@ test_resolve_matches_quoted_blocked_by_edges() { hold_absent=$(run_decisions "$home" hold "$origin" edge-absent \ --title "Absent edge decision" --reason "captain absent pending" --repo sample) \ || fail "could not register absent-edge hold" + run_decisions "$home" complete "$origin" edge-first edge-mid edge-last edge-absent >/dev/null \ + || fail "could not record dispatch-bound cleanup receipts for quote-edge holds" tasks_in "$home" add pad-a "Pad A" --kind ship --repo sample >/dev/null \ || fail "could not create pad-a blocker" @@ -556,7 +777,10 @@ test_scout_teardown_always_requires_inventory_verification test_structured_holds_survive_teardown_and_route_resolution test_origin_slug_validation_precedes_path_construction test_visual_review_uses_shared_completion_owner +test_post_teardown_handwritten_resolution_refuses_completion test_none_inventory_and_resolved_prose_do_not_create_holds test_terminal_single_owner_status_decision_does_not_block_empty_inventory test_secondmate_hold_stays_in_authoritative_home test_resolve_matches_quoted_blocked_by_edges +test_historical_open_inventory_diagnostics_name_safe_remedies +test_resolution_receipt_attack_constructions_refuse diff --git a/tests/fm-model-capacity-hold.test.sh b/tests/fm-model-capacity-hold.test.sh new file mode 100755 index 00000000000..c5696545fcb --- /dev/null +++ b/tests/fm-model-capacity-hold.test.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# Behavior tests for durable model-capacity holds on Firstmate-controlled paths. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +TMP_ROOT=$(fm_test_tmproot fm-model-capacity-hold) +HOLD="$ROOT/bin/fm-model-capacity-hold.sh" +SEND="$ROOT/bin/fm-send.sh" +SPAWN="$ROOT/bin/fm-spawn.sh" + +make_home() { + local home=$1 + mkdir -p "$home/state" "$home/data" "$home/fakebin" + fm_write_meta "$home/state/lane.meta" 'window=test:fm-lane' 'kind=ship' 'harness=codex' + cat > "$home/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + display-message) printf '1\n' ;; + capture-pane) printf '╭────╮\n│ │\n╰────╯\n' ;; + send-keys) printf '%s\n' "$*" >> "$FM_HOLD_SEND_LOG" ;; +esac +SH + fm_fake_exit0 "$home/fakebin" sleep + chmod +x "$home/fakebin/tmux" +} + +test_registered_hold_blocks_send_until_digest_bound_release() { + local home log err authority rc + home="$TMP_ROOT/home" + make_home "$home" + log="$home/send.log" + err="$home/send.err" + authority="$home/release-authority.txt" + : > "$log" + + FM_HOME="$home" "$HOLD" register reserve-night \ + --reason 'captain reserved model capacity' --dispatch-ref 'cm31r2 dispatch 2026-07-31' >/dev/null \ + || fail "could not register a model-capacity hold" + rc=0 + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$home" \ + FM_HOLD_SEND_LOG="$log" FM_SEND_SETTLE=0 "$SEND" lane 'new model work' \ + >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "registered model-capacity hold allowed a text steer" + [ ! -s "$log" ] || fail "held text steer reached the backend" + assert_grep 'model-capacity hold reserve-night is active' "$err" \ + "held send did not identify the active durable hold" + + printf 'Captain released reserve-night after the protected dispatch ended.\n' > "$authority" + FM_HOME="$home" "$HOLD" release reserve-night --authority-file "$authority" >/dev/null \ + || fail "could not release the registered hold" + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$home" \ + FM_HOLD_SEND_LOG="$log" FM_SEND_SETTLE=0 "$SEND" lane 'new model work' \ + >/dev/null 2>"$err" || fail "released hold still blocked text delivery" + [ -s "$log" ] || fail "released send never reached the backend" + find "$home/data/model-capacity-holds" -name 'reserve-night.registered' -type f | grep . >/dev/null \ + || fail "registration receipt disappeared on release" + release_receipt=$(find "$home/data/model-capacity-holds" -name 'reserve-night.released' -type f | head -1) + assert_present "$release_receipt" "release has no durable receipt" + authority_digest=$(shasum -a 256 "$authority" | awk '{print $1}') + assert_grep "authority_sha256=$authority_digest" "$release_receipt" \ + "release receipt is not bound to its authority object" + pass "registered model-capacity hold blocks sends until a digest-bound release" +} + +test_registered_hold_blocks_spawn_before_fleet_mutation() { + local home err rc + home="$TMP_ROOT/spawn-home" + make_home "$home" + err="$home/spawn.err" + FM_HOME="$home" "$HOLD" register reserve-spawn \ + --reason 'capacity reserved for another dispatch' --dispatch-ref 'dispatch reserve-spawn' >/dev/null \ + || fail "could not register the spawn hold" + + rc=0 + FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" "$SPAWN" new-lane sample \ + >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "registered model-capacity hold allowed a spawn" + assert_grep 'model-capacity hold reserve-spawn is active' "$err" \ + "held spawn did not identify the active durable hold" + assert_absent "$home/state/new-lane.meta" "held spawn mutated fleet metadata before refusing" + assert_absent "$home/data/new-lane" "held spawn created task data before refusing" + pass "registered model-capacity hold blocks spawn before fleet mutation" +} + +test_registered_hold_blocks_send_until_digest_bound_release +test_registered_hold_blocks_spawn_before_fleet_mutation diff --git a/tests/fm-record-reconcile.test.sh b/tests/fm-record-reconcile.test.sh new file mode 100755 index 00000000000..6d4b396d573 --- /dev/null +++ b/tests/fm-record-reconcile.test.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# Behavior tests for non-destructive fleet-record reconciliation. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +TMP_ROOT=$(fm_test_tmproot fm-record-reconcile) +RECONCILE="$ROOT/bin/fm-record-reconcile.sh" + +test_terminal_row_reconciles_while_unlanded_work_remains() { + local home wt dirty_wt receipt head + home="$TMP_ROOT/home" + wt="$home/projects/sample-terminal" + mkdir -p "$home/data" "$home/state" "$home/projects" "$wt" + cp "$ROOT/.tasks.toml" "$home/.tasks.toml" + cat > "$home/data/backlog.md" <<'EOF' +## In flight +- [ ] sample-terminal - Completed producer awaiting an independent gate (repo: sample) (kind: ship) (since 2026-07-31) +- [ ] captain-wait - Completed producer awaiting the captain (repo: sample) (kind: ship) (since 2026-07-31) +- [ ] dirty-terminal - Terminal event with uncommitted work (repo: sample) (kind: ship) (since 2026-07-31) +- [ ] missing-meta - Preserve a row whose endpoint metadata is missing (repo: sample) (kind: ship) (since 2026-07-31) + +## Queued + +## Done +EOF + git -C "$wt" init -q + git -C "$wt" config user.email test@example.com + git -C "$wt" config user.name Test + git -C "$wt" commit -q --allow-empty -m base + git -C "$wt" checkout -q -b fm/sample-terminal + git -C "$wt" commit -q --allow-empty -m 'unlanded completed work' + head=$(git -C "$wt" rev-parse HEAD) + fm_write_meta "$home/state/sample-terminal.meta" \ + 'window=test:fm-sample-terminal' "worktree=$wt" 'project=sample' 'kind=ship' 'endpoint_task_id=sample-terminal' + printf 'done: ready in branch fm/sample-terminal at %s\n' "$head" > "$home/state/sample-terminal.status" + fm_write_meta "$home/state/captain-wait.meta" \ + 'window=test:fm-captain-wait' "worktree=$wt" 'project=sample' 'kind=ship' 'endpoint_task_id=captain-wait' + printf 'awaiting-captain [key=gate]: approve the independent gate result\n' > "$home/state/captain-wait.status" + dirty_wt="$home/projects/dirty-terminal" + mkdir -p "$dirty_wt" + git -C "$dirty_wt" init -q + git -C "$dirty_wt" config user.email test@example.com + git -C "$dirty_wt" config user.name Test + git -C "$dirty_wt" commit -q --allow-empty -m base + printf 'uncommitted evidence\n' > "$dirty_wt/uncommitted.txt" + fm_write_meta "$home/state/dirty-terminal.meta" \ + 'window=test:fm-dirty-terminal' "worktree=$dirty_wt" 'project=sample' 'kind=ship' 'endpoint_task_id=dirty-terminal' + printf 'done: reported too early\n' > "$home/state/dirty-terminal.status" + fm_write_meta "$home/state/orphan-meta.meta" \ + 'window=test:fm-orphan-meta' "worktree=$home/projects/orphan" 'project=sample' 'kind=ship' + printf 'done: historical status with no metadata row\n' > "$home/state/orphan-status.status" + + FM_HOME="$home" "$RECONCILE" >/dev/null || fail "record reconciliation failed" + ! sed -n '/^## In flight/,/^## /p' "$home/data/backlog.md" | grep -F -- '- [ ] sample-terminal -' >/dev/null \ + || fail "terminal report still reads in flight" + sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- '- [x] sample-terminal -' >/dev/null \ + || fail "terminal report was not reconciled into Done" + sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- '- [x] captain-wait -' >/dev/null \ + || fail "complete awaiting-captain report was not reconciled into Done" + grep -F -- '- [ ] dirty-terminal -' "$home/data/backlog.md" >/dev/null \ + || fail "dirty terminal row was retired despite uncommitted evidence" + assert_present "$dirty_wt/uncommitted.txt" "reconciliation erased uncommitted evidence" + assert_present "$home/state/sample-terminal.meta" "reconciliation erased retained task metadata" + [ "$(git -C "$wt" rev-parse HEAD)" = "$head" ] || fail "reconciliation changed unlanded work" + assert_absent "$wt/.git/refs/remotes/origin" "fixture unexpectedly gained a landing remote" + grep -F -- '- [ ] missing-meta -' "$home/data/backlog.md" >/dev/null \ + || fail "reconciliation erased a row that proves missing metadata" + assert_present "$home/state/orphan-meta.meta" "reconciliation erased orphan metadata evidence" + receipt=$(find "$home/data/record-reconciliation" -type f -name '*.receipt' | head -1) + assert_present "$receipt" "reconciliation wrote no durable inventory receipt" + assert_grep $'terminal-retained\tsample-terminal' "$receipt" "receipt omitted the reconciled terminal row" + assert_grep $'terminal-retained\tcaptain-wait' "$receipt" "receipt omitted the reconciled captain-wait row" + assert_grep $'terminal-unreconciled\tdirty-terminal' "$receipt" "receipt omitted the dirty terminal refusal" + assert_grep $'missing-meta\tmissing-meta' "$receipt" "receipt omitted metadata-count drift" + assert_grep $'orphan-meta\torphan-meta' "$receipt" "receipt omitted the orphan metadata" + assert_grep $'orphan-status\torphan-status' "$receipt" "receipt omitted the orphan status record" + assert_grep 'in_flight_count=2' "$receipt" "receipt omitted the reconciled in-flight count" + assert_grep 'metadata_count=4' "$receipt" "receipt omitted the retained metadata count" + assert_grep 'metadata_minus_in_flight=2' "$receipt" "receipt omitted explicit metadata-count drift" + pass "terminal row reconciles without cleaning unlanded work or erasing drift evidence" +} + +test_terminal_row_reconciles_while_unlanded_work_remains diff --git a/tests/fm-send-strict.test.sh b/tests/fm-send-strict.test.sh index d65569c6199..3b7c0bae6bb 100755 --- a/tests/fm-send-strict.test.sh +++ b/tests/fm-send-strict.test.sh @@ -50,7 +50,11 @@ case "${1:-}" in printf '%%1\n' exit 0 ;; capture-pane) - printf '╭────╮\n│ │\n╰────╯\n' + if [ -n "${FM_TMUX_CAPTURE_FILE:-}" ]; then + cat "$FM_TMUX_CAPTURE_FILE" + else + printf '╭────╮\n│ │\n╰────╯\n' + fi exit 0 ;; list-windows) printf 'foreign:%s\n' "${FM_FAKE_TMUX_WINDOW:-fm-lost}" @@ -190,6 +194,29 @@ test_healthy_fm_id_send_still_works() { pass "fm-send strict: healthy fm-<id> sends still type once and submit" } +test_stale_composer_refuses_before_typing() { + local dir fb home err log capture rc got + dir="$TMP_ROOT/stale-composer"; mkdir -p "$dir" + fb=$(make_stubs "$dir"); home=$(setup_home stalecomposer); err="$dir/send.err"; log="$dir/tmux.log"; : > "$log" + capture="$dir/capture.txt" + printf '╭────────────────────╮\n│ stale prior order │\n╰────────────────────╯\n' > "$capture" + fm_write_meta "$home/state/stale.meta" "window=sess:fm-stale" "kind=ship" "harness=codex" + + rc=0 + PATH="$fb:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$home" FM_TMUX_LOG="$log" \ + FM_TMUX_CAPTURE_FILE="$capture" FM_SEND_SETTLE=0 \ + "$SEND" stale "fresh order" >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "stale composer accepted a second instruction" + got=$(cat "$log") + assert_not_contains "$got" 'literal=1 arg=fresh order' \ + "fm-send typed the fresh order into a composer that already contained stale text" + assert_not_contains "$got" 'literal=0 arg=Enter' \ + "fm-send submitted the stale composer while attempting the fresh order" + assert_contains "$(cat "$err")" 'composer is not affirmatively empty' \ + "stale-composer refusal did not explain the delivery gap" + pass "fm-send strict: stale composer text refuses before any typing or Enter" +} + test_exact_lane_id_send_still_works test_unset_fm_home_fails test_unresolvable_target_does_not_tmux_fallback @@ -197,3 +224,4 @@ test_prefixless_herdr_pane_id_fails test_unmatched_single_colon_target_must_exist test_fm_prefixed_herdr_session_is_an_explicit_target test_healthy_fm_id_send_still_works +test_stale_composer_refuses_before_typing diff --git a/tests/fm-session-start.test.sh b/tests/fm-session-start.test.sh index 1d5eb9e6d6b..5b42483f43d 100755 --- a/tests/fm-session-start.test.sh +++ b/tests/fm-session-start.test.sh @@ -771,6 +771,8 @@ SH i=1 while [ "$i" -le 40 ]; do ( + # Bash 3.2 keeps $$ fixed across subshells and has no BASHPID. A child + # shell's PPID is the portable actual PID of this concurrent subshell. harness_pid=$(sh -c 'printf "%s\n" "$PPID"') : > "$home/state/harness-$harness_pid" : > "$ready/$i" diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index b86eb9ac64e..74380d5d8e4 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -212,6 +212,33 @@ test_drain_dedupes_obvious_duplicates() { pass "drain collapses obvious duplicate heartbeat and signal records" } +test_drain_preserves_consumed_queue_evidence() { + local dir state data out archived receipt digest bytes + dir=$(make_case custody) + state="$dir/state" + data="$dir/data" + out="$dir/drain.out" + mkdir -p "$data" + append_wake "$state" signal task.status "signal: $state/task.status" || fail "signal append failed" + append_wake "$state" heartbeat heartbeat heartbeat || fail "heartbeat append failed" + digest=$(shasum -a 256 "$state/.wake-queue" | awk '{print $1}') + bytes=$(wc -c < "$state/.wake-queue" | tr -d '[:space:]') + + FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" "$DRAIN" > "$out" \ + || fail "custody-preserving drain failed" + archived=$(find "$data/wake-receipts" -type f ! -name '*.receipt' | head -1) + receipt=$(find "$data/wake-receipts" -type f -name '*.receipt' | head -1) + assert_present "$archived" "drain erased the consumed queue without preserving its bytes" + assert_present "$receipt" "drain erased the consumed queue without a custody receipt" + [ "$(shasum -a 256 "$archived" | awk '{print $1}')" = "$digest" ] \ + || fail "archived queue digest differs from the consumed queue" + assert_grep "sha256=$digest" "$receipt" "queue receipt digest does not bind the consumed bytes" + assert_grep "bytes=$bytes" "$receipt" "queue receipt byte count does not bind the consumed bytes" + assert_grep 'custody_class=consumed-wake-queue' "$receipt" "queue receipt omitted custody class" + [ ! -s "$state/.wake-queue" ] || fail "drain left consumed rows in the live queue" + pass "drain preserves consumed wake evidence before retiring live queue rows" +} + # The drain runs at the top of every wake-handling turn, so it also asserts # watcher liveness via fm-guard.sh: a lapsed re-arm chain then surfaces even on a # plain drain-and-handle turn that runs no other supervision script. It must warn @@ -394,9 +421,12 @@ test_slow_annotation_does_not_block_append_and_deleted_file_fails_open() { } test_interruption_before_and_after_raw_commit() { - local dir state before_out after_out replay_out empty_out pid rc count i + local dir state pre_data post_data before_out after_out replay_out empty_out pid rc count i dir=$(make_case interruption) state="$dir/state" + pre_data="$dir/pre-data" + post_data="$dir/post-data" + mkdir -p "$pre_data" "$post_data" before_out="$dir/before.out" after_out="$dir/after.out" replay_out="$dir/replay.out" @@ -404,7 +434,7 @@ test_interruption_before_and_after_raw_commit() { printf 'done: interruption fixture\n' > "$state/task.status" append_wake "$state" signal task.status "signal: task" || fail "pre-commit interruption wake append failed" - FM_STATE_OVERRIDE="$state" FM_WAKE_DRAIN_TEST_DELAY_BEFORE_COMMIT=5 "$DRAIN" > "$before_out" & + FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$pre_data" FM_WAKE_DRAIN_TEST_DELAY_BEFORE_COMMIT=5 "$DRAIN" > "$before_out" & pid=$! i=0 while [ "$i" -lt 100 ] && ! compgen -G "$state/.wake-queue.drain.*" >/dev/null; do @@ -418,23 +448,37 @@ test_interruption_before_and_after_raw_commit() { rc=$? set -e [ "$rc" -ne 0 ] || fail "pre-commit interruption unexpectedly succeeded" - FM_STATE_OVERRIDE="$state" "$DRAIN" > "$replay_out" || fail "restored pre-commit wake did not drain" + FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$pre_data" "$DRAIN" > "$replay_out" || fail "restored pre-commit wake did not drain" count=$(awk -F '\t' 'NF == 5 { count++ } END { print count + 0 }' "$replay_out") [ "$count" -eq 1 ] || fail "pre-commit interruption lost or duplicated the restored row" append_wake "$state" signal task.status "signal: task after commit" || fail "post-commit interruption wake append failed" - FM_STATE_OVERRIDE="$state" FM_WAKE_ENRICH_TEST_DELAY=5 "$DRAIN" > "$after_out" & + FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$post_data" FM_WAKE_ENRICH_TEST_DELAY=5 "$DRAIN" > "$after_out" & pid=$! wait_for_file_text "$after_out" "$(printf '\tsignal\ttask.status\t')" \ || { kill "$pid" 2>/dev/null || true; fail "post-commit drain did not print its raw row"; } + i=0 + while [ "$i" -lt 100 ] && ! compgen -G "$post_data/wake-receipts/.*.receipt" >/dev/null; do + sleep 0.05 + i=$((i + 1)) + done + compgen -G "$post_data/wake-receipts/.*.receipt" >/dev/null \ + || { kill "$pid" 2>/dev/null || true; fail "post-commit drain did not custody-receipt consumed bytes"; } + i=0 + while [ "$i" -lt 100 ] && [ -e "$state/.wake-queue.lock" ]; do + sleep 0.05 + i=$((i + 1)) + done + [ ! -e "$state/.wake-queue.lock" ] \ + || { kill "$pid" 2>/dev/null || true; fail "post-commit drain had not released queue ownership"; } kill -TERM "$pid" 2>/dev/null || fail "could not interrupt drain after raw commitment" set +e wait "$pid" set -e - FM_STATE_OVERRIDE="$state" "$DRAIN" > "$empty_out" || fail "drain after post-commit interruption failed" + FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$post_data" "$DRAIN" > "$empty_out" || fail "drain after post-commit interruption failed" count=$(awk -F '\t' 'NF == 5 { count++ } END { print count + 0 }' "$after_out" "$empty_out") [ "$count" -eq 1 ] || fail "post-commit interruption restored or duplicated the consumed row" - pass "interruptions restore before commitment and never replay after raw commitment" + pass "interruptions restore before custody commitment and never replay after receipt-bound retirement" } test_concurrent_append_and_drain @@ -444,6 +488,7 @@ test_not_working_stale_enqueue_before_suppressor test_check_output_is_queued test_atomic_double_drain test_drain_dedupes_obvious_duplicates +test_drain_preserves_consumed_queue_evidence test_drain_asserts_watcher_liveness test_structural_signal_enrichment_preserves_raw_rows test_enrichment_caps_and_status_file_failures diff --git a/tests/fm-watch-arm-ownership.test.sh b/tests/fm-watch-arm-ownership.test.sh new file mode 100755 index 00000000000..e71707f6ce1 --- /dev/null +++ b/tests/fm-watch-arm-ownership.test.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Regression for manual arm racing a Claude Stop-hook-owned arm cycle. +set -u + +# shellcheck source=tests/wake-helpers.sh +. "$(dirname "${BASH_SOURCE[0]}")/wake-helpers.sh" + +TMP_ROOT=$(fm_test_tmproot fm-watch-arm-ownership) +WATCH="$ROOT/bin/fm-watch.sh" +ARM="$ROOT/bin/fm-watch-arm.sh" +LIB="$ROOT/bin/fm-wake-lib.sh" + +test_manual_arm_does_not_attach_to_autoarm_owned_cycle() { + local dir state fakebin holder watcher arm out i + dir=$(make_case owner-race) + state="$dir/state" + fakebin="$dir/fakebin" + out="$dir/arm.out" + printf '%s\n' fm-pr-check-migration-scan-v1 > "$state/.pr-check-migration-scan-v1" + printf '%s\n' fm-pr-check-migration-v1 > "$state/.pr-check-migration-v1" + chmod 0600 "$state/.pr-check-migration-scan-v1" "$state/.pr-check-migration-v1" + + FM_STATE_OVERRIDE="$state" bash -c ' + . "$1" + fm_lock_try_acquire "$2" || exit 7 + sleep 20 + ' _ "$LIB" "$state/.claude-autoarm.lock" & + holder=$! + i=0 + while [ "$i" -lt 50 ] && [ ! -e "$state/.claude-autoarm.lock/pid" ]; do sleep 0.05; i=$((i + 1)); done + assert_present "$state/.claude-autoarm.lock/pid" "fixture autoarm owner lock was not published" + + PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$state" FM_POLL=5 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" >/dev/null 2>&1 & + watcher=$! + i=0 + while [ "$i" -lt 80 ] && [ ! -e "$state/.watch.lock/pid" ]; do sleep 0.05; i=$((i + 1)); done + assert_present "$state/.watch.lock/pid" "fixture watcher did not become live" + + PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$state" FM_ARM_CONFIRM_TIMEOUT=1 "$ARM" > "$out" 2>&1 & + arm=$! + if ! wait_for_exit "$arm" 30; then + kill "$arm" 2>/dev/null || true + wait "$arm" 2>/dev/null || true + kill "$watcher" "$holder" 2>/dev/null || true + wait "$watcher" "$holder" 2>/dev/null || true + fail "manual arm attached to the Stop-hook-owned cycle instead of deferring: $(cat "$out")" + fi + grep -F 'watcher: delegated to Claude auto-arm owner' "$out" >/dev/null \ + || fail "manual arm did not identify delegated cycle ownership: $(cat "$out")" + kill -0 "$watcher" 2>/dev/null || fail "ownership reconciliation stopped the live watcher" + kill "$watcher" "$holder" 2>/dev/null || true + wait "$watcher" "$holder" 2>/dev/null || true + pass "manual arm never attaches to a Claude auto-arm-owned watcher cycle" +} + +test_manual_arm_does_not_attach_to_autoarm_owned_cycle diff --git a/tests/fm-watch-triage.test.sh b/tests/fm-watch-triage.test.sh index c10565bc8af..2947956e44e 100755 --- a/tests/fm-watch-triage.test.sh +++ b/tests/fm-watch-triage.test.sh @@ -111,6 +111,15 @@ record_pi_busy() { # <state-dir> <id> reap() { kill "$1" 2>/dev/null || true; wait "$1" 2>/dev/null || true; } +make_clean_retained_worktree() { # <path> + local path=$1 + mkdir -p "$path" + git -C "$path" init -q + git -C "$path" config user.email test@example.com + git -C "$path" config user.name Test + git -C "$path" commit -q --allow-empty -m base +} + # --- pure classifier predicates (fm-classify-lib.sh) ------------------------ test_signal_reason_is_actionable_classifier() { @@ -456,6 +465,95 @@ test_terminal_stale_surfaced() { pass "a stale pane sitting on a terminal status is surfaced (queue + exit)" } +test_surfaced_done_lane_retires_from_future_stale_escalation() { + local dir state fakebin out capture_file window key sig pid terminal worktree head + dir=$(make_case terminal-retained); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt" + window="test:fm-complete" + terminal='done: ready in branch fm/complete at cf68fd9' + worktree="$dir/worktree" + make_clean_retained_worktree "$worktree" + head=$(git -C "$worktree" rev-parse HEAD) + printf 'idle composer after completion' > "$capture_file" + printf 'window=%s\nkind=ship\nworktree=%s\n' "$window" "$worktree" > "$state/complete.meta" + printf '%s\n' "$terminal" > "$state/complete.status" + sig=$(seen_sig "$state/complete.status"); printf '%s' "$sig" > "$state/.seen-complete_status" + printf '%s' "$terminal" > "$state/.hb-surfaced-complete" + key=$(printf '%s' "$window" | tr ':/.' '___') + + watch_bg "$state" "$fakebin" "$out" env FM_FAKE_TMUX_WINDOW="$window" \ + FM_FAKE_TMUX_CAPTURE="$capture_file" FM_STALE_ESCALATE_SECS=1 + pid=$! + if ! wait_live "$pid" 35; then + reap "$pid" + fail "surfaced done lane re-escalated as stale: $(cat "$out")" + fi + [ ! -s "$state/.wake-queue" ] || fail "surfaced done lane queued another stale wake" + assert_present "$state/.terminal-retained-$key" \ + "watcher did not record that the complete lane remains intentionally retained" + assert_grep "head=$head" "$state/.terminal-retained-$key" \ + "terminal retirement was not bound to the retained HEAD" + assert_grep 'tree_state=clean' "$state/.terminal-retained-$key" \ + "terminal retirement did not verify a clean worktree" + assert_present "$state/complete.meta" "terminal retirement erased task metadata" + reap "$pid" + + # A retained terminal lane is trusted only while the mechanically checked + # terminal/clean-tree/HEAD triple is unchanged. + printf 'new unreported work\n' > "$worktree/changed.txt" + : > "$out" + watch_bg "$state" "$fakebin" "$out" env FM_FAKE_TMUX_WINDOW="$window" \ + FM_FAKE_TMUX_CAPTURE="$capture_file" FM_STALE_ESCALATE_SECS=1 + pid=$! + wait_for_exit "$pid" 40 || fail "changed retained lane stayed hidden after its tree became dirty" + grep -F 'retained evidence changed' "$out" >/dev/null \ + || fail "retained lane change did not surface an actionable reason: $(cat "$out")" + pass "surfaced done lane retires from stale escalation while its worktree metadata remains" +} + +test_awaiting_captain_is_durable_but_resumed_work_still_wedges() { + local dir state fakebin out capture_file window key sig pid line worktree + dir=$(make_case awaiting-captain); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt" + window="test:fm-awaiting" + line='awaiting-captain [key=gate]: approve the independent gate result' + worktree="$dir/worktree" + make_clean_retained_worktree "$worktree" + printf 'idle complete worker' > "$capture_file" + printf 'window=%s\nkind=ship\nworktree=%s\n' "$window" "$worktree" > "$state/awaiting.meta" + printf '%s\n' "$line" > "$state/awaiting.status" + sig=$(seen_sig "$state/awaiting.status"); printf '%s' "$sig" > "$state/.seen-awaiting_status" + printf '%s' "$line" > "$state/.hb-surfaced-awaiting" + key=$(printf '%s' "$window" | tr ':/.' '___') + export FM_FAKE_CREW_STATE='state: awaiting-captain · source: status-log · approve the independent gate result' + + watch_bg "$state" "$fakebin" "$out" env FM_FAKE_TMUX_WINDOW="$window" \ + FM_FAKE_TMUX_CAPTURE="$capture_file" FM_STALE_ESCALATE_SECS=1 FM_PAUSE_RESURFACE_SECS=1 + pid=$! + if ! wait_live "$pid" 35; then + reap "$pid" + fail "awaiting-captain state resurfaced on a bounded stale cadence: $(cat "$out")" + fi + [ ! -s "$state/.wake-queue" ] || fail "awaiting-captain state queued a repeated stale wake" + assert_present "$state/.awaiting-captain-$key" "durable captain-wait marker was not recorded" + reap "$pid" + + # A stale awaiting-captain line cannot hide work that has resumed. Once the + # authoritative run-step says working, the normal wedge timer applies again. + export FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running)' + echo $(( $(date +%s) - 500 )) > "$state/.stale-since-$key" + printf 'identity-1\t100\n' > "$state/.progress-$key" + : > "$out" + watch_bg "$state" "$fakebin" "$out" env FM_FAKE_TMUX_WINDOW="$window" \ + FM_FAKE_TMUX_CAPTURE="$capture_file" FM_STALE_ESCALATE_SECS=1 \ + FM_PROCESS_PROGRESS_BIN="$fakebin/fm-no-progress" + pid=$! + wait_for_exit "$pid" 50 || fail "resumed worker hid forever behind awaiting-captain" + grep -F 'possible wedge' "$out" >/dev/null \ + || fail "resumed worker did not return to normal wedge supervision: $(cat "$out")" + pass "awaiting-captain stays quiet indefinitely but cannot hide resumed wedged work" +} + # --- stale pane, STALE terminal status overridden by an active run: absorbed --- # Regression for the 2026-07 herdr false-surface incidents: a crew's own status # log gets no new entry once firstmate hands it to a no-mistakes validation @@ -569,6 +667,49 @@ test_nonterminal_stale_provably_working_absorbed_then_escalated() { pass "provably-working non-terminal stale is absorbed on first sight, then wedge-escalated past the threshold" } +test_cumulative_cpu_delta_suppresses_false_wedge() { + local dir state fakebin out capture_file window key pane_hash sig pid progress counter + dir=$(make_case cumulative-progress); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; progress="$fakebin/progress-sample" + counter="$fakebin/progress-counter" + window="test:fm-progress" + printf 'unchanged terminal surface while tool runs' > "$capture_file" + printf 'window=%s\nkind=ship\n' "$window" > "$state/progress.meta" + printf 'working: long test run\n' > "$state/progress.status" + sig=$(seen_sig "$state/progress.status"); printf '%s' "$sig" > "$state/.seen-progress_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text 'unchanged terminal surface while tool runs') + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '2\n' > "$state/.count-$key" + printf '%s' "$pane_hash" > "$state/.stale-$key" + echo $(( $(date +%s) - 500 )) > "$state/.stale-since-$key" + printf 'identity-1\t100\n' > "$state/.progress-$key" + printf '100\n' > "$counter" + cat > "$progress" <<SH +#!/usr/bin/env bash +value=\$(cat '$counter') +value=\$((value + 20)) +printf '%s\n' "\$value" > '$counter' +printf 'identity-1\t%s\n' "\$value" +SH + chmod +x "$progress" + export FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running)' + + watch_bg "$state" "$fakebin" "$out" env FM_FAKE_TMUX_WINDOW="$window" \ + FM_FAKE_TMUX_CAPTURE="$capture_file" FM_PROCESS_PROGRESS_BIN="$progress" \ + FM_STALE_ESCALATE_SECS=1 + pid=$! + if ! wait_live "$pid" 30; then + reap "$pid" + fail "cumulative CPU delta was ignored and the progressing worker wedged: $(cat "$out")" + fi + [ ! -s "$state/.wake-queue" ] || fail "progressing worker queued a false stale wake" + grep -E '^identity-1[[:space:]][0-9]+$' "$state/.progress-$key" >/dev/null \ + || fail "watcher did not retain the new cumulative progress sample" + reap "$pid" + pass "cumulative CPU delta resets stale escalation for a demonstrably progressing worker" +} + # --- non-terminal stale, crew NOT provably working: surfaced immediately ------ # The key requirement: a crew with no running pipeline that has gone quiet (and is # not busy) has stopped - it may be done via interactive menus, waiting, or wedged. @@ -1813,8 +1954,11 @@ test_turn_ended_not_working_surfaced test_working_note_not_working_surfaced test_actionable_signal_surfaced test_terminal_stale_surfaced +test_surfaced_done_lane_retires_from_future_stale_escalation +test_awaiting_captain_is_durable_but_resumed_work_still_wedges test_stale_terminal_status_overridden_by_active_run test_nonterminal_stale_provably_working_absorbed_then_escalated +test_cumulative_cpu_delta_suppresses_false_wedge test_wedge_escalation_marks_demand_deep_inspection_after_threshold test_wedge_escalation_resets_when_pane_becomes_active test_busy_pane_below_turn_age_bound_is_absorbed From 33ffa01bd79c477382b506725928bed58b2c39d8 Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Mon, 3 Aug 2026 16:13:32 -0600 Subject: [PATCH 02/14] no-mistakes(review): Fix supervision state, progress, drain, and remote steering --- bin/fm-crew-state.sh | 1 + bin/fm-process-progress.sh | 56 ++++++++++++++++++++++++++----- bin/fm-record-reconcile.sh | 9 +++++ bin/fm-send.sh | 10 +++--- bin/fm-wake-drain.sh | 23 ++++++++++++- tests/fm-crew-state.test.sh | 8 ++++- tests/fm-process-progress.test.sh | 55 ++++++++++++++++++++++++++++++ tests/fm-record-reconcile.test.sh | 49 +++++++++++++++++++++++++-- tests/fm-send-strict.test.sh | 35 +++++++++++++++++++ tests/fm-wake-queue.test.sh | 24 +++++++++++++ 10 files changed, 254 insertions(+), 16 deletions(-) create mode 100644 tests/fm-process-progress.test.sh diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index c6143f1d95e..b7bf40d4464 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -356,6 +356,7 @@ nm_headless_worker_health() { return 0 fi case "$stat" in + *Z*) NM_WORKER_HEALTH=dead ;; *T*) NM_WORKER_HEALTH=suspended ;; *) NM_WORKER_HEALTH=live ;; esac diff --git a/bin/fm-process-progress.sh b/bin/fm-process-progress.sh index cda68c49bcc..8033751b0ae 100755 --- a/bin/fm-process-progress.sh +++ b/bin/fm-process-progress.sh @@ -12,6 +12,9 @@ set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" # shellcheck source=bin/fm-backend.sh . "$SCRIPT_DIR/fm-backend.sh" @@ -26,12 +29,14 @@ identity=$(LC_ALL=C ps -p "$root" -o lstart= -o command= 2>/dev/null) || exit 1 [ -n "$identity" ] || exit 1 if command -v shasum >/dev/null 2>&1; then identity_sha=$(printf '%s' "$root:$identity" | shasum -a 256 | awk '{print $1}') + target_sha=$(printf '%s' "$backend:$target" | shasum -a 256 | awk '{print $1}') else identity_sha=$(printf '%s' "$root:$identity" | sha256sum | awk '{print $1}') + target_sha=$(printf '%s' "$backend:$target" | sha256sum | awk '{print $1}') fi -rows=$(LC_ALL=C ps -axo pid=,ppid=,time=,command= 2>/dev/null) || exit 1 -cpu=$(printf '%s\n' "$rows" | awk -v root="$root" ' +rows=$(LC_ALL=C ps -axo pid=,ppid=,lstart=,time= 2>/dev/null) || exit 1 +current=$(printf '%s\n' "$rows" | awk -v root="$root" ' function centis(raw, d, rest, n, a, h, m, s) { d = 0 rest = raw @@ -48,9 +53,9 @@ cpu=$(printf '%s\n' "$rows" | awk -v root="$root" ' return int((((d * 24 + h) * 60 + m) * 60 + s) * 100 + 0.5) } { - pid[NR] = $1 parent[$1] = $2 - value[$1] = centis($3) + started[$1] = $3 " " $4 " " $5 " " $6 " " $7 + value[$1] = centis($8) seen[$1] = 1 } END { @@ -65,10 +70,45 @@ cpu=$(printf '%s\n' "$rows" | awk -v root="$root" ' } } } - total = 0 - for (p in member) total += value[p] - print total + 0 + for (p in member) print p "|" started[p] "\t" value[p] } ') -case "$cpu" in ''|*[!0-9]*) exit 1 ;; esac +mkdir -p "$STATE" +ledger="$STATE/.process-progress-$target_sha" +prior=/dev/null +if [ -f "$ledger" ] && [ ! -L "$ledger" ] \ + && [ "$(awk -F '\t' '$1 == "root" { print $2; exit }' "$ledger" 2>/dev/null)" = "$identity_sha" ]; then + prior=$ledger +fi +tmp=$(mktemp "$STATE/.process-progress.XXXXXX") || exit 1 +total_file=$(mktemp "$STATE/.process-progress-total.XXXXXX") || { rm -f "$tmp"; exit 1; } +printf 'root\t%s\n' "$identity_sha" > "$tmp" +awk -F '\t' -v total_file="$total_file" ' + FILENAME == ARGV[1] { + if ($1 == "retired") retired = $2 + 0 + else if ($1 == "proc") previous[$2] = $3 + 0 + next + } + { + current[$1] = $2 + 0 + } + END { + for (key in previous) { + if (!(key in current)) retired += previous[key] + } + total = retired + print "retired\t" retired + for (key in current) { + value = current[key] + if ((key in previous) && previous[key] > value) value = previous[key] + print "proc\t" key "\t" value + total += value + } + print total + 0 > total_file + } +' "$prior" <(printf '%s\n' "$current") >> "$tmp" || { rm -f "$tmp" "$total_file"; exit 1; } +cpu=$(cat "$total_file") +rm -f "$total_file" +case "$cpu" in ''|*[!0-9]*) rm -f "$tmp"; exit 1 ;; esac +mv "$tmp" "$ledger" || { rm -f "$tmp"; exit 1; } printf '%s\t%s\n' "$identity_sha" "$cpu" diff --git a/bin/fm-record-reconcile.sh b/bin/fm-record-reconcile.sh index d6cbfa0446f..8af70a77c06 100755 --- a/bin/fm-record-reconcile.sh +++ b/bin/fm-record-reconcile.sh @@ -20,6 +20,7 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" BACKLOG="$DATA/backlog.md" RECEIPT_DIR="$DATA/record-reconciliation" +FM_CREW_STATE_BIN=${FM_CREW_STATE_BIN:-$SCRIPT_DIR/fm-crew-state.sh} # shellcheck source=bin/fm-classify-lib.sh . "$SCRIPT_DIR/fm-classify-lib.sh" @@ -76,6 +77,14 @@ if fm_tasks_axi_compatible; then [ "$kind" != secondmate ] || continue last=$(last_status_line "$STATE/$id.status") status_is_done "$last" || status_is_awaiting_captain "$last" || continue + current=$("$FM_CREW_STATE_BIN" "$id" 2>/dev/null || true) + case "$current" in + 'state: done '*|'state: awaiting-captain '*) ;; + *) + append_event terminal-unreconciled "$id" "terminal event retained in flight because current state is ${current:-unavailable}" + continue + ;; + esac if ! terminal_tree_is_clean "$meta"; then append_event terminal-unreconciled "$id" 'terminal status retained in flight because worktree identity or clean-tree evidence is unavailable' continue diff --git a/bin/fm-send.sh b/bin/fm-send.sh index c93a59e654a..68dc095ebbe 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -283,10 +283,12 @@ if [ "${1:-}" = "--key" ]; then else MESSAGE=$* fm_model_capacity_hold_refuse "text submission to $T" || exit 1 - composer_state=$(fm_backend_composer_state "$TARGET_BACKEND" "$T" "$EXPECTED_LABEL") - if [ "$composer_state" != empty ]; then - echo "error: text not sent to $T because its composer is not affirmatively empty (verdict=${composer_state:-unknown}; backend=$TARGET_BACKEND; tried $RESOLUTION_TRIED). Preserve the pane and reconcile its pending input before retrying." >&2 - exit 1 + if [ "$TARGET_BACKEND" != remote ]; then + composer_state=$(fm_backend_composer_state "$TARGET_BACKEND" "$T" "$EXPECTED_LABEL") + if [ "$composer_state" != empty ]; then + echo "error: text not sent to $T because its composer is not affirmatively empty (verdict=${composer_state:-unknown}; backend=$TARGET_BACKEND; tried $RESOLUTION_TRIED). Preserve the pane and reconcile its pending input before retrying." >&2 + exit 1 + fi fi if [ "$MARK_FROM_FIRSTMATE" = 1 ]; then # Reuse an existing correlation id for recovery resends; otherwise create a diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index d6c2f7672e2..4aff4dec8f1 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -13,6 +13,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$SCRIPT_DIR/fm-custody-lib.sh" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +FM_RECORD_RECONCILE_BIN=${FM_RECORD_RECONCILE_BIN:-$SCRIPT_DIR/fm-record-reconcile.sh} DRAIN_TMP= DRAIN_LOCK_HELD=false @@ -80,6 +81,20 @@ EOF fi } +terminal_wake_present() { # <annotations> + local annotations=$1 event + case "$annotations" in + *"annotations omitted"*) return 0 ;; + esac + while IFS= read -r event; do + [ -n "$event" ] || continue + if status_is_done "$event" || status_is_awaiting_captain "$event"; then + return 0 + fi + done < <(printf '%s\n' "$annotations" | sed -n 's/^wake annotation: latest wake-EVENT observed at drain, not current state\(; historical \/ not necessarily the triggering event\)\{0,1\}: [A-Za-z0-9._-]*\.status: //p') + return 1 +} + # shellcheck disable=SC2317,SC2329 # Invoked by trap handlers below. cleanup() { local status=$? @@ -134,7 +149,13 @@ DRAIN_LOCK_HELD=false # Raw output and queue deletion are authoritative. Everything below is # best-effort and cannot restore, duplicate, hide, or fail the consumed rows. -(fm_wake_print_annotations "$RAW_ROWS") || true +annotations=$(fm_wake_print_annotations "$RAW_ROWS") || annotations= +if [ -n "$annotations" ]; then + printf '%s\n' "$annotations" +fi +if terminal_wake_present "$annotations"; then + "$FM_RECORD_RECONCILE_BIN" >/dev/null 2>&1 || true +fi (print_open_decisions_section) || true assert_watcher_liveness exit 0 diff --git a/tests/fm-crew-state.test.sh b/tests/fm-crew-state.test.sh index 9c7948c434c..40155a6b6f0 100755 --- a/tests/fm-crew-state.test.sh +++ b/tests/fm-crew-state.test.sh @@ -408,7 +408,13 @@ test_dead_or_suspended_pipeline_worker_is_not_recorded_working() { assert_not_contains "$out" 'state: working' "suspended step worker inherited the stale running row" assert_contains "$out" 'headless pipeline worker suspended pid=4242' \ "suspended step worker was not distinguished from a live worker" - pass "dead and suspended headless workers never inherit a stale running verdict" + + FM_FAKE_WORKER_PS_STATE='Z+' + out=$(run_crew_state "$d" headless-dead) + assert_not_contains "$out" 'state: working' "zombie step worker inherited the stale running row" + assert_contains "$out" 'headless pipeline worker dead pid=4242' \ + "zombie step worker was not classified as dead" + pass "dead, suspended, and zombie workers never inherit a stale running verdict" } # (b) needs-decision log + a resumed (running/fixing) run = SUPERSEDED diff --git a/tests/fm-process-progress.test.sh b/tests/fm-process-progress.test.sh new file mode 100644 index 00000000000..51b4873c957 --- /dev/null +++ b/tests/fm-process-progress.test.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +TMP_ROOT=$(fm_test_tmproot fm-process-progress) +FAKEBIN=$(fm_fakebin "$TMP_ROOT/fakebin") +SNAPSHOT="$TMP_ROOT/snapshot" +PROGRESS="$ROOT/bin/fm-process-progress.sh" + +cat > "$FAKEBIN/ps" <<'SH' +#!/usr/bin/env bash +case "$*" in + '-p 42 -o lstart= -o command=') + printf 'Mon Aug 3 12:00:00 2026 root-process\n' + ;; + '-axo pid=,ppid=,lstart=,time=') + cat "$FM_FAKE_PS_SNAPSHOT" + ;; + *) + exit 1 + ;; +esac +SH +chmod +x "$FAKEBIN/ps" + +sample() { + PATH="$FAKEBIN:$PATH" FM_STATE_OVERRIDE="$TMP_ROOT/state" \ + FM_PROCESS_PROGRESS_ROOT_PID=42 FM_FAKE_PS_SNAPSHOT="$SNAPSHOT" \ + "$PROGRESS" tmux test:fm-progress +} + +mkdir -p "$TMP_ROOT/state" +cat > "$SNAPSHOT" <<'EOF' + 42 1 Mon Aug 3 12:00:00 2026 0:00.20 + 43 42 Mon Aug 3 12:00:01 2026 0:00.80 +EOF +first=$(sample) || fail "initial cumulative sample failed" +cat > "$SNAPSHOT" <<'EOF' + 42 1 Mon Aug 3 12:00:00 2026 0:00.20 +EOF +second=$(sample) || fail "post-child-exit sample failed" +cat > "$SNAPSHOT" <<'EOF' + 42 1 Mon Aug 3 12:00:00 2026 0:00.25 +EOF +third=$(sample) || fail "subsequent root-progress sample failed" + +first_cpu=${first#*$'\t'} +second_cpu=${second#*$'\t'} +third_cpu=${third#*$'\t'} +[ "$first_cpu" -eq 100 ] || fail "initial descendant total was $first_cpu, expected 100" +[ "$second_cpu" -eq "$first_cpu" ] || fail "exited child made cumulative CPU fall from $first_cpu to $second_cpu" +[ "$third_cpu" -gt "$second_cpu" ] || fail "new CPU after child exit did not advance the cumulative total" +pass "process progress remains monotonic across descendant exit" diff --git a/tests/fm-record-reconcile.test.sh b/tests/fm-record-reconcile.test.sh index 6d4b396d573..aa7f64d8582 100755 --- a/tests/fm-record-reconcile.test.sh +++ b/tests/fm-record-reconcile.test.sh @@ -9,7 +9,7 @@ TMP_ROOT=$(fm_test_tmproot fm-record-reconcile) RECONCILE="$ROOT/bin/fm-record-reconcile.sh" test_terminal_row_reconciles_while_unlanded_work_remains() { - local home wt dirty_wt receipt head + local home wt dirty_wt receipt head crew_state home="$TMP_ROOT/home" wt="$home/projects/sample-terminal" mkdir -p "$home/data" "$home/state" "$home/projects" "$wt" @@ -51,8 +51,17 @@ EOF fm_write_meta "$home/state/orphan-meta.meta" \ 'window=test:fm-orphan-meta' "worktree=$home/projects/orphan" 'project=sample' 'kind=ship' printf 'done: historical status with no metadata row\n' > "$home/state/orphan-status.status" + crew_state="$home/crew-state" + cat > "$crew_state" <<'SH' +#!/usr/bin/env bash +case "$1" in + captain-wait) printf 'state: awaiting-captain · source: status-log\n' ;; + *) printf 'state: done · source: status-log\n' ;; +esac +SH + chmod +x "$crew_state" - FM_HOME="$home" "$RECONCILE" >/dev/null || fail "record reconciliation failed" + FM_HOME="$home" FM_CREW_STATE_BIN="$crew_state" "$RECONCILE" >/dev/null || fail "record reconciliation failed" ! sed -n '/^## In flight/,/^## /p' "$home/data/backlog.md" | grep -F -- '- [ ] sample-terminal -' >/dev/null \ || fail "terminal report still reads in flight" sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- '- [x] sample-terminal -' >/dev/null \ @@ -82,4 +91,40 @@ EOF pass "terminal row reconciles without cleaning unlanded work or erasing drift evidence" } +test_resumed_worker_overrides_stale_terminal_event() { + local home wt crew_state receipt + home="$TMP_ROOT/resumed" + wt="$home/projects/resumed" + mkdir -p "$home/data" "$home/state" "$wt" + cp "$ROOT/.tasks.toml" "$home/.tasks.toml" + cat > "$home/data/backlog.md" <<'EOF' +## In flight +- [ ] resumed - Producer resumed after an earlier terminal event (repo: sample) (kind: ship) (since 2026-07-31) + +## Queued + +## Done +EOF + git -C "$wt" init -q + git -C "$wt" config user.email test@example.com + git -C "$wt" config user.name Test + git -C "$wt" commit -q --allow-empty -m base + fm_write_meta "$home/state/resumed.meta" "window=test:fm-resumed" "worktree=$wt" "project=sample" "kind=ship" + printf 'done: earlier completion event\n' > "$home/state/resumed.status" + crew_state="$home/crew-state" + cat > "$crew_state" <<'SH' +#!/usr/bin/env bash +printf 'state: working · source: run-step · validating (running)\n' +SH + chmod +x "$crew_state" + + FM_HOME="$home" FM_CREW_STATE_BIN="$crew_state" "$RECONCILE" >/dev/null || fail "resumed-worker reconciliation failed" + grep -F -- '- [ ] resumed -' "$home/data/backlog.md" >/dev/null \ + || fail "stale terminal event retired a worker that had resumed" + receipt=$(find "$home/data/record-reconciliation" -type f -name '*.receipt' | head -1) + assert_grep $'terminal-unreconciled\tresumed' "$receipt" "receipt omitted the authoritative active-state refusal" + pass "active current state overrides a stale terminal event" +} + test_terminal_row_reconciles_while_unlanded_work_remains +test_resumed_worker_overrides_stale_terminal_event diff --git a/tests/fm-send-strict.test.sh b/tests/fm-send-strict.test.sh index 3b7c0bae6bb..2c6ad2d48cd 100755 --- a/tests/fm-send-strict.test.sh +++ b/tests/fm-send-strict.test.sh @@ -217,6 +217,40 @@ test_stale_composer_refuses_before_typing() { pass "fm-send strict: stale composer text refuses before any typing or Enter" } +test_remote_send_uses_host_local_composer_check() { + local dir fb home err log rc decoded + dir="$TMP_ROOT/remote"; mkdir -p "$dir" + fb=$(make_stubs "$dir"); home=$(setup_home remote); err="$dir/send.err"; log="$dir/ssh.log"; decoded="$dir/decoded.log" + mkdir -p "$home/data" + cat > "$home/data/secondmates.md" <<'EOF' +- ios - iOS delivery (host: remote-mac; root: /remote/root; home: /remote/home; scope: iOS work; projects: alpha; added 2026-08-02) +EOF + fm_write_meta "$home/state/ios.meta" "remote_host=remote-mac" "kind=ship" "harness=codex" + cat > "$fb/ssh" <<'SH' +#!/usr/bin/env bash +set -u +while [ "$#" -gt 0 ]; do + case "$1" in -o) shift 2 ;; --) shift; break ;; *) exit 90 ;; esac +done +printf '%s\n' "$*" > "$FM_SSH_LOG" +perl -MMIME::Base64=decode_base64 -e ' + my $data = decode_base64($ARGV[0]); + $data =~ s/\0/\n/g; + print $data; +' "$6" > "$FM_SSH_DECODED" +SH + chmod +x "$fb/ssh" + + rc=0 + PATH="$fb:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" FM_SSH_BIN="$fb/ssh" \ + FM_SSH_LOG="$log" FM_SSH_DECODED="$decoded" FM_SEND_SETTLE=0 \ + "$SEND" fm-ios "remote order" >/dev/null 2>"$err" || rc=$? + expect_code 0 "$rc" "remote task selector should delegate the composer check to the remote host" + assert_contains "$(cat "$decoded")" "fm-remote-secondmate-control.sh" "remote send did not use the host-local control path" + assert_contains "$(cat "$decoded")" "remote order" "remote send lost the requested text" + pass "fm-send strict: remote text reaches the host-local guarded sender" +} + test_exact_lane_id_send_still_works test_unset_fm_home_fails test_unresolvable_target_does_not_tmux_fallback @@ -225,3 +259,4 @@ test_unmatched_single_colon_target_must_exist test_fm_prefixed_herdr_session_is_an_explicit_target test_healthy_fm_id_send_still_works test_stale_composer_refuses_before_typing +test_remote_send_uses_host_local_composer_check diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index 74380d5d8e4..6c67446a6fb 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -481,6 +481,29 @@ test_interruption_before_and_after_raw_commit() { pass "interruptions restore before custody commitment and never replay after receipt-bound retirement" } +test_terminal_signal_invokes_record_reconciliation() { + local dir state reconcile marker out + dir=$(make_case terminal-reconcile) + state="$dir/state" + reconcile="$dir/reconcile" + marker="$dir/reconciled" + out="$dir/drain.out" + cat > "$reconcile" <<SH +#!/usr/bin/env bash +: > '$marker' +SH + chmod +x "$reconcile" + printf 'working: still active\n' > "$state/task.status" + append_wake "$state" signal task.status "signal: $state/task.status" || fail "nonterminal signal append failed" + FM_STATE_OVERRIDE="$state" FM_RECORD_RECONCILE_BIN="$reconcile" "$DRAIN" > "$out" || fail "nonterminal drain failed" + assert_absent "$marker" "nonterminal signal invoked record reconciliation" + printf 'done: producer complete\n' >> "$state/task.status" + append_wake "$state" signal task.status "signal: $state/task.status" || fail "terminal signal append failed" + FM_STATE_OVERRIDE="$state" FM_RECORD_RECONCILE_BIN="$reconcile" "$DRAIN" > "$out" || fail "terminal drain failed" + assert_present "$marker" "terminal signal bypassed record reconciliation" + pass "terminal signals reconcile records at the shared drain boundary" +} + test_concurrent_append_and_drain test_signal_catchup_without_running_watcher test_stale_enqueue_before_suppressor @@ -494,3 +517,4 @@ test_structural_signal_enrichment_preserves_raw_rows test_enrichment_caps_and_status_file_failures test_slow_annotation_does_not_block_append_and_deleted_file_fails_open test_interruption_before_and_after_raw_commit +test_terminal_signal_invokes_record_reconciliation From 2191f6706931c72d99a7a51a90ba20d5b871fc7f Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Mon, 3 Aug 2026 16:22:39 -0600 Subject: [PATCH 03/14] no-mistakes(review): Reconcile truncated terminal wake annotations --- bin/fm-wake-drain.sh | 2 +- tests/fm-wake-queue.test.sh | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index 4aff4dec8f1..2a953ebed7c 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -84,7 +84,7 @@ EOF terminal_wake_present() { # <annotations> local annotations=$1 event case "$annotations" in - *"annotations omitted"*) return 0 ;; + *"annotations omitted"*|*"[truncated]"*) return 0 ;; esac while IFS= read -r event; do [ -n "$event" ] || continue diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index 6c67446a6fb..4386ec1cd86 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -501,7 +501,13 @@ SH append_wake "$state" signal task.status "signal: $state/task.status" || fail "terminal signal append failed" FM_STATE_OVERRIDE="$state" FM_RECORD_RECONCILE_BIN="$reconcile" "$DRAIN" > "$out" || fail "terminal drain failed" assert_present "$marker" "terminal signal bypassed record reconciliation" - pass "terminal signals reconcile records at the shared drain boundary" + rm -f "$marker" + awk 'BEGIN { printf "done: "; for (i = 0; i < 9000; i++) printf "x"; printf "\n" }' > "$state/task.status" + append_wake "$state" signal task.status "signal: $state/task.status" || fail "truncated terminal signal append failed" + FM_STATE_OVERRIDE="$state" FM_RECORD_RECONCILE_BIN="$reconcile" "$DRAIN" > "$out" || fail "truncated terminal drain failed" + assert_grep '[truncated]' "$out" "oversized terminal fixture did not exercise annotation truncation" + assert_present "$marker" "truncated terminal annotation bypassed record reconciliation" + pass "terminal signals reconcile records even when annotations truncate" } test_concurrent_append_and_drain From 69476d53f93d0a5dea023bb6a2cadb375793a1ad Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Fri, 31 Jul 2026 23:15:11 -0600 Subject: [PATCH 04/14] fix: reconcile terminal wakes immediately --- bin/fm-crew-state.sh | 3 +- bin/fm-wake-drain.sh | 51 ++++++++++++++++++------------- docs/architecture.md | 2 +- docs/scripts.md | 2 +- tests/fm-crew-state.test.sh | 27 ++++++++++++++-- tests/fm-record-reconcile.test.sh | 21 +++++++++++-- tests/fm-wake-queue.test.sh | 16 ++++++++-- 7 files changed, 89 insertions(+), 33 deletions(-) diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index b7bf40d4464..80a555d30c3 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -589,7 +589,8 @@ if [ "$HAVE_RUN" = 1 ]; then RUN_DETAIL="run record still active${SEP}headless worker pid=$NM_WORKER_PID has unrecognized identity" ;; *) - RUN_DETAIL="$RUN_DETAIL${SEP}headless worker identity unavailable; run record only" + RUN_STATE=unknown + RUN_DETAIL="run record still active${SEP}headless worker identity unavailable; live/dead state is indeterminate" ;; esac fi diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index 2a953ebed7c..5479c49a3b8 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash # Atomically drain durable watcher wake records, preserve the consumed queue and -# its digest-bound receipt, optionally annotate validated signal status keys -# after raw consumption commits, then assert liveness. +# its digest-bound receipt, reconcile verified terminal status signals, optionally +# annotate validated status keys after raw consumption commits, then assert +# liveness. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -81,18 +82,26 @@ EOF fi } -terminal_wake_present() { # <annotations> - local annotations=$1 event - case "$annotations" in - *"annotations omitted"*|*"[truncated]"*) return 0 ;; - esac - while IFS= read -r event; do - [ -n "$event" ] || continue - if status_is_done "$event" || status_is_awaiting_captain "$event"; then +# A terminal status wake is the ordinary completion path, so reconcile its +# structured row in the same turn rather than waiting for session restart. +# Status keys are structurally mapped and read without following symlinks; queue +# payload prose never supplies a path. Nonterminal signals remain read-only. +reconcile_terminal_status_wakes() { # <deduped-raw-rows> + local rows=$1 epoch seq kind key payload receipt + while IFS=$(printf '\t') read -r epoch seq kind key payload; do + [ "$kind" = signal ] || continue + fm_wake_status_key_map "$key" || continue + fm_wake_latest_event "$STATE/$FM_WAKE_STATUS_KEY" 65536 || continue + if status_is_done "$FM_WAKE_EVENT_LINE" || status_is_awaiting_captain "$FM_WAKE_EVENT_LINE"; then + receipt=$(FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_DATA_OVERRIDE="$DATA" \ + "$FM_RECORD_RECONCILE_BIN") || return 1 + [ -z "$receipt" ] || printf 'record reconciliation: %s\n' "$receipt" return 0 fi - done < <(printf '%s\n' "$annotations" | sed -n 's/^wake annotation: latest wake-EVENT observed at drain, not current state\(; historical \/ not necessarily the triggering event\)\{0,1\}: [A-Za-z0-9._-]*\.status: //p') - return 1 + done <<EOF +$rows +EOF + return 0 } # shellcheck disable=SC2317,SC2329 # Invoked by trap handlers below. @@ -147,15 +156,13 @@ DRAIN_TMP= fm_lock_release "$FM_WAKE_QUEUE_LOCK" DRAIN_LOCK_HELD=false -# Raw output and queue deletion are authoritative. Everything below is -# best-effort and cannot restore, duplicate, hide, or fail the consumed rows. -annotations=$(fm_wake_print_annotations "$RAW_ROWS") || annotations= -if [ -n "$annotations" ]; then - printf '%s\n' "$annotations" -fi -if terminal_wake_present "$annotations"; then - "$FM_RECORD_RECONCILE_BIN" >/dev/null 2>&1 || true -fi +# Raw output, custody, and queue retirement are authoritative. Reconciliation +# below may fail the command visibly but can never restore, duplicate, or hide +# the already receipt-bound consumed rows. Annotation and liveness remain +# best-effort even after reconciliation failure. +reconcile_status=0 +reconcile_terminal_status_wakes "$RAW_ROWS" || reconcile_status=$? +(fm_wake_print_annotations "$RAW_ROWS") || true (print_open_decisions_section) || true assert_watcher_liveness -exit 0 +exit "$reconcile_status" diff --git a/docs/architecture.md b/docs/architecture.md index de4ae5a56d2..d8499897827 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -27,7 +27,7 @@ Its initial normal-mode status signal still surfaces through the no-verb path, w Fresh stale panes use the same current-state read before trusting the status log, so an active run or a proven busy worker outranks an old captain-relevant status-log line left behind before validation. No-change heartbeats are also benign. Absorbed wakes advance their suppression markers, log to `state/.watch-triage.log`, and keep the watcher blocking without a queue record or LLM turn. -After each drain, `fm-wake-drain.sh` first custody-versions the exact consumed queue bytes and writes a digest, byte count, source reference, custody class, and timestamp receipt, then runs the same liveness guard as the supervision scripts, so a lapsed watcher chain surfaces even on a turn that only drains and handles queued wakes. +After each drain, `fm-wake-drain.sh` first custody-versions the exact consumed queue bytes and writes a digest, byte count, source reference, custody class, and timestamp receipt, then reconciles any safely read terminal status signal through `fm-record-reconcile.sh` in that same turn before running the liveness guard, so neither a terminal row nor a lapsed watcher chain waits for session restart. Routine watcher polling, supervision no-ops, elapsed waiting time, and absorbed benign wakes stay silent. A declared external wait trades that silence for one bounded recheck per pause window, so a forgotten pause cannot remain invisible indefinitely. Crew status files are append-only wake-event logs, not current-state fields. diff --git a/docs/scripts.md b/docs/scripts.md index d0368a0c31c..59fcbf10900 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -87,7 +87,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-tasks-axi-lib.sh` | Shared backlog-backend selector and `tasks-axi` compatibility probe | | `fm-quota-axi-lib.sh` | Shared `quota-axi` compatibility floor for the bootstrap diagnostic | | `fm-vendor-auth-probe.sh`| Run one hard-bounded, non-destructive authentication probe of a named vendor CLI and report the fact | -| `fm-wake-drain.sh` | Atomically drain and custody-version queued wakes, emit bounded status annotations and a fleet-wide OPEN DECISIONS section, then assert watcher liveness | +| `fm-wake-drain.sh` | Atomically drain and custody-version queued wakes, reconcile verified terminal status signals, emit bounded annotations and a fleet-wide OPEN DECISIONS section, then assert watcher liveness | | `fm-wake-lib.sh` | Shared durable wake queue, portable locks, and watcher identity/health helpers | | `fm-classify-lib.sh` | Shared wake-classification vocabulary and durable keyed-decision folds and scans | | `fm-send.sh` | Send one verified literal line or supported key through the target's recorded backend | diff --git a/tests/fm-crew-state.test.sh b/tests/fm-crew-state.test.sh index 40155a6b6f0..eef3c460588 100755 --- a/tests/fm-crew-state.test.sh +++ b/tests/fm-crew-state.test.sh @@ -71,6 +71,9 @@ case "${1:-}" in if [ "${1:-}" = --run ]; then printf '%s\n' "${FM_FAKE_AXI_STATUS_RUN:-}" else printf '%s\n' "${FM_FAKE_AXI_STATUS:-}"; fi ;; logs) + if [ "${FM_FAKE_WORKER_LOG_UNAVAILABLE:-0}" != 1 ]; then + printf 'codex started pid=4242\n' + fi printf '%s\n' "${FM_FAKE_CI_LOGS:-}" ;; esac ;; @@ -180,10 +183,11 @@ reset_fakes() { FM_FAKE_HERDR_MISSING=0 FM_FAKE_HERDR_AGENT_STATUS="" FM_FAKE_CI_LOGS="" - FM_FAKE_WORKER_PS_STATE="" + FM_FAKE_WORKER_PS_STATE='S+' + FM_FAKE_WORKER_LOG_UNAVAILABLE=0 export FM_FAKE_AXI_STATUS FM_FAKE_AXI_STATUS_RUN FM_FAKE_RUNS_LIST FM_FAKE_BUSY FM_FAKE_BUSY_TEXT FM_FAKE_TMUX_MISSING export FM_FAKE_HERDR_BUSY FM_FAKE_HERDR_MISSING FM_FAKE_HERDR_AGENT_STATUS FM_FAKE_CI_LOGS - export FM_FAKE_WORKER_PS_STATE + export FM_FAKE_WORKER_PS_STATE FM_FAKE_WORKER_LOG_UNAVAILABLE } # --- run-object fixtures (TOON, as `no-mistakes axi status` emits) ----------- @@ -417,6 +421,24 @@ test_dead_or_suspended_pipeline_worker_is_not_recorded_working() { pass "dead, suspended, and zombie workers never inherit a stale running verdict" } +test_unbound_pipeline_worker_is_indeterminate() { + reset_fakes + local d out + d=$(new_case headless-unbound) + make_repo_on_branch "$d/wt" fm/headless-unbound + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/headless-unbound.meta" "window=fm:fm-headless-unbound" "worktree=$d/wt" "kind=ship" "harness=codex" + FM_FAKE_AXI_STATUS="$(run_running fm/headless-unbound)" + FM_FAKE_CI_LOGS='pipeline log shape without a native worker pid' + FM_FAKE_WORKER_LOG_UNAVAILABLE=1 + out=$(run_crew_state "$d" headless-unbound) + assert_contains "$out" 'state: unknown' \ + "PID-unbound run inherited a live verdict from the stale running ledger" + assert_contains "$out" 'headless worker identity unavailable' \ + "PID-unbound run did not report its process-level evidence limit" + pass "PID-unbound pipeline workers remain indeterminate rather than guessed live or dead" +} + # (b) needs-decision log + a resumed (running/fixing) run = SUPERSEDED test_stale_needs_decision_superseded() { reset_fakes @@ -1371,6 +1393,7 @@ test_missing_run_head_falls_back_to_current_state() { test_active_run_is_authoritative test_headless_pipeline_worker_liveness_overrides_pane_interruption test_dead_or_suspended_pipeline_worker_is_not_recorded_working +test_unbound_pipeline_worker_is_indeterminate test_stale_needs_decision_superseded test_stale_blocked_superseded test_genuine_parked_not_superseded diff --git a/tests/fm-record-reconcile.test.sh b/tests/fm-record-reconcile.test.sh index aa7f64d8582..70be7d8ac39 100755 --- a/tests/fm-record-reconcile.test.sh +++ b/tests/fm-record-reconcile.test.sh @@ -6,10 +6,17 @@ set -u . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" TMP_ROOT=$(fm_test_tmproot fm-record-reconcile) -RECONCILE="$ROOT/bin/fm-record-reconcile.sh" +DRAIN="$ROOT/bin/fm-wake-drain.sh" + +append_status_wake() { # <state-dir> <task-id> + FM_STATE_OVERRIDE="$1" bash -c ' + . "$1" + fm_wake_append signal "$2.status" "signal: $3/$2.status" + ' _ "$ROOT/bin/fm-wake-lib.sh" "$2" "$1" +} test_terminal_row_reconciles_while_unlanded_work_remains() { - local home wt dirty_wt receipt head crew_state + local home wt dirty_wt receipt head guard_root drain_out home="$TMP_ROOT/home" wt="$home/projects/sample-terminal" mkdir -p "$home/data" "$home/state" "$home/projects" "$wt" @@ -61,7 +68,15 @@ esac SH chmod +x "$crew_state" - FM_HOME="$home" FM_CREW_STATE_BIN="$crew_state" "$RECONCILE" >/dev/null || fail "record reconciliation failed" + guard_root="$home/guard-root" + drain_out="$home/drain.out" + mkdir -p "$guard_root" + touch "$home/state/.last-watcher-beat" + append_status_wake "$home/state" sample-terminal || fail "could not queue terminal status wake" + FM_HOME="$home" FM_ROOT_OVERRIDE="$guard_root" "$DRAIN" > "$drain_out" \ + || fail "terminal wake drain and record reconciliation failed" + grep "$(printf '\tsignal\tsample-terminal.status\t')" "$drain_out" >/dev/null \ + || fail "terminal wake did not traverse the ordinary drain path" ! sed -n '/^## In flight/,/^## /p' "$home/data/backlog.md" | grep -F -- '- [ ] sample-terminal -' >/dev/null \ || fail "terminal report still reads in flight" sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- '- [x] sample-terminal -' >/dev/null \ diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index 4386ec1cd86..c6f858d1ffa 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -328,7 +328,7 @@ SH } test_enrichment_caps_and_status_file_failures() { - local dir state out fake_perl_log perl_bin i raw_count annotation_bytes annotation_count oversized_lines perl_reads + local dir state out fake_perl_log perl_bin i raw_count annotation_bytes annotation_count oversized_lines perl_reads terminal_reads dir=$(make_case caps) state="$dir/state" out="$dir/drain.out" @@ -337,7 +337,13 @@ test_enrichment_caps_and_status_file_failures() { cat > "$dir/fakebin/perl" <<'SH' #!/usr/bin/env bash if [ "${1:-}" = -MFcntl=:DEFAULT ]; then - printf 'read\n' >> "$FM_WAKE_ENRICH_PERL_LOG" + previous= + current= + for arg in "$@"; do + previous=$current + current=$arg + done + printf 'read\t%s\n' "$previous" >> "$FM_WAKE_ENRICH_PERL_LOG" fi exec "$FM_WAKE_ENRICH_REAL_PERL" "$@" SH @@ -374,7 +380,11 @@ SH annotation_count=$(grep -c '^wake annotation: latest' "$out" || true) [ "$annotation_count" -lt 9 ] || fail "global cap did not omit any of the nine readable status annotations" perl_reads=$(wc -l < "$fake_perl_log" | tr -d ' ') - [ "$perl_reads" -eq 8 ] || fail "enrichment read cap allowed $perl_reads safe reads instead of 8" + [ "$perl_reads" -eq 9 ] \ + || fail "terminal detection plus capped enrichment used $perl_reads safe reads instead of exactly 9" + terminal_reads=$(awk -F '\t' -v path="$state/huge.status" '$2 == path { count++ } END { print count + 0 }' "$fake_perl_log") + [ "$terminal_reads" -eq 2 ] \ + || fail "terminal status was not read exactly once for reconciliation and once inside the eight-read annotation cap" grep -E '^wake annotation: [1-9][0-9]* annotations omitted \(enrichment read cap\)$' "$out" >/dev/null \ || fail "enrichment read-cap omission marker was not emitted" if grep -E ': (empty|missing|malformed|unreadable)\.status:' "$out" >/dev/null; then From 8dc58690f547d09d56f04c8ae38854278924fc28 Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Sat, 1 Aug 2026 14:01:18 -0600 Subject: [PATCH 05/14] fix: fail closed on watcher syntax checks --- bin/fm-arm-command-policy.mjs | 76 +++++++++++++++++++++--------- docs/arm-pretool-check.md | 7 +-- tests/fm-arm-pretool-check.test.sh | 19 ++++++++ 3 files changed, 78 insertions(+), 24 deletions(-) diff --git a/bin/fm-arm-command-policy.mjs b/bin/fm-arm-command-policy.mjs index a1f30577755..53a1cec5ea5 100755 --- a/bin/fm-arm-command-policy.mjs +++ b/bin/fm-arm-command-policy.mjs @@ -619,47 +619,35 @@ function shellInvocation(position) { const name = basename(position.command.value); if (!["sh", "bash", "zsh"].includes(name)) return null; const words = position.words; - let noexec = false; for (let i = position.index + 1; i < words.length; i += 1) { const option = words[i]; if (/^-[A-Za-z]*c[A-Za-z]*$/.test(option.value)) { - if (option.value.slice(1).includes("n")) noexec = true; let payloadIndex = i + 1; if (words[payloadIndex]?.value === "--") payloadIndex += 1; - return { kind: "command", payload: words[payloadIndex] || null, noexec }; + return { kind: "command", payload: words[payloadIndex] || null }; } if (/^[-+]O$/.test(option.value)) { i += 1; continue; } - if (option.value === "--noexec") { - noexec = true; - continue; - } if (option.value === "--") { const payload = words[i + 1] || null; - return payload ? { kind: "script", payload, noexec } : { kind: "stdin", payload: null, noexec }; - } - if (/^[-+][A-Za-z]+$/.test(option.value)) { - if (option.value.slice(1).includes("n")) noexec = option.value.startsWith("-"); - continue; + return payload ? { kind: "script", payload } : { kind: "stdin", payload: null }; } if (/^[-+]/.test(option.value)) continue; - return { kind: "script", payload: option, noexec }; + return { kind: "script", payload: option }; } - return { kind: "stdin", payload: null, noexec }; + return { kind: "stdin", payload: null }; } function shellHeredocPayloads(tokens, position) { - const shell = shellInvocation(position); - if (shell?.kind !== "stdin" || shell.noexec) return []; + if (shellInvocation(position)?.kind !== "stdin") return []; const heredocs = tokens.filter((token) => token.type === "redir" && token.fd === 0 && typeof token.heredoc === "string"); return heredocs.length === 0 ? [] : [heredocs.at(-1).heredoc]; } function shellHereStringPayloads(tokens, position) { - const shell = shellInvocation(position); - if (shell?.kind !== "stdin" || shell.noexec) return []; + if (shellInvocation(position)?.kind !== "stdin") return []; const payloads = []; for (let i = 0; i < tokens.length; i += 1) { const token = tokens[i]; @@ -706,6 +694,16 @@ function hasDynamicExecutionPayload(position, context) { return false; } +// Treat every statically visible watcher-script word passed to a shell as a +// protected execution candidate, regardless of the options preceding it. This +// deliberately does not model bash/zsh's open-ended option grammar: options +// such as --rcfile and --init-file consume values, so inference from an `n` +// byte is unsafe. decision() carries the sole exact no-execute allow-list. +function shellCarriesProtectedWatcher(position, context) { + if (!position.command || !["sh", "bash", "zsh"].includes(basename(position.command.value))) return false; + return position.words.slice(position.index + 1).some((word) => protectedIdentity(word.value, context.root) === "watch"); +} + function assignmentName(word) { const match = word.value.match(/^([A-Za-z_][A-Za-z0-9_]*)=/); return match ? match[1] : ""; @@ -733,6 +731,16 @@ function nodeHasRedirection(tokens) { return tokens.some((token) => token.type === "redir"); } +function nodeHasProtectedOutputTarget(tokens, context) { + for (let i = 0; i < tokens.length; i += 1) { + const token = tokens[i]; + if (token.type !== "redir" || token.inlineTarget || ![">", ">>", ">&"].includes(token.value)) continue; + const target = tokens[i + 1]; + if (target?.type === "word" && (protectedIdentity(target.value, context.root) || wordReferencesAny(target, context.protectedVariables))) return true; + } + return false; +} + function nodeHasUnsafeSubstitution(tokens) { return tokens.some((token) => token.type === "word" && token.subs.length > 0); } @@ -803,8 +811,8 @@ function analyzeProgram(command, context, depth = 0) { } const shell = shellInvocation(position); - const shellPayload = shell?.kind === "command" && !shell.noexec ? shell.payload : null; - const shellScript = shell?.kind === "script" && !shell.noexec ? shell.payload : null; + const shellPayload = shell?.kind === "command" ? shell.payload : null; + const shellScript = shell?.kind === "script" ? shell.payload : null; const sourceScript = sourcedScript(position); const literalEvalPayload = evalPayload(position); const heredocPayloads = shellHeredocPayloads(tokens, position); @@ -836,6 +844,8 @@ function analyzeProgram(command, context, depth = 0) { const protectedKind = protectedIdentity(executable, context.root); if (hasUnclassifiableProtectedExpansion(position.command, context.root)) unclassifiableProtected = true; const commandName = basename(executable); + const shellProtectedWatcher = shellCarriesProtectedWatcher(position, nodeContext); + nodeNestedProtected ||= shellProtectedWatcher; const args = position.words.slice(position.index + 1); if (commandName === "pkill" && args.some((word) => /fm-watch/.test(word.value) || wordReferencesAny(word, nodeContext.watcherPatterns))) broadKill = true; if (commandName === "kill" && (nodePgrepWatcher || args.some((word) => wordReferencesAny(word, nodeContext.watcherPids)))) broadKill = true; @@ -854,8 +864,10 @@ function analyzeProgram(command, context, depth = 0) { tokens, position, protectedKind, + shellProtectedWatcher, nestedProtected: nodeNestedProtected, redirection: nodeHasRedirection(tokens), + protectedOutputRedirection: nodeHasProtectedOutputTarget(tokens, nodeContext), substitution: nodeHasUnsafeSubstitution(tokens), }); } @@ -880,6 +892,23 @@ function ordinaryWordsOnly(tokens) { return tokens.every((token) => token.type === "word" && token.subs.length === 0); } +// The entire no-execute exception is an allow-list. Only these exact top-level +// forms are accepted: +// bash -n <fm-watch.sh> +// bash --noexec <fm-watch.sh> +// Any wrapper, extra option/argument, separator, substitution, or redirection +// falls through to the normal protected-command denials. +function allowedWatcherSyntaxCheck(analysis, context) { + if (analysis.nodeInfos.length !== 1 || analysis.program.separators.length !== 0) return false; + const info = analysis.nodeInfos[0]; + if (!info.shellProtectedWatcher || info.redirection || info.substitution) return false; + if (!ordinaryWordsOnly(info.tokens) || info.position.prefixAssignments > 0 || info.position.wrappers.length > 0) return false; + if (basename(info.position.command?.value || "") !== "bash") return false; + const args = info.position.words.slice(info.position.index + 1); + if (args.length !== 2 || !["-n", "--noexec"].includes(args[0].value)) return false; + return protectedIdentity(args[1].value, context.root) === "watch"; +} + function setupKind(info, context) { const { tokens, position } = info; if (!ordinaryWordsOnly(tokens) || position.prefixAssignments > 0 || position.wrappers.length > 0) return ""; @@ -920,9 +949,12 @@ function decision(command, root, home) { const analysis = analyzeProgram(command, context); if (analysis.broadKill) return deny("broad-watcher-kill"); if (analysis.error && analysis.protectedFound) return deny("unclassifiable-protected-command"); + // A protected output target is independently unsafe even when the command + // itself is unrelated. Check it before every allow path so `: > watcher` can + // never truncate a script while data-only heredocs remain ordinary data. + if (analysis.nodeInfos?.some((info) => info.protectedOutputRedirection)) return deny("watcher-redirection"); if (!analysis.protectedFound) return { decision: "allow" }; if (analysis.nodeInfos?.some((info) => info.protectedKind === "watch")) return deny("watcher-direct"); - if (analysis.nestedProtected) return deny("watcher-nested"); const separators = analysis.program.separators; if (separators.includes("&") || analysis.nodeInfos.some((info) => info.position.wrappers.includes("nohup")) || analysis.nodeInfos.some((info) => basename(info.position.words[0]?.value || "") === "disown")) { @@ -931,6 +963,8 @@ function decision(command, root, home) { if (separators.includes("|") || separators.includes("|&")) return deny("watcher-pipeline"); if (analysis.nodeInfos.some((info) => info.redirection)) return deny("watcher-redirection"); if (analysis.nodeInfos.some((info) => info.substitution)) return deny("watcher-nested"); + if (allowedWatcherSyntaxCheck(analysis, context)) return { decision: "allow" }; + if (analysis.nestedProtected) return deny("watcher-nested"); if (blessedProgram(analysis, context)) return { decision: "allow" }; if (analysis.nodeInfos.some((info) => info.position.prefixAssignments > 0 || info.position.wrappers.some((wrapper) => wrapper !== "exec"))) { return deny("watcher-nested"); diff --git a/docs/arm-pretool-check.md b/docs/arm-pretool-check.md index 8bca04d9509..b27d2ce2435 100644 --- a/docs/arm-pretool-check.md +++ b/docs/arm-pretool-check.md @@ -75,9 +75,10 @@ This covers statically-visible literal words in command position; opaque dynamic `bin/fm-watch.sh` is protected but is not a blessed entry point. A direct `bin/fm-watch.sh` execution - relative, `<code-root>`-anchored, `$VAR`-prefixed, or `~`-prefixed - always denies with `watcher-direct`, whose reason points the caller at `bin/fm-watch-arm.sh` and `bin/fm-watch-checkpoint.sh`. -Shell no-execute syntax checks such as `bash -n bin/fm-watch.sh`, combined short flags containing `n`, and `bash --noexec bin/fm-watch.sh` are read-only data uses and are allowed. -The exception belongs to the semantic classifier, not the byte prefilter: the same path executed as `bash bin/fm-watch.sh` remains a denied direct watcher launch. -The regression matrix covers both halves through every adapter transport, alongside the full dangerous-command deny corpus, so a syntax-validation allowance cannot weaken execution protection. +The no-execute exception is an explicit allow-list containing only `bash -n <fm-watch.sh>` and `bash --noexec <fm-watch.sh>` as sole top-level commands with no wrapper, extra option or argument, separator, substitution, or redirection. +The exception belongs to the semantic classifier, not the byte prefilter: every other shell invocation carrying the watcher script is denied, including `--rcfile`, `--init-file`, combined flags, and direct execution. +Independently, every redirection on a protected execution and every output redirection targeting a protected script is denied before an allow decision, so a syntax check or unrelated command cannot truncate the script it is checking. +The regression matrix covers the two allowed forms and the executing/destructive counterforms through every adapter transport, alongside the full dangerous-command deny corpus. The same bytes in an argument, comment, assertion, documentation query, Python string, `printf`, or `tmux send-keys` payload are data and do not make the outer command relevant. diff --git a/tests/fm-arm-pretool-check.test.sh b/tests/fm-arm-pretool-check.test.sh index c03869436e7..94869240c7f 100755 --- a/tests/fm-arm-pretool-check.test.sh +++ b/tests/fm-arm-pretool-check.test.sh @@ -65,6 +65,7 @@ matrix_case R18 allow "sh -c 'tmux send-keys -t lab \"bin/fm-watch-arm.sh &\" En matrix_case R19 allow "eval 'printf \"%s\\n\" \"bin/fm-watch-arm.sh &\"'" matrix_case R20 allow 'bash -n bin/fm-watch.sh' matrix_case R21 allow 'sed -n '\''1,20p'\'' bin/fm-watch.sh' +matrix_case R22 allow 'bash --noexec bin/fm-watch.sh' matrix_case D01 deny 'bin/fm-watch-arm.sh &' matrix_case D02 deny 'nohup bin/fm-watch-arm.sh' @@ -125,6 +126,18 @@ matrix_case D56 deny 'for x in 1; do pkill -f fm-watch; done' matrix_case D57 deny 'case x in x) pkill -f fm-watch ;; esac' matrix_case D58 deny 'until false; do kill $(pgrep -f fm-watch); done' matrix_case D59 deny 'bash bin/fm-watch.sh' +matrix_case D60 deny 'bash --rcfile -n bin/fm-watch.sh' +matrix_case D61 deny 'bash --init-file -n bin/fm-watch.sh' +matrix_case D62 deny 'sh --rcfile -n bin/fm-watch.sh' +matrix_case D63 deny 'bash -n bin/fm-watch.sh > bin/fm-watch.sh' +matrix_case D64 deny ': > bin/fm-watch.sh' +matrix_case D65 deny 'bash -n bin/fm-watch.sh > /tmp/fm-watch-syntax.out' +matrix_case D66 deny 'bash -in bin/fm-watch.sh' +matrix_case D67 deny 'bash -n -- bin/fm-watch.sh' +matrix_case D68 deny 'bash -n bin/fm-watch.sh &' +matrix_case D69 deny 'bash -o xtrace bin/fm-watch.sh' +matrix_case D70 deny 'bash --rcfile /dev/null bin/fm-watch.sh' +matrix_case D71 deny 'target=bin/fm-watch.sh; : > "$target"' matrix_case E01 allow "bin/fm-watch-checkpoint.sh --seconds '180;still-one-arg'" matrix_case E02 allow "bin/fm-watch-checkpoint.sh --label 'fm-watch-arm.sh; literal argument'" @@ -237,7 +250,13 @@ test_direct_policy_contract() { assert_policy direct-watch-expanded $'deny\twatcher-direct' '$FM_HOME/bin/fm-watch.sh' assert_policy direct-watch-safe-shape $'deny\twatcher-direct' 'cd /tmp; bin/fm-watch.sh' assert_policy direct-watch-syntax-read allow 'bash -n bin/fm-watch.sh' + assert_policy direct-watch-noexec-read allow 'bash --noexec bin/fm-watch.sh' assert_policy direct-watch-script-wrapper $'deny\twatcher-nested' 'bash bin/fm-watch.sh' + assert_policy direct-watch-rcfile-value $'deny\twatcher-nested' 'bash --rcfile -n bin/fm-watch.sh' + assert_policy direct-watch-init-file-value $'deny\twatcher-nested' 'bash --init-file -n bin/fm-watch.sh' + assert_policy direct-watch-syntax-truncate $'deny\twatcher-redirection' 'bash -n bin/fm-watch.sh > bin/fm-watch.sh' + assert_policy direct-watch-leading-truncate $'deny\twatcher-redirection' ': > bin/fm-watch.sh' + assert_policy direct-watch-variable-truncate $'deny\twatcher-redirection' 'target=bin/fm-watch.sh; : > "$target"' heredoc_data=$'cat <<\'EOF\'\nbin/fm-watch-arm.sh &\nEOF' heredoc_watcher=$'bin/fm-watch-arm.sh <<\'EOF\'\ndata only\nEOF' assert_policy direct-heredoc-data allow "$heredoc_data" From 25be547769994acfb1b39fcd8aee903a61741f0a Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Sat, 1 Aug 2026 18:41:55 -0600 Subject: [PATCH 06/14] docs: disclose bash -s watcher guard gap --- docs/arm-pretool-check.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/arm-pretool-check.md b/docs/arm-pretool-check.md index b27d2ce2435..cf52deb914f 100644 --- a/docs/arm-pretool-check.md +++ b/docs/arm-pretool-check.md @@ -76,7 +76,11 @@ This covers statically-visible literal words in command position; opaque dynamic A direct `bin/fm-watch.sh` execution - relative, `<code-root>`-anchored, `$VAR`-prefixed, or `~`-prefixed - always denies with `watcher-direct`, whose reason points the caller at `bin/fm-watch-arm.sh` and `bin/fm-watch-checkpoint.sh`. The no-execute exception is an explicit allow-list containing only `bash -n <fm-watch.sh>` and `bash --noexec <fm-watch.sh>` as sole top-level commands with no wrapper, extra option or argument, separator, substitution, or redirection. -The exception belongs to the semantic classifier, not the byte prefilter: every other shell invocation carrying the watcher script is denied, including `--rcfile`, `--init-file`, combined flags, and direct execution. +The exception belongs to the semantic classifier, not the byte prefilter: every other statically recognized shell invocation carrying the watcher script is denied, including `--rcfile`, `--init-file`, combined flags, and direct execution. +The known [`bash -s <operand>` gap](https://github.com/kunchenguid/firstmate/issues/1489) makes shell-invocation analysis mistake the operand for a script path, so a heredoc or here-string payload is not inspected and can execute `bin/fm-watch.sh`. +The guard therefore does not cover every payload-hiding shell form. +The three discovered holes - `--rcfile`, `--init-file`, and `-s` - are all argument-consuming or payload-hiding invocation forms, and that class is demonstrably not exhausted. +That open-ended class is why the syntax exception remains an allow-list: a fourth unmodeled form that reaches protected-invocation classification must fall through to denial instead of acquiring read-only status. Independently, every redirection on a protected execution and every output redirection targeting a protected script is denied before an allow decision, so a syntax check or unrelated command cannot truncate the script it is checking. The regression matrix covers the two allowed forms and the executing/destructive counterforms through every adapter transport, alongside the full dangerous-command deny corpus. From 97e4828ee2388e69760538471c99c845e16746cf Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Mon, 3 Aug 2026 18:37:01 -0600 Subject: [PATCH 07/14] fix: keep completed-but-unlanded rows in flight Done means merged, never green-open. Reconciliation previously transitioned a terminal producer row from In flight to Done as soon as the worker reported complete with a clean tree, which released dependents against work that had not landed. Record the terminal-retention evidence (exact head, clean tree) on the row instead and leave it In flight. Teardown, which separately refuses unlanded work, stays the only path that retires it. The write is idempotent so repeated terminal wakes do not churn the row body. Also restores the reconcile-binary and current-state seams the two lanes' tests depend on, so the resumed-worker refusal is still exercised through the drain path. --- bin/fm-record-reconcile.sh | 27 ++++++++++++++------ bin/fm-session-start.sh | 7 +++--- tests/fm-record-reconcile.test.sh | 41 ++++++++++++++++++++----------- 3 files changed, 49 insertions(+), 26 deletions(-) diff --git a/bin/fm-record-reconcile.sh b/bin/fm-record-reconcile.sh index 8af70a77c06..9aab5a09dcc 100755 --- a/bin/fm-record-reconcile.sh +++ b/bin/fm-record-reconcile.sh @@ -5,10 +5,13 @@ # # Usage: fm-record-reconcile.sh # -# A `done:` producer or a producer declaring the complete-only -# `awaiting-captain:` state while still In flight moves to Done with --no-prune -# and an explicit retained-lifecycle note. Its endpoint metadata and worktree -# remain; teardown separately refuses unlanded work. Scout rows move only after +# Done means merged, never green-open: a terminal producer report proves work is +# complete, not landed, so no row here is transitioned. A `done:` producer, or a +# producer declaring the complete-only `awaiting-captain:` state, keeps its In +# flight row and gains durable terminal-retention evidence (exact head, clean +# tree) recorded both on the row and in the receipt. Because the row stays In +# flight it releases no dependent, and only teardown - which separately refuses +# unlanded work - retires it after landing. Scout rows are annotated only after # the report exists and the decision-hold completion gate verifies. Missing # metadata and orphan metadata are accounted in the receipt and preserved. set -eu @@ -100,10 +103,18 @@ if fm_tasks_axi_compatible; then continue fi fi - (cd "$FM_HOME" && tasks-axi 'done' "$id" --no-prune \ - --note "producer terminal status reconciled at $TERMINAL_HEAD; endpoint metadata and worktree retained pending landing, independent gate, or captain answer") >/dev/null \ - || { append_event terminal-unreconciled "$id" 'tasks-axi transition failed; row preserved'; continue; } - append_event terminal-retained "$id" "moved from In flight to Done at head=$TERMINAL_HEAD with tree=clean; metadata and worktree retained" + # Done means merged. A terminal producer report proves the work is complete, + # not that it landed, so the row stays In flight and keeps releasing nothing + # until teardown confirms landing. Record the retention evidence in place + # instead of transitioning, and keep the write idempotent so repeated + # terminal wakes do not churn the row's body. + marker="terminal-retained: head=$TERMINAL_HEAD tree=clean" + if ! (cd "$FM_HOME" && tasks-axi show "$id" --full 2>/dev/null) | grep -Fq "$marker"; then + (cd "$FM_HOME" && tasks-axi update "$id" --archive-body --body \ + "$marker; producer reported complete and its worktree is clean. Endpoint metadata and worktree are retained, and this row stays In flight - releasing no dependent - until teardown confirms the work landed.") >/dev/null \ + || { append_event terminal-unreconciled "$id" 'tasks-axi retention annotation failed; row preserved'; continue; } + fi + append_event terminal-retained "$id" "retained in flight at head=$TERMINAL_HEAD with tree=clean; metadata and worktree retained pending landing" done < <(in_flight_ids) fi diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index 0f290d1cc0d..fcf84e3695a 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -34,9 +34,10 @@ # secondmate liveness, pending remote handoff retry, # X-mode artifact writes, fleet sync) also run only when # locked. -# 3. record reconcile - retires surfaced terminal rows into Done without -# pruning and receipts every metadata/backlog mismatch. -# It only runs while locked. +# 3. record reconcile - records terminal-retention evidence on surfaced +# terminal rows, which stay In flight until teardown +# confirms landing, and receipts every metadata/backlog +# mismatch. It only runs while locked. # 4. wake-drain - mutates the durable wake queue, so it also only runs # when locked. # 5. context digest - data/projects.md, data/secondmates.md, data/captain.md, diff --git a/tests/fm-record-reconcile.test.sh b/tests/fm-record-reconcile.test.sh index 70be7d8ac39..64165544860 100755 --- a/tests/fm-record-reconcile.test.sh +++ b/tests/fm-record-reconcile.test.sh @@ -7,6 +7,7 @@ set -u TMP_ROOT=$(fm_test_tmproot fm-record-reconcile) DRAIN="$ROOT/bin/fm-wake-drain.sh" +RECONCILE="$ROOT/bin/fm-record-reconcile.sh" append_status_wake() { # <state-dir> <task-id> FM_STATE_OVERRIDE="$1" bash -c ' @@ -15,8 +16,8 @@ append_status_wake() { # <state-dir> <task-id> ' _ "$ROOT/bin/fm-wake-lib.sh" "$2" "$1" } -test_terminal_row_reconciles_while_unlanded_work_remains() { - local home wt dirty_wt receipt head guard_root drain_out +test_terminal_row_retained_in_flight_until_it_lands() { + local home wt dirty_wt receipt head guard_root drain_out crew_state home="$TMP_ROOT/home" wt="$home/projects/sample-terminal" mkdir -p "$home/data" "$home/state" "$home/projects" "$wt" @@ -58,6 +59,9 @@ EOF fm_write_meta "$home/state/orphan-meta.meta" \ 'window=test:fm-orphan-meta' "worktree=$home/projects/orphan" 'project=sample' 'kind=ship' printf 'done: historical status with no metadata row\n' > "$home/state/orphan-status.status" + # Reconciliation refuses any row whose worker has resumed, so the drain path + # needs a current-state source. Export it so it survives into the reconcile + # child the drain spawns. crew_state="$home/crew-state" cat > "$crew_state" <<'SH' #!/usr/bin/env bash @@ -73,16 +77,23 @@ SH mkdir -p "$guard_root" touch "$home/state/.last-watcher-beat" append_status_wake "$home/state" sample-terminal || fail "could not queue terminal status wake" - FM_HOME="$home" FM_ROOT_OVERRIDE="$guard_root" "$DRAIN" > "$drain_out" \ + FM_HOME="$home" FM_ROOT_OVERRIDE="$guard_root" FM_CREW_STATE_BIN="$crew_state" \ + "$DRAIN" > "$drain_out" \ || fail "terminal wake drain and record reconciliation failed" grep "$(printf '\tsignal\tsample-terminal.status\t')" "$drain_out" >/dev/null \ || fail "terminal wake did not traverse the ordinary drain path" - ! sed -n '/^## In flight/,/^## /p' "$home/data/backlog.md" | grep -F -- '- [ ] sample-terminal -' >/dev/null \ - || fail "terminal report still reads in flight" - sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- '- [x] sample-terminal -' >/dev/null \ - || fail "terminal report was not reconciled into Done" - sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- '- [x] captain-wait -' >/dev/null \ - || fail "complete awaiting-captain report was not reconciled into Done" + # Done means merged. A complete-but-unlanded producer keeps its In flight row + # so it releases no dependent, and carries durable retention evidence instead. + sed -n '/^## In flight/,/^## /p' "$home/data/backlog.md" | grep -F -- '- [ ] sample-terminal -' >/dev/null \ + || fail "terminal report was released from In flight before it landed" + sed -n '/^## In flight/,/^## /p' "$home/data/backlog.md" | grep -F -- '- [ ] captain-wait -' >/dev/null \ + || fail "complete awaiting-captain report was released from In flight before it landed" + ! sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- 'sample-terminal' >/dev/null \ + || fail "unlanded terminal report was recorded as Done" + ! sed -n '/^## Done/,$p' "$home/data/backlog.md" | grep -F -- 'captain-wait' >/dev/null \ + || fail "unlanded awaiting-captain report was recorded as Done" + grep -F "terminal-retained: head=$head tree=clean" "$home/data/backlog.md" >/dev/null \ + || fail "retained terminal row carries no terminal-retention evidence" grep -F -- '- [ ] dirty-terminal -' "$home/data/backlog.md" >/dev/null \ || fail "dirty terminal row was retired despite uncommitted evidence" assert_present "$dirty_wt/uncommitted.txt" "reconciliation erased uncommitted evidence" @@ -94,16 +105,16 @@ SH assert_present "$home/state/orphan-meta.meta" "reconciliation erased orphan metadata evidence" receipt=$(find "$home/data/record-reconciliation" -type f -name '*.receipt' | head -1) assert_present "$receipt" "reconciliation wrote no durable inventory receipt" - assert_grep $'terminal-retained\tsample-terminal' "$receipt" "receipt omitted the reconciled terminal row" - assert_grep $'terminal-retained\tcaptain-wait' "$receipt" "receipt omitted the reconciled captain-wait row" + assert_grep $'terminal-retained\tsample-terminal' "$receipt" "receipt omitted the retained terminal row" + assert_grep $'terminal-retained\tcaptain-wait' "$receipt" "receipt omitted the retained captain-wait row" assert_grep $'terminal-unreconciled\tdirty-terminal' "$receipt" "receipt omitted the dirty terminal refusal" assert_grep $'missing-meta\tmissing-meta' "$receipt" "receipt omitted metadata-count drift" assert_grep $'orphan-meta\torphan-meta' "$receipt" "receipt omitted the orphan metadata" assert_grep $'orphan-status\torphan-status' "$receipt" "receipt omitted the orphan status record" - assert_grep 'in_flight_count=2' "$receipt" "receipt omitted the reconciled in-flight count" + assert_grep 'in_flight_count=4' "$receipt" "receipt omitted the retained in-flight count" assert_grep 'metadata_count=4' "$receipt" "receipt omitted the retained metadata count" - assert_grep 'metadata_minus_in_flight=2' "$receipt" "receipt omitted explicit metadata-count drift" - pass "terminal row reconciles without cleaning unlanded work or erasing drift evidence" + assert_grep 'metadata_minus_in_flight=0' "$receipt" "receipt omitted explicit metadata-count drift" + pass "terminal row is retained in flight with retention evidence until it lands" } test_resumed_worker_overrides_stale_terminal_event() { @@ -141,5 +152,5 @@ SH pass "active current state overrides a stale terminal event" } -test_terminal_row_reconciles_while_unlanded_work_remains +test_terminal_row_retained_in_flight_until_it_lands test_resumed_worker_overrides_stale_terminal_event From 265821254f07bcca9e6b8eee1c9fda9f8c9fe001 Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Mon, 3 Aug 2026 19:16:34 -0600 Subject: [PATCH 08/14] no-mistakes(review): Fix worker identity and capacity hold races --- bin/fm-crew-state.sh | 45 +++++++----- bin/fm-model-capacity-hold.sh | 94 ++++++++++++++++++++----- bin/fm-process-identity-lib.sh | 38 ++++++++++ bin/fm-wake-lib.sh | 43 +----------- tests/fm-crew-state.test.sh | 69 +++++++++++++----- tests/fm-model-capacity-hold.test.sh | 101 +++++++++++++++++++++++++++ 6 files changed, 295 insertions(+), 95 deletions(-) create mode 100644 bin/fm-process-identity-lib.sh diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index 80a555d30c3..b791aa8a373 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -35,11 +35,11 @@ # checks" from "checks green, waiting on merge" (see nm_ci_checks_state) - # a ci-step log-tail check overrides working -> done once checks read # green, so a green PR is never silently read as still-validating. -# For an active full run whose step log names its native worker PID, the -# process is checked independently of the pane: live remains working, -# stopped/suspended or absent never inherits the stale running row, and an -# unrecognized reused PID becomes unknown. Pane interruption cannot stop or -# prove the state of this detached worker. +# For an active full run whose step log binds its native worker PID to an +# exact process-birth identity, the process is checked independently of the +# pane: live remains working, while stopped, suspended, absent, or replaced +# never inherits the stale running row. A PID without that binding leaves +# the authoritative run-step mapping unchanged. # 3. Reconcile the status log: if its last line says needs-decision/blocked but # the run-step shows the run moved on, the log is deterministically stale and # is flagged superseded. A genuinely parked run plus a needs-decision log @@ -113,6 +113,9 @@ if [ -z "$WT" ] || [ ! -d "$WT" ]; then emit unknown none "worktree gone (torn down?)" fi +# shellcheck source=bin/fm-process-identity-lib.sh +. "$SCRIPT_DIR/fm-process-identity-lib.sh" + # --- status log ------------------------------------------------------------ # Last non-empty status line, and its leading verb (the word before the colon). @@ -315,13 +318,12 @@ nm_ci_checks_state() { } # Resolve native no-mistakes step-worker health when the current step log exposes -# a start PID. Absence of such an identity is an explicit unsupported shape and -# leaves the run-record mapping unchanged; a named PID is never allowed to do so -# when the process is dead, suspended, or has been reused by an unrelated command. +# a PID bound to the exact process identity captured when it started. An absent +# binding leaves the run-record mapping unchanged. NM_WORKER_HEALTH=unavailable NM_WORKER_PID= nm_headless_worker_health() { - local run_id step log started pid related ps_out stat command + local run_id step log started pid expected_identity current_identity related ps_out stat NM_WORKER_HEALTH=unavailable NM_WORKER_PID= run_id=$(strip_quotes "$(nm_field id)") @@ -333,11 +335,13 @@ nm_headless_worker_health() { log=$(nm_run axi logs --step "$step" --run "$run_id") [ -n "$log" ] || return 0 started=$(printf '%s\n' "$log" \ - | grep -Ei '(codex|claude|opencode|grok|kimi|pi|agent).*started.*pid[=: ]+[0-9]+' \ + | grep -Ei '(codex|claude|opencode|grok|kimi|pi|agent).*started.*pid[=: ]+[0-9]+.*pid-identity-hex=[0-9a-f]+' \ | tail -1) [ -n "$started" ] || return 0 pid=$(printf '%s\n' "$started" | sed -nE 's/.*pid[=: ]+([0-9]+).*/\1/p') case "$pid" in ''|*[!0-9]*) return 0 ;; esac + expected_identity=$(printf '%s\n' "$started" | sed -nE 's/.*pid-identity-hex=([0-9a-fA-F]+).*/\1/p' | tr 'A-F' 'a-f') + case "$expected_identity" in ''|*[!0-9a-f]*) return 0 ;; esac NM_WORKER_PID=$pid related=$(printf '%s\n' "$log" \ | grep -Ei "((started|exited|stopped).*(pid[=: ]+)?$pid)|((pid[=: ]+)?$pid.*(started|exited|stopped))" \ @@ -345,16 +349,20 @@ nm_headless_worker_health() { case "$related" in *exited*|*stopped*) NM_WORKER_HEALTH=dead; return 0 ;; esac - ps_out=$(ps -p "$pid" -o stat= -o command= 2>/dev/null) || { + current_identity=$(fm_pid_identity "$pid") || { NM_WORKER_HEALTH=dead return 0 } - stat=$(printf '%s\n' "$ps_out" | awk 'NR == 1 { print $1 }') - command=${ps_out#*"$stat"} - if ! printf '%s\n' "$command" | grep -Eiq 'codex|claude|opencode|grok|kimi|(^|[ /])pi([ /]|$)|agent'; then - NM_WORKER_HEALTH=unknown + current_identity=$(printf '%s' "$current_identity" | od -An -v -tx1 | tr -d '[:space:]') + if [ "$current_identity" != "$expected_identity" ]; then + NM_WORKER_HEALTH=reused return 0 fi + ps_out=$(ps -p "$pid" -o stat= -o command= 2>/dev/null) || { + NM_WORKER_HEALTH=dead + return 0 + } + stat=$(printf '%s\n' "$ps_out" | awk 'NR == 1 { print $1 }') case "$stat" in *Z*) NM_WORKER_HEALTH=dead ;; *T*) NM_WORKER_HEALTH=suspended ;; @@ -584,13 +592,16 @@ if [ "$HAVE_RUN" = 1 ]; then RUN_STATE=blocked RUN_DETAIL="run record still active${SEP}headless pipeline worker suspended pid=$NM_WORKER_PID" ;; + reused) + RUN_STATE=blocked + RUN_DETAIL="run record still active${SEP}headless pipeline worker identity replaced pid=$NM_WORKER_PID" + ;; unknown) RUN_STATE=unknown RUN_DETAIL="run record still active${SEP}headless worker pid=$NM_WORKER_PID has unrecognized identity" ;; *) - RUN_STATE=unknown - RUN_DETAIL="run record still active${SEP}headless worker identity unavailable; live/dead state is indeterminate" + RUN_DETAIL="$RUN_DETAIL${SEP}headless worker identity unavailable; run record only" ;; esac fi diff --git a/bin/fm-model-capacity-hold.sh b/bin/fm-model-capacity-hold.sh index 3988670c0ef..b5594800fe7 100755 --- a/bin/fm-model-capacity-hold.sh +++ b/bin/fm-model-capacity-hold.sh @@ -22,11 +22,14 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" MARKER="$STATE/.model-capacity-hold" RECEIPTS="$DATA/model-capacity-holds" +LIFECYCLE_LOCK="$STATE/.model-capacity-hold.lock" # shellcheck source=bin/fm-custody-lib.sh . "$SCRIPT_DIR/fm-custody-lib.sh" # shellcheck source=bin/fm-model-capacity-hold-lib.sh . "$SCRIPT_DIR/fm-model-capacity-hold-lib.sh" +# shellcheck source=bin/fm-wake-lib.sh +. "$SCRIPT_DIR/fm-wake-lib.sh" fail() { printf 'fm-model-capacity-hold: %s\n' "$*" >&2 @@ -59,6 +62,38 @@ write_atomic() { # <path> <exclusive:0|1>, content on stdin mv "$tmp" "$path" || { rm -f "$tmp"; fail "could not publish $path"; } } +LOCK_HELD=0 +release_lifecycle_lock() { + [ "$LOCK_HELD" -eq 1 ] || return 0 + fm_lock_release "$LIFECYCLE_LOCK" + LOCK_HELD=0 +} + +acquire_lifecycle_lock() { + fm_lock_acquire_wait "$LIFECYCLE_LOCK" + LOCK_HELD=1 + trap release_lifecycle_lock EXIT +} + +registration_matches() { # <receipt> <id> <reason> <dispatch> + local receipt=$1 id=$2 reason=$3 dispatch=$4 schema receipt_id receipt_reason receipt_dispatch + [ -f "$receipt" ] && [ ! -L "$receipt" ] || return 1 + schema=$(fm_model_capacity_hold_value "$receipt" schema) || return 1 + receipt_id=$(fm_model_capacity_hold_value "$receipt" hold_id) || return 1 + receipt_reason=$(fm_model_capacity_hold_value "$receipt" reason) || return 1 + receipt_dispatch=$(fm_model_capacity_hold_value "$receipt" dispatch_ref) || return 1 + [ "$schema" = fm-model-capacity-hold.v1 ] \ + && [ "$receipt_id" = "$id" ] \ + && [ "$receipt_reason" = "$reason" ] \ + && [ "$receipt_dispatch" = "$dispatch" ] +} + +publish_marker() { # <receipt> + local receipt=$1 staged="$MARKER.tmp.${BASHPID:-$$}" + cp "$receipt" "$staged" || fail "could not stage active marker" + mv "$staged" "$MARKER" || fail "could not publish active marker" +} + command_register() { local id reason dispatch timestamp receipt existing_id id=${1:-} @@ -78,6 +113,7 @@ command_register() { validate_line reason "$reason" validate_line dispatch-ref "$dispatch" mkdir -p "$STATE" "$RECEIPTS" + acquire_lifecycle_lock if [ -e "$MARKER" ]; then existing_id=$(fm_model_capacity_hold_value "$MARKER" hold_id) || fail "active hold marker is malformed" [ "$existing_id" = "$id" ] || fail "another model-capacity hold is active: $existing_id" @@ -86,15 +122,23 @@ command_register() { fi timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ') receipt="$RECEIPTS/$id.registered" - { - printf 'schema=fm-model-capacity-hold.v1\n' - printf 'hold_id=%s\n' "$id" - printf 'reason=%s\n' "$reason" - printf 'dispatch_ref=%s\n' "$dispatch" - printf 'registered_at=%s\n' "$timestamp" - } | write_atomic "$receipt" 1 - cp "$receipt" "$MARKER.tmp.${BASHPID:-$$}" || fail "could not stage active marker" - mv "$MARKER.tmp.${BASHPID:-$$}" "$MARKER" || fail "could not publish active marker" + if [ -e "$receipt" ]; then + [ ! -e "$RECEIPTS/$id.released" ] && [ ! -e "$RECEIPTS/$id.active-marker-retired" ] \ + || fail "hold registration was already released: $id" + registration_matches "$receipt" "$id" "$reason" "$dispatch" \ + || fail "existing registration receipt does not match retry: $receipt" + else + { + printf 'schema=fm-model-capacity-hold.v1\n' + printf 'hold_id=%s\n' "$id" + printf 'reason=%s\n' "$reason" + printf 'dispatch_ref=%s\n' "$dispatch" + printf 'registered_at=%s\n' "$timestamp" + } | write_atomic "$receipt" 1 + fi + publish_marker "$receipt" + release_lifecycle_lock + trap - EXIT printf 'registered: %s\n' "$id" } @@ -107,7 +151,7 @@ command_status() { } command_release() { - local id authority active_id digest timestamp registered receipt + local id authority active_id digest timestamp registered receipt registration_digest id=${1:-} authority= [ "$#" -ge 1 ] || fail "release requires a hold id" @@ -121,23 +165,37 @@ command_release() { done validate_id "$id" [ -f "$authority" ] && [ ! -L "$authority" ] || fail "authority file must be a regular non-symlink file" + mkdir -p "$STATE" "$RECEIPTS" + acquire_lifecycle_lock active_id=$(fm_model_capacity_hold_value "$MARKER" hold_id) || fail "no valid active model-capacity hold" [ "$active_id" = "$id" ] || fail "active hold is $active_id, not $id" registered="$RECEIPTS/$id.registered" [ -f "$registered" ] && [ ! -L "$registered" ] || fail "registration receipt is missing" digest=$(fm_custody_sha256 "$authority") || fail "could not digest release authority" + registration_digest=$(fm_custody_sha256 "$registered") || fail "could not digest registration receipt" timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ') receipt="$RECEIPTS/$id.released" - { - printf 'schema=fm-model-capacity-hold-release.v1\n' - printf 'hold_id=%s\n' "$id" - printf 'registration_sha256=%s\n' "$(fm_custody_sha256 "$registered")" - printf 'authority_path=%s\n' "$authority" - printf 'authority_sha256=%s\n' "$digest" - printf 'released_at=%s\n' "$timestamp" - } | write_atomic "$receipt" 1 + if [ -e "$receipt" ]; then + [ "$(fm_model_capacity_hold_value "$receipt" schema || true)" = fm-model-capacity-hold-release.v1 ] \ + && [ "$(fm_model_capacity_hold_value "$receipt" hold_id || true)" = "$id" ] \ + && [ "$(fm_model_capacity_hold_value "$receipt" registration_sha256 || true)" = "$registration_digest" ] \ + && [ "$(fm_model_capacity_hold_value "$receipt" authority_path || true)" = "$authority" ] \ + && [ "$(fm_model_capacity_hold_value "$receipt" authority_sha256 || true)" = "$digest" ] \ + || fail "existing release receipt does not match retry: $receipt" + else + { + printf 'schema=fm-model-capacity-hold-release.v1\n' + printf 'hold_id=%s\n' "$id" + printf 'registration_sha256=%s\n' "$registration_digest" + printf 'authority_path=%s\n' "$authority" + printf 'authority_sha256=%s\n' "$digest" + printf 'released_at=%s\n' "$timestamp" + } | write_atomic "$receipt" 1 + fi mv "$MARKER" "$RECEIPTS/$id.active-marker-retired" \ || fail "release receipt exists but active marker could not be retired" + release_lifecycle_lock + trap - EXIT printf 'released: %s\n' "$id" } diff --git a/bin/fm-process-identity-lib.sh b/bin/fm-process-identity-lib.sh new file mode 100644 index 00000000000..a1b10aa91d7 --- /dev/null +++ b/bin/fm-process-identity-lib.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Portable exact process-birth identity shared by process-bound lifecycle checks. + +# Resolved once at source time because identity checks run inside short confirm +# and attach polls on platforms where each process fork is measurable. +_FM_UNAME=$(uname 2>/dev/null || echo unknown) + +fm_pid_identity() { + local pid=$1 out proc_root stat_line starttime cmdline_hex identity_key + local -a stat_fields + case "$pid" in + ''|*[!0-9]*) return 1 ;; + esac + proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc} + # Prefer a Linux-compatible /proc when present: stat field 22 is immune to + # wall-clock steps, and the full command line distinguishes a tick collision. + if [ -r "$proc_root/$pid/stat" ] && [ -r "$proc_root/$pid/cmdline" ]; then + stat_line=$(cat "$proc_root/$pid/stat" 2>/dev/null) || return 1 + # After the final comm delimiter, array index 19 is proc stat field 22. + read -r -a stat_fields <<< "${stat_line##*)}" + [ "${#stat_fields[@]}" -ge 20 ] || return 1 + starttime=${stat_fields[19]} + case "$starttime" in + ''|*[!0-9]*) return 1 ;; + esac + cmdline_hex=$(od -An -v -tx1 "$proc_root/$pid/cmdline" 2>/dev/null | tr -d '[:space:]') || return 1 + [ -n "$cmdline_hex" ] || return 1 + identity_key=proc-starttime + [ "$_FM_UNAME" != Linux ] || identity_key=linux-starttime + printf '%s=%s cmdline-hex=%s\n' "$identity_key" "$starttime" "$cmdline_hex" + return 0 + fi + # Pin the locale so the fallback identity is stable when it is recorded and + # later compared under different ambient locale settings. + out=$(LC_ALL=C ps -p "$pid" -o lstart= -o command= 2>/dev/null) || return 1 + [ -n "$out" ] || return 1 + printf '%s\n' "$out" | sed 's/^[[:space:]]*//' +} diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 3af1642b319..72c0793b019 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -9,10 +9,8 @@ STATE="${FM_STATE_OVERRIDE:-${STATE:-$FM_HOME/state}}" FM_WAKE_QUEUE="${FM_WAKE_QUEUE:-$STATE/.wake-queue}" FM_WAKE_QUEUE_LOCK="${FM_WAKE_QUEUE_LOCK:-$STATE/.wake-queue.lock}" FM_LOCK_STALE_AFTER="${FM_LOCK_STALE_AFTER:-2}" -# Resolved once at source time: fm_pid_identity and fm_path_mtime run inside 0.2s -# confirm and 0.5s attach polls, and forking uname per call is a measurable cost on -# the platform (Git Bash/MSYS) that already pays the highest fork price. -_FM_UNAME=$(uname 2>/dev/null || echo unknown) +# shellcheck source=bin/fm-process-identity-lib.sh +. "$FM_WAKE_LIB_DIR/fm-process-identity-lib.sh" mkdir -p "$STATE" fm_current_pid() { @@ -27,43 +25,6 @@ fm_pid_alive() { kill -0 "$pid" 2>/dev/null } -fm_pid_identity() { - local pid=$1 out proc_root stat_line starttime cmdline_hex identity_key - local -a stat_fields - case "$pid" in - ''|*[!0-9]*) return 1 ;; - esac - proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc} - # Prefer a Linux-compatible /proc when present: stat field 22 (starttime, clock ticks since boot) is - # immune to the wall-clock steps that re-render the ps lstart fallback's date - # (observed as WSL2 btime drift) and would evict a live watcher; combining the - # full NUL-separated cmdline keeps PID reuse a mismatch even on a tick collision. - # Git Bash/MSYS exposes these compatible files but its Cygwin ps rejects the - # portable fallback's -o fields, so capability detection must not key on uname. - if [ -r "$proc_root/$pid/stat" ] && [ -r "$proc_root/$pid/cmdline" ]; then - stat_line=$(cat "$proc_root/$pid/stat" 2>/dev/null) || return 1 - # After the final comm delimiter, array index 19 is proc stat field 22. - read -r -a stat_fields <<< "${stat_line##*)}" - [ "${#stat_fields[@]}" -ge 20 ] || return 1 - starttime=${stat_fields[19]} - case "$starttime" in - ''|*[!0-9]*) return 1 ;; - esac - cmdline_hex=$(od -An -v -tx1 "$proc_root/$pid/cmdline" 2>/dev/null | tr -d '[:space:]') || return 1 - [ -n "$cmdline_hex" ] || return 1 - identity_key=proc-starttime - [ "$_FM_UNAME" != Linux ] || identity_key=linux-starttime - printf '%s=%s cmdline-hex=%s\n' "$identity_key" "$starttime" "$cmdline_hex" - return 0 - fi - # Pin LC_ALL=C so lstart's date format is locale-invariant: the identity is - # written under one locale but re-read under the machine's ambient locale, which - # would otherwise mismatch on a non-C locale (e.g. ko_KR) and reject a live watcher. - out=$(LC_ALL=C ps -p "$pid" -o lstart= -o command= 2>/dev/null) || return 1 - [ -n "$out" ] || return 1 - printf '%s\n' "$out" | sed 's/^[[:space:]]*//' -} - fm_path_mtime() { if [ "$_FM_UNAME" = Darwin ]; then stat -f %m "$1" 2>/dev/null diff --git a/tests/fm-crew-state.test.sh b/tests/fm-crew-state.test.sh index eef3c460588..5c0a78bfbe1 100755 --- a/tests/fm-crew-state.test.sh +++ b/tests/fm-crew-state.test.sh @@ -71,9 +71,7 @@ case "${1:-}" in if [ "${1:-}" = --run ]; then printf '%s\n' "${FM_FAKE_AXI_STATUS_RUN:-}" else printf '%s\n' "${FM_FAKE_AXI_STATUS:-}"; fi ;; logs) - if [ "${FM_FAKE_WORKER_LOG_UNAVAILABLE:-0}" != 1 ]; then - printf 'codex started pid=4242\n' - fi + printf '%s\n' "${FM_FAKE_WORKER_LOG:-}" printf '%s\n' "${FM_FAKE_CI_LOGS:-}" ;; esac ;; @@ -130,7 +128,11 @@ SH set -u if printf '%s\n' "$*" | grep -F -- '-p 4242' >/dev/null; then [ -n "${FM_FAKE_WORKER_PS_STATE:-}" ] || exit 1 - printf '%s codex native worker\n' "$FM_FAKE_WORKER_PS_STATE" + if printf '%s\n' "$*" | grep -F -- 'lstart=' >/dev/null; then + printf '%s\n' "${FM_FAKE_WORKER_IDENTITY:-Mon Aug 3 12:00:00 2026 codex native worker}" + else + printf '%s codex native worker\n' "$FM_FAKE_WORKER_PS_STATE" + fi exit 0 fi exec /bin/ps "$@" @@ -184,10 +186,17 @@ reset_fakes() { FM_FAKE_HERDR_AGENT_STATUS="" FM_FAKE_CI_LOGS="" FM_FAKE_WORKER_PS_STATE='S+' - FM_FAKE_WORKER_LOG_UNAVAILABLE=0 + FM_FAKE_WORKER_IDENTITY='Mon Aug 3 12:00:00 2026 codex native worker' + FM_FAKE_WORKER_LOG="" export FM_FAKE_AXI_STATUS FM_FAKE_AXI_STATUS_RUN FM_FAKE_RUNS_LIST FM_FAKE_BUSY FM_FAKE_BUSY_TEXT FM_FAKE_TMUX_MISSING export FM_FAKE_HERDR_BUSY FM_FAKE_HERDR_MISSING FM_FAKE_HERDR_AGENT_STATUS FM_FAKE_CI_LOGS - export FM_FAKE_WORKER_PS_STATE FM_FAKE_WORKER_LOG_UNAVAILABLE + export FM_FAKE_WORKER_PS_STATE FM_FAKE_WORKER_IDENTITY FM_FAKE_WORKER_LOG +} + +worker_start_log() { + local identity_hex + identity_hex=$(printf '%s' "$FM_FAKE_WORKER_IDENTITY" | od -An -v -tx1 | tr -d '[:space:]') + printf 'codex started pid=4242 pid-identity-hex=%s\n' "$identity_hex" } # --- run-object fixtures (TOON, as `no-mistakes axi status` emits) ----------- @@ -382,7 +391,7 @@ test_headless_pipeline_worker_liveness_overrides_pane_interruption() { make_fakebin "$d" >/dev/null fm_write_meta "$d/state/headless-live.meta" "window=fm:fm-headless-live" "worktree=$d/wt" "kind=ship" "harness=codex" FM_FAKE_AXI_STATUS="$(run_running fm/headless-live)" - FM_FAKE_CI_LOGS='codex started pid=4242' + FM_FAKE_WORKER_LOG=$(worker_start_log) FM_FAKE_WORKER_PS_STATE='S+' FM_FAKE_TMUX_MISSING=1 out=$(run_crew_state "$d" headless-live) @@ -400,7 +409,7 @@ test_dead_or_suspended_pipeline_worker_is_not_recorded_working() { make_fakebin "$d" >/dev/null fm_write_meta "$d/state/headless-dead.meta" "window=fm:fm-headless-dead" "worktree=$d/wt" "kind=ship" "harness=codex" FM_FAKE_AXI_STATUS="$(run_running fm/headless-dead)" - FM_FAKE_CI_LOGS='codex started pid=4242' + FM_FAKE_WORKER_LOG=$(worker_start_log) FM_FAKE_WORKER_PS_STATE='' out=$(run_crew_state "$d" headless-dead) assert_not_contains "$out" 'state: working' "dead step worker inherited the stale running row" @@ -421,22 +430,43 @@ test_dead_or_suspended_pipeline_worker_is_not_recorded_working() { pass "dead, suspended, and zombie workers never inherit a stale running verdict" } -test_unbound_pipeline_worker_is_indeterminate() { +test_reused_pipeline_worker_pid_is_not_live() { reset_fakes local d out + d=$(new_case headless-reused) + make_repo_on_branch "$d/wt" fm/headless-reused + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/headless-reused.meta" "window=fm:fm-headless-reused" "worktree=$d/wt" "kind=ship" "harness=codex" + FM_FAKE_AXI_STATUS="$(run_running fm/headless-reused)" + FM_FAKE_WORKER_LOG=$(worker_start_log) + FM_FAKE_WORKER_IDENTITY='Mon Aug 3 12:01:00 2026 codex unrelated replacement' + out=$(run_crew_state "$d" headless-reused) + assert_contains "$out" 'state: blocked' "reused allowed-family PID was reported live" + assert_contains "$out" 'headless pipeline worker identity replaced pid=4242' \ + "reused PID did not contradict the bound worker identity" + pass "reused worker PID cannot impersonate the recorded process birth" +} + +test_unbound_pipeline_worker_preserves_authoritative_state() { + reset_fakes + local d out fixture expected d=$(new_case headless-unbound) make_repo_on_branch "$d/wt" fm/headless-unbound make_fakebin "$d" >/dev/null fm_write_meta "$d/state/headless-unbound.meta" "window=fm:fm-headless-unbound" "worktree=$d/wt" "kind=ship" "harness=codex" - FM_FAKE_AXI_STATUS="$(run_running fm/headless-unbound)" - FM_FAKE_CI_LOGS='pipeline log shape without a native worker pid' - FM_FAKE_WORKER_LOG_UNAVAILABLE=1 - out=$(run_crew_state "$d" headless-unbound) - assert_contains "$out" 'state: unknown' \ - "PID-unbound run inherited a live verdict from the stale running ledger" - assert_contains "$out" 'headless worker identity unavailable' \ - "PID-unbound run did not report its process-level evidence limit" - pass "PID-unbound pipeline workers remain indeterminate rather than guessed live or dead" + FM_FAKE_CI_LOGS='pipeline log shape without a native worker identity' + for fixture in running fixing ci; do + case "$fixture" in + running) FM_FAKE_AXI_STATUS="$(run_running fm/headless-unbound)"; expected='validating (running)' ;; + fixing) FM_FAKE_AXI_STATUS="$(run_fixing fm/headless-unbound)"; expected='validating (fixing)' ;; + ci) FM_FAKE_AXI_STATUS="$(run_top_level_ci fm/headless-unbound)"; expected='ci running' ;; + esac + out=$(run_crew_state "$d" headless-unbound) + assert_contains "$out" 'state: working' "PID-unbound $fixture run lost its authoritative working state" + assert_contains "$out" "$expected" "PID-unbound $fixture run lost its run-step detail" + assert_not_contains "$out" 'state: unknown' "PID-unbound $fixture run manufactured indeterminacy" + done + pass "PID-unbound running, fixing, and CI states remain authoritatively working" } # (b) needs-decision log + a resumed (running/fixing) run = SUPERSEDED @@ -1393,7 +1423,8 @@ test_missing_run_head_falls_back_to_current_state() { test_active_run_is_authoritative test_headless_pipeline_worker_liveness_overrides_pane_interruption test_dead_or_suspended_pipeline_worker_is_not_recorded_working -test_unbound_pipeline_worker_is_indeterminate +test_reused_pipeline_worker_pid_is_not_live +test_unbound_pipeline_worker_preserves_authoritative_state test_stale_needs_decision_superseded test_stale_blocked_superseded test_genuine_parked_not_superseded diff --git a/tests/fm-model-capacity-hold.test.sh b/tests/fm-model-capacity-hold.test.sh index c5696545fcb..f573e37ae76 100755 --- a/tests/fm-model-capacity-hold.test.sh +++ b/tests/fm-model-capacity-hold.test.sh @@ -84,5 +84,106 @@ test_registered_hold_blocks_spawn_before_fleet_mutation() { pass "registered model-capacity hold blocks spawn before fleet mutation" } +test_registration_serializes_competing_hold_ids() { + local home fakebin first_pid second_pid first_rc second_rc wait_count + home="$TMP_ROOT/race-home" + make_home "$home" + fakebin="$home/hold-fakebin" + mkdir -p "$fakebin" + cat > "$fakebin/cp" <<'SH' +#!/usr/bin/env bash +set -u +case "${2:-}" in + *.model-capacity-hold.tmp.*) + : > "$FM_HOLD_CP_ENTERED" + while [ ! -e "$FM_HOLD_CP_RELEASE" ]; do sleep 0.02; done + ;; +esac +exec /bin/cp "$@" +SH + chmod +x "$fakebin/cp" + first_rc="$home/first.rc" + second_rc="$home/second.rc" + PATH="$fakebin:$PATH" FM_HOME="$home" FM_HOLD_CP_ENTERED="$home/first.entered" \ + FM_HOLD_CP_RELEASE="$home/cp.release" "$HOLD" register reserve-first \ + --reason 'first reservation' --dispatch-ref 'dispatch first' >/dev/null 2>&1 & + first_pid=$! + wait_count=0 + while [ ! -e "$home/first.entered" ] && [ "$wait_count" -lt 100 ]; do + sleep 0.02 + wait_count=$((wait_count + 1)) + done + [ -e "$home/first.entered" ] || fail "first registration never reached marker publication" + PATH="$fakebin:$PATH" FM_HOME="$home" FM_HOLD_CP_ENTERED="$home/second.entered" \ + FM_HOLD_CP_RELEASE="$home/cp.release" "$HOLD" register reserve-second \ + --reason 'second reservation' --dispatch-ref 'dispatch second' >/dev/null 2>&1 & + second_pid=$! + sleep 0.2 + assert_absent "$home/data/model-capacity-holds/reserve-second.registered" \ + "competing registration published a receipt before the active lifecycle completed" + : > "$home/cp.release" + wait "$first_pid"; printf '%s\n' "$?" > "$first_rc" + if wait "$second_pid"; then printf '0\n' > "$second_rc"; else printf '%s\n' "$?" > "$second_rc"; fi + [ "$(cat "$first_rc")" -eq 0 ] || fail "first serialized registration failed" + [ "$(cat "$second_rc")" -ne 0 ] || fail "competing hold id also registered successfully" + assert_grep 'hold_id=reserve-first' "$home/state/.model-capacity-hold" \ + "serialized registration did not retain the first active hold" + pass "competing hold registrations serialize behind one active marker" +} + +test_matching_orphan_registration_is_repaired() { + local home receipt out + home="$TMP_ROOT/orphan-home" + make_home "$home" + receipt="$home/data/model-capacity-holds/reserve-orphan.registered" + mkdir -p "${receipt%/*}" + { + printf 'schema=fm-model-capacity-hold.v1\n' + printf 'hold_id=reserve-orphan\n' + printf 'reason=interrupted reservation\n' + printf 'dispatch_ref=dispatch orphan\n' + printf 'registered_at=2026-08-03T12:00:00Z\n' + } > "$receipt" + out=$(FM_HOME="$home" "$HOLD" register reserve-orphan \ + --reason 'interrupted reservation' --dispatch-ref 'dispatch orphan') \ + || fail "matching orphan registration was not repairable" + assert_contains "$out" 'registered: reserve-orphan' "orphan repair did not report success" + cmp -s "$receipt" "$home/state/.model-capacity-hold" \ + || fail "orphan repair did not publish the exact durable receipt" + pass "matching interrupted registration receipt repairs the active marker" +} + +test_matching_orphan_release_is_repaired() { + local home authority registered receipt registration_digest authority_digest + home="$TMP_ROOT/release-orphan-home" + make_home "$home" + authority="$home/release-authority.txt" + printf 'Captain released the interrupted hold.\n' > "$authority" + FM_HOME="$home" "$HOLD" register reserve-release \ + --reason 'release interruption' --dispatch-ref 'dispatch release' >/dev/null \ + || fail "could not prepare interrupted release" + registered="$home/data/model-capacity-holds/reserve-release.registered" + receipt="$home/data/model-capacity-holds/reserve-release.released" + registration_digest=$(shasum -a 256 "$registered" | awk '{print $1}') + authority_digest=$(shasum -a 256 "$authority" | awk '{print $1}') + { + printf 'schema=fm-model-capacity-hold-release.v1\n' + printf 'hold_id=reserve-release\n' + printf 'registration_sha256=%s\n' "$registration_digest" + printf 'authority_path=%s\n' "$authority" + printf 'authority_sha256=%s\n' "$authority_digest" + printf 'released_at=2026-08-03T12:01:00Z\n' + } > "$receipt" + FM_HOME="$home" "$HOLD" release reserve-release --authority-file "$authority" >/dev/null \ + || fail "matching interrupted release was not repairable" + assert_absent "$home/state/.model-capacity-hold" "repaired release left the active marker in place" + assert_present "$home/data/model-capacity-holds/reserve-release.active-marker-retired" \ + "repaired release did not retire the active marker" + pass "matching interrupted release receipt retires the active marker" +} + test_registered_hold_blocks_send_until_digest_bound_release test_registered_hold_blocks_spawn_before_fleet_mutation +test_registration_serializes_competing_hold_ids +test_matching_orphan_registration_is_repaired +test_matching_orphan_release_is_repaired From f86f0c9a514b3396c753f35b2011e9129e5e7070 Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Mon, 3 Aug 2026 19:33:42 -0600 Subject: [PATCH 09/14] no-mistakes(review): Document PID containment and secure hold retries --- AGENTS.md | 3 +- bin/fm-crew-state.sh | 90 +------------------------- bin/fm-model-capacity-hold.sh | 6 +- bin/fm-process-identity-lib.sh | 38 ----------- bin/fm-wake-lib.sh | 43 ++++++++++++- docs/architecture.md | 7 ++- tests/fm-crew-state.test.sh | 94 +--------------------------- tests/fm-model-capacity-hold.test.sh | 51 +++++++++++++++ 8 files changed, 105 insertions(+), 227 deletions(-) delete mode 100644 bin/fm-process-identity-lib.sh diff --git a/AGENTS.md b/AGENTS.md index 50bb4c2c603..f7c38e1399a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -330,7 +330,6 @@ Resume fleet supervision immediately after the decision lands. Judge validation by the current-code-matched run step through `bin/fm-crew-state.sh`, not by shell liveness or the last status event. Running, fixing, or CI states remain working; parked approval or fix-review states require the worker to follow the active gate help; passed or checks-passed is done; failed or cancelled is failed. -Pane lifecycle and headless pipeline lifecycle are independent: an interrupt or exit affects only the pane, so a bound live native worker keeps the task working, while a dead or suspended bound native worker contradicts a stale `running` ledger and must be reconciled before the task is called live. A worker hand-editing, committing, aborting, or restarting during an active validation run duplicates pipeline ownership outside the supersession sequence above; steer it back to the gate response flow. The worker reports the PR when CI first becomes green rather than waiting for merge monitoring to finish. @@ -378,7 +377,7 @@ An `awaiting-captain:` event means completed work is retained for an unbounded c Handle actionable wakes as follows: 1. For `signal:`, read the listed event lines first, then reconcile current state only where action depends on it. -2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. Never infer that a detached pipeline stopped from a pane interrupt or infer that a dead bound native worker is live from a `running` row. +2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. Never infer that a detached pipeline stopped from a pane interrupt. 3. For `check:`, act on the named poll result, including merges, X-mode events, and process-to-event source results. 4. For `heartbeat:`, review the whole fleet from the structured fleet view, reconcile suspicious tasks and PR state, update the backlog, and never report an unchanged fleet as progress. diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index b791aa8a373..30cc2272b65 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -35,11 +35,8 @@ # checks" from "checks green, waiting on merge" (see nm_ci_checks_state) - # a ci-step log-tail check overrides working -> done once checks read # green, so a green PR is never silently read as still-validating. -# For an active full run whose step log binds its native worker PID to an -# exact process-birth identity, the process is checked independently of the -# pane: live remains working, while stopped, suspended, absent, or replaced -# never inherits the stale running row. A PID without that binding leaves -# the authoritative run-step mapping unchanged. +# Native worker PIDs are not birth-bound and do not override this mapping; +# docs/architecture.md owns the resulting reuse boundary and closure requirement. # 3. Reconcile the status log: if its last line says needs-decision/blocked but # the run-step shows the run moved on, the log is deterministically stale and # is flagged superseded. A genuinely parked run plus a needs-decision log @@ -113,9 +110,6 @@ if [ -z "$WT" ] || [ ! -d "$WT" ]; then emit unknown none "worktree gone (torn down?)" fi -# shellcheck source=bin/fm-process-identity-lib.sh -. "$SCRIPT_DIR/fm-process-identity-lib.sh" - # --- status log ------------------------------------------------------------ # Last non-empty status line, and its leading verb (the word before the colon). @@ -317,58 +311,6 @@ nm_ci_checks_state() { esac } -# Resolve native no-mistakes step-worker health when the current step log exposes -# a PID bound to the exact process identity captured when it started. An absent -# binding leaves the run-record mapping unchanged. -NM_WORKER_HEALTH=unavailable -NM_WORKER_PID= -nm_headless_worker_health() { - local run_id step log started pid expected_identity current_identity related ps_out stat - NM_WORKER_HEALTH=unavailable - NM_WORKER_PID= - run_id=$(strip_quotes "$(nm_field id)") - [ -n "$run_id" ] || return 0 - step=$(printf '%s\n' "$RUN_OUT" \ - | awk -F, '$2 ~ /^[[:space:]]*"?(running|fixing)"?[[:space:]]*$/ { gsub(/^[[:space:]]+|[[:space:]]+$/, "", $1); found=$1 } END { print found }') - [ -n "$step" ] || step=$(strip_quotes "$(nm_field status)") - [ -n "$step" ] || return 0 - log=$(nm_run axi logs --step "$step" --run "$run_id") - [ -n "$log" ] || return 0 - started=$(printf '%s\n' "$log" \ - | grep -Ei '(codex|claude|opencode|grok|kimi|pi|agent).*started.*pid[=: ]+[0-9]+.*pid-identity-hex=[0-9a-f]+' \ - | tail -1) - [ -n "$started" ] || return 0 - pid=$(printf '%s\n' "$started" | sed -nE 's/.*pid[=: ]+([0-9]+).*/\1/p') - case "$pid" in ''|*[!0-9]*) return 0 ;; esac - expected_identity=$(printf '%s\n' "$started" | sed -nE 's/.*pid-identity-hex=([0-9a-fA-F]+).*/\1/p' | tr 'A-F' 'a-f') - case "$expected_identity" in ''|*[!0-9a-f]*) return 0 ;; esac - NM_WORKER_PID=$pid - related=$(printf '%s\n' "$log" \ - | grep -Ei "((started|exited|stopped).*(pid[=: ]+)?$pid)|((pid[=: ]+)?$pid.*(started|exited|stopped))" \ - | tail -1) - case "$related" in - *exited*|*stopped*) NM_WORKER_HEALTH=dead; return 0 ;; - esac - current_identity=$(fm_pid_identity "$pid") || { - NM_WORKER_HEALTH=dead - return 0 - } - current_identity=$(printf '%s' "$current_identity" | od -An -v -tx1 | tr -d '[:space:]') - if [ "$current_identity" != "$expected_identity" ]; then - NM_WORKER_HEALTH=reused - return 0 - fi - ps_out=$(ps -p "$pid" -o stat= -o command= 2>/dev/null) || { - NM_WORKER_HEALTH=dead - return 0 - } - stat=$(printf '%s\n' "$ps_out" | awk 'NR == 1 { print $1 }') - case "$stat" in - *Z*) NM_WORKER_HEALTH=dead ;; - *T*) NM_WORKER_HEALTH=suspended ;; - *) NM_WORKER_HEALTH=live ;; - esac -} # Coarse fallback for cross-branch attribution. `no-mistakes axi status` (bare) # reports the active-or-most-recent run for the CURRENT branch when one # exists, else falls back to some other branch's run purely as informational @@ -578,34 +520,6 @@ if [ "$HAVE_RUN" = 1 ]; then fi fi - if [ "$RUN_STATE" = working ] && [ "$RUN_SOURCE" = full ]; then - nm_headless_worker_health - case "$NM_WORKER_HEALTH" in - live) - RUN_DETAIL="$RUN_DETAIL${SEP}headless pipeline worker live pid=$NM_WORKER_PID; pane state is independent" - ;; - dead) - RUN_STATE=blocked - RUN_DETAIL="run record still active${SEP}headless pipeline worker dead pid=$NM_WORKER_PID" - ;; - suspended) - RUN_STATE=blocked - RUN_DETAIL="run record still active${SEP}headless pipeline worker suspended pid=$NM_WORKER_PID" - ;; - reused) - RUN_STATE=blocked - RUN_DETAIL="run record still active${SEP}headless pipeline worker identity replaced pid=$NM_WORKER_PID" - ;; - unknown) - RUN_STATE=unknown - RUN_DETAIL="run record still active${SEP}headless worker pid=$NM_WORKER_PID has unrecognized identity" - ;; - *) - RUN_DETAIL="$RUN_DETAIL${SEP}headless worker identity unavailable; run record only" - ;; - esac - fi - # Reconcile the status log. A needs-decision/blocked log line that the run-step # has moved past (anything but a genuinely parked run) is deterministically # stale: the gate resolved and the run resumed or finished. diff --git a/bin/fm-model-capacity-hold.sh b/bin/fm-model-capacity-hold.sh index b5594800fe7..999ea15e2ac 100755 --- a/bin/fm-model-capacity-hold.sh +++ b/bin/fm-model-capacity-hold.sh @@ -114,14 +114,18 @@ command_register() { validate_line dispatch-ref "$dispatch" mkdir -p "$STATE" "$RECEIPTS" acquire_lifecycle_lock + receipt="$RECEIPTS/$id.registered" if [ -e "$MARKER" ]; then existing_id=$(fm_model_capacity_hold_value "$MARKER" hold_id) || fail "active hold marker is malformed" [ "$existing_id" = "$id" ] || fail "another model-capacity hold is active: $existing_id" + registration_matches "$MARKER" "$id" "$reason" "$dispatch" \ + || fail "active hold metadata does not match retry: $id" + registration_matches "$receipt" "$id" "$reason" "$dispatch" \ + || fail "active hold registration receipt does not match retry: $receipt" printf 'registered: %s already active\n' "$id" return 0 fi timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ') - receipt="$RECEIPTS/$id.registered" if [ -e "$receipt" ]; then [ ! -e "$RECEIPTS/$id.released" ] && [ ! -e "$RECEIPTS/$id.active-marker-retired" ] \ || fail "hold registration was already released: $id" diff --git a/bin/fm-process-identity-lib.sh b/bin/fm-process-identity-lib.sh deleted file mode 100644 index a1b10aa91d7..00000000000 --- a/bin/fm-process-identity-lib.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# Portable exact process-birth identity shared by process-bound lifecycle checks. - -# Resolved once at source time because identity checks run inside short confirm -# and attach polls on platforms where each process fork is measurable. -_FM_UNAME=$(uname 2>/dev/null || echo unknown) - -fm_pid_identity() { - local pid=$1 out proc_root stat_line starttime cmdline_hex identity_key - local -a stat_fields - case "$pid" in - ''|*[!0-9]*) return 1 ;; - esac - proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc} - # Prefer a Linux-compatible /proc when present: stat field 22 is immune to - # wall-clock steps, and the full command line distinguishes a tick collision. - if [ -r "$proc_root/$pid/stat" ] && [ -r "$proc_root/$pid/cmdline" ]; then - stat_line=$(cat "$proc_root/$pid/stat" 2>/dev/null) || return 1 - # After the final comm delimiter, array index 19 is proc stat field 22. - read -r -a stat_fields <<< "${stat_line##*)}" - [ "${#stat_fields[@]}" -ge 20 ] || return 1 - starttime=${stat_fields[19]} - case "$starttime" in - ''|*[!0-9]*) return 1 ;; - esac - cmdline_hex=$(od -An -v -tx1 "$proc_root/$pid/cmdline" 2>/dev/null | tr -d '[:space:]') || return 1 - [ -n "$cmdline_hex" ] || return 1 - identity_key=proc-starttime - [ "$_FM_UNAME" != Linux ] || identity_key=linux-starttime - printf '%s=%s cmdline-hex=%s\n' "$identity_key" "$starttime" "$cmdline_hex" - return 0 - fi - # Pin the locale so the fallback identity is stable when it is recorded and - # later compared under different ambient locale settings. - out=$(LC_ALL=C ps -p "$pid" -o lstart= -o command= 2>/dev/null) || return 1 - [ -n "$out" ] || return 1 - printf '%s\n' "$out" | sed 's/^[[:space:]]*//' -} diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 72c0793b019..3af1642b319 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -9,8 +9,10 @@ STATE="${FM_STATE_OVERRIDE:-${STATE:-$FM_HOME/state}}" FM_WAKE_QUEUE="${FM_WAKE_QUEUE:-$STATE/.wake-queue}" FM_WAKE_QUEUE_LOCK="${FM_WAKE_QUEUE_LOCK:-$STATE/.wake-queue.lock}" FM_LOCK_STALE_AFTER="${FM_LOCK_STALE_AFTER:-2}" -# shellcheck source=bin/fm-process-identity-lib.sh -. "$FM_WAKE_LIB_DIR/fm-process-identity-lib.sh" +# Resolved once at source time: fm_pid_identity and fm_path_mtime run inside 0.2s +# confirm and 0.5s attach polls, and forking uname per call is a measurable cost on +# the platform (Git Bash/MSYS) that already pays the highest fork price. +_FM_UNAME=$(uname 2>/dev/null || echo unknown) mkdir -p "$STATE" fm_current_pid() { @@ -25,6 +27,43 @@ fm_pid_alive() { kill -0 "$pid" 2>/dev/null } +fm_pid_identity() { + local pid=$1 out proc_root stat_line starttime cmdline_hex identity_key + local -a stat_fields + case "$pid" in + ''|*[!0-9]*) return 1 ;; + esac + proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc} + # Prefer a Linux-compatible /proc when present: stat field 22 (starttime, clock ticks since boot) is + # immune to the wall-clock steps that re-render the ps lstart fallback's date + # (observed as WSL2 btime drift) and would evict a live watcher; combining the + # full NUL-separated cmdline keeps PID reuse a mismatch even on a tick collision. + # Git Bash/MSYS exposes these compatible files but its Cygwin ps rejects the + # portable fallback's -o fields, so capability detection must not key on uname. + if [ -r "$proc_root/$pid/stat" ] && [ -r "$proc_root/$pid/cmdline" ]; then + stat_line=$(cat "$proc_root/$pid/stat" 2>/dev/null) || return 1 + # After the final comm delimiter, array index 19 is proc stat field 22. + read -r -a stat_fields <<< "${stat_line##*)}" + [ "${#stat_fields[@]}" -ge 20 ] || return 1 + starttime=${stat_fields[19]} + case "$starttime" in + ''|*[!0-9]*) return 1 ;; + esac + cmdline_hex=$(od -An -v -tx1 "$proc_root/$pid/cmdline" 2>/dev/null | tr -d '[:space:]') || return 1 + [ -n "$cmdline_hex" ] || return 1 + identity_key=proc-starttime + [ "$_FM_UNAME" != Linux ] || identity_key=linux-starttime + printf '%s=%s cmdline-hex=%s\n' "$identity_key" "$starttime" "$cmdline_hex" + return 0 + fi + # Pin LC_ALL=C so lstart's date format is locale-invariant: the identity is + # written under one locale but re-read under the machine's ambient locale, which + # would otherwise mismatch on a non-C locale (e.g. ko_KR) and reject a live watcher. + out=$(LC_ALL=C ps -p "$pid" -o lstart= -o command= 2>/dev/null) || return 1 + [ -n "$out" ] || return 1 + printf '%s\n' "$out" | sed 's/^[[:space:]]*//' +} + fm_path_mtime() { if [ "$_FM_UNAME" = Darwin ]; then stat -f %m "$1" 2>/dev/null diff --git a/docs/architecture.md b/docs/architecture.md index d8499897827..ac9d89be1b7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -33,9 +33,10 @@ A declared external wait trades that silence for one bounded recheck per pause w Crew status files are append-only wake-event logs, not current-state fields. Because of that, a per-wake read of only the latest line can bury an earlier still-open `needs-decision`/`blocked` under later unrelated appends; `fm-wake-drain.sh` prints a separate, fleet-wide OPEN DECISIONS section on every drain (including the empty-queue path session-start relies on), built from `fm-classify-lib.sh`'s `status_open_decisions` fold so the buried decision keeps surfacing until it is explicitly resolved. `bin/fm-crew-state.sh <id>` is the cheap current-state read for an actionable heartbeat review: it attributes a no-mistakes run, active or terminal, only when it matches the crew's branch and current code identity, then keeps that run-step authoritative even if the pane has closed. -For a running step with a recorded native-agent PID in the current step log, that process identity must still be live and unsuspended; a pane interrupt does not stop it, and a dead or suspended bound PID is reported as a pipeline contradiction rather than working. -Older or unsupported run shapes without a verifiable PID remain explicitly indeterminate at the process layer instead of borrowing pane liveness. -The script header owns the exact run-head ancestry rules. +No-mistakes currently records native-agent worker PIDs without a process-birth identity, so `fm-crew-state.sh` does not use those PIDs to override an authoritative working run-step. +If a recorded worker dies without an exit marker and that PID is reused by Codex, Claude, or another allowed-family agent process, the stale run can still be reported as working. +Closing this boundary requires the no-mistakes launcher to emit an exact process-birth identity and a compatible minimum version that Firstmate can require; that producer change is outside this repository. +The script header owns the exact run-head ancestry rules and points here for the worker-PID containment boundary. During no-mistakes' `ci` monitor phase, it also reads the ci step log tail because `axi status` reports both "still waiting on checks" and "checks green, waiting on merge" as `ci,running`. The most recent recognized ci log marker wins, so checks-green monitoring reports done while a later re-arm, failed-check, or issue marker returns the crew to working. Only when no matching run exists does it consult semantic busy state; exact busy reports working, exact idle permits fallback to a status-log event whose verb maps to a recognized run-state, and unknown or a dead pane stays unknown instead of trusting a stale log. diff --git a/tests/fm-crew-state.test.sh b/tests/fm-crew-state.test.sh index 5c0a78bfbe1..25d689a450e 100755 --- a/tests/fm-crew-state.test.sh +++ b/tests/fm-crew-state.test.sh @@ -71,7 +71,6 @@ case "${1:-}" in if [ "${1:-}" = --run ]; then printf '%s\n' "${FM_FAKE_AXI_STATUS_RUN:-}" else printf '%s\n' "${FM_FAKE_AXI_STATUS:-}"; fi ;; logs) - printf '%s\n' "${FM_FAKE_WORKER_LOG:-}" printf '%s\n' "${FM_FAKE_CI_LOGS:-}" ;; esac ;; @@ -123,21 +122,7 @@ case "${1:-}" in esac exit 0 SH - cat > "$fb/ps" <<'SH' -#!/usr/bin/env bash -set -u -if printf '%s\n' "$*" | grep -F -- '-p 4242' >/dev/null; then - [ -n "${FM_FAKE_WORKER_PS_STATE:-}" ] || exit 1 - if printf '%s\n' "$*" | grep -F -- 'lstart=' >/dev/null; then - printf '%s\n' "${FM_FAKE_WORKER_IDENTITY:-Mon Aug 3 12:00:00 2026 codex native worker}" - else - printf '%s codex native worker\n' "$FM_FAKE_WORKER_PS_STATE" - fi - exit 0 -fi -exec /bin/ps "$@" -SH - chmod +x "$fb/no-mistakes" "$fb/tmux" "$fb/herdr" "$fb/ps" + chmod +x "$fb/no-mistakes" "$fb/tmux" "$fb/herdr" printf '%s\n' "$fb" } @@ -185,18 +170,8 @@ reset_fakes() { FM_FAKE_HERDR_MISSING=0 FM_FAKE_HERDR_AGENT_STATUS="" FM_FAKE_CI_LOGS="" - FM_FAKE_WORKER_PS_STATE='S+' - FM_FAKE_WORKER_IDENTITY='Mon Aug 3 12:00:00 2026 codex native worker' - FM_FAKE_WORKER_LOG="" export FM_FAKE_AXI_STATUS FM_FAKE_AXI_STATUS_RUN FM_FAKE_RUNS_LIST FM_FAKE_BUSY FM_FAKE_BUSY_TEXT FM_FAKE_TMUX_MISSING export FM_FAKE_HERDR_BUSY FM_FAKE_HERDR_MISSING FM_FAKE_HERDR_AGENT_STATUS FM_FAKE_CI_LOGS - export FM_FAKE_WORKER_PS_STATE FM_FAKE_WORKER_IDENTITY FM_FAKE_WORKER_LOG -} - -worker_start_log() { - local identity_hex - identity_hex=$(printf '%s' "$FM_FAKE_WORKER_IDENTITY" | od -An -v -tx1 | tr -d '[:space:]') - printf 'codex started pid=4242 pid-identity-hex=%s\n' "$identity_hex" } # --- run-object fixtures (TOON, as `no-mistakes axi status` emits) ----------- @@ -383,70 +358,6 @@ test_active_run_is_authoritative() { pass "active run-step is authoritative" } -test_headless_pipeline_worker_liveness_overrides_pane_interruption() { - reset_fakes - local d out - d=$(new_case headless-live) - make_repo_on_branch "$d/wt" fm/headless-live - make_fakebin "$d" >/dev/null - fm_write_meta "$d/state/headless-live.meta" "window=fm:fm-headless-live" "worktree=$d/wt" "kind=ship" "harness=codex" - FM_FAKE_AXI_STATUS="$(run_running fm/headless-live)" - FM_FAKE_WORKER_LOG=$(worker_start_log) - FM_FAKE_WORKER_PS_STATE='S+' - FM_FAKE_TMUX_MISSING=1 - out=$(run_crew_state "$d" headless-live) - assert_contains "$out" 'state: working' "live detached worker was recorded halted with its pane gone" - assert_contains "$out" 'headless pipeline worker live pid=4242' \ - "run-step did not distinguish the detached worker from its interrupted pane" - pass "live headless pipeline worker remains working independently of pane interruption" -} - -test_dead_or_suspended_pipeline_worker_is_not_recorded_working() { - reset_fakes - local d out - d=$(new_case headless-dead) - make_repo_on_branch "$d/wt" fm/headless-dead - make_fakebin "$d" >/dev/null - fm_write_meta "$d/state/headless-dead.meta" "window=fm:fm-headless-dead" "worktree=$d/wt" "kind=ship" "harness=codex" - FM_FAKE_AXI_STATUS="$(run_running fm/headless-dead)" - FM_FAKE_WORKER_LOG=$(worker_start_log) - FM_FAKE_WORKER_PS_STATE='' - out=$(run_crew_state "$d" headless-dead) - assert_not_contains "$out" 'state: working' "dead step worker inherited the stale running row" - assert_contains "$out" 'headless pipeline worker dead pid=4242' \ - "dead step worker was not distinguished from the run record" - - FM_FAKE_WORKER_PS_STATE='T' - out=$(run_crew_state "$d" headless-dead) - assert_not_contains "$out" 'state: working' "suspended step worker inherited the stale running row" - assert_contains "$out" 'headless pipeline worker suspended pid=4242' \ - "suspended step worker was not distinguished from a live worker" - - FM_FAKE_WORKER_PS_STATE='Z+' - out=$(run_crew_state "$d" headless-dead) - assert_not_contains "$out" 'state: working' "zombie step worker inherited the stale running row" - assert_contains "$out" 'headless pipeline worker dead pid=4242' \ - "zombie step worker was not classified as dead" - pass "dead, suspended, and zombie workers never inherit a stale running verdict" -} - -test_reused_pipeline_worker_pid_is_not_live() { - reset_fakes - local d out - d=$(new_case headless-reused) - make_repo_on_branch "$d/wt" fm/headless-reused - make_fakebin "$d" >/dev/null - fm_write_meta "$d/state/headless-reused.meta" "window=fm:fm-headless-reused" "worktree=$d/wt" "kind=ship" "harness=codex" - FM_FAKE_AXI_STATUS="$(run_running fm/headless-reused)" - FM_FAKE_WORKER_LOG=$(worker_start_log) - FM_FAKE_WORKER_IDENTITY='Mon Aug 3 12:01:00 2026 codex unrelated replacement' - out=$(run_crew_state "$d" headless-reused) - assert_contains "$out" 'state: blocked' "reused allowed-family PID was reported live" - assert_contains "$out" 'headless pipeline worker identity replaced pid=4242' \ - "reused PID did not contradict the bound worker identity" - pass "reused worker PID cannot impersonate the recorded process birth" -} - test_unbound_pipeline_worker_preserves_authoritative_state() { reset_fakes local d out fixture expected @@ -1421,9 +1332,6 @@ test_missing_run_head_falls_back_to_current_state() { } test_active_run_is_authoritative -test_headless_pipeline_worker_liveness_overrides_pane_interruption -test_dead_or_suspended_pipeline_worker_is_not_recorded_working -test_reused_pipeline_worker_pid_is_not_live test_unbound_pipeline_worker_preserves_authoritative_state test_stale_needs_decision_superseded test_stale_blocked_superseded diff --git a/tests/fm-model-capacity-hold.test.sh b/tests/fm-model-capacity-hold.test.sh index f573e37ae76..dda254f953d 100755 --- a/tests/fm-model-capacity-hold.test.sh +++ b/tests/fm-model-capacity-hold.test.sh @@ -182,8 +182,59 @@ test_matching_orphan_release_is_repaired() { pass "matching interrupted release receipt retires the active marker" } +test_active_same_id_requires_matching_metadata() { + local home out err rc receipt + home="$TMP_ROOT/same-id-home" + make_home "$home" + FM_HOME="$home" "$HOLD" register reserve-shared \ + --reason 'first owner reservation' --dispatch-ref 'dispatch first owner' >/dev/null \ + || fail "could not prepare same-id retry case" + out=$(FM_HOME="$home" "$HOLD" register reserve-shared \ + --reason 'first owner reservation' --dispatch-ref 'dispatch first owner') \ + || fail "matching same-id registration was not idempotent" + assert_contains "$out" 'registered: reserve-shared already active' \ + "matching same-id registration did not report the active reservation" + + err="$home/reason-mismatch.err" + rc=0 + FM_HOME="$home" "$HOLD" register reserve-shared \ + --reason 'second owner reservation' --dispatch-ref 'dispatch first owner' \ + >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "same-id registration accepted a different reason" + assert_grep 'active hold metadata does not match retry' "$err" \ + "same-id reason collision did not identify the metadata mismatch" + + err="$home/dispatch-mismatch.err" + rc=0 + FM_HOME="$home" "$HOLD" register reserve-shared \ + --reason 'first owner reservation' --dispatch-ref 'dispatch second owner' \ + >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "same-id registration accepted a different dispatch reference" + assert_grep 'active hold metadata does not match retry' "$err" \ + "same-id dispatch collision did not identify the metadata mismatch" + + receipt="$home/data/model-capacity-holds/reserve-shared.registered" + { + printf 'schema=fm-model-capacity-hold.v1\n' + printf 'hold_id=reserve-shared\n' + printf 'reason=first owner reservation\n' + printf 'dispatch_ref=tampered receipt owner\n' + printf 'registered_at=2026-08-03T12:02:00Z\n' + } > "$receipt" + err="$home/receipt-mismatch.err" + rc=0 + FM_HOME="$home" "$HOLD" register reserve-shared \ + --reason 'first owner reservation' --dispatch-ref 'dispatch first owner' \ + >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "same-id registration accepted a mismatched durable receipt" + assert_grep 'active hold registration receipt does not match retry' "$err" \ + "same-id receipt collision did not identify the durable metadata mismatch" + pass "active same-id retries require matching reservation metadata" +} + test_registered_hold_blocks_send_until_digest_bound_release test_registered_hold_blocks_spawn_before_fleet_mutation test_registration_serializes_competing_hold_ids test_matching_orphan_registration_is_repaired test_matching_orphan_release_is_repaired +test_active_same_id_requires_matching_metadata From 10e71c61cf740a01a88923f98df838cc6198591f Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Mon, 3 Aug 2026 19:53:58 -0600 Subject: [PATCH 10/14] no-mistakes(document): Refresh supervision lifecycle documentation --- AGENTS.md | 12 ++++--- README.md | 4 +-- docs/architecture.md | 3 +- docs/decision-hold-lifecycle.md | 57 +++++++++++++-------------------- 4 files changed, 33 insertions(+), 43 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f7c38e1399a..6f67f0c7c38 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -129,7 +129,7 @@ Treat `data/captain.md` as the domain-local record of captain preferences, optio Run `bin/fm-session-start.sh` exactly once at session start. Its header is the single owner of composed commands, ordering, and digest contents. `bin/fm-supervision-instructions.sh` renders the emitted supervision block from `docs/supervision-protocols/`. -Do not reimplement it by separately running its lock, bootstrap, or initial wake-drain components. +Do not reimplement it by separately running its lock, bootstrap, record-reconciliation, or initial wake-drain components. Tracked native session-open adapters only nudge this command; `docs/sessionstart-nudge.md` owns their current behavior and compatibility. Read the complete digest once and trust it as this turn's startup and recovery input. @@ -144,14 +144,16 @@ A lock-refused session must not spawn, steer, merge, drain the wake queue, repai When the lock could not be acquired, the worktree-tangle check uses read-only advisory wording without a checkout repair command. Home-local stale Herdr projection cleanup and the six bootstrap MUTATING sweeps - non-executing legacy PR-check migration, fleet sync, secondmate convergence, secondmate liveness, pending remote handoff retry, and X-mode artifact writes - run only when this session actually holds the lock from step 1. The secondmate liveness sweep deterministically accounts for every registered secondmate: it relaunches only from the recovery-grade `dead` or `missing` states, preserves ambiguous, unreadable, or unreachable remote targets, and reports skipped or failed guarantees as `SECONDMATE_LIVENESS:` lines (`bin/fm-bootstrap.sh`; `bin/fm-backend.sh`'s `fm_backend_agent_state`; `docs/remote-secondmates.md`). -3. **Wake queue** - when locked, drains the durable wake queue and prints the raw records prominently as this turn's first work queue; a bounded, clearly labeled historical status-event annotation may follow a valid `signal` record but never replaces it or current-state reconciliation, and a lapsed watcher chain still surfaces here via the same guard alarm. +3. **Record reconciliation** - when locked, records exact-head and clean-tree retention evidence for eligible terminal producer rows and inventories metadata/backlog drift without deleting evidence or moving unlanded work to Done. + A lock-refused session skips this mutation, and the producer row remains In flight until teardown independently confirms landing. +4. **Wake queue** - when locked, drains the durable wake queue, custody-versions the consumed bytes, and prints the raw records prominently as this turn's first work queue; a structurally mapped terminal signal triggers record reconciliation in the same turn, while a bounded, clearly labeled historical status-event annotation may follow but never replaces the raw record or current-state reconciliation, and a lapsed watcher chain still surfaces here via the same guard alarm. Every locked drain also prints a bounded fleet-wide `OPEN DECISIONS` section when durable decision records remain open, including when the queue itself is empty; reconcile those entries before continuing. When the lock could not be acquired and verified, the queue is left untouched because no session mutation is authorized, and the guard's tangle/watcher-liveness alarms still print in read-only advisory mode without drain, supervision repair, or checkout repair commands. -4. **Context digest** - the full contents of `data/projects.md`, `data/secondmates.md`, `data/captain.md`, `data/captain-shared.md`, and `data/learnings.md`, each clearly delimited. +5. **Context digest** - the full contents of `data/projects.md`, `data/secondmates.md`, `data/captain.md`, `data/captain-shared.md`, and `data/learnings.md`, each clearly delimited. A file that does not exist prints an explicit `ABSENT` marker, never confused with an empty-but-present file: absence is meaningful (`captain.md` absent means use the firstmate repo's built-in defaults, `projects.md` absent means rebuild it from the clones under `projects/`, etc.). -5. **Fleet-state digest** - the compact backlog listing owned by `bin/fm-session-start.sh`; every `state/<id>.meta`; a bounded tail of each task's `state/<id>.status` (labeled as wake-EVENT history, not current state, with the full log path printed for a deeper read); the `state/.afk` flag; and one cheap alive/dead read of each task's recorded backend endpoint. +6. **Fleet-state digest** - the compact backlog listing owned by `bin/fm-session-start.sh`; every `state/<id>.meta`; a bounded tail of each task's `state/<id>.status` (labeled as wake-EVENT history, not current state, with the full log path printed for a deeper read); the `state/.afk` flag; and one cheap alive/dead read of each task's recorded backend endpoint. That liveness line is a fast presence check only, not a full state read - when you need a crew's actual current state (a run-step, not just "is the pane there"), read it with `bin/fm-crew-state.sh <id>` as before; the digest deliberately skips that deeper, slower read for every task so it stays fast and bounded. -6. **Supervision operating instructions and next step** - after the wake queue and before context, the digest emits exactly one operating block for the detected primary harness. +7. **Supervision operating instructions and next step** - after the wake queue and before context, the digest emits exactly one operating block for the detected primary harness. The closing reminder points back to that emitted block and preserves only the lock, afk, X-mode, and read-once reminders. The script itself never starts supervision; the emitted harness protocol owns the exact wait or wake mechanism. diff --git a/README.md b/README.md index 2264fdab978..0fdd6ed6f5a 100644 --- a/README.md +++ b/README.md @@ -171,7 +171,7 @@ Claude and grok use the slash form shown here; codex uses the same names with `$ | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | `/afk` | Enter away-mode supervision: the sub-supervisor self-handles routine notifications in bash, escalates captain-relevant events and bounded declared-external-wait rechecks as batched digests, and actively alerts if delivery gets stuck while you step away | | `/ahoy` | Recap visible session events since the prior real captain message plus visibly unanswered captain decisions, falling back to Bearings when invoked as the session's first real captain message | -| `/bearings` | Generate a concise four-section chat digest from bounded local fleet and registered-secondmate state; use `/bearings file` to also replace today's dated report in `data/`, and add `include PRs` when live PR enrichment is wanted | +| `/bearings` | Generate a concise four-section chat digest from bounded local fleet and registered-secondmate state; use `/bearings file` to install today's dated report after custody-versioning any same-day predecessor, and add `include PRs` when live PR enrichment is wanted | | `/updatefirstmate` | Self-update the running firstmate and its secondmates to the latest from origin with fast-forward-only pulls, then re-read instructions and nudge secondmates | | `/stow` | Sweep the session for uncaptured durable knowledge, route each finding to its disk home per AGENTS.md, file undone next steps to the backlog, and report what is now safe to reset | @@ -179,7 +179,7 @@ Bearings invocation examples: - `/bearings` returns the fresh four-section digest in chat only. - `/bearings include PRs` keeps chat-only mode and opts into live PR enrichment. -- `/bearings file` replaces today's `data/status-report-<YYYY-MM-DD>.md` from scratch and links it from the four-section chat digest. +- `/bearings file` custody-versions any same-day predecessor, installs a complete replacement at `data/status-report-<YYYY-MM-DD>.md`, and links it from the four-section chat digest. - `/bearings file include PRs` combines the dated report with live PR enrichment. Agent-only reference skills live under `.agents/skills/` and are loaded by firstmate at the trigger points named in [`AGENTS.md`](AGENTS.md). diff --git a/docs/architecture.md b/docs/architecture.md index ac9d89be1b7..7a5bd1544ce 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -91,7 +91,8 @@ The daemon injects only into an affirmatively `empty` composer, so both `pending Unsupported supervisor backends refuse at daemon startup. Stalled escalation delivery writes `state/.subsuper-inject-wedged` and attempts a configured backend-independent active alert after `FM_MAX_DEFER_SECS` instead of silently deferring forever. On an unmarked return, `bin/fm-afk-return.sh` owns ordered shutdown, durable catch-up evidence, and the fail-closed gate that keeps ordinary work behind every live firstmate-actionable blocker. -`fm-send.sh` selects a pre-Enter popup-settle for slash commands and for codex `$...` skill invocations using metadata-routed target `harness=` values, then adds its own `FM_SEND_SETTLE` pause after successful text sends so immediate peeks catch the receiving turn starting; the sub-supervisor uses only the shared submit core and does not pay that post-submit pause. +Before typing a local steer, `fm-send.sh` requires the shared composer classifier to return exactly `empty`; pending or unreadable input refuses before mutation so a stale order cannot be extended or mistaken for the new message. +It then selects a pre-Enter popup-settle for slash commands and for codex `$...` skill invocations using metadata-routed target `harness=` values, and adds its own `FM_SEND_SETTLE` pause after successful text sends so immediate peeks catch the receiving turn starting; the sub-supervisor uses only the shared submit core and does not pay that post-submit pause. ## Busy state is semantic, per adapter diff --git a/docs/decision-hold-lifecycle.md b/docs/decision-hold-lifecycle.md index 234055aec3f..68740e2c89c 100644 --- a/docs/decision-hold-lifecycle.md +++ b/docs/decision-hold-lifecycle.md @@ -14,19 +14,28 @@ It creates a kind `captain` backlog item when absent and invokes `tasks-axi hold It rejects an identity collision, a changed title, and attempts to reopen an already resolved identity. The `complete` subcommand unions the reviewed keys into `decision_keys=` and appends `decisions_reviewed=1` while originating task metadata is live. -A post-teardown visual review can complete against the surviving report and durable holds without recreating volatile task metadata. It accepts `--none` as an explicit semantic inventory result, not as inferred absence. It verifies every listed identity against tasks-axi before recording completion. +Every live completion also binds the exact `endpoint_task_id=` dispatch to the report path and digest in a script-owned origin receipt, including for `--none`. +For each unresolved hold, `complete` requires a fresh Bearings snapshot that exposes the exact hold, then records a cleanup receipt bound to the origin, dispatch, hold identity, backlog-object digest, and Bearings-evidence digest. +An exact post-teardown visual-review pass can add an open hold and rerun `complete` only while the retained report still matches that trusted origin receipt and its dispatch binding. +An open historical hold with a surviving exact binding can reconstruct a missing cleanup receipt through the same full-inventory retry. +Missing or mismatched binding evidence, and already-resolved historical or hand-written rows without script-owned receipts, remain preserved and refused because no safe automatic migration is shipped. For an open keyed status decision, it appends a `captain-held [key=<key>]: ...` transfer event only after the matching backlog hold is durable. `bin/fm-classify-lib.sh` recognizes that transfer as closing the live status copy without claiming that the captain has answered it. Scout teardown calls the script's read-only `verify` subcommand after checking for the report and before removing any source state. +For an unresolved hold, verification requires the exact dispatch binding, trusted origin and cleanup receipts, a matching nonzero backlog-object digest, and a fresh Bearings appearance. +For a resolved hold, verification instead requires the trusted resolution chain described below. The `--force` path remains the explicit captain-approved discard escape hatch. The `resolve` subcommand requires a decision file and at least one existing dependent task whose structured `blocked-by` edge points to the hold. -It records the decision digest and routed task identities as a retry identity in the hold body, clears each dependency edge through tasks-axi, and marks the hold Done only after those writes succeed. -An exact retry can finish a partial routing operation, while a changed decision or routed-task set is rejected. -A failed intermediate step leaves the hold open. +It first requires the trusted cleanup receipt, retains the captain's exact bytes in a canonical decision object, and records the decision digest and complete routed-task identity set as the retry identity in the hold body. +It clears each dependency edge through tasks-axi and marks the hold Done only after those writes succeed. +After closure it writes a resolution receipt bound to the origin, dispatch, hold, canonical decision object and digest, routed identities, and unique live-or-archived row object and digest. +An exact retry can finish a partial routing operation or a missing resolution receipt only while the cleanup receipt and canonical decision object survive; a changed decision or routed-task set is rejected. +A failed intermediate step leaves the hold open unless the row already closed, in which case verification still refuses until the matching script-owned resolution receipt exists. +The read-only `verify-resolution` subcommand revalidates that full chain and every routed task. ## Structured read surfaces @@ -40,14 +49,13 @@ The projection remains read-only and does not inspect historical prose. ## Verification record -Verification date: 2026-07-14. -Additional quoted `blocked_by` regression verification date: 2026-07-17. -Plural blocker-readiness and mixed-home projection verification date: 2026-07-22. +Verification date: 2026-08-03. The focused end-to-end regression uses only synthetic `sample` identities and decision text. It begins with a completed investigation and visual review whose genuine unresolved choice exists only in the report. -The initial Bearings snapshot correctly has no open decision, and the new teardown gate refuses to erase the source. -A later regression covers tasks-axi's quoted multi-entry `blocked_by` output so `resolve` matches the first, middle, and last ids and rejects a genuinely absent id. +The initial Bearings snapshot correctly has no open decision, and the teardown gate refuses to erase the source. +The regression also covers tasks-axi's quoted multi-entry `blocked_by` output so `resolve` matches the first, middle, and last ids and rejects a genuinely absent id. +It exercises origin and dispatch binding, fresh Bearings visibility, nonzero object digests, post-teardown review, partial resolution retry, canonical decision retention, live-or-archived row uniqueness, and refusal of malformed, duplicated, hand-written, or historically unprovable authority objects. The final verification commands and their exact summarized outputs follow. @@ -58,34 +66,13 @@ ok - non-forced scout teardown always requires durable inventory verification ok - captain holds are idempotent, distinct, teardown-safe, Bearings-visible, and durably routed before close ok - completion and verification validate origins before constructing paths ok - ended visual review follows the same decision-hold completion owner +ok - post-teardown completion rejects hand-written resolved rows ok - resolved findings and decision-like prose do not create false holds ok - terminal single-owner stale status decisions do not block empty inventory ok - main-home and secondmate-home captain holds remain correctly routed ok - resolve matches first/middle/last in quoted blocked_by and rejects a genuinely absent id - -$ bash tests/fm-fleet-snapshot-view.test.sh -ok - backlog normalization preserves strict roles and resolves every blocker compatibly -ok - durable captain-held transfer closes the duplicate live status decision -ok - snapshot parses tasks-axi rows and respects operational overrides - -$ bash tests/fm-bearings-snapshot.test.sh -ok - a completed scout with decision-like report prose is a pointer, not pending -ok - action-free items (working/done/queued/landed) do not leak into Captain's Call -ok - mixed secondmate roles, partial state, and captain readiness project independently -ok - main and secondmate captain actionability use the same blocker readiness - -$ bash tests/fm-brief.test.sh -ok - fm-brief.sh: investigation and visual-review completions load the shared decision policy - -$ bash tests/fm-teardown.test.sh -all teardown safety cases passed - -$ bin/fm-lint.sh -fm-lint.sh: ShellCheck 0.11.0 (pinned 0.11.0) - -$ git diff --check -(no output) - -$ for test_script in tests/*.test.sh; do bash "$test_script"; done -ALL 71 TEST SCRIPTS PASSED +ok - historical open and binding-less refusals name only safe operator remedies +ok - absent, malformed, duplicate, wrong-origin, hand-written, and zero-digest resolution rows refuse ``` + +`bin/fm-doc-audience-check.sh` and `git diff --check` are the documentation-phase structural checks. From 1ab64f149a734bbdcc6862398bfbfba98f1f34c7 Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Tue, 4 Aug 2026 15:35:06 -0600 Subject: [PATCH 11/14] test: model Orca composer preflight in route fixture --- tests/fm-backend-orca.test.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/fm-backend-orca.test.sh b/tests/fm-backend-orca.test.sh index 5ee3fd570d5..274806b1987 100755 --- a/tests/fm-backend-orca.test.sh +++ b/tests/fm-backend-orca.test.sh @@ -711,13 +711,14 @@ test_peek_send_and_crew_state_route_through_orca_meta() { FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_SEND_SETTLE=0 \ "$ROOT/bin/fm-peek.sh" "fm-$id" 10 ) [ "$out" = ready ] || fail "fm-peek should read through Orca metadata, got '$out'" - printf '{"ok":true,"result":{"send":{"handle":"term-io","accepted":true}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"terminal":{"tail":["│ > │"]}}}\n' > "$RESP/2.out" printf '{"ok":true,"result":{"send":{"handle":"term-io","accepted":true}}}\n' > "$RESP/3.out" - printf '{"ok":true,"result":{"terminal":{"tail":["│ > │"]}}}\n' > "$RESP/4.out" + printf '{"ok":true,"result":{"send":{"handle":"term-io","accepted":true}}}\n' > "$RESP/4.out" + printf '{"ok":true,"result":{"terminal":{"tail":["│ > │"]}}}\n' > "$RESP/5.out" PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ FM_ROOT_OVERRIDE="$neutral" FM_HOME="$neutral" FM_STATE_OVERRIDE="$state" FM_SEND_SETTLE=0 \ "$ROOT/bin/fm-send.sh" "fm-$id" "hello orca" - printf '{"ok":true,"result":{"terminal":{"tail":["idle prompt"]}}}\n' > "$RESP/5.out" + printf '{"ok":true,"result":{"terminal":{"tail":["idle prompt"]}}}\n' > "$RESP/6.out" out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" "$ROOT/bin/fm-crew-state.sh" "$id" ) assert_contains "$out" "state: unknown" "crew-state should fall back cleanly for an idle Orca scout" From aa4103a1fda297607990d8e39b10a6a016ab5003 Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Tue, 4 Aug 2026 19:51:37 -0600 Subject: [PATCH 12/14] fix: preserve zellij composer fallback --- bin/fm-backend.sh | 36 ++++++++++++++++++-- bin/fm-send.sh | 18 +++++++--- tests/fm-backend-zellij.test.sh | 30 +++++++++++++++++ tests/fm-send-strict.test.sh | 60 ++++++++++++++++++++++++++++++++- 4 files changed, 136 insertions(+), 8 deletions(-) diff --git a/bin/fm-backend.sh b/bin/fm-backend.sh index 7830114fc48..a92cc415bf1 100644 --- a/bin/fm-backend.sh +++ b/bin/fm-backend.sh @@ -793,6 +793,37 @@ fm_backend_busy_state() { # <backend> <target> esac } +# fm_backend_composer_classifier_capability: the static, adapter-owned answer to +# whether <backend> exposes a composer classifier. This is intentionally decided +# only from the verified backend identity, before any target, actor, pane, or +# classifier output is consulted. A newly added or unrecognized backend reports +# unknown so callers refuse until its capability is classified deliberately. +fm_backend_composer_classifier_capability() { # <backend> -> present|absent|unknown + case "$1" in + tmux|herdr|orca|cmux) printf 'present' ;; + zellij) printf 'absent' ;; + *) printf 'unknown' ;; + esac +} + +# fm_backend_composer_preflight_state: bind the static capability to the runtime +# classifier verdict without letting classifier output impersonate the +# no-classifier state. Every result from a present classifier is namespaced under +# classified:, so only this function's static absent arm can emit unclassified. +fm_backend_composer_preflight_state() { # <backend> <target> [expected-label] + local backend=$1 capability state + shift + capability=$(fm_backend_composer_classifier_capability "$backend") + case "$capability" in + present) + state=$(fm_backend_composer_state "$backend" "$@") || state=unknown + printf 'classified:%s' "${state:-unknown}" + ;; + absent) printf 'unclassified' ;; + *) printf 'capability-unknown' ;; + esac +} + # fm_backend_composer_state: classify the composer/input row of <target> as # empty|pending|pending-unproven|unknown for callers that need a pre-submit # input guard or an adapter's conservative submit fallback. It is exposed so a @@ -803,8 +834,9 @@ fm_backend_busy_state() { # <backend> <target> # fm_backend_herdr_composer_state), as do orca and cmux # (fm_backend_orca_composer_state, fm_backend_cmux_composer_state); zellij's # submit path uses an internal content-diff approach with no separately named -# classifier, so it reports unknown here - callers fall back to their own -# policy, exactly as an unknown fm_backend_busy_state already does. +# classifier, so its static capability reports absent and callers fall back to +# its own submit-verification policy. A present classifier that returns unknown +# remains a runtime failure to prove emptiness, never a no-classifier signal. fm_backend_composer_state() { # <backend> <target> -> empty|pending|pending-unproven|unknown local backend=$1 shift diff --git a/bin/fm-send.sh b/bin/fm-send.sh index 68dc095ebbe..c3558e90cec 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -284,11 +284,19 @@ else MESSAGE=$* fm_model_capacity_hold_refuse "text submission to $T" || exit 1 if [ "$TARGET_BACKEND" != remote ]; then - composer_state=$(fm_backend_composer_state "$TARGET_BACKEND" "$T" "$EXPECTED_LABEL") - if [ "$composer_state" != empty ]; then - echo "error: text not sent to $T because its composer is not affirmatively empty (verdict=${composer_state:-unknown}; backend=$TARGET_BACKEND; tried $RESOLUTION_TRIED). Preserve the pane and reconcile its pending input before retrying." >&2 - exit 1 - fi + composer_preflight=$(fm_backend_composer_preflight_state "$TARGET_BACKEND" "$T" "$EXPECTED_LABEL") + case "$composer_preflight" in + classified:empty|unclassified) : ;; + classified:*) + composer_state=${composer_preflight#classified:} + echo "error: text not sent to $T because its composer is not affirmatively empty (verdict=${composer_state:-unknown}; backend=$TARGET_BACKEND; tried $RESOLUTION_TRIED). Preserve the pane and reconcile its pending input before retrying." >&2 + exit 1 + ;; + *) + echo "error: text not sent to $T because composer-classifier capability is not established for backend=$TARGET_BACKEND (capability=${composer_preflight:-unknown}; tried $RESOLUTION_TRIED)." >&2 + exit 1 + ;; + esac fi if [ "$MARK_FROM_FIRSTMATE" = 1 ]; then # Reuse an existing correlation id for recovery resends; otherwise create a diff --git a/tests/fm-backend-zellij.test.sh b/tests/fm-backend-zellij.test.sh index 5039379f8b1..a918bdffebd 100755 --- a/tests/fm-backend-zellij.test.sh +++ b/tests/fm-backend-zellij.test.sh @@ -979,6 +979,35 @@ test_send_text_submit_send_failed_when_pane_absent() { pass "fm_backend_zellij_send_text_submit: reports 'send-failed' when the target pane is absent" } +test_fm_send_text_falls_back_to_zellij_submit_policy() { + local dir state fb neutral rc out + dir="$TMP_ROOT/fm-send-text-fallback"; state="$dir/state" + mkdir -p "$state" "$dir/responses" + neutral="$dir/neutral-root"; mkdir -p "$neutral" + fm_write_meta "$state/zellij-fallback.meta" "window=firstmate:7" "backend=zellij" "kind=ship" + touch "$state/.last-watcher-beat" + zellij_pane_response "$dir" 1 7 3 + zellij_pane_response "$dir" 3 7 3 + zellij_pane_response "$dir" 5 7 3 + zellij_pane_response "$dir" 7 7 3 + printf '%s' $'❯ steer zellij' > "$dir/responses/4.out" + printf '%s' $'steer zellij\n❯' > "$dir/responses/8.out" + fb=$(make_zellij_fakebin "$dir") + fm_fake_exit0 "$fb" sleep + + rc=0 + out=$( PATH="$fb:$PATH" FM_HOME="$neutral" FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" \ + FM_ZELLIJ_LOG="$dir/log" FM_ZELLIJ_RESPONSES="$dir/responses" FM_ZELLIJ_SESSION_LIST="firstmate" \ + FM_SEND_RETRIES=1 FM_SEND_SETTLE=0 \ + "$ROOT/bin/fm-send.sh" firstmate:7 "steer zellij" 2>&1 ) || rc=$? + expect_code 0 "$rc" "fm-send should use zellij's own submit verification when no composer classifier exists: $out" + assert_contains "$(cat "$dir/log")" $'\x1f''paste'$'\x1f''--pane-id'$'\x1f''7'$'\x1f''--'$'\x1f''steer zellij' \ + "fm-send did not reach zellij's submit policy after the no-classifier fallback" + assert_contains "$(cat "$dir/log")" $'\x1f''send-keys'$'\x1f''--pane-id'$'\x1f''7'$'\x1f''Enter' \ + "zellij's fallback submit policy did not send Enter" + pass "fm-send: a statically unclassified zellij backend uses its own submit verification" +} + # --- fm-*.sh script routing via explicit backend-tagged meta ------------------ test_scripts_route_explicit_target_through_meta_backend() { @@ -1115,6 +1144,7 @@ test_send_text_submit_detects_landed_send test_send_text_submit_detects_swallowed_enter test_send_text_submit_send_failed_when_session_absent test_send_text_submit_send_failed_when_pane_absent +test_fm_send_text_falls_back_to_zellij_submit_policy test_scripts_route_explicit_target_through_meta_backend test_scripts_verify_label_for_fm_targets test_scripts_reject_fm_target_label_mismatch diff --git a/tests/fm-send-strict.test.sh b/tests/fm-send-strict.test.sh index 2c6ad2d48cd..86e2eecc1da 100755 --- a/tests/fm-send-strict.test.sh +++ b/tests/fm-send-strict.test.sh @@ -46,7 +46,7 @@ case "${1:-}" in if [ -n "${FM_FAKE_TMUX_DEAD_TARGET:-}" ] && [ "$target" = "$FM_FAKE_TMUX_DEAD_TARGET" ]; then exit 1 fi - [ "$cursor" = 1 ] && { printf '1\n'; exit 0; } + [ "$cursor" = 1 ] && { printf '%s\n' "${FM_FAKE_TMUX_CURSOR:-1}"; exit 0; } printf '%%1\n' exit 0 ;; capture-pane) @@ -88,6 +88,41 @@ setup_home() { # <name> -> echoes home dir printf '%s\n' "$home" } +test_composer_classifier_capability_matrix() { + ( + local backend expected expected_preflight observed observed_preflight seen='' + # shellcheck source=bin/fm-backend.sh + . "$ROOT/bin/fm-backend.sh" + fm_backend_composer_state() { printf 'unknown'; } + for backend in $FM_BACKEND_KNOWN; do + case "$backend" in + tmux|herdr|orca|cmux) expected=present; expected_preflight=classified:unknown ;; + zellij) expected=absent; expected_preflight=unclassified ;; + *) fail "composer-classifier matrix has no expected row for known backend '$backend'" ;; + esac + observed=$(fm_backend_composer_classifier_capability "$backend") \ + || fail "composer-classifier capability query failed for known backend '$backend'" + [ "$observed" = "$expected" ] \ + || fail "composer-classifier capability for $backend: expected $expected, got $observed" + observed_preflight=$(fm_backend_composer_preflight_state "$backend" fixture-target) \ + || fail "composer preflight-state query failed for known backend '$backend'" + [ "$observed_preflight" = "$expected_preflight" ] \ + || fail "composer preflight for $backend with classifier verdict unknown: expected $expected_preflight, got $observed_preflight" + seen="$seen $backend=$observed/$observed_preflight" + done + [ "$seen" = " tmux=present/classified:unknown herdr=present/classified:unknown zellij=absent/unclassified orca=present/classified:unknown cmux=present/classified:unknown" ] \ + || fail "composer-classifier matrix did not cover the exact known-backend enumeration:$seen" + [ "$(fm_backend_composer_classifier_capability bogus)" = unknown ] \ + || fail "an unrecognized backend must not inherit the no-classifier fallback" + [ "$(fm_backend_composer_preflight_state bogus fixture-target)" = capability-unknown ] \ + || fail "an unrecognized backend must fail closed at preflight" + fm_backend_composer_state() { printf 'unclassified'; } + [ "$(fm_backend_composer_preflight_state tmux fixture-target)" = classified:unclassified ] \ + || fail "a classified backend's actor-influenced verdict impersonated the static no-classifier state" + ) || fail "composer-classifier capability matrix failed" + pass "fm-backend: every known backend declares a static composer-classifier capability" +} + test_exact_lane_id_send_still_works() { local dir fb home err log rc got dir="$TMP_ROOT/exact"; mkdir -p "$dir" @@ -217,6 +252,27 @@ test_stale_composer_refuses_before_typing() { pass "fm-send strict: stale composer text refuses before any typing or Enter" } +test_unknown_classified_composer_refuses_before_typing() { + local dir fb home err log rc got + dir="$TMP_ROOT/unknown-classified-composer"; mkdir -p "$dir" + fb=$(make_stubs "$dir"); home=$(setup_home unknownclassified); err="$dir/send.err"; log="$dir/tmux.log"; : > "$log" + fm_write_meta "$home/state/unknown.meta" "window=sess:fm-unknown" "kind=ship" "harness=codex" + + rc=0 + PATH="$fb:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$home" FM_TMUX_LOG="$log" \ + FM_FAKE_TMUX_CURSOR=unreadable FM_SEND_SETTLE=0 \ + "$SEND" unknown "fresh order" >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "unknown verdict from a classified backend was accepted" + got=$(cat "$log") + assert_not_contains "$got" 'literal=1 arg=fresh order' \ + "fm-send typed after a classified backend returned unknown" + assert_not_contains "$got" 'literal=0 arg=Enter' \ + "fm-send submitted after a classified backend returned unknown" + assert_contains "$(cat "$err")" 'verdict=unknown' \ + "classified-backend refusal did not preserve the unknown verdict" + pass "fm-send strict: unknown remains unsafe when a backend has a composer classifier" +} + test_remote_send_uses_host_local_composer_check() { local dir fb home err log rc decoded dir="$TMP_ROOT/remote"; mkdir -p "$dir" @@ -251,6 +307,7 @@ SH pass "fm-send strict: remote text reaches the host-local guarded sender" } +test_composer_classifier_capability_matrix test_exact_lane_id_send_still_works test_unset_fm_home_fails test_unresolvable_target_does_not_tmux_fallback @@ -258,5 +315,6 @@ test_prefixless_herdr_pane_id_fails test_unmatched_single_colon_target_must_exist test_fm_prefixed_herdr_session_is_an_explicit_target test_healthy_fm_id_send_still_works +test_unknown_classified_composer_refuses_before_typing test_stale_composer_refuses_before_typing test_remote_send_uses_host_local_composer_check From dd9a9a8c3e0fc39fff566ee4bf003e8bdcd663ae Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Tue, 4 Aug 2026 22:03:22 -0600 Subject: [PATCH 13/14] test: cover unknown composer capability refusal --- docs/architecture.md | 2 +- tests/fm-send-strict.test.sh | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/docs/architecture.md b/docs/architecture.md index 7a5bd1544ce..6757983040f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -91,7 +91,7 @@ The daemon injects only into an affirmatively `empty` composer, so both `pending Unsupported supervisor backends refuse at daemon startup. Stalled escalation delivery writes `state/.subsuper-inject-wedged` and attempts a configured backend-independent active alert after `FM_MAX_DEFER_SECS` instead of silently deferring forever. On an unmarked return, `bin/fm-afk-return.sh` owns ordered shutdown, durable catch-up evidence, and the fail-closed gate that keeps ordinary work behind every live firstmate-actionable blocker. -Before typing a local steer, `fm-send.sh` requires the shared composer classifier to return exactly `empty`; pending or unreadable input refuses before mutation so a stale order cannot be extended or mistaken for the new message. +Before typing a local steer, `fm-send.sh` requires a statically classified backend's shared composer classifier to return exactly `empty`; a backend statically declared without a classifier falls back to its adapter's submit-verification policy, while pending or unreadable input and unknown classifier capability refuse before mutation so a stale order cannot be extended or mistaken for the new message. It then selects a pre-Enter popup-settle for slash commands and for codex `$...` skill invocations using metadata-routed target `harness=` values, and adds its own `FM_SEND_SETTLE` pause after successful text sends so immediate peeks catch the receiving turn starting; the sub-supervisor uses only the shared submit core and does not pay that post-submit pause. ## Busy state is semantic, per adapter diff --git a/tests/fm-send-strict.test.sh b/tests/fm-send-strict.test.sh index 86e2eecc1da..fe587ca6db7 100755 --- a/tests/fm-send-strict.test.sh +++ b/tests/fm-send-strict.test.sh @@ -273,6 +273,34 @@ test_unknown_classified_composer_refuses_before_typing() { pass "fm-send strict: unknown remains unsafe when a backend has a composer classifier" } +test_unknown_composer_classifier_capability_refuses_before_typing() { + local dir fb home err log fixture_root rc got + dir="$TMP_ROOT/unknown-composer-capability"; mkdir -p "$dir" + fb=$(make_stubs "$dir"); home=$(setup_home unknowncapability); err="$dir/send.err"; log="$dir/tmux.log"; : > "$log" + fixture_root="$dir/fixture-root" + cp -R "$ROOT/bin" "$fixture_root" + cat >> "$fixture_root/fm-backend.sh" <<'SH' + +fm_backend_composer_classifier_capability() { printf 'unknown'; } +SH + fm_write_meta "$home/state/unknown-capability.meta" "window=sess:fm-unknown-capability" "kind=ship" "harness=codex" + + rc=0 + PATH="$fb:$PATH" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" FM_TMUX_LOG="$log" FM_SEND_SETTLE=0 \ + "$fixture_root/fm-send.sh" unknown-capability "fresh order" >/dev/null 2>"$err" || rc=$? + [ "$rc" -ne 0 ] || fail "unknown composer-classifier capability was accepted" + got=$(cat "$log") + assert_not_contains "$got" 'literal=1 arg=fresh order' \ + "fm-send typed after the backend's composer-classifier capability became unknown" + assert_not_contains "$got" 'literal=0 arg=Enter' \ + "fm-send submitted after the backend's composer-classifier capability became unknown" + assert_contains "$(cat "$err")" 'composer-classifier capability is not established' \ + "unknown-capability refusal did not explain the delivery gap" + assert_contains "$(cat "$err")" 'capability=capability-unknown' \ + "unknown-capability refusal did not preserve the preflight verdict" + pass "fm-send strict: unknown composer-classifier capability refuses before any typing or Enter" +} + test_remote_send_uses_host_local_composer_check() { local dir fb home err log rc decoded dir="$TMP_ROOT/remote"; mkdir -p "$dir" @@ -316,5 +344,6 @@ test_unmatched_single_colon_target_must_exist test_fm_prefixed_herdr_session_is_an_explicit_target test_healthy_fm_id_send_still_works test_unknown_classified_composer_refuses_before_typing +test_unknown_composer_classifier_capability_refuses_before_typing test_stale_composer_refuses_before_typing test_remote_send_uses_host_local_composer_check From 0fddbae64cb4fbd8b0a0b3dbe276e4ff5a3a9c87 Mon Sep 17 00:00:00 2001 From: 420tombombadil <dijongui@gmail.com> Date: Tue, 4 Aug 2026 22:24:13 -0600 Subject: [PATCH 14/14] no-mistakes(document): Clarify composer classifier documentation --- docs/scripts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/scripts.md b/docs/scripts.md index 59fcbf10900..57a7cda9656 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -52,7 +52,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-spawn.sh` | Spawn crewmates, scouts, `id=repo` batches, and secondmates on the resolved harness and runtime backend | | `fm-backend.sh` | Runtime-backend selection, meta helpers, selector resolution, and operation dispatch | | `fm-backend-hometag-lib.sh` | Shared per-installation home-tag derivation for zellij tab and cmux workspace titles | -| `fm-composer-lib.sh` | Single fleet-wide owner of composer-content classification for all backends | +| `fm-composer-lib.sh` | Shared composer-content classifier used by statically classified backends | | `backends/tmux.sh` | Verified tmux session-provider adapter | | `backends/herdr.sh` | Experimental herdr session-provider adapter | | `backends/zellij.sh` | Experimental zellij session-provider adapter |