diff --git a/.agents/skills/afk/SKILL.md b/.agents/skills/afk/SKILL.md index 9562ff91573..65b402f14c3 100644 --- a/.agents/skills/afk/SKILL.md +++ b/.agents/skills/afk/SKILL.md @@ -2,7 +2,7 @@ name: afk description: >- Enter the away posture when the captain invokes /afk, says they are going afk, `state/.afk-contract` or `state/.afk` exists, an incoming message starts with `FM_INJECT_MARK`, or any `state/.subsuper-*` marker is involved. - It writes the durable away-posture record with the captain's away words verbatim as the whole mandate in the same turn as /afk, before any other work and without waiting for a further go, reads the words back in plain sentences after entry, announces hold-for-return only at entry, keeps the one supervision session running in the away posture (on Pi the supervision branch acts on the words by its own judgment and takes every safe actionable wake with main parked; the daemon still delivers batched digests on the other harnesses for now), and on the first unmarked message renders the return brief from durable records before ordinary work resumes. + It writes the durable away-posture record with the captain's away words verbatim as the whole mandate in the same turn as /afk, before any other work and without waiting for a further go, reads the words back in plain sentences after entry, announces hold-for-return only at entry, keeps the one supervision session running in the away posture (on Pi the supervision branch acts on the words by its own judgment and takes every safe actionable wake with main parked, as the supervision host does on a non-Pi home that opted into it; the daemon still delivers batched digests elsewhere for now), and on the first unmarked message renders the return brief from durable records before ordinary work resumes. user-invocable: true metadata: internal: true @@ -31,14 +31,15 @@ Hold-for-return is the default and the only reach profile this release records: The away daemon is no longer launched on Pi; the ordinary supervision session (`docs/pi-supervision-branch.md`) keeps running with the record present, and `bin/fm-afk-launch.sh start` refuses on these harnesses. With the record present main is parked: the supervision branch takes every safe actionable wake, captain outcomes accumulate for the return brief, and main's standing authority relocates to the branch through the guarded scripts (`docs/pi-supervision-branch.md` "Postures"); only a wake the branch declines (including a broken branch or unsafe scan) or a watcher failure wakes main. `/quiet` needs nothing extra on Pi: the attended branch already keeps routine wakes out of this conversation, so quiet-while-present is the attended posture's own shape there. - - **Claude with `config/supervision-host`**: nothing to launch for `/afk`; go on to the announcement. - The supervision host (`docs/supervision-host.md`) is the away session there: it runs the branch's contract on a headless engine under the record while main is parked, and `bin/fm-afk-launch.sh start-native` refuses the away daemon on that home. + - **Claude, Cursor, OpenCode, omp, Grok, or Codex with `config/supervision-host`**: nothing to launch for `/afk`; go on to the announcement. + The supervision host (`docs/supervision-host.md`) is the away session there: it runs the branch's contract on a headless engine under the record while main is parked, and `bin/fm-afk-launch.sh start` and `start-native` refuse the away daemon on that home. + If `enter` printed a `Supervision host: no engine ...` line, every away wake reaches this conversation instead; say so in the announcement. `/quiet` is unchanged there and still launches the daemon below. - - **Harness WITH a native in-pane tracked-background tool** (claude's background bash without the supervision host, grok's background tool): run `bin/fm-afk-launch.sh start-native`, then run `FM_AFK_STATE_PREPARED=1 bin/fm-afk-start.sh` through that native tool. + - **Harness WITH a native in-pane tracked-background tool** (claude's and grok's, without the supervision host): run `bin/fm-afk-launch.sh start-native`, then run `FM_AFK_STATE_PREPARED=1 bin/fm-afk-start.sh` through that native tool. This is a deliberate no-separate-terminal exception because the harness-hosted job creates no terminal or layout mutation, and a shell launcher cannot invoke a harness-native background tool. If the native launch fails, run `bin/fm-afk-launch.sh stop` to roll back the prepared lifecycle. Do not wrap it in `nohup ... &` (Codex/herdr can reap fire-and-forget shell children after a tool call returns). - - **Every other harness** (codex, opencode, omp, kimi, cursor): run `bin/fm-afk-launch.sh start`. + - **Every other harness** (codex, opencode, omp, and cursor without the supervision host, and kimi): run `bin/fm-afk-launch.sh start`. It is the single owner of the daemon terminal: it creates a NON-VISIBLE tracked terminal for the current backend and passes the captain pane in as `FM_SUPERVISOR_TARGET` so the daemon injects into the captain, not its own new pane (docs/herdr-backend.md "Away-mode supervisor support"). Both daemon paths require the record `enter` wrote and share `bin/fm-afk-start.sh` as the daemon entry. The daemon is **presence-gated**: it injects escalations only while `state/.afk` exists, and stays quiet otherwise. @@ -59,7 +60,7 @@ Hold-for-return is the default and the only reach profile this release records: Destructive, irreversible, and security-sensitive actions are never pre-authorizable whatever the words say, and ask-user findings keep the `ask-user-authority` policy unless the words pre-answer the exact decision; anything else that needs the captain holds for their return. - On Pi, main is parked and the supervision branch handles every safe actionable wake under main's standing authority, through the same guarded scripts main would use: any pull request green at its live head may merge (which one the words meant is the branch's reading), queued work whose blockers cleared - already queued, or filed by the branch because the words explicitly call for it - dispatches within the spend cap, and a decision is answered with the captain's own pre-stated answer or under `ask-user-authority`. Anything else holds for the return, a red merge never proceeds while away, local-only landing always waits for the captain, and only a wake the branch declines (including a broken branch or unsafe scan) or a watcher failure wakes main (`docs/pi-supervision-branch.md` "Postures"). -- On a Claude home with `config/supervision-host`, the host's engine is that branch under the same rules, and a wake it hands back reaches main as `Stop hook feedback` with a `supervision-host:` line: that is automatic supervision, never the captain's return, so handle it under the away posture ([supervision protocol](../../../docs/supervision-protocols/supervision-host.md)). +- On a non-Pi home with `config/supervision-host`, the host's engine is that branch under the same rules, and a wake it hands back reaches main through that harness's own wake path (`Stop hook feedback` on Claude, a `watcher` follow-up on Cursor, OpenCode, and omp, the arm's background-task-completed notification on Grok, the checkpoint's output on Codex) with a `supervision-host:` line: that is automatic supervision, never the captain's return, so handle it under the away posture ([supervision protocol](../../../docs/supervision-protocols/supervision-host.md)). - The session-start digest reports the posture under its AFK subsection, so a restart re-enters the posture from the record, not from memory. ## How to exit: the return @@ -77,7 +78,7 @@ No `/back` is needed. The first genuine message is the return signal: Acting on the fleet - dispatching, steering, merging, or any other ordinary captain work - still waits until the check exits successfully. Once it does, close every task the brief lists under "Landed, cleanup due" through ordinary teardown (`bin/fm-teardown.sh `, never forced; a refusal is a stop-and-investigate result) and tell the captain those workers are closed in outcome language. - A message **with** the current operational prefix (`FM_OPERATIONAL_PREFIX`, U+2063 INVISIBLE SEPARATOR followed by `FIRSTMATE_OP: `), or a legacy bare `FM_INJECT_MARK` daemon escalation -> stay away and process it. -- A `Stop hook feedback` wake from the Stop hook or the supervision host -> stay away and process it; it is automatic supervision, not a message from the captain. +- A `Stop hook feedback` wake from the Stop hook or the supervision host, or a Grok background-task-completed notification for the arm -> stay away and process it; it is automatic supervision, not a message from the captain. - Re-invoking `/afk` while already away -> stay away (refresh); this does **not** trigger an exit. Bias ambiguous cases toward exit: a present captain beats token savings, and a false exit is self-correcting (the captain re-runs `/afk`). @@ -98,7 +99,7 @@ Destructive, irreversible, and security-sensitive actions are never pre-authoriz ## The daemon, where it still runs -On the harnesses that still launch the daemon (every verified harness except Pi and pi-signed, and except away mode on a Claude home with `config/supervision-host`), the mechanics below are unchanged. +On the harnesses that still launch the daemon (every verified harness except Pi and pi-signed, and except away mode on a home with `config/supervision-host`), the mechanics below are unchanged. ### Operational prefix contract diff --git a/.agents/skills/harness-adapters/references/harness/codex.md b/.agents/skills/harness-adapters/references/harness/codex.md index d68486f12e2..2dd3e4b33b7 100644 --- a/.agents/skills/harness-adapters/references/harness/codex.md +++ b/.agents/skills/harness-adapters/references/harness/codex.md @@ -50,4 +50,5 @@ The tracked hook anchors to `pwd -P`, verifies that root is Firstmate-shaped and Codex's primary watcher protocol is `../../../bin/fm-watch-checkpoint.sh --seconds "${FM_CODEX_WATCH_CHECKPOINT:-180}"`, not `../../../bin/fm-watch-arm.sh`. Codex cannot reason while a foreground tool call is running, so the checkpoint is deliberately foreground and bounded to return control regularly for user messages and queued notifications. +In a home with `config/supervision-host` the checkpoint runs the supervision host instead of the watcher, with Claude's print mode as its headless engine, and holds for at least an hour while away; [`supervision-host.md`](../../../../../docs/supervision-host.md) owns the host and that bound. Codex's PreToolUse watcher-arm seatbelt blocks directly through its project hook. diff --git a/.agents/skills/harness-adapters/references/harness/cursor.md b/.agents/skills/harness-adapters/references/harness/cursor.md index 0bdede0f20a..4906f178308 100644 --- a/.agents/skills/harness-adapters/references/harness/cursor.md +++ b/.agents/skills/harness-adapters/references/harness/cursor.md @@ -68,6 +68,7 @@ Example: `../../../bin/fm-spawn.sh --scout --harness cursor ## Primary integration Primary supervision is the stop-hook park in `../../../docs/supervision-protocols/cursor.md` through tracked `.cursor/hooks.json`; primary and secondmate launches require `--trust` or hooks do not load. +In a home with `config/supervision-host` the park runs the supervision host instead of `../../../bin/fm-watch-arm.sh`, with Claude's print mode as its headless engine; [`supervision-host.md`](../../../../../docs/supervision-host.md) owns the host. Cursor exposes 20 project events plus a Claude-Code compatibility map that loads `.claude/settings.json`. Tracked hooks register `stop`, `sessionStart`, and two `preToolUse` seatbelts through `$CURSOR_PROJECT_DIR`; Claude entries stand down on Cursor payloads under `../../../docs/turnend-guard.md`. diff --git a/.agents/skills/harness-adapters/references/harness/grok.md b/.agents/skills/harness-adapters/references/harness/grok.md index 82e6ec1c19f..cbaa7075389 100644 --- a/.agents/skills/harness-adapters/references/harness/grok.md +++ b/.agents/skills/harness-adapters/references/harness/grok.md @@ -66,4 +66,5 @@ The exact running Stop payload selects same-process continuation on 0.2.112; 0.2 Grok also loads Claude project settings, so Claude entries for Grok-covered events stand down under `GROK_AGENT` or `GROK_HOOK_EVENT`; that owner records the exact set and why `GROK_SESSION_ID` is excluded. Project-local hooks require launch-time `--trust`; without it the guard steps aside and `../../../bin/fm-guard.sh` is the next-command alarm. Watcher supervision remains tracked background notification around `../../../bin/fm-watch-arm.sh`, not Pi-style extension ownership. +In a home with `config/supervision-host` the session-start block renders that background call as `../../../bin/fm-supervision-host.sh park`, with Claude's print mode as its headless engine; [`supervision-host.md`](../../../../../docs/supervision-host.md) owns the host. PreToolUse blocks directly, but every `$VAR` in a hook command needs inline `:-default` or Grok refuses the hook. diff --git a/.agents/skills/harness-adapters/references/harness/omp.md b/.agents/skills/harness-adapters/references/harness/omp.md index ee78d1b1bba..d07be220ddc 100644 --- a/.agents/skills/harness-adapters/references/harness/omp.md +++ b/.agents/skills/harness-adapters/references/harness/omp.md @@ -50,7 +50,7 @@ There is no `agent_settled` event; `agent_end` plus `willContinue` replaces it. The omp primary follows the Pi extension-owned watcher model through `../../../docs/supervision-protocols/omp.md`: `.omp/extensions/fm-primary-omp-watch.ts` arms `bin/fm-watch-arm.sh --restart` through the `fm_watch_arm_omp` tool and owns every successor, and `.omp/extensions/fm-primary-turnend-guard.ts` answers omp's blocking `session_stop` hook by forcing one continuation when `../../../bin/fm-turnend-guard.sh` returns 2, bounded per turn by omp's `stop_hook_active` flag. The same file ports the `tool_call` seatbelts and delivers the session-start digest through `before_agent_start` on the Run tier; omp's `session_start` carries no reason, so the source is derived (first start `startup` or `resume` from the launch line, later in-process starts `clear`, `session_compact` as `compact`). omp has no asynchronous Stop-hook equivalent, so the Claude auto-arm model does not apply; `fm_supervision_model` classifies omp as `extension`, and `fm_omp_extension_owns_supervision` in `../../../bin/fm-wake-lib.sh` is the ownership proof that tolerates the extension's own watcher hand-off. -The Pi supervision branch is out of scope for omp; every actionable wake is delivered to main. +The Pi supervision branch does not run on omp; without the supervision host every actionable wake is delivered to main, and in a home with `config/supervision-host` the watch extension spawns the host instead of the arm, with Claude's print mode as its headless engine ([`supervision-host.md`](../../../../../docs/supervision-host.md)). Launch a primary with plain `omp` inside the home (`FM_OMP_HARNESS=omp omp` when starting from a Claude pane); `../../../bin/fm-session-start.sh` prints `OMP_WATCH_EXTENSION: not loaded` when the running session has not loaded both tracked extensions. `FM_OMP_LIVE_E2E=1 ../../../tests/fm-omp-primary-live-e2e.test.sh` is the opt-in live guard; `../../../tests/fm-omp-harness.test.sh` is the portable regression. A secondmate registered with `remote=1` in `data/secondmates.md`, spawned through the ordinary `../../../bin/fm-spawn.sh --secondmate` path, is refused on omp until a remote host verifies it, as is `../../../bin/fm-remote-secondmate-control.sh launch`; there is no `--remote` flag. diff --git a/.agents/skills/harness-adapters/references/harness/opencode.md b/.agents/skills/harness-adapters/references/harness/opencode.md index ca9ff18b3f5..66229475f0e 100644 --- a/.agents/skills/harness-adapters/references/harness/opencode.md +++ b/.agents/skills/harness-adapters/references/harness/opencode.md @@ -37,6 +37,7 @@ The primary integration was verified on 2026-07-08 with OpenCode 1.17.6. `.opencode/plugins/fm-primary-turnend-guard.js` listens for `session.idle`. Throwing from `session.idle` does not block `opencode run`, so the primary adapter treats the event as passive and uses `client.session.promptAsync` to force one follow-up turn when `../../../bin/fm-turnend-guard.sh` returns 2. The follow-up was verified in the interactive TUI. +In a home with `config/supervision-host` the watch-arm plugin spawns the supervision host instead of `../../../bin/fm-watch-arm.sh`, with Claude's print mode as its headless engine; [`supervision-host.md`](../../../../../docs/supervision-host.md) owns the host. `opencode run` can exit before displaying a queued follow-up, so the adapter steps aside in headless mode. On native Windows, the operational-input adapter runs its Bash helper through `bash`; macOS and Linux invoke it directly. diff --git a/.omp/extensions/fm-primary-omp-watch.ts b/.omp/extensions/fm-primary-omp-watch.ts index 6d908d258d7..383d3c7b8fc 100644 --- a/.omp/extensions/fm-primary-omp-watch.ts +++ b/.omp/extensions/fm-primary-omp-watch.ts @@ -23,6 +23,18 @@ // hooks exist. // - The arming tool is fm_watch_arm_omp and its human fallback // /fm-watch-arm-omp; the loaded-build marker is state/.omp-watch-extension-loaded. +// - Supervision host: a home opted in with config/supervision-host +// (docs/configuration.md "Supervision host" owns the opt-in) spawns +// bin/fm-supervision-host.sh park --restart in the arm's place, which +// takes away-posture wakes itself and closes only when main is needed; its +// header owns the output read here. A "supervision-host:" line is +// actionable like a wake line, and the message delivered at the host's +// close carries every such line in order while wake lines keep an +// eight-line cap. The host +// prints the first cycle's status line as soon as it is verified, so +// readiness and the handling handoff work as they do for the arm, with a +// longer readiness budget for the host's own startup. Without the file +// nothing below changes. // // Session-generation ownership (stated once here): // omp emits session_shutdown for ordinary same-process replacements (/new, @@ -46,7 +58,7 @@ // replacement handoff. import { spawn, spawnSync, type ChildProcess } from "node:child_process"; import { createHash } from "node:crypto"; -import { mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs"; +import { existsSync, mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs"; import { dirname, resolve } from "node:path"; import { fileURLToPath } from "node:url"; // typebox resolves inside omp's extension loader (verified, omp 18.1.11); the @@ -128,6 +140,7 @@ const fmRoot = process.env.FM_ROOT_OVERRIDE || root; const state = process.env.FM_STATE_OVERRIDE || `${fmHome}/state`; const config = process.env.FM_CONFIG_OVERRIDE || `${fmHome}/config`; const armScript = `${fmRoot}/bin/fm-watch-arm.sh`; +const hostScript = `${fmRoot}/bin/fm-supervision-host.sh`; const marker = `${state}/.omp-watch-extension-loaded`; const handoffDir = `${state}/extensions/omp-primary-watch`; const actionableHandoff = `${handoffDir}/session-replacement-actionable.json`; @@ -142,6 +155,7 @@ const armReadyTimeoutMs = positiveInteger( "FM_OMP_ARM_READY_TIMEOUT_MS", process.platform === "win32" ? 35000 : 12000, ); +const hostReadyTimeoutMs = Math.max(armReadyTimeoutMs, 30000); const armRetireTimeoutMs = positiveInteger("FM_WATCH_ARM_RETIRE_TIMEOUT_MS", 1000); const repairOnlyHint = "call fm_watch_arm_omp again only after a later notification says the cycle is missing, failed, or unhealthy"; const shuttingDownMessage = "watcher: not armed - omp session is shutting down"; @@ -186,6 +200,7 @@ const armClose = new WeakMap>(); const armRetired = new WeakSet(); const armRecovery = new WeakMap(); const armPendingActionable = new WeakMap(); +const armHostMode = new WeakMap(); function positiveInteger(name: string, fallback: number): number { const value = Number(process.env[name]); @@ -241,6 +256,25 @@ function completedActionableLine(output: string): string { return newline < 0 ? "" : actionableLine(output.slice(0, newline + 1)); } +// The host-mode wake message: every "supervision-host:" line in order, wake +// lines capped at eight, and the away note while the posture record exists. +function hostWakeMessage(output: string): string { + let shown = 0; + const lines = output.split(/\r?\n/).filter((line) => { + if (/^supervision-host:/.test(line)) return true; + if (/^(signal:|stale:|check:|heartbeat($|:))/.test(line) && shown < 8) { + shown += 1; + return true; + } + return false; + }); + if (lines.length === 0) return ""; + if (existsSync(`${state}/.afk-contract`)) { + lines.push("This wake comes from automatic supervision under the away-posture record, not from the captain: it is not a return, so handle it under the away posture."); + } + return lines.join("\n"); +} + // The text omp carries in a user message_start: sendUserMessage wraps a string // as one text part, so the joined text parts equal the sent content. function userMessageText(content: unknown): string { @@ -281,7 +315,8 @@ function validatePendingActionable(value: unknown): PendingActionableClose { typeof (value as { token?: unknown }).token !== "string" || !/^[0-9]+-[0-9]+-[0-9]+$/.test((value as { token: string }).token) || typeof (value as { message?: unknown }).message !== "string" || - !actionableLine((value as { message: string }).message) || + (!actionableLine((value as { message: string }).message) && + !/^supervision-host:/m.test((value as { message: string }).message)) || typeof (value as { predecessorArmPid?: unknown }).predecessorArmPid !== "string" || !/^[0-9]*$/.test((value as { predecessorArmPid: string }).predecessorArmPid) || ((value as { delivered?: unknown }).delivered !== undefined && @@ -372,8 +407,20 @@ function clearReplacementHandoff(pending: PendingActionableClose): void { } } -function classifyClose(stdout: string, stderr: string, code: number | null, signal: NodeJS.Signals | null): CloseClassification { +function classifyClose( + hostMode: boolean, + stdout: string, + stderr: string, + code: number | null, + signal: NodeJS.Signals | null, +): CloseClassification { const combined = `${stdout}\n${stderr}`.trim(); + if (hostMode) { + const message = hostWakeMessage(combined); + if (message) return { kind: "actionable", message }; + const stoodDown = combined.split(/\r?\n/).find((line) => /^supervision-host stood down:/.test(line)); + if (stoodDown) return { kind: "failure", message: `watcher: FAILED - ${stoodDown}` }; + } const reason = actionableLine(combined); if (reason) return { kind: "actionable", message: reason }; const healthy = combined.split(/\r?\n/).find((line) => /^watcher: healthy\b/.test(line)); @@ -392,9 +439,10 @@ function classifyClose(stdout: string, stderr: string, code: number | null, sign }; } if (code && code !== 0) { + const script = hostMode ? "fm-supervision-host.sh" : "fm-watch-arm.sh"; return { kind: "failure", - message: `watcher: FAILED - fm-watch-arm.sh exited ${code}${combined ? `\n${combined}` : ""}`, + message: `watcher: FAILED - ${script} exited ${code}${combined ? `\n${combined}` : ""}`, }; } return { @@ -783,8 +831,9 @@ export default function (pi: ExtensionAPI) { function waitForReadiness(armChild: ChildProcess): Promise { const readiness = armReadiness.get(armChild); if (!readiness) return Promise.resolve(false); + const timeout = armHostMode.get(armChild) ? hostReadyTimeoutMs : armReadyTimeoutMs; return new Promise((resolveReady) => { - const timer = setTimeout(() => resolveReady(false), armReadyTimeoutMs); + const timer = setTimeout(() => resolveReady(false), timeout); timer.unref(); void readiness.then((ready) => { clearTimeout(timer); @@ -888,19 +937,23 @@ export default function (pi: ExtensionAPI) { }; } const id = ++owner.seq; - const env = { + const hostMode = existsSync(`${config}/supervision-host`); + const env: NodeJS.ProcessEnv = { ...process.env, FM_HOME: fmHome, FM_ROOT_OVERRIDE: fmRoot, FM_CONFIG_OVERRIDE: config, - FM_WATCH_ARM_SCRIPT: armScript, + FM_WATCH_ARM_SCRIPT: hostMode ? hostScript : armScript, FM_WATCH_PREDECESSOR_ARM_PID: predecessorArmPid, }; - const armChild = spawn("bash", ["-lc", "config_dir=\"${FM_CONFIG_OVERRIDE:-$FM_HOME/config}\"; [ -f \"$config_dir/x-mode.env\" ] && . \"$config_dir/x-mode.env\"; exec \"$FM_WATCH_ARM_SCRIPT\" --restart"], { + if (hostMode) env.FM_SUPERVISION_HOST_PRIMARY = "omp"; + const command = hostMode ? "exec \"$FM_WATCH_ARM_SCRIPT\" park --restart" : "exec \"$FM_WATCH_ARM_SCRIPT\" --restart"; + const armChild = spawn("bash", ["-lc", `config_dir="\${FM_CONFIG_OVERRIDE:-$FM_HOME/config}"; [ -f "$config_dir/x-mode.env" ] && . "$config_dir/x-mode.env"; ${command}`], { cwd: fmRoot, env, stdio: ["ignore", "pipe", "pipe"], }); + armHostMode.set(armChild, hostMode); owner.child = armChild; let stdout = ""; let stderr = ""; @@ -930,6 +983,7 @@ export default function (pi: ExtensionAPI) { if (/^watcher: (?:started|attached)\b/m.test(combined)) { settleReadiness(true); } + if (hostMode) return; const reason = completedActionableLine(stdout) || completedActionableLine(stderr); if (reason && !armPendingActionable.has(armChild)) { const pending = createPendingActionable(reason, String(armChild.pid ?? "")); @@ -954,7 +1008,7 @@ export default function (pi: ExtensionAPI) { resolveClosed(); settleReadiness(false); releaseChild(); - const classification = classifyClose(stdout, stderr, code, signal); + const classification = classifyClose(hostMode, stdout, stderr, code, signal); const predecessor = String(armChild.pid ?? ""); if (classification.kind === "actionable") { const pending = armPendingActionable.get(armChild) ?? createPendingActionable(classification.message, predecessor); diff --git a/.opencode/plugins/fm-primary-watch-arm.js b/.opencode/plugins/fm-primary-watch-arm.js index d4e8850bb21..d2147967398 100644 --- a/.opencode/plugins/fm-primary-watch-arm.js +++ b/.opencode/plugins/fm-primary-watch-arm.js @@ -3,12 +3,25 @@ import { existsSync, readFileSync, readdirSync, realpathSync } from "node:fs"; import { resolve } from "node:path"; import { encodeFirstmateOperationalInput } from "./lib/fm-operational-input.js"; +// Supervision host: a home opted in with config/supervision-host +// (docs/configuration.md "Supervision host" owns the opt-in) spawns +// bin/fm-supervision-host.sh park --restart in the arm's place, which takes +// away-posture wakes itself and closes only when main is needed; its header +// owns the output read here. A "supervision-host:" line is actionable like a +// wake line, and the delivered message carries every such line in order while +// wake lines keep an eight-line cap. The host prints the first cycle's status +// line as soon as it is verified, so readiness and the handling handoff work +// as they do for the arm, with a longer readiness budget for the host's own +// startup. Without the file nothing below changes. const COORDINATOR_KEY = "__firstmateOpenCodeWatchArm"; // 35s on Windows so the budget stays above arm's MSYS confirm default (30s in // bin/fm-watch-arm.sh): a slow but successful Git Bash cold start must not be // SIGTERMed mid-confirmation. Conditioned on win32 so other platforms keep 12s. const ARM_READY_TIMEOUT_DEFAULT_MS = process.platform === "win32" ? 35000 : 12000; const ARM_READY_TIMEOUT_MS = positiveInteger("FM_OPENCODE_ARM_READY_TIMEOUT_MS", ARM_READY_TIMEOUT_DEFAULT_MS); +const HOST_READY_TIMEOUT_MS = Math.max(ARM_READY_TIMEOUT_MS, 30000); +const WAKE_LINE = /^(signal:|stale:|check:|heartbeat($|:))/; +const HOST_LINE = /^supervision-host:/; const ARM_RETIRE_TIMEOUT_MS = positiveInteger("FM_WATCH_ARM_RETIRE_TIMEOUT_MS", 1000); const REARM_RETRY_BASE_MS = positiveInteger("FM_WATCH_REARM_RETRY_BASE_MS", 250); const REARM_RETRY_MAX_MS = positiveInteger("FM_WATCH_REARM_RETRY_MAX_MS", 4000); @@ -23,6 +36,7 @@ let restorationInFlight = null; let armClose = new WeakMap(); let armReadiness = new WeakMap(); let armRecovery = new WeakMap(); +let armHostMode = new WeakMap(); function positiveInteger(name, fallback) { const value = Number(process.env[name]); @@ -37,8 +51,9 @@ function setArmStatus(status) { function waitForArmReady(armChild) { const readiness = armReadiness.get(armChild); if (!readiness) return Promise.resolve("failed"); + const timeout = armHostMode.get(armChild) ? HOST_READY_TIMEOUT_MS : ARM_READY_TIMEOUT_MS; return new Promise((resolve) => { - const timer = setTimeout(() => resolve("timeout"), ARM_READY_TIMEOUT_MS); + const timer = setTimeout(() => resolve("timeout"), timeout); timer.unref(); void readiness.then((status) => { clearTimeout(timer); @@ -130,9 +145,34 @@ async function sessionOwnsLock(paths) { return false; } -function classifyArmClose(stdout, stderr, code, signal) { +// The host-mode wake message: every "supervision-host:" line in order, wake +// lines capped at eight, and the away note while the posture record exists. +function hostWakeMessage(paths, combined) { + let shown = 0; + const lines = combined.split(/\r?\n/).filter((line) => { + if (HOST_LINE.test(line)) return true; + if (WAKE_LINE.test(line) && shown < 8) { + shown += 1; + return true; + } + return false; + }); + if (lines.length === 0) return ""; + if (existsSync(`${paths.state}/.afk-contract`)) { + lines.push("This wake comes from automatic supervision under the away-posture record, not from the captain: it is not a return, so handle it under the away posture."); + } + return lines.join("\n"); +} + +function classifyArmClose(paths, hostMode, stdout, stderr, code, signal) { const combined = `${stdout}\n${stderr}`; - const reason = combined.split(/\r?\n/).find((line) => /^(signal:|stale:|check:|heartbeat($|:))/.test(line)); + if (hostMode) { + const message = hostWakeMessage(paths, combined); + if (message) return { kind: "actionable", message }; + const stoodDown = combined.split(/\r?\n/).find((line) => /^supervision-host stood down:/.test(line)); + if (stoodDown) return { kind: "failure", message: `watcher: FAILED - ${stoodDown}` }; + } + const reason = combined.split(/\r?\n/).find((line) => WAKE_LINE.test(line)); if (reason) return { kind: "actionable", message: reason }; const healthy = combined.split(/\r?\n/).find((line) => /^watcher: healthy\b/.test(line)); if (healthy) { @@ -150,9 +190,10 @@ function classifyArmClose(stdout, stderr, code, signal) { }; } if (code && code !== 0) { + const script = hostMode ? "fm-supervision-host.sh" : "fm-watch-arm.sh"; return { kind: "failure", - message: `watcher: FAILED - fm-watch-arm.sh exited ${code}${combined.trim() ? `\n${combined.trim()}` : ""}`, + message: `watcher: FAILED - ${script} exited ${code}${combined.trim() ? `\n${combined.trim()}` : ""}`, }; } return { @@ -161,9 +202,9 @@ function classifyArmClose(stdout, stderr, code, signal) { }; } -function observeArmOutput(stdout, stderr, settleReadiness) { +function observeArmOutput(hostMode, stdout, stderr, settleReadiness) { const combined = `${stdout}\n${stderr}`; - if (combined.split(/\r?\n/).some((line) => /^(signal:|stale:|check:|heartbeat($|:))/.test(line))) { + if (combined.split(/\r?\n/).some((line) => WAKE_LINE.test(line) || (hostMode && HOST_LINE.test(line)))) { setArmStatus("wake"); settleReadiness("wake"); return; @@ -335,6 +376,7 @@ async function scheduleRetry(paths, sessionID, client, reason, predecessorArmPid function spawnArm(paths, sessionID, client, predecessorArmPid = "") { setArmStatus("starting"); + const hostMode = existsSync(`${paths.config}/supervision-host`); const env = { ...process.env, FM_HOME: paths.home, @@ -342,11 +384,14 @@ function spawnArm(paths, sessionID, client, predecessorArmPid = "") { FM_CONFIG_OVERRIDE: paths.config, FM_WATCH_PREDECESSOR_ARM_PID: predecessorArmPid, }; - const armChild = spawn("bash", ["-lc", 'config_dir="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}"; [ -f "$config_dir/x-mode.env" ] && . "$config_dir/x-mode.env"; exec "$FM_ROOT_OVERRIDE/bin/fm-watch-arm.sh" --restart'], { + if (hostMode) env.FM_SUPERVISION_HOST_PRIMARY = "opencode"; + const command = hostMode ? '"$FM_ROOT_OVERRIDE/bin/fm-supervision-host.sh" park --restart' : '"$FM_ROOT_OVERRIDE/bin/fm-watch-arm.sh" --restart'; + const armChild = spawn("bash", ["-lc", `config_dir="\${FM_CONFIG_OVERRIDE:-$FM_HOME/config}"; [ -f "$config_dir/x-mode.env" ] && . "$config_dir/x-mode.env"; exec ${command}`], { cwd: paths.root, env, stdio: ["ignore", "pipe", "pipe"], }); + armHostMode.set(armChild, hostMode); child = armChild; let stdout = ""; let stderr = ""; @@ -377,19 +422,19 @@ function spawnArm(paths, sessionID, client, predecessorArmPid = "") { armChild.stdout.on("data", (chunk) => { stdout += chunk.toString(); observeRecovery(); - observeArmOutput(stdout, stderr, settleReadiness); + observeArmOutput(hostMode, stdout, stderr, settleReadiness); }); armChild.stderr.on("data", (chunk) => { stderr += chunk.toString(); observeRecovery(); - observeArmOutput(stdout, stderr, settleReadiness); + observeArmOutput(hostMode, stdout, stderr, settleReadiness); }); armChild.on("close", (code, signal) => { if (settled) return; settled = true; resolveClosed(); releaseChild(); - const classification = classifyArmClose(stdout, stderr, code, signal); + const classification = classifyArmClose(paths, hostMode, stdout, stderr, code, signal); settleReadiness(classification.kind === "actionable" ? "wake" : "failed"); const predecessor = String(armChild.pid ?? ""); if (classification.kind === "actionable") { diff --git a/AGENTS.md b/AGENTS.md index 256e5c536de..44caa8ae2cd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,7 +78,7 @@ config/backlog-backend backlog backend override; LOCAL, gitignored; absent or " config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), herdr has its own required CI lane (docs/herdr-backend.md), while zellij, orca, and cmux remain experimental with no dedicated real-backend CI lane (docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning config/calm Calm presentation preference shared by the Pi extension and the Claude Code mod; LOCAL, gitignored, and not inherited; see docs/configuration.md "Calm preference" config/supervision-branch-model config/supervision-branch-effort Pi supervision-branch model and reasoning-effort pins written by /supervision-model; LOCAL, gitignored, independently settable, and not inherited; see docs/configuration.md "Pi supervision branch model and effort" -config/supervision-host optional opt-in to the supervision host, which runs the supervision branch's contract on a headless engine beside a Claude primary in the away posture; LOCAL, gitignored, not inherited; absent changes nothing; see docs/configuration.md "Supervision host" +config/supervision-host optional opt-in to the supervision host, which runs the supervision branch's contract on a headless engine beside a non-Pi primary in the away posture; LOCAL, gitignored, not inherited; absent changes nothing; see docs/configuration.md "Supervision host" config/startup-memory-budget primary-authoritative per-home startup-memory budget; LOCAL, gitignored, materialized as 7,500 estimated tokens by locked primary bootstrap and inherited into secondmate homes; see docs/configuration.md "Startup memory budget" config/stow-pass-horizon optional presence flag opting this home in to /stow's default-off pass-count decay horizon; LOCAL, gitignored, and not inherited; see docs/configuration.md "Stow pass horizon" config/herdr-presentation-spaces optional "off" opt-out from, or "on" opt-in to, Herdr's default-on disposable single-task visual projection, which is unconfigured-default-on only at or above a Herdr version floor; LOCAL, gitignored; inherited by secondmate homes; see docs/herdr-backend.md "Presentation spaces" @@ -479,7 +479,7 @@ Each skill owns its own daemon procedure, which is otherwise identical; these sa - `state/.afk-contract` is the away posture, written in the same turn as `/afk` before any other work, because `/afk` is itself the go: no read-back gates entry or waits for a go; entry announces hold-for-return only, and the away session acts on those words by its own judgment through the guarded scripts under standing authority, holding for the return on doubt. - While `state/.afk` exists, the daemon owns supervision; do not arm a separate watcher. The daemon is never launched on Pi, where the ordinary supervision session continues under the record with main parked: the branch takes every safe actionable wake it can, and only a declined wake (including a broken branch or unsafe scan) or a watcher failure wakes main. - Away mode on a Claude home with `config/supervision-host` works the same way with the supervision host as the branch; a wake it hands back arrives as Stop hook feedback and is never the captain's return. + Away mode on a non-Pi home with `config/supervision-host` works the same way with the supervision host as the branch; a wake it hands back arrives through that harness's own wake path and is never the captain's return. - A marked message while away or quiet mode is active is internal escalation and does not exit that mode. - A message beginning `/afk` refreshes away mode; a message beginning `/quiet` refreshes quiet mode. - Any other unmarked message means the captain returned in away mode (load `/afk`, run the return owner, and do not process that message as ordinary work until its durable catch-up gate clears), or, in quiet mode, is simply answered as ordinary work with the flag and daemon left untouched until an explicit `/quiet off`. diff --git a/README.md b/README.md index a4faabeaa13..3b9ab871e2f 100644 --- a/README.md +++ b/README.md @@ -183,7 +183,7 @@ Claude and grok use the slash form shown here; codex uses the same names with `$ | Skill | What it does | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | -| `/afk` | Enter away-mode supervision: Pi's in-process branch, an [opt-in Claude supervision host](docs/configuration.md#supervision-host-configsupervision-host), or the daemon handles wakes while you step away; see the [away procedure](.agents/skills/afk/SKILL.md) for the posture and return contract | +| `/afk` | Enter away-mode supervision: Pi's in-process branch, an [opt-in supervision host](docs/configuration.md#supervision-host-configsupervision-host) beside the other primaries, or the daemon handles wakes while you step away; see the [away procedure](.agents/skills/afk/SKILL.md) for the posture and return contract | | `/quiet` | Enter quiet supervision mode: the same token-saving sub-supervisor tradeoff as `/afk`, for a captain who is staying and chatting - ordinary messages do not exit it, only an explicit `/quiet off` does | | `/ahoy` | Recap visible session events since the prior real captain message plus visibly unanswered captain decisions, then guide the captain through any open decisions one at a time in agent-judged impact order; fall back to Bearings when invoked as the session's first real captain message | | `/bearings` | Generate a concise four-section chat digest from bounded fleet state, including registered remote-home ledgers and measured follow-up for owned contributions; use `/bearings file` to also replace today's dated report in `data/`, and add `include PRs` for live GitHub enrichment | diff --git a/bin/fm-afk-launch.sh b/bin/fm-afk-launch.sh index 6c4441b34c1..829d362146c 100755 --- a/bin/fm-afk-launch.sh +++ b/bin/fm-afk-launch.sh @@ -17,10 +17,12 @@ # On Pi and pi-signed the entry ENDS there: the away daemon is no longer launched # on Pi, the ordinary supervision session keeps running in both postures, and # `start` refuses on those harnesses. The same holds for away mode (not quiet -# mode) on a Claude primary whose home opted into the supervision host -# (config/supervision-host), where the host runs the away session. Every other -# harness still runs the daemon for now, so `start` and `start-native` require -# the record `enter` wrote before they launch the daemon. +# mode) on a claude, cursor, opencode, omp, grok, or codex primary whose home +# opted into the supervision host (config/supervision-host), where the host +# runs the away session; `enter` there adds one line when the host has no +# engine, because every away wake then reaches main. Every other harness still +# runs the daemon for now, so `start` and `start-native` require the record +# `enter` wrote before they launch the daemon. # `stop` (the return, driven by bin/fm-afk-return.sh) shuts the daemon down, # clears state/.afk last, and archives the record under state/afk-contracts/. # @@ -191,12 +193,21 @@ fm_afk_launch_primary_harness() { "$FM_AFK_LAUNCH_DIR/fm-harness.sh" 2>/dev/null || printf unknown } -# The away daemon is no longer launched on Pi, nor for away mode on a Claude -# primary whose home opted into the supervision host (config/supervision-host, +# The primary harnesses whose arm owner runs the supervision host when the +# home opted in (docs/supervision-host.md). +fm_afk_launch_host_primary() { # + case "$1" in + claude|cursor|opencode|omp|grok|codex) return 0 ;; + esac + return 1 +} + +# The away daemon is no longer launched on Pi, nor for away mode on a primary +# whose home opted into the supervision host (config/supervision-host, # docs/supervision-host.md): the posture record is the whole entry there and # the ordinary supervision session runs in both postures. Quiet mode still -# runs the daemon on that Claude home, so a quiet entry or a refresh of a -# running quiet daemon is allowed. +# runs the daemon on that home, so a quiet entry or a refresh of a running +# quiet daemon is allowed. fm_afk_launch_daemon_allowed() { local harness mode harness=$(fm_afk_launch_primary_harness) @@ -204,17 +215,34 @@ fm_afk_launch_daemon_allowed() { pi|pi-signed) fm_afk_launch_log "the away daemon is no longer launched on $harness; the away-posture record is the posture there (run bin/fm-afk-launch.sh enter and stop)" return 1 ;; - claude) - [ -f "${FM_CONFIG_OVERRIDE:-$FM_HOME/config}/supervision-host" ] || return 0 - mode=${FM_AFK_MODE:-} - if [ -z "$mode" ] && [ -f "$FM_AFK_LAUNCH_STATE/.afk" ]; then - mode=$(head -n 1 "$FM_AFK_LAUNCH_STATE/.afk" 2>/dev/null || true) - fi - [ "$mode" != quiet ] || return 0 - fm_afk_launch_log "the away daemon is not launched on this claude home, which runs the supervision host (config/supervision-host); the away-posture record is the posture here (run bin/fm-afk-launch.sh enter and stop)" - return 1 ;; esac - return 0 + fm_afk_launch_host_primary "$harness" || return 0 + [ -f "${FM_CONFIG_OVERRIDE:-$FM_HOME/config}/supervision-host" ] || return 0 + mode=${FM_AFK_MODE:-} + if [ -z "$mode" ] && [ -f "$FM_AFK_LAUNCH_STATE/.afk" ]; then + mode=$(head -n 1 "$FM_AFK_LAUNCH_STATE/.afk" 2>/dev/null || true) + fi + [ "$mode" != quiet ] || return 0 + fm_afk_launch_log "the away daemon is not launched on this $harness home, which runs the supervision host (config/supervision-host); the away-posture record is the posture here (run bin/fm-afk-launch.sh enter and stop)" + return 1 +} + +# One line for the entry when this home runs the supervision host but the host +# has no engine (bin/fm-supervision-engine-lib.sh owns the opt-in parse), so +# the away posture would hand every wake to main. +fm_afk_launch_host_engine_note() { + local harness config + [ "${FM_AFK_MODE:-}" != quiet ] || return 0 + config=${FM_CONFIG_OVERRIDE:-$FM_HOME/config} + [ -f "$config/supervision-host" ] || return 0 + harness=$(fm_afk_launch_primary_harness) + fm_afk_launch_host_primary "$harness" || return 0 + # shellcheck source=bin/fm-supervision-engine-lib.sh + . "$FM_AFK_LAUNCH_DIR/fm-supervision-engine-lib.sh" || return 0 + fm_supervision_host_config "$config" "$harness" || return 0 + [ -z "$FM_SUPERVISION_ENGINE" ] || return 0 + printf 'Supervision host: no engine runs the away session on this home (%s), so every away wake reaches this conversation; name a verified engine in config/supervision-host (for example "claude").\n' \ + "$FM_SUPERVISION_ENGINE_PROBLEM" } fm_afk_launch_catchup_pending() { @@ -240,7 +268,8 @@ fm_afk_launch_record_require() { fm_afk_launch_enter() { fm_afk_launch_catchup_pending && return 1 - "$FM_AFK_CONTRACT_CMD" enter "$@" + "$FM_AFK_CONTRACT_CMD" enter "$@" || return + fm_afk_launch_host_engine_note } # The command run inside the created terminal. Real launch runs the shared diff --git a/bin/fm-afk-return.sh b/bin/fm-afk-return.sh index c2e086b19a2..93dcd1f8684 100755 --- a/bin/fm-afk-return.sh +++ b/bin/fm-afk-return.sh @@ -539,7 +539,7 @@ EOF # 6. handled while away. Every outcome the away session recorded in the # store during the window counts as handled. On Pi the supervision branch, - # and on a Claude home the supervision host (docs/supervision-host.md), took + # and on an opted-in home the supervision host (docs/supervision-host.md), took # every safe actionable wake it could while main was parked; wakes it # declined still fell back to main. The captain rows are listed above. printf 'Handled while away:\n' diff --git a/bin/fm-branch-report.sh b/bin/fm-branch-report.sh index d0d997510d9..3d71640a3fb 100755 --- a/bin/fm-branch-report.sh +++ b/bin/fm-branch-report.sh @@ -28,6 +28,14 @@ # ended, or from any other shell, is refused. Exit codes: 0 recorded, 1 the # store refused or failed (nothing recorded), 2 usage, 3 refused (actor, turn, # or scope). +# +# A row recorded after the captain returned (the away-posture record is gone) +# may be missing from the return brief, so it is also queued for MAIN as a +# durable check wake keyed supervision-host-return:, presented by the +# drain until MAIN acknowledges it. bin/fm-afk-return.sh archives the record +# before it reads the store and this check follows the append, so every row is +# in the brief, queued, or both: the relay does not depend on the host +# surviving its turn or on its owner delivering the host's own handback. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -111,4 +119,15 @@ printf '%s\t%s\t%s\t%s\n' "$TURN" "$SEQ" "$VERDICT" "$TASK" >> "$RECEIPTS" || { echo "recorded seq $SEQ, but the host receipt could not be written; the host will hand this wake to MAIN" >&2 exit 1 } +if [ ! -f "$STATE/.afk-contract" ]; then + # shellcheck source=bin/fm-wake-lib.sh + . "$SCRIPT_DIR/fm-wake-lib.sh" + if ! fm_wake_append check "supervision-host-return:$SEQ" \ + "check: supervision-host outcome $SEQ for $TASK [$VERDICT] was recorded after the captain returned, so the return brief may not show it; relay it to the captain: $SUMMARY"; then + printf 'recorded seq %s [%s], but the captain has returned and its relay to MAIN could not be queued; the host hands this turn to MAIN\n' "$SEQ" "$VERDICT" >&2 + exit 0 + fi + printf 'recorded seq %s [%s]; the captain has returned, so it is queued for MAIN to relay\n' "$SEQ" "$VERDICT" + exit 0 +fi printf 'recorded seq %s [%s]; it waits in the outcome store for MAIN\n' "$SEQ" "$VERDICT" diff --git a/bin/fm-guard.sh b/bin/fm-guard.sh index ba9ee330465..ee922dc509e 100755 --- a/bin/fm-guard.sh +++ b/bin/fm-guard.sh @@ -38,7 +38,13 @@ # The ordinary warning also stays silent for the supervision branch # actor (FM_SUPERVISION_ACTOR=branch), because that actor runs guarded commands # while handling exactly the queued rows its grant covers and can drain nothing -# else. Always exits 0: the guard warns, it never blocks. +# else. The watcher-down banner and its reminder stay silent for that actor too, +# and its calls leave the episode state alone: the branch never owns watcher +# continuity (Pi main or the supervision host restarts the watcher once the +# branch's turn ends, and a successor cycle that closed on a newer wake mid-turn +# is that host's normal gap), while the repair line names the primary's own arm +# command, which under a supervision host's primary pin is the host itself or +# the plain arm. Always exits 0: the guard warns, it never blocks. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -201,8 +207,11 @@ fi # No fresh watcher with tasks in flight is the dangerous state: emit a prominent, # bordered banner FIRST so it reads as an alarm, not a buried stderr line. Later -# calls in the same episode get a one-line reminder only. -if [ "$watcher_healthy" = false ]; then +# calls in the same episode get a one-line reminder only. The supervision branch +# actor neither sees nor advances an episode (header). +if [ "$GUARD_ACTOR" = branch ]; then + : +elif [ "$watcher_healthy" = false ]; then episode_key=$(fm_guard_stale_episode_key "$watcher_down_reason") episode_key=${episode_key%$'\n'} print_full_banner=0 diff --git a/bin/fm-supervision-host.sh b/bin/fm-supervision-host.sh index e7284678f4f..19404fbbf98 100755 --- a/bin/fm-supervision-host.sh +++ b/bin/fm-supervision-host.sh @@ -4,13 +4,34 @@ # non-Pi primary (docs/supervision-host.md owns the design). # # Usage: -# fm-supervision-host.sh park +# fm-supervision-host.sh park [--restart] # # A primary's arm owner runs this in place of bin/fm-watch-arm.sh when the home -# opted in (config/supervision-host); today that owner is the Claude Stop -# auto-arm (bin/fm-claude-stop-autoarm.sh). To that owner it IS an arm: it -# prints the arm's own lines and exits only when main is needed, and stays -# parked across every close it handled itself. +# opted in (config/supervision-host): the Claude Stop auto-arm +# (bin/fm-claude-stop-autoarm.sh), the Cursor stop-hook park +# (bin/fm-turnend-guard-cursor.sh), the OpenCode TUI plugin +# (.opencode/plugins/fm-primary-watch-arm.js), the omp watch extension +# (.omp/extensions/fm-primary-omp-watch.ts), Grok's model-owned background arm +# (docs/supervision-protocols/grok.md), and Codex's foreground checkpoint +# (bin/fm-watch-checkpoint.sh). To that owner it IS an arm: it prints the +# arm's own lines and exits only when main is needed, and stays parked across +# every close it handled itself. Each owner passes its harness as +# FM_SUPERVISION_HOST_PRIMARY, which the engine carries as the primary pin. +# +# OUTPUT, the contract every owner reads. The first cycle's status line +# ("watcher: started ..." or "watcher: attached ...") is printed as soon as the +# arm prints it, so an owner that waits for arm readiness sees it at once; +# everything else is printed in one write when the host exits: the close as +# the arm printed it (without that status line), then any "supervision-host:" +# lines. A "supervision-host:" line is a wake in its own right (the park +# boundary prints nothing else); "supervision-host stood down: ..." means this +# session or generation no longer owns supervision and the owner stands down +# silently; an exit status above 128, or no output at all, means the host +# itself died and the owner retries it. Any other close is judged exactly as +# the arm's. --restart starts the first cycle with fm-watch-arm.sh --restart, +# and an FM_WATCH_PREDECESSOR_ARM_PID the owner passes reaches that first +# cycle only, for owners that start their own successor after every close +# (OpenCode, omp). # # THE LOOP. It owns watcher cycles through bin/fm-watch-arm.sh. On each # actionable close: @@ -39,10 +60,14 @@ # existed, so the host exits with the close, one "supervision-host:" line # naming them, and one line per outcome, for main to relay. The host injects # nothing and has no delivery path of its own; the owner's existing wake path -# is the only way main hears from it. +# is the only way main hears from it. That handoff is only a prompt: each +# outcome recorded after the return is already a durable queued wake +# (bin/fm-branch-report.sh), so it still reaches main when the host dies at the +# turn's end or its owner drops the handoff, as a superseded Cursor park does. # # THE PARK BOUNDARY. Claude drops the exit 2 of a Stop hook it terminated at -# the hook's configured timeout (docs/verification/supervision.md), and a host +# the hook's configured timeout (docs/verification/supervision.md), Cursor's +# stop hook carries the same tracked 28800-second registration, and a host # that handles its own wakes is not shortened by them, so the host ends its # own park before that timeout: after FM_SUPERVISION_HOST_PARK_SECONDS (default # 27000, under the tracked 28800-second registration) it stops this home's @@ -50,10 +75,13 @@ # owner delivers as an ordinary wake; main drains, acknowledges, and ends its # turn, and that turn end starts the next park. The boundary is checked on # every loop pass, however many closes are already waiting, and an away close -# whose engine turn could no longer finish before the boundary (the turn bound +# whose engine turn could still be running at the turn limit (the turn bound # plus the engine grace), judged when the close arrives and again just before # the turn starts, is not handled: the host exits through the same boundary -# with that close printed ahead of the line. +# with that close printed ahead of the line. The turn limit is the boundary +# itself unless the owner sets FM_SUPERVISION_HOST_PARK_LIMIT later: Codex's +# checkpoint, whose bound is the park itself rather than a harness timeout, +# lets a turn that starts before the boundary finish after it. # # OWNERSHIP. Before activation, every successor cycle, and every engine turn # the host proves this session still holds the fleet lock @@ -67,8 +95,10 @@ # the primary's harness pin, and this turn's report id, so every guarded # script applies the same partition, leases, and away relocation it applies to # the Pi branch. At activation the host stops anything a crashed predecessor -# left running (recorded with identities, never by name) and releases the -# branch actor's leases; it releases them again after every engine turn. +# left running (recorded with identities, never by name), including the +# engine descendants its turn recorded, removes that turn's files, and +# releases the branch actor's leases; it releases them again after every +# engine turn. # # STATE (all under state/, owned here): .supervision-host (this host's pid and # the processes it runs), .supervision-host-engine (the engine conversation: @@ -81,7 +111,9 @@ # # Tunables (environment): FM_SUPERVISION_HOST_PARK_SECONDS (27000; a positive # integer below the 28800-second registration, any other value is the default), -# FM_SUPERVISION_HOST_TURN_TIMEOUT (1200), FM_SUPERVISION_HOST_ROTATE_TURNS (20: +# FM_SUPERVISION_HOST_PARK_LIMIT (the park boundary; a later value below the +# registration lets turns run past the boundary up to it, any other value is +# the boundary), FM_SUPERVISION_HOST_TURN_TIMEOUT (1200), FM_SUPERVISION_HOST_ROTATE_TURNS (20: # a new engine conversation after this many turns; every main session start # also opens a new one), FM_SUPERVISION_HOST_READY_TIMEOUT (25: how long a # successor cycle may take to verify), FM_SUPERVISION_HOST_POLL (1). @@ -102,10 +134,17 @@ CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" # shellcheck source=bin/fm-supervision-engine-lib.sh . "$SCRIPT_DIR/fm-supervision-engine-lib.sh" +FIRST_ARM_RESTART=0 case "${1:-}" in - park) ;; + park) + case "$#:${2:-}" in + 1:) ;; + 2:--restart) FIRST_ARM_RESTART=1 ;; + *) echo "usage: fm-supervision-host.sh park [--restart]" >&2; exit 2 ;; + esac + ;; -h|--help) sed -n '2,/^set -u/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;; - *) echo "usage: fm-supervision-host.sh park" >&2; exit 2 ;; + *) echo "usage: fm-supervision-host.sh park [--restart]" >&2; exit 2 ;; esac numeric_or() { # @@ -116,6 +155,8 @@ GRACE=${FM_GUARD_GRACE:-$(fm_poll_derived_grace)} ENGINE_GRACE=$(numeric_or "${FM_SUPERVISION_ENGINE_GRACE:-}" 30) PARK_SECONDS=$(numeric_or "${FM_SUPERVISION_HOST_PARK_SECONDS:-}" 27000) [ "$PARK_SECONDS" -lt 28800 ] 2>/dev/null || PARK_SECONDS=27000 +PARK_LIMIT=$(numeric_or "${FM_SUPERVISION_HOST_PARK_LIMIT:-}" "$PARK_SECONDS") +{ [ "$PARK_LIMIT" -lt 28800 ] && [ "$PARK_LIMIT" -ge "$PARK_SECONDS" ]; } 2>/dev/null || PARK_LIMIT=$PARK_SECONDS TURN_TIMEOUT=$(numeric_or "${FM_SUPERVISION_HOST_TURN_TIMEOUT:-}" 1200) ROTATE_TURNS=$(numeric_or "${FM_SUPERVISION_HOST_ROTATE_TURNS:-}" 20) READY_TIMEOUT=$(numeric_or "${FM_SUPERVISION_HOST_READY_TIMEOUT:-}" 25) @@ -124,6 +165,9 @@ AUTOARM_GEN=${FM_SUPERVISION_HOST_AUTOARM_GEN:-} AUTOARM_OWNER=${FM_SUPERVISION_HOST_OWNER_PID:-} PRIMARY=${FM_SUPERVISION_HOST_PRIMARY:-} [ -n "$PRIMARY" ] || PRIMARY=$("$SCRIPT_DIR/fm-harness.sh" 2>/dev/null || printf unknown) +# The owner's predecessor arm belongs to the first cycle only. +OWNER_PREDECESSOR=${FM_WATCH_PREDECESSOR_ARM_PID:-} +case "$OWNER_PREDECESSOR" in *[!0-9]*) OWNER_PREDECESSOR= ;; esac unset FM_WATCH_PREDECESSOR_ARM_PID FM_SUPERVISION_ACTOR FM_BRANCH_REPORT_TURN HOST_RECORD="$STATE/.supervision-host" @@ -150,6 +194,14 @@ ENGINE_SUBSHELL= SUCCESSOR_PID= SUCCESSOR_OUT= ENGINE_RUNNING=0 +# The running turn's result and diagnostics files, removed by the cleanup when +# the host is stopped mid-turn. +TURN_RESULT= +TURN_ERRORS= +# The first cycle's status line, printed as soon as the arm prints it +# (header, OUTPUT) and left out of that cycle's close. +READY_PENDING=1 +READY_LINE= log_line() { # local tmp @@ -243,7 +295,15 @@ activate() { [ "$role" = arm ] && stop_recorded "$pid" "$identity" 10 done < "$HOST_RECORD" fi - rm -f "$STATE"/.supervision-host-arm.* "$TURN_FILE" 2>/dev/null || true + # A predecessor killed outright ran no cleanup: reap the engine descendants + # its turn recorded, then drop that turn's files. + local ledger + for ledger in "$STATE"/.supervision-host-descendants.*; do + case "$ledger" in *.pids|*.next) continue ;; esac + [ -f "$ledger" ] && _fm_engine_reap "$ledger" + done + rm -f "$STATE"/.supervision-host-arm.* "$STATE"/.supervision-host-descendants.* "$STATE"/.supervision-host-result.* \ + "$STATE"/.supervision-host-errors.* "$STATE"/.supervision-host-readback.* "$TURN_FILE" 2>/dev/null || true printf 'host\t%s\t%s\n' "$HOST_PID" "$(identity_of "$HOST_PID")" > "$HOST_RECORD" || return 1 release_branch_leases } @@ -268,7 +328,7 @@ stop_engine_turn() { # shellcheck disable=SC2329 # Invoked by the EXIT trap. cleanup() { - local rc=$? + local rc=$? f trap - EXIT HUP TERM INT if [ "$ENGINE_RUNNING" -eq 1 ]; then stop_engine_turn @@ -281,6 +341,9 @@ cleanup() { fi release_branch_leases rm -f "$TURN_FILE" "$ENGINE_PID_FILE" 2>/dev/null || true + for f in "$TURN_RESULT" "$TURN_ERRORS"; do + case "$f" in "$STATE"/.supervision-host-*) rm -f "$f" 2>/dev/null || true ;; esac + done if [ -f "$HOST_RECORD" ] && [ "$(awk -F '\t' '$1 == "host" { print $2; exit }' "$HOST_RECORD" 2>/dev/null)" = "$HOST_PID" ]; then rm -f "$HOST_RECORD" 2>/dev/null || true fi @@ -312,13 +375,14 @@ host_still_owner() { && [ "$FM_AUTOARM_OUTCOME" = arming ] } -start_arm() { # ; sets the started pid/output +start_arm() { # [--restart]; sets the started pid/output local predecessor=$1 out pid + shift out=$(mktemp "$STATE/.supervision-host-arm.XXXXXX") || return 1 if [ -n "$predecessor" ]; then - FM_WATCH_PREDECESSOR_ARM_PID=$predecessor FM_GUARD_GRACE="$GRACE" "$SCRIPT_DIR/fm-watch-arm.sh" >"$out" 2>&1 & + FM_WATCH_PREDECESSOR_ARM_PID=$predecessor FM_GUARD_GRACE="$GRACE" "$SCRIPT_DIR/fm-watch-arm.sh" "$@" >"$out" 2>&1 & else - FM_GUARD_GRACE="$GRACE" "$SCRIPT_DIR/fm-watch-arm.sh" >"$out" 2>&1 & + FM_GUARD_GRACE="$GRACE" "$SCRIPT_DIR/fm-watch-arm.sh" "$@" >"$out" 2>&1 & fi pid=$! record_process arm "$pid" @@ -330,9 +394,10 @@ boundary_reached() { [ $(( $(date +%s) - HOST_STARTED )) -ge "$PARK_SECONDS" ] } -# True when an engine turn started now could still be running at the boundary. +# True when an engine turn started now could still be running at the turn +# limit (the boundary unless the owner set a later one). turn_crosses_boundary() { - [ $(( $(date +%s) - HOST_STARTED + TURN_TIMEOUT + ENGINE_GRACE )) -ge "$PARK_SECONDS" ] + [ $(( $(date +%s) - HOST_STARTED + TURN_TIMEOUT + ENGINE_GRACE )) -ge "$PARK_LIMIT" ] } # End the park at the boundary: stop the current and successor arms and this @@ -346,22 +411,40 @@ boundary_exit() { SUCCESSOR_PID= SUCCESSOR_OUT= "$SCRIPT_DIR/fm-watch-arm.sh" --stop >/dev/null 2>&1 || true - print_close log_line "boundary after $(( $(date +%s) - HOST_STARTED ))s" - printf 'supervision-host: cycle boundary - the host ended its park before the Stop hook timeout; drain, acknowledge, and end the turn, and the next park starts on its own\n' + emit 'supervision-host: cycle boundary - the host ended its park at its bound; drain, acknowledge, and end the turn, and the next park starts on its own' exit 0 } +# Print the first cycle's status line once the arm has written it in full. +stream_ready_line() { + local complete line + complete=$(wc -l < "$ARM_OUT" 2>/dev/null | tr -d ' ') + case "$complete" in ''|0|*[!0-9]*) return 0 ;; esac + line=$(head -n "$complete" "$ARM_OUT" 2>/dev/null | grep -E -m 1 '^watcher: (started|attached) ' || true) + [ -n "$line" ] || return 0 + printf '%s\n' "$line" + READY_LINE=$line + READY_PENDING=0 +} + # Wait for the current arm to close. Returns 0 with ARM_TEXT set, # or 1 when the park boundary arrives first. await_close() { while fm_pid_alive "$ARM_PID"; do refresh_process "$ARM_PID" + [ "$READY_PENDING" -eq 0 ] || stream_ready_line boundary_reached && return 1 sleep "$POLL" done wait "$ARM_PID" 2>/dev/null || true ARM_TEXT=$(cat "$ARM_OUT" 2>/dev/null || true) + if [ -n "$READY_LINE" ]; then + # Already printed: drop its first occurrence from this first close. + ARM_TEXT=$(printf '%s\n' "$ARM_TEXT" | awk -v line="$READY_LINE" '!dropped && $0 == line { dropped = 1; next } { print }') + READY_LINE= + fi + READY_PENDING=0 forget_process "$ARM_PID" rm -f "$ARM_OUT" 2>/dev/null || true CLOSED_ARM_PID=$ARM_PID @@ -370,8 +453,15 @@ await_close() { return 0 } -print_close() { - [ -z "$ARM_TEXT" ] || printf '%s\n' "$ARM_TEXT" +# Print the close read so far, then the given lines, in one write (header, +# OUTPUT), so an owner reading a stream sees the whole exit at once. +emit() { # [line...] + local text=$ARM_TEXT line + for line in "$@"; do + [ -n "$line" ] || continue + text=${text:+$text$'\n'}$line + done + [ -z "$text" ] || printf '%s\n' "$text" } # Hand the close to main: stop the successor cycle (the state main's own turn @@ -384,10 +474,8 @@ exit_to_main() { # [further lines] SUCCESSOR_OUT= "$SCRIPT_DIR/fm-watch-arm.sh" --stop >/dev/null 2>&1 || true fi - print_close - printf 'supervision-host: %s\n' "$1" - [ -z "${2:-}" ] || printf '%s\n' "$2" log_line "to-main $1" + emit "supervision-host: $1" "${2:-}" exit 0 } @@ -414,9 +502,8 @@ turn_outcome_lines() { # } stand_down() { # - print_close - printf 'supervision-host stood down: %s\n' "$1" log_line "stand-down $1" + emit "supervision-host stood down: $1" exit 0 } @@ -576,6 +663,8 @@ handle_away() { # fi result=$(mktemp "$STATE/.supervision-host-result.XXXXXX") || result=/dev/null errors=$(mktemp "$STATE/.supervision-host-errors.XXXXXX") || errors=/dev/null + TURN_RESULT=$result + TURN_ERRORS=$errors ENGINE_RUNNING=1 # Backgrounded and waited, so a signal to the host is handled at once # instead of after the whole turn; the cleanup stops the engine. @@ -606,11 +695,13 @@ handle_away() { # receipts=$(awk -F '\t' -v turn="$turn" '$1 == turn { n++ } END { print n + 0 }' "$RECEIPTS" 2>/dev/null) usage=$(fm_supervision_engine_result "$FM_SUPERVISION_ENGINE" "$result" "${ENGINE_COST:-0}" 2>/dev/null || true) [ "$result" = /dev/null ] || rm -f "$result" + TURN_RESULT= if [ "$rc" -eq 0 ] && [ "${receipts:-0}" -gt 0 ] && [ -z "$unacked" ] \ && [ -n "$usage" ] && [ "${usage#error=0}" != "$usage" ]; then write_engine_record $((ENGINE_TURNS + 1)) "$(printf '%s\n' "$usage" | sed -n 's/.* conversation_cost=\([^ ]*\).*/\1/p')" \ || rm -f "$ENGINE_RECORD" [ "$errors" = /dev/null ] || rm -f "$errors" + TURN_ERRORS= log_line "handled turn=$turn rc=$rc reports=$receipts $usage $first" return 0 fi @@ -619,6 +710,7 @@ handle_away() { # rm -f "$ENGINE_RECORD" log_line "failed turn=$turn rc=$rc reports=${receipts:-0} unacked=${unacked:-none} ${usage:-no-result} $(head -c 300 "$errors" 2>/dev/null | tr '\t\n' ' ') $first" [ "$errors" = /dev/null ] || rm -f "$errors" + TURN_ERRORS= if fm_timed_out "$rc"; then HANDLE_WHY="the engine turn hit its ${TURN_TIMEOUT}s bound" elif [ "$rc" -eq 127 ]; then @@ -648,7 +740,11 @@ activate || { echo "supervision-host stood down: the host record could not be wr log_line "start gen=$GEN primary=$PRIMARY" # The first cycle. -start_arm "" || { echo "watcher: FAILED - the supervision host could not start a watcher cycle"; exit 1; } +if [ "$FIRST_ARM_RESTART" -eq 1 ]; then + start_arm "$OWNER_PREDECESSOR" --restart +else + start_arm "$OWNER_PREDECESSOR" +fi || { echo "watcher: FAILED - the supervision host could not start a watcher cycle"; exit 1; } ARM_PID=$STARTED_ARM_PID ARM_OUT=$STARTED_ARM_OUT @@ -663,18 +759,18 @@ while :; do # status above 128 tells the owner the host itself died. if [ -z "$REASON" ]; then log_line "pass-through a close without a wake" - print_close + emit exit 0 fi if [ -e "$STATE/.afk" ]; then log_line "pass-through the away daemon's flag exists $(printf '%s\n' "$REASON" | head -n 1)" - print_close + emit exit 0 fi # Attended: every wake is main's, as without the host. if [ ! -f "$STATE/.afk-contract" ]; then log_line "pass-through attended $(printf '%s\n' "$REASON" | head -n 1)" - print_close + emit exit 0 fi if ! host_still_owner; then diff --git a/bin/fm-supervision-instructions.sh b/bin/fm-supervision-instructions.sh index 913e2ef3e02..4d2d373bbde 100755 --- a/bin/fm-supervision-instructions.sh +++ b/bin/fm-supervision-instructions.sh @@ -1,10 +1,12 @@ #!/usr/bin/env bash # Render the primary-harness supervision operating block for session start and -# the short repair line used by guards and turn-end hooks. On a Claude primary -# whose home opted into the supervision host (config/supervision-host), the -# block adds one state line and the host's main-side protocol -# (docs/supervision-protocols/supervision-host.md); without that file the -# output is unchanged. +# the short repair line used by guards and turn-end hooks. On a non-Pi primary +# with a supervision protocol (claude, cursor, opencode, omp, grok, codex) whose +# home opted into the supervision host (config/supervision-host), the block +# adds one state line and the host's main-side protocol +# (docs/supervision-protocols/supervision-host.md, whose lines tagged +# "{,...} " render only for the listed harnesses), and Grok's arm +# command becomes the host; without that file the output is unchanged. set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -102,9 +104,15 @@ case "$HARNESS" in esac [ -f "$SNIPPET" ] || SNIPPET="$DOC_DIR/unknown.md" HOST_SNIPPET= -if [ "$HARNESS" = claude ] && [ -f "$CONFIG/supervision-host" ]; then - HOST_SNIPPET="$DOC_DIR/supervision-host.md" -fi +grok_arm='bin/fm-watch-arm.sh' +case "$HARNESS" in + claude|cursor|opencode|omp|grok|codex) + if [ -f "$CONFIG/supervision-host" ]; then + HOST_SNIPPET="$DOC_DIR/supervision-host.md" + grok_arm='bin/fm-supervision-host.sh park' + fi + ;; +esac checkpoint_seconds=${FM_CODEX_WATCH_CHECKPOINT:-180} pi_ext="$FM_ROOT/.pi/extensions/fm-primary-pi-watch.ts" @@ -126,14 +134,23 @@ if [ "$X_MODE" -eq 0 ] && [ -f "$x_mode_env" ]; then fi render_snippet() { # [snippet] - local line snippet=${1:-$SNIPPET} + local line tags snippet=${1:-$SNIPPET} while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + '{'*'} '*) + tags=${line%%\} *} + tags=${tags#\{} + case ",$tags," in *",$HARNESS,"*) ;; *) continue ;; esac + line=${line#*\} } + ;; + esac line=${line//__FM_PI_EXT__/$pi_ext} line=${line//__FM_PI_TURNEND_EXT__/$pi_turnend_ext} line=${line//__FM_OMP_EXT__/$omp_ext} line=${line//__FM_OMP_TURNEND_EXT__/$omp_turnend_ext} line=${line//__FM_X_MODE_ENV_SH__/$x_mode_env_sh} line=${line//__FM_X_MODE_ENV__/$x_mode_env} + line=${line//__FM_GROK_ARM__/$grok_arm} printf '%s\n' "$line" done < "$snippet" } @@ -177,7 +194,7 @@ repair_line() { printf '%s%s\n' "$prefix" 'repair missing watcher supervision by letting the OpenCode TUI plugin arm after idle; use bin/fm-watch-arm.sh only as a manual recovery probe if the plugin reports failure.' ;; grok) - printf '%s%s\n' "$prefix" 'repair missing watcher supervision with bin/fm-watch-arm.sh as its own Grok tracked background task, never shell &.' + printf '%s%s%s%s\n' "$prefix" 'repair missing watcher supervision with ' "$grok_arm" ' as its own Grok tracked background task, never shell &.' ;; cursor) printf '%s%s\n' "$prefix" 'watcher supervision is owned by the stop-hook park; inspect the hook registration and watcher startup path before ending the turn.' @@ -206,7 +223,7 @@ ordinary_wake_line() { printf '%s\n' '- Ordinary wake: the OpenCode TUI plugin already owns watcher continuity; do not arm manually.' ;; grok) - printf '%s\n' '- Ordinary wake: re-arm exactly one bin/fm-watch-arm.sh Grok tracked background task as directed below.' + printf '%s%s%s\n' '- Ordinary wake: re-arm exactly one ' "$grok_arm" ' Grok tracked background task as directed below.' ;; cursor) printf '%s\n' '- Ordinary wake: the stop-hook park (bin/fm-turnend-guard-cursor.sh) already owns watcher continuity; drain and handle the wake, and do not arm another cycle yourself.' diff --git a/bin/fm-turnend-guard-cursor.sh b/bin/fm-turnend-guard-cursor.sh index e09bdba3763..5c101c808e1 100755 --- a/bin/fm-turnend-guard-cursor.sh +++ b/bin/fm-turnend-guard-cursor.sh @@ -27,6 +27,16 @@ # 1. an actionable watcher wake from the park; # 2. the bounded repair instruction when supervision could not be established. # +# SUPERVISION HOST. A home opted in with config/supervision-host +# (docs/configuration.md "Supervision host" owns the opt-in) parks on +# bin/fm-supervision-host.sh in the arm's place, which takes away-posture wakes +# itself and exits only when main is needed; its header owns the output this +# park reads. A "supervision-host:" line is actionable like a wake line, and +# the follow-up carries every such line in order while wake lines keep the +# eight-line cap; "supervision-host stood down:" ends the park silently; a host +# that died without a close is retried instead of being judged by the +# healthy-watcher predicate. Without the file nothing below changes. +# # LOOP BOUNDING IS DOUBLE, because either bound alone is insufficient: # - `loop_limit` in .cursor/hooks.json is Cursor's own ceiling. Once # loop_count reaches it Cursor stops INVOKING this hook at all, so it is the @@ -297,6 +307,13 @@ ARM_PID= ACTIONABLE=0 HEALTHY=0 STAND_DOWN=0 +HOST_MODE=0 +HOST_RC=0 +ACTIONABLE_RE='^(signal:|stale:|check:|heartbeat($|:))' +if [ -f "$CONFIG/supervision-host" ]; then + HOST_MODE=1 + ACTIONABLE_RE='^(signal:|stale:|check:|heartbeat($|:)|supervision-host:)' +fi # Never leave an arm child or its capture file behind, on any exit path. trap '[ -n "$ARM_PID" ] && kill "$ARM_PID" 2>/dev/null; [ -n "$ARM_OUT" ] && rm -f "$ARM_OUT" 2>/dev/null; :' EXIT @@ -306,7 +323,9 @@ while [ "$attempt" -lt "$ARM_ATTEMPTS" ]; do current_session_still_ours || exit 0 attempt=$((attempt + 1)) ARM_OUT=$(mktemp "$STATE/.cursor-park-output.XXXXXX") || ARM_OUT= - if [ -n "$ARM_OUT" ]; then + if [ "$HOST_MODE" -eq 1 ]; then + FM_SUPERVISION_HOST_PRIMARY=cursor "$SCRIPT_DIR/fm-supervision-host.sh" park >"${ARM_OUT:-/dev/null}" 2>&1 & + elif [ -n "$ARM_OUT" ]; then "$SCRIPT_DIR/fm-watch-arm.sh" >"$ARM_OUT" 2>&1 & else "$SCRIPT_DIR/fm-watch-arm.sh" >/dev/null 2>&1 & @@ -326,7 +345,8 @@ while [ "$attempt" -lt "$ARM_ATTEMPTS" ]; do ARM_PID= exit 0 fi - wait "$ARM_PID" 2>/dev/null || true + HOST_RC=0 + wait "$ARM_PID" 2>/dev/null || HOST_RC=$? ARM_PID= # Away mode may have been entered while parked: the daemon owns triage now. @@ -334,10 +354,27 @@ while [ "$attempt" -lt "$ARM_ATTEMPTS" ]; do ACTIONABLE=0 if [ -n "$ARM_OUT" ]; then - grep -Eq '^(signal:|stale:|check:|heartbeat($|:))' "$ARM_OUT" 2>/dev/null && ACTIONABLE=1 + grep -Eq "$ACTIONABLE_RE" "$ARM_OUT" 2>/dev/null && ACTIONABLE=1 fi [ "$ACTIONABLE" -eq 1 ] && break + if [ "$HOST_MODE" -eq 1 ]; then + # The host stood down because this session no longer owns supervision: + # whoever does owns continuity now. + if [ -n "$ARM_OUT" ] && grep -q '^supervision-host stood down:' "$ARM_OUT" 2>/dev/null; then + exit 0 + fi + # A host that died without a close may have left its cycle running with + # no owner to deliver the close; retrying lets the next host stop what it + # left and own a fresh cycle, which the healthy-watcher predicate cannot. + if [ "$HOST_RC" -gt 128 ] || [ -z "$ARM_OUT" ] || [ ! -s "$ARM_OUT" ]; then + [ "$attempt" -lt "$ARM_ATTEMPTS" ] || break + [ -z "$ARM_OUT" ] || rm -f "$ARM_OUT" 2>/dev/null + ARM_OUT= + continue + fi + fi + # A non-actionable close is benign when another verified watcher already owns # this home and is still beating inside the shared grace window. if fm_watcher_healthy "$STATE" "$WATCH" "$GRACE" "$FM_HOME"; then @@ -357,7 +394,15 @@ if ! fm_supervision_needed "$STATE" "$GRACE"; then fi if [ "$ACTIONABLE" -eq 1 ]; then - WAKE=$(grep -E '^(signal:|stale:|check:|heartbeat)' "$ARM_OUT" 2>/dev/null | head -8) + if [ "$HOST_MODE" -eq 1 ]; then + WAKE=$(awk '/^supervision-host:/ { print; next } /^(signal:|stale:|check:|heartbeat)/ && shown++ < 8' "$ARM_OUT" 2>/dev/null) + if [ -e "$STATE/.afk-contract" ]; then + WAKE="$WAKE +This wake comes from automatic supervision under the away-posture record, not from the captain: it is not a return, so handle it under the away posture." + fi + else + WAKE=$(grep -E '^(signal:|stale:|check:|heartbeat)' "$ARM_OUT" 2>/dev/null | head -8) + fi emit_followup watcher "firstmate watcher wake - one supervision event needs a handling turn now. $WAKE diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index 8268bb917fa..3613d4335c3 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -139,16 +139,19 @@ write_rows_file_locked() { # _fm_atomic_replace "$source" "$target" } +# claim_main_rows_locked []: claim every unreserved queued row for main, +# or with a cutoff only the unreserved rows at or below it. Rows main already +# owns stay owned either way. claim_main_rows_locked() { DRAIN_TMP=$(mktemp "$STATE/.main-eligible-rows.tmp.XXXXXX") || return 1 - awk -F '\t' -v branch="$ELIGIBLE_ROWS_FILE" -v main="$MAIN_ROWS_FILE" ' + awk -F '\t' -v branch="$ELIGIBLE_ROWS_FILE" -v main="$MAIN_ROWS_FILE" -v cutoff="${1:-}" ' BEGIN { while ((getline line < branch) > 0) reserved[line]=1 while ((getline line < main) > 0) owned[line]=1 } NF >= 5 && $2 ~ /^[0-9]+$/ { present[$2]=1 - if (!($2 in reserved)) owned[$2]=1 + if (!($2 in reserved) && (cutoff == "" || $2 + 0 <= cutoff + 0)) owned[$2]=1 } END { for (seq in owned) if (seq in present) print seq } ' "$FM_WAKE_QUEUE" | LC_ALL=C sort -n > "$DRAIN_TMP" || return 1 @@ -650,13 +653,15 @@ if [ -n "$ACK_THROUGH" ]; then PRESENTED_MAX=$(presented_max_row "$MAIN_ROWS_FILE") || exit 1 fi if [ "$ACTOR" = main ]; then - # Preserve main's original whole-cutoff acknowledgement contract: rows may - # arrive after presentation but before the printed ack runs, and a direct - # or replayed main ack still owns every unreserved row through its cutoff. - # Claim again under the queue lock so those rows cannot be stranded merely - # because they were not present during the earlier drain. A live branch + # Preserve main's original whole-cutoff acknowledgement contract: a direct + # or replayed main ack still owns every unreserved row through its cutoff, + # so claim those again under the queue lock and none is stranded merely + # because it was not present during the earlier drain. A row above the + # cutoff arrived after presentation and was never shown to main, so it + # stays unowned for whichever actor presents it next; claiming it here + # would hand every later away-session wake back to main. A live branch # grant remains excluded by claim_main_rows_locked. - claim_main_rows_locked || exit 1 + claim_main_rows_locked "$ACK_THROUGH" || exit 1 fi if [ "$ACTOR" = branch ]; then # check-kind rows (inactive-outcome receipts, secondmate stall markers) diff --git a/bin/fm-watch-checkpoint.sh b/bin/fm-watch-checkpoint.sh index 35280f1f6f4..45162017f74 100755 --- a/bin/fm-watch-checkpoint.sh +++ b/bin/fm-watch-checkpoint.sh @@ -1,9 +1,26 @@ #!/usr/bin/env bash # Run one bounded foreground watcher checkpoint for harnesses that should not # rely on background-task completion to wake the model. +# +# SUPERVISION HOST. A home opted in with config/supervision-host +# (docs/configuration.md "Supervision host" owns the opt-in) runs +# bin/fm-supervision-host.sh in the watcher's place for the checkpoint's bound, +# as the host's park boundary; the host takes away-posture wakes itself and +# returns only when main is needed (its header owns the output read here). +# While the away-posture record state/.afk-contract exists, the bound is +# raised to FM_CODEX_WATCH_CHECKPOINT_AWAY (default 3600) when that is longer, +# so a parked main is not woken every few minutes; an engine turn that starts +# before the bound may finish after it. A close that carries a wake or a +# "supervision-host:" line other than the park boundary passes through as a +# wake; the boundary alone is the ordinary quiet checkpoint. Without the file +# nothing below changes. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" SECONDS_ARG=${FM_CODEX_WATCH_CHECKPOINT:-180} usage() { @@ -51,7 +68,7 @@ ERR=$(mktemp "${TMPDIR:-/tmp}/fm-watch-checkpoint.err.XXXXXX") || { } trap 'rm -f "$OUT" "$ERR"' EXIT -run_with_perl_timeout() { +run_with_perl_timeout() { # perl -e ' my $seconds = shift; my $pid = fork; @@ -77,20 +94,62 @@ run_with_perl_timeout() { waitpid $pid, 0; alarm 0; exit($? >> 8); - ' "$SECONDS_ARG" "$SCRIPT_DIR/fm-watch.sh" + ' "$@" } -set +e -if command -v timeout >/dev/null 2>&1; then - timeout "$SECONDS_ARG" "$SCRIPT_DIR/fm-watch.sh" >"$OUT" 2>"$ERR" - RC=$? -elif command -v gtimeout >/dev/null 2>&1; then - gtimeout "$SECONDS_ARG" "$SCRIPT_DIR/fm-watch.sh" >"$OUT" 2>"$ERR" - RC=$? -else - run_with_perl_timeout >"$OUT" 2>"$ERR" +run_bounded() { # + if command -v timeout >/dev/null 2>&1; then + timeout "$@" + elif command -v gtimeout >/dev/null 2>&1; then + gtimeout "$@" + else + run_with_perl_timeout "$@" + fi +} + +positive_or() { # + case "$1" in ''|0*|*[!0-9]*) printf '%s\n' "$2" ;; *) printf '%s\n' "$1" ;; esac +} + +if [ -f "$CONFIG/supervision-host" ]; then + BOUND=$SECONDS_ARG + if [ -f "$STATE/.afk-contract" ]; then + AWAY_BOUND=$(positive_or "${FM_CODEX_WATCH_CHECKPOINT_AWAY:-}" 3600) + [ "$AWAY_BOUND" -le "$BOUND" ] 2>/dev/null || BOUND=$AWAY_BOUND + fi + # The host's park boundary stays below the 28800-second registration. + [ "$BOUND" -lt 27000 ] 2>/dev/null || BOUND=27000 + LIMIT=$(( BOUND + $(positive_or "${FM_SUPERVISION_HOST_TURN_TIMEOUT:-}" 1200) + $(positive_or "${FM_SUPERVISION_ENGINE_GRACE:-}" 30) )) + set +e + # The host ends its own park; the outer bound only catches a host that + # outlived every one of its own bounds. + FM_SUPERVISION_HOST_PRIMARY=codex FM_SUPERVISION_HOST_PARK_SECONDS=$BOUND FM_SUPERVISION_HOST_PARK_LIMIT=$LIMIT \ + run_bounded $((LIMIT + 120)) "$SCRIPT_DIR/fm-supervision-host.sh" park >"$OUT" 2>"$ERR" RC=$? + set -e + if grep -E '^(signal:|stale:|check:|heartbeat($|:)|supervision-host:)' "$OUT" 2>/dev/null \ + | grep -Ev '^supervision-host: cycle boundary' >/dev/null; then + grep -Ev '^watcher: (started|attached) ' "$OUT" + [ ! -s "$ERR" ] || cat "$ERR" >&2 + exit 0 + fi + if grep -E '^supervision-host: cycle boundary' "$OUT" >/dev/null 2>&1; then + printf 'checkpoint: no actionable wake within %ss\n' "$BOUND" + exit 124 + fi + [ ! -s "$OUT" ] || cat "$OUT" + [ ! -s "$ERR" ] || cat "$ERR" >&2 + if [ "$RC" -eq 124 ]; then + echo "checkpoint: the supervision host outlived its own bound of ${BOUND}s" >&2 + exit 1 + fi + [ "$RC" -ne 0 ] || RC=1 + exit "$RC" fi + +set +e +run_bounded "$SECONDS_ARG" "$SCRIPT_DIR/fm-watch.sh" >"$OUT" 2>"$ERR" +RC=$? set -e if grep -E '^(signal:|stale:|check:|heartbeat($|:))' "$OUT" >/dev/null 2>&1; then diff --git a/docs/architecture.md b/docs/architecture.md index af9b3a6e67c..52616c70d47 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -141,7 +141,7 @@ On a Pi primary, supervision is default-on: the watcher extension can hand eligi The branch handles those rows, stores the outcome durably, and merges it back into main. A captain-facing outcome persists as one exact, sequence-keyed visible transcript entry and then opens one sequence-keyed processing turn on main, which only main's sequence-bound acknowledgement closes. [docs/pi-supervision-branch.md](pi-supervision-branch.md) owns row eligibility, dispatch architecture, deterministic outcome delivery, and processing re-presentation, while the generated [Pi supervision protocol](supervision-protocols/pi.md) owns MAIN's merged-event handling and acknowledgement duty. -For the opt-in Claude away-posture exception to the other harnesses' wake-to-main path, see [supervision-host.md](supervision-host.md). +For the opt-in away-posture exception to the non-Pi harnesses' wake-to-main path, see [supervision-host.md](supervision-host.md). ### Registered secondmate current state @@ -184,7 +184,7 @@ What stays mechanical is exactly what a script can check without reading words: The record's presence is the posture on every harness, `bin/fm-afk-launch.sh` owns entry and exit, and `bin/fm-afk-return.sh` archives the record and owns the return brief's ordered sections, including landed live task records that still owe cleanup, rendered from durable state. While the record exists neither supervisor rechecks an item held for the captain, and a declared external wait names when it clears with `until` for a condition-aware recheck in both postures that occurs at the declared time or the hours-long `FM_PAUSE_RESURFACE_SECS` bound, whichever comes first. On Pi and pi-signed the away daemon is no longer launched: the ordinary supervision session continues under the record with main parked, so the supervision branch takes every actionable wake, captain outcomes accumulate for the return brief, and main's standing authority relocates to the branch through the guarded scripts, each keeping its own gate ([`pi-supervision-branch.md`](pi-supervision-branch.md#postures)); a wake the branch cannot take and a watcher failure still reach main. -On an opted-in Claude home, the [supervision host](supervision-host.md) runs the away session instead of the daemon. +On an opted-in non-Pi home, the [supervision host](supervision-host.md) runs the away session instead of the daemon. A presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) still extends walk-away supervision on the remaining harnesses: the `/afk` skill starts it through the tracked foreground helper `bin/fm-afk-start.sh` once the record exists, after which the watcher reverts to daemon-managed one-shot mode and the daemon self-handles routine wakes in bash. The watcher and daemon share `bin/fm-classify-lib.sh` for captain-relevant status verbs, declared-wait vocabulary (a `paused:` external wait and a verified `captain-held` transfer alike, through one combined predicate), and status-scan primitives. Terminal verbs remain captain-relevant, while a nonterminal progress verb cannot become terminal merely because its prose contains a legacy free-text token such as `merged`; bare legacy free-text lines remain compatible. @@ -499,4 +499,4 @@ Use `/stow` before an intentional reset when the conversation may hold durable k ## Development notes The current watcher reliability work combines always-on bash triage with a durable queue for actionable wakes, generation-bound post-handling acknowledgement, deterministic re-arm recovery after watcher downtime, a race-proof singleton lock, duplicate self-eviction, drain-time liveness assertion, and a self-verifying tracked-child arm wrapper. -The away posture is the record `bin/fm-afk-contract.sh` owns; see [supervision-host.md](supervision-host.md) for the opt-in Claude away session and the `/afk` skill for the remaining daemon-backed harnesses. +The away posture is the record `bin/fm-afk-contract.sh` owns; see [supervision-host.md](supervision-host.md) for the opt-in non-Pi away session and the `/afk` skill for the remaining daemon-backed harnesses. diff --git a/docs/configuration.md b/docs/configuration.md index f72b6ba3156..0ac83242a3e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -99,13 +99,16 @@ Both choices are local to each Firstmate home and are not part of secondmate inh ## Supervision host (config/supervision-host) The optional local, gitignored `config/supervision-host` opts this home into the supervision host, which runs the supervision branch's contract on a headless engine session beside a non-Pi primary; [docs/supervision-host.md](supervision-host.md) owns the design, its current scope, and the verified engines. -Today only a Claude primary runs it, and only for the away posture: with the file present, the Claude Stop hook runs the host in the watcher arm's place, the host handles wakes on the engine while the away-posture record `state/.afk-contract` exists, and `/afk` launches no away daemon on that home, while `/quiet` still does. +Today a Claude, Cursor, OpenCode, omp, Grok, or Codex primary runs it, and only for the away posture: with the file present, that primary's arm owner runs the host in the watcher arm's place, the host handles wakes on the engine while the away-posture record `state/.afk-contract` exists, and `/afk` launches no away daemon on that home, while `/quiet` still does. Absence leaves the home exactly as it is without the host, on every harness; a Pi primary keeps its in-process supervision branch whether or not the file exists. +A Grok primary reads the file when its session-start block renders, so a change takes effect at its next session start; every other owner reads it at every arm. The file may be empty, or hold one line ` []`: - empty or `default` selects the primary harness's own engine at that engine's default model (`sonnet` for the Claude engine); - ` []` names a verified engine, currently only `claude`, and optionally the engine's own model name or alias; `default ` selects the primary harness's engine with that model. +Only Claude has a verified engine of its own, so a Cursor, OpenCode, omp, Grok, or Codex home names `claude` in the file. + An engine that is not verified, a primary with no verified engine, or a malformed line leaves the host with no engine: it takes no wake, every wake reaches main as it would without the host, and each away-posture wake carries a line naming the problem. The file is read at every wake, so a change applies at the next one without a restart. It is local to each home and not part of secondmate inherited configuration. @@ -1216,6 +1219,7 @@ FM_PROCEVENT_LAUNCH_FLOOR_SECONDS=1 # minimum interval between launches of o FM_PROCEVENT_LAUNCH_CONFIRM_SECONDS=3 # how long reconcile waits for the runners it started to prove they are running; 1..600, keep well below FM_POLL FM_WHEN_OUTPUT_TAIL_BYTES=8192 # bound on the command-output tail inside one condition->action outcome document FM_CODEX_WATCH_CHECKPOINT=180 # seconds per foreground watcher checkpoint in Codex primary supervision +FM_CODEX_WATCH_CHECKPOINT_AWAY=3600 # requested away checkpoint bound on a home with config/supervision-host; longer of this and attended bound, capped at 27000 FM_CREW_STATE_NM_TIMEOUT=10 # seconds allowed per no-mistakes query inside fm-crew-state.sh, and per state-database run-inventory read behind a capped AXI overview FM_TEARDOWN_NM_TIMEOUT=10 # seconds allowed per no-mistakes query or abort inside fm-teardown.sh FM_CREW_STATE_RUNS_LIMIT=200 # plain runs-ledger rows scanned for fallback attribution; does not change the CLI's AXI overview window (selection owner: bin/fm-nm-run-lib.sh) diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index 64b0fa77a1c..b0d507f4df5 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -332,7 +332,7 @@ The pane-independent max-defer alert is configured in [`wedge-alarm.md`](wedge-a Harnesses with native tracked background execution can run the daemon in their terminal. Pi and pi-signed no longer launch the away daemon; their ordinary supervision session continues under the posture record. -An opted-in Claude home also skips the daemon for `/afk`; see [supervision-host.md](supervision-host.md). +An opted-in non-Pi home also skips the daemon for `/afk`; see [supervision-host.md](supervision-host.md). For another harness without native tracked background execution, `bin/fm-afk-launch.sh` creates a dedicated unfocused Herdr workspace, runs the daemon there with an explicit supervisor target and backend, records the exact daemon pane, and closes only that pane on stop. It never splits the captain's active tab and never uses shell `&`. Recovery reconciles only the recorded exact id. diff --git a/docs/pi-supervision-branch.md b/docs/pi-supervision-branch.md index 265b9a31c31..94c790160ab 100644 --- a/docs/pi-supervision-branch.md +++ b/docs/pi-supervision-branch.md @@ -20,7 +20,7 @@ This in-process supervision branch is Pi-only by construction: A home on any harness that already has an outcome store still receives the shared drain compatibility recovery described in [Lost-wake outcome backstop](#lost-wake-outcome-backstop). - It does not change which harness is primary and never moves a home to Pi. -On an opted-in Claude home, the supervision host runs the away branch beside the primary; [supervision-host.md](supervision-host.md) owns its scope and mechanism. +On an opted-in non-Pi home, the supervision host runs the away branch beside the primary; [supervision-host.md](supervision-host.md) owns its scope and mechanism. ## Components and their owners diff --git a/docs/supervision-host.md b/docs/supervision-host.md index e249869b3d0..e5a28c4ba36 100644 --- a/docs/supervision-host.md +++ b/docs/supervision-host.md @@ -8,26 +8,40 @@ It is one architecture with Pi's, not a second one: the same branch prompt, the The host is opt-in per home through `config/supervision-host`; [configuration.md](configuration.md#supervision-host-configsupervision-host) owns the file. Without the file every home behaves exactly as it does without the host. -Today it runs only on a Claude primary and only takes wakes in the away posture: +Today it runs beside a Claude, Cursor, OpenCode, omp, Grok, or Codex primary and only takes wakes in the away posture: - Attended (no away-posture record `state/.afk-contract`), the host is a pass-through: every close reaches main exactly as the plain watcher arm delivers it. - Away (the record exists), the host hands each close to the engine, and main stays parked unless the host hands the wake back. -- `/afk` launches no away daemon on an opted-in Claude home, because the host is the away session there; `/quiet` still launches the daemon, and while its flag `state/.afk` exists the host stands aside exactly as the plain arm does. +- `/afk` launches no away daemon on an opted-in home of those harnesses, because the host is the away session there; `/quiet` still launches the daemon, and while its flag `state/.afk` exists the host stands aside exactly as the plain arm does. - Pi keeps its in-process branch whether or not the file exists, and no Pi engine is built. +- Kimi has no primary supervision protocol, so it has no arm owner to run the host. -Attended supervision on the host, other primary harnesses, `/quiet` on the host, and the daemon's retirement are later steps of the same design; until they land, their current behavior stays as described in their own owners. +Attended supervision on the host, `/quiet` on the host, and the daemon's retirement are later steps of the same design; until they land, their current behavior stays as described in their own owners. ## Components and their owners - The loop: `bin/fm-supervision-host.sh`, whose header owns the per-close order, the park boundary, ownership checks, predecessor cleanup, state files, and tunables. -- The arm owner: `bin/fm-claude-stop-autoarm.sh` runs the host in place of `bin/fm-watch-arm.sh` for an opted-in home, inside its existing single-flight generation, and delivers the host's output through the same exit-2 rewake; its header owns how host output is classified. +- The arm owners: each primary's existing arm owner runs the host in place of its watcher command for an opted-in home and delivers a handed-back wake through the wake path that harness already trusts; the host's header owns the output contract they read. + + | Primary | Arm owner | A handed-back wake reaches main as | + |---|---|---| + | Claude | the Stop auto-arm, `bin/fm-claude-stop-autoarm.sh`, inside its single-flight generation | the hook's exit-2 rewake (`Stop hook feedback`) | + | Cursor | the `stop` hook park, `bin/fm-turnend-guard-cursor.sh` | the park's `watcher` follow-up | + | OpenCode | the TUI plugin, `.opencode/plugins/fm-primary-watch-arm.js`, which restarts its own successor after each close | a `watcher` prompt through `promptAsync` | + | omp | the watch extension, `.omp/extensions/fm-primary-omp-watch.ts`, which restarts its own successor after each close | the extension's `watcher` follow-up | + | Grok | the model's tracked background call, rendered as `bin/fm-supervision-host.sh park` at session start | the background task's completion notification | + | Codex | the foreground checkpoint, `bin/fm-watch-checkpoint.sh`, in the watcher's place | the checkpoint's own output | + + Hook, plugin, extension, and checkpoint owners pass their harness as the primary pin; Grok's model-owned call relies on primary detection. + The host pins dispatched work to the primary's crew harness rather than the engine's. + Grok's arm command is fixed when the session-start block renders, so adding or removing the file on a Grok home takes effect at the next session start; the other owners read the file at every arm. - The engine: `bin/fm-supervision-engine-lib.sh` owns the opt-in parse, the verified-engine list, and one bounded engine turn, including the reap of engine tool processes that outlive it. - Row eligibility: `bin/fm-branch-dispatch.mjs` is the command entry to `.pi/extensions/lib/fm-branch-dispatch.ts`, so the host and the Pi extension compute branch-claimable rows and their task scope from one owner; it also renders the wake message with the same away-posture tail. - The grant and the drain: `bin/fm-wake-grant.sh` publishes the branch's rows bound to the host's own process, and [watcher-continuity.md](watcher-continuity.md#per-actor-acknowledgement) owns the per-actor drain and acknowledgement the engine runs. - The prompt: `bin/fm-branch-prompt.sh` emits the same byte-stable prompt the Pi branch runs; each wake names its host's report surface. -- The report surface: `bin/fm-branch-report.sh` is the command twin of the Pi branch's `fm_branch_report` tool, with the same task scoping, and it appends to the outcome store (`bin/fm-branch-outcome.sh`) plus a per-turn receipt the host requires. +- The report surface: `bin/fm-branch-report.sh` is the command twin of the Pi branch's `fm_branch_report` tool, with the same task scoping, and it appends to the outcome store (`bin/fm-branch-outcome.sh`) plus a per-turn receipt the host requires; a row recorded after the captain returned is also queued for main as a durable check wake. - Leases and authority: `bin/fm-lease-lib.sh` owns the per-task leases, the main-owned role partition, and the away relocation; the host's engine runs with `FM_SUPERVISION_ACTOR=branch`, the session-lock holder as `FM_LEASE_HOLDER_PID`, and the primary's harness pin, so every guarded script treats it exactly as it treats the Pi branch. -- The main side: [supervision-protocols/supervision-host.md](supervision-protocols/supervision-host.md) is what main reads at session start on an opted-in Claude home. +- The main side: [supervision-protocols/supervision-host.md](supervision-protocols/supervision-host.md) is what main reads at session start on an opted-in home, rendered for its harness. ## One away wake @@ -37,29 +51,36 @@ The engine drains, handles, reports through `bin/fm-branch-report.sh`, and ackno The host counts the wake handled only when the turn exited cleanly, recorded at least one report, and left none of its granted rows in the wake queue; it releases the branch's leases and grant either way and parks on the successor only for a handled wake. A handled wake never reaches main, whether its outcome was routine or captain: captain outcomes wait in the outcome store, and the return brief (`bin/fm-afk-return.sh`) presents them. The one exception is a captain who returns while a turn is still running: the return brief was rendered before that turn's outcomes existed, so the host hands the close to main with those outcomes for main to relay, whether or not the turn handled its wake. +That handoff is only the prompt delivery: each outcome recorded after the return is already a queued `check` wake, because the return owner archives the record before it reads the store and the report surface queues any row it records once the record is gone. +So the outcome reaches main's drain even when the handoff is lost, as when a Cursor park superseded by the return turn's own end stops its host as the engine turn finishes. ## Failure direction Every path that cannot finish an away wake on the engine hands that wake to main, with one `supervision-host: ` line after the close. -Before handing it back, the host stops its successor cycle, so main's next turn end starts from the same state as without the host and the wake stays durable in the queue. +Before handing it back, the host stops its successor cycle, so the owner's next arm starts from the same state as without the host and the wake stays durable in the queue. That covers an unverified successor, a refused handoff, an unreadable queue, rows main already claimed, a missing engine or node, a turn that timed out or failed, a turn that recorded no report, and a turn that reported but left any of its granted rows unacknowledged. The last names those rows, which stay durable in the queue for main's drain. A turn that fails also starts the next wake on a fresh engine conversation. When the captain returned during a failed turn that recorded outcomes, the handback carries those outcomes too, for main to relay. When the host loses session-lock ownership or its auto-arm generation, it stands down silently and leaves continuity to whoever owns it now. A host that starts without that ownership stands down before activation, so it never stops the owner's host or watcher or releases its leases. -A host that dies without a close is retried by the auto-arm, and the next host stops, by recorded identity, whatever its predecessor left running before it arms. +A host that dies without a close is retried by its owner (Grok's model and Codex's checkpoint see it as a failed cycle and start the next one), and the next host stops, by recorded identity, whatever its predecessor left running, including the engine descendants a killed turn recorded, and removes that turn's files before it arms. ## The park boundary -Claude drops the exit 2 of a Stop hook it terminated at the hook timeout ([verification](verification/supervision.md#claude-drops-the-exit-2-of-a-hook-it-timed-out-2026-09-23)). -A plain watcher park rarely lasts that long, because heartbeat closes wake main, but a host absorbs its own wakes, so it ends its park itself before the tracked 28,800-second registration. +Claude drops the exit 2 of a Stop hook it terminated at the hook timeout ([verification](verification/supervision.md#claude-drops-the-exit-2-of-a-hook-it-timed-out-2026-09-23)), and Cursor's `stop` hook carries the same tracked 28,800-second registration. +A plain watcher park rarely lasts that long, because heartbeat closes wake main, but a host absorbs its own wakes, so it ends its park itself before that registration. `FM_SUPERVISION_HOST_PARK_SECONDS` sets that boundary (default 27,000), and a value that is not a positive integer below 28,800 is treated as the default. -At the boundary it stops the home's watcher and exits with one `supervision-host: cycle boundary` line; main drains, acknowledges, and ends its turn, and that turn end starts the next park. +The OpenCode, omp, and Grok owners have no hook timeout and keep the same default, so their parks end on the same cadence. +At the boundary it stops the home's watcher and exits with one `supervision-host: cycle boundary` line; main drains and acknowledges, and the owner starts the next park (at the next turn end on Claude and Cursor, at once for OpenCode and omp, and at the model's re-arm on Grok). The host checks the boundary on every loop pass, so closes that are already waiting cannot carry it past the boundary. It also starts no engine turn that could still be running at the boundary (the turn bound plus the engine grace), judged when the close arrives and again just before the turn starts: that close reaches main ahead of the boundary line instead, and its wake stays durable in the queue. One short main turn per boundary is the cost of never losing the park silently. +Codex has no asynchronous wake, so its checkpoint's own bound is the park: the checkpoint passes it as the boundary and reports the boundary as its ordinary quiet line (`checkpoint: no actionable wake within s`). +Attended the bound stays `FM_CODEX_WATCH_CHECKPOINT` (default 180 seconds); while the away record exists it is raised to `FM_CODEX_WATCH_CHECKPOINT_AWAY` (default 3,600) if longer, then capped at 27,000 seconds so a parked main is not woken every few minutes. +Because that bound is not a harness timeout, the checkpoint also sets `FM_SUPERVISION_HOST_PARK_LIMIT`, which lets an engine turn that starts before the boundary finish after it; a captain message typed during the park waits for the checkpoint to return, at most the bound plus one engine turn, unless the captain interrupts it. + ## Engine conversations The engine keeps one conversation across wakes so the byte-stable prompt stays cached, keyed to the current main session: every main session start opens a new one, and so does every `FM_SUPERVISION_HOST_ROTATE_TURNS` turns, because each wake adds history and the per-wake cost grows with it. @@ -83,9 +104,11 @@ Today the only verified engine is Claude's print mode, measured on Claude Code 2 - From inside the engine's shell the primary is not in the harness ancestry, so the engine can never act as the session-lock owner. The default model is `sonnet`, which handled every measured wake correctly at a fraction of a larger model's cost; `config/supervision-host` can name another. +The Claude engine runs beside any of the six primaries, but only a Claude primary selects it by default: a Cursor, OpenCode, omp, Grok, or Codex home names it (`claude`, optionally with a model) in `config/supervision-host`, and `/afk` there says so when the file selects no engine. ## Verification `tests/fm-supervision-host.test.sh` drives the real host, auto-arm, grant, drain, report, and lease scripts against a stub engine. +Each arm owner's own suite covers its host mode against a stub host: `tests/fm-claude-stop-autoarm.test.sh`, `tests/fm-cursor-primary.test.sh`, `tests/fm-pi-watch-extension.test.sh` (the OpenCode plugin), `tests/fm-omp-harness.test.sh`, `tests/fm-watch-checkpoint.test.sh`, and `tests/fm-supervision-instructions.test.sh` (the rendered protocol, including Grok's arm command). `tests/fm-supervision-host-live-e2e.test.sh` runs a real engine turn and is opt-in because it spends tokens. [verification/supervision.md](verification/supervision.md#supervision-host) records the dated live results. diff --git a/docs/supervision-protocols/grok.md b/docs/supervision-protocols/grok.md index 305e1802a16..98be3e1b722 100644 --- a/docs/supervision-protocols/grok.md +++ b/docs/supervision-protocols/grok.md @@ -7,7 +7,7 @@ When this session owns supervision and away mode is not active: 3. First cycle: arm with Grok's tracked background tool, as its own call: `run_terminal_command` with `background: true` on: - `[ -f __FM_X_MODE_ENV_SH__ ] && . __FM_X_MODE_ENV_SH__; exec bin/fm-watch-arm.sh` + `[ -f __FM_X_MODE_ENV_SH__ ] && . __FM_X_MODE_ENV_SH__; exec __FM_GROK_ARM__` 4. Trust only the arm's one-line status. 5. `watcher: started ...` or `watcher: attached ...` means a live cycle exists. @@ -25,7 +25,7 @@ When you see a background-task-completed system reminder for the arm: 1. Run `bin/fm-wake-drain.sh` first. 2. Optionally fetch arm output with `get_command_or_subagent_output()` for the reason line. 3. Handle `signal`, `stale`, `check`, or `heartbeat` using the harness-neutral contract in `AGENTS.md`. -4. Ordinary wake: re-arm the next cycle with the same background `bin/fm-watch-arm.sh` call if the home still needs supervision, as `bin/fm-supervision-lib.sh` defines it. +4. Ordinary wake: re-arm the next cycle with the same background `__FM_GROK_ARM__` call if the home still needs supervision, as `bin/fm-supervision-lib.sh` defines it. 5. Do not invent a wake from an attach-status line alone. Drain the queue and act only on real wake records, the drain's `OPEN DECISIONS` and `UNREAD STATUS` entries, or a real watcher reason line. Re-arm attaches to an existing healthy cycle when one is already present and follows its verified successor chain. @@ -35,5 +35,5 @@ The primary project Stop hook runs `bin/fm-turnend-guard-grok.sh` as a backstop, [`turnend-guard.md`](../turnend-guard.md) owns its running-payload capability selection between native same-process blocking and the pre-native bounded resume fallback. After any forced continuation, arm the watcher with the background protocol above. -Interactive TUI primary sessions are the supported supervision host. +Interactive TUI sessions are the supported Grok primary surface. Headless `grok -p` may wait for background process exit but does not reliably surface full auto-wake model output; do not run the primary firstmate as a one-shot headless process. diff --git a/docs/supervision-protocols/omp.md b/docs/supervision-protocols/omp.md index eddacc3ff6d..8548475c044 100644 --- a/docs/supervision-protocols/omp.md +++ b/docs/supervision-protocols/omp.md @@ -23,7 +23,7 @@ When this session owns supervision and away mode is not active: The turn-end guard on omp is structural, not advisory: `__FM_OMP_TURNEND_EXT__` answers omp's blocking `session_stop` hook, and when `bin/fm-turnend-guard.sh` returns 2 it forces one continuation carrying the guard text, bounded to one per turn by the `stop_hook_active` flag omp sets on the continuation's own stop. An interrupted turn never raises `session_stop`, so a supervisor-initiated interrupt is not guarded; `bin/fm-control.sh` owns that postcondition. -The Pi supervision branch (`docs/pi-supervision-branch.md`) is out of scope for the omp primary: every actionable wake is delivered to this conversation, exactly as on Claude, and the lease, outcome-store, and `fm_branch_processed` contracts do not apply here. +The Pi supervision branch (`docs/pi-supervision-branch.md`) is Pi's in-process conversation and does not run on omp: without the supervision host every actionable wake is delivered to this conversation and the lease, outcome-store, and `fm_branch_processed` contracts do not apply here, while a home with `config/supervision-host` runs the host's away session ([`supervision-host.md`](../supervision-host.md)). The turn-end guard extension lives at `__FM_OMP_TURNEND_EXT__`. The watcher extension lives at `__FM_OMP_EXT__`. diff --git a/docs/supervision-protocols/supervision-host.md b/docs/supervision-protocols/supervision-host.md index cef71a8059d..ab5395cb0d2 100644 --- a/docs/supervision-protocols/supervision-host.md +++ b/docs/supervision-protocols/supervision-host.md @@ -1,11 +1,26 @@ Supervision host: on for this home (`config/supervision-host`; [`supervision-host.md`](../supervision-host.md) owns the design). -The Stop hook runs the supervision host in the arm's place, and everything above still holds with these additions: +{claude} The Stop hook runs the supervision host in the arm's place, and everything above still holds with these additions: +{cursor} The `stop` hook park runs the supervision host in the arm's place, and everything above still holds with these additions: +{opencode} The OpenCode TUI plugin runs the supervision host in the arm's place, and everything above still holds with these additions: +{omp} The omp watch extension runs the supervision host in the arm's place, and everything above still holds with these additions: +{grok} Your tracked background arm above runs the supervision host (`bin/fm-supervision-host.sh park`) in the plain arm's place, and everything above still holds with these additions: +{codex} Every foreground checkpoint runs the supervision host in the watcher's place, and everything above still holds with these additions: 1. Attended (no away-posture record `state/.afk-contract`): every wake reaches you exactly as above. 2. Away (the record exists and no daemon runs): the host hands each wake to a headless away session that runs the supervision branch's contract under the record, and you are parked. - Only a wake the host hands back reaches you, as `Stop hook feedback` carrying the close plus one `supervision-host: ` line. +{claude} Only a wake the host hands back reaches you, as `Stop hook feedback` carrying the close plus one `supervision-host: ` line. +{cursor,opencode,omp} Only a wake the host hands back reaches you, as a `watcher` follow-up carrying the close plus one `supervision-host: ` line. +{grok} Only a wake the host hands back reaches you, as the arm's background-task-completed notification whose output carries the close plus one `supervision-host: ` line. +{codex} Only a wake the host hands back reaches you, as checkpoint output carrying the close plus one `supervision-host: ` line. +{codex} While the record exists each checkpoint uses the longer away bound (`FM_CODEX_WATCH_CHECKPOINT_AWAY`, default 3600s, subject to the host's park cap; see [`supervision-host.md`](../supervision-host.md#the-park-boundary)), so a captain message waits until the checkpoint returns unless the captain interrupts it. That wake is automatic supervision, not the captain's return: drain and handle it under the away posture, and never run the return from it. After the return, a `supervision-host:` line naming the captain's return during a turn means that turn's outcomes missed the return brief, whether the wake was handled or handed back: relay every following `supervision-host: outcome ...` line to the captain (the rows also remain in `bin/fm-branch-outcome.sh list`), then drain and handle any queued wake before acknowledging. -3. `supervision-host: cycle boundary ...` means the host ended its park before the Stop hook timeout: run `bin/fm-wake-drain.sh`, handle whatever it presents, run its printed acknowledgement (an empty queue prints `--ack-through 0`), and end the turn; the next park starts at that turn end. + Each such outcome is also a queued `check: supervision-host outcome ... was recorded after the captain returned` wake, which the drain presents until acknowledged: relay each outcome once, whichever arrives first. +{claude,cursor} 3. `supervision-host: cycle boundary ...` means the host ended its park at its bound: run `bin/fm-wake-drain.sh`, handle whatever it presents, run its printed acknowledgement (an empty queue prints `--ack-through 0`), and end the turn; the next park starts at that turn end. +{opencode,omp} 3. `supervision-host: cycle boundary ...` means the host ended its park at its bound and the next park has already started: run `bin/fm-wake-drain.sh`, handle whatever it presents, and run its printed acknowledgement (an empty queue prints `--ack-through 0`). +{grok} 3. `supervision-host: cycle boundary ...` means the host ended its park at its bound: run `bin/fm-wake-drain.sh`, handle whatever it presents, run its printed acknowledgement (an empty queue prints `--ack-through 0`), and re-arm the same background host call. +{codex} 3. The host's park boundary returns as the checkpoint's ordinary `checkpoint: no actionable wake within s` line; handle it as step 5 above says. 4. A guarded command that exits 6 naming the branch actor's lease means the away session is handling that task right now: leave the lease alone and retry after it releases, which it does when its turn ends. 5. Captain outcomes the away session records wait in the outcome store for the return brief (`bin/fm-afk-return.sh`); nothing processes them in this conversation before the return. -6. `/afk` writes only the record here (`bin/fm-afk-launch.sh start-native` refuses the away daemon on this home), while `/quiet` still launches the daemon, which then owns supervision as above. +{claude,grok} 6. `/afk` writes only the record here (`bin/fm-afk-launch.sh start-native` refuses the away daemon on this home), while `/quiet` still launches the daemon, which then owns supervision as above. +{cursor,opencode,omp,codex} 6. `/afk` writes only the record here (`bin/fm-afk-launch.sh start` refuses the away daemon on this home), while `/quiet` still launches the daemon, which then owns supervision as above. +{grok} 7. The pre-tool seatbelt does not classify the host command, so keep it exactly the one background call above: never shell `&`, a pipe, or another command bundled onto it. diff --git a/docs/verification/supervision.md b/docs/verification/supervision.md index c8de6dcadfd..d952702ef4a 100644 --- a/docs/verification/supervision.md +++ b/docs/verification/supervision.md @@ -629,6 +629,62 @@ tests/fm-supervision-instructions.test.sh tests/fm-watch-arm.test.sh ``` + +### Non-Pi primaries + +This supports the per-primary routing in [supervision-host.md](../supervision-host.md): with `config/supervision-host`, the Cursor, OpenCode, Grok, and Codex arm owners run the host with the Claude engine, and without it nothing changes. +It was measured on 2026-09-24 on macOS 26.6.2 arm64 with Claude Code 2.1.281 as the engine (`sonnet`), codex-cli 0.155.1, cursor-agent 2026.09.23-86fc751, OpenCode 1.18.32, grok 1.0.41, and Claude Code 2.1.281 as primaries, and Pi 0.87.0 workers on `openai-codex/gpt-5.6-sol`, in disposable lab homes on private tmux sockets. +omp is not installed on the measuring machine, so its routing rests on `tests/fm-omp-harness.test.sh` alone. +OpenCode's OpenAI login on that machine rejects `gpt-5.6-sol` and has exhausted its `gpt-5.5` usage, so the OpenCode primary ran on OpenCode's free `opencode/big-pickle`. + +Each primary supervised the same session: an attended worker it landed itself, three gated workers, and an away window whose words pre-answered a keyed decision, named a steer, asked for a stopped worker to be told to continue, and ordered a queued item dispatched once another finished. +The lab opened the gates, interrupted a worker mid-turn, tried a main-actor steer on a task the engine held, let the park reach its boundary under shortened bounds, and returned while an engine turn was in flight. + +| Case | Observed | +| --- | --- | +| Away entry | every primary wrote the away record and started no daemon; Codex's checkpoint reported its raised bound under `FM_CODEX_WATCH_CHECKPOINT_AWAY=900` (bound 900, limit 2130) | +| Decision the words pre-answered | the engine answered it on all five primaries and reported `per your away instructions:`; main stayed parked | +| Steer the words named | the engine delivered it on all five, and the finished work carries it | +| Worker stopped mid-task | the engine told it to continue and confirmed it working again (Codex, Cursor, Grok, OpenCode) | +| Main steer during an engine turn | `fm-send.sh` exited 6 with `task ... is leased to the branch supervision actor ... retry after that actor releases it` on all five | +| Dispatch the words ordered with no override | the worker launched on the primary's own runtime (`codex`, `cursor`, `grok`, `claude`); on OpenCode the home's saved default dispatch profile (Pi) took precedence, as configured | +| Boundary | main drained, acknowledged, and re-parked on every primary | +| Return during an engine turn | the finished turn's outcome reached main: Codex and Grok through the host's hand-back line, Cursor through the queued `check: supervision-host outcome ... was recorded after the captain returned` wake after the captain's message superseded the park, Claude through both, and OpenCode in the return brief | +| Malformed engine result (Claude) | handed back as Stop-hook feedback that kept the `supervision-host:` line and named itself not a return | +| A wake that lands between main's drain and its acknowledgement | main's acknowledgement claims only rows at or below its cutoff, so the away session can still take a later row; `tests/fm-wake-queue.test.sh` pins this, and no live run reached that window after the change | + +Engine turns cost $0.06 to $0.79 each; whole away windows cost $0.66 (Claude), $1.21 (Cursor), $1.64 (OpenCode), $2.51 (Grok), and $3.76 (Codex, two windows). +An engine-dispatched Grok 1.0.41 worker stops on Grok's workspace-trust prompt for a project under `/private/tmp`; the engine held it for the captain rather than answering it. + +Without `config/supervision-host`, attended and away sessions on Codex, Cursor, and Grok primaries ran identically on the tree before this change (`9284978f`) and with it: the attended worker landed and was cleaned up, `/afk` started the daemon, the away finish was delivered, the return brief rendered, and nothing landed. +The OpenCode pair could not run, because every primary turn hit the model rejection or usage limit above in both trees. +The live guards gave the same results in both trees: + +| Guard | Before | After | +| --- | --- | --- | +| `FM_CLAUDE_LIVE_E2E=1 tests/fm-claude-stop-autoarm-live-e2e.test.sh` | ok | ok | +| `FM_SUPERVISION_HOST_LIVE_E2E=1 tests/fm-supervision-host-live-e2e.test.sh` | ok | ok | +| `FM_CURSOR_PRIMARY_LIVE_E2E=1 tests/fm-cursor-primary-live-e2e.test.sh` | 7 of 7 ok | 7 of 7 ok | +| `FM_CODEX_LIVE_E2E=1 tests/fm-codex-continuity-live-e2e.test.sh` | ok | ok | +| `FM_GROK_LIVE_E2E=1 tests/fm-grok-continuity-live-e2e.test.sh` | ok | ok | +| `FM_GROK_STOP_LIVE_E2E=1 tests/fm-grok-stop-live-e2e.test.sh` (native 1.0.41, legacy 0.2.102) | `not ok - native path expected two Stop payloads, got 3` | same | +| `FM_OPENCODE_LIVE_E2E=1 tests/fm-opencode-primary-live-e2e.test.sh` | `not ok - ... "The usage limit has been reached","statusCode":429` | same | + +The Grok stop guard was last verified on 0.2.112 and has drifted from Grok 1.0.41 in both trees. + +Deterministic entry points: + +```sh +tests/fm-supervision-host.test.sh +tests/fm-wake-queue.test.sh +tests/fm-cursor-primary.test.sh +tests/fm-pi-watch-extension.test.sh +tests/fm-omp-harness.test.sh +tests/fm-watch-checkpoint.test.sh +tests/fm-supervision-instructions.test.sh +tests/fm-afk-launch.test.sh +``` + ## Wedge-alarm channels The two real notification channels were bounded manually on 2026-07-10 on macOS 26.5.2 with Herdr 0.7.3. diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index 3e10ec270d0..5cc0080689d 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -86,6 +86,7 @@ A main drain with nothing of its own left, and a live grant still holding the qu A row that lost the five appended fields or its numeric sequence can never be claimed, presented, or named by an `--ack-through` cutoff, so a main drain retires it under the queue lock and reports how many it removed together with those rows verbatim, bounded to the first 20 and a count of the rest, because the queue was their only durable record; a branch drain never does, because a grant can only name sequences that were structurally valid when it was published. A retirement that cannot be read or written is reported and never fails the drain: the rows that remain usable are still presented with their acknowledgement command, the unusable ones stay queued for a later drain to retire, and failing the whole drain would strand the usable rows too. Its `--ack-through ` deletes only claimed main rows at or below the cutoff, while a branch acknowledgement deletes only claimed branch rows at or below its cutoff. +A main acknowledgement first claims every unreserved row at or below its cutoff, so none is stranded, and leaves a row above the cutoff that arrived after presentation unowned, so an away-session grant can still take it rather than handing every later wake back to main. Every settled branch prompt releases any residual grant, so an omitted or failed acknowledgement leaves the durable row available to a later main drain; a successful acknowledgement has already removed it. An acknowledgement whose cutoff removes none of the actor's rows while a presented row above the cutoff still waits is reported as having acknowledged nothing, together with the exact `--ack-through` and `--recovery-generation` command for that presented row; the presented set is read before any re-claim, so a row that arrived after presentation is never named for unseen acknowledgement. If a branch offer loses the claim race to main, it rejects its settlement so the watcher retains the actionable close until Pi accepts its main follow-up. diff --git a/tests/fm-afk-launch.test.sh b/tests/fm-afk-launch.test.sh index d9029bc1a21..7a8e435e775 100755 --- a/tests/fm-afk-launch.test.sh +++ b/tests/fm-afk-launch.test.sh @@ -833,6 +833,52 @@ unit_supervision_host_claude_home_runs_no_away_daemon() { rm -rf "$st" } +# Every non-Pi primary with an arm owner runs the host under the same file, so +# away mode launches no daemon there, quiet mode still does, and a harness with +# no arm owner (kimi) keeps the daemon. `enter` says so when the file selects +# no engine for that primary. +unit_supervision_host_other_harnesses_run_no_away_daemon() { + local st harness out rc + st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-host-harness.XXXXXX") + mkdir -p "$st/state" "$st/config" + daemon_allowed() { # [mode] + FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" FM_TEST_HARNESS="$1" FM_AFK_MODE="${2:-}" \ + bash -c '. "$1"; fm_afk_launch_primary_harness() { printf "%s" "$FM_TEST_HARNESS"; }; fm_afk_launch_daemon_allowed' _ "$LAUNCH" 2>&1 + } + for harness in cursor opencode omp grok codex; do + daemon_allowed "$harness" >/dev/null || fail "$harness: a home without config/supervision-host must keep the away daemon" + done + : > "$st/config/supervision-host" + for harness in cursor opencode omp grok codex; do + out=$(daemon_allowed "$harness"); rc=$? + [ "$rc" -ne 0 ] || fail "$harness: an opted-in home must refuse the away daemon" + printf '%s' "$out" | grep -F "not launched on this $harness home, which runs the supervision host" >/dev/null \ + || fail "$harness: the refusal must name the host: $out" + daemon_allowed "$harness" quiet >/dev/null || fail "$harness: quiet mode must still launch the daemon on an opted-in home" + done + daemon_allowed kimi >/dev/null || fail "kimi has no arm owner to run the host, so it must keep the away daemon" + pass "supervision host: away mode on an opted-in cursor, opencode, omp, grok, or codex home launches no daemon" + + enter_with() { # + rm -f "$st/state/.afk-contract" "$st/config/supervision-host" + [ "$2" = - ] || printf '%s\n' "$2" > "$st/config/supervision-host" + FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" FM_TEST_HARNESS="$1" \ + bash -c '. "$1"; fm_afk_launch_primary_harness() { printf "%s" "$FM_TEST_HARNESS"; }; fm_afk_launch_main enter --words "watch the fleet"' _ "$LAUNCH" 2>&1 + } + out=$(enter_with cursor ''); rc=$? + [ "$rc" -eq 0 ] && [ -f "$st/state/.afk-contract" ] || fail "enter on an opted-in cursor home failed (rc=$rc): $out" + printf '%s' "$out" | grep -F "Supervision host: no engine runs the away session on this home (the primary harness 'cursor' has no verified supervision engine)" >/dev/null \ + || fail "enter must say when the host has no engine for this primary: $out" + out=$(enter_with cursor claude) + printf '%s' "$out" | grep -F 'Supervision host: no engine' >/dev/null && fail "enter must stay quiet when the file names a verified engine: $out" + out=$(enter_with cursor -) + printf '%s' "$out" | grep -F 'Supervision host' >/dev/null && fail "enter must stay quiet on a home without the file: $out" + out=$(enter_with claude '') + printf '%s' "$out" | grep -F 'Supervision host: no engine' >/dev/null && fail "a claude home's own engine must count as an engine: $out" + pass "supervision host: enter names a missing engine on an opted-in home and says nothing otherwise" + rm -rf "$st" +} + unit_native_entry_preserves_prepared_state() { local st st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-native-entry.XXXXXX") @@ -1290,6 +1336,7 @@ unit_readiness_failure_preserves_unconfirmed_record unit_tmux_absence_distinguishes_probe_failure unit_native_lifecycle unit_supervision_host_claude_home_runs_no_away_daemon +unit_supervision_host_other_harnesses_run_no_away_daemon unit_native_entry_preserves_prepared_state unit_close_failure_preserves_record unit_record_publication_atomic diff --git a/tests/fm-cursor-primary.test.sh b/tests/fm-cursor-primary.test.sh index fb872c520fd..62a0d4cfc14 100755 --- a/tests/fm-cursor-primary.test.sh +++ b/tests/fm-cursor-primary.test.sh @@ -471,6 +471,97 @@ test_park_inert_when_afk() { pass "cursor park: inert while away mode is active" } +# A supervision host fixture standing in for bin/fm-supervision-host.sh: it +# records its primary pin and arguments, then closes the way says. +write_host_fixture() { # + local dir=$1 kind=$2 + { + printf '#!/usr/bin/env bash\n' + printf 'printf "%%s\\t%%s\\t%%s\\n" "$$" "${FM_SUPERVISION_HOST_PRIMARY:-}" "$*" >> "$FM_HOME/state/host-ran"\n' + case "$kind" in + handback) + printf 'printf "watcher: started pid=%%s (beacon fresh)\\n" "$$"\n' + printf 'for i in 1 2 3 4 5 6 7 8 9 10; do printf "stale: fixture-win %%s\\n" "$i"; done\n' + printf 'printf "supervision-host: the away session could not take this wake: fixture; this wake is yours\\n"\n' + printf 'for i in 1 2 3 4 5 6 7 8 9 10; do printf "supervision-host: outcome %%s for demo [routine]: fixture\\n" "$i"; done\n' + ;; + boundary) + printf 'printf "supervision-host: cycle boundary - fixture\\n"\n' + ;; + stood-down) + printf 'printf "supervision-host stood down: this session no longer owns supervision\\n"\n' + ;; + dies-once) + printf '[ "$(wc -l < "$FM_HOME/state/host-ran")" -gt 1 ] || kill -KILL $$\n' + printf 'printf "stale: fixture-win after a retry\\n"\n' + ;; + esac + printf 'exit 0\n' + } > "$dir/bin/fm-supervision-host.sh" + chmod +x "$dir/bin/fm-supervision-host.sh" +} + +test_park_runs_the_supervision_host_only_when_opted_in() { + local dir out body + dir=$(make_primary_dir "$TMP_ROOT/park-host-off") + : > "$dir/state/task1.meta" + write_arm_fixture "$dir" actionable + write_host_fixture "$dir" handback + out=$(run_park "$dir") + [ -e "$dir/state/arm-ran" ] || fail "a home without config/supervision-host must park on the arm" + [ ! -e "$dir/state/host-ran" ] || fail "a home without config/supervision-host ran the supervision host" + + dir=$(make_primary_dir "$TMP_ROOT/park-host-on") + : > "$dir/state/task1.meta" + : > "$dir/state/.afk-contract" + mkdir -p "$dir/config" + : > "$dir/config/supervision-host" + write_arm_fixture "$dir" actionable + write_host_fixture "$dir" handback + out=$(run_park "$dir") + [ ! -e "$dir/state/arm-ran" ] || fail "an opted-in home ran the plain arm" + [ "$(cut -f2,3 "$dir/state/host-ran")" = "$(printf 'cursor\tpark')" ] \ + || fail "the park must run the host as 'park' with the cursor primary pin: $(cat "$dir/state/host-ran")" + [ "$(kind_of_followup "$out")" = watcher ] || fail "a handed-back wake must arrive as a watcher-kind follow-up, got: $out" + body=$(followup_of "$out") + [ "$(printf '%s\n' "$body" | grep -c '^supervision-host:')" -eq 11 ] \ + || fail "the follow-up must carry every supervision-host line: $body" + [ "$(printf '%s\n' "$body" | grep -c '^stale: fixture-win')" -eq 8 ] \ + || fail "the follow-up must keep the eight-line cap on wake lines: $body" + case "$body" in *'not from the captain: it is not a return'*) ;; *) fail "an away handback must say it is not the captain's return: $body" ;; esac + pass "cursor park: an opted-in home parks on the supervision host and relays every host line" +} + +test_park_host_boundary_stand_down_and_death() { + local dir out + dir=$(make_primary_dir "$TMP_ROOT/park-host-boundary") + : > "$dir/state/task1.meta" + mkdir -p "$dir/config" + : > "$dir/config/supervision-host" + write_host_fixture "$dir" boundary + out=$(run_park "$dir") + case "$(followup_of "$out")" in *'supervision-host: cycle boundary - fixture'*) ;; *) fail "the park boundary must reach the session as a follow-up: $out" ;; esac + + dir=$(make_primary_dir "$TMP_ROOT/park-host-stood-down") + : > "$dir/state/task1.meta" + mkdir -p "$dir/config" + : > "$dir/config/supervision-host" + write_host_fixture "$dir" stood-down + out=$(run_park "$dir") + [ -z "$out" ] || fail "a host that stood down must end the park silently: $out" + [ "$(wc -l < "$dir/state/host-ran" | tr -d ' ')" -eq 1 ] || fail "a host that stood down must not be retried" + + dir=$(make_primary_dir "$TMP_ROOT/park-host-died") + : > "$dir/state/task1.meta" + mkdir -p "$dir/config" + : > "$dir/config/supervision-host" + write_host_fixture "$dir" dies-once + out=$(run_park "$dir") + [ "$(wc -l < "$dir/state/host-ran" | tr -d ' ')" -eq 2 ] || fail "a host that died without a close must be retried: $(cat "$dir/state/host-ran")" + case "$(followup_of "$out")" in *'stale: fixture-win after a retry'*) ;; *) fail "the retried host's wake was not delivered: $out" ;; esac + pass "cursor park: the host's boundary wakes, its stand-down is silent, and a host that died is retried" +} + test_park_inert_under_pi_coding_agent() { local dir out payload dir=$(make_primary_dir "$TMP_ROOT/park-pi-host") @@ -695,6 +786,8 @@ test_park_stands_down_when_superseded test_park_serializes_supersession_with_followup_commit test_superseded_park_does_not_consume_nag_budget test_park_inert_when_afk +test_park_runs_the_supervision_host_only_when_opted_in +test_park_host_boundary_stand_down_and_death test_park_inert_under_pi_coding_agent test_park_still_parks_with_pi_leak_and_cursor_identity test_park_stands_down_when_away_mode_activates_before_commit diff --git a/tests/fm-guard-stale-banner.test.sh b/tests/fm-guard-stale-banner.test.sh index e5bfb9e8762..16443316750 100755 --- a/tests/fm-guard-stale-banner.test.sh +++ b/tests/fm-guard-stale-banner.test.sh @@ -160,6 +160,38 @@ $haystack EOF } +# The same persistent-model call from the supervision branch actor, as the +# supervision host's engine turn runs every guarded command. +run_guard_case_as_branch() { + local dir=$1 + FM_ROOT_OVERRIDE="$(case_root "$dir")" \ + FM_HOME="$(case_home "$dir")" \ + FM_GUARD_GRACE=999 \ + FM_SUPERVISION_MODEL=persistent \ + FM_SUPERVISION_ACTOR=branch \ + "$ROOT/bin/fm-guard.sh" 2>&1 +} + +# The branch actor never owns watcher continuity, so a down watcher is never an +# instruction to it, and its calls neither open nor end main's down-episode. +test_branch_actor_is_never_told_to_repair_the_watcher() { + local dir out + dir=$(make_guard_case branch-watcher-down) + out=$(run_guard_case_as_branch "$dir") + assert_not_contains "$out" "WATCHER DOWN" "the branch actor was shown the watcher-down banner: $out" + assert_not_contains "$out" "watcher still down" "the branch actor was shown the watcher-down reminder: $out" + assert_not_contains "$out" "repair" "the branch actor was given a watcher repair instruction: $out" + out=$(run_guard_case "$dir") + [ "$(count_text "$out" "WATCHER DOWN - SUPERVISION IS OFF")" -eq 1 ] \ + || fail "a branch call must not consume main's full banner for the episode: $out" + out=$(run_guard_case_as_branch "$dir") + [ -z "$out" ] || fail "the branch actor must stay silent inside main's episode: $out" + out=$(run_guard_case "$dir") + assert_contains "$out" "full banner already printed this episode" \ + "a branch call must not end main's down-episode" + pass "fm-guard stale banner: the branch actor is never told to repair the watcher and leaves main's episode alone" +} + test_first_stale_call_prints_full_banner() { local dir out dir=$(make_guard_case first-stale) @@ -922,6 +954,7 @@ test_extension_ownership_needs_every_signal test_extension_stale_beacon_alarms_despite_live_session test_extension_handoff_keeps_queued_wake_warning test_branch_actor_is_not_told_to_drain_queued_wakes +test_branch_actor_is_never_told_to_repair_the_watcher test_persistent_model_ignores_pi_extension_evidence test_extension_live_watcher_is_healthy_without_ownership_evidence test_autoarm_fresh_beacon_without_watcher_is_healthy diff --git a/tests/fm-lint.test.sh b/tests/fm-lint.test.sh index 75edfe85bae..21df1d028ce 100755 --- a/tests/fm-lint.test.sh +++ b/tests/fm-lint.test.sh @@ -572,7 +572,7 @@ test_changed_mode_drops_external_sources_and_excludes_cross_file_codes() { "changed-mode local lint did not disclose dropped source following" assert_grep $'analysis_mode\tlocal' "$telemetry" \ "telemetry did not record local analysis mode" - assert_grep $'source_directives\t4' "$telemetry" \ + assert_grep $'source_directives\t5' "$telemetry" \ "telemetry did not count the changed root's source directives" assert_grep $'source_followed_directives\t0' "$telemetry" \ "telemetry reported followed sources in no-external-sources mode" diff --git a/tests/fm-omp-harness.test.sh b/tests/fm-omp-harness.test.sh index 7c25848eedd..757c01b5b59 100755 --- a/tests/fm-omp-harness.test.sh +++ b/tests/fm-omp-harness.test.sh @@ -574,6 +574,221 @@ EOF pass ".omp watch extension: fm_watch_arm_omp arms once, repeats as a no-op, and delivers an actionable close as one follow-up" } +# An opted-in home spawns the supervision host in the arm's place; its streamed +# status line drives readiness and the handling handoff, and a handed-back +# wake is delivered with every host line and the away note. +test_watch_extension_runs_the_supervision_host() { + local repo home log out status + repo="$TMP_ROOT/watch-host/repo"; home="$TMP_ROOT/watch-host/home"; log="$TMP_ROOT/watch-host/arm.log" + install_omp_extension_fixture "$repo" + mkdir -p "$home/state" "$home/config" + : > "$home/config/supervision-host" + : > "$home/state/.afk-contract" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'confirmed generation=%s watcher=%s\n' "$2" "$4" >> "${FM_ARM_LOG:?}" + exit 0 +fi +printf 'plain-arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" +exit 1 +SH + cat > "$repo/bin/fm-supervision-host.sh" <<'SH' +#!/usr/bin/env bash +printf 'host=%s args=%s primary=%s predecessor=%s\n' "$$" "$*" "${FM_SUPERVISION_HOST_PRIMARY:-}" \ + "${FM_WATCH_PREDECESSOR_ARM_PID:-none}" >> "${FM_ARM_LOG:?}" +if [ "$(grep -c '^host=' "$FM_ARM_LOG")" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + sleep 1 + printf 'signal: omp-host done\nsupervision-host: the away session could not take this wake: fixture; this wake is yours\nsupervision-host: outcome 1 for demo [captain]: fixture\n' + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=gen-2\n' "$$" +sleep 30 +SH + chmod +x "$repo/bin/fm-watch-arm.sh" "$repo/bin/fm-supervision-host.sh" + out=$(FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_WATCH_REARM_RETRY_LIMIT=1 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 \ + EXT="$repo/.omp/extensions/fm-primary-omp-watch.ts" node --input-type=module 2>&1 <<'EOF' +import { pathToFileURL } from "node:url"; +import { writeFileSync, readFileSync } from "node:fs"; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const handlers = new Map(); let tool = null; const sent = []; +const pi = { + on(e, h) { handlers.set(e, h); }, + registerCommand() {}, + registerTool(t) { tool = t; }, + sendUserMessage(m, o) { sent.push({ m, o }); return undefined; }, +}; +const mod = await import(pathToFileURL(process.env.EXT).href); +mod.default(pi); +await tool.execute(); +for (let i = 0; i < 60 && sent.length < 1; i += 1) await new Promise((r) => setTimeout(r, 100)); +const rows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); +if (rows.some((row) => row.startsWith("plain-arm="))) throw new Error(`an opted-in home ran the plain arm: ${rows.join(" | ")}`); +const hosts = rows.filter((row) => row.startsWith("host=")); +if (hosts.length !== 2) throw new Error(`expected the host and one successor host, got: ${rows.join(" | ")}`); +if (!hosts.every((row) => / args=park --restart primary=omp /.test(row))) throw new Error(`the host must run as 'park --restart' with the omp pin: ${hosts.join(" | ")}`); +if (!/predecessor=[0-9]+$/.test(hosts[1])) throw new Error(`the successor host did not receive the closed host as its predecessor: ${hosts[1]}`); +if (!rows.includes(`confirmed generation=gen-2 watcher=${hosts[1].replace(/^host=([0-9]+).*/, "$1")}`)) { + throw new Error(`the handling handoff was not confirmed against the successor host's cycle: ${rows.join(" | ")}`); +} +if (sent.length !== 1) throw new Error(`expected one follow-up wake, saw ${sent.length}: ${JSON.stringify(sent)}`); +for (const needle of [ + "signal: omp-host done", + "supervision-host: the away session could not take this wake: fixture; this wake is yours", + "supervision-host: outcome 1 for demo [captain]: fixture", + "not from the captain: it is not a return", +]) { + if (!sent[0].m.includes(needle)) throw new Error(`the follow-up lacks '${needle}': ${sent[0].m}`); +} +await handlers.get("before_agent_start")({ type: "before_agent_start", prompt: sent[0].m }, {}); +await handlers.get("session_shutdown")({}, {}); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "omp watch extension host mode: $out" + [ -z "$out" ] || fail "omp watch extension host test printed output: $out" + pass ".omp watch extension: an opted-in home runs the supervision host and relays every host line" +} + +# A host cycle boundary can close with only a "supervision-host:" line; left +# unconsumed across a session replacement it rides the persisted handoff and +# the successor session loads and replays it. +test_watch_extension_replays_a_host_only_boundary_across_replacement() { + local repo home log out status + repo="$TMP_ROOT/watch-host-handoff/repo"; home="$TMP_ROOT/watch-host-handoff/home"; log="$TMP_ROOT/watch-host-handoff/arm.log" + install_omp_extension_fixture "$repo" + mkdir -p "$home/state" "$home/config" + : > "$home/config/supervision-host" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" = --handling-delivered ] && exit 0 +exit 1 +SH + cat > "$repo/bin/fm-supervision-host.sh" <<'SH' +#!/usr/bin/env bash +printf 'host=%s\n' "$$" >> "${FM_ARM_LOG:?}" +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=gen-%s\n' "$$" "$$" +if [ "$(grep -c '^host=' "$FM_ARM_LOG")" -eq 1 ]; then + sleep 1 + printf 'supervision-host: outcome 1 for demo [captain]: fixture boundary\n' + exit 0 +fi +sleep 30 +SH + chmod +x "$repo/bin/fm-watch-arm.sh" "$repo/bin/fm-supervision-host.sh" + out=$(FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_WATCH_REARM_RETRY_LIMIT=1 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 \ + EXT="$repo/.omp/extensions/fm-primary-omp-watch.ts" node --input-type=module 2>&1 <<'EOF' +import { pathToFileURL } from "node:url"; +import { writeFileSync, readFileSync, existsSync } from "node:fs"; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const handoff = `${process.env.FM_HOME}/state/extensions/omp-primary-watch/session-replacement-actionable.json`; +const handlers = new Map(); let tool = null; const sent = []; +const pi = { + on(e, h) { handlers.set(e, h); }, + registerCommand() {}, + registerTool(t) { tool = t; }, + sendUserMessage(m, o) { sent.push({ m, o }); return undefined; }, +}; +const mod = await import(pathToFileURL(process.env.EXT).href); +mod.default(pi); +await tool.execute(); +for (let i = 0; i < 60 && sent.length < 1; i += 1) await new Promise((r) => setTimeout(r, 100)); +if (sent.length !== 1) throw new Error(`expected one boundary follow-up, saw ${sent.length}: ${JSON.stringify(sent)}`); +const boundary = "supervision-host: outcome 1 for demo [captain]: fixture boundary"; +if (!sent[0].m.includes(boundary)) throw new Error(`the follow-up lacks the boundary line: ${sent[0].m}`); +// The session is replaced before omp consumes the boundary follow-up. +await handlers.get("session_shutdown")({}, {}); +const stored = JSON.parse(readFileSync(handoff, "utf8")); +if (stored.pending.length !== 1 || !stored.pending[0].message.includes(boundary)) { + throw new Error(`the unconsumed boundary did not ride the handoff: ${JSON.stringify(stored)}`); +} +await handlers.get("session_start")({ type: "session_start" }, {}); +for (let i = 0; i < 60 && sent.length < 2; i += 1) await new Promise((r) => setTimeout(r, 100)); +const replays = sent.slice(1); +if (replays.some((item) => item.m.includes("watcher: FAILED"))) throw new Error(`the successor failed to load the handoff: ${JSON.stringify(replays)}`); +if (replays.length !== 1 || !replays[0].m.includes(boundary)) throw new Error(`the successor did not replay the boundary: ${JSON.stringify(replays)}`); +await handlers.get("before_agent_start")({ type: "before_agent_start", prompt: replays[0].m }, {}); +await handlers.get("session_shutdown")({}, {}); +if (existsSync(handoff)) throw new Error("a consumed replay must not ride the replacement handoff again"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "omp watch extension host-only handoff: $out" + [ -z "$out" ] || fail "omp watch extension host-only handoff test printed output: $out" + pass ".omp watch extension: a host-only boundary rides the replacement handoff and replays in the successor session" +} + +# A host whose exit reaches the extension in separate stream chunks is +# delivered once at its close: a successor host whose status and signal lines +# land while the previous wake is still being delivered, with its outcome lines +# after a pause, reaches main as one follow-up carrying both. +test_watch_extension_delivers_a_split_host_close_whole() { + local repo home log out status + repo="$TMP_ROOT/watch-host-split/repo"; home="$TMP_ROOT/watch-host-split/home"; log="$TMP_ROOT/watch-host-split/arm.log" + install_omp_extension_fixture "$repo" + mkdir -p "$home/state" "$home/config" + : > "$home/config/supervision-host" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" = --handling-delivered ] && exit 0 +exit 1 +SH + cat > "$repo/bin/fm-supervision-host.sh" <<'SH' +#!/usr/bin/env bash +printf 'host=%s\n' "$$" >> "${FM_ARM_LOG:?}" +started="watcher: started pid=$$ (beacon fresh) recovery-generation=gen-$$" +case "$(grep -c '^host=' "$FM_ARM_LOG")" in + 1) + printf '%s\n' "$started" + sleep 1 + printf 'signal: omp-host first\n' + exit 0 + ;; + 2) + printf '%s\nsignal: omp-host second\n' "$started" + sleep 1 + printf 'supervision-host: outcome 2 for demo [captain]: fixture split\n' + exit 0 + ;; +esac +printf '%s\n' "$started" +sleep 30 +SH + chmod +x "$repo/bin/fm-watch-arm.sh" "$repo/bin/fm-supervision-host.sh" + out=$(FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_WATCH_REARM_RETRY_LIMIT=1 FM_WATCH_REARM_RETRY_BASE_MS=5 FM_WATCH_REARM_RETRY_MAX_MS=10 \ + EXT="$repo/.omp/extensions/fm-primary-omp-watch.ts" node --input-type=module 2>&1 <<'EOF' +import { pathToFileURL } from "node:url"; +import { writeFileSync } from "node:fs"; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const handlers = new Map(); let tool = null; const sent = []; +const pi = { + on(e, h) { handlers.set(e, h); }, + registerCommand() {}, + registerTool(t) { tool = t; }, + sendUserMessage(m, o) { sent.push({ m, o }); return undefined; }, +}; +const mod = await import(pathToFileURL(process.env.EXT).href); +mod.default(pi); +await tool.execute(); +for (let i = 0; i < 80 && sent.length < 2; i += 1) await new Promise((r) => setTimeout(r, 100)); +const second = sent.filter((item) => item.m.includes("signal: omp-host second")); +if (second.length !== 1) throw new Error(`expected one follow-up for the split close, saw ${second.length}: ${JSON.stringify(sent)}`); +if (!second[0].m.includes("supervision-host: outcome 2 for demo [captain]: fixture split")) { + throw new Error(`the split close was delivered without its outcome line: ${second[0].m}`); +} +await handlers.get("session_shutdown")({}, {}); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "omp watch extension split host close: $out" + [ -z "$out" ] || fail "omp watch extension split host close test printed output: $out" + pass ".omp watch extension: a host close split across stream chunks reaches main as one whole follow-up" +} + test_detection_anchored_name_and_marker_precedence test_lock_identity_and_liveness_classification test_spawn_launch_line_and_worker_wiring @@ -585,3 +800,6 @@ test_control_composer_and_model_tables test_ownership_proof_is_omp_keyed test_turnend_guard_extension_compels_one_continuation test_watch_extension_arms_and_delivers +test_watch_extension_runs_the_supervision_host +test_watch_extension_replays_a_host_only_boundary_across_replacement +test_watch_extension_delivers_a_split_host_close_whole diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index e984875b098..c774a6c58bd 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -3661,6 +3661,85 @@ EOF pass "OpenCode watcher plugin starts one successor before wake prompt delivery settles" } +# An opted-in home spawns the supervision host in the arm's place; its +# streamed status line drives readiness and the handling handoff, and a +# handed-back wake is delivered with every host line and the away note. +test_opencode_primary_watch_plugin_runs_the_supervision_host() { + local plugin repo home log stop out status + plugin="$ROOT/.opencode/plugins/fm-primary-watch-arm.js" + repo="$TMP_ROOT/opencode-host-root" + home="$TMP_ROOT/opencode-host-home" + log="$TMP_ROOT/opencode-host.log" + stop="$TMP_ROOT/opencode-host.stop" + mkdir -p "$repo/bin" "$home/state" "$home/config" + git init -q "$repo" + : > "$repo/AGENTS.md" + : > "$home/state/task.meta" + : > "$home/state/.afk-contract" + : > "$home/config/supervision-host" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'confirmed generation=%s watcher=%s\n' "$2" "$4" >> "${FM_ARM_LOG:?}" + exit 0 +fi +printf 'plain-arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" +exit 1 +SH + cat > "$repo/bin/fm-supervision-host.sh" <<'SH' +#!/usr/bin/env bash +printf 'host=%s args=%s primary=%s predecessor=%s\n' "$$" "$*" "${FM_SUPERVISION_HOST_PRIMARY:-}" \ + "${FM_WATCH_PREDECESSOR_ARM_PID:-none}" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^host=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + sleep 0.3 + printf 'signal: synthetic wake\nsupervision-host: the away session could not take this wake: fixture; this wake is yours\nsupervision-host: outcome 1 for demo [captain]: fixture\n' + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" "$repo/bin/fm-supervision-host.sh" + out=$(PLUGIN="$plugin" WORKTREE="$repo" FM_HOME="$home" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" node 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +const prompts = []; +const client = { session: { promptAsync: async (request) => { prompts.push(request.body.parts[0].text); } } }; +const hooks = await mod.FmPrimaryWatchArm({ client, directory: process.env.WORKTREE, worktree: process.env.WORKTREE }); +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +await hooks.event({ event: { type: "session.idle", properties: { sessionID: "session-test" } } }); +for (let i = 0; i < 400 && prompts.length < 1; i += 1) await new Promise((resolve) => setTimeout(resolve, 10)); +const rows = existsSync(process.env.FM_ARM_LOG) ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n") : []; +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +if (rows.some((row) => row.startsWith("plain-arm="))) throw new Error(`an opted-in home ran the plain arm: ${rows.join(" | ")}`); +const hosts = rows.filter((row) => row.startsWith("host=")); +if (hosts.length !== 2) throw new Error(`expected the host and one successor host, got: ${rows.join(" | ")}`); +if (!hosts.every((row) => / args=park --restart primary=opencode /.test(row))) throw new Error(`the host must run as 'park --restart' with the opencode pin: ${hosts.join(" | ")}`); +if (!/predecessor=[0-9]+$/.test(hosts[1])) throw new Error(`the successor host did not receive the closed host as its predecessor: ${hosts[1]}`); +if (!rows.some((row) => row === "confirmed generation=fixture-generation watcher=" + hosts[1].replace(/^host=([0-9]+).*/, "$1"))) { + throw new Error(`the handling handoff was not confirmed against the successor host's cycle: ${rows.join(" | ")}`); +} +if (prompts.length !== 1) throw new Error(`expected one wake prompt, got ${prompts.length}`); +for (const needle of [ + "signal: synthetic wake", + "supervision-host: the away session could not take this wake: fixture; this wake is yours", + "supervision-host: outcome 1 for demo [captain]: fixture", + "not from the captain: it is not a return", +]) { + if (!prompts[0].includes(needle)) throw new Error(`the wake prompt lacks '${needle}': ${prompts[0]}`); +} +EOF + ) + status=$? + [ "$status" -eq 0 ] || fail "OpenCode watch plugin must run the supervision host on an opted-in home: $out" + [ -z "$out" ] || fail "OpenCode host test printed output: $out" + pass "OpenCode watcher plugin runs the supervision host on an opted-in home and relays every host line" +} + test_opencode_pre_ready_actionable_close_preserves_its_successor() { local plugin repo home log release retired stop out status plugin="$ROOT/.opencode/plugins/fm-primary-watch-arm.js" @@ -4355,6 +4434,7 @@ test_opencode_primary_watch_plugin_sources_effective_config test_opencode_primary_watch_plugin_requires_session_lock test_opencode_watch_arm_coordinator_respects_primary_scope test_opencode_primary_watch_plugin_rearms_after_wake +test_opencode_primary_watch_plugin_runs_the_supervision_host test_opencode_pre_ready_actionable_close_preserves_its_successor test_opencode_hung_successor_falls_back_to_typed_wake test_opencode_unretired_successor_falls_back_without_retry diff --git a/tests/fm-supervision-host.test.sh b/tests/fm-supervision-host.test.sh index 904f48aff78..2ed1b866f4b 100755 --- a/tests/fm-supervision-host.test.sh +++ b/tests/fm-supervision-host.test.sh @@ -37,6 +37,8 @@ FAKE_CLAUDE="$FAKEBIN/claude" # return handle, but the captain returns (the record is archived) before # the turn ends # return-fail the same, then exit nonzero without a result +# return-first the captain returns first, then handle, then block until the +# host is stopped (an owner killing its host at the turn's end) # noack the same as handle, but skip the acknowledgement # chain handle, then append a status line, so the next close is already # waiting when the turn ends @@ -66,7 +68,8 @@ ack=$(printf '%s\n' "$drain" | sed -n 's/^WAKE_ACK_REQUIRED: after handling comp task=$(sed -n 's/^tasks=//p' "$STATE/.supervision-host-turn" | awk '{ print $1 }') [ -n "$task" ] || task=fleet case "$mode" in - handle|hold-lease|return|return-fail|noack|chain|emptyresult) + handle|hold-lease|return|return-fail|return-first|noack|chain|emptyresult) + [ "$mode" != return-first ] || "$FM_REPO/bin/fm-afk-contract.sh" archive >> "$FM_HOME/engine-return.log" 2>&1 "$FM_REPO/bin/fm-lease.sh" claim "$task" >> "$FM_HOME/engine-lease.log" 2>&1 "$FM_REPO/bin/fm-branch-report.sh" --task "$task" --verdict routine --summary "stub handled $task" \ >> "$FM_HOME/engine-report.log" 2>&1 @@ -78,6 +81,7 @@ case "$mode" in chain) printf 'working [at=%s]: chained %s\n' "$(date +%s)" "$n" >> "$STATE/demo.status" ;; esac [ "$mode" != return-fail ] || exit 3 + [ "$mode" != return-first ] || sleep "$FM_TEST_STUB_MAX_BLOCK_SECONDS" [ "$mode" != emptyresult ] || { printf '{}\n'; exit 0; } result ;; @@ -98,7 +102,9 @@ export FM_POLL=1 FM_SIGNAL_GRACE=0 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 export FM_ARM_CONFIRM_TIMEOUT=30 unset FM_SUPERVISION_ACTOR FM_BRANCH_REPORT_TURN FM_LEASE_HOLDER_PID PI_CODING_AGENT -HOMES=() +# Homes are registered in a file: make_home runs in a command substitution, +# whose variables never reach this shell. +HOMES_FILE="$TMP_ROOT/homes" # Stop whatever a case left running, by the exact pids its home recorded. stop_home_processes() { # local home=$1 pid @@ -115,9 +121,9 @@ stop_home_processes() { # } suite_cleanup() { local home - for home in "${HOMES[@]:-}"; do + while IFS= read -r home; do [ -n "$home" ] && stop_home_processes "$home" - done + done < <(cat "$HOMES_FILE" 2>/dev/null) fm_test_cleanup } trap suite_cleanup EXIT @@ -138,21 +144,22 @@ make_home() { # [config line] FM_HOME="$home" "$CONTRACT" enter --words 'watch the fleet; merge nothing' >/dev/null 2>&1 \ || fail "fixture: could not record the away posture" fi - HOMES+=("$home") + printf '%s\n' "$home" >> "$HOMES_FILE" printf '%s\n' "$home" } # Run the host under the fake harness that holds the home's session lock. -start_host() { # +start_host() { # [park options...] local home=$1 + shift FM_HOME="$home" FM_CREW_STATE_BIN="$home/fakebin/fm-crew-state.sh" PATH="$home/fakebin:$PATH" \ "$FAKE_CLAUDE" -c ' printf "%s\n" "$$" > "$FM_HOME/state/.lock" printf "%s\n" "$$" >> "$FM_HOME/claude-pids" rm -f "$FM_HOME/host.rc" - "$0" park > "$FM_HOME/host.out" 2>&1 + "$0" park "$@" > "$FM_HOME/host.out" 2>&1 printf "%s\n" "$?" > "$FM_HOME/host.rc" - ' "$HOST" 2>> "$home/claude.err" & + ' "$HOST" "$@" 2>> "$home/claude.err" & } # Extended-regex twins of tests/lib.sh's fixed-string assert_grep pair. @@ -226,6 +233,36 @@ test_report_surface_enforces_actor_turn_and_scope() { pass "report surface: only the branch actor's current turn may report, and only on the tasks its wake names" } +# The return brief is rendered after the record is archived, so a report made +# after that may be missing from it: the report itself queues the relay for +# main, durably, while a report made during the away window only waits for the +# brief. +test_report_after_the_return_is_queued_for_main() { + local home state out rc drained + home="$TMP_ROOT/report-return" + state="$home/state" + mkdir -p "$state" + FM_HOME="$home" "$CONTRACT" enter --words 'watch the fleet' >/dev/null 2>&1 || fail "fixture: could not record the away posture" + printf 'turn=t1\nrows=4\ntasks=alpha\nunscoped=0\nwake=signal: alpha.status\n' > "$state/.supervision-host-turn" + + out=$(FM_HOME="$home" FM_SUPERVISION_ACTOR=branch FM_BRANCH_REPORT_TURN=t1 "$REPORT" --task alpha --verdict routine --summary 'steered while away' 2>&1); rc=$? + expect_code 0 "$rc" "a report during the away window must be recorded" + assert_contains "$out" "it waits in the outcome store for MAIN" "a report during the away window waits for the return brief" + ! grep -qs 'supervision-host-return' "$state/.wake-queue" || fail "a report during the away window must not be queued for main" + + FM_HOME="$home" "$CONTRACT" archive >/dev/null 2>&1 || fail "fixture: could not archive the away posture" + out=$(FM_HOME="$home" FM_SUPERVISION_ACTOR=branch FM_BRANCH_REPORT_TURN=t1 "$REPORT" --task alpha --verdict captain --summary 'PR ready for review' 2>&1); rc=$? + expect_code 0 "$rc" "a report after the return must be recorded" + assert_contains "$out" "recorded seq 2 [captain]; the captain has returned, so it is queued for MAIN to relay" \ + "a report after the return must say it is queued for main" + assert_re $'\tcheck\tsupervision-host-return:2\tcheck: supervision-host outcome 2 for alpha \\[captain\\] was recorded after the captain returned.*relay it to the captain: PR ready for review$' \ + "$state/.wake-queue" "the late outcome must be a durable check wake for main" + drained=$(FM_HOME="$home" "$ROOT/bin/fm-wake-drain.sh" 2>&1) + assert_contains "$drained" "supervision-host outcome 2 for alpha [captain] was recorded after the captain returned" \ + "main's drain must present the late outcome" + pass "report surface: an outcome recorded after the captain returned is queued durably for main" +} + # --- dispatch entry ----------------------------------------------------------- test_dispatch_entry_scopes_rows_and_renders_the_away_tail() { @@ -301,7 +338,8 @@ test_away_wake_is_handled_on_the_engine_and_never_reaches_main() { fail "the host did not release the lease the engine left held: $(FM_HOME="$home" "$LEASE" check demo)" fi [ ! -s "$home/host.rc" ] || fail "a handled away wake reached main: $(cat "$home/host.out")" - [ ! -s "$home/host.out" ] || fail "a handled away wake printed to main: $(cat "$home/host.out")" + [ "$(grep -cv '^watcher: started pid=' "$home/host.out")" -eq 0 ] \ + || fail "a handled away wake printed more than the first cycle's status to main: $(cat "$home/host.out")" watcher_live "$home" || fail "the host is not parked on a live successor cycle" echo handle > "$home/stub-mode" @@ -368,6 +406,70 @@ test_return_during_an_engine_turn_hands_its_outcomes_to_main() { pass "host: a captain return during an engine turn hands that turn's outcomes to main" } +# The live failure this guards: a Cursor park superseded by the captain's +# return kills its host as the engine turn ends, so the host's own handoff is +# never printed. The outcome still reaches main: the next host's first cycle +# resurfaces the durable queue and main's drain presents it. +test_outcome_after_the_return_survives_a_host_killed_at_the_turn_end() { + local home host rc drained + home=$(make_home away-return-first away) + echo return-first > "$home/stub-mode" + start_host "$home" + wait_until 150 watcher_live "$home" || fail "return-first: the host never started a watcher cycle" + append_status "$home" 'finishing while the captain comes back' + wait_until 250 grep -qs 'supervision-host-return:1' "$home/state/.wake-queue" \ + || fail "return-first: the late outcome was never queued: $(cat "$home/engine-report.log" 2>/dev/null)" + host=$(awk -F '\t' '$1 == "host" { print $2 }' "$home/state/.supervision-host") + kill -TERM "$host" + wait_until 250 host_exited "$home" || fail "return-first: the stopped host did not exit" + rc=$(cat "$home/host.rc") + [ "$rc" -gt 128 ] || fail "fixture: the host was not stopped mid-turn (rc=$rc): $(cat "$home/host.out")" + assert_no_re '^supervision-host: ' "$home/host.out" "fixture: the stopped host printed a handoff, so this case proves nothing" + for f in "$home"/state/.supervision-host-result.* "$home"/state/.supervision-host-errors.*; do + [ -e "$f" ] && fail "a host stopped mid-turn left its turn file behind: $f" + done + assert_grep 'supervision-host-return:1' "$home/state/.wake-queue" "the late outcome must stay queued after its host died" + + rm -f "$home/host.rc" + start_host "$home" + wait_until 250 host_exited "$home" || fail "return-first: the next host did not resurface the queued outcome" + assert_re '^check: rearm-resurface$' "$home/host.out" "the next host's first cycle must resurface the queue" + assert_re ' pass-through attended check: rearm-resurface' "$home/state/.supervision-host.log" "the attended resurface must reach main" + drained=$(FM_HOME="$home" "$ROOT/bin/fm-wake-drain.sh" 2>&1) + assert_contains "$drained" "supervision-host outcome 1 for demo [routine] was recorded after the captain returned" \ + "main's drain must present the outcome the killed host never handed off" + pass "host: an outcome recorded after the return reaches main even when its host dies at the turn's end" +} + +# A host killed outright mid-turn runs no cleanup; the next host's activation +# stops the engine it left and removes that turn's files. +test_next_host_clears_a_turn_its_killed_predecessor_left() { + local home host engine + home=$(make_home away-killed-mid-turn away) + echo return-first > "$home/stub-mode" + start_host "$home" + wait_until 150 watcher_live "$home" || fail "killed: the host never started a watcher cycle" + append_status "$home" 'mid-turn when its host is killed' + wait_until 250 grep -qs 'supervision-host-return:1' "$home/state/.wake-queue" || fail "killed: the turn never reported" + host=$(awk -F '\t' '$1 == "host" { print $2 }' "$home/state/.supervision-host") + engine=$(cut -f1 "$home/state/.supervision-host.engine-pid") + kill -KILL "$host" + wait_until 100 host_exited "$home" || fail "killed: the host did not die" + ls "$home"/state/.supervision-host-result.* >/dev/null 2>&1 || fail "fixture: the killed turn left no result file, so this case proves nothing" + kill -0 "$engine" 2>/dev/null || fail "fixture: the engine died with its host, so this case proves nothing" + + rm -f "$home/host.rc" + start_host "$home" + wait_until 250 host_exited "$home" || fail "killed: the next host did not resurface the queued outcome" + wait_until 100 sh -c '! kill -0 "$1" 2>/dev/null' _ "$engine" || fail "the next host left its killed predecessor's engine running" + for f in "$home"/state/.supervision-host-result.* "$home"/state/.supervision-host-errors.* \ + "$home"/state/.supervision-host-descendants.* "$home/state/.supervision-host-turn"; do + [ -e "$f" ] && fail "the next host left its killed predecessor's turn file behind: $f" + done + assert_re '^check: rearm-resurface$' "$home/host.out" "the next host's first cycle must resurface the queue" + pass "host: the next host stops the engine a killed predecessor left mid-turn and removes that turn's files" +} + test_report_without_acknowledgement_hands_the_wake_to_main() { local home home=$(make_home away-noack away) @@ -558,6 +660,90 @@ test_park_seconds_at_or_beyond_the_hook_registration_fall_back_to_the_default() pass "host: a park at or beyond the Stop-hook registration falls back to the default boundary" } +# An owner whose own bound is the park lets a turn run past the boundary up to +# its limit; a limit below the boundary or at the registration is the boundary. +test_park_limit_lets_a_turn_outlive_the_boundary() { + local cases name limit want home + cases='limit-later:28000:handled limit-earlier:50:boundary limit-registration:28800:boundary limit-absent::boundary' + for c in $cases; do + name=${c%%:*}; limit=${c#*:}; want=${limit#*:}; limit=${limit%%:*} + home=$(make_home "$name" away) + FM_SUPERVISION_HOST_PARK_SECONDS=100 FM_SUPERVISION_HOST_PARK_LIMIT=$limit FM_SUPERVISION_HOST_TURN_TIMEOUT=200 \ + FM_SUPERVISION_ENGINE_GRACE=10 start_host "$home" + wait_until 150 watcher_live "$home" || fail "$name: the host never started a watcher cycle" + append_status "$home" 'one close' + wait_until 250 sh -c '[ -s "$1/host.rc" ] || grep -q " handled " "$1/state/.supervision-host.log" 2>/dev/null' _ "$home" \ + || fail "$name: the close was neither handled nor handed to main: $(cat "$home/state/.supervision-host.log")" + if host_exited "$home"; then + grep -q '^supervision-host: cycle boundary - ' "$home/host.out" || fail "$name: the host exited without the boundary: $(cat "$home/host.out")" + [ "$want" = boundary ] || fail "$name: a turn inside the owner's limit was refused at the boundary" + else + [ "$want" = handled ] || fail "$name: a turn past the boundary ran without a later limit" + kill -TERM "$(awk -F '\t' '$1 == "host" { print $2 }' "$home/state/.supervision-host")" + wait_until 200 host_exited "$home" || fail "$name: the host did not stop on TERM" + fi + done + pass "host: an owner's later park limit lets a turn outlive the boundary, and no other limit does" +} + +# The first cycle's status line reaches the owner before any close and only +# once; --restart replaces a watcher it would otherwise attach to, and the +# owner's predecessor arm makes the first cycle a handling successor. +test_first_cycle_status_streams_and_owner_options_reach_it() { + local home stale fresh generation predecessor + home=$(make_home stream attended) + start_host "$home" + wait_until 150 grep -qs '^watcher: started pid=' "$home/host.out" \ + || fail "stream: the first cycle's status did not reach the owner before a close: $(cat "$home/host.out")" + host_exited "$home" && fail "stream: the host exited before any close: $(cat "$home/host.out")" + append_status "$home" 'fixture finished' 'done' + wait_until 200 host_exited "$home" || fail "stream: the attended close did not reach main" + [ "$(grep -c '^watcher: ' "$home/host.out")" -eq 1 ] || fail "stream: the status line must be printed once: $(cat "$home/host.out")" + [ "$(sed -n '1p' "$home/host.out" | cut -c1-17)" = 'watcher: started ' ] || fail "stream: the status line must come first" + assert_re '^signal: .*demo.status' "$home/host.out" "stream: the close must follow the status line" + # Main handles that close, so the next cycle has no episode to resurface. + FM_HOME="$home" "$ROOT/bin/fm-wake-drain.sh" >/dev/null 2> "$home/drain.err" || fail "stream: main's drain failed" + ack_drain_err "$home/state" "$home/drain.err" >/dev/null 2>&1 || fail "stream: main's acknowledgement failed: $(cat "$home/drain.err")" + + # A watcher a dead arm left behind, holding this home's watcher lock. + FM_HOME="$home" PATH="$home/fakebin:$PATH" perl -e 'setpgrp(0, 0); exec @ARGV' "$ROOT/bin/fm-watch-arm.sh" \ + > "$home/stale-arm.out" 2>&1 & + wait_until 150 watcher_live "$home" || fail "stream: the fixture watcher never started" + kill -KILL "$!" 2>/dev/null || true + wait "$!" 2>/dev/null || true + stale=$(cat "$home/state/.watch.lock/pid") + rm -f "$home/host.out" "$home/host.rc" + start_host "$home" --restart + # Stopping the old watcher opens a downtime episode, so the fresh cycle may + # close on its resurface before the arm confirms it, and the arm then prints + # only that close (bin/fm-watch-arm.sh): either order is the owner's cycle. + wait_until 150 sh -c 'grep -qs "^watcher: started pid=" "$1/host.out" || [ -s "$1/host.rc" ]' _ "$home" \ + || fail "stream: the restarting host never reported its cycle: $(cat "$home/host.out" "$home/claude.err" 2>/dev/null)" + fresh=$(sed -n 's/^watcher: started pid=\([0-9]*\).*/\1/p' "$home/host.out") + [ "$fresh" != "$stale" ] || fail "stream: --restart attached to the watcher it should have replaced" + wait_until 100 sh -c '! kill -0 "$1" 2>/dev/null' _ "$stale" || fail "stream: --restart left the old watcher running" + wait_until 200 host_exited "$home" || append_status "$home" 'second close' 'done' + wait_until 200 host_exited "$home" || fail "stream: the restarting host's close did not reach main" + assert_re '^(signal: .*demo.status|check: rearm-resurface)$' "$home/host.out" "stream: the restarting host's close must reach main" + + # That close left an unacknowledged downtime episode; a host the owner starts + # as the closed arm's successor takes it over as a handling successor + # instead of re-announcing it. + generation=$(sed -n 's/^[a-z]*:[a-z]*://p' "$home/state/.watcher-down") + [ -n "$generation" ] || fail "fixture: the close left no downtime episode: $(cat "$home/state/.watcher-down")" + predecessor=$(sed -n '1p' "$home/claude-pids") + rm -f "$home/host.out" "$home/host.rc" + FM_WATCH_PREDECESSOR_ARM_PID=$predecessor start_host "$home" --restart + wait_until 150 grep -qs '^watcher: started pid=' "$home/host.out" || fail "stream: the successor host never reported its cycle" + assert_re "^watcher: started pid=[0-9]+ \\(beacon fresh\\) recovery-generation=$generation\$" "$home/host.out" \ + "the owner's predecessor must make the first cycle a handling successor of the pending generation" + sleep 3 + host_exited "$home" && fail "a handling successor re-announced the pending episode: $(cat "$home/host.out")" + kill -TERM "$(awk -F '\t' '$1 == "host" { print $2 }' "$home/state/.supervision-host")" + wait_until 200 host_exited "$home" || fail "stream: the successor host did not stop on TERM" + pass "host: the first cycle's status streams once, --restart replaces a stale watcher, and an owner predecessor makes a handling successor" +} + test_unverified_engine_hands_every_away_wake_to_main() { local home home=$(make_home no-engine away 'pi') @@ -626,11 +812,14 @@ test_superseded_host_leaves_the_owner_untouched() { } test_report_surface_enforces_actor_turn_and_scope +test_report_after_the_return_is_queued_for_main test_dispatch_entry_scopes_rows_and_renders_the_away_tail test_attended_close_passes_straight_to_main test_away_wake_is_handled_on_the_engine_and_never_reaches_main test_away_turn_without_a_report_hands_the_wake_to_main test_return_during_an_engine_turn_hands_its_outcomes_to_main +test_outcome_after_the_return_survives_a_host_killed_at_the_turn_end +test_next_host_clears_a_turn_its_killed_predecessor_left test_report_without_acknowledgement_hands_the_wake_to_main test_return_during_a_failed_turn_still_hands_its_outcomes_to_main test_incomplete_engine_result_hands_the_wake_to_main @@ -640,6 +829,8 @@ test_park_boundary_ends_the_park_before_the_hook_timeout test_park_boundary_holds_under_back_to_back_closes test_park_boundary_rechecked_just_before_the_engine_turn test_park_seconds_at_or_beyond_the_hook_registration_fall_back_to_the_default +test_park_limit_lets_a_turn_outlive_the_boundary +test_first_cycle_status_streams_and_owner_options_reach_it test_unverified_engine_hands_every_away_wake_to_main test_host_outside_the_lock_owner_stands_down test_superseded_host_leaves_the_owner_untouched diff --git a/tests/fm-supervision-instructions.test.sh b/tests/fm-supervision-instructions.test.sh index bd341092115..10a5050f427 100755 --- a/tests/fm-supervision-instructions.test.sh +++ b/tests/fm-supervision-instructions.test.sh @@ -34,11 +34,53 @@ test_supervision_host_protocol_only_on_an_opted_in_claude_home() { assert_contains "$hosted" "never run the return from it" "the host protocol did not say a handed-back wake is not the captain's return" [ "$(printf '%s\n' "$hosted" | grep -vF -e '- Supervision host: on;' | head -n "$(printf '%s\n' "$plain" | wc -l)")" = "$plain" ] \ || fail "the host protocol changed the claude block it should only append to" - other=$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness codex) - assert_not_contains "$other" "Supervision host" "a non-claude primary rendered the host protocol" + other=$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness pi) + assert_not_contains "$other" "Supervision host" "a pi primary rendered the host protocol" pass "renderer adds the supervision-host protocol only on an opted-in claude home, leaving the claude block intact" } +# Each non-Pi arm owner gets the host protocol in its own terms, and only its +# own terms; Grok's model-owned arm command becomes the host; a home without +# the file renders exactly what it did before, with no tag or placeholder. +test_supervision_host_protocol_on_every_arm_owner() { + local home config harness plain hosted body + home="$TMP_ROOT/host-owners-home" + config="$TMP_ROOT/host-owners-config" + mkdir -p "$home/state" "$config" + for harness in claude cursor opencode omp grok codex; do + rm -f "$config/supervision-host" + plain=$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness "$harness") + assert_not_contains "$plain" "Supervision host" "$harness: a home without config/supervision-host rendered the host protocol" + assert_not_contains "$plain" "__FM_" "$harness: a placeholder leaked into the rendered block" + : > "$config/supervision-host" + hosted=$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness "$harness") + assert_contains "$hosted" "- Supervision host: on;" "$harness: an opted-in home did not render the host state line" + body=$(printf '%s\n' "$hosted" | sed -n '/^Supervision host: on for this home/,$p') + [ -n "$body" ] || fail "$harness: the host protocol is missing" + printf '%s\n' "$body" | grep -E '^\{[a-z,]+\} ' >/dev/null && fail "$harness: a harness tag leaked into the rendered protocol: $body" + [ "$(printf '%s\n' "$body" | grep -c 'runs the supervision host')" -eq 1 ] \ + || fail "$harness: the protocol must name exactly one arm owner: $body" + [ "$(printf '%s\n' "$body" | grep -c '^ *Only a wake the host hands back reaches you')" -eq 1 ] \ + || fail "$harness: the protocol must name exactly one wake path: $body" + [ "$(printf '%s\n' "$body" | grep -c '^3\. ')" -eq 1 ] || fail "$harness: the protocol must say once how the park boundary arrives: $body" + [ "$(printf '%s\n' "$body" | grep -c '^6\. ./afk. writes only the record here')" -eq 1 ] \ + || fail "$harness: the protocol must say once what /afk does here: $body" + done + rm -f "$config/supervision-host" + plain=$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness grok) + assert_contains "$plain" 'exec bin/fm-watch-arm.sh`' "grok without the file must arm the plain watcher" + : > "$config/supervision-host" + hosted=$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness grok) + assert_contains "$hosted" 'exec bin/fm-supervision-host.sh park`' "grok with the file must arm the supervision host" + assert_not_contains "$hosted" 'fm-watch-arm.sh` call' "grok with the file must re-arm the supervision host, not the plain arm" + assert_contains "$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness grok --repair-line)" \ + 'bin/fm-supervision-host.sh park as its own Grok tracked background task' "grok's repair line must name the host" + hosted=$(FM_HOME="$home" FM_CONFIG_OVERRIDE="$config" "$RENDER" --harness codex) + assert_contains "$hosted" 'FM_CODEX_WATCH_CHECKPOINT_AWAY' "codex must learn that an away checkpoint holds longer" + assert_contains "$hosted" 'checkpoint: no actionable wake within' "codex must learn how the park boundary arrives" + pass "renderer gives each non-Pi arm owner the host protocol in its own terms, and grok arms the host" +} + test_unknown_fallback() { local out out=$("$RENDER" --harness not-real) @@ -239,6 +281,7 @@ test_pi_snippet_uses_effective_extension_path() { } test_supervision_host_protocol_only_on_an_opted_in_claude_home +test_supervision_host_protocol_on_every_arm_owner test_selected_harness_block_only test_unknown_fallback test_conditional_stanzas diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index 93513bcc288..a4e43acd1eb 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -1602,6 +1602,44 @@ test_branch_grant_refuses_rows_already_claimed_by_main() { pass "branch grant cannot take a row already claimed by main" } +# A wake that lands between main's drain and its acknowledgement was never +# presented to main and sits above the printed cutoff, so the acknowledgement +# must leave it unowned: an away-session grant can still take it, and main's +# next drain still presents it. Claiming it for main instead handed every later +# away wake back to main until main drained again. +test_main_ack_leaves_a_row_that_arrived_after_its_drain_unclaimed() { + local dir state sequence generation rc + dir=$(make_case main-ack-leaves-late-row) + state="$dir/state" + + append_wake "$state" signal "task-a.status" "signal: task-a" || fail "first signal append failed" + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/main.out" 2> "$dir/main.err" \ + || fail "main presentation failed" + sequence=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation [A-Za-z0-9._-][A-Za-z0-9._-]*$/\1/p' "$dir/main.err") + generation=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through [0-9][0-9]* --recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$dir/main.err") + [ "$sequence" = 1 ] || fail "main was not asked to acknowledge exactly its presented row: $(cat "$dir/main.err")" + + append_wake "$state" signal "task-b.status" "signal: task-b" || fail "late signal append failed" + FM_STATE_OVERRIDE="$state" "$DRAIN" --ack-through "$sequence" --recovery-generation "$generation" \ + > "$dir/ack.out" 2> "$dir/ack.err" || fail "main acknowledgement failed: $(cat "$dir/ack.err")" + grep -Fq "$(printf '\tsignal\ttask-b.status\t')" "$state/.wake-queue" \ + || fail "main's acknowledgement consumed a row it was never shown" + + FM_STATE_OVERRIDE="$state" "$GRANT" activate "$$" late-row || fail "branch owner activation failed" + rc=0 + FM_STATE_OVERRIDE="$state" "$GRANT" publish late-row 2 || rc=$? + [ "$rc" -eq 0 ] || fail "an away-session grant could not take a row main never saw: rc=$rc" + FM_STATE_OVERRIDE="$state" "$GRANT" release late-row || fail "branch grant release failed" + FM_STATE_OVERRIDE="$state" "$GRANT" deactivate "$$" late-row || fail "branch owner deactivation failed" + + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$dir/main2.out" 2> "$dir/main2.err" \ + || fail "main's next drain failed" + grep -Fq "$(printf '\tsignal\ttask-b.status\t')" "$dir/main2.out" \ + || fail "main's next drain did not present the late row: $(cat "$dir/main2.out" "$dir/main2.err")" + + pass "main's acknowledgement leaves a row that arrived after its drain for whichever actor takes it next" +} + test_actor_filter_precedes_same_key_deduplication() { local dir state main_sequence main_generation branch_sequence branch_generation dir=$(make_case actor-dedup-order) @@ -2729,6 +2767,7 @@ test_main_is_never_told_to_drain_rows_only_the_branch_owns test_uncountable_queue_still_raises_the_pending_alarm test_unconsumable_rows_are_retired_instead_of_wedging_the_queue test_branch_grant_refuses_rows_already_claimed_by_main +test_main_ack_leaves_a_row_that_arrived_after_its_drain_unclaimed test_actor_filter_precedes_same_key_deduplication test_main_reclaims_a_grant_whose_branch_owner_exited test_branch_actor_without_eligible_snapshot_refuses diff --git a/tests/fm-watch-checkpoint.test.sh b/tests/fm-watch-checkpoint.test.sh index 7424aaba3c8..34d03f612e8 100755 --- a/tests/fm-watch-checkpoint.test.sh +++ b/tests/fm-watch-checkpoint.test.sh @@ -81,7 +81,101 @@ test_existing_singleton_watcher_is_not_success() { pass "checkpoint rejects an existing watcher singleton as unowned" } +# A home opted into the supervision host whose checkpoint runs a stub host in +# a fixture code root: the stub records the bound it was given, then closes +# the way $FM_HOME/host-kind says. +make_host_home() { # + local home + home=$(make_home "$1") + mkdir -p "$home/root/bin" + cp "$CHECKPOINT" "$home/root/bin/fm-watch-checkpoint.sh" + cat > "$home/root/bin/fm-supervision-host.sh" <<'SH' +#!/usr/bin/env bash +printf 'args=%s\nprimary=%s\npark=%s\nlimit=%s\n' "$*" "${FM_SUPERVISION_HOST_PRIMARY:-}" \ + "${FM_SUPERVISION_HOST_PARK_SECONDS:-}" "${FM_SUPERVISION_HOST_PARK_LIMIT:-}" > "$FM_HOME/host-env" +case "$(cat "$FM_HOME/host-kind")" in + boundary) printf 'supervision-host: cycle boundary - fixture\n' ;; + handback) + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: demo.status\nsupervision-host: the away session could not take this wake: fixture; this wake is yours\n' + ;; + stood-down) printf 'supervision-host stood down: this session no longer owns supervision\n' ;; +esac +SH + chmod +x "$home/root/bin/fm-watch-checkpoint.sh" "$home/root/bin/fm-supervision-host.sh" + : > "$home/config/supervision-host" + printf '%s\n' "$home" +} + +run_host_checkpoint() { # [checkpoint args...]; sets STATUS + local home=$1 + printf '%s\n' "$2" > "$home/host-kind" + shift 2 + STATUS=0 + FM_HOME="$home" "$home/root/bin/fm-watch-checkpoint.sh" "$@" >"$home/out.txt" 2>"$home/err.txt" || STATUS=$? +} + +test_host_checkpoint_bounds_the_park_by_posture() { + local home + home=$(make_host_home host-bound) + run_host_checkpoint "$home" boundary --seconds 5 + expect_code 124 "$STATUS" "a host park that reached its bound is a quiet checkpoint" + assert_contains "$(cat "$home/out.txt")" "checkpoint: no actionable wake within 5s" "the boundary must read as the ordinary quiet line" + assert_contains "$(cat "$home/host-env")" $'args=park\nprimary=codex\npark=5\nlimit=1235' \ + "attended, the host must park for the checkpoint's own bound with the codex pin and a turn limit past it" + : > "$home/state/.afk-contract" + run_host_checkpoint "$home" boundary --seconds 5 + expect_code 124 "$STATUS" "an away park that reached its bound is a quiet checkpoint" + assert_contains "$(cat "$home/out.txt")" "checkpoint: no actionable wake within 3600s" "away, the bound must be raised" + assert_contains "$(cat "$home/host-env")" 'park=3600' "away, the host must park for the away bound" + FM_CODEX_WATCH_CHECKPOINT_AWAY=900 run_host_checkpoint "$home" boundary --seconds 5 + assert_contains "$(cat "$home/host-env")" 'park=900' "the away bound must be configurable" + FM_CODEX_WATCH_CHECKPOINT_AWAY=900 run_host_checkpoint "$home" boundary --seconds 1000 + assert_contains "$(cat "$home/host-env")" 'park=1000' "the away bound must never shorten a longer checkpoint" + pass "checkpoint: an opted-in home runs the host for the checkpoint's bound, raised while away" +} + +test_host_checkpoint_passes_a_handback_and_reports_a_stand_down() { + local home + home=$(make_host_home host-handback) + run_host_checkpoint "$home" handback --seconds 5 + expect_code 0 "$STATUS" "a handed-back wake is an actionable checkpoint" + assert_contains "$(cat "$home/out.txt")" $'signal: demo.status\nsupervision-host: the away session could not take this wake' \ + "the wake and its host line must pass through" + assert_not_contains "$(cat "$home/out.txt")" "watcher: started" "the host's cycle status is not part of the wake" + run_host_checkpoint "$home" stood-down --seconds 5 + expect_code 1 "$STATUS" "a host that stood down is a failed checkpoint" + assert_contains "$(cat "$home/out.txt")" "supervision-host stood down" "the stand-down must be shown" + pass "checkpoint: a handed-back wake passes through, and a host stand-down is a failure" +} + +# The real host under a fake Codex harness that holds the home's session lock. +# shellcheck disable=SC2016 # the fake harness's script expands in its own shell +test_real_host_checkpoint_ends_quietly_at_its_bound() { + local home fakebin status + home=$(make_home host-real) + : > "$home/config/supervision-host" + fakebin="$TMP_ROOT/host-real-bin" + mkdir -p "$fakebin" + ln -s /bin/bash "$fakebin/codex" + status=0 + FM_HOME="$home" FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$fakebin/codex" -c ' + printf "%s\n" "$$" > "$FM_HOME/state/.lock" + "$0" --seconds 4 + ' "$CHECKPOINT" >"$home/out.txt" 2>"$home/err.txt" || status=$? + expect_code 124 "$status" "a quiet host checkpoint: $(cat "$home/out.txt" "$home/err.txt")" + assert_contains "$(cat "$home/out.txt")" "checkpoint: no actionable wake within 4s" "the real host's boundary must read as the quiet line" + assert_grep ' boundary ' "$home/state/.supervision-host.log" "the host must have ended its own park" + if [ -e "$home/state/.watch.lock/pid" ] && kill -0 "$(cat "$home/state/.watch.lock/pid")" 2>/dev/null; then + fail "a host checkpoint left its watcher running" + fi + pass "checkpoint: the real host ends its park at the checkpoint bound as a quiet checkpoint" +} + test_quiet_checkpoint_exits_124_cleanly test_signal_passes_through_and_exits_zero test_registered_check_uses_preserved_watcher_environment test_existing_singleton_watcher_is_not_success +test_host_checkpoint_bounds_the_park_by_posture +test_host_checkpoint_passes_a_handback_and_reports_a_stand_down +test_real_host_checkpoint_ends_quietly_at_its_bound