diff --git a/.agents/skills/captain-hold-lifecycle/SKILL.md b/.agents/skills/captain-hold-lifecycle/SKILL.md index eaea0c27411..438f6353b2a 100644 --- a/.agents/skills/captain-hold-lifecycle/SKILL.md +++ b/.agents/skills/captain-hold-lifecycle/SKILL.md @@ -19,6 +19,7 @@ The agent performs the semantic inventory because scripts must not infer captain Every unresolved question that belongs to the captain and is discovered while producing, reading, presenting, or ending an investigation or visual review must be carried by a captain-held task in the authoritative backlog of the home that owns the originating work before that work or review may be treated as complete. For a Lavish board-backed handoff, pass the reply through `bin/fm-procevent-lavish.sh arm --agent-reply-file` before appending the status; the adapter owns version-specific acceptance ordering. Prefer holding the work item the question gates over minting a new row; create a new task only when no work item exists to hold. +The originating investigation or review is never its own inventory entry, so hold a separate task for the call and pass `--origin ` so `complete` can check it. Put the question and its options in the hold reason, and keep one held task per genuine gate: a multi-question review is one held task pointing at its report, not a row per question. Represent that task with exactly one board card that consolidates its questions and options; never fan one task id into duplicate same-key cards. Register or re-hold through `bin/fm-captain-hold.sh hold`, which is idempotent per task id. After inventorying the whole report and review surface, run `bin/fm-captain-hold.sh complete` with every captain-held task id, or with `--none` only when the reviewed surface leaves nothing waiting on the captain. diff --git a/.agents/skills/harness-adapters/references/harness/pi.md b/.agents/skills/harness-adapters/references/harness/pi.md index dd06c7dd8cf..085cc869929 100644 --- a/.agents/skills/harness-adapters/references/harness/pi.md +++ b/.agents/skills/harness-adapters/references/harness/pi.md @@ -18,7 +18,6 @@ Verified on 2026-07-27 with Pi and Pi-signed 0.82.0 unless a fact gives another Native Codex sessions may request `ultra` through the native extension flag described by `../../../bin/fm-spawn.sh`; it is separate from Pi's thinking levels. Pi has no permission system, so workers are always autonomous. -Pi's installed `packages/coding-agent/docs/settings.md` UI and display section documents `regular` as the `tuiMode` default and `fullscreen` as experimental. Fullscreen can bury steering messages by rewriting scrollback, so Firstmate avoids it when the installed CLI supports the override. `../../../bin/fm-spawn.sh --help` owns the executable-pinning and version-safe launch mechanics. @@ -33,7 +32,10 @@ Multiple positional arguments become separate queued messages; the spawn templat A fresh Pi or Pi-signed spawn for a ship, scout, or task registers its exact isolated worktree in the Pi trust store under the spawn's `PI_CODING_AGENT_DIR` (default `~/.pi/agent/trust.json`, or the pinned root's `trust.json` under a worker account pin), so the worker does not pause for the project-trust dialog. Relaunches and secondmates do not register trust. -For a manually launched Pi in an untrusted directory, accept the dialog with Enter and verify the instructions begin processing. +A project trust dialog can appear on the first Pi run in any not-yet-trusted directory that holds a trust-requiring resource such as `.pi/extensions/`, including a clean worktree and a freshly seeded secondmate home. +Accept it with Enter and verify the instructions begin processing. +The decision persists per path in `~/.pi/agent/trust.json`, or in the pinned root's `trust.json` under a worker account pin, so later spawns in the same pooled slot under that root skip it. +For unattended seeded-secondmate launches, `../../../bin/fm-spawn.sh --help` owns the capability-gated project-trust approval mechanics; [runtime verification](../../../../../docs/verification/runtime-backends.md#pi-seeded-secondmate-project-trust) owns the regression evidence. ## Worker turn-end extension diff --git a/.agents/skills/operational-home-layout/SKILL.md b/.agents/skills/operational-home-layout/SKILL.md index e9930c506a8..87a17dd85f8 100644 --- a/.agents/skills/operational-home-layout/SKILL.md +++ b/.agents/skills/operational-home-layout/SKILL.md @@ -42,6 +42,7 @@ config/trace-context optional presence flag enabling default-off native W3C tra config/lavish-axi-host optional one-line per-machine Lavish server address; LOCAL, gitignored, inherited by secondmate homes, and exported into every worker launch; see docs/configuration.md "Lavish server address" for opening versus polling config/brief-include.md optional standing worker instructions appended verbatim as the last section of every ship and scout scaffold; LOCAL, gitignored, and not inherited; keep its text out of `## Firstmate spec`; see docs/configuration.md "Home brief include" config/fleet-ledger optional presence flag opting this home in to the default-off fleet activity ledger state/fleet-ledger.jsonl that outside tools can follow; LOCAL, gitignored, and not inherited; see docs/fleet-ledger.md +config/wait-no-turns optional presence flag opting this home into default-off waiting-worker behavior (brief waiting section, foreground pipeline drive, pending-reply hold, one fire-and-forget retry ring); LOCAL, gitignored, and not inherited; see docs/configuration.md "Waiting worker spends no turns" config/turnend-churn-absorb optional presence flag opting this home into the default-off absorb of bare turn-end wakes on pane churn; LOCAL, gitignored, and not inherited; see docs/configuration.md "Turn-end pane-churn absorb" config/wedge-defer-parked-gate optional presence flag opting this home into the default-off deferral of a wedge escalation for a lane parked at a validation gate awaiting the supervisor's own still-open decision; LOCAL, gitignored, and not inherited; see docs/configuration.md "Parked-gate wait deferral" config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") diff --git a/.pi/extensions/fm-branch-supervision.ts b/.pi/extensions/fm-branch-supervision.ts index 91c65224d1e..0e9defcd9a8 100644 --- a/.pi/extensions/fm-branch-supervision.ts +++ b/.pi/extensions/fm-branch-supervision.ts @@ -653,10 +653,16 @@ export default function (pi: ExtensionAPI) { // queued for the captain's next prompt. The durable truth is the store's // processed marker; this only paces re-presentation and resets with the // session generation. - type ProcessingState = { sequences: string; through: number; triggered: number; pending: boolean; nextTurnQueued: boolean }; + type ProcessingState = { sequences: string; through: number; triggered: number; pending: boolean; nextTurnQueued: boolean; visibleFinals: Set }; let processing: ProcessingState | null = null; let queuedProcessingContent: string | null = null; let processingOpenedThisRun = false; + // Compare only replies to the same consumed sequence set. A retry can be + // the first real handling, so only an empty or exact-repeat final is hidden. + // Buffer retry streaming until message_end can make that decision; tool + // messages always keep their prose, and a user message ends this scope. + let activeProcessing: { request: ProcessingState; retry: boolean } | null = null; + let userMessageThisTurn = false; let processedInitializedGeneration = -1; // One revision for BOTH selections: a model or effort change invalidates an // in-flight branch build exactly the same way. @@ -1095,7 +1101,7 @@ export default function (pi: ExtensionAPI) { } if (processing?.pending) return true; if (!processing || processing.sequences !== sequences) { - processing = { sequences, through, triggered: 0, pending: false, nextTurnQueued: false }; + processing = { sequences, through, triggered: 0, pending: false, nextTurnQueued: false, visibleFinals: new Set() }; } // A presentation already sent is consumed by the run it joins or opens; // until that run settles, sending a widened or identical copy would hand @@ -1677,7 +1683,6 @@ ${context.command} // duplicate suppression. Operational extension injections are not dialog. const prompt = event.prompt; processingOpenedThisRun = queuedProcessingContent !== null && prompt === queuedProcessingContent; - if (processingOpenedThisRun) queuedProcessingContent = null; const trimmed = prompt.trim(); if (!trimmed || isOperationalUserText(trimmed)) return; const file = currentMainSession.getSessionFile() ?? ""; @@ -1692,6 +1697,48 @@ ${context.command} // so a fresh copy may be queued again once this run settles unacknowledged. if (processing) processing.nextTurnQueued = false; }); + pi.on?.("turn_start", () => { + userMessageThisTurn = false; + }); + pi.on?.("message_start", (event) => { + if (event.message.role === "user") { + userMessageThisTurn = true; + activeProcessing = null; + } else if ( + event.message.role === "custom" && + isProcessingCustomMessage(event.message) && + queuedProcessingContent !== null && + event.message.content === queuedProcessingContent + ) { + // message_start covers both an idle custom prompt and a follow-up + // consumed inside an existing run; neither needs before_agent_start. + activeProcessing = !userMessageThisTurn && processing ? { request: processing, retry: processing.triggered > 1 } : null; + queuedProcessingContent = null; + } + }); + pi.registerMarkdownTransformer?.((markdown, context) => + activeProcessing?.retry && context.isStreaming && context.messageType !== "user" ? "" : markdown, + ); + pi.on?.("message_end", (event) => { + if (!activeProcessing || event.message.role !== "assistant") return; + // message_end runs before tool execution. Keep the whole message when + // it carries a call, including prose alongside fm_branch_processed. + if (event.message.content.some((part) => part.type === "toolCall")) return; + const text = event.message.content.filter((part) => part.type === "text").map((part) => part.text).join("\n").trim(); + const { request, retry } = activeProcessing; + if (!retry || (text && !request.visibleFinals.has(text))) { + if (text) request.visibleFinals.add(text); + return; + } + // Pi applies the replacement before persistence and transcript rendering. + // Preserve the message envelope, including provider usage accounting. + return { + message: { + ...event.message, + content: [], + }, + }; + }); pi.on?.("context", (event, ctx) => { if (!afkPostureRecordPresent(state)) return; const messages = event.messages ?? []; @@ -1714,6 +1761,7 @@ ${context.command} mainStreaming = false; queuedProcessingContent = null; processingOpenedThisRun = false; + activeProcessing = null; if (processing) processing.pending = false; const settledGeneration = generation; await enqueueDelivery(async () => { @@ -1773,6 +1821,8 @@ ${context.command} consecutiveProviderErrors = 0; providerRecovery = null; generation += 1; + activeProcessing = null; + userMessageThisTurn = false; mirrorCollection.collectAnchor = null; mirrorCollection.pendingCursor = null; mirrorCollection.stagedCaptain = null; @@ -1821,6 +1871,9 @@ ${context.command} shuttingDown = true; generation += 1; processing = null; + queuedProcessingContent = null; + activeProcessing = null; + userMessageThisTurn = false; pendingMirror.length = 0; currentMainSession = null; mirrorCollection.collectAnchor = null; @@ -2369,6 +2422,9 @@ ${context.command} }; } const remaining = await readUnprocessedOutcomes(acknowledgedGeneration); + if (acknowledgedGeneration === generation && activeProcessing && through >= activeProcessing.request.through) { + activeProcessing = null; + } if (remaining !== null && remaining.length === 0) processing = null; const open = remaining === null ? "the remaining outcomes could not be read" diff --git a/.pi/extensions/fm-primary-pi-watch.ts b/.pi/extensions/fm-primary-pi-watch.ts index 2ba5d6a190f..e2043e3ba8a 100644 --- a/.pi/extensions/fm-primary-pi-watch.ts +++ b/.pi/extensions/fm-primary-pi-watch.ts @@ -149,6 +149,8 @@ const armScript = `${fmRoot}/bin/fm-watch-arm.sh`; const marker = `${state}/.pi-watch-extension-loaded`; const handoffDir = `${state}/extensions/pi-primary-watch`; const actionableHandoff = `${handoffDir}/session-replacement-actionable.json`; +const extensionLog = `${state}/.watch-extension.log`; +const extensionLogMaxLines = extensionLogKeepLines(); const extensionVersion = `sha256:${createHash("sha256").update(readFileSync(extensionFile)).digest("hex")}`; const retryBaseMs = positiveInteger("FM_WATCH_REARM_RETRY_BASE_MS", 250); const retryMaxMs = positiveInteger("FM_WATCH_REARM_RETRY_MAX_MS", 4000); @@ -213,6 +215,18 @@ function positiveInteger(name: string, fallback: number): number { return Math.floor(value); } +// Opt-in bound for the extension diagnostic log: only a positive +// FM_WATCH_EXTENSION_LOG_KEEP_LINES enables logging, so the default run +// writes nothing. Unset, empty, non-numeric, zero, and negative values +// disable the log entirely instead of falling back to a silent default. +function extensionLogKeepLines(): number { + const raw = process.env.FM_WATCH_EXTENSION_LOG_KEEP_LINES; + if (raw === undefined || raw.trim() === "") return 0; + const value = Math.floor(Number(raw)); + if (!Number.isFinite(value) || value <= 0) return 0; + return value; +} + function parentPid(pid: string): string { const result = spawnSync("ps", ["-o", "ppid=", "-p", pid], { encoding: "utf8" }); if (result.status !== 0) return ""; @@ -228,6 +242,21 @@ function pidAlive(pid: string): boolean { } } +// An arm child whose process is gone but whose close event has not fired yet +// (stdio pipes still held) must not keep the single-flight slot: neither a +// repair call nor a scheduled retry would start anything until that close +// finally fires. Callers that gate on slot occupancy use this instead of +// owner.child so both paths can always recover. + +function liveArmChild(owner: SessionGeneration): ChildProcess | null { + const child = owner.child; + if (!child) return null; + if (child.exitCode !== null || child.signalCode !== null) return null; + const pid = child.pid; + if (pid === undefined || !pidAlive(String(pid))) return null; + return child; +} + function lockOwnership(): LockOwnership { let lockPid = ""; try { @@ -314,6 +343,35 @@ function nodeErrorCode(error: unknown): string { : ""; } +// Bounded diagnostic record for restore attempts, readiness timeouts, and +// handling-confirmation targets and results. Opt-in through +// FM_WATCH_EXTENSION_LOG_KEEP_LINES and off by default: a disabled log +// returns before touching the filesystem, so it never creates its file. +// Purely observational: a logging failure never changes supervision +// behavior. docs/watcher-continuity.md owns what the arm layer already +// records; this file is the extension side. +function appendExtensionLog(detail: string): void { + if (extensionLogMaxLines <= 0) return; + try { + mkdirSync(state, { recursive: true }); + const cleaned = detail.replace(/[\r\n\t]+/g, " ").slice(0, 512); + const line = `${new Date().toISOString()} pid=${process.pid} ${cleaned}`; + let previous = ""; + try { + previous = readFileSync(extensionLog, "utf8"); + } catch (error) { + if (nodeErrorCode(error) !== "ENOENT") return; + } + const joined = `${previous}${previous === "" || previous.endsWith("\n") ? "" : "\n"}${line}\n`; + const kept = joined.split("\n").slice(-(extensionLogMaxLines + 1)).join("\n"); + const temporary = `${extensionLog}.tmp-${process.pid}`; + writeFileSync(temporary, kept, { mode: 0o600 }); + renameSync(temporary, extensionLog); + } catch { + // Diagnostic only: never fail supervision for observability. + } +} + function createPendingActionable(message: string, predecessorArmPid: string): PendingActionableClose { return { version: 1, @@ -612,6 +670,7 @@ export default function (pi: ExtensionAPI) { ok: boolean; handled?: boolean; detail: string; + superseded?: boolean; } { try { const result = spawnSync( @@ -627,6 +686,11 @@ export default function (pi: ExtensionAPI) { // The arm owner proves exact-generation acknowledgement under the queue // lock. Neither an empty queue nor an arbitrary RPC failure is enough. if (result.status === 4) return { ok: true, handled: true, detail: "" }; + if (result.status === 3) { + // The marker advanced past this restoration's generation mid-restore, + // so a newer pipeline owns the episode now: superseded, not rejected. + return { ok: false, detail: "", superseded: true }; + } const stderr = (result.stderr || "").trim(); return { ok: false, @@ -642,16 +706,15 @@ export default function (pi: ExtensionAPI) { } function confirmHandlingDeliveryWithRetry( - owner: SessionGeneration, recovery: { generation: string; watcherPid: string }, - ): { ok: boolean; handled?: boolean; detail: string } { - const snapshot = (): { generation: string; watcherPid: string } => { - const current = owner.child ? armRecovery.get(owner.child) : undefined; - return current ?? recovery; - }; - const first = confirmHandlingDelivery(snapshot()); - if (first.ok) return first; - return confirmHandlingDelivery(snapshot()); + ): { ok: boolean; handled?: boolean; detail: string; superseded?: boolean } { + // Confirm the restoration's own recovery token, never a fresh snapshot of + // the current arm child: a successor replaced during the restore window + // must not turn this delivery into a false rejection, and a retry must + // not retire a newer healthy watcher. + const first = confirmHandlingDelivery(recovery); + if (first.ok || first.superseded) return first; + return confirmHandlingDelivery(recovery); } function offerWakeToBranch(message: string): Promise | null { @@ -672,12 +735,26 @@ export default function (pi: ExtensionAPI) { ): Promise { if (!generationIsLive(owner)) return false; if (recovery) { - const confirmed = confirmHandlingDeliveryWithRetry(owner, recovery); + const confirmed = confirmHandlingDeliveryWithRetry(recovery); + appendExtensionLog( + `confirm generation=${recovery.generation} watcherPid=${recovery.watcherPid} result=${confirmed.handled ? "handled" : confirmed.ok ? "confirmed" : confirmed.superseded ? "superseded" : "rejected"}`, + ); if (confirmed.handled) return true; - if (!confirmed.ok) { - const watcherPid = recovery.watcherPid; - if (!pidAlive(watcherPid)) { - await retireArm(owner.child); + // A superseded result means a newer pipeline owns this episode now: it + // routes like a confirmed delivery below, with no failure appended, and + // retires nothing. + if (!confirmed.ok && !confirmed.superseded) { + const failedPid = recovery.watcherPid; + const current = owner.child; + const currentRecovery = current ? armRecovery.get(current) : undefined; + if ( + current && + currentRecovery?.watcherPid === failedPid && + currentRecovery?.generation === recovery.generation && + !pidAlive(failedPid) + ) { + appendExtensionLog(`retire pid=${failedPid} reason=confirm-failure`); + await retireArm(current); } return await sendWake(owner, `${message}\n\n${confirmed.detail}`, pending); } @@ -855,7 +932,7 @@ export default function (pi: ExtensionAPI) { // been idle. const deferred = owner.deferredClose; owner.deferredClose = null; - if (deferred && !owner.child && !owner.retryTimer) { + if (deferred && !liveArmChild(owner) && !owner.retryTimer) { scheduleRetry(owner, deferred.message, deferred.predecessorArmPid); } } @@ -917,10 +994,14 @@ export default function (pi: ExtensionAPI) { if (!generationIsLive(owner)) return { failure: "" }; const replacement = startArm(owner, predecessorArmPid); const successorChild = owner.child; + appendExtensionLog( + `restore attempt=${attempt} predecessor=${predecessorArmPid || "none"} start=${replacement.ok ? `ok pid=${successorChild?.pid ?? "none"}` : "failed"}`, + ); if (replacement.ok && successorChild && await waitForReadiness(successorChild)) { return { failure: "", recovery: armRecovery.get(successorChild) }; } if (replacement.ok) { + appendExtensionLog(`restore attempt=${attempt} readiness=timeout pid=${successorChild?.pid ?? "none"}`); failure = "watcher: FAILED - Pi extension could not verify a ready successor watcher"; if (!(await retireArm(successorChild))) { return { @@ -936,11 +1017,12 @@ export default function (pi: ExtensionAPI) { if (attempt === retryLimit) break; await waitForRetry(attempt + 1); } + appendExtensionLog(`restore exhausted attempts=${retryLimit + 1} outcome=hand-to-main`); return { failure: `${failure}\nwatcher: FAILED - Pi extension could not restore watcher continuity after ${retryLimit} retries` }; } function scheduleRetry(owner: SessionGeneration, message: string, predecessorArmPid: string): void { - if (!generationIsLive(owner) || owner.child || owner.retryTimer) return; + if (!generationIsLive(owner) || liveArmChild(owner) || owner.retryTimer) return; const ownership = lockOwnership(); if (ownership !== "owned") { surfaceFailure(owner, `watcher: FAILED - Pi extension cannot restore continuity because this session no longer owns the lock\n${message}`); @@ -974,7 +1056,7 @@ export default function (pi: ExtensionAPI) { }; } publishGenerationOwner(owner, "active"); - if (owner.child) { + if (liveArmChild(owner)) { return { ok: true, message: `watcher: unchanged - Pi extension already owns an arm child; no manual re-arm needed; ${repairOnlyHint}`, diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index 401ce1c8be9..f452de16829 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -3375,6 +3375,11 @@ fm_backend_herdr_proof_lines() { # # viewport is the one bound that always contains the composer. # Styled capture is preferred. An empty or failed styled read falls through to # the plain capture so a missing ANSI format does not look like an empty draft. +# This read serves only the Claude payload proof, so the grok-tuned +# dark-truecolor ghost strip is off (FM_COMPOSER_GHOST_LUMA_MAX=0): Claude +# 2.1.283 draws a typed slash command in muted grey 38;2;112;112;112 (verified +# live), which that strip dropped, judging a typed /exit unsent. Claude's own +# ghost suggestion is SGR-2 dim and is still stripped. fm_backend_herdr_composer_content() { # local target=$1 cap caps if cap=$(fm_backend_herdr_visible_capture_ansi "$target" 2>/dev/null) && [ -n "$cap" ]; then @@ -3384,7 +3389,7 @@ fm_backend_herdr_composer_content() { # else return 1 fi - fm_composer_extract_selected_content "$caps" "$cap" + FM_COMPOSER_GHOST_LUMA_MAX=0 fm_composer_extract_selected_content "$caps" "$cap" } # fm_backend_herdr_composer_payload_shown: 0 when , read from a diff --git a/bin/fm-afk-return.sh b/bin/fm-afk-return.sh index 99e6bc86ac7..b162e6bba40 100755 --- a/bin/fm-afk-return.sh +++ b/bin/fm-afk-return.sh @@ -71,6 +71,9 @@ RETURN_GRACE=${FM_GUARD_GRACE:-300} # shellcheck source=bin/fm-afk-contract.sh . "$SCRIPT_DIR/fm-afk-contract.sh" CONTRACT="$SCRIPT_DIR/fm-afk-contract.sh" +# Functions only: decodes the stored hold reasons the catch-up listing shows. +# shellcheck source=bin/fm-hold-reason-lib.sh +. "$SCRIPT_DIR/fm-hold-reason-lib.sh" usage() { sed -n '2,11p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' @@ -591,7 +594,7 @@ render_return_brief() { # } fm_backend_source() { # - local name=$1 adapter rel path siblings + local name=$1 adapter rel sibling fm_backend_validate "$name" || return 1 adapter="$FM_BACKEND_LIB_DIR/backends/$name.sh" + # The sibling list rides in the positional parameters: zsh does not + # word-split an unquoted expansion, so a space-separated string is one path. case "$name" in tmux) - siblings="fm-tmux-lib.sh fm-composer-lib.sh fm-cursor-lib.sh fm-session-lock-lib.sh fm-agent-process-lib.sh fm-gemini-lib.sh" + set -- fm-tmux-lib.sh fm-composer-lib.sh fm-cursor-lib.sh fm-session-lock-lib.sh fm-agent-process-lib.sh fm-gemini-lib.sh ;; herdr) - siblings="fm-composer-lib.sh fm-transition-lib.sh fm-agent-process-lib.sh fm-session-lock-lib.sh fm-gemini-lib.sh" + set -- fm-composer-lib.sh fm-transition-lib.sh fm-agent-process-lib.sh fm-session-lock-lib.sh fm-gemini-lib.sh ;; zellij) - siblings="fm-backend-hometag-lib.sh fm-composer-lib.sh" + set -- fm-backend-hometag-lib.sh fm-composer-lib.sh ;; orca) - siblings="fm-composer-lib.sh" + set -- fm-composer-lib.sh ;; cmux) - siblings="fm-backend-hometag-lib.sh fm-composer-lib.sh" + set -- fm-backend-hometag-lib.sh fm-composer-lib.sh ;; *) return 1 ;; esac fm_backend_source_readable "$adapter" || return 1 - # shellcheck disable=SC2086 # sibling names are a fixed space-separated list - for rel in $siblings; do - path="$FM_BACKEND_LIB_DIR/$rel" - fm_backend_source_readable "$path" || return 1 + for rel in "$@"; do + sibling="$FM_BACKEND_LIB_DIR/$rel" + fm_backend_source_readable "$sibling" || return 1 done case "$name" in tmux) diff --git a/bin/fm-backlog-transition-lib.sh b/bin/fm-backlog-transition-lib.sh index d7dc67bee53..7d73826034f 100644 --- a/bin/fm-backlog-transition-lib.sh +++ b/bin/fm-backlog-transition-lib.sh @@ -78,6 +78,13 @@ FM_BACKLOG_CLOSE_REPLAY_RESULT= # library does not source fm-tasks-axi-lib.sh does not apply. # shellcheck source=bin/fm-timeout-lib.sh disable=SC1091 . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-timeout-lib.sh" +# fm-pr-lib.sh owns which URL is a Gerrit change. It is functions and empty +# globals only, so it is sourced once rather than re-initialising a caller's +# parsed identity. +if ! declare -F fm_pr_url_parse >/dev/null 2>&1; then + # shellcheck source=bin/fm-pr-lib.sh disable=SC1091 + . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-pr-lib.sh" +fi # Latched when a row read hits its bound. fm_backlog_row_show runs inside a # command substitution, so the subshell can READ this latch but cannot set it; @@ -509,16 +516,34 @@ fm_backlog_start() { # fm_backlog_mutate "$1" start "$2" } +# tasks-axi takes a --pr link only as a canonical GitHub or Forgejo pull request +# and refuses anything else, so a Gerrit change URL is recorded on the row as a +# note instead. The subshell keeps the parse from overwriting a caller's +# FM_PR_* identity. +fm_backlog_pr_is_gerrit_change() { # + ( fm_pr_url_parse "$1" && [ "$FM_PR_PROVIDER" = gerrit ] ) +} + fm_backlog_done() { # [flag...] - local data=$1 id=$2 + local data=$1 id=$2 arg previous_arg='' + local -a done_args=() shift 2 - fm_backlog_mutate "$data" "done" "$id" "$@" + for arg in "$@"; do + if [ "$previous_arg" = --pr ] && fm_backlog_pr_is_gerrit_change "$arg"; then + done_args[${#done_args[@]}-1]=--note + done_args+=("Gerrit change $arg") + else + done_args+=("$arg") + fi + previous_arg=$arg + done + fm_backlog_mutate "$data" "done" "$id" "${done_args[@]+"${done_args[@]}"}" } fm_backlog_row_artifact_supported() { local id=$1 flag=${2:-} value=${3:-} case "$flag" in - --pr) return 0 ;; + --pr) ! fm_backlog_pr_is_gerrit_change "$value" ;; --report) [ "$value" = "data/$id/report.md" ] ;; *) return 1 ;; esac @@ -550,8 +575,12 @@ fm_backlog_retain() { # [flag...] fi ;; --pr) - deliverable="${deliverable:+$deliverable; }PR $arg" - row_args=(--pr "$arg") + if fm_backlog_row_artifact_supported "$id" --pr "$arg"; then + deliverable="${deliverable:+$deliverable; }PR $arg" + row_args=(--pr "$arg") + else + deliverable="${deliverable:+$deliverable; }Gerrit change $arg" + fi ;; --note) deliverable="${deliverable:+$deliverable; }$arg" ;; esac diff --git a/bin/fm-brief.sh b/bin/fm-brief.sh index c72e7859e61..c17f158b9ff 100755 --- a/bin/fm-brief.sh +++ b/bin/fm-brief.sh @@ -354,17 +354,48 @@ INBOX_DIR=$(shell_quote "$STATE/$ID.inbox") # The receive-and-ack half of the steering-inbox contract, included in every # scaffold kind. The record format, doorbell line, and re-ring ladder are -# owned by bin/fm-task-inbox-lib.sh; the doorbell itself is self-describing, -# so this section is reinforcement for the natural-checkpoint habit, not the -# only carrier of the instruction. +# owned by bin/fm-task-inbox-lib.sh. The doorbell names the inbox as +# "$FM_TASK_INBOX", which bin/fm-spawn.sh exports into every launch; the full +# path here remains the fallback for a worker launched without that export. +# The doorbell itself is self-describing, so this section is reinforcement +# for the natural-checkpoint habit, not the only carrier of the instruction. +# config/wait-no-turns (docs/configuration.md) adds the line that a waiting +# worker does not poll the inbox: checkpoint checks happen during active work, +# so waiting still spends no turns. IFS= read -r -d '' INBOX_SECTION < --watch`, or `until ; do sleep 30; done` for anything else. +Never spend turns on `sleep` followed by a status check, and never background a command in order to poll it. +In Claude Code that `until` loop in a single Bash call is the sanctioned foreground wait: when the harness refuses a sleep-then-check command and points you at backgrounding instead, reissue the wait as the loop rather than accepting the background. +Bound that command by what your harness lets one command run: in Pi pass the bash tool a `timeout` of at most 2700 seconds, because Pi sets none by default; in Claude Code pass the Bash tool its maximum `timeout` of 600000 ms, because its default is 2 minutes; in Codex keep waiting on a still-running command with empty `write_stdin` polls of up to 300000 ms; elsewhere pass your shell tool its largest timeout and assume at most 10 minutes. +Give any `--wait` a duration a little under that bound. +When the bound passes with nothing changed, run the same blocking command again, with no status check in between. +The one exception is `respond`: it sent its answer before it began waiting, so reattach with `no-mistakes axi run --wait` instead, and never send the same `respond` again, because it would answer whichever gate parks next without you reading it. +A wait your shell can watch this way needs no `paused:` line, except your own pipeline run, a long foreground command, or your own validation round, which you declare once just before its blocking hold: append `paused:` once just before its first blocking command, then stay in the command, and never append it again as you reissue that command. +EOF +WAIT_SECTION=${WAIT_SECTION%$'\n'} +WAIT_BLOCK= +if [ -e "$CONFIG/wait-no-turns" ]; then + WAIT_BLOCK="$WAIT_SECTION"$'\n\n' +fi + if [ "$KIND" = secondmate ]; then SECONDMATE_PROJECTS="" idx=1 @@ -596,7 +627,7 @@ $CREWMATE_PAUSE_INSTRUCTIONS $SHARED_INFRA_RULE 8. $(fm_jev_first_rule) -$INBOX_SECTION$WIKI_GUIDE_SECTION +$WAIT_BLOCK$INBOX_SECTION$WIKI_GUIDE_SECTION # Definition of done Write your findings to \`$DATA/$ID/report.md\`. @@ -675,7 +706,7 @@ $ASK_USER_BLOCK $SHARED_INFRA_RULE 8. $(fm_jev_first_rule) -$INBOX_SECTION +$WAIT_BLOCK$INBOX_SECTION # Project memory A project's \`AGENTS.md\` or \`CLAUDE.md\` is loaded into every agent session in that project, so edit it only to correct information that is factually wrong - including information your own change made wrong - and never to add knowledge because it is missing. diff --git a/bin/fm-captain-hold.sh b/bin/fm-captain-hold.sh index 3872ed7292a..289b27da5e8 100755 --- a/bin/fm-captain-hold.sh +++ b/bin/fm-captain-hold.sh @@ -49,6 +49,10 @@ # a UTC `Captain hold set:` timestamp in the task body: repeating an active # hold preserves the existing timestamp, while re-holding released work starts # a new lifecycle. A task already closed is refused rather than reopened. +# `--origin` also records the origin on a `Captain hold origin:` body line, which +# `complete` and `verify` check. The reason may hold parentheses and line breaks: +# tasks-axi refuses them, so `hold` escapes them where it writes the reason and +# readers decode them (bin/fm-hold-reason-lib.sh owns the encoding). # `--until` records the captain's own deferral date through `tasks-axi hold # --until`, so a "revisit later" answer is stored as a date instead of a live # card. @@ -138,7 +142,10 @@ # `--none` is an explicit semantic attestation that the just-reviewed surface # has no unresolved captain call, and is refused while the origin still has an # open keyed status decision. With a non-empty inventory, every listed task is -# verified durable (actively captain-held, or closed with a recorded answer), +# verified durable (captain-held, or carrying a recorded resolution), +# is never the origin itself, and, when `hold --origin` recorded one, was held +# for this origin; a hold with no recorded origin is accepted on durability +# alone and named in the output, # the inventory is unioned idempotently into the metadata, and every still-open # keyed status decision is transferred to its durable owner with a # `captain-held [key=...]` status close naming the inventory. Later review @@ -146,7 +153,8 @@ # surviving report and tasks without recreating task state. # `verify` is read-only and is called by scout teardown, so teardown cannot # erase a source before this gate has succeeded: every recorded inventory -# entry must still be durable and no keyed status decision may be open. +# entry must still satisfy the same durability and origin checks as `complete`, +# and no keyed status decision may be open. # Metadata compatibility: the attestation keeps the historical # `decisions_reviewed=1` and `decision_keys=` keys, and an inventory entry that # names no existing task resolves through the legacy `-decision-` @@ -228,6 +236,9 @@ CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" # shellcheck source=bin/fm-wake-lib.sh # shellcheck disable=SC1091 . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-hold-reason-lib.sh +# shellcheck disable=SC1091 +. "$SCRIPT_DIR/fm-hold-reason-lib.sh" # shellcheck source=bin/fm-parent-channel-lib.sh # shellcheck disable=SC1091 . "$SCRIPT_DIR/fm-parent-channel-lib.sh" @@ -802,27 +813,106 @@ write_hold_set_stamp() { # + printf '%s\n' "$1" | sed -n 's/^Captain hold origin: \(.*\)$/\1/p' | head -1 +} + +task_identity() { + local id=$1 + if task_show "$id"; then + id=$(show_field_value "$TASK_SHOW_OUTPUT" id) + validate_slug backend-task-id "$id" + elif ! printf '%s\n' "$TASK_SHOW_OUTPUT" | grep -q '^code: NOT_FOUND$'; then + fail "could not resolve the backend identity of $id" + fi + printf '%s' "$id" +} + +write_hold_origin() { # + local id=$1 body=$2 origin=$3 stamp rest new_body tmp + body=$(decode_shown_value "$body") \ + || fail "could not decode the existing body for $id" + stamp=$(printf '%s\n' "$body" | sed -n 1p) + [ -n "$(body_hold_set_timestamp "$body")" ] \ + || fail "task $id lost its hold-set stamp before its origin was recorded" + rest=$(printf '%s\n' "$body" | sed 1d | awk '!/^Captain hold origin: /' \ + | awk 'NF || started { started = 1; print }') + new_body=$stamp + if [ -n "$origin" ]; then + new_body=$(printf '%s\nCaptain hold origin: %s' "$stamp" "$origin") + fi + if [ -n "$rest" ]; then + new_body=$(printf '%s\n\n%s' "$new_body" "$rest") + fi + tmp=$(umask 077; mktemp "${TMPDIR:-/tmp}/fm-captain-hold-origin.XXXXXX") \ + || fail "cannot stage the hold origin" + if ! printf '%s\n' "$new_body" > "$tmp"; then + rm -f -- "$tmp" + fail "cannot stage the hold origin for $id" + fi + if ! tasks_axi update "$id" --body-file "$tmp" >/dev/null; then + rm -f -- "$tmp" + fail "could not record the hold origin on $id" + fi + rm -f -- "$tmp" +} + +refuse_self_inventory() { + local origin=$1 entry=$2 meta="$STATE/$1.meta" + if list_has_key "$(meta_value "$meta" decision_keys)" "$entry"; then + fail "origin $origin cannot be its own captain-call inventory entry; historical decision_keys in $meta still contains $entry; hold a separate captain task with --origin $origin, replace only $entry in the final decision_keys= line with that task id while preserving all other entries, then re-run complete $origin " + fi + fail "origin $origin cannot be its own captain-call inventory entry; hold a separate captain task for the call and list that task" +} + # Resolve one entry and verify the row it names is durably captain-held. A # resolution failure that is not the read bound keeps resolve_entry's own # status - its stderr already named the entry; 124 means the backend never # answered, which is not the same as an unknown entry and must not be spent -# as absence. On success prints " " so the caller can keep the -# attestation evidence. -verify_entry_durable() { # ; prints " " - local origin=$1 entry=$2 resolved resolve_status=0 +# as absence. The result carries the attestation evidence and whether an +# origin was recorded, so completion can disclose the legacy fallback. +verify_entry_durable() { # ; prints " " + local origin=$1 entry=$2 resolved resolve_status=0 id how stored origin_state=unrecorded origin_id stored_id + # The origin task is never its own captain-call inventory: it is the work the + # calls were found in, so accepting it would let a refused hold look recorded. + if [ -n "$origin" ] && [ "$origin" != "$BINDING_ANY" ] && [ "$entry" = "$origin" ]; then + refuse_self_inventory "$origin" "$entry" + fi resolved=$(resolve_entry "$origin" "$entry") || resolve_status=$? if [ "$resolve_status" -ne 0 ]; then [ "$resolve_status" -ne 124 ] \ || fail "the backlog backend exceeded its read bound resolving $entry" exit "$resolve_status" fi - printf '%s\n' "$resolved" - verify_hold_durable "${resolved%% *}" + id=${resolved%% *} + how=${resolved##* } + verify_hold_durable "$id" + id=$(show_field_value "$TASK_SHOW_OUTPUT" id) + validate_slug backend-task-id "$id" + stored=$(body_hold_origin "$(decode_shown_value "$(show_field "$TASK_SHOW_OUTPUT" body)")") + origin_id=$origin + if [ -n "$origin" ] && [ "$origin" != "$BINDING_ANY" ]; then + origin_id=$(task_identity "$origin") || exit $? + [ "$id" != "$origin_id" ] || refuse_self_inventory "$origin" "$entry" + fi + if [ -n "$stored" ]; then + if [ -n "$origin" ] && [ "$origin" != "$BINDING_ANY" ]; then + stored_id=$(task_identity "$stored") || exit $? + if [ "$stored_id" != "$origin_id" ]; then + fail "captain-held task $id was held for origin $stored, not $origin; hold a task for $origin or list the right one" + fi + fi + origin_state=recorded + fi + printf '%s %s %s\n' "$id" "$how" "$origin_state" } command_hold() { local id=${1:-} title='' reason='' repo='' origin='' until='' show state existing_title body='' hold_kind hold_set occurrence - local existing_hold_kind='' existing_held='' preserve_hold_set=0 + local existing_hold_kind='' existing_held='' preserve_hold_set=0 stored_reason previous_origin='' hold_status=0 [ "$#" -ge 1 ] || { usage >&2; exit 2; } shift while [ "$#" -gt 0 ]; do @@ -837,8 +927,9 @@ command_hold() { shift done validate_slug task-id "$id" - validate_one_line reason "$reason" - case "$reason" in *'('*|*')'*) fail "reason must not contain parentheses (tasks-axi hold contract)" ;; esac + [ -n "$reason" ] || fail "reason must not be empty" + # bin/fm-hold-reason-lib.sh owns the storage constraint and reversible encoding. + stored_reason=$(fm_hold_reason_encode "$reason") || fail "could not encode the hold reason" if [ -n "$origin" ]; then validate_slug origin-id "$origin" fi @@ -899,12 +990,25 @@ command_hold() { task_show_or_fail "$id" "task $id disappeared while recording its hold-set stamp" [ -n "$(body_hold_set_timestamp "$(show_field_value "$show" body)")" ] \ || fail "task $id did not retain its hold-set stamp" + if [ -n "$origin" ]; then + origin=$(task_identity "$origin") || exit $? + previous_origin=$(body_hold_origin "$(show_field_value "$show" body)") + write_hold_origin "$id" "$(show_field "$show" body)" "$origin" || exit $? + fi if [ -n "$until" ]; then - tasks_axi hold "$id" --reason "$reason" --kind captain --until "$until" >/dev/null \ - || fail "could not hold task $id for the captain" + tasks_axi hold "$id" --reason "$stored_reason" --kind captain --until "$until" >/dev/null \ + || hold_status=$? else - tasks_axi hold "$id" --reason "$reason" --kind captain >/dev/null \ - || fail "could not hold task $id for the captain" + tasks_axi hold "$id" --reason "$stored_reason" --kind captain >/dev/null \ + || hold_status=$? + fi + if [ "$hold_status" -ne 0 ]; then + # A refused re-hold must not associate the previous hold or answer with a + # new origin. Restore the old line verbatim, without resolving it again. + if [ -n "$origin" ]; then + write_hold_origin "$id" "$(show_field "$show" body)" "$previous_origin" || exit $? + fi + fail "could not hold task $id for the captain" fi task_show "$id" || fail "task $id disappeared while holding it" show=$TASK_SHOW_OUTPUT @@ -958,6 +1062,7 @@ report_retained_artifact_failure() { # apply_pending_retained_artifact() { # local id=$1 marker local -a args=() + RETAINED_CLOSE_ARGS=() marker=$(fm_backlog_close_marker_path "$STATE" "$id") || return 1 [ -e "$marker" ] || [ -L "$marker" ] || return 0 fm_backlog_close_marker_validate "$marker" "$DATA" "$id" "$STATE" \ @@ -966,6 +1071,10 @@ apply_pending_retained_artifact() { # args=("${FM_BACKLOG_CLOSE_VALIDATED_ARGS[@]+"${FM_BACKLOG_CLOSE_VALIDATED_ARGS[@]}"}") case "${args[0]-}" in --pr|--report) + if [ "${args[0]}" = --pr ] && fm_backlog_pr_is_gerrit_change "${args[1]-}"; then + RETAINED_CLOSE_ARGS=(--note "Gerrit change ${args[1]}") + return 0 + fi fm_backlog_row_artifact_supported "$id" "${args[@]}" || return 0 fm_backlog_mutate "$DATA" update "$id" "${args[@]}" \ || { report_retained_artifact_failure "$id" "$marker"; return 1; } @@ -978,7 +1087,7 @@ close_answered() { # tasks_axi unhold "$1" >/dev/null else apply_pending_retained_artifact "$1" || return 1 - tasks_axi "done" "$1" >/dev/null + tasks_axi "done" "$1" "${RETAINED_CLOSE_ARGS[@]+"${RETAINED_CLOSE_ARGS[@]}"}" >/dev/null fi } @@ -1646,7 +1755,7 @@ reconcile_note() { command_complete() { local origin=${1:-} meta previous='' supplied='' keys='' entry key status_file open has_meta=0 transfer_rc transfers=() resolved - local resolved_how attested_by_prefix='' + local resolved_how attested_by_prefix='' origin_state unrecorded_origin='' [ "$#" -ge 2 ] || { usage >&2; exit 2; } validate_slug origin-id "$origin" shift @@ -1678,8 +1787,13 @@ command_complete() { while IFS= read -r entry; do [ -n "$entry" ] || continue resolved=$(verify_entry_durable "$origin" "$entry") || exit $? + origin_state=${resolved##* } + resolved=${resolved% *} resolved_how=${resolved##* } resolved=${resolved%% *} + if [ "$origin_state" = unrecorded ]; then + unrecorded_origin="${unrecorded_origin}${unrecorded_origin:+ }$resolved" + fi if [ "$resolved_how" = migrated-prefix ]; then attested_by_prefix="${attested_by_prefix}${attested_by_prefix:+ }$entry=$resolved" fi @@ -1721,8 +1835,9 @@ EOF fi fi fi - printf 'complete: %s captain-call inventory reviewed%s%s\n' "$origin" "${keys:+ ($keys)}" \ - "${attested_by_prefix:+ [attested through the configured prefix: $attested_by_prefix]}" + printf 'complete: %s captain-call inventory reviewed%s%s%s\n' "$origin" "${keys:+ ($keys)}" \ + "${attested_by_prefix:+ [attested through the configured prefix: $attested_by_prefix]}" \ + "${unrecorded_origin:+ [no recorded origin on: $unrecorded_origin; not checked against $origin]}" } command_verify() { diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh index 274d5b40816..823908a4b97 100755 --- a/bin/fm-classify-lib.sh +++ b/bin/fm-classify-lib.sh @@ -960,6 +960,25 @@ EOF printf '%s\n' "$current" } +# The subset of status_open_decisions the task raised about its own work: a +# reserved-namespace key is raised by a supervisor library about the task (a +# pending-reply escalation), a `remote-reply-continuity-` key is the parent's +# own blocker about a broken remote reply mirror +# (bin/fm-procevent-remote-reply.sh), and a `captain-hold-` key relays a child +# decision a secondmate escalated to the captain (bin/fm-captain-hold.sh) while +# it keeps working, so the task is not waiting on any of them. Pending-reply +# recovery and a fire-and-forget retry ring consult this set and leave a task +# alone while it is non-empty. +status_own_open_decisions() { # + local line prefix + status_open_decisions "$1" | while IFS= read -r line || [ -n "$line" ]; do + for prefix in ${FM_CLASSIFY_RESERVED_KEY_PREFIXES:-$FM_CLASSIFY_RESERVED_KEY_PREFIXES_DEFAULT} remote-reply-continuity- captain-hold-; do + case "$line" in "$prefix"*) continue 2 ;; esac + done + printf '%s\n' "$line" + done +} + # 0 when the fold above still holds at least one decision OPENED by # `needs-decision` - the status side's own record that a human was asked # something and has not answered. A `blocked` record is deliberately not this: a @@ -1076,6 +1095,28 @@ EOF printf '%s' "$verb" } +# The status file inside that is this home's outbound parent channel +# rather than a self-home task status log, printed; empty when there is none. +# Only a remote mate home resolves one - its state/parent-replies.status is the +# parent channel (bin/fm-parent-channel-lib.sh owns that resolution, sourced +# lazily here because that library sources this one at its top level, so a +# top-level source would be circular). A main home, a local mate - whose +# channel lives in the parent home - or an unusable identity or binding keeps +# every file, so ordinary task logs fold and wake exactly as before. The home +# is the directory containing , the /state layout every caller of +# these fleet-wide scans shares; a state dir outside such a home excludes +# nothing. Callers compare the resolved path, never the file name, so a +# parent-replies.status in any other home shape stays an ordinary task log. +status_scan_parent_channel_exclude() { # + local state=$1 exclude + if ! command -v fm_parent_channel_outbound_status >/dev/null 2>&1; then + # shellcheck source=bin/fm-parent-channel-lib.sh + . "$_FM_CLASSIFY_LIB_DIR/fm-parent-channel-lib.sh" + fi + exclude=$(fm_parent_channel_outbound_status "$(dirname "$state")" "$state") || return 0 + printf '%s\n' "$exclude" +} + # Fleet-wide wrapper around status_open_decisions: scans every task's status # log under and prefixes each still-open decision with its owning task # id, so a per-wake or per-session surface can print the consolidated open set @@ -1084,9 +1125,11 @@ EOF # one "\t\t\t" line per open decision, in glob (task id) # order; prints nothing when none are open. scan_open_decisions() { # - local state=$1 f task open line + local state=$1 f task open line exclude + exclude=$(status_scan_parent_channel_exclude "$state") for f in "$state"/*.status; do [ -e "$f" ] || continue + [ "$f" = "$exclude" ] && continue task=$(basename "$f"); task="${task%.status}" open=$(status_open_decisions "$f") || continue [ -n "$open" ] || continue @@ -1377,9 +1420,11 @@ status_open_decisions_incremental() { # [] # the whole-file status_open_decisions, so a fleet-wide per-drain scan stays # bounded by new appends rather than total lifetime log size across every task. scan_open_decisions_incremental() { # - local state=$1 f task open line + local state=$1 f task open line exclude + exclude=$(status_scan_parent_channel_exclude "$state") for f in "$state"/*.status; do [ -e "$f" ] || continue + [ "$f" = "$exclude" ] && continue task=$(basename "$f"); task="${task%.status}" open=$(status_open_decisions_incremental "$f") || continue [ -n "$open" ] || continue @@ -1394,9 +1439,11 @@ EOF } status_presentation_snapshot() { # - local state=$1 f task size ident + local state=$1 f task size ident exclude + exclude=$(status_scan_parent_channel_exclude "$state") for f in "$state"/*.status; do [ -e "$f" ] || continue + [ "$f" = "$exclude" ] && continue [ -f "$f" ] && [ -r "$f" ] && [ ! -L "$f" ] || continue task=$(basename "$f"); task="${task%.status}" size=$(_fm_status_file_size "$f") || return 1 @@ -2008,9 +2055,11 @@ status_line_is_unread_surface() { # # Prints nothing when none are unread. Directory scan rejects status symlinks # the same way scan_open_decisions does. scan_unread_surface_lines() { # - local state=$1 f task lines line + local state=$1 f task lines line exclude + exclude=$(status_scan_parent_channel_exclude "$state") for f in "$state"/*.status; do [ -e "$f" ] || continue + [ "$f" = "$exclude" ] && continue task=$(basename "$f"); task="${task%.status}" lines=$(status_new_lines_since_cursor "$f") || return 1 [ -n "$lines" ] || continue diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index d058f896901..5e3d7ee69d0 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -537,6 +537,22 @@ if [ "$ACTIONABLE" -eq 1 ]; then [ -z "$OUT" ] || rm -f "$OUT" 2>/dev/null || true exit 2 fi + if [ "$HOST_MODE" -eq 1 ] && fm_autoarm_still_owner "$STATE" "$MY_GEN" \ + && fm_recovery_marker_snapshot "$STATE/.watcher-down" \ + && [[ "$FM_RECOVERY_MARKER_TOKEN" == pending:handling:* || "$FM_RECOVERY_MARKER_TOKEN" == announced:handling:* ]] \ + && ! fm_watcher_healthy "$STATE" "$SCRIPT_DIR/fm-watch.sh" "$GRACE" "$FM_HOME"; then + LOST_HANDBACK_COMMITTED=0 + if [ ! -e "$FAILURE_NOTICE" ]; then + printf 'firstmate watcher auto-arm FAILED - the supervision host returned an actionable wake, but its rewake could not be committed.\n' >&2 + autoarm_commit failed "$FAILURE_NOTICE" && LOST_HANDBACK_COMMITTED=1 + else + autoarm_commit failed-suppressed && LOST_HANDBACK_COMMITTED=1 + fi + if [ "$LOST_HANDBACK_COMMITTED" -eq 1 ]; then + [ -z "$OUT" ] || rm -f "$OUT" 2>/dev/null || true + exit 2 + fi + fi [ -z "$OUT" ] || rm -f "$OUT" 2>/dev/null || true exit 0 fi diff --git a/bin/fm-composer-lib.sh b/bin/fm-composer-lib.sh index f0b91698704..cf5139c020b 100644 --- a/bin/fm-composer-lib.sh +++ b/bin/fm-composer-lib.sh @@ -794,6 +794,37 @@ _fm_composer_pi_separator_row() { # return 1 } +# _fm_composer_titled_rule_row: 0 when a trimmed row is a composer rule with a +# session title burned into it (Claude Code draws a named session's title into +# its composer's TOP rule: `──────── ─`, issues #5601 and #5558), proven +# by collapsing to exactly the column width of , the partner +# closing rule already mapped to spaces. +# +# This is deliberately NOT a relaxation of _fm_composer_pi_separator_row, and +# the two must not be merged: that predicate also feeds the pi identity +# conjunction, so it stays strictly dashes-only. This one has the single +# consumer _fm_composer_bare_rule_sandwich. +# +# The row must OPEN with the same 8-column dash run the strict separator +# requires. Width is proven by comparing canonical space strings, never by +# `${#row}`, which counts characters under UTF-8 and bytes under LC_ALL=C +# (issue #1988). Title text is ASCII-printable only, the same boundary +# _fm_composer_titled_bottom_ok holds; any other glyph leaves residue, and the +# verdict stays `unknown`, the safe direction. +_fm_composer_titled_rule_row() { # + local row=$1 expected=$2 spaces + case "$row" in + ────────*) ;; + *) return 1 ;; + esac + spaces=${row//─/ } + spaces=$(printf '%s' "$spaces" | LC_ALL=C sed 's/[!-~]/ /g') + case "$spaces" in + *[![:space:]]*) return 1 ;; + esac + [ "$spaces" = "$expected" ] +} + # Row-scan results are returned through FM_COMPOSER_SCAN_* globals (bash 3.2 # has no nameref); they are internal to this owner. _fm_composer_scan_screen() { # [extract-wrap] @@ -1463,6 +1494,28 @@ _fm_composer_locate_footer_zone() { # && [ "$FM_COMPOSER_SCAN_BARE_ROW" -le "$FM_COMPOSER_FOOTER_LAST" ] } +# _fm_composer_bare_rule_sandwich: 0 when bare agent-glyph sits in its +# own titled composer: a titled rule directly above it and the screen's only +# unmatched separator directly below it, which is that composer's closing rule. +# +# The cursorless staleness rule reads an unmatched separator BELOW a candidate +# as proof the candidate is scrollback. A titled top rule never opens the +# separator pair, so the composer's own closing rule becomes that unmatched +# separator and a genuinely idle composer read `unknown`. Adjacency on BOTH +# edges keeps the staleness rule intact everywhere else: a glyph stranded in +# scrollback has transcript rows, not its own rules, around it. +_fm_composer_bare_rule_sandwich() { # + local plain=$1 row=$2 above below + [ "$row" -ge 1 ] || return 1 + [ "$FM_COMPOSER_SCAN_PI_LAST_SEPARATOR" -eq "$((row + 1))" ] || return 1 + below=$(_fm_composer_screen_row "$((row + 1))" "$plain") + fm_composer_normalize_trim_var below + _fm_composer_pi_separator_row "$below" || return 1 + above=$(_fm_composer_screen_row "$((row - 1))" "$plain") + fm_composer_normalize_trim_var above + _fm_composer_titled_rule_row "$above" "${below//─/ }" +} + _fm_composer_select_cursorless() { # [harness] local plain=$1 harness=${2:-} generic=-1 next boundary raw trimmed glyph bare footer=0 FM_COMPOSER_SELECTED_KIND= @@ -1521,10 +1574,14 @@ _fm_composer_select_cursorless() { # [harness] # Devin's native idle composer is a bare `❭` row closed by one solid rule. # That closing rule is not a Pi pair, so preserve this verified shape only # for Devin without weakening the fleet-wide Pi staleness safeguard. + # Otherwise spare only a bare glyph inside its own titled composer rules; + # see _fm_composer_bare_rule_sandwich for why that shape is not scrollback. if [ "$harness" = devin ] && [ "$FM_COMPOSER_SELECTED_KIND" = bare ] \ && [ "$FM_COMPOSER_SCAN_PI_LAST_SEPARATOR" -eq "$((generic + 1))" ]; then : - else + elif ! { [ "$FM_COMPOSER_SELECTED_KIND" = bare ] \ + && [ "$generic" = "$FM_COMPOSER_SCAN_BARE_ROW" ] \ + && _fm_composer_bare_rule_sandwich "$plain" "$FM_COMPOSER_SCAN_BARE_ROW"; }; then FM_COMPOSER_SELECTED_KIND= return 1 fi diff --git a/bin/fm-contributions.sh b/bin/fm-contributions.sh index 4e27e33e8a1..12bfc5fffcf 100755 --- a/bin/fm-contributions.sh +++ b/bin/fm-contributions.sh @@ -5,7 +5,7 @@ # fm-contributions.sh snapshot [--all] # fm-contributions.sh poll # fm-contributions.sh pending -# fm-contributions.sh verdict +# fm-contributions.sh verdict # fm-contributions.sh ack # fm-contributions.sh arm [--if-owned] # @@ -23,9 +23,10 @@ # checks/reviews). Checks are normalized by name, id, started_at, status and # conclusion; projection picks the newest attempt per distinct name. The last # observation's lane names also disclose a lane absent from the next head. -# A verdict records the EXACT judged head, source URL, actor and summary. A -# comment's arrival time never supplies its judged head. Record a prose verdict -# only after its source identifies that head; otherwise leave it unbound and +# A verdict records the EXACT judged head, source URL, actor and summary. The +# actor is exactly one of captain, fleet, maintainer or nobody; any other value +# is refused. A comment's arrival time never supplies its judged head. Record a +# prose verdict only after its source identifies that head; otherwise leave it unbound and # triage its signal. Formal reviews carry GitHub's own commit_id. Neither kind # can grant merge authority. Captain-actor prose requires an existing live hold; # an eligible merge remains a captain call, never an automatic forge action. @@ -473,7 +474,7 @@ case "${1:-}" in else [ "$#" -eq 4 ] || fail 'verdict needs judged-head, source-url, actor and summary' fm_pr_head_valid "$1" || fail 'an exact judged commit is required' - case "$3" in captain|fleet|maintainer|nobody) ;; *) fail 'invalid required actor' ;; esac + case "$3" in captain|fleet|maintainer|nobody) ;; *) fail "invalid required actor '$3'; expected one of: captain, fleet, maintainer, nobody" ;; esac case "$2" in "$url"\#*) ;; *) fail 'verdict source must be a comment or review on this contribution' ;; esac jq --arg head "$1" --arg source "$2" --arg actor "$3" --arg summary "$4" \ '.verdict={head:$head,source:$source,actor:$actor,summary:$summary}' "$TMP/row.json" > "$TMP/update.json" diff --git a/bin/fm-dod-lib.sh b/bin/fm-dod-lib.sh index b738c4dbbdb..7aa66e563b0 100755 --- a/bin/fm-dod-lib.sh +++ b/bin/fm-dod-lib.sh @@ -372,12 +372,28 @@ EOF # Written once; only the two sentences about a green PR depend on the forge, # because on gerrit the ci step is skipped and there is no PR to report. fm_nm_driving_block() { # - local pr_return_line='' pr_reattach_clause=';' + local pr_return_line='' pr_reattach_clause=';' drive_block wait_cfg if [ "$1" != gerrit ]; then pr_return_line="Only a drive call's return reports the green PR: \`no-mistakes axi status\` shows progress but never reports \`checks-passed\` while the ci step is still monitoring the PR for merge, so never wait on a status poll for the next gate or outcome. " pr_reattach_clause="; once checks are green it returns \`checks-passed\` immediately, and" fi + # config/wait-no-turns selects the foreground drive. Absent, the text matches + # the backgrounded drive a home had before that flag. + wait_cfg=${CONFIG:-${FM_CONFIG_OVERRIDE:-${FM_HOME:-}/config}} + if [ -e "$wait_cfg/wait-no-turns" ]; then + drive_block="Drive the run with ONE foreground \`no-mistakes axi run\` and let it block. +It bounds its own hold for you: \`--wait\` (default 8m) exists precisely so a harness with a ten-minute command cap gets a structured return instead of being killed mid-hold. +Declare that wait using the brief's status-reporting rule before the foreground drive call. +Never background a wait, and never arm a timer to stand in for one: a backgrounded call returns in milliseconds, so it does not wait at all, and every timer left behind fires later as a paid wake for nothing. +${pr_return_line}Whenever a drive call returns without a gate or an outcome - its own wait elapsed, or it was killed or timed out - that is not a failure: reattach at once by re-running \`no-mistakes axi run\` without flags, and issue the same foreground call again, one at a time, until a gate or outcome comes back${pr_reattach_clause} if it refuses because no run is active, read the finished outcome from \`no-mistakes axi status\`." + else + drive_block="One drive call blocks until the next gate or outcome, which routinely outlives what your harness lets a single command run: Claude Code kills a command at ten minutes maximum, while one fix round is capped around thirty minutes and up to three rounds chain. +So background the drive call instead of sitting in one blocking hold your harness will kill, and read its return when it finishes. +Declare that wait using the brief's status-reporting rule before waiting on the backgrounded drive call. +Where a harness's own command limit is not established, assume it bounds commands and use that same backgrounded shape. +${pr_return_line}Whenever a drive call returns without a gate or an outcome - its own wait elapsed, or it was killed or timed out - reattach at once by re-running \`no-mistakes axi run\` without flags, backgrounded the same way${pr_reattach_clause} if it refuses because no run is active, read the finished outcome from \`no-mistakes axi status\`." + fi cat < grep -Eo 'https?://[^[:space:])"]+/pull/[0-9]+' "$1" 2>/dev/null | head -1 } -backlog_json() { # [] - defaults to this home's $BACKLOG +backlog_json() ( # [] - defaults to this home's $BACKLOG local backlog=${1:-$BACKLOG} if [ ! -f "$backlog" ]; then jq -n --arg path "$backlog" '{path:$path,present:false,records:[]}' return 0 fi + set -o pipefail # shellcheck disable=SC2094 jq -Rn --arg path "$backlog" --arg today "$SNAPSHOT_TODAY" --arg now "$SNAPSHOT_NOW" \ --argjson age_days "$FM_SNAPSHOT_UNDATED_HOLD_AGE_DAYS" ' @@ -579,8 +582,8 @@ backlog_json() { # [] - defaults to this home's $BACKLOG | .captain_actionable = (.hold_bucket == "live") else . end) | del(.section,.order) - ' < "$backlog" -} + ' < "$backlog" | fm_hold_reason_decode_stream json +) # Reuse this parser for execution reconciliation without probing any endpoints, # remote homes, transcripts, or forge APIs. diff --git a/bin/fm-fleet-sync.sh b/bin/fm-fleet-sync.sh index f8cc3054591..91b76f78555 100755 --- a/bin/fm-fleet-sync.sh +++ b/bin/fm-fleet-sync.sh @@ -319,10 +319,12 @@ sync_project() { echo "$label: skipped: not a git repo" return 0 fi - # Both sides are physical paths (git resolves --show-toplevel through symlinks), - # so a symlinked clone dir still compares equal to its own root. + # Compare filesystem identity, not spelling: the question is whether git's root + # and $PROJ are the same directory, and a string compare of the two paths also + # fails when they merely differ in case (case-insensitive volume) or in how a + # symlink is spelled. proj_abs=$(cd "$PROJ" && pwd -P) || proj_abs="" - if [ "$proj_top" != "$proj_abs" ]; then + if [ -z "$proj_abs" ] || ! [ "$proj_top" -ef "$proj_abs" ]; then echo "$label: skipped: not a clone root (git would act on $proj_top)" return 0 fi diff --git a/bin/fm-git-strip-ai-trailers.sh b/bin/fm-git-strip-ai-trailers.sh index 5469dfe5557..dfa01616177 100755 --- a/bin/fm-git-strip-ai-trailers.sh +++ b/bin/fm-git-strip-ai-trailers.sh @@ -19,8 +19,9 @@ # because git -c core.hooksPath= (or a child process that # inherits it) carries the override there, and a lookup that honored it # would find this directory again and never run the repository's own -# hook - a skipped pre-push guard. A lookup that fails exits nonzero -# rather than skipping the repository's hook. Does not touch the +# hook - a skipped pre-push guard. An empty core.hooksPath means no +# repository hook, as in plain git; any other failed lookup exits +# nonzero rather than skipping the repository's hook. Does not touch the # project's git config; the caller prefixes the pane with # GIT_CONFIG_COUNT / GIT_CONFIG_KEY_0 / GIT_CONFIG_VALUE_0. # @@ -153,14 +154,21 @@ write_executable() { # is the other environment channel that can carry this directory as # core.hooksPath; only the repository's config files name its own hooks. Skip # when the lookup still names this launch's own hooks dir, meaning those files -# point here, so the wrapper cannot recurse into itself. +# point here, so the wrapper cannot recurse into itself. An empty +# core.hooksPath makes that lookup fail, but plain git reads it as "no hooks", +# so the wrapper runs none; any other failure reruns the lookup to show git's +# error and refuses. runtime_chain_body() { local ours=$1 cat </dev/null) || { + if hooks_path=\$(unset GIT_CONFIG_PARAMETERS; git config --get --type=path core.hooksPath 2>/dev/null) && [ -z "\$hooks_path" ]; then + exit 0 + fi + (unset GIT_CONFIG_PARAMETERS; git rev-parse --path-format=absolute --git-path hooks >/dev/null) echo "fm-git-strip-ai-trailers: cannot resolve this repository's hooks directory; refusing to skip its \$name hook" >&2 exit 1 } diff --git a/bin/fm-hold-reason-lib.sh b/bin/fm-hold-reason-lib.sh new file mode 100644 index 00000000000..6b6b6a6d9ff --- /dev/null +++ b/bin/fm-hold-reason-lib.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# fm-hold-reason-lib.sh - the one reversible encoding of a captain-hold reason. +# +# tasks-axi stores a hold reason as one markdown line inside a parenthesised tag, +# so its own `hold` refuses parentheses and line breaks. A decision reason is +# ordinary prose, so bin/fm-captain-hold.sh encodes the reason where it writes +# it and every reader that shows it decodes it again, instead of banning the +# characters. Stored reasons use the reserved fm-hold-v1: prefix followed by +# base64-encoded UTF-8 text. Unmarked reasons are plain text. Readers decode only +# the hold-reason field, once, and keep line breaks in quoted output strings. +# +# Source this file; it defines functions only. + +# fm_hold_reason_encode : print the storable form, no trailing newline. +fm_hold_reason_encode() { + printf '%s' "$1" | perl -MMIME::Base64=encode_base64 -0777 -ne \ + 'print "fm-hold-v1:", encode_base64($_, "")' +} + +# fm_hold_reason_decode_stream [toon|markdown|json]: decode marked reason fields. +fm_hold_reason_decode_stream() { + perl -MJSON::PP -MMIME::Base64=encode_base64,decode_base64 -MEncode=decode,FB_CROAK -e ' + use strict; + use warnings; + binmode STDIN, ":encoding(UTF-8)"; + binmode STDOUT, ":encoding(UTF-8)"; + my $format = shift; + my $json = JSON::PP->new->allow_nonref; + sub decode_reason { + my ($value) = @_; + return $value unless defined($value) && $value =~ /^fm-hold-v1:(.*)\z/s; + my $payload = $1; + my $bytes = decode_base64($payload); + return $value unless encode_base64($bytes, "") eq $payload; + # Historical literals with valid base64 and UTF-8 remain indistinguishable + # from encoded reasons; malformed payloads retain their stored text. + my $decoded = eval { decode("UTF-8", $bytes, FB_CROAK) }; + return $@ ? $value : $decoded; + } + sub decode_field { + my ($raw) = @_; + my $value = $raw =~ /^"/ ? $json->decode($raw) : $raw; + my $decoded = decode_reason($value); + return $decoded eq $value ? $raw : $json->encode($decoded); + } + if ($format eq "json") { + local $/; + my $snapshot = $json->decode(); + for my $record (@{$snapshot->{records}}) { + $record->{hold_reason} = decode_reason($record->{hold_reason}) + if exists $record->{hold_reason}; + } + print $json->encode($snapshot), "\n"; + exit; + } + my ($column, $task); + while (my $line = ) { + if ($format eq "markdown") { + $line =~ s{^([-*] .*\(hold:\s*)(fm-hold-v1:[A-Za-z0-9+/]*={0,2})(\).*)$} + {$1 . decode_field($2) . $3}e; + } elsif ($line =~ /^tasks\[\d+\]\{([^}]*)\}:\n?$/) { + my @names = split /,/, $1; + ($column) = grep { $names[$_] eq "hold_reason" } 0 .. $#names; + $task = 0; + } elsif (defined($column) && $line =~ /^ (.*)\n?$/) { + my @fields = $1 =~ /("(?:[^"\\]|\\.)*"|[^,]+)/g; + $fields[$column] = decode_field($fields[$column]); + $line = " " . join(",", @fields) . "\n"; + } elsif ($task && $line =~ /^ hold_reason: (.*)\n?$/) { + $line = " hold_reason: " . decode_field($1) . "\n"; + } elsif ($line !~ /^ /) { + $column = undef; + $task = $line eq "task:\n"; + } + print $line; + } + ' "${1:-toon}" +} diff --git a/bin/fm-lint.sh b/bin/fm-lint.sh index 8e0f1d5c946..6d2cc9030fb 100755 --- a/bin/fm-lint.sh +++ b/bin/fm-lint.sh @@ -7,13 +7,14 @@ # both use this owner without duplicating lint configuration. # The explicit --fast mode is local-only and disables ShellCheck's extended # dataflow analysis while preserving ordinary shell lint checks and source -# following. CI, main, and merge-base-less runs keep --norc --external-sources -# --source-path=tests with full dataflow over the whole canonical set. An -# ordinary local branch (changed-file mode, including the no-mistakes lint -# step) drops --external-sources, keeps dataflow, and excludes SC1091, -# SC2034, SC2153, and SC2329, the codes that need library context. Those -# codes still run in CI over the whole set. Explicit paths keep -# --external-sources --source-path=tests with the selected dataflow mode. +# following. CI, main, and merge-base-less runs attempt --norc +# --external-sources --source-path=tests with full dataflow for each canonical +# root. An ordinary local branch (changed-file mode, including the no-mistakes +# lint step) drops --external-sources, keeps dataflow, and excludes SC1091, +# SC2034, SC2153, and SC2329, the codes that need library context. CI checks +# those codes on source-following attempts (see the memory fallback below). +# Explicit paths attempt --external-sources --source-path=tests with the +# selected dataflow mode. # Tests stop source analysis at imported production modules because CI analyzes # every production shell separately as a canonical, source-aware root. # The default (no explicit-path) path also runs bin/fm-lint-workflows.sh so a @@ -25,8 +26,9 @@ # - In CI (GITHUB_ACTIONS=true or CI=true), on the main branch, or when no # merge-base against origin/main (or local main) can be found, it lints # the full canonical set: bin/*.sh bin/backends/*.sh tests/*.sh, with -# --external-sources --source-path=tests and full dataflow. This is what -# CI always runs, so CI coverage never depends on a local diff. +# --external-sources --source-path=tests and full dataflow first. CI +# coverage never depends on a local diff; memory failures may take the +# narrower retry below. # - Otherwise (an ordinary local branch with a real merge-base) it lints # only the canonical-set files changed since that merge-base, including # uncommitted local edits, via plain local `git diff` (no network, no @@ -50,9 +52,10 @@ # two CI runners, each with those same concurrency-limited workers. # Partitions are complete, disjoint, and byte-weight balanced; --list-files # exposes their actual roots. -# Partition mode is always full source-aware analysis, never changed-only or -# --fast, and does not accept explicit paths. Each partition also runs workflow -# lint and backend-purity checks, keeping either invocation independently useful. +# Partition mode starts with full source-aware analysis, never changed-only +# or --fast, and does not accept explicit paths. Each partition also runs +# workflow lint and backend-purity checks, keeping either invocation +# independently useful. # # With FM_LINT_REQUIRE_BOUNDS=1, which CI sets, every per-root ShellCheck # process runs under an enforced envelope: a wall deadline @@ -71,29 +74,42 @@ # cannot apply the address-space limit at all) each root still runs in its # own ShellCheck process with identical diagnostics, just unbounded. # +# If a source-following root exits with a memory failure, it is retried once +# without --external-sources under the same memory limit and only the time +# left in that root's original deadline; with under a second left, the +# memory failure stands without a retry. A clean retry passes +# with an explicit memory-fallback reason and warning; only the same +# cross-file-dependent codes omitted in local no-source lint are excluded. +# Other findings and failed retries still fail lint. The retry's diagnostics +# replace the failed attempt's output; peak RSS is the maximum of both attempts. +# # Per-root evidence is incremental: workers append begin/end records (root, -# mode, shard, start, end, duration, exit status, reason, and peak RSS when -# measured) to a roots log as each root completes, so a mid-run kill still -# leaves the completed record and names the root in flight as -# begun-but-unfinished. With --telemetry the log is retained at +# mode, shard, start, end, duration, final exit status, reason, peak RSS when +# measured, and whether the final attempt followed sources) to a roots log +# as each root completes, so a mid-run kill still leaves the completed record +# and names the root in flight as begun-but-unfinished. With --telemetry the +# log is retained at # .roots.tsv (or .roots.tsv if there is no # .tsv suffix); otherwise it lives only in the -# run's scratch dir. Reason values are ok, findings, timeout, memory, -# signal:, limit-unavailable, or error:. Memory requires process-level -# evidence (a GHC exhaustion status or runtime error on stderr), not an echoed -# source excerpt or an OOM phrase in a filename. In partition mode begin/end +# run's scratch dir. Reason values are ok, findings, memory-fallback, +# timeout, memory, signal:, limit-unavailable, or error:. +# Memory requires process-level evidence (a GHC exhaustion status or runtime +# error on stderr), not an echoed source excerpt or an OOM phrase in a +# filename. In partition mode begin/end # lines also stream to stderr, and an abnormal root end is always reported # there. # # Optional quiet telemetry writes one bounded TSV snapshot of content and source # graph identity, wall/CPU/RSS, shard load, and competing ShellCheck processes. +# source_followed_directives counts directives only for roots whose final +# attempt followed sources, not roots that passed or failed a no-source retry. # # Usage: # fm-lint.sh lint the context-selected file set (see above) # fm-lint.sh --fast [path]... local lint with extended analysis disabled # fm-lint.sh ... lint explicit roots with the same config # fm-lint.sh --jobs <1|2> [path]... override concurrent worker count -# fm-lint.sh --partition <1of2|2of2> lint one full-rigor canonical CI partition +# fm-lint.sh --partition <1of2|2of2> lint one canonical CI partition (see fallback above) # fm-lint.sh --telemetry ... write a quiet metrics snapshot # fm-lint.sh --required-version print the ShellCheck pin # fm-lint.sh --list-files print the file set that would be linted @@ -101,8 +117,8 @@ set -u REQUIRED_SHELLCHECK=0.11.0 -# Cross-file codes that need --external-sources. Local changed-file mode -# cannot judge them, so they stay CI-only. +# Cross-file codes that need --external-sources. No-source checks (local +# changed-file mode and memory fallback) cannot judge them. LOCAL_NOX_EXCLUDE=SC1091,SC2034,SC2153,SC2329 SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" SELF="$SELF_DIR/fm-lint.sh" @@ -162,6 +178,17 @@ fm_lint_root_rss() { # printf '%s\n' "${kib:-unavailable}" } +fm_lint_max_root_rss() { # + local first=$1 second=$2 + case "$first" in ''|unavailable|*[!0-9]*) printf '%s\n' "$second"; return ;; esac + case "$second" in ''|unavailable|*[!0-9]*) printf '%s\n' "$first"; return ;; esac + if [ "$first" -gt "$second" ]; then + printf '%s\n' "$first" + else + printf '%s\n' "$second" + fi +} + # Map a root's exit status onto the reported reason vocabulary without # pretending every signal or nonzero exit is a memory kill: only process-level # memory-failure evidence earns the memory reason - GHC's heap-exhaustion @@ -211,24 +238,11 @@ fm_lint_classify_root() { # esac } -# Run one selected root in its own ShellCheck process, record its lifecycle -# in the roots log, and append its diagnostics to the shard output. -fm_lint_run_root() { # - local index=$1 path=$2 output_dir=$3 shard_index=$4 - local root_out="$output_dir/root.$shard_index.$index.out" - local root_err="$output_dir/root.$shard_index.$index.err" - local rss_file="$output_dir/root.$shard_index.$index.rss" - local start_ms end_ms duration_ms invocation_rc=0 reason rss_kib - start_ms=$(fm_lint_now_ms) - if [ -n "${FM_LINT_INTERNAL_ROOTS_LOG:-}" ]; then - printf 'begin\t%s\t%s\t%s\t%s\t%s\n' \ - "$index" "$path" "$shard_index" "${FM_LINT_INTERNAL_MODE:-}" "$start_ms" \ - >> "$FM_LINT_INTERNAL_ROOTS_LOG" - fi - if [ "${FM_LINT_INTERNAL_PROGRESS:-0}" = 1 ]; then - printf 'fm-lint: begin %s (shard %s, %s mode)\n' \ - "$path" "$shard_index" "${FM_LINT_INTERNAL_MODE:-unknown}" >&2 - fi +# Run one ShellCheck invocation under the given deadline and the per-root +# address-space limit, returning its exit status in FM_LINT_LAST_RC. +fm_lint_exec_root() { # + local path=$1 root_out=$2 root_err=$3 rss_file=$4 seconds=$5 invocation_rc=0 + shift 5 if [ "${FM_LINT_INTERNAL_BOUNDED:-none}" != none ]; then # The watchdog runs in a process group of its own (the same setpgrp hop the # workers use), so the owner's TERM-then-KILL group sweep cannot kill it @@ -239,33 +253,105 @@ fm_lint_run_root() { # # the watchdog is still starting is detected too. ( FM_EXEC_TIMED_OWNER_PID=$$ exec "${FM_LINT_PERL_BIN:-perl}" -e 'setpgrp(0, 0) or die "setpgrp: $!"; exec @ARGV or die "exec: $!"' \ "${BASH:-bash}" "$SELF" --internal-timed \ - "$FM_LINT_INTERNAL_ROOT_SECS" "$FM_LINT_INTERNAL_GRACE" \ + "$seconds" "$FM_LINT_INTERNAL_GRACE" \ "${BASH:-bash}" "$SELF" --internal-root "$rss_file" "$FM_LINT_INTERNAL_MEMORY_KIB" \ - "$FM_LINT_SHELLCHECK" "${FM_LINT_WORKER_ARGS[@]}" -- "$path" ) > "$root_out" 2> "$root_err" & + "$FM_LINT_SHELLCHECK" "$@" -- "$path" ) > "$root_out" 2> "$root_err" & FM_LINT_WORKER_RUN_PID=$! wait "$FM_LINT_WORKER_RUN_PID" || invocation_rc=$? FM_LINT_WORKER_RUN_PID= else - "$FM_LINT_SHELLCHECK" "${FM_LINT_WORKER_ARGS[@]}" -- "$path" > "$root_out" 2> "$root_err" & + "$FM_LINT_SHELLCHECK" "$@" -- "$path" > "$root_out" 2> "$root_err" & FM_LINT_WORKER_RUN_PID=$! wait "$FM_LINT_WORKER_RUN_PID" || invocation_rc=$? FM_LINT_WORKER_RUN_PID= fi + FM_LINT_LAST_RC=$invocation_rc +} + +# Run one selected root, retry memory failures without source following, record +# its lifecycle in the roots log, and append the final diagnostics. +fm_lint_run_root() { # + local index=$1 path=$2 output_dir=$3 shard_index=$4 + local root_out="$output_dir/root.$shard_index.$index.out" + local root_err="$output_dir/root.$shard_index.$index.err" + local rss_file="$output_dir/root.$shard_index.$index.rss" + local fallback_out="$output_dir/root.$shard_index.$index.fallback.out" + local fallback_err="$output_dir/root.$shard_index.$index.fallback.err" + local fallback_rss="$output_dir/root.$shard_index.$index.fallback.rss" + local start_ms end_ms duration_ms invocation_rc=0 reason rss_kib initial_rc initial_reason + local fallback_secs + local final_follow_sources=${FM_LINT_INTERNAL_FOLLOW_SOURCES:-1} + local -a fallback_args + start_ms=$(fm_lint_now_ms) + if [ -n "${FM_LINT_INTERNAL_ROOTS_LOG:-}" ]; then + printf 'begin\t%s\t%s\t%s\t%s\t%s\n' \ + "$index" "$path" "$shard_index" "${FM_LINT_INTERNAL_MODE:-}" "$start_ms" \ + >> "$FM_LINT_INTERNAL_ROOTS_LOG" + fi + if [ "${FM_LINT_INTERNAL_PROGRESS:-0}" = 1 ]; then + printf 'fm-lint: begin %s (shard %s, %s mode)\n' \ + "$path" "$shard_index" "${FM_LINT_INTERNAL_MODE:-unknown}" >&2 + fi + fm_lint_exec_root "$path" "$root_out" "$root_err" "$rss_file" \ + "$FM_LINT_INTERNAL_ROOT_SECS" "${FM_LINT_WORKER_ARGS[@]}" + invocation_rc=$FM_LINT_LAST_RC + reason=$(fm_lint_classify_root "$invocation_rc" "$root_err") + initial_rc=$invocation_rc + initial_reason=$reason + # The retry spends what is left of this root's one deadline rather than a + # fresh one, so both attempts together still fit the budget CI sized its job + # timeout around. + fallback_secs=$(( (start_ms + FM_LINT_INTERNAL_ROOT_SECS * 1000 - $(fm_lint_now_ms)) / 1000 )) + if [ "$reason" = memory ] \ + && [ "${FM_LINT_INTERNAL_FOLLOW_SOURCES:-1}" -eq 1 ] \ + && [ "${FM_LINT_INTERNAL_BOUNDED:-none}" != none ] \ + && [ "$fallback_secs" -lt 1 ]; then + printf 'fm-lint: %s hit the memory ceiling with --external-sources (reason=%s rc=%s); no time left in its %ss deadline to retry without it\n' \ + "$path" "$initial_reason" "$initial_rc" "$FM_LINT_INTERNAL_ROOT_SECS" >> "$output_dir/shard.$shard_index.out" + rss_kib=$(fm_lint_root_rss "$rss_file") + cat "$root_out" "$root_err" >> "$output_dir/shard.$shard_index.out" + elif [ "$reason" = memory ] \ + && [ "${FM_LINT_INTERNAL_FOLLOW_SOURCES:-1}" -eq 1 ]; then + fallback_args=() + for arg in "${FM_LINT_WORKER_ARGS[@]}"; do + [ "$arg" = --external-sources ] || fallback_args+=("$arg") + done + [ -z "$LOCAL_NOX_EXCLUDE" ] || fallback_args+=("--exclude=$LOCAL_NOX_EXCLUDE") + fm_lint_exec_root "$path" "$fallback_out" "$fallback_err" "$fallback_rss" \ + "$fallback_secs" "${fallback_args[@]}" + final_follow_sources=0 + invocation_rc=$FM_LINT_LAST_RC + reason=$(fm_lint_classify_root "$invocation_rc" "$fallback_err") + rss_kib=$(fm_lint_max_root_rss \ + "$(fm_lint_root_rss "$rss_file")" "$(fm_lint_root_rss "$fallback_rss")") + printf 'fm-lint: %s hit the memory ceiling with --external-sources (reason=%s rc=%s); retried without it' \ + "$path" "$initial_reason" "$initial_rc" >> "$output_dir/shard.$shard_index.out" + if [ "$reason" = ok ]; then + reason=memory-fallback + invocation_rc=0 + printf '; fallback passed with cross-file codes excluded (%s)\n' "$LOCAL_NOX_EXCLUDE" \ + >> "$output_dir/shard.$shard_index.out" + else + printf '; fallback reason=%s rc=%s\n' "$reason" "$invocation_rc" \ + >> "$output_dir/shard.$shard_index.out" + fi + cat "$fallback_out" "$fallback_err" >> "$output_dir/shard.$shard_index.out" + else + rss_kib=$(fm_lint_root_rss "$rss_file") + cat "$root_out" "$root_err" >> "$output_dir/shard.$shard_index.out" + fi end_ms=$(fm_lint_now_ms) duration_ms=$((end_ms - start_ms)) - rss_kib=$(fm_lint_root_rss "$rss_file") - reason=$(fm_lint_classify_root "$invocation_rc" "$root_err") if [ -n "${FM_LINT_INTERNAL_ROOTS_LOG:-}" ]; then - printf 'end\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ + printf 'end\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ "$index" "$path" "$shard_index" "${FM_LINT_INTERNAL_MODE:-}" \ - "$start_ms" "$end_ms" "$duration_ms" "$invocation_rc" "$reason" "$rss_kib" \ + "$start_ms" "$end_ms" "$duration_ms" "$invocation_rc" "$reason" "$rss_kib" "$final_follow_sources" \ >> "$FM_LINT_INTERNAL_ROOTS_LOG" fi - if [ "${FM_LINT_INTERNAL_PROGRESS:-0}" = 1 ] || { [ "$reason" != ok ] && [ "$reason" != findings ]; }; then + if [ "${FM_LINT_INTERNAL_PROGRESS:-0}" = 1 ] || { [ "$reason" != ok ] && [ "$reason" != findings ] && [ "$reason" != memory-fallback ]; }; then printf 'fm-lint: end %s reason=%s rc=%s duration_ms=%s rss_kib=%s\n' \ "$path" "$reason" "$invocation_rc" "$duration_ms" "$rss_kib" >&2 fi - cat "$root_out" "$root_err" >> "$output_dir/shard.$shard_index.out" return "$invocation_rc" } @@ -1207,7 +1293,11 @@ if [ -n "$TELEMETRY" ]; then : > "$TMP_ROOT/source-targets" source_directives=0 source_boundaries=0 + source_followed=0 + awk -F '\t' '$1 == "end" && $12 == 0 { print $2 }' "$ROOTS_LOG" > "$TMP_ROOT/no-source-indices" + root_index=0 for path in "${ROOTS[@]}"; do + root_index=$((root_index + 1)) if [ -f "$path" ]; then bytes=$(wc -c < "$path" 2>/dev/null | tr -d '[:space:]') case "$bytes" in ''|*[!0-9]*) bytes=0 ;; esac @@ -1220,17 +1310,18 @@ if [ -n "$TELEMETRY" ]; then sub(/[[:space:]].*$/, "", target) print target } - ' "$path" >> "$TMP_ROOT/source-targets" + ' "$path" > "$TMP_ROOT/root-source-targets" + cat "$TMP_ROOT/root-source-targets" >> "$TMP_ROOT/source-targets" + if [ "$FOLLOW_SOURCES" -eq 1 ] \ + && ! grep -qx "$root_index" "$TMP_ROOT/no-source-indices"; then + followed_here=$(grep -cv '^/dev/null$' "$TMP_ROOT/root-source-targets" || true) + source_followed=$((source_followed + followed_here)) + fi fi done source_directives=$(wc -l < "$TMP_ROOT/source-targets" | tr -d '[:space:]') source_boundaries=$(grep -c '^/dev/null$' "$TMP_ROOT/source-targets" 2>/dev/null || true) case "$source_boundaries" in ''|*[!0-9]*) source_boundaries=0 ;; esac - if [ "$FOLLOW_SOURCES" -eq 1 ]; then - source_followed=$((source_directives - source_boundaries)) - else - source_followed=0 - fi source_targets=$(LC_ALL=C sort -u "$TMP_ROOT/source-targets" | wc -l | tr -d '[:space:]') content_cksum=$(cksum "$TMP_ROOT/content-cksums" | awk '{print $1 "-" $2}') git_head=$(git rev-parse HEAD 2>/dev/null || printf 'unavailable') diff --git a/bin/fm-parent-channel-lib.sh b/bin/fm-parent-channel-lib.sh index 24718258317..160d580ac02 100644 --- a/bin/fm-parent-channel-lib.sh +++ b/bin/fm-parent-channel-lib.sh @@ -123,6 +123,28 @@ fm_parent_channel_destination() { # esac } +# The outbound parent-channel status path that lives INSIDE , printed, +# when is a remote mate; non-zero for a main home, a local mate, or an +# unusable identity or binding. Only the remote route resolves the channel into +# the mate's own state dir, so parent-replies.status there is the mate's parent +# channel rather than a self-home task status file: a home's own status scans +# and decision folds exclude exactly this resolved path (the same special case +# fm-pending-reply-lib.sh's wrong-home detection applies). A local mate's +# channel lives in the parent home's state/.status, which the parent's +# scans must keep classifying, so only the remote route resolves here. +fm_parent_channel_outbound_status() { # + local home=$1 state=$2 destination rc=0 + destination=$(fm_parent_channel_destination "$home" "$state") || rc=$? + [ "$rc" -eq 0 ] || return 1 + # The substitution above ran the resolver in a subshell, so its route global + # died with it; resolve once more in this shell (stdout discarded, the same + # shape fm-pending-reply-lib.sh's wrong-home detection uses) so the route + # check reads the resolver's own verdict rather than re-deriving it. + fm_parent_channel_destination "$home" "$state" >/dev/null || return 1 + [ "$FM_PARENT_CHANNEL_ROUTE" = remote ] || return 1 + printf '%s\n' "$destination" +} + # Fold onto one bounded line, so a note copied from a child ledger or a # hold reason cannot break the channel's line framing. fm_parent_channel_clean_note() { # diff --git a/bin/fm-pending-reply-lib.sh b/bin/fm-pending-reply-lib.sh index 46eeb3372fa..e9685063861 100755 --- a/bin/fm-pending-reply-lib.sh +++ b/bin/fm-pending-reply-lib.sh @@ -11,8 +11,10 @@ # Safety property (captain direction 2026-07-22): a secondmate agent may ignore # the marker and answer only in its visible conversation. The parent must notice # the missing correlated report without scraping that conversation, send exactly -# one automatic recovery request asking for a repost through the parent channel, -# and escalate once if the recovery turn also completes without a correlated +# one automatic recovery request asking for a repost through the parent channel +# (held back, when config/wait-no-turns is present, while the mate waits on its +# own open decision or blocker), and +# escalate once if the recovery turn also completes without a correlated # report. Never loop, never repeatedly inject, never silently expire unresolved # records, and never treat wrong-home or structured-home heuristics as # acknowledgement. A same-basename restatement-copy of the mate home's @@ -963,6 +965,11 @@ fm_pending_reply_send_recovery() { # task_id=$(fm_pending_reply_get "$rec" task_id) # A remote mate's report may exist and simply not have been mirrored yet. fm_pending_reply_missing_report_is_evidence "$state" "$task_id" "$completed" || return 1 + # config/wait-no-turns: a mate waiting on its own open decision or blocker + # is never poked. The recovery stays unattempted until the answer lands. + if [ -e "${FM_CONFIG_OVERRIDE:-${FM_HOME:-}/config}/wait-no-turns" ]; then + [ -z "$(status_own_open_decisions "$state/$task_id.status")" ] || return 1 + fi status_file=$(fm_pending_reply_get "$rec" parent_status) parent_home=$(fm_pending_reply_get "$rec" parent_home) msg=$(fm_pending_reply_recovery_message "$rec") diff --git a/bin/fm-remote-delta-read.sh b/bin/fm-remote-delta-read.sh index d4c26bd6697..84aef13d051 100755 --- a/bin/fm-remote-delta-read.sh +++ b/bin/fm-remote-delta-read.sh @@ -10,6 +10,16 @@ # the source. A shortened or changed prefix returns a structured continuity-break # result instead of silently rebasing the cursor. # +# The log is sampled every FM_REMOTE_DELTA_POLL_SECONDS (default 0.5 seconds). +# A complete line is visible on the next sample, and the window deadline can +# overshoot by that interval plus snapshot and scheduling work. +# Each sample of an existing log stats it once. The first sample always runs +# the bounded capture and hashing; later samples skip that work only when the +# size, subsecond mtime and ctime, inode, and device key is unchanged. If either +# timestamp lacks a nonzero subsecond fraction, every sample captures the log +# rather than trusting a coarse key that could hide a same-second rewrite. +# The wait remains an ordinary child sleep; signal handling is unchanged. +# # Exit 75 means the wait window closed with no complete line. SIGTERM exits the # same way after cleanup. The remote job worker preempts this read-only poll to # unblock any queued command other than another reply long-poll, then publishes @@ -19,7 +29,7 @@ set -eu FM_HOME=${FM_HOME:?FM_HOME is required} MAX_BYTES=${FM_REMOTE_DELTA_MAX_BYTES:-65536} -POLL_SECONDS=${FM_REMOTE_DELTA_POLL_SECONDS:-0.2} +POLL_SECONDS=${FM_REMOTE_DELTA_POLL_SECONDS:-0.5} die() { printf 'error: %s\n' "$1" >&2; exit 1; } usage() { sed -n '2,11p' "$0" | sed 's/^# \{0,1\}//'; exit 2; } @@ -79,6 +89,30 @@ snapshot_log() { # ) } +delta_subsecond() { # : digits, one dot, and a nonzero fraction + case "$1" in *[!0-9.]* | *.*.*) return 1 ;; esac + case "$1" in [0-9]*.*[1-9]*) ;; *) return 1 ;; esac +} + +# The file identity a snapshot was taken against: GNU and BSD stat spell the +# fields differently, so the poll selects the syntax once by capability. The +# mtime and ctime keep their subsecond fraction; a key without one (a stat or +# filesystem with whole-second timestamps) is discarded, because it cannot tell +# a same-second same-size rewrite apart, and that poll takes a full snapshot. +delta_log_key() { # : sets KEY to "size:mtime:ctime:inode:device" or empty + local rest mtime ctime + if [ "$DELTA_KEY_GNU_STAT" = 1 ]; then + KEY=$(stat -c '%s:%.9Y:%.9Z:%i:%d' "$1" 2>/dev/null) || KEY= + else + KEY=$(stat -f '%z:%Fm:%Fc:%i:%d' "$1" 2>/dev/null) || KEY= + fi + rest=${KEY#*:} + mtime=${rest%%:*} + rest=${rest#*:} + ctime=${rest%%:*} + delta_subsecond "$mtime" && delta_subsecond "$ctime" || KEY= +} + resolve_log() { # local rel=$1 home_real parent_real parent base path case "$rel" in ''|/*|*'//'*) die "log must be a nonempty relative path" ;; esac @@ -129,61 +163,69 @@ trap 'rm -rf -- "$TMP"' EXIT trap 'exit 75' TERM : > "$TMP/empty" EMPTY_HASH=$(sha256_file "$TMP/empty") -START=$(date +%s) +if stat -c '%s' / >/dev/null 2>&1; then DELTA_KEY_GNU_STAT=1; else DELTA_KEY_GNU_STAT=0; fi +START=$SECONDS +LAST_KEY= while :; do if [ -e "$LOG" ] || [ -L "$LOG" ]; then [ -f "$LOG" ] && [ ! -L "$LOG" ] || die "log changed into an unsafe file: $REL" - snapshot_log "$LOG" "$TMP/source" "$TMP/size" \ - || die "log could not be captured safely: $REL" - SIZE=$(tr -d ' ' < "$TMP/size") - if [ "$SIZE" -lt "$OFFSET" ]; then - copy_prefix "$TMP/source" "$SIZE" "$TMP/prefix" - ACTUAL=$(sha256_file "$TMP/prefix") - emit_break truncated "$SIZE" "$ACTUAL" - exit 0 - fi - copy_prefix "$TMP/source" "$OFFSET" "$TMP/prefix" - ACTUAL=$(sha256_file "$TMP/prefix") - if [ "$ACTUAL" != "$PREFIX" ]; then - emit_break prefix-changed "$SIZE" "$ACTUAL" - exit 0 - fi - if [ "$SIZE" -gt "$OFFSET" ]; then - tail -c "+$((OFFSET + 1))" "$TMP/source" | head -c "$MAX_BYTES" > "$TMP/chunk" || true - COMPLETE_BYTES=$(LC_ALL=C od -An -v -tu1 "$TMP/chunk" | awk ' - { for (i = 1; i <= NF; i++) { bytes++; if ($i == 10) complete=bytes } } - END { print complete + 0 } - ') - if [ "$COMPLETE_BYTES" -eq 0 ]; then : > "$TMP/payload"; else head -c "$COMPLETE_BYTES" "$TMP/chunk" > "$TMP/payload"; fi - BYTES=$(LC_ALL=C wc -c < "$TMP/payload" | tr -d ' ') - if [ "$BYTES" -gt 0 ]; then - TO=$((OFFSET + BYTES)) - copy_prefix "$TMP/source" "$TO" "$TMP/to-prefix" - TO_HASH=$(sha256_file "$TMP/to-prefix") - PAYLOAD_HASH=$(sha256_file "$TMP/payload") - printf 'schema=fm-remote-delta.v1\n' - printf 'status=delta\n' - printf 'path=%s\n' "$REL" - printf 'from_offset=%s\n' "$OFFSET" - printf 'to_offset=%s\n' "$TO" - printf 'from_prefix_sha256=%s\n' "$PREFIX" - printf 'to_prefix_sha256=%s\n' "$TO_HASH" - printf 'payload_sha256=%s\n' "$PAYLOAD_HASH" - printf 'payload_bytes=%s\n' "$BYTES" - printf 'reason=\n\n' - cat "$TMP/payload" + delta_log_key "$LOG" + if [ -z "$KEY" ] || [ "$KEY" != "$LAST_KEY" ]; then + snapshot_log "$LOG" "$TMP/source" "$TMP/size" \ + || die "log could not be captured safely: $REL" + # The gate stat precedes the capture, so the snapshot is at least as new + # as its key: a log that moved in between changes the key and is + # captured again on the next poll, never mistaken for stable. + LAST_KEY=$KEY + IFS= read -r SIZE < "$TMP/size" + if [ "$SIZE" -lt "$OFFSET" ]; then + copy_prefix "$TMP/source" "$SIZE" "$TMP/prefix" + ACTUAL=$(sha256_file "$TMP/prefix") + emit_break truncated "$SIZE" "$ACTUAL" exit 0 fi - if [ $((SIZE - OFFSET)) -ge "$MAX_BYTES" ]; then - emit_break line-exceeds-bound "$SIZE" "$ACTUAL" + copy_prefix "$TMP/source" "$OFFSET" "$TMP/prefix" + ACTUAL=$(sha256_file "$TMP/prefix") + if [ "$ACTUAL" != "$PREFIX" ]; then + emit_break prefix-changed "$SIZE" "$ACTUAL" exit 0 fi + if [ "$SIZE" -gt "$OFFSET" ]; then + tail -c "+$((OFFSET + 1))" "$TMP/source" | head -c "$MAX_BYTES" > "$TMP/chunk" || true + COMPLETE_BYTES=$(LC_ALL=C od -An -v -tu1 "$TMP/chunk" | awk ' + { for (i = 1; i <= NF; i++) { bytes++; if ($i == 10) complete=bytes } } + END { print complete + 0 } + ') + if [ "$COMPLETE_BYTES" -eq 0 ]; then : > "$TMP/payload"; else head -c "$COMPLETE_BYTES" "$TMP/chunk" > "$TMP/payload"; fi + BYTES=$(LC_ALL=C wc -c < "$TMP/payload" | tr -d ' ') + if [ "$BYTES" -gt 0 ]; then + TO=$((OFFSET + BYTES)) + copy_prefix "$TMP/source" "$TO" "$TMP/to-prefix" + TO_HASH=$(sha256_file "$TMP/to-prefix") + PAYLOAD_HASH=$(sha256_file "$TMP/payload") + printf 'schema=fm-remote-delta.v1\n' + printf 'status=delta\n' + printf 'path=%s\n' "$REL" + printf 'from_offset=%s\n' "$OFFSET" + printf 'to_offset=%s\n' "$TO" + printf 'from_prefix_sha256=%s\n' "$PREFIX" + printf 'to_prefix_sha256=%s\n' "$TO_HASH" + printf 'payload_sha256=%s\n' "$PAYLOAD_HASH" + printf 'payload_bytes=%s\n' "$BYTES" + printf 'reason=\n\n' + cat "$TMP/payload" + exit 0 + fi + if [ $((SIZE - OFFSET)) -ge "$MAX_BYTES" ]; then + emit_break line-exceeds-bound "$SIZE" "$ACTUAL" + exit 0 + fi + fi fi elif [ "$OFFSET" -ne 0 ] || [ "$PREFIX" != "$EMPTY_HASH" ]; then emit_break missing 0 "$EMPTY_HASH" exit 0 fi - NOW=$(date +%s) - [ $((NOW - START)) -lt "$WAIT" ] || exit 75 + [ $((SECONDS - START)) -lt "$WAIT" ] || exit 75 sleep "$POLL_SECONDS" done diff --git a/bin/fm-remote-job-lib.sh b/bin/fm-remote-job-lib.sh index 9a526c5be99..0b6076c96b8 100755 --- a/bin/fm-remote-job-lib.sh +++ b/bin/fm-remote-job-lib.sh @@ -55,6 +55,18 @@ # Abandoned .stage.* staging litter older than # FM_REMOTE_JOB_STAGE_REAP_SECONDS is reaped by the worker's stale sweep. # +# Result consumers and active-command monitors sample every 0.25 seconds by +# default; the dispatcher's post-activity burst still samples every 0.05 seconds. +# FM_REMOTE_JOB_ACTIVE_POLL_SECONDS overrides the active/result interval; an +# explicitly supplied FM_REMOTE_JOB_POLL_SECONDS remains the legacy fallback +# for both intervals. Resolve the active default before filling the dispatcher +# default, and retain it when the library is sourced again. +# Once-per-second cancellation, preemption, and disconnect checks can overshoot +# their due time by one sampling interval plus work/scheduling time, as can the +# active command's timeout check. Completion and result collection can each add +# one interval. Sleeps stay ordinary child processes: existing signal handlers +# and the separate cancellation/preemption TERM-to-KILL grace are unchanged. +# # The worker accepts only a tracked, non-symlink executable named fm-*.sh below # its configured FM_ROOT/bin. Every child receives env -i with the composed # PATH, HOME, FM_HOME, FM_ROOT_OVERRIDE, and FM_REMOTE_JOB_ACTIVE=1. The PATH @@ -89,6 +101,7 @@ FM_REMOTE_JOB_MAX_BYTES=${FM_REMOTE_JOB_MAX_BYTES:-1048576} FM_REMOTE_JOB_QUEUE_TIMEOUT=${FM_REMOTE_JOB_QUEUE_TIMEOUT:-360} FM_REMOTE_JOB_TIMEOUT=${FM_REMOTE_JOB_TIMEOUT:-360} FM_REMOTE_JOB_WAIT_GRACE=${FM_REMOTE_JOB_WAIT_GRACE:-30} +FM_REMOTE_JOB_ACTIVE_POLL_SECONDS=${FM_REMOTE_JOB_ACTIVE_POLL_SECONDS:-${FM_REMOTE_JOB_POLL_SECONDS:-0.25}} FM_REMOTE_JOB_POLL_SECONDS=${FM_REMOTE_JOB_POLL_SECONDS:-0.05} FM_REMOTE_JOB_REAP_SECONDS=${FM_REMOTE_JOB_REAP_SECONDS:-3600} FM_REMOTE_JOB_STAGE_REAP_SECONDS=${FM_REMOTE_JOB_STAGE_REAP_SECONDS:-600} @@ -489,15 +502,43 @@ fm_remote_job_write_state() { # queued|running|done mv -f -- "$tmp" "$job/state" } -fm_remote_job_read_state() { # - local job=$1 value extra - fm_remote_job_regular_bounded "$job/state" 64 || return 1 - IFS= read -r value < "$job/state" || return 1 - if IFS= read -r extra < <(tail -n +2 "$job/state"); then - : "$extra" - return 1 +# Reads a one-line record bounded to bytes with builtins only, matching +# fm_remote_job_regular_bounded plus the former read/tail checks: a regular +# non-symlink file of at most bytes, one newline-terminated line, a +# tolerated unterminated tail, no carriage returns, and a non-empty value. +# The -d '' -n read treats NUL as the delimiter, so an ordinary +# record (no NULs) is pulled whole at once: the read fails at end of file, +# and success means either bytes landed (the file busts the +# bound) or a NUL stopped it early (already malformed). -N cannot do this: +# the stock /bin/bash on macOS is 3.2, which has -n but no -N. The local +# LC_ALL=C makes -n count bytes rather than multibyte characters, so the byte +# bound holds in a UTF-8 locale. +fm_remote_job_read_line() { # + local file=$1 max=$2 result_var=$3 content + local LC_ALL=C + [ -f "$file" ] && [ ! -L "$file" ] || return 1 + ! IFS= read -r -d '' -n "$((max + 1))" content < "$file" 2>/dev/null || return 1 + case "$content" in *$'\r'* | *$'\n'*$'\n'*) return 1 ;; esac + case "$content" in *$'\n'*) ;; *) return 1 ;; esac + content=${content%%$'\n'*} + [ -n "$content" ] || return 1 + printf -v "$result_var" '%s' "$content" +} + +# Reads the one-word state record with builtins only: the result consumers and +# the lane preemption scan call this once per sample, so it cannot afford the +# bounded-size subshell or a tail process substitution. Passing a result +# variable name avoids the command substitution fork; without one the value is +# printed as before. +fm_remote_job_read_state() { # [result-variable] + local job=$1 result_var=${2:-} read_value + fm_remote_job_read_line "$job/state" 64 read_value || return 1 + case "$read_value" in queued|running|'done') ;; *) return 1 ;; esac + if [ -n "$result_var" ]; then + printf -v "$result_var" '%s' "$read_value" + else + printf '%s\n' "$read_value" fi - case "$value" in queued|running|'done') printf '%s\n' "$value" ;; *) return 1 ;; esac } fm_remote_job_read_number() { # queue_deadline|timeout|deadline|seq @@ -680,7 +721,7 @@ fm_remote_job_stage() { # [args...]; stdi fm_remote_job_wait() { # ; honors FM_REMOTE_JOB_DISCONNECT_PROBE local account_home=$1 id=$2 job state queue_deadline execution_timeout wait_deadline exit_value - local now next_probe=0 + local deadline_ticks next_probe=0 fm_remote_job_prepare_state "$account_home" || return 1 job=$(fm_remote_job_job_dir "$id") || { FM_REMOTE_JOB_ERROR="remote job record disappeared or became unsafe" @@ -699,8 +740,12 @@ fm_remote_job_wait() { # ; honors FM_REMOTE_JOB_DISCONNECT_PR return 1 } wait_deadline=$((queue_deadline + execution_timeout + FM_REMOTE_JOB_WAIT_GRACE)) + # SECONDS is the loop's clock so no time child runs per sample: one date + # read here converts the epoch deadline into the shell's own tick counter + # with the same whole-second granularity. + deadline_ticks=$((SECONDS + wait_deadline - $(date +%s))) while :; do - state=$(fm_remote_job_read_state "$job" 2>/dev/null || true) + fm_remote_job_read_state "$job" state 2>/dev/null || state= case "$state" in 'done') if ! fm_remote_job_regular_bounded "$job/stdout" "$FM_REMOTE_JOB_MAX_BYTES" || @@ -723,20 +768,19 @@ fm_remote_job_wait() { # ; honors FM_REMOTE_JOB_DISCONNECT_PR queued|running) ;; *) FM_REMOTE_JOB_ERROR="remote job state is invalid"; return 1 ;; esac - now=$(date +%s) - if [ "$now" -ge "$wait_deadline" ]; then + if [ "$SECONDS" -ge "$deadline_ticks" ]; then FM_REMOTE_JOB_ERROR="remote job did not complete within its bounded wait" return 1 fi - if [ -n "${FM_REMOTE_JOB_DISCONNECT_PROBE:-}" ] && [ "$now" -ge "$next_probe" ]; then - next_probe=$((now + 1)) + if [ -n "${FM_REMOTE_JOB_DISCONNECT_PROBE:-}" ] && [ "$SECONDS" -ge "$next_probe" ]; then + next_probe=$((SECONDS + 1)) if ! "$FM_REMOTE_JOB_DISCONNECT_PROBE"; then fm_remote_job_cancel "$account_home" "$id" 2>/dev/null || true FM_REMOTE_JOB_ERROR="remote job caller disconnected; the job was cancelled" return 1 fi fi - sleep "$FM_REMOTE_JOB_POLL_SECONDS" + sleep "$FM_REMOTE_JOB_ACTIVE_POLL_SECONDS" done } diff --git a/bin/fm-remote-job-worker.sh b/bin/fm-remote-job-worker.sh index 291ec9fa45b..3a9c52dca23 100755 --- a/bin/fm-remote-job-worker.sh +++ b/bin/fm-remote-job-worker.sh @@ -29,11 +29,12 @@ # does not depend on the worker it is consolidating. # # The serving loop does not busy-poll an idle queue. After a lane starts or is -# reaped it rescans every FM_REMOTE_JOB_POLL_SECONDS for 20 passes, so a home +# reaped it rescans every FM_REMOTE_JOB_POLL_SECONDS for four passes, so a home # whose lane just finished starts its next job promptly; otherwise it sleeps -# one second between passes. That bound is how long newly staged or cancelled -# work, a lane that died, an orphaned claim, or an expired queue deadline can -# wait for the next pass, and it refreshes the readiness heartbeat about once +# one second between passes. Work arriving after the four-pass burst may wait +# for that quiet scan. Newly staged or cancelled work, a lane that died, an +# orphaned claim, or an expired queue deadline can wait that interval plus +# scan work and scheduling time. It refreshes the readiness heartbeat about once # per second, far inside the probe's 10-second freshness bound. The stale # sweep, whose state preparation also re-applies the queue directories' 0700 # modes, runs at startup and then at most every 60 seconds, never more rarely @@ -67,7 +68,7 @@ FM_REMOTE_JOB_ORPHAN_GRACE_SECONDS=$(worker_bounded_setting "${FM_REMOTE_JOB_ORP FM_REMOTE_JOB_SUPERVISOR_MAX_RESTARTS=$(worker_bounded_setting "${FM_REMOTE_JOB_SUPERVISOR_MAX_RESTARTS:-}" 20) FM_REMOTE_JOB_SUPERVISOR_MAX_BACKOFF_SECONDS=$(worker_bounded_setting "${FM_REMOTE_JOB_SUPERVISOR_MAX_BACKOFF_SECONDS:-}" 5) FM_REMOTE_JOB_SUPERVISOR_HEALTHY_SECONDS=$(worker_bounded_setting "${FM_REMOTE_JOB_SUPERVISOR_HEALTHY_SECONDS:-}" 10) -WORKER_FAST_PASSES=20 +WORKER_FAST_PASSES=4 WORKER_IDLE_WAIT_SECONDS=1 WORKER_SWEEP_SECONDS=60 @@ -749,7 +750,7 @@ worker_run_with_timeout() { # [args...] fi next_check=$((SECONDS + 1)) fi - sleep "$FM_REMOTE_JOB_POLL_SECONDS" + sleep "$FM_REMOTE_JOB_ACTIVE_POLL_SECONDS" done wait "$group_pid" 2>/dev/null rc=$? @@ -760,25 +761,49 @@ worker_run_with_timeout() { # [args...] return "$rc" } -worker_job_command() { # ; the first argv element of a staged record - local job=$1 first= - fm_remote_job_regular_bounded "$job/argv" "$FM_REMOTE_JOB_MAX_BYTES" || return 1 - IFS= read -r -d '' first < "$job/argv" || [ -n "$first" ] || return 1 - printf '%s\n' "$first" -} - worker_preempting_waiter_exists() { # - local lane_home=$1 job state command job_home + local lane_home=$1 job state command job_home field_terminated remaining chunk + # The argv byte bound counts with read -n and ${#...}, which count bytes only + # in the C locale. + local LC_ALL=C for job in "$FM_REMOTE_JOB_JOBS"/job-*; do [ -d "$job" ] && [ ! -L "$job" ] || continue - state=$(fm_remote_job_read_state "$job" 2>/dev/null || true) + fm_remote_job_read_state "$job" state 2>/dev/null || continue [ "$state" = queued ] || continue fm_remote_job_cancelled "$job" && continue # Lanes are per home, so only a waiter for this lane's own home may - # preempt; another home's queue drains through its own lane. - job_home=$(worker_read_text "$job" home 8192 2>/dev/null || true) + # preempt; another home's queue drains through its own lane. The record + # fields are read with builtins only: this scan runs once a second in + # every lane that executes a preemptible long poll, so no field read may + # spawn a child process. + fm_remote_job_read_line "$job/home" 8192 job_home 2>/dev/null || job_home= [ "$job_home" = "$lane_home" ] || continue - command=$(worker_job_command "$job" 2>/dev/null || true) + # The staged argv record must fit within FM_REMOTE_JOB_MAX_BYTES: bound + # the first NUL-delimited field, then walk the remaining NUL-terminated + # fields and any unterminated tail, still with builtins only. -d '' -n + # is the bounded read on the macOS stock bash (3.2 has -n but no -N); + # never pass -n 0, whose behavior diverges across bash versions. + command= + if [ -f "$job/argv" ] && [ ! -L "$job/argv" ]; then + { field_terminated= + IFS= read -r -d '' -n "$((FM_REMOTE_JOB_MAX_BYTES + 1))" command && field_terminated=1 + if [ -n "$field_terminated" ]; then + if [ "${#command}" -gt "$FM_REMOTE_JOB_MAX_BYTES" ]; then + false + else + remaining=$((FM_REMOTE_JOB_MAX_BYTES - ${#command} - 1)) + chunk= + while [ "$remaining" -ge 0 ] && IFS= read -r -d '' -n "$((remaining + 1))" chunk; do + [ "${#chunk}" -le "$remaining" ] || break + remaining=$((remaining - ${#chunk} - 1)) + done + remaining=$((remaining - ${#chunk})) + [ "$remaining" -ge 0 ] + fi + else + [ -n "$command" ] + fi; } < "$job/argv" 2>/dev/null || command= + fi fm_remote_job_command_preemptible "$command" || return 0 done return 1 diff --git a/bin/fm-send.sh b/bin/fm-send.sh index af09392a4d0..19562680313 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -51,7 +51,9 @@ # watcher re-rings an unacknowledged message while its endpoint remains # available, escalates after the bounded ladder, and instead routes a positively # dead or missing endpoint directly to recovery without typing. An explicit -# fire-and-forget record is excluded from that ladder. +# fire-and-forget record is excluded from that ladder; when config/wait-no-turns +# is present and its ring here was skipped or failed, the watcher rings it +# exactly once more. # bin/fm-task-inbox-lib.sh owns the record format, the doorbell line, and the # re-ring ladder. The composer pre-check before the ring is ADVISORY only: when # the composer visibly holds pending text the ring is skipped with a notice and @@ -1085,9 +1087,22 @@ else # bounded re-ring ladder or direct unavailable-endpoint recovery. ring_rc=0 fm_task_inbox_ring "$TARGET_BACKEND" "$T" "$INBOX_RECORD" "$EXPECTED_LABEL" || ring_rc=$? + ring_retry="the watcher will re-ring" + if [ -n "$FIRE_AND_FORGET_ID" ] \ + && [ -e "${FM_CONFIG_OVERRIDE:-$FM_HOME/config}/wait-no-turns" ]; then + case "$ring_rc" in + 1|2) + if fm_task_inbox_mark_retry "$STATE" "$INBOX_TASK_ID" "$INBOX_RECORD"; then + ring_retry="the watcher will ring it once more" + else + ring_retry="its one retry ring could not be recorded, so nothing will ring it again" + fi + ;; + esac + fi case "$ring_rc" in - 1) echo "fm-send: doorbell skipped (composer visibly holds pending text); the steer is durably recorded at $INBOX_RECORD and the watcher will re-ring" >&2 ;; - 2) echo "fm-send: doorbell did not reach $T; the steer is durably recorded at $INBOX_RECORD and the watcher will re-ring" >&2 ;; + 1) echo "fm-send: doorbell skipped (composer visibly holds pending text); the steer is durably recorded at $INBOX_RECORD and $ring_retry" >&2 ;; + 2) echo "fm-send: doorbell did not reach $T; the steer is durably recorded at $INBOX_RECORD and $ring_retry" >&2 ;; 3) echo "fm-send: doorbell not typed because the agent in $T has exited; the steer is durably recorded at $INBOX_RECORD for recovery (stuck-crewmate-recovery), and the watcher will not re-ring a dead pane" >&2 ;; esac exit 0 diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index bb69d371525..652b1da808c 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -383,6 +383,8 @@ PRIMARY_HARNESS=$("$SCRIPT_DIR/fm-harness.sh" 2>/dev/null || printf unknown) . "$SCRIPT_DIR/fm-wake-lib.sh" # shellcheck source=bin/fm-line-cap-lib.sh . "$SCRIPT_DIR/fm-line-cap-lib.sh" +# shellcheck source=bin/fm-hold-reason-lib.sh +. "$SCRIPT_DIR/fm-hold-reason-lib.sh" # One tasks-axi compatibility verdict per session start. The probe costs three # tasks-axi subprocesses and this digest needs the same answer twice - here for @@ -482,7 +484,7 @@ print_backlog_manual_compact() { } } } - ' "$path" + ' "$path" | fm_hold_reason_decode_stream markdown } # tasks-axi closes every listing with its own help block. This section composes @@ -534,11 +536,11 @@ print_backlog_tasks_axi_compact() { printf 'compact backlog listing (tasks-axi; done rows omitted; every in-flight, held, and blocked row shown in full; ready queued bounded to %s; task bodies omitted)\n' \ "$QUEUED_LIMIT" printf '\nin flight:\n' - printf '%s\n' "$in_flight" | strip_axi_help + printf '%s\n' "$in_flight" | fm_hold_reason_decode_stream | strip_axi_help printf '\nheld (captain- or time-gated; an in-flight item that is also held appears in both groups):\n' - printf '%s\n' "$held" | strip_axi_help + printf '%s\n' "$held" | fm_hold_reason_decode_stream | strip_axi_help printf '\nblocked queued:\n' - printf '%s\n' "$blocked" | strip_axi_help + printf '%s\n' "$blocked" | fm_hold_reason_decode_stream | strip_axi_help printf '\nready queued (dispatchable now):\n' print_ready_queued_bounded "$ready" return 0 diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 2e5a089b191..375d91eebf4 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -196,6 +196,12 @@ # name from PATH once, probes that concrete path with --help, and launches the # same path. It adds --tui-mode regular only when that help advertises the flag; # a failed or inconclusive probe omits it so older Pi versions remain launchable. +# A --secondmate launch of a Firstmate-seeded home (the existing +# .fm-secondmate-home marker validate_firstmate_home_for_spawn already requires) +# also adds --approve when that help advertises it, so the first unattended +# launch does not stall on Pi's "Trust project folder?" dialog for that home +# path; --approve is session-scoped to the launch cwd and does not rewrite the +# operator's trust.json. Ordinary Pi worker launches never receive --approve. # A missing selected executable refuses before endpoint creation, and pi-signed # never falls back to pi. # Devin is worker-only: the recorded --permission-mode (dangerous unless @@ -337,7 +343,9 @@ # pins to 1 with a literal assignment so it survives the cleared environment # even on a host that never had it set. # An enabled task trace also retains TRACEPARENT. Explicit Firstmate launch -# assignments still apply inside the filtered environment. Raw commands must +# assignments still apply inside the filtered environment, including the +# FM_TASK_INBOX export every launch carries (the absolute state/.inbox +# path the steering doorbell names). Raw commands must # be POSIX sh compatible under this opt-in; the absent-file path is unchanged. # This is an exec environment boundary, not a sandbox for the pane's startup # shell, credential files, same-user processes, or later shell initialization. @@ -376,6 +384,9 @@ # supplies its own trailing space, empty never used) # __PIBIN__ quoted concrete Pi-family executable path resolved from PATH # __PITUIMODE__ optional --tui-mode regular when that executable advertises it +# __PIAPPROVE__ optional --approve on a seeded Pi/pi-signed secondmate when +# that executable advertises the flag (empty otherwise; session +# trust for the launch cwd only, never a trust.json rewrite) # __PIRESUME__ optional relaunch-only `--session ` that keeps a # Pi replacement on the session the endpoint's runtime already # reports (relaunch_resume_args below owns it; it supplies its @@ -1951,6 +1962,17 @@ pi_supports_tui_mode() { printf '%s\n' "$help" | grep -Eq -- '(^|[[:space:]])--tui-mode([[:space:]=]|$)' } +# Same help-probe shape as pi_supports_tui_mode for the session-scoped project +# trust flag. A seeded secondmate home carries tracked .pi/extensions that gate +# Pi behind "Trust project folder?" on first launch; --approve trusts that +# launch cwd for the run without rewriting ~/.pi/agent/trust.json. +pi_supports_approve() { + local executable=$1 help + help=$("$executable" --help 2>&1) || return 1 + # Pi prints "--approve, -a"; allow comma (and any non-token char) after the name. + printf '%s\n' "$help" | grep -Eq -- '(^|[[:space:]])--approve([^[:alnum:]_-]|$)' +} + # omp pre-launch model validation. `omp models --json` (omp 18.1.11) prints # {"models":[{"provider","id","selector":"/",...}]} for built-in and # auto-discovered providers only; it never lists a provider an extension @@ -2105,7 +2127,7 @@ launch_template() { ;; opencode) printf '%s' 'OPENCODE_CONFIG_CONTENT='\''{"permission":{"*":"allow"}__EFFORTFLAG__}'\'' opencode __MODELFLAG__--prompt "$(__OPINPUT__ encode launch-brief < __BRIEF__)"' ;; pi | pi-signed) - printf '%s' '__PIBIN____PITUIMODE____PIRESUME__' + printf '%s' '__PIBIN____PITUIMODE____PIAPPROVE____PIRESUME__' if [ "$kind" = secondmate ]; then printf '%s' ' __MODELFLAG____EFFORTFLAG__-e __PITURNEND__ -e __PIWATCH__ "$(__OPINPUT__ encode launch-brief < __BRIEF__)"' else @@ -2373,6 +2395,15 @@ pi | pi-signed) PI_TUI_MODE=' --tui-mode regular' fi LAUNCH=${LAUNCH//__PITUIMODE__/$PI_TUI_MODE} + # Seeded-home signal is .fm-secondmate-home (required by + # validate_firstmate_home_for_spawn before any secondmate launch reaches + # the pane). Session-only --approve; never expand to a parent path or + # rewrite the operator trust store. + PI_APPROVE= + if [ "$KIND" = secondmate ] && pi_supports_approve "$PI_BIN"; then + PI_APPROVE=' --approve' + fi + LAUNCH=${LAUNCH//__PIAPPROVE__/$PI_APPROVE} LAUNCH="FM_PI_HARNESS=$HARNESS $LAUNCH" ;; cursor) @@ -5511,6 +5542,12 @@ fi if [ "$LAVISH_AXI_HOST_CONFIG_PRESENT" = 1 ]; then LAUNCH="export LAVISH_AXI_HOST=$(shell_quote "$LAVISH_AXI_HOST"); $LAUNCH" fi +# Every launch also exports the absolute path of this task's steering inbox, so +# the constant doorbell line (bin/fm-task-inbox-lib.sh) can name +# "$FM_TASK_INBOX" instead of a path that grows with the home's depth. Like the +# kill switch below it is an export statement, so it survives a compound raw +# launch and the launch-env-allowlist `env -i` wrapper. +LAUNCH="export FM_TASK_INBOX=$(shell_quote "$STATE_REAL/$ID.inbox"); $LAUNCH" LAUNCH="export COMPACT_ADVISER_DISABLE=1; $LAUNCH" # When the live-harness gate has exported DISABLE_AUTOUPDATER into this spawn's # own environment, carry it into the launch command text so Claude Code's diff --git a/bin/fm-supervise-daemon.sh b/bin/fm-supervise-daemon.sh index dbd33c2abd9..07ae71e7375 100755 --- a/bin/fm-supervise-daemon.sh +++ b/bin/fm-supervise-daemon.sh @@ -69,9 +69,10 @@ # undelivered past FM_MAX_DEFER_SECS, the daemon retries a normal flush and # writes state/.subsuper-inject-wedged and attempts a configurable active # alert if submit still cannot be confirmed. -# - Cheap heartbeat catch-all: every HEARTBEAT_SCAN_SECS the daemon greps all -# state/*.status for a captain-relevant line the per-wake classifier might -# have missed (e.g. a status verb outside CAPTAIN_RE) and escalates it. +# - Cheap heartbeat catch-all: every HEARTBEAT_SCAN_SECS the daemon greps the +# state dir's task status logs for a captain-relevant line the per-wake +# classifier might have missed (e.g. a status verb outside CAPTAIN_RE) and +# escalates it. # # The robustness shell from the prior always-inject version is preserved: # single-instance lock (portable helper, no flock dependency), crash-loop @@ -1215,8 +1216,8 @@ _oldest_line_age() { # -> seconds since the oldest buffered item first ar # re-peek; gone -> clear; still declaring the wait, on an idle OR a busy pane # -> escalate a recheck digest naming which human the wait is on, and reset # the window (repeating bounded re-surface, never a wedge). -# 3) heartbeat scan: every HEARTBEAT_SCAN_SECS, grep state/*.status for a -# captain-relevant line the per-wake classifier missed and escalate it. +# 3) heartbeat scan: every HEARTBEAT_SCAN_SECS, run the catch-all status scan in +# the block below and escalate what it finds; that block owns its file set. housekeeping() { # local state=$1 now due f key task win marker age last max_defer oldest pause_secs marker_epoch until bounded_until pause_reason run_id jsf jts jage now=$(_now) @@ -1400,11 +1401,17 @@ housekeeping() { # # because the event this backstop most needs to catch is precisely one a # later routine append has already moved past; fm-classify-lib.sh's span # read decides relevance, and the classified-through offset is the dedup. + # A remote mate's own parent channel is not a self-home task status log, + # so it is excluded here exactly as in the watcher's twin backstop + # (fm-watch.sh heartbeat_scan_finds_actionable); the home-shape-aware + # resolution lives in status_scan_parent_channel_exclude. if [ "$(_file_age "$state/.subsuper-last-scan")" -ge "${FM_HEARTBEAT_SCAN_SECS:-$HEARTBEAT_SCAN_SECS_DEFAULT}" ]; then _now > "$state/.subsuper-last-scan" - local event record rest endpoint ident rc + local event record rest endpoint ident rc exclude + exclude=$(status_scan_parent_channel_exclude "$state") for f in "$state"/*.status; do [ -e "$f" ] || [ -L "$f" ] || continue + [ "$f" = "$exclude" ] && continue task=$(basename "$f"); task="${task%.status}" record='' status_span_first_actionable_record "$f" \ diff --git a/bin/fm-supervision-host.sh b/bin/fm-supervision-host.sh index c0dd9905075..26555b69633 100755 --- a/bin/fm-supervision-host.sh +++ b/bin/fm-supervision-host.sh @@ -54,8 +54,11 @@ # before the close is printed, so supervision continues when the session # drops the handoff. It confirms no handling handoff, so the recovery # marker still reads downtime and the re-arm owner delivers the close to -# main. The watcher singleton lock makes the session's next arm attach to -# that cycle instead of starting a second one; +# main. The host records that successor's arm before relinquishing it +# (detach_successor owns the persistence check and failure path). The +# session's next park without --restart requests a take-over of its cycle +# rather than an ordinary attach; bin/fm-watch-arm.sh's --take-over header owns the +# conditions under which that restores a single owner and the fallback; # - away (an away record exists): every close goes to the engine. # Every turn that starts attended meets that rule again at its start, so a # close accepted away whose turn starts attended (the captain returned in @@ -132,7 +135,12 @@ # left running (recorded with identities, never by name), including the # engine descendants its turn recorded, removes that turn's files, and # releases the branch actor's leases; it releases them again after every -# engine turn. +# engine turn. It also reads the record of a successor a pass-through left for +# main: while that arm still runs under its recorded identity, the first cycle +# without --restart requests a take-over rather than an ordinary attach. +# Activation removes the +# record only once that identity is no longer alive, so a later host retries a +# take-over that left it running. # # STATE (all under state/, owned here): .supervision-host (this host's pid and # the processes it runs), .supervision-host-engine (the engine conversation: @@ -141,7 +149,8 @@ # report scope and the reports it recorded), .supervision-host-prompt and # .supervision-host-wake (the prompt and wake text of the current turn), # .supervision-host-mirror (the dialog-mirror feed while an attended wake is -# rendered), +# rendered), .supervision-host-left (the pid and identity of the successor arm a +# pass-through left running for main, until that arm is gone), # .supervision-host-health (the latch: errors, cooldown, and probe time, keyed # to the main session, engine, and model), and .supervision-host.log (a bounded # ledger of where every close went, with each engine turn's usage and @@ -155,10 +164,15 @@ # a new engine conversation after this many turns; every main session start # also opens a new one), FM_SUPERVISION_HOST_READY_TIMEOUT (25: how long a # successor cycle may take to verify), FM_SUPERVISION_HOST_POLL (1). +# Park duration uses Bash's process-relative SECONDS counter (including Bash +# 3.2), while durable timestamps still use epoch time. This is not a portable +# monotonic-clock guarantee. Arm exit probes use ordinary 0.5-second child +# sleeps within the unchanged POLL-cadence maintenance and boundary checks; +# close observation and a shell-only caught signal may wait that interval plus +# work/scheduling time. No stop-signal disposition or cleanup bound changes. # FM_TEST_SUPERVISION_HOST_CLOCK names a file holding the park's elapsed -# seconds, which the park and turn boundary checks read in place of the wall -# clock only when FM_TEST_SEAM=1; tests/lib.sh arms the marker for isolated -# suites. +# seconds, which the park and turn boundary checks read in place of SECONDS +# only when FM_TEST_SEAM=1; tests/lib.sh arms the marker for isolated suites. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -226,8 +240,10 @@ HOST_LOG="$STATE/.supervision-host.log" ENGINE_PID_FILE="$STATE/.supervision-host.engine-pid" HEALTH_FILE="$STATE/.supervision-host-health" MIRROR_FEED="$STATE/.supervision-host-mirror" +LEFT_RECORD="$STATE/.supervision-host-left" HOST_PID=$$ +HOST_STARTED_SECONDS=$SECONDS HOST_STARTED=$(date +%s) GEN="host-$HOST_PID-$HOST_STARTED" TURN_SEQ=0 @@ -245,7 +261,12 @@ HANDLE_RC=0 ENGINE_SUBSHELL= SUCCESSOR_PID= SUCCESSOR_OUT= +SUCCESSOR_WATCHER= +SUCCESSOR_GENERATION= ENGINE_RUNNING=0 +# The successor arm a predecessor's pass-through left for main, which the +# first cycle takes over. +LEFT_ARM= # The running turn's result and diagnostics files, removed by the cleanup when # the host is stopped mid-turn. TURN_RESULT= @@ -356,6 +377,18 @@ activate() { done rm -f "$STATE"/.supervision-host-arm.* "$STATE"/.supervision-host-descendants.* "$STATE"/.supervision-host-result.* \ "$STATE"/.supervision-host-errors.* "$STATE"/.supervision-host-readback.* "$TURN_FILE" "$MIRROR_FEED" 2>/dev/null || true + # The successor a pass-through left for main: the first cycle takes it over + # while it still answers to its recorded identity, and its record goes only + # once it does not. + if [ -f "$LEFT_RECORD" ]; then + pid='' identity='' + IFS="$(printf '\t')" read -r pid identity < "$LEFT_RECORD" || true + if fm_pid_alive "$pid" && [ -n "$identity" ] && [ "$(identity_of "$pid")" = "$identity" ]; then + LEFT_ARM=$pid + else + rm -f "$LEFT_RECORD" + fi + fi printf 'host\t%s\t%s\n' "$HOST_PID" "$(identity_of "$HOST_PID")" > "$HOST_RECORD" || return 1 release_branch_leases } @@ -442,22 +475,24 @@ start_arm() { # [--restart]; sets the started pi STARTED_ARM_OUT=$out } -park_elapsed() { +park_elapsed() { # Sets PARK_ELAPSED without a production clock/helper fork. if [ "${FM_TEST_SEAM:-}" = 1 ] && [ -n "${FM_TEST_SUPERVISION_HOST_CLOCK:-}" ]; then - numeric_or "$(cat "$FM_TEST_SUPERVISION_HOST_CLOCK" 2>/dev/null)" 0 + PARK_ELAPSED=$(numeric_or "$(cat "$FM_TEST_SUPERVISION_HOST_CLOCK" 2>/dev/null)" 0) return fi - printf '%s\n' $(( $(date +%s) - HOST_STARTED )) + PARK_ELAPSED=$((SECONDS - HOST_STARTED_SECONDS)) } boundary_reached() { - [ "$(park_elapsed)" -ge "$PARK_SECONDS" ] + park_elapsed + [ "$PARK_ELAPSED" -ge "$PARK_SECONDS" ] } # True when an engine turn started now could still be running at the turn # limit (the boundary unless the owner set a later one). turn_crosses_boundary() { - [ $(( $(park_elapsed) + TURN_TIMEOUT + ENGINE_GRACE )) -ge "$PARK_LIMIT" ] + park_elapsed + [ $((PARK_ELAPSED + TURN_TIMEOUT + ENGINE_GRACE)) -ge "$PARK_LIMIT" ] } # End the park at the boundary: stop the current and successor arms and this @@ -471,7 +506,8 @@ boundary_exit() { SUCCESSOR_PID= SUCCESSOR_OUT= "$SCRIPT_DIR/fm-watch-arm.sh" --stop >/dev/null 2>&1 || true - log_line "boundary after $(park_elapsed)s" + park_elapsed + log_line "boundary after ${PARK_ELAPSED}s" emit 'supervision-host: cycle boundary - the host ended its park at its bound; drain, acknowledge, and end the turn, and the next park starts on its own' exit 0 } @@ -496,12 +532,11 @@ await_close() { refresh_process "$ARM_PID" [ "$READY_PENDING" -eq 0 ] || stream_ready_line boundary_reached && return 1 - # The arm's exit is probed at a tenth of a second between POLL-cadence - # checks: the close is read as soon as the arm dies instead of up to POLL - # seconds late, while refresh keeps its per-second cadence. - i=$((POLL * 10)) + # Probe the arm's exit twice a second between POLL-cadence checks, without + # changing the outer identity refresh, readiness, or boundary cadence. + i=$((POLL * 2)) while [ "$i" -gt 0 ] && fm_pid_alive "$ARM_PID"; do - sleep 0.1 + sleep 0.5 i=$((i - 1)) done done @@ -545,10 +580,17 @@ retire_successor() { # Hand the close to main: stop the successor cycle, print the close, why, and # any further "supervision-host:" lines, and exit. exit_to_main() { # [further lines] + local lines=${2:-} rc=0 retire_successor + if [ -n "$SUCCESSOR_GENERATION" ] \ + && ! fm_recovery_marker_publish "$STATE/.watcher-down" downtime >/dev/null 2>&1; then + log_line "to-main downtime-unrestored $1" + lines=${lines:+$lines$'\n'}"supervision-host: watcher downtime could not be restored for the main hand-back" + rc=1 + fi log_line "to-main $1" - emit "supervision-host: $1" "${2:-}" - exit 0 + emit "supervision-host: $1" "$lines" + exit "$rc" } # The outcome store (bin/fm-branch-outcome.sh) owns and validates these rows. @@ -629,13 +671,27 @@ start_successor() { # done } -# Drop the successor from this host's cleanup without stopping it. The shell -# signals background jobs when it exits, and this arm's handler would then -# stop the watcher, so disown it first. The capture file stays tracked so the -# EXIT trap unlinks it; the arm already holds that descriptor and keeps -# waiting on the watcher. +# Record the successor for the next host to take over, then drop it from this +# host's cleanup without stopping it. A successor whose record does not read +# back as a regular file holding exactly its pid and identity stays tracked, +# so the cleanup stops it and main's next turn end arms a fresh cycle; that +# returns 1. The shell signals background jobs when it exits, and this arm's +# handler would then stop the watcher, so disown it first. The capture file +# stays tracked so the EXIT trap unlinks it; the arm already holds that +# descriptor and keeps waiting on the watcher. detach_successor() { + local identity tmp= [ -n "${SUCCESSOR_PID:-}" ] || return 0 + identity=$(identity_of "$SUCCESSOR_PID") + if [ -z "$identity" ] || ! tmp=$(mktemp "$LEFT_RECORD.tmp.XXXXXX" 2>/dev/null) \ + || ! printf '%s\t%s\n' "$SUCCESSOR_PID" "$identity" > "$tmp" 2>/dev/null \ + || ! mv -f "$tmp" "$LEFT_RECORD" 2>/dev/null \ + || [ -L "$LEFT_RECORD" ] || [ ! -f "$LEFT_RECORD" ] \ + || [ "$(cat "$LEFT_RECORD" 2>/dev/null)" != "$SUCCESSOR_PID"$'\t'"$identity" ]; then + [ -z "$tmp" ] || rm -f "$tmp" "$LEFT_RECORD/${tmp##*/}" 2>/dev/null || true + log_line "pass-through successor-unrecorded $(printf '%s\n' "$REASON" | head -n 1)" + return 1 + fi disown "$SUCCESSOR_PID" 2>/dev/null || true forget_process "$SUCCESSOR_PID" SUCCESSOR_PID= @@ -1000,6 +1056,9 @@ log_line "start gen=$GEN primary=$PRIMARY" # The first cycle. if [ "$FIRST_ARM_RESTART" -eq 1 ]; then start_arm "$OWNER_PREDECESSOR" --restart +elif [ -n "$LEFT_ARM" ]; then + log_line "take-over arm=$LEFT_ARM" + start_arm "$OWNER_PREDECESSOR" --take-over "$LEFT_ARM" else start_arm "$OWNER_PREDECESSOR" fi || { echo "watcher: FAILED - the supervision host could not start a watcher cycle"; exit 1; } diff --git a/bin/fm-task-inbox-lib.sh b/bin/fm-task-inbox-lib.sh index 27c3aeda623..257da1dd54d 100644 --- a/bin/fm-task-inbox-lib.sh +++ b/bin/fm-task-inbox-lib.sh @@ -30,11 +30,14 @@ # .inbox/.ring-state watcher re-ring ladder: "\t\t" # .inbox/.escalated oldest-message name already surfaced as stale, # so later polls suppress another escalation +# .inbox/.retry-ring name of a fire-and-forget record still owed its +# one retry ring (fm_task_inbox_mark_retry) # # Record format (fm_task_inbox_write / fm_task_inbox_body): # schema=fm-task-inbox.v1 # at= # delivery=fire-and-forget present only when the re-ring ladder must ignore it +# (it still gets one retry ring; see below) # -- # @@ -59,7 +62,20 @@ # crash or marker failure may produce a rare duplicate rather than silently lose # a wake. # -# Inbox paths containing bytes outside printable ASCII are unsupported. The +# Retry ring (fm_task_inbox_mark_retry): only while config/wait-no-turns is +# present. A fire-and-forget record never enters the ladder, but when +# fm-send's ring at enqueue did not land +# (fm_task_inbox_ring returned 1 or 2) it marks the record, and one grace later +# the due action is `retry`: once the worker has no open decision of its own, +# the watcher rings once more and spends the mark +# whatever the result, so the record never rings a third time and never +# escalates. A waiting worker does not poll its inbox (bin/fm-brief.sh), so +# without this retry the record could sit unread until a checkpoint. A pending ordinary record's +# ladder rings the same inbox, so the retry waits behind it, and an +# acknowledged record drops its mark. The remote steer leg has no watcher +# ladder and owes no retry. +# +# Inbox names containing bytes outside printable ASCII are unsupported. The # doorbell refuses them rather than sending terminal control bytes to a pane. # # fm_task_inbox_ring requires bin/fm-backend.sh's dispatch (sourced below); the @@ -252,22 +268,30 @@ fm_task_inbox_body() { # } # The constant self-describing doorbell line for the inbox containing a record. -# Self-describing on purpose: a worker whose brief predates the inbox contract -# still receives the complete instruction in the line itself. The leading `: ` -# is the POSIX shell no-op, so the same line typed into a pane whose agent has -# exited (a bare shell) runs nothing; see the dead-pane note in the header. -# A non-printable path fails without output so terminal controls never reach -# the pane's line discipline. +# It names the inbox by the literal "$FM_TASK_INBOX", which bin/fm-spawn.sh +# exports into every launch as the inbox's absolute path, so the worker can +# resolve it from its own environment even after losing its brief context. +# The short `.inbox` name follows as the fallback for a worker launched +# before that export, whose brief carries the full path (bin/fm-dod-lib.sh +# role contract, bin/fm-brief.sh inbox section). No absolute path is printed, +# so the line's length never grows with the home's depth: a long line wraps +# past what a harness composer read can prove, and a Herdr submit then reports +# it did not reach the pane on every re-ring. The leading `: ` is the POSIX +# shell no-op, so the same line typed into a pane whose agent has exited (a +# bare shell) runs nothing; see the dead-pane note in the header. A +# non-printable inbox name fails without output so terminal controls never +# reach the pane's line discipline. fm_task_inbox_doorbell_line() { # - local dir=${1%/*} abs quoted LC_ALL=C + local dir=${1%/*} abs name quoted LC_ALL=C abs=$(cd "$dir" 2>/dev/null && pwd) || abs=$dir abs=${abs%/handled} - case "$abs" in - *[![:print:]]*) return 1 ;; + name=${abs##*/} + case "$name" in + ''|*[![:print:]]*) return 1 ;; esac - quoted=$(printf '%s' "$abs" | sed "s/'/'\\\\''/g") - printf ": Firstmate instruction waiting: list '%s'/*.msg and, in numeric order, read and act on each, then mv each handled file to '%s'/handled/." \ - "$quoted" "$quoted" + quoted=$(printf '%s' "$name" | sed "s/'/'\\\\''/g") + printf ": Firstmate instruction waiting: list \"\$FM_TASK_INBOX\"/*.msg in your '%s' steering inbox, read and act on each in numeric order, then mv each into its handled/." \ + "$quoted" } # Ring the doorbell, best-effort: one endpoint-liveness pre-check, one advisory @@ -361,11 +385,30 @@ fm_task_inbox_oldest_unhandled() { # printf '%s' "$best" } +# Owe a fire-and-forget record its one retry ring (see the header). A newer +# mark replaces an older one: a ring names the whole inbox, not one record. +fm_task_inbox_mark_retry() { # + local dir + dir=$(fm_task_inbox_dir "$1" "$2") + { printf '%s\n' "${3##*/}" > "$dir/.retry-ring"; } 2>/dev/null +} + +# Spend the retry mark after its ring, only while it still names that record: +# a newer mark written meanwhile is owed its own retry and survives. Fails only +# when the processed record's mark stays behind. +fm_task_inbox_clear_retry() { # + local dir + dir=$(fm_task_inbox_dir "$1" "$2") + [ "$(cat "$dir/.retry-ring" 2>/dev/null)" = "${3##*/}" ] || return 0 + rm -f "$dir/.retry-ring" 2>/dev/null +} + # The re-ring ladder decision for one task. Prints exactly one of: # quiet nothing due (healthy, within grace or spacing, # or already escalated for the current oldest) # ring one doorbell re-ring is due # escalate attempt budget spent; surface as stale +# retry a fire-and-forget record's one retry ring is due # An empty inbox also resets the ladder bookkeeping so the next message starts # a fresh ladder. fm_task_inbox_due_action() { # @@ -373,6 +416,17 @@ fm_task_inbox_due_action() { # dir=$(fm_task_inbox_dir "$1" "$2") if ! oldest=$(fm_task_inbox_oldest_unhandled "$1" "$2"); then rm -f "$dir/.ring-state" "$dir/.escalated" 2>/dev/null || true + # The one retry ring exists only while config/wait-no-turns is present. + # Absent, a mark is left untouched and the inbox stays quiet, as before. + if [ -e "${FM_CONFIG_OVERRIDE:-${FM_HOME:-}/config}/wait-no-turns" ]; then + base=$(cat "$dir/.retry-ring" 2>/dev/null || true) + if ! fm_task_inbox_seq_of "$base" >/dev/null || [ ! -f "$dir/$base" ]; then + rm -f "$dir/.retry-ring" 2>/dev/null || true + elif [ "$(fm_path_age "$dir/.retry-ring")" -ge "$(fm_task_inbox_grace_secs)" ]; then + printf 'retry %s' "$dir/$base" + return 0 + fi + fi printf 'quiet' return 0 fi diff --git a/bin/fm-tasks-axi.sh b/bin/fm-tasks-axi.sh index b773014a115..7f0dc1822c9 100755 --- a/bin/fm-tasks-axi.sh +++ b/bin/fm-tasks-axi.sh @@ -14,6 +14,10 @@ # stores it verbatim as a link, which lifecycle transitions record relative to # that same root. # +# `show` (including `view`) and `list` decode stored captain-hold reasons +# through bin/fm-hold-reason-lib.sh, which owns the field-only decoding contract. +# Decoded reasons use quoted strings so embedded line breaks remain intact. +# # Why it exists: a bare `tasks-axi` resolves the tracked `.tasks.toml` paths # against its working directory, so from the code root it forks the queue # whenever the home lives elsewhere; docs/configuration.md ("Backlog backend") @@ -46,7 +50,8 @@ # - a markdown `/backlog.md` that is itself a symlink, because the # first write would replace the link with a private copy, exactly the fork # this command exists to prevent. Lifecycle transitions refuse the same file. -# Otherwise the exit status is tasks-axi's own. +# Otherwise the exit status is tasks-axi's own, unless decoding a read fails; +# in that case the decoder's nonzero status is returned. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -57,6 +62,8 @@ DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" # shellcheck source=bin/fm-backlog-transition-lib.sh disable=SC1091 . "$SCRIPT_DIR/fm-backlog-transition-lib.sh" +# shellcheck source=bin/fm-hold-reason-lib.sh disable=SC1091 +. "$SCRIPT_DIR/fm-hold-reason-lib.sh" usage() { awk ' @@ -137,4 +144,11 @@ else fi cd "$FM_BACKLOG_AXI_ROOT" || fail "cannot enter the backlog root $FM_BACKLOG_AXI_ROOT" +case "${1:-}" in + show|view|list) + set -o pipefail + tasks-axi ${ARGS[@]+"${ARGS[@]}"} | fm_hold_reason_decode_stream + exit $? + ;; +esac exec tasks-axi ${ARGS[@]+"${ARGS[@]}"} diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index e0e3e115ae3..1c2ee5b3bb5 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -102,7 +102,10 @@ # cleanup step, teardown verifies record exclusivity: no OTHER task record in # this home or any locally registered Firstmate home may name the same live path # in its worktree= or home=. One live path with two task records is the reuse -# collision itself, whichever record is stale. +# collision itself, whichever record is stale. The one exception is a slot whose +# owner claim (below) names another task: this teardown is then records-only and +# touches nothing under the slot, so the scan is skipped rather than stranding +# the stale record and, with it, the claimant's own teardown. # That scan alone cannot prove THIS record is the current owner, because the task # that took the slot next may leave no record it can reach - its own worker may # have exited and its record been cleaned up, or it may live in a home this @@ -2318,6 +2321,12 @@ require_exclusive_worktree_slot_record() { local record_meta=$1 record_id=$2 record_state=$3 worktree=$4 local slot conflict other_id field slot=$(fm_canonical_existing_dir "$worktree") || return 0 + # A slot whose owner claim names another task was reassigned, so this record's + # teardown is records-only and touches nothing under it; another record naming + # the slot is then no hazard, and refusing would strand this stale record and + # block the claimant's own teardown behind it. + fm_treehouse_slot_owner_state "$slot" "$record_id" + [ "$FM_TREEHOUSE_SLOT_OWNER" != other ] || return 0 fm_collect_local_firstmate_states "$record_state" "nothing was changed" || return 1 conflict=$(fm_task_record_conflicts_on_path "$slot" "$record_meta") || return 1 conflict=${conflict%%$'\n'*} @@ -2339,11 +2348,11 @@ require_exclusive_task_worktree_slot() { # Positive slot ownership, read from the claim the task that took the slot wrote # into the slot itself (bin/fm-wake-lib.sh owns the claim and its states). # -# The record scan above proves that no OTHER task record names this slot. It -# cannot prove that THIS record is not the stale one, because the task that took -# the slot next may leave no record this scan can reach: its own worker may have -# exited and its record been cleaned up, or it may belong to a home this machine -# does not register. The claim closes that gap from the other side - it names the +# For a slot this task still claims, or one with no claim, the record scan above +# proves that no OTHER task record names it. It cannot prove that THIS record is +# not the stale one, because the task that took the slot next may leave no record +# this scan can reach: its own worker may have exited and its record been cleaned +# up, or it may belong to a home this machine does not register. The claim closes that gap from the other side - it names the # task that actually took the slot, and it is written under the same project lock # that allocates it - so a claim naming another task is proof the slot was # reassigned after this record was written. diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index d74fdba61e2..889d1703031 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -315,6 +315,7 @@ family_for_basename() { ;; fm-approved-execution.test.sh|fm-daemon.test.sh|fm-guard-stale-banner.test.sh|fm-pi-watch-extension.test.sh|\ fm-session-lock-ancestry.test.sh|fm-cursor-primary.test.sh|\ + fm-parent-channel-scan-exclusion.test.sh|\ fm-supervision-events.test.sh|fm-turnend-guard.test.sh|fm-wake-daemon-lifecycle-e2e.test.sh|\ fm-wake-drain-unread-status.test.sh|\ fm-tool-update-check.test.sh|\ @@ -372,6 +373,7 @@ family_for_basename() { fm-harness-liveness-drift-live-e2e.test.sh|\ fm-devin-signals-live-e2e.test.sh|fm-muse-signals-live-e2e.test.sh|fm-devin-live-e2e.test.sh|fm-rovo-signals-live-e2e.test.sh|fm-agy-signals-live-e2e.test.sh|\ fm-launch-prompt-signals-live-e2e.test.sh|\ + fm-pi-seeded-home-trust-live-e2e.test.sh|\ fm-herdr-version-floor-live-e2e.test.sh|\ fm-herdr-pi-stale-registration-live-e2e.test.sh|\ fm-worker-account-live-e2e.test.sh|\ @@ -845,6 +847,7 @@ tests/fm-pi-codex-native.test.sh 75 tests/fm-pi-primary-live-e2e.test.sh 72 tests/fm-pi-role-agents-live-e2e.test.sh 82 tests/fm-pi-role-agents.test.sh 918 +tests/fm-pi-seeded-home-trust-live-e2e.test.sh 45 tests/fm-pi-watch-extension.test.sh 56515 tests/fm-pi-windows-shell-invocation.test.sh 5121 tests/fm-pr-check-security.test.sh 300675 @@ -1772,7 +1775,7 @@ families_for_changed_path() { bin/fm-lint.sh|bin/fm-lint-workflows.sh|bin/fm-install-shellcheck.sh|\ bin/fm-install-actionlint.sh|\ bin/fm-brief.sh|bin/fm-ensure-agents-md.sh|bin/fm-crew-state.sh|\ - bin/fm-captain-hold.sh|bin/fm-decision-hold.sh|bin/fm-supervision*|bin/fm-transition-lib.sh|\ + bin/fm-captain-hold.sh|bin/fm-hold-reason-lib.sh|bin/fm-decision-hold.sh|bin/fm-supervision*|bin/fm-transition-lib.sh|\ bin/fm-tmux-lib.sh|bin/fm-marker-lib.sh|bin/fm-operational-input.sh|bin/fm-tasks-axi-lib.sh|\ bin/fm-vendor-auth-probe.sh|\ bin/fm-primary-scope-lib.sh|bin/fm-project-mode.sh|bin/fm-forge-detect.sh|bin/fm-promote.sh|\ diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index b1468310c33..149da9ff551 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -971,9 +971,64 @@ _fm_recovery_marker_reopen_announced() { fm_lock_release "$FM_WAKE_QUEUE_LOCK" } +# The handover rule for a watcher stopped by bin/fm-watch-arm.sh --take-over +# (docs/watcher-continuity.md "Generation reuse" owns it). The snapshot reads +# the marker token and the queue's append sequence under both locks before the +# stop; handover-restore puts an acknowledged token back only while that +# sequence is unchanged and the marker reads the fresh pending downtime the +# stopped watcher's own close published. +FM_RECOVERY_HANDOVER_TOKEN= +FM_RECOVERY_HANDOVER_SEQ= +fm_recovery_marker_handover_snapshot() { # + local marker=$1 lock + FM_RECOVERY_HANDOVER_TOKEN= + FM_RECOVERY_HANDOVER_SEQ= + lock="${marker}.lock" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" || return 1 + if ! fm_lock_acquire_wait "$lock"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + fi + if fm_recovery_marker_read "$marker"; then + # shellcheck disable=SC2034 # Read by callers after this function returns. + FM_RECOVERY_HANDOVER_TOKEN=$FM_RECOVERY_MARKER_TOKEN + fi + # shellcheck disable=SC2034 # Read by callers after this function returns. + FM_RECOVERY_HANDOVER_SEQ=$(cat "$STATE/.wake-queue.seq" 2>/dev/null || true) + fm_lock_release "$lock" + fm_lock_release "$FM_WAKE_QUEUE_LOCK" +} + +_fm_recovery_marker_handover_restore() { + local marker=$1 token=$2 seq=$3 lock status=0 + case "$token" in acked:*) ;; *) return 0 ;; esac + lock="${marker}.lock" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" || return 1 + if ! fm_lock_acquire_wait "$lock"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + fi + if [ "$(cat "$STATE/.wake-queue.seq" 2>/dev/null || true)" = "$seq" ] \ + && fm_recovery_marker_read "$marker"; then + case "$FM_RECOVERY_MARKER_TOKEN" in + pending:downtime:*) + if [ "${FM_RECOVERY_MARKER_TOKEN##*:}" != "${token##*:}" ]; then + _fm_recovery_marker_restore_token_locked "$marker" "$token" || status=1 + fi + ;; + esac + fi + fm_lock_release "$lock" + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return "$status" +} + fm_recovery_transition() { local marker=$1 action=$2 target=${3:-} value=${4:-} bound=${5:-} case "$action" in + handover-restore) + _fm_recovery_marker_handover_restore "$marker" "$target" "$value" + ;; publish) _fm_recovery_marker_publish "$marker" "${target:-downtime}" "$bound" ;; @@ -1035,6 +1090,10 @@ fm_recovery_marker_reopen_announced() { fm_recovery_transition "$1" reopen-announced } +fm_recovery_marker_handover_restore() { # + fm_recovery_transition "$1" handover-restore "$2" "$3" +} + # fm_lock_reap_dead_link # Remove a link lock whose owner is dead without a nested mutex. Renaming the # dead owner directory to this process's tombstone elects exactly one reaper, diff --git a/bin/fm-watch-arm.sh b/bin/fm-watch-arm.sh index 0d79044f0bf..3a04cfb95b4 100755 --- a/bin/fm-watch-arm.sh +++ b/bin/fm-watch-arm.sh @@ -74,6 +74,17 @@ # bin/fm-watch.sh`: that pattern matches every firstmate home's watcher # (secondmate homes run the same script) and would kill siblings. # +# --take-over : own the cycle that arm owns, for an owner +# that left a successor cycle running through main's turn and now parks again +# (bin/fm-supervision-host.sh). Only when this home's healthy watcher is that +# arm's own child, it stops that watcher by its locked identity: a cycle that +# delivered a reason before the stop landed reports it exactly as an attached +# arm would, and otherwise this arm owns a fresh cycle as a plain arm does. +# Recovery restoration follows docs/watcher-continuity.md "Generation reuse"; +# an unconfirmed stop leaves downtime for the fresh cycle's recovery check. +# Any other watcher, or one that outlives the stop, +# is attached to exactly as a plain arm attaches. +# # --stop: the same home-scoped stop without re-arming, for an owner that ends # its own supervision cycle on purpose (the supervision host's park boundary, # bin/fm-supervision-host.sh). The stopped watcher publishes downtime exactly @@ -143,9 +154,10 @@ ARM_PID=${BASHPID:-$$} case "$CYCLE_LOG_MAX_BYTES" in ''|*[!0-9]*|0) CYCLE_LOG_MAX_BYTES=262144 ;; esac case "$CYCLE_LOG_KEEP_LINES" in ''|*[!0-9]*|0) CYCLE_LOG_KEEP_LINES=1000 ;; esac -# The lifecycle ledger is diagnostic evidence, not a supervision dependency. -# Writes are bounded and best-effort so an observability failure cannot stall an -# otherwise healthy watcher cycle. +# Lifecycle writes are bounded and best-effort so an observability failure +# cannot stall an otherwise healthy watcher cycle. Take-over also uses the +# owner's row as stop evidence; missing evidence takes the safe recovery path +# (docs/watcher-continuity.md "Generation reuse"). cycle_clean_field() { printf '%s' "$1" | tr '\t\r\n' ' ' | cut -c1-512 } @@ -243,9 +255,10 @@ cycle_log_append() { # A persistent adapter passes the arm pid that just closed. Once this new arm # verifies its watcher, update that predecessor's final record in place so the # one-record-per-cycle ledger captures the actual successor outcome without an -# extra synthetic lifecycle row. +# extra synthetic lifecycle row. A taking-over arm names itself instead, so its +# record of the cycle it took over names the cycle it started. cycle_mark_predecessor_successor() { - local successor=$1 predecessor=${FM_WATCH_PREDECESSOR_ARM_PID:-} i tmp + local successor=$1 predecessor=${2:-${FM_WATCH_PREDECESSOR_ARM_PID:-}} i tmp case "$predecessor" in ''|*[!0-9]*) return 0 ;; esac @@ -330,31 +343,35 @@ fail_unexplained_cycle() { return 1 } -# Close a cycle whose reason line this arm could not read against the bounded -# terminal-delivery ledger the watcher publishes before releasing its lock. -close_unobserved_cycle() { - local i reason clean_identity record_pid record_identity record_reason +# Read the reason the current cycle's watcher recorded in the bounded +# terminal-delivery ledger it publishes before releasing its lock. Sets +# DELIVERED_REASON; fails when no record matches the cycle's pid and identity. +DELIVERED_REASON= +cycle_delivered_reason() { + local i clean_identity record_pid record_identity record_reason + DELIVERED_REASON= clean_identity=$(printf '%s' "$cycle_watcher_identity" | tr '\t\r\n' ' ') i=0 while ! fm_lock_try_acquire "$WATCH_DELIVERY_LOCK"; do - [ "$i" -lt 20 ] || { - fail_unexplained_cycle - return 1 - } + [ "$i" -lt 20 ] || return 1 sleep 0.02 i=$((i + 1)) done - reason= if [ -f "$WATCH_DELIVERY_LOG" ]; then while IFS=$'\t' read -r record_pid record_identity record_reason; do if [ "$record_pid" = "$cycle_watcher_pid" ] && [ "$record_identity" = "$clean_identity" ]; then - reason=$record_reason + DELIVERED_REASON=$record_reason fi done < "$WATCH_DELIVERY_LOG" fi fm_lock_release "$WATCH_DELIVERY_LOCK" - if [ -n "$reason" ]; then - printf '%s\n' "$reason" + [ -n "$DELIVERED_REASON" ] +} + +# Close a cycle whose reason line this arm could not read against that ledger. +close_unobserved_cycle() { + if cycle_delivered_reason; then + printf '%s\n' "$DELIVERED_REASON" return 0 fi fail_unexplained_cycle @@ -467,10 +484,17 @@ handling_successor_generation() { mode=arm handling_generation= handling_watcher_pid= +take_over_arm_pid= case "${1:-}" in ''|arm|--arm) mode=arm ;; --restart) mode=restart ;; --stop) mode=stop ;; + --take-over) + mode=take-over + take_over_arm_pid=${2:-} + case "$take_over_arm_pid" in ''|*[!0-9]*) echo "watcher: invalid take-over arm pid" >&2; exit 2 ;; esac + [ "$#" -eq 2 ] || { echo "watcher: unexpected take-over arguments" >&2; exit 2; } + ;; --handling-delivered) mode=handling-delivered handling_generation=${2:-} @@ -480,7 +504,7 @@ case "${1:-}" in case "$handling_watcher_pid" in ''|*[!0-9]*) echo "watcher: invalid successor watcher pid" >&2; exit 2 ;; esac [ "$#" -eq 4 ] || { echo "watcher: unexpected handling delivery arguments" >&2; exit 2; } ;; - *) echo "usage: $(basename "$0") [--restart | --stop | --handling-delivered GENERATION --watcher-pid PID]" >&2; exit 2 ;; + *) echo "usage: $(basename "$0") [--restart | --stop | --take-over ARM_PID | --handling-delivered GENERATION --watcher-pid PID]" >&2; exit 2 ;; esac if [ "$mode" = handling-delivered ]; then @@ -494,7 +518,7 @@ if [ "$mode" = handling-delivered ]; then handling_status=0 fm_recovery_marker_begin_handling "$STATE/.watcher-down" "$handling_generation" || handling_status=$? if [ "$handling_status" -ne 0 ] \ - && [ -f "$FM_WAKE_QUEUE" ] && [ ! -L "$FM_WAKE_QUEUE" ] && [ ! -s "$FM_WAKE_QUEUE" ] \ + && { [ ! -e "$FM_WAKE_QUEUE" ] || { [ -f "$FM_WAKE_QUEUE" ] && [ ! -L "$FM_WAKE_QUEUE" ] && [ ! -s "$FM_WAKE_QUEUE" ]; }; } \ && fm_recovery_marker_snapshot "$STATE/.watcher-down"; then case "$FM_RECOVERY_MARKER_TOKEN" in "acked:handling:$handling_generation"|"acked:downtime:$handling_generation") handling_status=4 ;; @@ -544,6 +568,67 @@ if [ "$mode" = stop ]; then exit 0 fi +# Stop the watcher the named arm owns, by its locked identity, and wait for it +# to exit (header, --take-over). Returns 3 after printing the reason that cycle +# delivered before the stop landed, 0 once it stopped without delivering, and +# 1 when it was not stopped (its handover state was unreadable, or it outlived +# the stop), which leaves it to the plain attach below. +take_over_cycle() { # + local pid=$1 i owner_signal + cycle_begin "$pid" attached "$2" + fm_recovery_marker_handover_snapshot "$STATE/.watcher-down" || return 1 + if attached_holder_live "$pid"; then + kill -TERM "$pid" 2>/dev/null || true + fi + i=0 + while [ "$i" -lt 50 ] && fm_pid_alive "$pid"; do + sleep 0.1 + i=$((i + 1)) + done + if fm_pid_alive "$pid"; then + return 1 + fi + if cycle_delivered_reason; then + cycle_log_append unknown unknown taken-over-delivered-wake none + printf '%s\n' "$DELIVERED_REASON" + return 3 + fi + # Only the owner can wait on this watcher and distinguish our TERM from a + # self-exit that raced the stop. Give its post-wait ledger append a short bound. + i=0 + owner_signal= + while [ "$i" -lt 50 ]; do + owner_signal=$(awk -F '\t' -v arm="$take_over_arm_pid" -v watcher="$pid" ' + $1 == "arm_pid=" arm && $2 == "watcher_pid=" watcher { signal = $7 } + END { sub(/^signal=/, "", signal); print signal } + ' "$CYCLE_LOG" 2>/dev/null || true) + [ -z "$owner_signal" ] || break + sleep 0.02 + i=$((i + 1)) + done + if [ "$owner_signal" = TERM ]; then + fm_recovery_marker_handover_restore "$STATE/.watcher-down" \ + "$FM_RECOVERY_HANDOVER_TOKEN" "$FM_RECOVERY_HANDOVER_SEQ" || true + cycle_log_append unknown unknown taken-over none + else + cycle_log_append unknown unknown taken-over-unconfirmed-stop none + fi + return 0 +} + +TAKEN_OVER=0 +if [ "$mode" = take-over ]; then + mode=arm + if healthy_watcher \ + && [ "$(ps -o ppid= -p "$HEALTHY_PID" 2>/dev/null | tr -d ' ')" = "$take_over_arm_pid" ]; then + take_over_cycle "$HEALTHY_PID" "$HEALTHY_IDENTITY" + case $? in + 0) TAKEN_OVER=1 ;; + 3) exit 0 ;; + esac + fi +fi + # If a genuinely live+fresh watcher already holds the lock, do not start a second # one - attach to that cycle and wait until it ends so the harness notify fires # then, not as an immediate empty wake. (--restart skips this: it just stopped @@ -692,6 +777,7 @@ while :; do exit 1 fi cycle_mark_predecessor_successor "started:$child" + [ "$TAKEN_OVER" -eq 0 ] || cycle_mark_predecessor_successor "started:$child" "$ARM_PID" if [ -n "$handling_generation" ]; then echo "watcher: started pid=$child (beacon fresh) recovery-generation=$handling_generation" else diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 78410a95c26..10c72ed54f7 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -538,7 +538,10 @@ inbox_steer_escalate_unavailable() { # # stale path instead of silently re-ringing forever; acknowledgement or teardown # still makes the race quiet. The attempt is data-plane typing or a # composer-protected skip, never a wake, so normal retries keep the watcher -# blocking. Runs for secondmates +# blocking. A fire-and-forget record's one retry ring follows the same busy +# wait, also waits while the worker has an open decision or blocker of its own +# (status_own_open_decisions), and never escalates: a dead pane just spends it. +# Runs for secondmates # too: their pane-staleness exemption is about quiet panes being healthy, # while an unacknowledged instruction past the ladder is a stuck steer. inbox_steer_check() { # @@ -546,6 +549,9 @@ inbox_steer_check() { # action=$(fm_task_inbox_due_action "$STATE" "$task") || return 0 verb=${action%% *} [ "$verb" != quiet ] || return 0 + if [ "$verb" = retry ] && [ -n "$(status_own_open_decisions "$STATE/$task.status")" ]; then + return 0 + fi rec=${action#* } count= case "$verb" in @@ -558,7 +564,11 @@ inbox_steer_check() { # agent_state=$(fm_backend_agent_state "$backend" "$w" 2>/dev/null || true) case "$agent_state" in dead|missing) - inbox_steer_escalate_unavailable "$w" "$task" "$rec" + if [ "$verb" = retry ]; then + fm_task_inbox_clear_retry "$STATE" "$task" "$rec" || true + else + inbox_steer_escalate_unavailable "$w" "$task" "$rec" + fi return 0 ;; esac @@ -587,6 +597,16 @@ inbox_steer_check() { # fi triage_log "steer-inbox delivery attempt: $task ${rec##*/} result=$ring_rc" ;; + retry) + ring_rc=0 + fm_task_inbox_ring "$backend" "$w" "$rec" "$(window_label "$w")" || ring_rc=$? + if ! fm_task_inbox_clear_retry "$STATE" "$task" "$rec" && [ -f "$rec" ]; then + reason="stale: $w (steering-inbox retry mark unremovable: ${rec%/*}/.retry-ring cannot be removed, so $rec would ring on every poll - inspect the inbox directory)" + fm_wake_append stale "$w" "$reason" || exit 1 + wake "$reason" + fi + triage_log "steer-inbox retry ring: $task ${rec##*/} result=$ring_rc" + ;; escalate) reason="stale: $w (unread firstmate instruction: $rec still unhandled after $count doorbell delivery attempts with an idle pane; inspect the worker)" if [ ! -d "${rec%/*}" ] || [ ! -f "$rec" ]; then @@ -2112,8 +2132,13 @@ age_of() { # seconds since file mtime; "due immediately" if missing # The caller records reported state only after surfacing or intentional absorption, # and commits a status classification position only after a successful span read. scan_signals() { - local f sig sf + local f sig sf exclude + # A remote mate's own parent channel is not a self-home task status log; the + # home-shape-aware exclusion and its precedent live in + # status_scan_parent_channel_exclude (fm-classify-lib.sh). + exclude=$(status_scan_parent_channel_exclude "$STATE") for f in "$STATE"/*.status "$STATE"/*.turn-ended; do + [ "$f" = "$exclude" ] && continue if [ ! -e "$f" ]; then case "$f" in *.status) [ -L "$f" ] || continue ;; *) continue ;; esac fi @@ -2382,10 +2407,14 @@ EOF # is absorbed; it surfaces only an event the per-wake path absorbed by mistake - # the fail-safe backstop. heartbeat_scan_finds_actionable() { - local f task record rest endpoint ident rc found=1 sig marker + local f task record rest endpoint ident rc found=1 sig marker exclude + # Same self-home exclusion as scan_signals: a remote mate's parent channel + # must not come back through the heartbeat fail-safe backstop. + exclude=$(status_scan_parent_channel_exclude "$STATE") FM_HEARTBEAT_SURFACE_ENDPOINTS='' for f in "$STATE"/*.status; do [ -e "$f" ] || [ -L "$f" ] || continue + [ "$f" = "$exclude" ] && continue task=$(basename "$f"); task="${task%.status}" # `jev` here too: the backstop re-classifies spans past the hb-surfaced # marker to catch the same escalation-only misses the signal path consults diff --git a/docs/architecture.md b/docs/architecture.md index 59838fcb577..860da363eba 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -447,7 +447,7 @@ A later merged poll consumes only that matching persisted value; with no match i Teardown is fail-closed for ship worktrees: dirty worktrees refuse, and committed work must be landed before the worktree is returned. A pool worktree is only taken or returned after the slot-ownership proof passes: a task record naming the assigned path refuses a fresh ship or scout before it claims the slot and refuses teardown before it returns one, a supported live endpoint refuses teardown, and no discard authority relaxes either. [`bin/fm-wake-lib.sh`](../bin/fm-wake-lib.sh)'s header owns the shared scan over every local home's task records and the canonical spelling it compares, while [`bin/fm-spawn.sh`](../bin/fm-spawn.sh)'s header owns the spawn-side refusal. -A slot's own owner claim, written by the spawn that takes it under the allocation lock and owned by [`bin/fm-wake-lib.sh`](../bin/fm-wake-lib.sh), covers a slot reassigned to a task that left no record the scan could reach: a claim naming a different task releases nothing - teardown warns, names the claimant, and finishes only the task's own cleanup - because Treehouse's own live process lease cannot answer ownership once the worker's exit releases it. +A slot's own owner claim, written by the spawn that takes it under the allocation lock and owned by [`bin/fm-wake-lib.sh`](../bin/fm-wake-lib.sh), covers a slot reassigned to another task, including one that left no record the scan could reach: a claim naming a different task releases nothing, even alongside that task's contradictory record - teardown warns, names the claimant, and finishes only the task's own cleanup - because Treehouse's own live process lease cannot answer ownership once the worker's exit releases it. A fleet clone refuses a fresh ship or scout before allocation when its plain local-path origin has no `origin` remote and already owns a linked Treehouse pool slot, naming that pool-owning repository to pass instead; [`bin/fm-wake-lib.sh`](../bin/fm-wake-lib.sh)'s `fm_treehouse_pool_origin_root` owns the detection and [`tests/fm-spawn-truth-path.test.sh`](../tests/fm-spawn-truth-path.test.sh) pins it. Allocation and return serialize on one project lock per machine-local Firstmate tree: every home reachable through local parent links shares that lock, and a home seeded from another machine anchors its own, because a lock taken on this filesystem is neither held nor observable across that boundary. Before the worktree is returned, teardown concludes the task's own no-mistakes run when it is parked at a gate, including a run whose head the task copy cannot resolve - the shared runs-ledger continuation proof is the only recognition for that case, so cleanup never orphans a parked run the pipeline advanced past the submitted head. diff --git a/docs/captain-hold-lifecycle.md b/docs/captain-hold-lifecycle.md index 5942ff6ade3..bf745c16aab 100644 --- a/docs/captain-hold-lifecycle.md +++ b/docs/captain-hold-lifecycle.md @@ -51,8 +51,9 @@ It works in this order: 1. It uses an existing task, or creates one when nothing exists to hold. 2. It records the task's UTC hold-set timestamp as the leading line of the task body. -3. It invokes the underlying tasks-axi hold operation. -4. It verifies both records. +3. When `--origin` is supplied, it records the origin on the task, replacing any previous association. +4. It invokes the underlying tasks-axi hold operation. +5. It verifies the hold and timestamp. Publishing the stamp first ensures a snapshot cannot observe a newly captain-held task without the timestamp that defines its age. @@ -62,6 +63,10 @@ Repeat and edge cases: - Re-holding released work starts a new timestamped lifecycle. - A closed task is refused rather than reopened. - `--until` stores the captain's own deferral date through tasks-axi's date gate. +- Before the backend hold runs, `--origin` records the origin the call is held for on its own `Captain hold origin:` body line, which `complete` and `verify` check using backend identities rather than alias spellings. + If that write fails, the backend hold is not attempted. +- The reason may contain parentheses, semicolons, quotes, and line breaks. + [`bin/fm-hold-reason-lib.sh`](../bin/fm-hold-reason-lib.sh) owns the storage encoding and compatibility rules; [`bin/fm-tasks-axi.sh --help`](../bin/fm-tasks-axi.sh) owns the public read commands and output contract. ### Answering a call (`answer`) @@ -104,6 +109,10 @@ A post-teardown visual review can complete against the surviving report and dura `complete` accepts `--none` as an explicit semantic inventory result. `--none` is refused while the origin still has a lifecycle-open keyed status decision. Before recording completion, `complete` verifies every listed task against tasks-axi. +The origin is never its own inventory entry, so a hold that failed cannot be vouched for by the origin row. +For a historical inventory that names its own origin, hold a separate captain task with `--origin`, replace only the invalid entry in the final `decision_keys=` line of the origin metadata with that task id while preserving all other entries, and re-run `complete`. +An entry whose recorded origin differs from the one being completed is refused. +An entry with no recorded origin, such as a hold made before origins were recorded or without `--origin`, is accepted on the durability check alone and named in the output. With a non-empty inventory, `complete` appends a `captain-held [key=]` transfer event for every still-open keyed status decision. The event names the reviewed inventory. @@ -115,7 +124,7 @@ Scout teardown calls the read-only `verify` subcommand after checking for the re `verify` checks three things: - The recorded attestation exists. -- Every recorded inventory entry is still durable: actively captain-held, or carrying a recorded answer. +- Every recorded inventory entry still passes the [completion inventory checks](#recording-a-reviewed-inventory-complete). - No keyed status decision opened after the last `complete`. A keyed status decision opened after the last `complete` makes `verify` fail, and re-running `complete` is the repair. @@ -141,6 +150,7 @@ After cleanup, and still under the task's own lock, teardown does three things: - It records one `Deliverable of the finished work: ...` line at the end of the task body. - It copies a supported pull request or canonical `data//report.md` into the row's structured artifact fields. + A Gerrit change URL is not a pull request tasks-axi accepts, so it appears only in the deliverable line. - It runs `tasks-axi reopen`. The row returns to Queued with its hold intact. @@ -153,6 +163,7 @@ That record carries the retention intent as a `mode=retain` line. An interrupted cleanup therefore replays the retention at the next session start through the same record, validator, and lock as an ordinary close, and never closes the row. If the captain answers before replay, `answer` validates that record and copies any supported retained pull request or report into the row before closing it. +A retained Gerrit change URL is instead recorded as a `Gerrit change ` note on that close. Replay then retires the record. ### Known retained-delivery gaps @@ -478,7 +489,7 @@ It then finishes any still-recorded dependency-edge cleanup without rewriting th ## Verification record -The focused end-to-end regression suite is `tests/fm-captain-hold-lifecycle.test.sh`, using only synthetic `sample` identities and decision text. +The focused end-to-end regression suite is `tests/fm-captain-hold-lifecycle.test.sh`, using only synthetic identities and decision text. It proves the behaviors below. The suite does not test the accepted merge-to-cleanup re-hold window or asynchronous queued-forge landing because those events occur after the locally serialized merge command has returned. @@ -527,7 +538,8 @@ The suite does not test the accepted merge-to-cleanup re-hold window or asynchro ### Legacy paths -- Every legacy path works: composed identities through the shim, pre-collapse `decision_keys=` metadata, routed-resolution replay, and a concrete-origin binding. +- Composed identities through the shim, valid pre-collapse `decision_keys=` inventories, routed-resolution replay, and a concrete-origin binding remain supported. + Historical self-inventories require the [documented repair](#recording-a-reviewed-inventory-complete). ### Task-body read-back cases diff --git a/docs/configuration.md b/docs/configuration.md index fa78ace6317..a14fd79690a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -625,6 +625,13 @@ See [`trace-context.md`](trace-context.md) for carrier semantics, supported rout See [`fleet-ledger.md`](fleet-ledger.md) for the opt-in setup, record contract, and limits. +## Waiting worker spends no turns (config/wait-no-turns) + +The optional local, gitignored `config/wait-no-turns` presence flag opts this home into keeping a waiting worker from spending turns until it is answered. +With it present, ship and scout briefs gain the `# Waiting` section and the foreground no-mistakes drive text, every brief's inbox section keeps the natural-checkpoint check and adds that a waiting worker does not poll or list its inbox because a waiting instruction rings, a pending-reply recovery waits while that mate has its own open decision or blocker, and a fire-and-forget steer whose doorbell did not land gets one later ring. +With the file absent, generated briefs omit the waiting section and the no-poll inbox line, the drive text backgrounds the call, recovery sends during an open decision, and a fire-and-forget steer is not owed a retry ring. +The flag is a home-local preference and is not inherited by secondmate homes. + ## Turn-end pane-churn absorb (config/turnend-churn-absorb) The optional local, gitignored `config/turnend-churn-absorb` presence flag opts this home into a default-off third form of positive work evidence in watcher triage. @@ -871,6 +878,7 @@ The Kimi installer requires an existing regular non-symlink `~/.kimi-code/config Its `remove` action excises only the marker-delimited Firstmate region and removes Firstmate's hook files. For Pi and pi-signed secondmate launches, `fm-spawn.sh` starts the selected executable with `-e` pointed at the secondmate home's own tracked `.pi/extensions/fm-primary-pi-watch.ts` and `.pi/extensions/fm-primary-turnend-guard.ts`, both already present from the secondmate home's git worktree. +Pi-family secondmates can start unattended in Firstmate-seeded homes without accepting project trust manually; [`fm-spawn.sh --help`](../bin/fm-spawn.sh) owns the capability requirement, session-only approval scope, and older-version fallback, with [regression evidence](verification/runtime-backends.md#pi-seeded-secondmate-project-trust). For omp secondmate launches, `fm-spawn.sh` passes no `-e` at all: omp auto-discovers the home's tracked `.omp/extensions/` with no trust gate, and naming a discovered file with `-e` as well loads it twice; every omp launch instead carries the tracked `.omp/fm-worker-overlay.yml` posture overlay through `--config`, which [`fm-spawn.sh --help`](../bin/fm-spawn.sh) owns. @@ -1085,7 +1093,7 @@ The optional local, gitignored `config/keep-ai-trailers` presence flag opts this With the flag absent, every Claude launch's inline `--settings` JSON carries `"attribution":{"commit":"","pr":"","sessionUrl":false}`, every Devin worker config sets `"attribution": false`, and every fleet launch receives a pane-scoped `GIT_CONFIG` `core.hooksPath` pointing at `state/.git-hooks`, where git's `commit-msg` hook strips known AI trailers even when a runtime injects them after the typed message. When the flag is present, Claude launches omit those attribution-off settings, Devin worker configs keep the user config's `attribution` setting (Devin's default is on), and fleet launches do not install or select the strip hooks, so Git uses the repository's configured hooks directly. `bin/fm-git-strip-ai-trailers.sh` owns the identities, the install, and chaining the hooks of whichever repository git is running in, including when `git -c core.hooksPath` supplies the pane's hook override, so a project hook such as husky still runs when stripping is enabled. -If the wrapper cannot resolve that repository's hooks directory, the git operation fails rather than silently skipping a project hook such as a pre-push guard. +A repository whose config sets `core.hooksPath` to the empty string runs no project hook, as in plain git; if the wrapper otherwise cannot resolve that repository's hooks directory, the git operation fails rather than silently skipping a project hook such as a pre-push guard. When stripping is enabled, the hooks directory is read-only, so a hook manager run inside a fleet pane (lefthook's npm postinstall, `pre-commit install`) fails instead of displacing the strip; install a project's hooks from outside the pane, where the wrappers chain them. The flag is a home-wide attribution choice, so it is inherited into secondmate homes under the [`secondmate-provisioning`](../.agents/skills/secondmate-provisioning/SKILL.md) inherited-local-material contract and a secondmate's own workers keep AI trailers too. Per-machine Cursor `cli-config.json` attribution-off is not this contract: it does not travel with Firstmate, defaults back to on when unset, and only feeds the CLI's request to the server, so it suppresses the trailer rather than preventing it. @@ -2650,6 +2658,7 @@ FM_BACKEND= # optional runtime backend override for new spawns; tmux FM_TRACE_CONTEXT= # optional trace-context override; see "Trace context propagation" FM_WIKIS_ROOT= # optional wikis root override; see "Wiki context in briefs" FM_TASK_ID= # internal task-worker marker fm-spawn.sh exports into ship and scout panes, never set by hand; bin/fm-test-run.sh refuses to execute in the repository primary checkout while it is set +FM_TASK_INBOX= # internal: absolute path of the task's steering inbox (state/.inbox) that fm-spawn.sh exports into every ship, scout, and secondmate launch, never set by hand; the steering doorbell names "$FM_TASK_INBOX" HERDR_SESSION=default # herdr-only: named session for normal backend ops; not enough for destructive cleanup (docs/herdr-backend.md) FM_BACKEND_HERDR_SUBMIT_POLLS=6 # herdr-only: agent-state samples spread across each Enter attempt's budget when confirming a submit (docs/herdr-backend.md "Current transport behavior") FM_BACKEND_HERDR_SUBMIT_MIN_SLEEP=0.6 # herdr-only: minimum per-Enter confirmation budget before polling agent-state after an idle baseline @@ -2758,6 +2767,7 @@ FM_WATCH_REARM_RETRY_MAX_MS=4000 # Pi/OpenCode adapter cap for exponential con FM_WATCH_REARM_RETRY_LIMIT=5 # Pi/OpenCode adapter launch-failure retries before surfacing restoration failure FM_WATCH_CYCLE_LOG_MAX_BYTES=262144 # size cap for the arm-owned watcher lifecycle ledger FM_WATCH_CYCLE_LOG_KEEP_LINES=1000 # newest complete lifecycle rows considered when the ledger is capped +FM_WATCH_EXTENSION_LOG_KEEP_LINES=0 # opt-in Pi extension diagnostic log (state/.watch-extension.log); unset, empty, non-numeric, zero, or negative disables logging, a positive value keeps that many newest rows; logging never changes supervision behavior FM_WATCHER_STALE_GRACE=300 # defaults to FM_GUARD_GRACE if set, else the poll-derived grace (docs/turnend-guard.md "Guard grace and the poll cadence"); seconds before a fresh arm refuses a live holder's stale beacon (attached arms: FM_WATCHER_STALL_BOUND) FM_WATCHER_STALL_BOUND= # live-holder stall bound; default and arm/re-arm behavior: docs/turnend-guard.md "Guard grace and the poll cadence" FM_SIGNAL_GRACE=30 # seconds to coalesce nearby status and turn-end signals into one wake diff --git a/docs/fm-test-portable-shards.md b/docs/fm-test-portable-shards.md index 8dbb29b67f2..89f306ee133 100644 --- a/docs/fm-test-portable-shards.md +++ b/docs/fm-test-portable-shards.md @@ -156,11 +156,11 @@ Portable shards, each portable serial shard, and the Herdr lane upload runner-ge ## Lint partitions and end-to-end latency -`bin/fm-lint.sh` owns two canonical CI partitions, each running full source-aware ShellCheck analysis, workflow validation, and backend-purity checks. -CI requires its per-root bounds, so an unenforceable deadline or address-space limit refuses lint rather than running uncapped; the script header owns the envelope and per-root execution contract. -Its `--list-files` interface exposes partition membership; `tests/fm-lint.test.sh` verifies complete/disjoint executed roots and unchanged analysis flags. +`bin/fm-lint.sh` owns two canonical CI partitions, each attempting full source-aware ShellCheck analysis and running workflow validation and backend-purity checks. +CI requires its per-root bounds, so an unenforceable deadline or address-space limit refuses lint rather than running uncapped; the script header owns the envelope, per-root execution contract, and memory fallback. +Its `--list-files` interface exposes partition membership; `tests/fm-lint.test.sh` verifies complete/disjoint executed roots, initial analysis flags, and fallback reporting. The workflow uploads each partition's quiet telemetry plus its per-root lifecycle sidecar to distinguish analysis cost, memory use, and host contention. -No fast mode, path skips, reduced checks, or paid runner provisioning is part of this layout. +No fast mode, path skips, or paid runner provisioning is part of this layout. The longer-term performance objective remains a complete green run under fifteen minutes including start delay, but the current watch-triage floor alone exceeds that objective. The immediate packing target is the runner's modeled script budget, not a claim that more shards alone can make an indivisible script faster. diff --git a/docs/pi-supervision-branch.md b/docs/pi-supervision-branch.md index 59328039ec3..c1f016d73ce 100644 --- a/docs/pi-supervision-branch.md +++ b/docs/pi-supervision-branch.md @@ -430,6 +430,14 @@ Nothing else advances that marker. An unrelated reply, an empty reply, or a reply that paraphrases the outcome leaves the sequence unprocessed. The extension presents the current unprocessed sequence set again at the next main run boundary and at every session start. +The first presentation of a sequence set is an ordinary turn whose response stays visible, including prose alongside `fm_branch_processed`. +From the second triggered presentation of that same set on, until its listed outcomes are acknowledged, an assistant final is removed before persistence only when its text is empty after trimming or exactly matches a final already visible for that sequence set after trimming. +Differing replies stay visible, including the first real handling after an empty or unrelated reply. +Messages carrying tool calls always retain their prose, signed reasoning, and usage accounting. +Pi's Markdown transformer API buffers retry prose while streaming on versions that expose it, so the complete reply can be compared before rendering. +A retained reply renders when the message ends. +Successful acknowledgement releases subsequent assistant output, and a real user message restores ordinary output immediately, including when a processing request rides that prompt. + ### Re-presentation pacing A presentation already pending its run boundary is not resent or widened. @@ -634,7 +642,7 @@ At that moment the branch reports any refusal instead of concluding there is "no - The new branch conversation at every main session start with continuation inside one session, and the mirror re-anchor that pairs with it. - Requested-versus-unsolicited delivery, exact visible entry content, and no unkeyed model turn. - The sequence-keyed processing request and its acknowledgement. -- Re-presentation after an empty reply and after an unrelated prior answer, the triggered-then-next-turn pacing, and session-start re-presentation. +- Suppression of empty or exact-repeat retry finals with differing replies preserved, preserved tool calls and user responses, the triggered-then-next-turn pacing, and session-start re-presentation. - Routine outcomes staying turn-free, task-level no-change notes staying hidden, absent-marker re-presentation, and malformed-age reporting without acknowledgement. - Idle and busy main state, and incident-shaped compaction and unrelated-assistant context. - Cold-start post-lock recovery, crash-before-cursor reload recovery, and repeated-reload idempotency. diff --git a/docs/remote-secondmates.md b/docs/remote-secondmates.md index 93f4057c863..418d7a8aaf8 100644 --- a/docs/remote-secondmates.md +++ b/docs/remote-secondmates.md @@ -82,7 +82,8 @@ On macOS the worker is `dev.firstmate.remote-job`, an Aqua-scoped LaunchAgent at After that bootstrap, every non-doctor `fm-on.sh` target runs through that worker in the remote account's GUI session. It never runs in the SSH process or a Herdr pane. Linux uses the same queue and worker protocol without the Aqua-session requirement. -When idle, the worker checks for newly staged work about once per second; after a lane starts or finishes it checks more frequently for a short period. +The [`fm-remote-job-worker.sh` header](../bin/fm-remote-job-worker.sh) owns dispatch cadence and the quiet-scan latency for work arriving after its post-activity burst. +Active-command and result waits use a separate sampling interval; the [`fm-remote-job-lib.sh` header](../bin/fm-remote-job-lib.sh) owns its defaults, overrides, and completion, cancellation, and timeout latency contract. ### Job lanes and preemption @@ -490,6 +491,7 @@ When deduplication finds that the worker already moved the matching record into The remote host runs no doorbell re-ring ladder of its own. A swallowed doorbell for an ordinary reply-bearing request surfaces through the parent's pending-reply recovery and escalation. Its recovery request rings the doorbell again when it is enqueued. +A fire-and-forget record, such as a reconcile ask, gets its single retry ring only on the local plane, and only when `config/wait-no-turns` is present: the remote steer leg owes no re-ring, so a swallowed remote doorbell for one waits for the next ring into that inbox, and a remote-side retry is known follow-up scope. ### Remote reads @@ -514,6 +516,7 @@ A process-event source takes these steps: - It does not carry blank separators. The listener holds its claim across an empty wait and across a delta it re-arms, so a line appended during either is collected without waiting for the next supervision cycle. +The [`fm-remote-delta-read.sh` header](../bin/fm-remote-delta-read.sh) owns snapshot sampling and its line-visibility and wait-window latency contract. It stops when that registration is retired, the registered command changes, or the home's owner lease lapses. `bin/fm-procevent.sh` owns the generic relisten rule, and `bin/fm-procevent-remote-reply.sh` owns this adapter's answer. diff --git a/docs/secondmate-parent-channel.md b/docs/secondmate-parent-channel.md index a15a163107c..614a26fca83 100644 --- a/docs/secondmate-parent-channel.md +++ b/docs/secondmate-parent-channel.md @@ -38,6 +38,8 @@ A duplicate line is harmless and a missed one is not, so the mate may still appe For marked replies, the report helper accepts no caller-selected destination and uses the channel resolver for both local and remote homes; its script header owns the exact invocation contract. The pending-reply guard may restate only the correlated line from a local mate's `state/.status` onto the parent channel, which repairs the common parent-home versus mate-home mixup without accepting arbitrary mate-home sightings as acknowledgement. Other correlated mate-home status lines remain wrong-home evidence, while a remote home's routed `state/parent-replies.status` is already the parent channel and is not classified as wrong-home. +The mate home's own status scans treat that remote channel the same way: `status_scan_parent_channel_exclude` in `bin/fm-classify-lib.sh` resolves the outbound path through the same `bin/fm-parent-channel-lib.sh` binding, and the watcher's signal scan and heartbeat backstop, the away-mode daemon's catch-all scan, and the fleet-wide folds skip exactly that resolved path, never a file name. +The remote reply adapter already mirrors every channel line into the parent home, so folding the channel again here would only spin spurious wakes and a phantom `parent-replies` task, while a `parent-replies.status` in a main home or in a local mate is an ordinary task log that keeps folding and waking. A missed-reply escalation includes the complete first sighting path and line number in readable shell-escaped form. ## What is deliberately not built @@ -55,6 +57,7 @@ A missed-reply escalation includes the complete first sighting path and line num `tests/fm-teardown.test.sh` covers teardown delivering a child's final line and refusing when the channel cannot be written. `tests/fm-brief.test.sh` pins the charter's channel rule. `tests/fm-pending-reply.test.sh` covers helper-selected local routing, remote-channel classification, same-basename restatement before false escalation, readable wrong-home diagnostics, and the rule that arbitrary mate-home sightings never acknowledge a reply. +`tests/fm-parent-channel-scan-exclusion.test.sh` covers the home-shape-aware scan exclusion against real remote, main-home, and local-mate fixtures: the watcher signal scan, both heartbeat backstops, the fleet-wide folds, and the real `fm-wake-drain.sh` end to end. ## Live verification diff --git a/docs/supervision-host.md b/docs/supervision-host.md index 6f905aa417f..d11a9ec51e9 100644 --- a/docs/supervision-host.md +++ b/docs/supervision-host.md @@ -41,7 +41,7 @@ Today it runs beside a Claude, Cursor, OpenCode, omp, Grok, or Codex primary: aw | Component | Owner | Role | |---|---|---| -| The loop | `bin/fm-supervision-host.sh` | Its header owns the per-close order, the park boundary, ownership checks, predecessor cleanup, state files, and tunables. | +| The loop | `bin/fm-supervision-host.sh` | Its header owns the per-close order, the park boundary and elapsed clock, arm-exit sampling and signal-observation latency, ownership checks, predecessor cleanup, state files, and tunables. | | The arm owners | Each primary's existing arm owner | Runs the host for a home that runs it and delivers a handed-back wake to main; see [Arm owners](#arm-owners). | | The engine | `bin/fm-supervision-engine-lib.sh` | Owns the home gate, including the default on Claude and the opt-out, the verified-engine list, and one bounded engine turn, including the reap of engine tool processes that outlive it. | | Row eligibility and the offer rule | `bin/fm-branch-dispatch.mjs` | The command entry to `.pi/extensions/lib/fm-branch-dispatch.ts`, so the host and the Pi extension compute branch-claimable rows, their task scope, and whether the branch may take a close (`branchOfferForWake`) from one owner; it also renders the wake message with the same away-posture tail, or the dialog mirror at its head. | @@ -104,7 +104,9 @@ The host asks the Pi branch's offer rule (`branchOfferForWake`, through `bin/fm- So a close reaches main off Pi exactly when it would on Pi: a check trigger, a decision-owned signal or stale trigger, and a scan that is unsafe or holds nothing for the branch stay main's. On that main-only pass-through the host starts the successor watcher cycle and leaves it running, then prints the close unchanged. It leaves the watcher's recovery marker reading downtime, confirming no handling handoff, because the re-arm owner delivers a close to main only while that marker reads downtime. -The watcher's singleton lock makes the session's next arm attach to that cycle instead of starting a second one. +The session's next park without `--restart` requests a take-over to restore a single host-owned arm; the [host header](../bin/fm-supervision-host.sh) owns successor persistence and cleanup, and the [arm header](../bin/fm-watch-arm.sh) owns take-over eligibility and fallback. +OpenCode and omp still launch the host with `--restart`, which takes precedence over recorded take-over and lacks its acknowledgement-preserving handover; changing that first-cycle path remains a follow-up. +The host-off Claude Stop hook's detached handling successor is also unchanged; see [Claude handling successor](watcher-continuity.md#claude-handling-successor). It also passes the close through unchanged, with no added line, when any of these holds (`fm_supervision_host_attended_ready` in `bin/fm-supervision-engine-lib.sh` owns the list): - The home names no usable engine. @@ -220,7 +222,10 @@ The captain row is still durable, and the next drain presents it until it is ack ## Failure direction Every path that cannot finish a wake the engine took hands that wake to main, with one `supervision-host: ` line after the close. -Before handing it back, the host stops its successor cycle. +Before handing it back, the host stops its successor cycle, and whenever a successor generation was recorded (confirmed or not), it explicitly republishes downtime for that generation. +That publication is required even when the successor already exited, because no watcher cleanup remains to make the close deliverable to the arm owner. +If that publication fails, the hand-back adds a `supervision-host: watcher downtime could not be restored` line and the host exits nonzero. +On Claude, a Stop hook whose rewake is refused while the recovery marker is still `pending:handling` and no watcher is live commits the auto-arm failure notice once per failure episode (`failed-suppressed` after that) and still exits 2, so the hand-back reaches main; every other refused rewake stays silent as before. So the owner's next arm starts from the same state as without the host, and the wake stays durable in the queue. ### Paths that hand the wake back diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index 63e508cbc99..ff46b31d120 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -545,6 +545,27 @@ The lab home was deleted and the test entry was removed from the store and verif That automated spawn case runs against a fake claude, so it asserts the store entry and the launch command and nothing more; the live arms above are what establish that the entry actually suppresses the dialog. The composer-classification record below observes the same gate from the other side, where an untrusted worktree left Claude, Grok, and Muse unverified because the guard reads a first-launch trust dialog as an unreadable composer. +## Pi seeded-secondmate project trust + +[`fm-spawn.sh --help`](../../bin/fm-spawn.sh) owns the seeded-secondmate project-trust approval contract and compatibility fallback. +The live guard below isolates Pi's trust-gate behavior in secondmate-shaped homes; portable launch-command coverage separately verifies that spawn selects the flag for the intended launches. + +Verified 2026-10-02 on pi 0.82.0 through the default-on live guard (disposable `PI_CODING_AGENT_DIR` / `HOME` only; never `~/.pi`): + +```sh +bash tests/fm-pi-seeded-home-trust-live-e2e.test.sh +``` + +``` +# live pi version: 0.82.0 +ok - fresh seeded Pi secondmate-shaped home stalls on Trust project folder? without --approve +ok - seeded home with --approve starts past the trust dialog without rewriting trust.json +ok - unseeded path without --approve still prompts on Trust project folder? +# all fm-pi-seeded-home-trust-live-e2e checks passed (3) +``` + +Portable launch-command coverage lives in `tests/fm-spawn-dispatch-profile.test.sh` (`test_pi_seeded_secondmate_preapproves_project_trust`, `test_pi_worker_launch_omits_seeded_home_approve`, `test_pi_approve_probe_omits_unsupported_flag`). + ## Launch-prompt backstop signatures `bin/fm-busy-lib.sh`'s launch-prompt backstop (`fm_busy_launch_prompt_parked`) reclassifies a launch whose busy record is still pinned at the fm-spawn seed as `unknown launch-prompt`, rather than `busy fm-spawn`, when the captured pane matches that harness's own recognized trust, sign-in, or first-run dialog. @@ -876,6 +897,61 @@ The current pending-composer ring contract is owned by `bin/fm-task-inbox-lib.sh Kimi was not installed on the verification machine; its receive path is the same one-line-plus-shell contract, and the portable ladder and enqueue regressions in `tests/fm-task-inbox.test.sh` and `tests/fm-send-inbox.test.sh` cover every harness-independent half. This guard is the refresh command after any harness upgrade; it spends a small number of real tokens per installed harness, reports an absent harness explicitly, and refuses a run that verified nothing. +The doorbell no longer prints the inbox's absolute path, so its length no longer grows with the home's depth. +It names the inbox as `"$FM_TASK_INBOX"`, which `bin/fm-spawn.sh` exports into every launch as the absolute `state/.inbox` path, followed by the short `.inbox` name; the brief's full path remains the fallback for a worker launched without that export. +The guard now launches each worker with `FM_TASK_INBOX` exported and no brief, so the worker must resolve the inbox from the doorbell and its environment alone. +It is the refresh command for that shape, which has not yet been recorded live here. +The run below, on 2026-09-30 on tmux 3.6, Linux (WSL2), with the same command, covered the earlier brief-primed shape, whose doorbell named only the short `.inbox` name and whose guard gave each worker the brief's steering-inbox sentence before the steer: + +```text +ok - claude (2.1.285 (Claude Code)): the doorbell reached a real worker, which acted and acked with the mv +ok - codex (codex-cli 0.157.0): the doorbell reached a real worker, which acted and acked with the mv +ok - opencode (1.18.33): the doorbell reached a real worker, which acted and acked with the mv +# harness absent, not verified here: grok +# harness absent, not verified here: kimi +# harness absent, not verified here: muse +``` + +OpenCode needed `FM_SEND_INBOX_LIVE_TIMEOUT=560` because its configured model was still mid-turn at the default 240 seconds. +Pi 0.87.1 was installed but not verified: its configured model returned an account error (`The 'gpt-5.6-sol' model is not supported when using Codex with a ChatGPT account`) before it read the inbox. + +## Waiting-worker command ceilings + +The `# Waiting` section of the ship and scout briefs (`bin/fm-brief.sh`) has a worker hold every external wait inside one blocking shell command, bounded by what its harness lets one command run. +That section is generated only when `config/wait-no-turns` is present. +Those bounds were read from the installed vendor code on 2026-09-11, macOS arm64, with Pi 0.85.1, codex-cli 0.154.0, and Claude Code 2.1.268. + +```sh +grep -n "Timeout in seconds" "$(npm root -g)/@earendil-works/pi-coding-agent/dist/core/tools/bash.js" +strings -n 20 "$(readlink -f "$(command -v codex)")" | grep -o "Non-empty writes default to [^.]*; empty polls wait [^.]*\." +strings -n 8 "$(readlink -f "$(command -v claude)")" | grep -oE '=120000,[A-Za-z0-9_$]+=600000;' | head -1 +``` + +Observed output: + +```text +28: timeout: Type.Optional(Type.Number({ description: "Timeout in seconds (optional, no default timeout)" })), +Non-empty writes default to 250 ms and cap at 30000 ms; empty polls wait 5000-300000 ms by default. +=120000,ARo=600000; +``` + +Pi's bash tool runs a command with no time limit unless the call passes `timeout`, so the brief asks for at most 2700 seconds, which stays under the watcher's 3600-second busy-turn bound. +Codex yields a still-running command back to the model, and one empty `write_stdin` poll then waits up to 300000 ms. +Claude Code's Bash tool defaults to 120000 ms and accepts at most 600000 ms; `BASH_DEFAULT_TIMEOUT_MS` and `BASH_MAX_TIMEOUT_MS` override those two values. + +Claude Code also constrains the shape of a wait, not only its length, so the brief has to name the shape that is allowed rather than only forbid the ones that are not. +Run as separate Bash tool calls on 2026-09-14 with Claude Code 2.1.268: + +```sh +until [ -e /tmp/fm-wait-probe ]; do sleep 30; done # ran to completion, rc=0 +sleep 61; echo "rc=$?" # rc=0 +sleep 40; echo "checked at $(date +%s)" # rc=0 +``` + +An earlier `sleep 60` chained ahead of a status check was refused before execution, with a message pointing at `Monitor` with an until-loop and at `run_in_background: true`, and adding "Do not chain shorter sleeps to work around this block". +The blocking foreground `until` loop is therefore the wait a Claude Code worker may use, and it is what the brief names, because the refusal's own `run_in_background` suggestion is the one shape a waiting worker must not take: a backgrounded call returns at once and so does not wait at all. +The brief's portable regression is `tests/fm-brief.test.sh`; rerun these commands after upgrading any of the three harnesses and update the numbers in the brief when they move. + ## Gemini The Gemini crewmate adapter was verified on 2026-09-04 with gemini-cli 0.58.0 on Linux, Node v24.20.0, tmux 3.4. @@ -2194,6 +2270,22 @@ FM_HARNESS_LIVENESS_DRIFT=1 bin/fm-test-run.sh tests/fm-harness-liveness-drift-l The supervision-branch extension (`.pi/extensions/fm-branch-supervision.ts`, [docs/pi-supervision-branch.md](../pi-supervision-branch.md)) builds its second session through the Pi SDK surface: `createAgentSession` (including its `model`, `modelRuntime`, and `thinkingLevel` options), `DefaultResourceLoader` with `extensionFactories`, `SessionManager`, `createBashToolDefinition` with a `spawnHook`, `sendCustomMessage` for routine notes, `appendEntry` and `registerEntryRenderer` for captain outcomes, the `before_provider_request` hook, the command context's model registry for picker candidates, a fresh `ModelRuntime` for isolated-branch resolution, and Pi's own `getSupportedThinkingLevels`/`clampThinkingLevel` plus its `getThinkingLevel` and `thinking_level_select` extension surface for effort. In TUI mode, its `/supervision-model` model list is drawn with Pi's own `SelectList`, `Input`, `fuzzyFilter`, and `DynamicBorder` through the extension context's `ui.custom` surface, which is what bounds and searches a long catalog. +Processing-retry visibility was verified on 2026-09-27 against Pi 0.87.1 with a local intercepted provider stream, without credentials or an external provider request: + +```sh +bin/fm-test-run.sh tests/fm-pi-branch-extension.test.sh +FM_PI_BRANCH_LIVE_E2E=1 npm exec --yes --package=typescript@5.9.3 -- bin/fm-test-run.sh tests/fm-pi-branch-live-e2e.test.sh tests/fm-pi-primary-types.test.sh +``` + +```text +ok - real Pi SDK 0.87.1 suppresses only empty or exact-repeat retry finals, retains first and differing replies after reopen, buffers retry streaming, and keeps outcomes retryable +ok - tracked Pi extensions pass strict no-emit typecheck against Pi 0.87.1 +``` + +The guard runs the extension through Pi's actual message event runner, renders its streamed replies with the stock assistant component, and checks both live agent state and a reopened session file. +The portable processing-turn case additionally covers whitespace-only replies, a one-character difference, prose alongside acknowledgment calls, signed reasoning and tool-call preservation, rejected and partial acknowledgements, busy follow-ups, user steering, and both orderings of a user message batched with a processing request. +Other primary harnesses do not load this Pi extension, and these event and persistence boundaries are independent of the runtime session backend. + Evidence produced 2026-08-25 on macOS 26.5.2 arm64, Node v24.13.1: - Historical real-SDK guard: `FM_PI_BRANCH_LIVE_E2E=1 bin/fm-test-run.sh tests/fm-pi-branch-live-e2e.test.sh` against the globally installed `@earendil-works/pi-coding-agent` 0.81.1 printed `ok - real Pi SDK 0.81.1 accepts the branch session construction and preserves an unpromptable wake`. diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index f09bd3c7cf8..1032808851b 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -42,6 +42,7 @@ Each adapter: - Preserves one child or scheduled retry at a time. - Applies bounded exponential retry after an unexpected or failed close. +Pi treats an arm child whose process is already gone as an empty slot even while its close event is still pending, so a repair call or a scheduled retry starts a fresh arm instead of answering unchanged. A failed follow-up never cancels continuity restoration. ### Pi session replacement @@ -121,7 +122,7 @@ The Claude turn-end guard owns that notice commit contract, the monotonic failur On a non-Pi primary, a home that runs the supervision host runs `bin/fm-supervision-host.sh` in place of the arm its re-arm owner would start. The host owns successive watcher cycles through the same arm. -The host's successor and pass-through lifecycle is owned by [supervision-host.md](supervision-host.md#postures); the arm's recovery and acknowledgement contracts below still apply. +[supervision-host.md](supervision-host.md#failure-direction) owns the hand-back's downtime restoration, including when the successor already exited; the arm's recovery and acknowledgement contracts below still apply. ## Actionable wake ordering @@ -133,14 +134,19 @@ After an actionable Pi, omp, or OpenCode child close, the adapter: 1. Waits for the predecessor process to close. 2. Starts and verifies one singleton successor. -3. Confirms the handling handoff against that successor before scheduling the follow-up. +3. Confirms the handling handoff before scheduling the follow-up: Pi confirms against the restoration's own recovery token, while omp and OpenCode confirm against the current successor. 4. Delivers the original wake. A complete Pi reason line can be observed while the predecessor is still finishing durable cleanup. That line is retained for replacement handoff, but the adapter never treats that already-ready predecessor as its own successor. -If the handoff confirmation fails, the adapter retries it once against the current generation and successor. -A failed confirmation is a restoration failure: the adapter classifies the error, retires a successor that is no longer alive, and surfaces exactly one typed message. +If the handoff confirmation fails, the adapter retries it once: Pi against that same token, omp and OpenCode against the current generation and successor. +A failed confirmation is a restoration failure: the adapter classifies the error and surfaces exactly one typed message. +Pi retires the current successor only when the failed token names its exact watcher pid and generation and that pid is no longer alive, while omp and OpenCode retire the current successor whenever the restoration's watcher pid is no longer alive. +On Pi a generation mismatch means a newer pipeline superseded this delivery mid-restore, so the wake routes like a confirmed delivery, with no failure appendix, and nothing is retired. +An already-acknowledged episode confirms as a no-op when the confirmation names its generation, because the drain acknowledged it after the successor started but before the confirmation ran. +The Pi extension diagnostic log is opt-in and off by default: only a positive FM_WATCH_EXTENSION_LOG_KEEP_LINES value appends restore attempts, readiness timeouts, and confirmation targets and results to state/.watch-extension.log, a bounded record that never changes supervision behavior. +docs/configuration.md owns the knob's default and accepted values. A failed confirmation is never swallowed. ### Readiness timeout and retry @@ -228,6 +234,9 @@ A downtime republication of a pending episode reuses its generation. A watcher close leaves an announced downtime episode announced, while a successful durable append opens a fresh pending generation so a live watcher can recover the new work. An announced handling episode becomes pending downtime on the same generation because its handling turn may have been interrupted. That handling republication gives a successor exactly one recovery presentation without orphaning the acknowledgement already printed for that generation. +A watcher stopped so an arm can take its cycle over (`bin/fm-watch-arm.sh --take-over`) publishes downtime like any close, but the taking arm restores an acknowledged episode that stop reopened only when the taken-over arm's cycle-ledger row for that exact arm and watcher records the watcher ending by the take-over's TERM and no wake was appended in between. +The taking arm waits within a short bound for that row; a missing row or any other signal leaves downtime for the fresh cycle's ordinary recovery wake, while take-over still proceeds. +Any other episode is left for the next cycle's arm check. ### What an acknowledgement retires @@ -424,6 +433,9 @@ The same suite covers ordinary same-process session replacement for `/new`, `/re - Repeated transitions with exactly one live cycle. - Disappearance of the shutting-down refusal after a valid replacement activates. - Terminal quit still refusing late rearm. +- A mid-restore marker advance that delivers the wake with no rejection appendix, offers it to an accepting supervision branch like a confirmed delivery, and records the attempt and the confirm result in the bounded extension log when opted in. +- A failed confirmation for a stale successor that spares a newer arm started by a repair. +- A repair, a scheduled retry, and a deferred close over a dead-but-unclosed arm child that each start a fresh arm instead of stalling. The guard and session-start suites prove that active generation evidence tolerates a fresh-beacon handoff. They also prove that a legacy or handoff-phase watcher marker from an absent replacement extension still raises the outage diagnostic. @@ -443,6 +455,9 @@ They also prove that a legacy or handoff-phase watcher marker from an absent rep - A watcher close inside the handling window that must leave the printed acknowledgement valid. - A re-arm whose recovery cycle is slowed after confirmation and must still surface rather than read as a watcher that stayed live. - The self-healing moved-generation acknowledgement that consumes its handled rows and names its remedy. +- The already-acknowledged confirmation no-op for a matching generation, with its mismatched-generation, dead-pid, and lock-mismatch rejections preserved. +- The manual-restart generation churn that makes a confirmation for the churned generation report a mismatch, which an arm check without a reopen leaves in place. +- A take-over that stays quiet after a confirmed TERM, still surfaces queued work and self-exit downtime, and attaches without stopping a cycle the named arm does not own. - The disposable-checkout arm refusal. - The home-gone and state-gone watcher exits. - The test reaper that stops a watcher armed for a temporary home. diff --git a/tests/fm-afk-return.test.sh b/tests/fm-afk-return.test.sh index 62978995b57..19c2cf7acbd 100755 --- a/tests/fm-afk-return.test.sh +++ b/tests/fm-afk-return.test.sh @@ -41,6 +41,7 @@ install_runner() { # cp "$ROOT/bin/fm-afk-contract.sh" "$dir/bin/" cp "$ROOT/bin/fm-branch-outcome.sh" "$dir/bin/" cp "$ROOT/bin/fm-tasks-axi-lib.sh" "$dir/bin/" + cp "$ROOT/bin/fm-hold-reason-lib.sh" "$dir/bin/" cp "$ROOT/bin/fm-backlog-transition-lib.sh" "$dir/bin/" # The merge-notification marker reader behind the brief's landed section. cp "$ROOT/bin/fm-pr-lib.sh" "$dir/bin/" diff --git a/tests/fm-backend-herdr.test.sh b/tests/fm-backend-herdr.test.sh index 87fca554ca4..b1bfcf21594 100755 --- a/tests/fm-backend-herdr.test.sh +++ b/tests/fm-backend-herdr.test.sh @@ -314,7 +314,13 @@ test_version_check_refuses_old_protocol() { test_version_check_refuses_missing_herdr() { local dir out status dir="$TMP_ROOT/version-missing"; mkdir -p "$dir/empty-fakebin" - out=$( PATH="$dir/empty-fakebin:/usr/bin:/bin" \ + # Hermetic PATH: the fakebin carries only bash (so the inner `bash -c` + # still resolves) and no system dir, so a real herdr installed under + # /usr/bin (or /bin -> usr/bin) cannot leak into this "not installed" + # simulation. fm_backend_herdr_tool_check needs no external tool on this + # path: `command -v` is a builtin and it short-circuits on herdr first. + ln -sf "$(command -v bash)" "$dir/empty-fakebin/bash" + out=$( PATH="$dir/empty-fakebin" \ bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_version_check' "$ROOT" 2>&1 ) status=$? [ "$status" -ne 0 ] || fail "version_check should refuse when herdr is not installed" @@ -5185,6 +5191,37 @@ test_send_text_submit_claude_slash_popup_composer_is_still_proven_and_submitted( pass "fm_backend_herdr_send_text_submit: a typed slash command hidden behind its popup is still proven and submitted" } +# Live Claude Code 2.1.283 draws a recognized typed slash command in muted +# truecolor grey (38;2;112;112;112, luminance 112), below the grok-tuned +# dark-foreground ghost threshold. Claude's own ghost suggestion is SGR-2 dim, +# so the Claude payload proof must not strip the grey command and judge the +# typed /exit unsent (the fm-control exit breakage, reproduced live). +test_send_text_submit_claude_grey_slash_command_is_proven_and_submitted() { + local dir log resp fb out enter_count text rule head + dir="$TMP_ROOT/submit-claude-grey-slash"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" + text='/exit' + herdr_submit_claude_prefix "$resp" "$text" + rule=$(printf '%0.s\xe2\x94\x80' $(seq 1 60)) + head=$(printf '%0.s\xe2\x94\x80' $(seq 1 19)) + { + printf ' \x1b[0m\x1b[38;2;112;112;112m/\x1b[0m\x1b[1m\x1b[38;2;112;112;112mexit\x1b[0m\x1b[38;2;112;112;112m Exit the CLI\x1b[0m\n' + printf '\x1b[0m\x1b[38;2;121;129;134m%s Firstmate operational input 1790546042 \xe2\x94\x80\x1b[0m\n' "$head" + printf '\xe2\x9d\xaf\xc2\xa0\x1b[0m\x1b[38;2;112;112;112m/exit\x1b[0m\n' + printf '\x1b[0m\x1b[38;2;121;129;134m%s\x1b[0m\n' "$rule" + printf ' \x1b[0m\x1b[38;2;86;93;96m\xe2\x8f\xb5\xe2\x8f\xb5 bypass permissions on\x1b[0m\n' + } > "$resp/4.out" + printf '{"result":{"agent":{"agent":"claude","agent_status":"idle"}}}\n' > "$resp/5.out" + printf '{"result":{"agent":{"agent_status":"working"}}}\n' > "$resp/7.out" + fb=$(make_herdr_fakebin "$dir") + out=$( PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" FM_BACKEND_HERDR_SUBMIT_POLLS=1 \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_send_text_submit default:w1:p2 "$1" 3 0.01 0.01' "$ROOT" "$text" ) + [ "$out" = empty ] || fail "a typed /exit drawn in Claude's grey slash-command colour must be proven and submitted, got '$out'" + enter_count=$(grep -c $'\x1f''pane'$'\x1f''send-keys'$'\x1f''w1:p2'$'\x1f''enter' "$log") + [ "$enter_count" -eq 1 ] || fail "the proven grey slash command should be submitted once, sent $enter_count Enter(s)" + [ "$(herdr_ctrl_u_count "$log")" -eq 0 ] || fail "a proven grey slash command must not be cleared" + pass "fm_backend_herdr_send_text_submit: a typed slash command Claude draws in muted truecolor grey is proven and submitted" +} + test_send_text_submit_lone_paste_placeholder_submits_the_long_payload() { local dir log resp fb out enter_count text dir="$TMP_ROOT/submit-paste-placeholder"; mkdir -p "$dir/responses"; log="$dir/log"; resp="$dir/responses"; : > "$log" @@ -6096,6 +6133,7 @@ test_send_text_submit_accepts_marked_payloads_whose_read_back_drops_u2063 test_send_text_submit_refuses_marked_digest_missing_its_head test_composer_state_claude_slash_popup_pushes_composer_above_tail_window test_send_text_submit_claude_slash_popup_composer_is_still_proven_and_submitted +test_send_text_submit_claude_grey_slash_command_is_proven_and_submitted test_send_text_submit_lone_paste_placeholder_submits_the_long_payload test_send_text_submit_multiline_paste_placeholder_submits_the_long_payload test_send_text_submit_refuses_placeholder_followed_by_a_literal_remainder diff --git a/tests/fm-backend.test.sh b/tests/fm-backend.test.sh index 96d00b10303..a9030018d8c 100755 --- a/tests/fm-backend.test.sh +++ b/tests/fm-backend.test.sh @@ -502,17 +502,32 @@ test_backend_validate_refuses_unknown() { } test_backend_source_shell_portable() { - local out status + local out status stub probe # zsh does not word-split unquoted expansions; sourcing fm-backend.sh from # an interactive zsh session must still recognize known backend names. + # The claim is name matching and the sibling precheck only: the adapters + # find their own siblings through BASH_SOURCE, so zsh is not a full load. if command -v zsh >/dev/null 2>&1; then - zsh -c "cd '$ROOT' && source bin/fm-backend.sh && fm_backend_source herdr && whence -w fm_backend_herdr_capture >/dev/null" 2>/dev/null \ - || fail "zsh: fm_backend_source herdr should load the adapter when sourced" + zsh -c "cd '$ROOT' && source bin/fm-backend.sh && fm_backend_source herdr" >/dev/null 2>&1 \ + || fail "zsh: fm_backend_source herdr should accept the known backend name and find its sibling libraries" out=$(zsh -c "cd '$ROOT' && source bin/fm-backend.sh && fm_backend_source bogus" 2>&1) \ && fail "zsh: fm_backend_source bogus should fail" assert_contains "$out" "unknown backend 'bogus'" \ "zsh: fm_backend_source did not reject bogus with the expected error" pass "zsh: fm_backend_source recognizes known backends and rejects unknown ones" + + # zsh ties the lowercase `path` array to PATH; a backend loaded while + # fm_backend_source clobbers PATH cannot resolve external commands. + stub="$TMP_ROOT/zsh-source-path" + probe="$stub/probe" + mkdir -p "$stub/backends" + printf 'command -v dirname > "%s"\n' "$probe" > "$stub/backends/orca.sh" + : > "$stub/fm-composer-lib.sh" + zsh -c "cd '$ROOT' && source bin/fm-backend.sh && FM_BACKEND_LIB_DIR='$stub' && fm_backend_source orca" >/dev/null 2>&1 \ + || fail "zsh: fm_backend_source orca should load a stub adapter" + [ -s "$probe" ] \ + || fail "zsh: fm_backend_source clobbered PATH while loading a backend adapter" + pass "zsh: fm_backend_source keeps PATH intact while loading a backend adapter" else pass "zsh: shell-portable backend matching skipped (zsh not found)" fi diff --git a/tests/fm-backlog-atomicity.test.sh b/tests/fm-backlog-atomicity.test.sh index 09c439826d9..8dc2f19281b 100755 --- a/tests/fm-backlog-atomicity.test.sh +++ b/tests/fm-backlog-atomicity.test.sh @@ -2083,6 +2083,45 @@ test_recovery_replays_a_close_an_interrupted_cleanup_left_open() { pass "session start finishes a close an interrupted cleanup recorded but never landed" } +test_recovery_replays_a_gerrit_close_with_its_change_url_as_a_note() { + local case_dir id out real_tasks_axi gerrit_url=https://gerrit.example.com/c/project/+/12345 + id=atomic-heal-gerrit-b9 + case_dir=$(make_home heal-pending-gerrit-close) + add_item "$case_dir" "$id" + start_item "$case_dir" "$id" + # The record a pre-fix teardown left: the Gerrit change URL as a --pr link. + printf 'id=%s\ndata=%s\nspawn_gen=spawn-heal-gerrit\narg=--pr\narg=%s\n' \ + "$id" "$(home_of "$case_dir")/data" "$gerrit_url" \ + > "$(home_of "$case_dir")/state/$id.backlog-close" + # Pin the refusal tasks-axi applies to a --pr link that is not a canonical + # GitHub pull request, so this case keeps reproducing whatever the installed + # release accepts. + real_tasks_axi=$(command -v tasks-axi) + cat > "$case_dir/fakebin/tasks-axi" </dev/null \ + || fail "the replayed Gerrit close did not record its change URL as a note" + assert_absent "$(home_of "$case_dir")/state/$id.backlog-close" \ + "a replayed Gerrit close left its record behind" + pass "session start replays a recorded Gerrit close with its change URL as a note" +} + test_recovery_backfills_a_recorded_link_on_an_already_done_item() { local case_dir id marker out id=atomic-heal-done-backfill-b9 @@ -3111,6 +3150,7 @@ test_recovery_marks_an_owned_record_in_flight test_recovery_rejects_an_internal_worker_record_symlink test_recovery_ignores_a_symlinked_worker_record test_recovery_replays_a_close_an_interrupted_cleanup_left_open +test_recovery_replays_a_gerrit_close_with_its_change_url_as_a_note test_recovery_backfills_a_recorded_link_on_an_already_done_item test_recovery_preserves_a_close_when_the_backlog_cannot_be_read test_recovery_retry_preserves_incomplete_cleanup_warning diff --git a/tests/fm-brief.test.sh b/tests/fm-brief.test.sh index 296a73ccdf4..9db2f637ccf 100755 --- a/tests/fm-brief.test.sh +++ b/tests/fm-brief.test.sh @@ -920,7 +920,8 @@ SIGNALS test_ship_and_scout_teach_validation_round_pause() { local home kind id brief home="$TMP_ROOT/validation-round-pause-home" - mkdir -p "$home/data" + mkdir -p "$home/data" "$home/config" + : > "$home/config/wait-no-turns" for kind in ship scout; do id="brief-validation-round-pause-$kind" @@ -930,6 +931,12 @@ test_ship_and_scout_teach_validation_round_pause() { FM_HOME="$home" "$ROOT/bin/fm-brief.sh" "$id" firstmate --mode no-mistakes >/dev/null 2>&1 fi brief="$home/data/$id/brief.md" + assert_grep "your own validation round, which you declare once just before its blocking hold" "$brief" \ + "$kind brief did not teach workers to declare their validation-round wait before holding it" + assert_grep "append \`paused:\` once just before its first blocking command, then stay in the command" "$brief" \ + "$kind brief's Waiting section does not declare the validation round once and then hold it" + assert_no_grep "is not a \`paused:\` wait" "$brief" \ + "$kind brief still tells workers never to declare a wait they hold in a command" assert_grep "your own validation round" "$brief" \ "$kind brief did not teach workers to declare their validation-round wait" assert_grep 'Before ending your turn with your own background shell or monitor still running' "$brief" \ @@ -941,7 +948,7 @@ test_ship_and_scout_teach_validation_round_pause() { assert_grep 'Do not declare active implementation or reasoning as a wait' "$brief" \ "$kind brief did not limit the declaration to actual waits" done - pass "fm-brief.sh: ship and scout scaffolds teach validation-round pauses" + pass "fm-brief.sh: ship and scout scaffolds declare a validation-round pause once, then hold it" } # Every ship mode and the scout scaffold carry the same advisory Jev-first rule @@ -1054,6 +1061,68 @@ test_scout_and_secondmate_scaffold() { pass "fm-brief: scout and secondmate code paths still scaffold well-formed briefs" } +# Contract: a waiting worker spends no turns. A decision wait ends the turn, an +# external wait sleeps in one bounded blocking shell command sized per harness, +# and a waiting worker neither polls its inbox nor polls a pipeline between holds. +test_workers_wait_without_spending_turns() { + local home id brief + home="$TMP_ROOT/wait-home" + mkdir -p "$home/data" "$home/config" + : > "$home/config/wait-no-turns" + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" brief-wait-ship some-proj --mode no-mistakes >/dev/null 2>&1 \ + || fail "fm-brief.sh ship scaffold exited non-zero" + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" brief-wait-scout some-proj --scout >/dev/null 2>&1 \ + || fail "fm-brief.sh scout scaffold exited non-zero" + for id in brief-wait-ship brief-wait-scout; do + brief="$home/data/$id/brief.md" + assert_grep "end your turn at once" "$brief" "$id: a decision wait must end the turn" + assert_grep "with ONE blocking shell command that returns when the state changes" "$brief" \ + "$id: an external wait must sleep in one blocking shell command" + assert_grep "gh pr checks --watch" "$brief" "$id: the CI wait primitive is missing" + assert_grep "a \`timeout\` of at most 2700 seconds" "$brief" "$id: the Pi ceiling is missing" + assert_grep "its maximum \`timeout\` of 600000 ms" "$brief" "$id: the Claude Code ceiling is missing" + assert_grep "empty \`write_stdin\` polls of up to 300000 ms" "$brief" "$id: the Codex ceiling is missing" + assert_grep "is the sanctioned foreground wait" "$brief" \ + "$id: the wait a Claude Code worker may use is not named" + assert_grep "reattach with \`no-mistakes axi run --wait\` instead, and never send the same \`respond\` again" "$brief" \ + "$id: a timed-out respond must reattach with axi run, never resend its answer" + assert_grep "Do not poll or list the inbox while waiting; a waiting instruction rings." "$brief" \ + "$id: polling the inbox while waiting is not forbidden" + assert_grep "natural checkpoint" "$brief" "$id: the flag dropped the natural-checkpoint inbox check" + done + brief="$home/data/brief-wait-ship/brief.md" + assert_grep "issue the same foreground call again" "$brief" \ + "the no-mistakes DOD must reattach with the same foreground call" + assert_no_grep "background the drive call" "$brief" "the no-mistakes DOD still backgrounds the drive call" + + FM_SECONDMATE_CHARTER='Supervise the alpha domain.' \ + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" brief-wait-sm --secondmate --no-projects >/dev/null 2>&1 \ + || fail "fm-brief.sh secondmate scaffold exited non-zero" + brief="$home/data/brief-wait-sm/brief.md" + assert_grep "Do not poll or list the inbox while waiting; a waiting instruction rings." "$brief" \ + "secondmate: polling the inbox while waiting is not forbidden" + assert_grep "natural checkpoint" "$brief" "secondmate: the flag dropped the natural-checkpoint inbox check" + pass "fm-brief: workers end the turn on a decision, wait in one bounded shell command, and never poll" +} + +# Without config/wait-no-turns the scaffold matches the pre-flag brief and drive text. +test_wait_no_turns_absent_keeps_the_previous_brief() { + local home brief + home="$TMP_ROOT/wait-off" + mkdir -p "$home/data" + [ ! -e "$home/config/wait-no-turns" ] + FM_HOME="$home" "$ROOT/bin/fm-brief.sh" brief-wait-off some-proj --mode no-mistakes >/dev/null 2>&1 \ + || fail "fm-brief.sh ship scaffold exited non-zero" + brief="$home/data/brief-wait-off/brief.md" + assert_no_grep "end your turn at once" "$brief" "an absent flag still added the waiting section" + assert_grep "natural checkpoint" "$brief" "an absent flag dropped the unprompted inbox check" + assert_no_grep "Do not poll or list the inbox while waiting" "$brief" "an absent flag still added the no-poll inbox line" + assert_grep "background the drive call" "$brief" "an absent flag replaced the backgrounded drive text" + assert_no_grep "issue the same foreground call again" "$brief" \ + "an absent flag still asked for the foreground reattach" + pass "fm-brief: without config/wait-no-turns the brief and drive text stay as they were" +} + test_worker_role_scope() { local kind home brief home="$TMP_ROOT/worker-role" @@ -1381,6 +1450,8 @@ test_ship_and_scout_carry_advisory_jev_rule test_scout_and_secondmate_load_decision_hold_policy test_scout_and_secondmate_scaffold test_scout_lavish_line_follows_presentation_floor +test_workers_wait_without_spending_turns +test_wait_no_turns_absent_keeps_the_previous_brief test_home_brief_include_is_appended_last test_ship_branch_prefix_defaults_to_legacy_fm test_ship_branch_prefix_override_is_consistent_across_modes diff --git a/tests/fm-calm-claude-mod.test.sh b/tests/fm-calm-claude-mod.test.sh index 69ab66558e0..ce5dcf8a869 100644 --- a/tests/fm-calm-claude-mod.test.sh +++ b/tests/fm-calm-claude-mod.test.sh @@ -33,6 +33,13 @@ run_node() { # node --input-type=module <"$1" } +# js_string : a JavaScript string literal for a shell value, for the +# generated scripts below. ${value@Q} would need Bash 4.4 and yields shell +# quoting; stock macOS Bash 3.2 reports a bad substitution. +js_string() { # + node -e 'process.stdout.write(JSON.stringify(process.argv[1]))' -- "$1" +} + test_plugin_shape() { local link resolved autoload link="$ROOT/.agents/skills/firstmate-calm" @@ -49,7 +56,7 @@ test_plugin_shape() { [ ! -e "$MOD/SKILL.md" ] || fail "the mod carries a SKILL.md and would load as a skill on every harness" cat >"$TMP_ROOT/shape.mjs" <"$TMP_ROOT/sprite.mjs" <"$TMP_ROOT/raster.mjs" < { if (!condition) throw new Error(message); }; for (let length = 0; length <= 80; length += 1) { const bytes = new Uint8Array(randomBytes(length)); @@ -235,8 +242,8 @@ test_presentation_policy() { local out cat >"$TMP_ROOT/policy.mjs" < { if (!condition) throw new Error(message); }; const plugin = "/repo/.claude/mods/firstmate-calm"; check(policy.calmPreferencePath({}, plugin) === "/repo/config/calm", "plugin-root fallback"); @@ -473,8 +480,8 @@ test_classifier_parity_with_shell_owner() { cat >"$TMP_ROOT/classify.mjs" <"$TMP_ROOT/doorbells.mjs" <&1 | grep -q -- '--tui-mode'; then + PI_TUI_MODE_ARGS='--tui-mode regular' +fi + find_chrome() { local candidate if [ -n "${FM_CHROME_BIN:-}" ] && [ -x "$FM_CHROME_BIN" ]; then @@ -1651,8 +1662,13 @@ for (const { name, actual } of rows) { async function assertStockHtmlRendering(command, submitData) { editorText = command; terminalInputHandler(submitData); + const resolveTool = (name) => tools.find((tool) => tool.name === name); const htmlRenderer = createToolHtmlRenderer({ - getToolDefinition: (name) => tools.find((tool) => tool.name === name), + getToolDefinition: resolveTool, + getToolRenderers: (name) => { + const tool = resolveTool(name); + return tool ? { renderCall: tool.renderCall, renderResult: tool.renderResult } : undefined; + }, theme, cwd: process.cwd(), }); @@ -1684,6 +1700,10 @@ editorText = "/export remapped.html"; terminalInputHandler("\r"); const unmatchedRenderer = createToolHtmlRenderer({ getToolDefinition: (name) => tools.find((tool) => tool.name === name), + getToolRenderers: (name) => { + const tool = tools.find((candidate) => candidate.name === name); + return tool ? { renderCall: tool.renderCall, renderResult: tool.renderResult } : undefined; + }, theme, cwd: process.cwd(), }); @@ -2289,7 +2309,7 @@ TS fi tmux -L "$TMUX_SOCKET" new-session -d -s "$TMUX_SESSION" -x 160 -y 36 \ - "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi --approve --no-context-files --no-skills --no-prompt-templates --no-extensions $extensions $session_arg; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" + "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi $PI_TUI_MODE_ARGS --approve --no-context-files --no-skills --no-prompt-templates --no-extensions $extensions $session_arg; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" i=0 while [ "$i" -lt 120 ]; do pane=$(tmux -L "$TMUX_SOCKET" capture-pane -p -t "$TMUX_SESSION" -S - 2>/dev/null || true) @@ -2428,7 +2448,7 @@ JS tmux -L "$TMUX_SOCKET" kill-session -t "$TMUX_SESSION" 2>/dev/null || true printf '%s\n' on >"$home/config/calm" tmux -L "$TMUX_SOCKET" new-session -d -s "$TMUX_SESSION" -x 160 -y 36 \ - "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi --approve --no-context-files --no-skills --no-prompt-templates --no-extensions -e ./.pi/extensions/fm-calm.ts -e ./followup-e2e.ts --session '$exact_session'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" + "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi $PI_TUI_MODE_ARGS --approve --no-context-files --no-skills --no-prompt-templates --no-extensions -e ./.pi/extensions/fm-calm.ts -e ./followup-e2e.ts --session '$exact_session'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" i=0 while [ "$i" -lt 120 ]; do pane=$(tmux -L "$TMUX_SOCKET" capture-pane -p -t "$TMUX_SESSION" -S - 2>/dev/null || true) @@ -2599,7 +2619,7 @@ TS printf '%s\n' "$calm_state" >"$home/config/calm" mkdir -p "$sessions/$label" tmux -L "$TMUX_SOCKET" new-session -d -s "$TMUX_SESSION" -x 160 -y 36 \ - "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' QUEUED_ESCAPE_HELD='$held' QUEUED_ESCAPE_STATUS_LOG='$sessions/$label/status.log' PI_OFFLINE=1 pi --approve --no-context-files --no-skills --no-prompt-templates --no-extensions -e ./.pi/extensions/fm-calm.ts -e ./queued-escape-e2e.ts --session-dir '$sessions/$label'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" + "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' QUEUED_ESCAPE_HELD='$held' QUEUED_ESCAPE_STATUS_LOG='$sessions/$label/status.log' PI_OFFLINE=1 pi $PI_TUI_MODE_ARGS --approve --no-context-files --no-skills --no-prompt-templates --no-extensions -e ./.pi/extensions/fm-calm.ts -e ./queued-escape-e2e.ts --session-dir '$sessions/$label'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" wait_for_text "$TMP_ROOT/queued-escape-pane" 'queued-escape-e2e.ts' \ || fail "Pi queued-row $label case did not reach the ready composer" tmux -L "$TMUX_SOCKET" send-keys -t "$TMUX_SESSION" -l "/queued-escape-e2e $label" @@ -2798,7 +2818,7 @@ TS local session_arg=$1 tmux -L "$TMUX_SOCKET" kill-session -t "$TMUX_SESSION" 2>/dev/null || true tmux -L "$TMUX_SOCKET" new-session -d -s "$TMUX_SESSION" -x 100 -y 44 \ - "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' PI_OFFLINE=1 pi --approve --no-context-files --no-prompt-templates --no-extensions -e ./.pi/extensions/fm-calm.ts -e ./geometry-provider.ts $session_arg; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" + "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' PI_OFFLINE=1 pi $PI_TUI_MODE_ARGS --approve --no-context-files --no-prompt-templates --no-extensions -e ./.pi/extensions/fm-calm.ts -e ./geometry-provider.ts $session_arg; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 20" } capture_geometry_viewport() { @@ -2880,7 +2900,7 @@ TS tmux -L "$TMUX_SOCKET" send-keys -t "$TMUX_SESSION" Enter wait_for_geometry_transition \ "$snapshot" \ - "Reloading keybindings, extensions, skills, prompts, themes, and context files..." \ + "Reloading" \ "CALM_GEOMETRY_FINAL" \ || fail "Pi Calm hidden-block geometry E2E did not complete the /reload viewport transition" assert_geometry_gap "$snapshot" "reloaded native Calm transcript" @@ -4190,7 +4210,7 @@ TS JSON tmux -L "$TMUX_SOCKET" new-session -d -s "$TMUX_SESSION" -x 180 -y 44 \ - "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi --approve --no-skills --no-prompt-templates --no-context-files --session '$session_file'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 30" + "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi $PI_TUI_MODE_ARGS --approve --no-skills --no-prompt-templates --no-context-files --session '$session_file'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 30" wait_for_text "$default_snapshot" "The deterministic tool example is complete." \ || fail "Pi calm E2E did not reach the restored session transcript" assert_contains "$(cat "$default_snapshot")" "CALM_E2E_OUTPUT" "calm mode was not off by default" @@ -4863,7 +4883,7 @@ JS tmux -L "$TMUX_SOCKET" kill-session -t "$TMUX_SESSION" 2>/dev/null || true tmux -L "$TMUX_SOCKET" new-session -d -s "$TMUX_SESSION" -x 180 -y 44 \ - "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi --approve --no-skills --no-prompt-templates --no-context-files --session '$session_file'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 30" + "cd '$project' && env FM_HOME='$home' PI_CODING_AGENT_DIR='$config' FM_OPERATIONAL_INPUT_SCRIPT='$OPERATIONAL_INPUT' PI_OFFLINE=1 pi $PI_TUI_MODE_ARGS --approve --no-skills --no-prompt-templates --no-context-files --session '$session_file'; rc=\$?; printf '\nPI_EXIT=%s\n' \"\$rc\"; sleep 30" wait_for_text "$restarted_snapshot" "CALM_WORKING_E2E_RESPONSE" \ || fail "Pi did not restore the persisted session after restart" assert_not_contains "$(cat "$restarted_snapshot")" "CALM_E2E_OUTPUT" "restart/resume reset Calm and restored a tool row" diff --git a/tests/fm-captain-hold-lifecycle.test.sh b/tests/fm-captain-hold-lifecycle.test.sh index 5ecd51fe9ba..2864a3e30eb 100755 --- a/tests/fm-captain-hold-lifecycle.test.sh +++ b/tests/fm-captain-hold-lifecycle.test.sh @@ -88,6 +88,15 @@ run_captain() { # FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-captain-hold.sh" "$@" } +# Completes 's captain-call inventory through a separate held task, because +# the origin task is never accepted as its own inventory entry. +complete_through_sibling() { # + local home=$1 id=$2 + run_captain "$home" hold "$id-call" --title "Sibling captain call for $id" \ + --reason "captain must decide the sibling call" --repo sample --origin "$id" >/dev/null \ + && run_captain "$home" complete "$id" "$id-call" +} + request_reconciles() { # ... local home=$1 source_id=$2 id shift 2 @@ -357,7 +366,7 @@ case "${1:-}" in show) case "${2:-}" in @KNOWN@) ;; - *) printf 'error: no task %s in this backlog\n' "${2:-}" >&2; exit 1 ;; + *) printf 'error: no task %s in this backlog\ncode: NOT_FOUND\n' "${2:-}" >&2; exit 1 ;; esac printf '%s\n' 'task:' printf ' id: %s\n' "$2" @@ -2575,7 +2584,7 @@ test_teardown_never_closes_a_captain_held_task() { run_captain "$home" hold "$id" \ --reason "captain must choose inline or by-reference attachments" >/dev/null \ || fail "could not hold the originating work item for the captain" - run_captain "$home" complete "$id" "$id" >/dev/null \ + complete_through_sibling "$home" "$id" >/dev/null \ || fail "completion gate failed with the origin as its own captain call" run_teardown "$home" "$id" > "$home/teardown.out" 2> "$home/teardown.err" \ @@ -2666,7 +2675,7 @@ test_retained_row_artifacts_survive_captain_answers() { > "$home/data/$retained_id/report.md" run_captain "$home" hold "$retained_id" --reason "captain must choose the report follow-up" \ >/dev/null || fail "could not hold the retained report" - run_captain "$home" complete "$retained_id" "$retained_id" >/dev/null \ + complete_through_sibling "$home" "$retained_id" >/dev/null \ || fail "completion gate failed for the retained report" run_teardown "$home" "$retained_id" > "$home/retained-teardown.out" \ 2> "$home/report-teardown.err" \ @@ -2687,7 +2696,7 @@ test_retained_row_artifacts_survive_captain_answers() { run_captain "$home" hold "$precedence_id" \ --reason "captain must choose the report follow-up" >/dev/null \ || fail "could not hold the report precedence fixture" - run_captain "$home" complete "$precedence_id" "$precedence_id" >/dev/null \ + complete_through_sibling "$home" "$precedence_id" >/dev/null \ || fail "completion gate failed for the report precedence fixture" run_teardown "$home" "$precedence_id" > "$home/precedence-teardown.out" \ 2> "$home/precedence-teardown.err" \ @@ -2815,7 +2824,7 @@ test_retained_row_artifacts_survive_captain_answers() { printf '# Released report\n' > "$home/data/$released_id/report.md" run_captain "$home" hold "$released_id" --reason "captain report release pending" \ >/dev/null || fail "could not hold the released report" - run_captain "$home" complete "$released_id" "$released_id" >/dev/null \ + complete_through_sibling "$home" "$released_id" >/dev/null \ || fail "completion gate failed for the released report" printf 'Release the completed report.\n' > "$home/released-answer.txt" run_captain "$home" answer "$released_id" --release \ @@ -2920,7 +2929,7 @@ test_interrupted_cleanup_keeps_the_captain_call_recoverable() { printf '# Failed cleanup\n\nThe captain call remains open.\n' > "$home/data/$id/report.md" run_captain "$home" hold "$id" --reason "captain must choose after cleanup retry" >/dev/null \ || fail "could not hold the cleanup-failure fixture" - run_captain "$home" complete "$id" "$id" >/dev/null \ + complete_through_sibling "$home" "$id" >/dev/null \ || fail "completion gate failed for the cleanup-failure fixture" cat > "$home/fakebin/treehouse" <<'SH' #!/usr/bin/env bash @@ -2979,7 +2988,7 @@ test_answer_before_cleanup_replay_preserves_the_retained_report() { printf '# Interrupted cleanup\n\nThe captain call remains open.\n' > "$home/data/$id/report.md" run_captain "$home" hold "$id" --reason "captain must choose after interrupted cleanup" \ >/dev/null || fail "could not hold the answer-before-replay fixture" - run_captain "$home" complete "$id" "$id" >/dev/null \ + complete_through_sibling "$home" "$id" >/dev/null \ || fail "completion gate failed for the answer-before-replay fixture" cat > "$home/fakebin/treehouse" <<'SH' #!/usr/bin/env bash @@ -3017,6 +3026,63 @@ SH pass "an answer before cleanup replay preserves the retained report" } +test_answer_before_cleanup_replay_notes_a_retained_gerrit_change() { + local home id repo wt rc show real_tasks_axi gerrit_url=https://gerrit.example.com/c/project/+/12345 + home=$(make_home answer-before-replay-gerrit) + id=sample-answer-before-replay-gerrit + repo="$home/projects/sample" + wt="$home/projects/$id" + fm_git_worktree "$repo" "$wt" fm/answer-before-replay-gerrit + tasks_in "$home" add "$id" "Ship the held Gerrit change" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the held Gerrit answer fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "pr=$gerrit_url" "spawn_gen=fixture-$id" + printf 'done: change landed\n' > "$home/state/$id.status" + run_captain "$home" hold "$id" --reason "captain must choose the follow-up" >/dev/null \ + || fail "could not hold the landed Gerrit task for the captain" + real_tasks_axi=$(command -v tasks-axi) + cat > "$home/fakebin/tasks-axi" < "$home/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +exit 1 +SH + chmod +x "$home/fakebin/treehouse" + + set +e + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" --force \ + > "$home/teardown.out" 2> "$home/teardown.err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "cleanup succeeded despite the failed worktree return" + assert_present "$home/state/$id.backlog-close" \ + "the interrupted cleanup lost its retained-artifact record" + + printf 'Proceed with the landed change.\n' > "$home/answer.txt" + run_captain "$home" answer "$id" --decision-file "$home/answer.txt" >/dev/null \ + || fail "the captain could not answer a Gerrit task before cleanup replay" + show=$(tasks_in "$home" show "$id" --full) || fail "the answered Gerrit row is gone" + assert_contains "$show" "state: done" "the answer did not close the Gerrit row" + assert_contains "$show" "Gerrit change $gerrit_url" \ + "the answer dropped the retained Gerrit change URL" + pass "an answer before cleanup replay notes the retained Gerrit change" +} + test_unusable_pending_close_record_names_its_reason() { local home id wt rc err marker home=$(make_home unusable-pending-close-reason) @@ -3033,7 +3099,7 @@ test_unusable_pending_close_record_names_its_reason() { printf '# Unusable pending close\n\nThe captain call remains open.\n' > "$home/data/$id/report.md" run_captain "$home" hold "$id" --reason "captain must choose after interrupted cleanup" \ >/dev/null || fail "could not hold the unusable pending-close fixture" - run_captain "$home" complete "$id" "$id" >/dev/null \ + complete_through_sibling "$home" "$id" >/dev/null \ || fail "completion gate failed for the unusable pending-close fixture" cat > "$home/fakebin/treehouse" <<'SH' #!/usr/bin/env bash @@ -3097,7 +3163,12 @@ EOF || fail "could not hold the relocated answer-before-replay fixture" PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$home/config" \ - "$ROOT/bin/fm-captain-hold.sh" complete "$id" "$id" >/dev/null \ + "$ROOT/bin/fm-captain-hold.sh" hold "$id-call" --title "Sibling captain call" \ + --reason "captain must decide the sibling call" --repo sample --origin "$id" >/dev/null \ + || fail "could not hold the sibling captain call" + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-captain-hold.sh" complete "$id" "$id-call" >/dev/null \ || fail "completion gate failed for the relocated answer-before-replay fixture" cat > "$home/fakebin/treehouse" <<'SH' #!/usr/bin/env bash @@ -3174,7 +3245,12 @@ EOF || fail "could not hold the relocated work item" PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$home/config" \ - "$ROOT/bin/fm-captain-hold.sh" complete "$id" "$id" >/dev/null \ + "$ROOT/bin/fm-captain-hold.sh" hold "$id-call" --title "Sibling captain call" \ + --reason "captain must decide the sibling call" --repo sample --origin "$id" >/dev/null \ + || fail "could not hold the sibling captain call" + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-captain-hold.sh" complete "$id" "$id-call" >/dev/null \ || fail "completion gate failed for the relocated captain hold" PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ @@ -3195,6 +3271,52 @@ EOF pass "cleanup retains captain calls in the configured backlog" } +test_teardown_retains_a_gerrit_captain_call_with_its_change_url() { + local home id repo wt show real_tasks_axi gerrit_url=https://gerrit.example.com/c/project/+/12345 + home=$(make_home teardown-held-gerrit) + id=sample-held-gerrit + repo="$home/projects/sample" + wt="$home/projects/$id" + fm_git_worktree "$repo" "$wt" fm/held-gerrit + tasks_in "$home" add "$id" "Ship the held Gerrit change" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the held Gerrit fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "pr=$gerrit_url" "spawn_gen=fixture-$id" + printf 'done: change landed\n' > "$home/state/$id.status" + run_captain "$home" hold "$id" --reason "captain must choose the follow-up" >/dev/null \ + || fail "could not hold the landed Gerrit task for the captain" + # Pin the refusal tasks-axi applies to a --pr link that is not a canonical + # GitHub pull request, so this case keeps reproducing whatever the installed + # release accepts. + real_tasks_axi=$(command -v tasks-axi) + cat > "$home/fakebin/tasks-axi" < "$home/teardown.out" 2> "$home/teardown.err" \ + || fail "cleanup of a captain-held Gerrit task failed: $(cat "$home/teardown.err")" + show=$(tasks_in "$home" show "$id" --full) || fail "the captain-held Gerrit row is gone after cleanup" + assert_contains "$show" "state: queued" "the held Gerrit row still reads as worked on" + assert_contains "$show" "hold_kind: captain" "cleanup dropped the captain hold" + assert_contains "$show" "Deliverable of the finished work: Gerrit change $gerrit_url" \ + "the Gerrit change URL was not recorded on the still-open row" + assert_absent "$home/state/$id.backlog-close" \ + "successful cleanup left its pending transition record behind" + pass "cleanup keeps a captain-held Gerrit task open and records its change URL" +} + test_merge_approval_releases_before_zero_done_retention() { local home id archive repo wt pr show home=$(make_home zero-done-retention) @@ -3958,7 +4080,7 @@ PM > "$home/data/$scout/report.md" run_captain "$home" hold "$scout" --reason "captain must choose" >/dev/null \ || fail "could not hold the investigation for the captain" - run_captain "$home" complete "$scout" "$scout" >/dev/null \ + complete_through_sibling "$home" "$scout" >/dev/null \ || fail "the completion gate failed with the origin as its own captain call" PERL5LIB="$shim" PERL5OPT=-MFmNoNonrefDefault \ run_teardown "$home" "$scout" > "$home/nonref.out" 2> "$home/nonref.err" \ @@ -3996,7 +4118,7 @@ retain_row_with_body() { # || fail "could not give $id a body carrying non-ASCII characters" run_captain "$home" hold "$id" --reason "captain must choose" >/dev/null \ || fail "could not hold $id for the captain" - run_captain "$home" complete "$id" "$id" >/dev/null \ + complete_through_sibling "$home" "$id" >/dev/null \ || fail "the completion gate failed for $id" run_teardown "$home" "$id" > "$home/$id.out" 2> "$home/$id.err" \ || fail "cleanup of captain-held $id failed: $(cat "$home/$id.err")" @@ -4034,6 +4156,485 @@ test_retained_body_keeps_its_utf8_bytes() { pass "cleanup preserves every byte of a retained body's non-ASCII characters" } +# A refused hold must never read as a recorded one. The gate used to accept the +# origin as its own inventory whenever the origin row looked durable, so a hold +# that failed just before `complete ` left a satisfied gate +# with no captain call recorded. +test_origin_is_never_its_own_inventory_entry() { + local home id + home=$(make_home origin-self-inventory) + id=sample-self-review + mkdir -p "$home/data/$id" + tasks_in "$home" add "$id" "Investigate sample self review" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the investigation fixture" + write_origin_meta "$home" "$id" + printf 'done: report complete\n' > "$home/state/$id.status" + if run_captain "$home" hold "$id" --reason "" >/dev/null 2> "$home/hold.err"; then + fail "hold accepted an empty reason" + fi + if run_captain "$home" complete "$id" "$id" > "$home/self.out" 2> "$home/self.err"; then + fail "complete accepted the origin as its own inventory after a failed hold" + fi + assert_grep "cannot be its own captain-call inventory entry" "$home/self.err" \ + "the refusal does not say why the origin was rejected" + assert_no_grep "decisions_reviewed=1" "$home/state/$id.meta" \ + "the refused completion recorded an inventory attestation" + + # Holding the origin row itself must not let it vouch for itself either. + run_captain "$home" hold "$id" --reason "captain must choose" >/dev/null \ + || fail "could not hold the origin row" + if run_captain "$home" complete "$id" "$id" > "$home/held.out" 2> "$home/held.err"; then + fail "complete accepted a held origin row as its own inventory" + fi + pass "complete refuses the origin as its own captain-call inventory" +} + +# `hold --origin` records which origin a call was held for, and `complete` +# refuses a task held for a different origin. A hold recorded before that +# record existed, or without --origin, still verifies and is flagged. +test_complete_refuses_an_entry_held_for_another_origin() { + local home id other o out + home=$(make_home origin-mismatch) + id=sample-first-review + other=sample-second-review + for o in "$id" "$other"; do + mkdir -p "$home/data/$o" + tasks_in "$home" add "$o" "Investigate $o" --kind scout --repo sample --start >/dev/null \ + || fail "could not create the $o fixture" + write_origin_meta "$home" "$o" + printf 'done: report complete\n' > "$home/state/$o.status" + done + run_captain "$home" hold sample-other-call --title "Call for the second review" \ + --reason "captain must decide" --repo sample --origin "$other" >/dev/null \ + || fail "could not hold the call recorded for the second review" + if run_captain "$home" complete "$id" sample-other-call > "$home/mismatch.out" 2> "$home/mismatch.err"; then + fail "complete accepted an entry held for a different origin" + fi + assert_grep "was held for origin $other, not $id" "$home/mismatch.err" \ + "the refusal does not name both origins" + assert_no_grep "decisions_reviewed=1" "$home/state/$id.meta" \ + "the refused completion recorded an inventory attestation" + + run_captain "$home" hold sample-own-call --title "Call for the first review" \ + --reason "captain must decide" --repo sample --origin "$id" >/dev/null \ + || fail "could not hold the call recorded for the first review" + out=$(run_captain "$home" complete "$id" sample-own-call) \ + || fail "complete refused an entry held for its own origin" + assert_not_contains "$out" "no recorded origin" \ + "an entry with a recorded origin was flagged as unrecorded" + + tasks_in "$home" add sample-old-call "Call held before origins were recorded" --kind captain --repo sample >/dev/null \ + || fail "could not create the older call" + tasks_in "$home" hold sample-old-call --reason "captain must decide" --kind captain >/dev/null \ + || fail "could not hold the older call" + out=$(run_captain "$home" complete "$other" sample-old-call) \ + || fail "complete refused an older hold with no recorded origin" + assert_contains "$out" "no recorded origin on: sample-old-call" \ + "an older hold with no recorded origin was not flagged" + pass "complete refuses an entry held for another origin and flags one with none recorded" +} + +test_hold_origins_precede_backend_holds() { + local home phase timing failure id shown origin until_args=() + for phase in new active released; do + for timing in plain dated; do + home=$(make_home "origin-failure-$phase-$timing") + id=sample-call + for origin in origin-a origin-b; do + tasks_in "$home" add "$origin" "Review $origin" --kind scout --repo sample >/dev/null \ + || fail "could not create $origin" + write_origin_meta "$home" "$origin" + done + if [ "$phase" != new ]; then + run_captain "$home" hold "$id" --title "Separate call" --reason "Choose for A" \ + --origin origin-a >/dev/null || fail "could not establish the original association" + fi + if [ "$phase" = released ]; then + printf 'Release this work.\n' > "$home/answer.txt" + run_captain "$home" answer "$id" --release --decision-file "$home/answer.txt" >/dev/null \ + || fail "could not release the original hold" + fi + cat > "$home/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = show ] && [ "${2:-}" = origin-b ] && [ -f "$FM_HOME/fail-lookup" ]; then + : > "$FM_HOME/lookup-refused" + printf 'error: origin read failed\ncode: READ_FAILED\n' >&2 + exit 2 +fi +if [ "${1:-}" = update ] && [ -f "$FM_HOME/fail-write" ]; then + previous='' + for arg in "$@"; do + if [ "$previous" = --body-file ] && grep -qx 'Captain hold origin: origin-b' "$arg"; then + : > "$FM_HOME/write-refused" + exit 9 + fi + previous=$arg + done +fi +if [ "${1:-}" = hold ] && [ "${2:-}" != --help ]; then + "$REAL_TASKS_AXI" show "$2" --full > "$FM_HOME/before-backend-hold" || exit $? + if [ -f "$FM_HOME/fail-hold" ]; then + : > "$FM_HOME/hold-refused" + exit 9 + fi +fi +exec "$REAL_TASKS_AXI" "$@" +SH + chmod +x "$home/fakebin/tasks-axi" + until_args=() + [ "$timing" != dated ] || until_args=(--until 2099-01-01) + for failure in lookup write hold; do + : > "$home/fail-$failure" + if run_captain "$home" hold "$id" --title "Separate call" --reason "Choose for B" \ + --origin origin-b ${until_args[@]+"${until_args[@]}"} > "$home/hold.out" 2> "$home/hold.err"; then + fail "$phase $timing hold succeeded despite an origin $failure failure" + fi + assert_present "$home/$failure-refused" "the failure did not reach the origin $failure" + if [ "$failure" = hold ]; then + assert_present "$home/before-backend-hold" "$phase $timing failure never reached the backend hold" + assert_grep 'Captain hold origin: origin-b' "$home/before-backend-hold" \ + "the failed backend hold did not see the new association" + rm "$home/before-backend-hold" + else + assert_absent "$home/before-backend-hold" "$phase $timing origin $failure failure reached the backend hold" + fi + shown=$(tasks_in "$home" show "$id" --full) + assert_not_contains "$shown" 'Captain hold origin: origin-b' \ + "$phase $timing origin $failure failure published the new association" + if [ "$phase" = active ]; then + assert_contains "$shown" 'held: yes' "an origin $failure failure lifted an existing hold" + else + assert_contains "$shown" 'held: no' "$phase $timing origin $failure failure left the task held" + fi + if [ "$phase" != new ]; then + assert_contains "$shown" 'Captain hold origin: origin-a' \ + "$phase $timing origin $failure failure lost the original association" + fi + rm "$home/fail-$failure" + if [ "$phase" = new ] && run_captain "$home" complete origin-a "$id" \ + > "$home/unrelated.out" 2> "$home/unrelated.err"; then + fail "$timing origin $failure failure satisfied an unrelated inventory" + fi + if run_captain "$home" complete origin-b "$id" > "$home/complete.out" 2> "$home/complete.err"; then + fail "$phase $timing origin $failure failure satisfied completion for B" + fi + printf 'decisions_reviewed=1\ndecision_keys=%s\n' "$id" >> "$home/state/origin-b.meta" + if run_captain "$home" verify origin-b > "$home/verify.out" 2> "$home/verify.err"; then + fail "$phase $timing origin $failure failure verified an inventory for B" + fi + if [ "$phase" != new ]; then + run_captain "$home" complete origin-a "$id" >/dev/null \ + || fail "$phase $timing origin $failure failure invalidated completion for A" + run_captain "$home" verify origin-a >/dev/null \ + || fail "$phase $timing origin $failure failure invalidated verification for A" + fi + done + run_captain "$home" hold "$id" --reason "Choose for B" --origin origin-b \ + ${until_args[@]+"${until_args[@]}"} >/dev/null || fail "$phase $timing successful retry failed" + assert_present "$home/before-backend-hold" "the successful retry did not reach the backend hold" + shown=$(cat "$home/before-backend-hold") + assert_contains "$shown" 'Captain hold origin: origin-b' "the backend hold ran before the new origin was recorded" + assert_not_contains "$shown" 'Captain hold origin: origin-a' "the backend hold ran with the old association" + shown=$(tasks_in "$home" show "$id" --full) + assert_contains "$shown" 'held: yes' "the successful retry did not hold the task" + assert_contains "$shown" 'Captain hold origin: origin-b' "a successful hold lost its association" + assert_not_contains "$shown" 'Captain hold origin: origin-a' "a successful hold retained the old association" + run_captain "$home" complete origin-b "$id" >/dev/null \ + || fail "a successful hold could not complete B" + run_captain "$home" verify origin-b >/dev/null || fail "a successful hold could not verify B" + if run_captain "$home" complete origin-a "$id" >/dev/null 2> "$home/old-origin.err"; then + fail "a successful reassociation still certified A" + fi + done + done + pass "new, active, and released holds require the origin first with and without deferral" +} + +test_historical_self_inventory_has_workable_repair() { + local home origin=sample-review keep=retained-call replacement=repair-call meta before out + home=$(make_home historical-self-inventory) + run_captain "$home" hold "$origin" --title "Old review call" --reason "Choose" >/dev/null \ + || fail "could not create the historical origin" + write_origin_meta "$home" "$origin" + for out in "$keep" "$replacement"; do + run_captain "$home" hold "$out" --title "Call $out" --reason "Choose" --origin "$origin" >/dev/null \ + || fail "could not create $out" + done + meta="$home/state/$origin.meta" + printf 'decisions_reviewed=1\ndecision_keys=%s,%s\n' "$origin" "$keep" >> "$meta" + before=$(cat "$meta") + for out in "$replacement" --none; do + if run_captain "$home" complete "$origin" "$out" > "$home/complete.out" 2> "$home/complete.err"; then + fail "complete accepted the historical self-inventory" + fi + assert_grep "historical decision_keys in $meta still contains $origin" "$home/complete.err" \ + "the historical refusal did not identify the persisted entry" + assert_grep 'replace only' "$home/complete.err" "the refusal omitted the repair instruction" + done + if run_captain "$home" verify "$origin" > "$home/verify.out" 2> "$home/verify.err"; then + fail "verify accepted the historical self-inventory" + fi + assert_grep "historical decision_keys in $meta still contains $origin" "$home/verify.err" \ + "verify omitted the historical repair instruction" + assert_equals "$before" "$(cat "$meta")" "refusing a historical inventory changed it" + sed "s/^decision_keys=$origin,$keep$/decision_keys=$replacement,$keep/" "$meta" > "$meta.repaired" + mv "$meta.repaired" "$meta" + run_captain "$home" complete "$origin" "$replacement" >/dev/null \ + || fail "the documented historical repair did not allow completion" + run_captain "$home" verify "$origin" >/dev/null || fail "the repaired inventory did not verify" + assert_equals "decision_keys=$replacement,$keep" "$(grep '^decision_keys=' "$meta" | tail -1)" \ + "repair lost a sibling inventory entry" + if run_captain "$home" complete "$origin" "$origin" >/dev/null 2> "$home/self.err"; then + fail "repair allowed a new self-inventory" + fi + pass "historical self-inventories name a workable repair that preserves sibling entries" +} + +test_inventory_compares_backend_identities() { + local home origin entry shown before + home=$(make_home backend-identities) + run_captain "$home" hold fm-o --title "Origin" --reason "Choose" >/dev/null \ + || fail "could not create the canonical origin" + tasks_in "$home" add fm-other "Other origin" --kind scout --repo sample >/dev/null \ + || fail "could not create the other origin" + cat > "$home/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = show ] && [ "${2:-}" = o ] && [ -f "$FM_HOME/fail-identity" ]; then + printf 'error: origin read failed\ncode: READ_FAILED\n' >&2 + exit 2 +fi +if [ "$#" -ge 2 ]; then + case "$2" in + o|call|other) set -- "$1" "fm-$2" "${@:3}" ;; + esac +fi +exec "$REAL_TASKS_AXI" "$@" +SH + chmod +x "$home/fakebin/tasks-axi" + for origin in fm-o o; do + for entry in fm-o o; do + write_origin_meta "$home" "$origin" + if run_captain "$home" complete "$origin" "$entry" > "$home/self.out" 2> "$home/self.err"; then + fail "complete accepted aliased self-inventory $origin/$entry" + fi + assert_grep 'cannot be its own captain-call inventory entry' "$home/self.err" \ + "the alias refusal did not identify self-inventory" + printf 'decisions_reviewed=1\ndecision_keys=%s\n' "$entry" >> "$home/state/$origin.meta" + if run_captain "$home" verify "$origin" > "$home/verify.out" 2> "$home/verify.err"; then + fail "verify accepted aliased self-inventory $origin/$entry" + fi + assert_grep 'historical decision_keys' "$home/verify.err" "the alias repair diagnostic was missing" + done + write_origin_meta "$home" "$origin" + done + run_captain "$home" hold fm-call --title "Separate call" --reason "Choose" --origin o >/dev/null \ + || fail "could not hold a call using the origin alias" + shown=$(tasks_in "$home" show fm-call --full) + assert_contains "$shown" 'Captain hold origin: fm-o' "hold did not store the backend origin identity" + printf '%s\n' "$shown" | sed -n 's/^ body: //p' | jq -r . \ + | sed 's/^Captain hold origin: fm-o$/Captain hold origin: o/' > "$home/legacy-origin.txt" + tasks_in "$home" update fm-call --body-file "$home/legacy-origin.txt" >/dev/null \ + || fail "could not create a legacy stored alias" + for origin in fm-o o; do + for entry in fm-call call; do + run_captain "$home" complete "$origin" "$entry" >/dev/null \ + || fail "complete refused equivalent origin spellings for $origin/$entry" + run_captain "$home" verify "$origin" >/dev/null \ + || fail "verify refused equivalent origin spellings for $origin/$entry" + done + done + for origin in fm-other other; do + write_origin_meta "$home" "$origin" + if run_captain "$home" complete "$origin" call > "$home/other.out" 2> "$home/other.err"; then + fail "complete accepted another origin through $origin" + fi + assert_grep "was held for origin o, not $origin" "$home/other.err" "the alias mismatch was not identified" + printf 'decisions_reviewed=1\ndecision_keys=call\n' >> "$home/state/$origin.meta" + if run_captain "$home" verify "$origin" >/dev/null 2> "$home/other-verify.err"; then + fail "verify accepted another origin through $origin" + fi + done + : > "$home/fail-identity" + before=$(cat "$home/state/o.meta") + if run_captain "$home" complete o fm-call >/dev/null 2> "$home/read.err"; then + fail "an unreadable backend identity was treated as an absent origin" + fi + assert_grep 'could not resolve the backend identity of o' "$home/read.err" "the identity read failure was hidden" + assert_equals "$before" "$(cat "$home/state/o.meta")" "a failed identity read changed the inventory" + rm "$home/fail-identity" + write_origin_meta "$home" report-only + run_captain "$home" hold report-call --title "Report call" --reason "Choose" --origin report-only >/dev/null \ + || fail "an origin with metadata but no backlog row could not record a call" + run_captain "$home" complete report-only report-call >/dev/null \ + || fail "an origin with metadata but no backlog row could not complete" + run_captain "$home" verify report-only >/dev/null \ + || fail "an origin with metadata but no backlog row could not verify" + pass "completion and verification compare backend identities for entries and current or stored origins" +} + +# tasks-axi refuses parentheses and line breaks in a hold reason and stores the +# rest on one markdown line. The reason is encoded where it is written and +# decoded wherever it is shown, so prose with every awkward character survives. +test_hold_reason_round_trips_awkward_characters() { + local home id reason stored json shown start verb fields out raw rc raw_rc mode + local title legacy body quoted_reason quoted_title quoted_legacy expected_reason until_args=() + local malformed index=0 malformed_reasons=( + 'fm-hold-v1:/w==' 'fm-hold-v1:bm9ydGg=$' 'fm-hold-v1:bm9ydGg' 'fm-hold-v1:Zh==' + ) + home=$(make_home reason-round-trip) + title='Investigate literal %28, "fm-hold-v1:bm9ydGg="' + legacy='Visit https://example.test/%28literal%29 and %0A; fm-hold-v1:bm9ydGg=' + body=$'fm-hold-v1:bm9ydGg=\n hold_reason: "%28"\n' + quoted_title=$(jq -cn --arg value "$title" '$value') + quoted_legacy=$(jq -cn --arg value "$legacy" '$value') + tasks_in "$home" add sample-legacy-call "$title" --kind captain --repo sample >/dev/null \ + || fail "could not create the legacy call" + tasks_in "$home" hold sample-legacy-call --reason "$legacy" --kind captain >/dev/null \ + || fail "could not hold the legacy call" + printf '%s' "$body" > "$home/legacy-body.txt" + tasks_in "$home" update sample-legacy-call --body-file "$home/legacy-body.txt" >/dev/null \ + || fail "could not write the legacy body" + + # Historical literal reasons are persisted input, not encoder output. + for malformed in "${malformed_reasons[@]}"; do + id="sample-malformed-$index" + index=$((index + 1)) + tasks_in "$home" add "$id" "Historical reason $index" --kind captain --repo sample >/dev/null \ + || fail "could not create $id" + tasks_in "$home" hold "$id" --reason "$malformed" --kind captain >/dev/null \ + || fail "could not store the historical literal reason" + for verb in show view; do + out=$(FM_HOME="$home" "$ROOT/bin/fm-tasks-axi.sh" "$verb" "$id" --full) \ + || fail "public $verb failed on historical literal $malformed" + raw=$(tasks_in "$home" "$verb" "$id" --full) + assert_equals "$raw" "$out" "public $verb changed historical literal $malformed" + done + done + + for id in sample-reason-call sample-dated-call; do + until_args=() + reason=$' Pick route (north); say "yes" or \'no\' - 100% sure %28x%29, café\t\\slash\r\nSecond line\n\n' + if [ "$id" = sample-dated-call ]; then + until_args=(--until 2099-01-01) + reason='fm-hold-v1:bm9ydGg=' + fi + quoted_reason=$(jq -cn --arg value "$reason" '$value') + run_captain "$home" hold "$id" --title "$title" --reason "$reason" \ + --repo sample ${until_args[@]+"${until_args[@]}"} >/dev/null \ + || fail "hold refused the reason for $id" + stored=$(grep "^- \[ \] $id " "$home/data/backlog.md") \ + || fail "the held row is not on one backlog line" + assert_contains "$stored" "(hold: fm-hold-v1:" "the persisted reason has no encoding marker" + assert_contains "$stored" "(hold-kind: captain)" "the reason broke the hold-kind tag" + + for verb in show view; do + out=$(FM_HOME="$home" "$ROOT/bin/fm-tasks-axi.sh" "$verb" "$id") \ + || fail "public $verb failed for $id" + shown=$(printf '%s\n' "$out" | sed -n 's/^ hold_reason: //p') + printf '%s\n' "$shown" | jq -e --arg reason "$reason" '. == $reason' >/dev/null \ + || fail "public $verb changed the reason for $id" + assert_contains "$out" " title: $quoted_title" "public $verb changed the title" + done + for fields in hold_reason,body body,hold_reason,hold_until; do + out=$(FM_HOME="$home" "$ROOT/bin/fm-tasks-axi.sh" list --fields "$fields") \ + || fail "public list failed with $fields" + assert_contains "$out" "$quoted_reason" "public list changed the reason with $fields" + assert_contains "$out" "$quoted_title" "public list changed the title with $fields" + raw=$(tasks_in "$home" list --fields "$fields" | grep '^ sample-legacy-call,') + shown=$(printf '%s\n' "$out" | grep '^ sample-legacy-call,') + assert_equals "$raw" "$shown" "public list changed legacy or unrelated fields" + for malformed in "${malformed_reasons[@]}"; do + assert_contains "$out" "$malformed" "public list changed historical literal $malformed" + done + done + json=$(PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$home/data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-fleet-snapshot.sh" --json) || fail "fleet snapshot failed" + printf '%s' "$json" | jq -e --arg id "$id" --arg reason "$reason" --arg title "$title" \ + '.backlog.records[] | select(.id == $id) | .hold_reason == $reason and .title == $title' >/dev/null \ + || fail "fleet changed the reason or title for $id" + printf '%s' "$json" | jq -e --arg reason "$legacy" --arg title "$title" \ + '.backlog.records[] | select(.id == "sample-legacy-call") | + .hold_reason == $reason and .title == $title and .body_lines[0] == "fm-hold-v1:bm9ydGg="' >/dev/null \ + || fail "fleet changed legacy or unrelated fields" + for malformed in "${malformed_reasons[@]}"; do + printf '%s' "$json" | jq -e --arg reason "$malformed" \ + 'any(.backlog.records[]; .hold_reason == $reason)' >/dev/null \ + || fail "fleet changed historical literal $malformed" + done + done + + out=$(FM_HOME="$home" "$ROOT/bin/fm-tasks-axi.sh" show sample-legacy-call --full) + raw=$(tasks_in "$home" show sample-legacy-call --full) + assert_equals "$raw" "$out" "public show changed legacy or unrelated fields" + out=$(FM_HOME="$home" "$ROOT/bin/fm-tasks-axi.sh" list) + raw=$(tasks_in "$home" list) + assert_equals "$raw" "$out" "public list changed output with no reason column" + for verb in show list; do + out=$(FM_HOME="$home" "$ROOT/bin/fm-tasks-axi.sh" "$verb" --help) + raw=$(tasks_in "$home" "$verb" --help) + assert_equals "$raw" "$out" "public $verb changed help output" + done + out=$(FM_HOME="$home" "$ROOT/bin/fm-tasks-axi.sh" show nonexistent-call 2>&1) + rc=$? + raw=$(tasks_in "$home" show nonexistent-call 2>&1) + raw_rc=$? + [ "$raw_rc" -ne 0 ] || fail "the missing-task fixture unexpectedly exists" + expect_code "$raw_rc" "$rc" "public show missing task" + assert_equals "$raw" "$out" "public show changed a read error" + + expected_reason=$' Pick route (north); say "yes" or \'no\' - 100% sure %28x%29, café\t\\slash\r\nSecond line\n\n' + quoted_reason=$(jq -cn --arg value "$expected_reason" '$value') + for mode in tool manual fallback; do + case "$mode" in + manual) printf 'manual\n' > "$home/config/backlog-backend" ;; + fallback) + rm "$home/config/backlog-backend" + cat > "$home/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = list ]; then + printf 'read failed: literal %%28 and fm-hold-v1:bm9ydGg=\n' >&2 + exit 1 +fi +exec "$REAL_TASKS_AXI" "$@" +SH + chmod +x "$home/fakebin/tasks-axi" + ;; + esac + start=$(PATH="$home/fakebin:$PATH" REAL_TASKS_AXI="$TASKS_AXI_BIN" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" FM_CONFIG_OVERRIDE="$home/config" \ + FM_BOOTSTRAP_NETWORK=skip "$ROOT/bin/fm-session-start.sh" 2>&1 || true) + assert_contains "$start" "$quoted_reason" "startup $mode changed the encoded reason" + assert_contains "$start" 'Investigate literal %28' "startup $mode changed the title" + assert_contains "$start" "$legacy" "startup $mode changed the legacy reason" + assert_contains "$start" '"fm-hold-v1:bm9ydGg="' "startup $mode decoded a reason twice" + for malformed in "${malformed_reasons[@]}"; do + assert_contains "$start" "$malformed" "startup $mode changed historical literal $malformed" + done + if [ "$mode" = fallback ]; then + assert_contains "$start" 'read failed: literal %28 and fm-hold-v1:bm9ydGg=' \ + "startup changed unrelated error text" + fi + done + rm "$home/fakebin/tasks-axi" + out=$(PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$home/data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-afk-return.sh" check 2>&1 || true) + assert_contains "$out" "$quoted_reason" "return brief changed the encoded reason" + assert_contains "$out" "$quoted_title" "return brief changed the title" + assert_contains "$out" "$quoted_legacy" "return brief changed the legacy reason" + for malformed in "${malformed_reasons[@]}"; do + assert_contains "$out" "$malformed" "return brief changed historical literal $malformed" + done + pass "marked hold reasons round-trip through public reads, fleet, startup, and return without changing other fields" +} + +test_hold_reason_round_trips_awkward_characters +test_hold_origins_precede_backend_holds +test_historical_self_inventory_has_workable_repair +test_inventory_compares_backend_identities +test_origin_is_never_its_own_inventory_entry +test_complete_refuses_an_entry_held_for_another_origin test_uninventoried_report_decision_refuses_completion test_hold_decodes_a_bare_scalar_body_without_the_nonref_default test_retained_body_keeps_its_utf8_bytes @@ -4066,9 +4667,11 @@ test_teardown_never_closes_a_captain_held_task test_retained_row_artifacts_survive_captain_answers test_interrupted_cleanup_keeps_the_captain_call_recoverable test_answer_before_cleanup_replay_preserves_the_retained_report +test_answer_before_cleanup_replay_notes_a_retained_gerrit_change test_unusable_pending_close_record_names_its_reason test_relocated_report_does_not_wedge_an_answer_before_replay test_teardown_retains_captain_calls_in_a_relocated_backlog +test_teardown_retains_a_gerrit_captain_call_with_its_change_url test_merge_approval_releases_before_zero_done_retention test_pr_merge_entrypoint_refuses_a_captain_held_task test_local_merge_entrypoint_refuses_a_captain_held_task diff --git a/tests/fm-claude-stop-autoarm.test.sh b/tests/fm-claude-stop-autoarm.test.sh index 1e97c2c41ab..309e3311d02 100755 --- a/tests/fm-claude-stop-autoarm.test.sh +++ b/tests/fm-claude-stop-autoarm.test.sh @@ -1418,6 +1418,24 @@ write_host_fixture() { stood-down) printf "printf 'supervision-host stood down: this session no longer owns supervision\\n'\n" ;; + lost-handback|lost-announced-handback) + local marker=pending + [ "$kind" = lost-handback ] || marker=announced + printf "printf '%s:handling:fixture-generation\\\\n' > \"\$FM_HOME/state/.watcher-down\"\\n" "$marker" + cat <<'SH' +printf 'signal: fixture.status\n' +printf 'supervision-host: branch-outcome: fixture\n' +printf 'supervision-host: watcher downtime could not be restored for the main hand-back\n' +exit 1 +SH + ;; + benign-refusal) + cat <<'SH' +printf 'acked:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" +printf 'signal: fixture.status\n' +printf 'supervision-host: branch-outcome: fixture\n' +SH + ;; handed-back-many) cat <<'SH' printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" @@ -1577,6 +1595,56 @@ test_host_stand_down_is_silent() { pass "auto-arm: a host that stood down closes silently without a retry" } +# Main already drained and acknowledged the wake, so the rewake is refused on a +# marker that is no longer downtime: that refusal stays silent and opens no +# failure episode. +test_host_benign_rewake_refusal_opens_no_failure_episode() { + local dir out status + dir=$(make_primary_dir "$TMP_ROOT/host-benign-refusal") + mkdir -p "$dir/config" + rm -f "$dir/config/supervision-host-off" + : > "$dir/state/task.meta" + write_host_fixture "$dir" benign-refusal + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + expect_code 0 "$status" "a refused rewake on an acknowledged marker must stay silent" + assert_not_contains "$out" "auto-arm FAILED" "a benign refusal must not deliver a failure notice" + assert_absent "$dir/state/.claude-autoarm-failure-notified" "a benign refusal opened a failure episode" + [ "$(epoch_outcome "$dir")" != failed ] || fail "a benign refusal must not record outcome=failed" + pass "auto-arm: a host rewake refused on an acknowledged marker opens no failure episode" +} + +# The host handed a wake back but left the marker in handling (pending or +# announced) with no live successor, so no rewake can commit: the hook delivers +# the failure notice once per episode and keeps exiting 2 without repeating it. +assert_host_lost_handback_notifies_once_per_episode() { + local kind=$1 dir out status + dir=$(make_primary_dir "$TMP_ROOT/host-$kind") + mkdir -p "$dir/config" + rm -f "$dir/config/supervision-host-off" + : > "$dir/state/task.meta" + write_host_fixture "$dir" "$kind" + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + expect_code 2 "$status" "a lost hand-back must reach main" + assert_contains "$out" "auto-arm FAILED - the supervision host returned an actionable wake" "a lost hand-back must deliver the failure notice" + assert_present "$dir/state/.claude-autoarm-failure-notified" "a lost hand-back did not record its failure episode" + [ "$(epoch_outcome "$dir")" = failed ] || fail "a lost hand-back must record outcome=failed, got: $(epoch_outcome "$dir")" + out=$(run_autoarm "$dir" 2>/dev/null); status=$? + expect_code 2 "$status" "a repeated lost hand-back must still reach main" + assert_not_contains "$out" "auto-arm FAILED" "a repeated lost hand-back must not repeat the failure notice" + [ "$(epoch_outcome "$dir")" = failed-suppressed ] \ + || fail "a repeated lost hand-back must record outcome=failed-suppressed, got: $(epoch_outcome "$dir")" +} + +test_host_lost_handback_notifies_once_per_episode() { + assert_host_lost_handback_notifies_once_per_episode lost-handback + pass "auto-arm: a lost host hand-back notifies once per failure episode" +} + +test_host_lost_announced_handback_notifies_once_per_episode() { + assert_host_lost_handback_notifies_once_per_episode lost-announced-handback + pass "auto-arm: a lost host hand-back on an announced marker notifies once per failure episode" +} + test_host_crash_is_retried_then_reported() { local dir out status dir=$(make_primary_dir "$TMP_ROOT/host-crash") @@ -1699,6 +1767,9 @@ test_host_handback_beside_a_quiet_record_carries_no_away_note test_plain_arm_banner_keeps_its_wake_line_cap test_host_handback_carries_every_host_line test_host_stand_down_is_silent +test_host_benign_rewake_refusal_opens_no_failure_episode +test_host_lost_handback_notifies_once_per_episode +test_host_lost_announced_handback_notifies_once_per_episode test_host_crash_is_retried_then_reported test_arguments_never_arm test_fm_lock_status_still_works_with_shared_lib diff --git a/tests/fm-claude-trust.test.sh b/tests/fm-claude-trust.test.sh index cafcb327dfd..70a807c04fa 100755 --- a/tests/fm-claude-trust.test.sh +++ b/tests/fm-claude-trust.test.sh @@ -626,11 +626,14 @@ test_refused_spawn_leaves_no_task_state() { } # Resolve the final prompt argument using the same shell argument splitting the -# pane sees after the two leading export statements. +# pane sees after the leading export statements. claude_launch_doorbell() { # - local command=${1#*; } + local command=$1 + while [[ "$command" == export\ *\;* ]]; do + command=${command#*; } + done ( - eval "set -- ${command#*; }" + eval "set -- $command" printf '%s' "${!#}" ) } diff --git a/tests/fm-composer-lib.test.sh b/tests/fm-composer-lib.test.sh index 0a9605b9eb1..a3a8c1e833f 100755 --- a/tests/fm-composer-lib.test.sh +++ b/tests/fm-composer-lib.test.sh @@ -764,6 +764,59 @@ test_matrix_grok_titled_bottom_border() { pass "matrix: grok's real oversized titled bottom is empty while typed and unproved panes stay safe" } +test_matrix_claude_titled_top_rule() { + # A named Claude Code session draws its title into the composer's TOP rule + # (issues #5601 and #5558; observed on herdr as + # `─── Firstmate operational input 1790546042 ─`). The strict separator + # predicate rejects that row, so the pair never opened, the closing rule + # read as a lower unmatched separator, and a visibly empty composer read + # `unknown` on every cursorless backend, refusing steers, exit, and relaunch. + local rule title top bottom footer screen ansi typed claude_idle + local scrollback short nonascii flush blank + claude_idle=$(printf 'claude\tidle') + rule='────────────────────────────────────────────────────────────' + title=' Firstmate operational input 1790546042 ' + top="${rule}───${title}─" + bottom="${rule}────────────────────────────────────────────" + footer=' ⏵⏵ bypass permissions on (shift+tab to cycle)' + screen="recap: earlier work"$'\n'"$top"$'\n❯'"$NBSP"$'\n'"$bottom"$'\n'"$footer" + ansi="${ESC}[38;2;128;130;131mrecap: earlier work${ESC}[0m"$'\n' + ansi+="${ESC}[0m${ESC}[38;2;121;129;134m${rule}─── ${ESC}[38;2;177;185;249m${title# }${ESC}[38;2;121;129;134m─${ESC}[0m"$'\n' + ansi+="${ESC}[0m${ESC}[38;2;128;130;131m❯${NBSP}${ESC}[0m"$'\n' + ansi+="${ESC}[0m${ESC}[38;2;121;129;134m${bottom}${ESC}[0m"$'\n'"$footer" + assert_screen "titled claude idle on herdr" empty "$CAPS_STYLED" "$screen" '' "$claude_idle" + assert_screen "titled claude idle on herdr (ansi)" empty "$CAPS_STYLED" "$ansi" '' "$claude_idle" + assert_screen "titled claude idle on zellij (ansi)" empty "$CAPS_STYLED_NOID" "$ansi" + assert_screen "titled claude idle on cmux/orca" empty "$CAPS_PLAIN" "$screen" + assert_screen "titled claude idle on tmux" empty "$CAPS_TMUX" "$ansi" 2 probe-absent + typed="$top"$'\n❯ fix the login bug\n'"$bottom"$'\n'"$footer" + assert_screen "titled claude typed on herdr" pending "$CAPS_STYLED" "$typed" '' "$claude_idle" + assert_screen "titled claude typed on zellij" pending "$CAPS_STYLED_NOID" "$typed" + assert_screen "titled claude typed on tmux" pending "$CAPS_TMUX" "$typed" 1 probe-absent + assert_screen "titled claude typed on plain backends" unknown "$CAPS_PLAIN" "$typed" + # The staleness rule still holds: a titled sandwich stranded in scrollback, + # with transcript rows between it and a lower unmatched rule, stays unknown. + scrollback="$top"$'\n❯'"$NBSP"$'\n'"$bottom"$'\nlater transcript output\n'"$bottom"$'\nmore output' + assert_screen "titled sandwich in scrollback" unknown "$CAPS_STYLED_NOID" "$scrollback" + # Width is proven, not assumed: a titled rule narrower than its closing rule + # is not that composer's top edge. + short="${rule}${title}─"$'\n❯'"$NBSP"$'\n'"$bottom" + assert_screen "mismatched titled rule width" unknown "$CAPS_STYLED_NOID" "$short" + # A non-ASCII title leaves residue and refuses rather than guessing width. + nonascii="${rule}─── ✳ Firstmate operational input 179054604 ─"$'\n❯'"$NBSP"$'\n'"$bottom" + assert_screen "non-ASCII titled rule" unknown "$CAPS_STYLED_NOID" "$nonascii" + # The rule must open with the strict separator's dash run. + flush=" Firstmate operational input 1790546042 ${rule}────"$'\n❯'"$NBSP"$'\n'"$bottom" + assert_screen "title flush at the rule's start" unknown "$CAPS_STYLED_NOID" "$flush" + # The strict blank-row posture is untouched: no glyph row, no proof. + blank="$top"$'\n\n'"$bottom" + assert_screen "titled rule over a blank row" unknown "$CAPS_STYLED_NOID" "$blank" + # The untitled pair keeps its verdict alongside the new shape. + assert_screen "untitled claude idle on herdr" empty "$CAPS_STYLED" \ + "$bottom"$'\n❯'"$NBSP"$'\n'"$bottom"$'\n'"$footer" '' "$claude_idle" + pass "matrix: claude's titled top rule proves an idle composer empty and a draft pending (#5601, #5558)" +} + test_matrix_kimi_bordered_shell_glyph_box() { # Kimi's bordered `│ > │` composer - the shape fm-spawn.sh's retired # spawn-local regex used to own. Now the shared owner proves it everywhere, @@ -1017,6 +1070,7 @@ test_matrix_pi_separated_needs_identity test_matrix_pi_dollar_status_footer_is_empty test_matrix_opencode_leftbar_signals test_matrix_grok_titled_bottom_border +test_matrix_claude_titled_top_rule test_matrix_kimi_bordered_shell_glyph_box test_matrix_claude_inside_zellij_ansi_dump test_strict_blank_row_divergence diff --git a/tests/fm-contributions.test.sh b/tests/fm-contributions.test.sh index aa14fe28a0e..ce87baefafb 100755 --- a/tests/fm-contributions.test.sh +++ b/tests/fm-contributions.test.sh @@ -299,6 +299,32 @@ test_verdict_retains_judged_head() { pass 'recorded judgment keeps its exact head and is stale immediately on a published replacement' } +test_verdict_actor_values_are_discoverable() { + local home help out actor + home=$(new_home verdict-actors) + forge_home "$home" + with_home "$home" "$ROOT/bin/fm-pr-check.sh" delivery https://github.com/o/r/pull/8 >/dev/null \ + || fail 'could not register delivery before judging its head' + help=$("$ROOT/bin/fm-contributions.sh" --help) || fail 'verdict help did not print' + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" verdict delivery https://github.com/o/r/pull/8 "$HEAD_A" \ + https://github.com/o/r/pull/8#issuecomment-99 bogus 'no such actor' 2>&1) \ + && fail 'an unknown actor was accepted' + [ "$(printf '%s\n' "$help" | sed -n '/^ fm-contributions.sh verdict /p')" = \ + ' fm-contributions.sh verdict ' ] \ + || fail "help usage does not name exactly the accepted actors: $help" + [ "$(printf '%s\n' "$help" | sed -n '/^actor is exactly one of /p')" = \ + 'actor is exactly one of captain, fleet, maintainer or nobody; any other value' ] \ + || fail "help explanation does not name exactly the accepted actors: $help" + [ "$out" = "fm-contributions: invalid required actor 'bogus'; expected one of: captain, fleet, maintainer, nobody" ] \ + || fail "refusal does not name exactly the accepted actors: $out" + for actor in captain fleet maintainer nobody; do + with_home "$home" "$ROOT/bin/fm-contributions.sh" verdict delivery https://github.com/o/r/pull/8 "$HEAD_A" \ + https://github.com/o/r/pull/8#issuecomment-99 "$actor" 'documented actor' >/dev/null \ + || fail "documented actor $actor was refused" + done + pass 'verdict help and refusal name exactly the actors the command accepts' +} + test_observed_replacement_refreshes_verdict() { local home home=$(new_home observed-replacement) @@ -606,17 +632,24 @@ set -eu printf '%s\n' "$*" >> "$FORGE/calls" fault=$(cat "$FORGE/fault" 2>/dev/null || true) case "$fault" in latency) sleep "${FORGE_LATENCY:-2}" ;; esac +# Concurrent forge callers each advance one shared clock. Truncating it in +# place races with the other callers and the fake date: an interleaved write +# can publish a half-written value (or the 6 an emptied read computes), and a +# caller then evaluates DEADLINE against torn arithmetic. Publish every new +# value by rename so each reader always sees one complete old-or-new clock. +clock_bump() { + local tmp + tmp=$(mktemp "$FORGE/clock.XXXXXX") + printf '%s\n' "$(( $(cat "$FORGE/clock") + $1 ))" > "$tmp" + mv -f "$tmp" "$FORGE/clock" +} case "$fault:$*" in # Advance once before the parallel read wave; its readers share this clock. - reserve:'api repos/o/r/issues/9') - printf '%s\n' "$(( $(cat "$FORGE/clock") + 6 ))" > "$FORGE/clock" ;; + reserve:'api repos/o/r/issues/9') clock_bump 6 ;; slow-wave:'api repos/o/r/pulls/8') sleep 3 ;; slow-wave:'api repos/o/r/pulls/8/reviews?'*) sleep 6 ;; - exhaust:'api repos/o/r/issues/8/comments?'*) - printf '%s\n' "$(( $(cat "$FORGE/clock") + 100 ))" > "$FORGE/clock" ;; - fail-late:'api repos/o/r/pulls/8/reviews?'*) - printf '%s\n' "$(( $(cat "$FORGE/clock") + 100 ))" > "$FORGE/clock" - printf 'HTTP 502\n' >&2; exit 1 ;; + exhaust:'api repos/o/r/issues/8/comments?'*) clock_bump 100 ;; + fail-late:'api repos/o/r/pulls/8/reviews?'*) clock_bump 100; printf 'HTTP 502\n' >&2; exit 1 ;; fail:'api repos/o/r/pulls/8/reviews?'*) printf 'HTTP 502\n' >&2; exit 1 ;; down:*) printf 'HTTP 502\n' >&2; exit 1 ;; hang:'api repos/o/r/pulls/8') sleep 4 ;; @@ -1066,7 +1099,7 @@ test_late_owner_keeps_failure_episode_suppressed() { } failures=0 -for test_name in test_actor_coverage test_stale_verdict test_unchecked_is_not_silence test_newest_check_has_no_verdict test_comment_wake test_review_wake test_inline_wake test_ready_issue_wake test_fresh_issue_requires_maintainer test_missing_lane_remains_missing test_partial_freshness_keeps_measured_rows test_malformed_record_cannot_prove_silence test_issue_timeline_and_exact_ack test_verdict_retains_judged_head test_observed_replacement_refreshes_verdict test_unobserved_head_leaves_verdict_unknown test_away_yolo_is_fleet_work test_away_yolo_cross_home_is_fleet_work test_retired_and_unsupported_coverage test_unsupported_forge_is_not_fleet_work test_held_unsupported_forge_is_not_captain_work test_shared_contribution_signal_wakes_once test_watcher_keeps_diagnostics_separate_from_contribution_wakes test_expired_child_unsupported_forge_stays_unmeasured test_watcher_surfaces_new_contribution_once test_home_summary_coverage test_unreadable_pending_is_not_empty test_record_task_identity_matches_dirname_basename test_read_only_views_create_no_state test_budget_refusal_between_calls test_budget_bounded_call_timeout test_genuine_failure_near_deadline_is_unavailable test_shared_url_observed_once test_terminal_contribution_settles test_late_owner_inherits_terminal_observation test_interrupted_multi_owner_poll_settles_every_owner test_done_task_open_pr_still_observed test_reservation_defers_later_url_when_fifteen_seconds_do_not_remain test_three_second_pr_reads_complete_fresh_in_one_cycle test_slow_read_deadline_kill_is_budget_refusal test_unmeasured_url_does_not_starve_the_tail test_budget_is_cut_down_to_the_watcher_check_bound test_arm_plumbs_a_configured_budget_into_the_check_shim test_unavailable_forge_records_error_and_wakes_once_per_episode test_late_owner_keeps_failure_episode_suppressed; do +for test_name in test_actor_coverage test_stale_verdict test_unchecked_is_not_silence test_newest_check_has_no_verdict test_comment_wake test_review_wake test_inline_wake test_ready_issue_wake test_fresh_issue_requires_maintainer test_missing_lane_remains_missing test_partial_freshness_keeps_measured_rows test_malformed_record_cannot_prove_silence test_issue_timeline_and_exact_ack test_verdict_retains_judged_head test_verdict_actor_values_are_discoverable test_observed_replacement_refreshes_verdict test_unobserved_head_leaves_verdict_unknown test_away_yolo_is_fleet_work test_away_yolo_cross_home_is_fleet_work test_retired_and_unsupported_coverage test_unsupported_forge_is_not_fleet_work test_held_unsupported_forge_is_not_captain_work test_shared_contribution_signal_wakes_once test_watcher_keeps_diagnostics_separate_from_contribution_wakes test_expired_child_unsupported_forge_stays_unmeasured test_watcher_surfaces_new_contribution_once test_home_summary_coverage test_unreadable_pending_is_not_empty test_record_task_identity_matches_dirname_basename test_read_only_views_create_no_state test_budget_refusal_between_calls test_budget_bounded_call_timeout test_genuine_failure_near_deadline_is_unavailable test_shared_url_observed_once test_terminal_contribution_settles test_late_owner_inherits_terminal_observation test_interrupted_multi_owner_poll_settles_every_owner test_done_task_open_pr_still_observed test_reservation_defers_later_url_when_fifteen_seconds_do_not_remain test_three_second_pr_reads_complete_fresh_in_one_cycle test_slow_read_deadline_kill_is_budget_refusal test_unmeasured_url_does_not_starve_the_tail test_budget_is_cut_down_to_the_watcher_check_bound test_arm_plumbs_a_configured_budget_into_the_check_shim test_unavailable_forge_records_error_and_wakes_once_per_episode test_late_owner_keeps_failure_episode_suppressed; do ( "$test_name" ) || failures=$((failures + 1)) done [ "$failures" -eq 0 ] || fail "$failures contribution regressions" diff --git a/tests/fm-extension-binding.test.sh b/tests/fm-extension-binding.test.sh index 22e23f1a645..ab7530a49df 100644 --- a/tests/fm-extension-binding.test.sh +++ b/tests/fm-extension-binding.test.sh @@ -108,6 +108,8 @@ extension_test_cleanup() { ( # worker.pid names the serving child; the copied remote helper stops its # known isolated supervisor tree so it cannot respawn during teardown. + # Production libraries are linted independently by fm-lint.sh. + # shellcheck source=/dev/null . "$REMOTE_ROOT/bin/fm-remote-job-lib.sh" fm_remote_job_stop_worker_tree "$(cat "$TMP_ROOT/remote-jobs/worker.pid")" ) 2>/dev/null || true diff --git a/tests/fm-fleet-sync.test.sh b/tests/fm-fleet-sync.test.sh index 68c32f50d30..0bee1668c3d 100755 --- a/tests/fm-fleet-sync.test.sh +++ b/tests/fm-fleet-sync.test.sh @@ -683,8 +683,8 @@ test_symlinked_clone_still_syncs() { home=$(new_home) clone=$(build_pair "$home" sigma) advance_origin "$home" sigma C1 - # A symlinked clone dir is a real clone root; the guard compares resolved paths, - # so it must not be mistaken for a directory nested in someone else's repo. + # A symlinked clone dir is a real clone root and must not be mistaken for a + # directory nested in someone else's repo. mv "$clone" "$home/real-sigma" ln -s "$home/real-sigma" "$clone" @@ -694,6 +694,38 @@ test_symlinked_clone_still_syncs() { pass "the clone-root guard accepts a symlinked clone directory" } +test_clone_root_named_by_another_spelling_still_syncs() { + local home clone fakebin alias out + home=$(new_home) + clone=$(build_pair "$home" tau) + advance_origin "$home" tau C1 + fakebin="$home/fb-rootalias"; rm -rf "$fakebin"; mkdir -p "$fakebin" + # git reports the clone's own root through an alias that is the same directory + # but a different string, as it does on a case-insensitive volume when the home + # was recorded with other casing. A symlink stands in for the case difference so + # the test also holds on a case-sensitive filesystem. + alias="$home/root-alias" + ln -s "$clone" "$alias" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +real=${REAL_GIT_FOR_TEST:?} +case " $* " in + *" rev-parse --show-toplevel "*) printf '%s\n' "${ROOT_ALIAS_FOR_TEST:?}"; exit 0 ;; +esac +exec "$real" "$@" +SH + chmod +x "$fakebin/git" + out="$home/out"; err="$home/err" + + ROOT_ALIAS_FOR_TEST="$alias" run_sync_guarded "$home" "$fakebin" "$out" "$err" tau || true + + assert_contains "$(cat "$out")" "tau: synced" \ + "a clone root that git names with another spelling must still fast-forward" + assert_not_contains "$(cat "$out")" "not a clone root" \ + "the guard must compare the directory itself, not the spelling of its path" + pass "the clone-root guard accepts a root named by a different spelling of the same directory" +} + test_non_signature_fetch_failure_is_not_retried() { local home fakebin clone out err home=$(new_home) @@ -741,3 +773,4 @@ test_non_signature_fetch_failure_is_not_retried test_non_clone_dir_never_syncs_the_enclosing_repo test_non_clone_dir_named_directly_never_syncs_the_enclosing_repo test_symlinked_clone_still_syncs +test_clone_root_named_by_another_spelling_still_syncs diff --git a/tests/fm-git-strip-ai-trailers.test.sh b/tests/fm-git-strip-ai-trailers.test.sh index c12124194ba..2f66b1008fe 100644 --- a/tests/fm-git-strip-ai-trailers.test.sh +++ b/tests/fm-git-strip-ai-trailers.test.sh @@ -234,6 +234,61 @@ test_pane_hookspath_does_not_reroute_another_repository() { pass "a pane GIT_CONFIG hooksPath still chains the repository git is actually in" } +test_empty_project_hookspath_runs_no_repository_hook() { + local repo hooks err + repo="$TMP_ROOT/empty-hookspath" + make_repo "$repo" + write_marker_hook "$repo/.git/hooks/pre-commit" default-pre-commit + git -C "$repo" config core.hooksPath '' + hooks="$TMP_ROOT/hooks-empty" + "$STRIP" install "$hooks" "$repo" || fail "install should succeed with an empty core.hooksPath" + printf 'note\n' >>"$repo/README.md" + git -C "$repo" add README.md + err=$(with_hooks_env "$hooks" git -C "$repo" commit -q --trailer 'Co-authored-by: Cursor ' -m 'fix: empty hooksPath' 2>&1) || + fail "a commit in a repo with an empty core.hooksPath was refused: $err" + assert_equals "" "$err" "an empty core.hooksPath commit printed errors" + [ -f "$repo/default-pre-commit.ran" ] && fail "a repository hook ran although core.hooksPath is empty" + assert_not_contains "$(git -C "$repo" log -1 --format=%B)" "Co-authored-by: Cursor" \ + "Cursor trailer survived an empty-hooksPath commit" + pass "an empty project core.hooksPath runs no repository hook and still strips the trailer" +} + +test_unresolvable_project_hookspath_still_refuses() { + local repo hooks head err + repo="$TMP_ROOT/unresolvable-hookspath" + make_repo "$repo" + printf 'note\n' >>"$repo/README.md" + git -C "$repo" add README.md + git -C "$repo" config core.hooksPath '~fm-no-such-user-6171/hooks' + hooks="$TMP_ROOT/hooks-unresolvable" + "$STRIP" install "$hooks" "$repo" || fail "install should succeed with an unresolvable core.hooksPath" + head=$(git -C "$repo" rev-parse HEAD) + err=$(with_hooks_env "$hooks" git -C "$repo" commit -q -m 'fix: unresolvable hooksPath' 2>&1) && + fail "a commit succeeded although the repository's hooks directory cannot be resolved" + assert_contains "$err" "refusing to skip its pre-commit hook" "the refusal did not name the skipped hook" + assert_equals 1 "$(printf '%s\n' "$err" | grep -c 'failed to expand user dir')" "git's lookup error was not shown exactly once" + assert_equals "$head" "$(git -C "$repo" rev-parse HEAD)" "a refused commit still moved HEAD" + pass "an unresolvable project core.hooksPath still refuses the commit" +} + +test_valueless_project_hookspath_still_refuses() { + local repo hooks head err + repo="$TMP_ROOT/valueless-hookspath" + make_repo "$repo" + hooks="$TMP_ROOT/hooks-valueless" + "$STRIP" install "$hooks" "$repo" || fail "install should succeed before the valueless key is written" + head=$(git -C "$repo" rev-parse HEAD) + printf 'note\n' >>"$repo/README.md" + git -C "$repo" add README.md + printf '[core]\n\thooksPath\n' >>"$repo/.git/config" + err=$(with_hooks_env "$hooks" git -C "$repo" commit -q -m 'fix: valueless hooksPath' 2>&1) && + fail "a commit succeeded although core.hooksPath has no value" + assert_contains "$err" "refusing to skip its pre-commit hook" "the refusal did not name the skipped hook" + assert_equals 1 "$(printf '%s\n' "$err" | grep -c "missing value for 'core.hookspath'")" "git's lookup error was not shown exactly once" + assert_equals "$head" "$(git -C "$repo" -c core.hooksPath=x rev-parse HEAD)" "a refused commit still moved HEAD" + pass "a valueless project core.hooksPath still refuses the commit" +} + write_refusing_pre_push() { # cat >"$1" < + local state + state=$(CDPATH='' cd -- "$1/state" && pwd -P) || fail "cannot resolve state dir $1/state" + printf "export FM_TASK_INBOX='%s'; " "$state/$2.inbox" +} + ai_trailer_hooks_prefix() { # local state state=$(CDPATH='' cd -- "$1/state" && pwd -P) || fail "cannot resolve state dir $1/state" @@ -304,7 +310,7 @@ test_kimi_launch_then_send_is_verified() { assert_contains "$out" "spawned $id harness=kimi" "kimi spawn did not report success" launch=$(cat "$CASE_DIR/launch.log") - [ "$launch" = "export COMPACT_ADVISER_DISABLE=1; $(ai_trailer_hooks_prefix "$HOME_DIR" "$id")FM_HOME='$HOME_DIR' env -u TYPESAFE_API_KEY -u OPENROUTER_API_KEY env -u CURSOR_AGENT -u CURSOR_INVOKED_AS -u GEMINI_CLI '$FAKEBIN_DIR/kimi' --model 'kimi-code/k3' --auto" ] \ + [ "$launch" = "export COMPACT_ADVISER_DISABLE=1; $(task_inbox_export "$HOME_DIR" "$id")$(ai_trailer_hooks_prefix "$HOME_DIR" "$id")FM_HOME='$HOME_DIR' env -u TYPESAFE_API_KEY -u OPENROUTER_API_KEY env -u CURSOR_AGENT -u CURSOR_INVOKED_AS -u GEMINI_CLI '$FAKEBIN_DIR/kimi' --model 'kimi-code/k3' --auto" ] \ || fail "kimi launch did not use the absolute binary, model, and --auto only: $launch" assert_not_contains "$launch" "--effort" "kimi launch emitted a nonexistent effort flag" assert_not_contains "$launch" "turn-ended" "kimi launch embedded a turn-end path" @@ -680,7 +686,7 @@ test_kimi_falls_back_to_expanded_home_binary() { rc=$? expect_code 0 "$rc" "Kimi HOME fallback spawn should succeed" launch=$(cat "$CASE_DIR/launch.log") - [ "$launch" = "export COMPACT_ADVISER_DISABLE=1; $(ai_trailer_hooks_prefix "$HOME_DIR" "$id")FM_HOME='$HOME_DIR' env -u TYPESAFE_API_KEY -u OPENROUTER_API_KEY env -u CURSOR_AGENT -u CURSOR_INVOKED_AS -u GEMINI_CLI '$fallback' --auto" ] \ + [ "$launch" = "export COMPACT_ADVISER_DISABLE=1; $(task_inbox_export "$HOME_DIR" "$id")$(ai_trailer_hooks_prefix "$HOME_DIR" "$id")FM_HOME='$HOME_DIR' env -u TYPESAFE_API_KEY -u OPENROUTER_API_KEY env -u CURSOR_AGENT -u CURSOR_INVOKED_AS -u GEMINI_CLI '$fallback' --auto" ] \ || fail "Kimi fallback did not expand HOME into an absolute executable: $launch" pass "fm-spawn: Kimi fallback expands the active HOME" } diff --git a/tests/fm-lint.test.sh b/tests/fm-lint.test.sh index 5cf04326472..40d7ef7db50 100755 --- a/tests/fm-lint.test.sh +++ b/tests/fm-lint.test.sh @@ -3,10 +3,9 @@ # # bin/fm-lint.sh is the single owner invoked by CI # (.github/workflows/ci.yml) and by the pre-push gate (.no-mistakes.yaml -# commands.lint). CI runs its two full-rigor canonical partitions; the local -# gate uses its context-selected default. Their selection differs deliberately, -# while this owner keeps analysis flags, configuration, and tool versions from -# drifting. +# commands.lint). CI and the local gate deliberately select different roots; +# bin/fm-lint.sh owns their analysis modes, memory fallback, configuration, +# and tool versions. # Regression origin: with no commands.lint configured, the local no-mistakes # lint step never ran the deterministic shell lint, so PRs passed local # validation yet failed CI on info/warning findings such as SC2015, SC1007, and @@ -1532,6 +1531,172 @@ test_root_memory_limit_reports_a_named_death() { pass "a root refused by its enforced memory limit fails by name with a memory reason" } +test_memory_failure_retries_without_external_sources() { + local tmp fakebin fixture out rc log rss_kib require_bounds=0 mode + local -a modes=(0) + if fm_lint_bounds_supported; then + require_bounds=1 + modes=(1 0) + fi + tmp=$(fm_test_tmproot fm-lint-memory-fallback) + fakebin=$(fm_fakebin "$tmp") + fixture="$tmp/teardown.sh" + log="$tmp/flags.log" + printf '#!/usr/bin/env bash\n# shellcheck source=lib.sh\nexit 0\n' > "$fixture" + cat > "$fakebin/shellcheck" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --version ]; then + printf 'ShellCheck - shell script analysis tool\nversion: 0.11.0\n' + exit 0 +fi +follow=no +exclude=none +while [ "$#" -gt 0 ] && [ "$1" != -- ]; do + case "$1" in + --external-sources) follow=yes ;; + --exclude=*) exclude=${1#--exclude=} ;; + esac + shift +done +shift +printf '%s\t%s\n' "$follow" "$exclude" >> "$FM_TEST_FALLBACK_LOG" +if [ "$follow" = yes ]; then + printf 'shellcheck: Heap exhausted;\n' >&2 + exit 251 +fi +exit 0 +SH + chmod +x "$fakebin/shellcheck" + + for mode in "${modes[@]}"; do + : > "$log" + rc=0 + out=$(PATH="$fakebin:$PATH" FM_LINT_JOBS=1 FM_LINT_REQUIRE_BOUNDS="$mode" \ + FM_TEST_FALLBACK_LOG="$log" "$LINT" --telemetry "$tmp/pass.$mode.tsv" "$fixture" 2>&1) || rc=$? + [ "$rc" -eq 0 ] || fail "a clean no-source fallback did not pass (bounded=$mode)"$'\n'"$out" + assert_grep $'source_directives\t1' "$tmp/pass.$mode.tsv" "telemetry lost the root's source directive" + assert_grep $'source_followed_directives\t0' "$tmp/pass.$mode.tsv" \ + "telemetry counted a source directive that the passing fallback did not follow" + [ "$(cat "$log")" = "$(printf 'yes\tnone\nno\tSC1091,SC2034,SC2153,SC2329')" ] \ + || fail "the memory failure did not retry without external sources and exclude only cross-file codes"$'\n'"$(cat "$log")" + assert_contains "$out" "hit the memory ceiling with --external-sources (reason=memory rc=251)" \ + "the fallback was not identified in the output" + assert_contains "$out" "fallback passed with cross-file codes excluded (SC1091,SC2034,SC2153,SC2329)" \ + "the narrower fallback result was not disclosed" + awk -F '\t' '$1 == "end" && $3 ~ /teardown\.sh$/ && $9 == 0 && $10 == "memory-fallback" { found=1 } END { exit !found }' \ + "$tmp/pass.$mode.roots.tsv" || fail "the clean fallback was not recorded distinctly" + rss_kib=$(awk -F '\t' '$1 == "end" && $3 ~ /teardown\.sh$/ { print $11 }' "$tmp/pass.$mode.roots.tsv") + if [ "$mode" -eq 1 ]; then + assert_grep $'meta\tbounds_enforced\t1' "$tmp/pass.$mode.roots.tsv" \ + "the bounded fallback did not enforce bounds" + case "$rss_kib" in ''|*[!0-9]*) fail "the fallback attempts lost per-root RSS reporting: $rss_kib" ;; esac + else + assert_grep $'meta\tbounds_enforced\t0' "$tmp/pass.$mode.roots.tsv" \ + "the unbounded fallback unexpectedly enforced bounds" + [ "$rss_kib" = unavailable ] || fail "the unbounded fallback unexpectedly reported RSS: $rss_kib" + fi + done + + if ! pinned_ready; then + pass "SKIP (ShellCheck $REQUIRED not resolved): real fallback finding check" + return + fi + local real_shellcheck + real_shellcheck=$(command -v shellcheck) + cat > "$fakebin/shellcheck" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --version ]; then + exec "$FM_REAL_SHELLCHECK" "$@" +fi +for arg in "$@"; do + if [ "$arg" = --external-sources ]; then + printf 'shellcheck: Heap exhausted;\n' >&2 + exit 251 + fi +done +exec "$FM_REAL_SHELLCHECK" "$@" +SH + chmod +x "$fakebin/shellcheck" + # shellcheck disable=SC2016 # The fixture intentionally contains an unexpanded parameter. + printf '#!/usr/bin/env bash\nx=$1\nprintf "%%s\\n" $x\n' > "$fixture" + rc=0 + out=$(PATH="$fakebin:$PATH" FM_REAL_SHELLCHECK="$real_shellcheck" \ + FM_LINT_JOBS=1 FM_LINT_REQUIRE_BOUNDS="$require_bounds" \ + "$LINT" --telemetry "$tmp/finding.tsv" "$fixture" 2>&1) || rc=$? + [ "$rc" -eq 1 ] || fail "a real ShellCheck finding in the fallback did not fail lint (exit $rc)"$'\n'"$out" + assert_contains "$out" "fallback reason=findings rc=1" \ + "the fallback finding was not identified" + assert_contains "$out" "SC2086" "the real fallback finding was not reported" + awk -F '\t' '$1 == "end" && $3 ~ /teardown\.sh$/ && $9 == 1 && $10 == "findings" { found=1 } END { exit !found }' \ + "$tmp/finding.roots.tsv" || fail "the fallback finding was not recorded as a failure" + pass "memory failures retry without source following, exclude cross-file codes, and preserve a real fallback finding" +} + +test_memory_fallback_spends_only_the_remaining_root_deadline() { + if ! fm_lint_bounds_supported; then + pass "SKIP (host cannot enforce the bounded envelope): fallback deadline check" + return + fi + local tmp fakebin fixture log out rc duration_ms + tmp=$(fm_test_tmproot fm-lint-fallback-deadline) + fakebin=$(fm_fakebin "$tmp") + fixture="$tmp/teardown.sh" + log="$tmp/attempts.log" + printf '#!/usr/bin/env bash\nexit 0\n' > "$fixture" + cat > "$fakebin/shellcheck" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --version ]; then + printf 'ShellCheck - shell script analysis tool\nversion: 0.11.0\n' + exit 0 +fi +for arg in "$@"; do + if [ "$arg" = --external-sources ]; then + printf 'follow\n' >> "$FM_TEST_ATTEMPT_LOG" + sleep "$FM_TEST_FIRST_SECS" + printf 'shellcheck: Heap exhausted;\n' >&2 + exit 251 + fi +done +printf 'fallback\n' >> "$FM_TEST_ATTEMPT_LOG" +sleep 60 +exit 0 +SH + chmod +x "$fakebin/shellcheck" + + # A 3s first attempt leaves about 3s of the 6s deadline, so the retry is + # killed there; a fresh deadline would let the root run for about 9s. + : > "$log" + rc=0 + out=$(PATH="$fakebin:$PATH" FM_LINT_JOBS=1 FM_LINT_REQUIRE_BOUNDS=1 \ + FM_LINT_ROOT_SECONDS=6 FM_LINT_ROOT_GRACE=1 \ + FM_TEST_ATTEMPT_LOG="$log" FM_TEST_FIRST_SECS=3 \ + "$LINT" --telemetry "$tmp/partial.tsv" "$fixture" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "a fallback cut off by the root deadline unexpectedly passed" + [ "$(cat "$log")" = "$(printf 'follow\nfallback')" ] \ + || fail "the root did not retry once with its remaining time"$'\n'"$(cat "$log")" + assert_contains "$out" "fallback reason=timeout" \ + "the fallback was not stopped by the root's remaining deadline"$'\n'"$out" + duration_ms=$(awk -F '\t' '$1 == "end" && $3 ~ /teardown\.sh$/ { print $8 }' "$tmp/partial.roots.tsv") + [ "$duration_ms" -lt 7000 ] \ + || fail "the first attempt and fallback together exceeded the root deadline plus grace: ${duration_ms}ms" + + # With under a second of the deadline left, no retry starts. + : > "$log" + rc=0 + out=$(PATH="$fakebin:$PATH" FM_LINT_JOBS=1 FM_LINT_REQUIRE_BOUNDS=1 \ + FM_LINT_ROOT_SECONDS=6 FM_LINT_ROOT_GRACE=1 \ + FM_TEST_ATTEMPT_LOG="$log" FM_TEST_FIRST_SECS=5.2 \ + "$LINT" --telemetry "$tmp/spent.tsv" "$fixture" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "a memory failure with no deadline left unexpectedly passed" + [ "$(cat "$log")" = follow ] \ + || fail "a fallback started with no time left in the root deadline"$'\n'"$(cat "$log")" + assert_contains "$out" "no time left in its 6s deadline to retry without it" \ + "the skipped fallback was not explained"$'\n'"$out" + awk -F '\t' '$1 == "end" && $3 ~ /teardown\.sh$/ && $10 == "memory" && $12 == 1 { found=1 } END { exit !found }' \ + "$tmp/spent.roots.tsv" || fail "the unretried memory failure was not recorded as a source-following memory failure" + pass "a memory fallback runs only within the time left in its root's original deadline" +} + test_memory_evidence_outranks_findings_and_signal_reasons() { local tmp fakebin roots_log out rc name reason bounded local -a roots modes @@ -1800,13 +1965,8 @@ test_seeded_module_boundary_parity() { pass "SKIP (ShellCheck $REQUIRED not resolved): seeded source-boundary parity check" return fi - local tmp rel adapter dispatcher dep owner test_root out rc - tmp=$(mktemp -d "$ROOT/.fm-lint-parity.XXXXXX") - if [ "${#FM_TEST_CLEANUP_DIRS[@]}" -eq 0 ]; then - trap fm_test_cleanup EXIT - fi - FM_TEST_CLEANUP_DIRS+=("$tmp") - rel=${tmp#"$ROOT/"} + local tmp adapter dispatcher dep owner test_root out rc + tmp=$(fm_test_tmproot fm-lint-parity) adapter="$tmp/adapter.sh" dispatcher="$tmp/dispatcher.sh" dep="$tmp/owner-dep.sh" @@ -1834,7 +1994,7 @@ owner_dependency_value=ok SH cat > "$owner" <: build a remote mate home whose state dir carries one +# genuine task log and the outbound parent channel, with one captain-facing +# decision, one reserved-key resolution, and one informational note on the +# channel - exactly the line shapes a mate home publishes mechanically. +seed_remote_mate() { # + local dir=$1 + mkdir -p "$dir/state" + printf '%s\n' mate > "$dir/.fm-secondmate-home" + printf 'schema=fm-secondmate-parent.v1\nroute=remote\nparent_host=remote.example\n' \ + > "$dir/.fm-secondmate-parent" + printf 'needs-decision [key=captain-hold-pr-7-1]: captain hold pr-7: merge the green PR?\n' \ + > "$dir/state/parent-replies.status" + printf 'resolved [key=captain-hold-pr-5-2]: captain chose the staged rollout\n' \ + >> "$dir/state/parent-replies.status" + printf 'note: the release branch is cut\n' >> "$dir/state/parent-replies.status" + printf 'needs-decision [key=api-shape]: pick REST or RPC\n' > "$dir/state/real-task.status" + printf 'note: benchmark results are in\n' >> "$dir/state/real-task.status" +} + +# seed_plain_home : a main home (no secondmate identity marker) whose +# state dir carries a parent-replies.status that merely shares the name. +seed_plain_home() { # + local dir=$1 + mkdir -p "$dir/state" + printf 'needs-decision [key=name-only]: an ordinary task file that shares the name\n' \ + > "$dir/state/parent-replies.status" + printf 'needs-decision [key=other-task]: a genuine sibling task decision\n' \ + > "$dir/state/other-task.status" +} + +# seed_local_mate : a LOCAL mate home - its parent channel +# lives in the parent home's state/.status, so a parent-replies.status in +# its own state dir is an ordinary self-home file. +seed_local_mate() { # + local dir=$1 parent_home=$2 + mkdir -p "$dir/state" + printf '%s\n' mate > "$dir/.fm-secondmate-home" + printf 'schema=fm-secondmate-parent.v1\nroute=local\nparent_home=%s\n' "$parent_home" \ + > "$dir/.fm-secondmate-parent" + printf 'needs-decision [key=local-shape]: still an ordinary self-home file\n' \ + > "$dir/state/parent-replies.status" +} + +REMOTE="$TMP_ROOT/remote-mate" +PLAIN="$TMP_ROOT/main-home" +LOCAL_MATE="$TMP_ROOT/local-mate" +seed_remote_mate "$REMOTE" +seed_plain_home "$PLAIN" +seed_local_mate "$LOCAL_MATE" "$PLAIN" +REMOTE_STATE="$REMOTE/state" +PLAIN_STATE="$PLAIN/state" +LOCAL_STATE="$LOCAL_MATE/state" + +# --- unit: the exclusion predicates ----------------------------------------- + +test_predicate_resolves_only_the_remote_channel() { + local out rc + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + status_scan_parent_channel_exclude "$2" + ' _ "$ROOT" "$REMOTE_STATE") \ + || fail "the remote mate's channel must resolve for exclusion, got rc=$?" + [ "$out" = "$REMOTE_STATE/parent-replies.status" ] \ + || fail "the exclusion must be the resolved channel path, got: $out" + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + status_scan_parent_channel_exclude "$2" + ' _ "$ROOT" "$PLAIN_STATE") + [ -z "$out" ] || fail "a main home must exclude nothing, got: $out" + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + status_scan_parent_channel_exclude "$2" + ' _ "$ROOT" "$LOCAL_STATE") + [ -z "$out" ] || fail "a local mate must exclude nothing, got: $out" + pass "only a remote mate home resolves its own parent channel for exclusion" +} + +test_resolver_predicates_on_home_shape_not_name() { + local out + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-parent-channel-lib.sh + . "$1/bin/fm-parent-channel-lib.sh" + fm_parent_channel_outbound_status "$2" "$3" + ' _ "$ROOT" "$REMOTE" "$REMOTE_STATE") \ + || fail "the remote mate's outbound status must resolve" + [ "$out" = "$REMOTE_STATE/parent-replies.status" ] \ + || fail "the remote route must resolve into the mate's own state dir, got: $out" + FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-parent-channel-lib.sh + . "$1/bin/fm-parent-channel-lib.sh" + fm_parent_channel_outbound_status "$2" "$3" + ' _ "$ROOT" "$PLAIN" "$PLAIN_STATE" \ + && fail "a main home has no outbound parent-channel status" + FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-parent-channel-lib.sh + . "$1/bin/fm-parent-channel-lib.sh" + fm_parent_channel_outbound_status "$2" "$3" + ' _ "$ROOT" "$LOCAL_MATE" "$LOCAL_STATE" \ + && fail "a local mate's channel lives in the parent home, not its own state dir" + pass "fm_parent_channel_outbound_status resolves only the remote route" +} + +# --- unit: the fleet-wide folds omit the channel and keep genuine tasks ----- + +test_remote_folds_omit_channel_and_keep_genuine_task() { + local dir out + dir="$TMP_ROOT/folds" + seed_remote_mate "$dir/home" + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + echo "WHOLE:"; scan_open_decisions "$2" + echo "SNAPSHOT:"; status_presentation_snapshot "$2" + echo "UNREAD:"; scan_unread_surface_lines "$2" + ' _ "$ROOT" "$dir/home/state") || fail "the remote-mate fold pass failed" + case "$out" in *parent-replies*) + fail "the channel leaked into the remote mate's folds: $out" ;; + esac + printf '%s\n' "$out" | sed -n '/^WHOLE:/,/^SNAPSHOT:/p' | grep -F 'api-shape' >/dev/null \ + || fail "the genuine task's open decision must still fold: $out" + printf '%s\n' "$out" | sed -n '/^SNAPSHOT:/,/^UNREAD:/p' | grep -F 'real-task' >/dev/null \ + || fail "the genuine task must stay in the presentation snapshot: $out" + printf '%s\n' "$out" | sed -n '/^UNREAD:/,$p' | grep -F 'benchmark results' >/dev/null \ + || fail "the genuine task's note must stay on the unread surface: $out" + pass "a remote mate's folds omit its channel and keep a genuine task" +} + +test_incremental_fold_omits_channel_and_keeps_genuine_task() { + local dir out + dir="$TMP_ROOT/folds-incremental" + seed_remote_mate "$dir/home" + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + scan_open_decisions_incremental "$2" + ' _ "$ROOT" "$dir/home/state") || fail "the incremental fold failed" + case "$out" in *parent-replies*) + fail "the channel leaked into the incremental fold: $out" ;; + esac + printf '%s\n' "$out" | grep -F 'api-shape' >/dev/null \ + || fail "the genuine task's decision must still fold incrementally: $out" + pass "the cursor-backed incremental fold omits a remote mate's channel" +} + +test_channel_lines_never_reach_the_remote_unread_surface() { + local dir out + dir="$TMP_ROOT/unread" + seed_remote_mate "$dir/home" + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + scan_unread_surface_lines "$2" + ' _ "$ROOT" "$dir/home/state") || fail "the unread-surface scan failed" + case "$out" in *parent-replies*|*captain-hold*|*release\ branch*) + fail "channel decision, resolution, or note surfaced as self-home unread status: $out" ;; + esac + pass "the channel's resolution and note lines stay off the remote unread surface" +} + +test_name_shared_file_folds_in_a_main_home() { + local out + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + scan_open_decisions "$2" + ' _ "$ROOT" "$PLAIN_STATE") || fail "the main-home fold failed" + printf '%s\n' "$out" | grep -F 'name-only' >/dev/null \ + || fail "a main home's parent-replies.status must keep folding as an ordinary task: $out" + printf '%s\n' "$out" | grep -F 'other-task' >/dev/null \ + || fail "the sibling task decision must keep folding: $out" + pass "a parent-replies.status in a main home still folds" +} + +test_name_shared_file_folds_in_a_local_mate() { + local out + out=$(FM_TEST_LIB_SOURCED=1 bash -c ' + # shellcheck source=bin/fm-classify-lib.sh + . "$1/bin/fm-classify-lib.sh" + scan_open_decisions "$2" + ' _ "$ROOT" "$LOCAL_STATE") || fail "the local-mate fold failed" + printf '%s\n' "$out" | grep -F 'local-shape' >/dev/null \ + || fail "a local mate's parent-replies.status must keep folding: $out" + pass "a parent-replies.status in a local mate still folds" +} + +# --- unit: the watcher's signal scan and heartbeat backstop ----------------- + +# Source the watcher once with an isolated state/home; its source guard returns +# before the lock/loop, so only the functions load. scan_signals and +# heartbeat_scan_finds_actionable read STATE at call time. FM_ROOT_OVERRIDE +# stays at the inert dir set above; the unit-called functions read STATE, not +# the repo root. +WATCH_STATE="$REMOTE_STATE" +export FM_STATE_OVERRIDE="$WATCH_STATE" +export FM_HOME="$REMOTE" +# Production modules are independently linted canonical roots. Keep this test's +# ShellCheck context local while preserving its unchanged runtime source path. +# shellcheck source=/dev/null +. "$ROOT/bin/fm-watch.sh" + +test_watcher_scan_skips_channel_and_keeps_task_in_remote_mate() { + local out rc + STATE="$REMOTE_STATE" + out=$(scan_signals) || fail "scan_signals failed over the remote mate state" + printf '%s\n' "$out" | cut -f3 | grep -F 'parent-replies.status' >/dev/null \ + && fail "the channel must not produce a signal wake: $out" + printf '%s\n' "$out" | cut -f3 | grep -F 'real-task.status' >/dev/null \ + || fail "the genuine task's status must still wake: $out" + pass "scan_signals skips a remote mate's channel and still reports its tasks" +} + +test_heartbeat_backstop_skips_channel_in_remote_mate() { + local dir rc + dir="$TMP_ROOT/heartbeat" + seed_remote_mate "$dir/home" + # A quiet task log keeps the first pass channel-only: the note: line is + # informational, so only the excluded channel could make the scan actionable. + printf 'note: benchmark results are in\n' > "$dir/home/state/real-task.status" + STATE="$dir/home/state" + heartbeat_scan_finds_actionable; rc=$? + [ "$rc" -eq 1 ] || fail "the channel must not surface through the heartbeat backstop (rc=$rc): $FM_HEARTBEAT_SURFACE_ENDPOINTS" + case "$FM_HEARTBEAT_SURFACE_ENDPOINTS" in + *parent-replies*) fail "the channel leaked into the heartbeat backstop: $FM_HEARTBEAT_SURFACE_ENDPOINTS" ;; + esac + # A genuine task's captain-relevant line must keep reaching the backstop. + printf 'blocked [key=wedge]: the crew is stuck\n' >> "$dir/home/state/real-task.status" + heartbeat_scan_finds_actionable; rc=$? + [ "$rc" -eq 0 ] || fail "a genuine task's decision must surface through the heartbeat backstop" + case "$FM_HEARTBEAT_SURFACE_ENDPOINTS" in + *real-task.status*) ;; + *) fail "the heartbeat backstop must name the genuine task: $FM_HEARTBEAT_SURFACE_ENDPOINTS" ;; + esac + case "$FM_HEARTBEAT_SURFACE_ENDPOINTS" in + *parent-replies*) fail "the channel leaked into the heartbeat backstop: $FM_HEARTBEAT_SURFACE_ENDPOINTS" ;; + esac + pass "the heartbeat backstop skips a remote mate's channel and keeps its tasks" +} + +test_watcher_scan_keeps_name_shared_files_outside_remote_mates() { + local out + STATE="$PLAIN_STATE" + out=$(scan_signals) || fail "scan_signals failed over the main-home state" + printf '%s\n' "$out" | cut -f3 | grep -F 'parent-replies.status' >/dev/null \ + || fail "a main home's parent-replies.status must keep waking: $out" + # shellcheck disable=SC2034 # read by the sourced watcher's scans at call time + STATE="$LOCAL_STATE" + out=$(scan_signals) || fail "scan_signals failed over the local-mate state" + printf '%s\n' "$out" | cut -f3 | grep -F 'parent-replies.status' >/dev/null \ + || fail "a local mate's parent-replies.status must keep waking: $out" + pass "scan_signals keeps parent-replies.status outside remote mate homes" +} + +# --- unit: the away-mode daemon's heartbeat catch-all backstop -------------- + +# The daemon runs the watcher's twin catch-all scan while a home is away, so it +# needs the same exclusion. Source it in a subshell - its BASH_SOURCE guard +# skips the main loop, and the isolation keeps its function table from +# colliding with the watcher already sourced above. +daemon_heartbeat_scan() { # + local home=$1 + rm -f "$home/state/.subsuper-last-scan" + FM_TEST_LIB_SOURCED=1 FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + bash -c ' + # shellcheck source=/dev/null + . "$1/bin/fm-supervise-daemon.sh" + housekeeping "$2" + ' _ "$ROOT" "$home/state" >/dev/null 2>&1 +} + +test_daemon_heartbeat_backstop_skips_channel_in_remote_mate() { + local dir buffer + dir="$TMP_ROOT/daemon-heartbeat" + seed_remote_mate "$dir/home" + # A quiet task log keeps the first pass channel-only, so only the excluded + # channel could put anything in the escalation buffer. + printf 'note: benchmark results are in\n' > "$dir/home/state/real-task.status" + daemon_heartbeat_scan "$dir/home" + buffer=$(cat "$dir/home/state/.subsuper-escalations" 2>/dev/null || true) + case "$buffer" in *parent-replies*|*captain-hold*|*release\ branch*) + fail "the channel leaked into the daemon's catch-all scan: $buffer" ;; + esac + [ -z "$(cat "$dir/home/state/.subsuper-seen-status-parent-replies" 2>/dev/null || true)" ] \ + || fail "the daemon tracked the channel as a phantom parent-replies task" + + # A genuine task's captain-relevant line must keep reaching the backstop. + printf 'blocked [key=wedge]: the crew is stuck\n' >> "$dir/home/state/real-task.status" + daemon_heartbeat_scan "$dir/home" + buffer=$(cat "$dir/home/state/.subsuper-escalations" 2>/dev/null || true) + printf '%s\n' "$buffer" | grep -F 'real-task.status' >/dev/null \ + || fail "a genuine task's decision must still surface through the daemon backstop: $buffer" + case "$buffer" in *parent-replies*) + fail "the channel leaked into the daemon's catch-all scan: $buffer" ;; + esac + pass "the daemon's catch-all scan skips a remote mate's channel and keeps its tasks" +} + +test_daemon_heartbeat_backstop_keeps_name_shared_file_in_a_main_home() { + local dir buffer + dir="$TMP_ROOT/daemon-heartbeat-main" + seed_plain_home "$dir/home" + printf 'blocked [key=name-only]: an ordinary task file that shares the name\n' \ + > "$dir/home/state/parent-replies.status" + daemon_heartbeat_scan "$dir/home" + buffer=$(cat "$dir/home/state/.subsuper-escalations" 2>/dev/null || true) + printf '%s\n' "$buffer" | grep -F 'parent-replies.status' >/dev/null \ + || fail "a main home's parent-replies.status must keep reaching the daemon backstop: $buffer" + pass "the daemon's catch-all scan keeps parent-replies.status outside remote mate homes" +} + +# --- end to end: the real drain over a remote mate home --------------------- + +test_drain_presents_no_channel_content_in_remote_mate() { + local dir out manifest + dir="$TMP_ROOT/drain" + seed_remote_mate "$dir/home" + mkdir -p "$dir/home/data" + FM_STATE_OVERRIDE="$dir/home/state" FM_HOME="$dir/home" \ + "$ROOT/bin/fm-wake-drain.sh" > "$dir/drain.out" \ + || fail "the drain failed over a remote mate home" + out=$(cat "$dir/drain.out") + case "$out" in *parent-replies*) + fail "the drain presented the remote mate's channel: $out" ;; + esac + printf '%s\n' "$out" | grep -F 'api-shape' >/dev/null \ + || fail "the genuine task's open decision must still surface in OPEN DECISIONS: $out" + printf '%s\n' "$out" | grep -F 'benchmark results' >/dev/null \ + || fail "the genuine task's note must still surface under UNREAD STATUS: $out" + # The presentation manifest is rebuilt from the excluded snapshot, so a + # channel row an older watcher recorded must not survive the drain. + manifest=$(cat "$dir/home/state/.status-presentation-cursor" 2>/dev/null || true) + case "$manifest" in *parent-replies*) + fail "the presentation manifest still tracks the channel: $manifest" ;; + esac + pass "the real drain presents no channel content from a remote mate home" +} + +test_drain_ignores_stale_channel_records_from_an_older_watcher() { + local dir out manifest + dir="$TMP_ROOT/drain-stale" + seed_remote_mate "$dir/home" + mkdir -p "$dir/home/data" + # An older watcher folded the channel and tracked it as a task; the fixed + # drain must drop both rather than present or choke on them. + printf 'needs-decision [key=old-phantom]: folded by the unfixed watcher\n' \ + > "$dir/home/state/.parent-replies.open-decisions-cursor" + printf 'parent-replies\tstrong:1:2:3\t99\t0\n' \ + > "$dir/home/state/.status-presentation-cursor" + FM_STATE_OVERRIDE="$dir/home/state" FM_HOME="$dir/home" \ + "$ROOT/bin/fm-wake-drain.sh" > "$dir/drain.out" \ + || fail "the drain failed over stale channel records" + out=$(cat "$dir/drain.out") + case "$out" in *parent-replies*|*old-phantom*) + fail "a stale channel fold resurfaced through the drain: $out" ;; + esac + manifest=$(cat "$dir/home/state/.status-presentation-cursor" 2>/dev/null || true) + case "$manifest" in *parent-replies*) + fail "the stale manifest row survived the drain: $manifest" ;; + esac + pass "stale channel records from an older watcher are dropped, not presented" +} + +test_predicate_resolves_only_the_remote_channel +test_resolver_predicates_on_home_shape_not_name +test_remote_folds_omit_channel_and_keep_genuine_task +test_incremental_fold_omits_channel_and_keeps_genuine_task +test_channel_lines_never_reach_the_remote_unread_surface +test_name_shared_file_folds_in_a_main_home +test_name_shared_file_folds_in_a_local_mate +test_watcher_scan_skips_channel_and_keeps_task_in_remote_mate +test_heartbeat_backstop_skips_channel_in_remote_mate +test_watcher_scan_keeps_name_shared_files_outside_remote_mates +test_daemon_heartbeat_backstop_skips_channel_in_remote_mate +test_daemon_heartbeat_backstop_keeps_name_shared_file_in_a_main_home +test_drain_presents_no_channel_content_in_remote_mate +test_drain_ignores_stale_channel_records_from_an_older_watcher diff --git a/tests/fm-pending-reply.test.sh b/tests/fm-pending-reply.test.sh index 350741fca2a..777d1c97c57 100755 --- a/tests/fm-pending-reply.test.sh +++ b/tests/fm-pending-reply.test.sh @@ -201,6 +201,75 @@ test_completed_turn_no_report_triggers_one_recovery() { pass "completed turn with no report triggers exactly one recovery" } +# A mate waiting on its own open decision is never poked by the recovery; the +# recovery stays unattempted and runs once the decision closes. +test_recovery_waits_while_the_mate_has_an_open_decision() { + local home state corr hook_log + home=$(setup_parent decision-wait) + state="$home/state" + hook_log="$TMP_ROOT/decision-wait-hook.log" + : > "$hook_log" + export FM_PENDING_REPLY_NOW=2500 + mkdir -p "$home/config" + : > "$home/config/wait-no-turns" + FM_CONFIG_OVERRIDE="$home/config" + # Invoked indirectly through FM_PENDING_REPLY_SEND_HOOK. + # shellcheck disable=SC2329 + decision_wait_hook() { + printf '%s\n' "$1" >> "$hook_log" + } + export -f decision_wait_hook + export FM_PENDING_REPLY_SEND_HOOK=decision_wait_hook + + corr=$(fm_pending_reply_create "$home" "$state" "hibit" "status of phase 8") + fm_pending_reply_mark_delivered "$state" "$corr" + fm_pending_reply_observe_busy "$state" "$corr" busy + fm_pending_reply_observe_busy "$state" "$corr" idle + printf 'needs-decision [key=scope]: narrow or wide?\n' >> "$state/hibit.status" + if fm_pending_reply_send_recovery "$state" "$corr" 2>/dev/null; then + fail "recovery must wait while the mate waits on its own decision" + fi + [ ! -s "$hook_log" ] || fail "recovery poked a mate waiting on its decision" + [ "$(phase_of "$state" "$corr")" = awaiting_report ] \ + || fail "a deferred recovery must stay unattempted, got $(phase_of "$state" "$corr")" + + printf 'resolved [key=scope]: answered: narrow\n' >> "$state/hibit.status" + fm_pending_reply_send_recovery "$state" "$corr" || fail "recovery should send once the decision closes" + [ "$(wc -l < "$hook_log" | tr -d ' ')" = 1 ] || fail "expected exactly one recovery send" + unset FM_PENDING_REPLY_SEND_HOOK + unset FM_CONFIG_OVERRIDE + pass "recovery never pokes a mate waiting on its own decision, and runs once it closes" +} + +# Without the flag, an open decision does not hold the recovery. +test_recovery_sends_during_an_open_decision_without_the_flag() { + local home state corr hook_log + home=$(setup_parent decision-wait-off) + state="$home/state" + hook_log="$TMP_ROOT/decision-wait-off-hook.log" + : > "$hook_log" + mkdir -p "$home/config" + FM_CONFIG_OVERRIDE="$home/config" + export FM_PENDING_REPLY_NOW=2500 + # shellcheck disable=SC2329 + decision_wait_off_hook() { + printf '%s\n' "$1" >> "$hook_log" + } + export -f decision_wait_off_hook + export FM_PENDING_REPLY_SEND_HOOK=decision_wait_off_hook + corr=$(fm_pending_reply_create "$home" "$state" "hibit" "status of phase 8") + fm_pending_reply_mark_delivered "$state" "$corr" + fm_pending_reply_observe_busy "$state" "$corr" busy + fm_pending_reply_observe_busy "$state" "$corr" idle + printf 'needs-decision [key=scope]: narrow or wide?\n' >> "$state/hibit.status" + fm_pending_reply_send_recovery "$state" "$corr" \ + || fail "recovery should send while a decision is open when the flag is absent" + [ "$(wc -l < "$hook_log" | tr -d ' ')" = 1 ] || fail "expected the recovery to send" + unset FM_PENDING_REPLY_SEND_HOOK + unset FM_CONFIG_OVERRIDE + pass "recovery sends during an open decision when config/wait-no-turns is absent" +} + test_recovery_grace_measures_from_turn_completion() { local home state corr hook_log lines home=$(setup_parent grace-from-completion) @@ -1927,6 +1996,8 @@ test_escalated_undelivered_correlation_stays_retryable() { test_normal_correlated_reply_resolves_once test_completed_turn_no_report_triggers_one_recovery +test_recovery_waits_while_the_mate_has_an_open_decision +test_recovery_sends_during_an_open_decision_without_the_flag test_recovery_grace_measures_from_turn_completion test_recovery_fresh_status_read_resolves_before_firing test_partial_resolve_write_blocks_firing diff --git a/tests/fm-pi-branch-extension.test.sh b/tests/fm-pi-branch-extension.test.sh index a541c1ac706..78b0d72989c 100644 --- a/tests/fm-pi-branch-extension.test.sh +++ b/tests/fm-pi-branch-extension.test.sh @@ -557,6 +557,7 @@ const mainUserMessages = []; const mainTools = []; const renderers = new Map(); const entryRenderers = new Map(); +const markdownTransformers = []; const mainEntries = []; const mainSessionManager = { getSessionFile: () => `${home}/main.jsonl`, @@ -581,6 +582,9 @@ const pi = { registerEntryRenderer(customType, renderer) { entryRenderers.set(customType, renderer); }, + registerMarkdownTransformer(transformer) { + markdownTransformers.push(transformer); + }, appendEntry(customType, data) { activeMainSession.getEntries().push({ type: "custom", customType, data }); }, @@ -1263,14 +1267,47 @@ test_captain_outcome_processing_turn_is_sequence_keyed_and_re_presented() { PLUGIN="$repo/.pi/extensions/fm-branch-supervision.ts" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ DRIVER_PRELUDE="$DRIVER_PRELUDE" node --input-type=module > "$TMP_ROOT/node-output" 2>&1 <<'EOF' const prelude = process.env.DRIVER_PRELUDE; -await eval(`(async () => { ${prelude}; globalThis.__t = { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home, bus }; })()`); -const { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home, bus } = globalThis.__t; +await eval(`(async () => { ${prelude}; globalThis.__t = { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home, bus, markdownTransformers }; })()`); +const { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home, bus, markdownTransformers } = globalThis.__t; import { writeFileSync } from "node:fs"; let requestsFloor = 0; const requests = () => sentToMain.filter((sent) => sent.message.customType === "fm-branch-process").slice(requestsFloor); const unprocessedSeqs = () => outcomeScript(["unprocessed"]).split("\n").filter(Boolean).map((line) => JSON.parse(line).seq); -const runOf = async (fn) => { await fire("agent_start", {}); await fn?.(); await fire("agent_end", {}); await fire("agent_settled", {}); }; +let consumedRequests = 0; +const consumeRequest = async (userFirst = false) => { + const pending = requests().at(-1); + if (!pending || consumedRequests === requests().length) return; + consumedRequests = requests().length; + if (userFirst || pending.options.deliverAs === "nextTurn") { + await fire("message_start", { message: { role: "user", content: "A new question" } }); + } + await fire("message_start", { message: { role: "custom", ...pending.message } }); +}; +const runOf = async (fn) => { + await fire("agent_start", {}); + await fire("turn_start", {}); + await consumeRequest(); + await fn?.(); + await fire("agent_end", {}); + await fire("agent_settled", {}); +}; +const render = (text, isStreaming = true, messageType = "assistant") => markdownTransformers.reduce( + (value, transform) => transform(value, { messageType, isStreaming, availableWidth: 80 }), text, +); +const finish = async (text, extra = []) => { + const message = { role: "assistant", content: [...(text ? [{ type: "text", text }] : []), ...extra], usage: { totalTokens: 7 }, stopReason: "stop" }; + await fire("message_start", { message }); + const replacement = await fire("message_end", { message }); + const stored = replacement?.message ?? message; + mainEntries.push({ type: "message", message: stored }); + if (stored.usage !== message.usage) throw new Error("suppression lost usage accounting"); + return stored; +}; +const visibleFinals = () => mainEntries.filter((entry) => entry.type === "message" && entry.message.role === "assistant") + .flatMap((entry) => entry.message.content.filter((part) => part.type === "text").map((part) => part.text)); +const priorResult = "Completed the requested work. The checks passed and the result is ready for review."; +await finish(priorResult); // A home with a delivered captain row and no processed marker (upgraded from // before the marker existed, or switched from the supervision host, whose @@ -1320,20 +1357,41 @@ if (request.options.triggerTurn !== true || request.options.deliverAs !== "follo if (!request.message.content.includes(`[seq ${seq}, recorded 0m ago] task-d: ${decision}`)) throw new Error(`the request lost its key or summary: ${request.message.content}`); if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error(`delivery did not leave seq ${seq} unprocessed: ${unprocessedSeqs()}`); -// Case A (timeline report 2026-08-31): the turn returns an EMPTY assistant -// message. The processed marker must not move, and the same sequence is -// presented again at the run boundary. -await runOf(() => mainEntries.push({ type: "message", message: { role: "assistant", content: [] } })); -if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error("an empty answer advanced the processed marker"); -if (requests().length !== 2) throw new Error(`an empty answer did not re-present the outcome: ${requests().length} requests`); -if (requests()[1].options.triggerTurn !== true) throw new Error("the first re-presentation must open its own turn"); +// The first presentation carries the one visible response for this outcome, +// even when it forgets to acknowledge. +await runOf(async () => { + if (render("Captain, task-d needs your call.") !== "Captain, task-d needs your call.") { + throw new Error("the first processing presentation hid its response while streaming"); + } + await finish("Captain, task-d needs your call."); +}); +if (visibleFinals().at(-1) !== "Captain, task-d needs your call.") throw new Error("the first processing presentation lost its final"); +if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error("an unacknowledged answer advanced the processed marker"); +if (requests().length !== 2 || requests()[1].options.triggerTurn !== true) throw new Error("the first re-presentation must open its own turn"); if (!requests()[1].message.content.includes(`[seq ${seq}, recorded 0m ago] task-d: ${decision}`)) throw new Error("the re-presentation changed the outcome"); - -// Case B: the turn repeats an unrelated prior answer. Same result: the marker -// holds, and the request is presented again - now riding the captain's next -// prompt because the triggered budget for this sequence set is spent. -await runOf(() => mainEntries.push({ type: "message", message: { role: "assistant", content: "The retry safe-stopped; diagnosis is underway." } })); -if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error("an unrelated answer advanced the processed marker"); +// The hidden retry repeats this set's prior reply instead of acknowledging. +// Exercise Pi's public message replacement and Markdown transformer surfaces: +// buffer streaming until the complete reply can be compared, then keep every +// differing reply, even one already visible outside this processing set. +await runOf(async () => { + if (render("Captain, task-d needs your call.") !== "" || render("prior reasoning", true, "assistant-thinking") !== "") { + throw new Error("a processing retry reply leaked while streaming"); + } + if (render(priorResult, false) !== priorResult || render("A question", true, "user") !== "A question") { + throw new Error("silencing a retry hid an earlier final or a user message"); + } + const repeated = await finish(" \nCaptain, task-d needs your call. \n"); + if (repeated.content.length) throw new Error("the exact trimmed repeat retained visible content"); + await finish(priorResult); + await finish("Captain, task-d needs your call!"); + const empty = await finish(""); + const whitespace = await finish(" \n\t"); + if (empty.content.length || whitespace.content.length) throw new Error("an empty retry retained visible content"); +}); +if (JSON.stringify(visibleFinals()) !== JSON.stringify([priorResult, "Captain, task-d needs your call.", priorResult, "Captain, task-d needs your call!"])) { + throw new Error(`retry comparison hid new prose or exposed a repeat: ${JSON.stringify(visibleFinals())}`); +} +if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error("a repeated or empty answer advanced the processed marker"); if (requests().length !== 3) throw new Error(`an unrelated answer did not re-present the outcome: ${requests().length} requests`); if (requests()[2].options.deliverAs !== "nextTurn" || requests()[2].options.triggerTurn) { throw new Error(`after the triggered budget the request must ride the next prompt: ${JSON.stringify(requests()[2].options)}`); @@ -1342,7 +1400,11 @@ if (requests()[2].options.deliverAs !== "nextTurn" || requests()[2].options.trig await fire("agent_settled", {}); if (requests().length !== 3) throw new Error("a duplicate next-turn copy was queued"); // The captain's next prompt consumes that copy; settling unacknowledged queues one more. -await runOf(() => mainEntries.push({ type: "message", message: { role: "assistant", content: "Captain, shipshape." } })); +await runOf(async () => { + if (render("The new answer") !== "The new answer") throw new Error("a nextTurn request hid the user response"); + await finish("The new answer"); +}); +if (visibleFinals().at(-1) !== "The new answer") throw new Error("a nextTurn request removed the user final"); if (requests().length !== 4 || requests()[3].options.deliverAs !== "nextTurn") throw new Error("the outcome stopped being re-presented on later prompts"); if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error("a paraphrase advanced the processed marker"); @@ -1354,6 +1416,32 @@ if (mainEntries.filter((entry) => entry.customType === "fm-branch-visible-outcom throw new Error("re-presentation duplicated the visible entry"); } +const call = { type: "toolCall", id: "ack-call", name: "fm_branch_processed", arguments: { through: seq } }; +const thinking = { type: "thinking", thinking: "reasoning for the tool", thinkingSignature: "provider-signature" }; +// The replacement's first presentation keeps prose sent alongside the +// acknowledgement call; this run's call is left unexecuted. +await runOf(async () => { + const firstWithTool = await finish("Captain, task-d still needs your call.", [thinking, call]); + if (firstWithTool.content.length !== 3 || firstWithTool.content[0].text !== "Captain, task-d still needs your call.") { + throw new Error("the first presentation dropped prose sent alongside its acknowledgement"); + } +}); +if (visibleFinals().at(-1) !== "Captain, task-d still needs your call.") throw new Error("the first presentation after replacement lost its final"); +if (requests().length !== 6 || requests()[5].options.triggerTurn !== true) throw new Error("the replacement did not retry the unacknowledged outcome"); +await fire("agent_start", {}); +await fire("turn_start", {}); +await consumeRequest(); +await finish("stale after reload"); +if (visibleFinals().at(-1) !== "stale after reload") throw new Error("session replacement hid differing retry prose"); +const repeatedAfterReload = await finish("stale after reload"); +if (repeatedAfterReload.content.length) throw new Error("session replacement lost retry comparison"); +const withTool = await finish("stale after reload", [thinking, call]); +if (withTool.content.length !== 3 || withTool.content[0].text !== "stale after reload" || withTool.content[1] !== thinking || withTool.content[2] !== call) { + throw new Error("retry comparison dropped prose, signed reasoning, or the acknowledgement call"); +} +await fire("turn_start", {}); +if (render("still unacknowledged") !== "") throw new Error("a tool continuation released suppression before acknowledgement"); + // Only the sequence-bound acknowledgement closes it. const nativeTools = new Map(); const messageTypes = new Set(); @@ -1376,9 +1464,13 @@ if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error const tooFar = await processed.execute("ack-too-far", { through: seq + 100 }, undefined, undefined, {}); if (!tooFar.isError) throw new Error("an acknowledgement beyond the read cursor was accepted"); if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seq])) throw new Error("a refused acknowledgement moved the marker"); +if (render("still refused") !== "") throw new Error("a refused acknowledgement released suppression"); const ack = await processed.execute("ack", { through: seq }, undefined, undefined, {}); if (ack.isError) throw new Error(`acknowledgement failed: ${JSON.stringify(ack)}`); if (unprocessedSeqs().length !== 0) throw new Error("the acknowledgement did not close the sequence"); +if (render("A newly processed response") !== "A newly processed response") throw new Error("successful acknowledgement did not release the response"); +await finish("A newly processed response"); +if (visibleFinals().at(-1) !== "A newly processed response") throw new Error("the acknowledged outcome lost its response"); const before = requests().length; await runOf(); if (requests().length !== before) throw new Error("an acknowledged outcome was presented again"); @@ -1431,9 +1523,13 @@ await runOf(); if (requests().length !== beforePairRepeat + 1 || requests().at(-1).options.triggerTurn !== true) { throw new Error("the second presentation of the widened sequence set did not open its own turn"); } +await fire("agent_start", {}); +await fire("turn_start", {}); +await consumeRequest(); const partial = await processed.execute("ack-partial", { through: seqE }, undefined, undefined, {}); if (partial.isError) throw new Error(`partial acknowledgement failed: ${JSON.stringify(partial)}`); if (JSON.stringify(unprocessedSeqs()) !== JSON.stringify([seqF])) throw new Error(`a partial acknowledgement did not keep the newer sequence open: ${unprocessedSeqs()}`); +if (render("partial response") !== "") throw new Error("partial acknowledgement released the remaining outcome's retry"); const beforeF = requests().length; await runOf(); if ( @@ -1447,6 +1543,62 @@ if ( const done = await processed.execute("ack-final", { through: seqF }, undefined, undefined, {}); if (done.isError || unprocessedSeqs().length !== 0) throw new Error("the final acknowledgement did not close the newer sequence"); +// A follow-up queued while main is busy must not hide the answer already +// underway. Suppression starts only when Pi consumes the custom message. +await fire("agent_start", {}); +await fire("turn_start", {}); +await fire("message_start", { message: { role: "user", content: "An active user request" } }); +await report2.execute("busy-report", { task: "task-g", verdict: "captain", summary: "A new decision" }, undefined, undefined, {}); +await finish("The busy user answer"); +if (visibleFinals().at(-1) !== "The busy user answer") throw new Error("queueing a retry hid an in-flight user answer"); +await fire("turn_start", {}); +await consumeRequest(); +await finish("Captain, task-g needs your call."); +if (visibleFinals().at(-1) !== "Captain, task-g needs your call.") throw new Error("a consumed busy follow-up hid its first presentation"); +// User steering in that same turn must immediately recover ordinary output. +await fire("message_start", { message: { role: "user", content: "A steering question" } }); +await finish("The steering answer"); +if (visibleFinals().at(-1) !== "The steering answer") throw new Error("processing suppression hid a steering response"); +await fire("agent_end", {}); +await fire("agent_settled", {}); +// Pi can also batch the user before the custom follow-up in one turn. +await fire("agent_start", {}); +await fire("turn_start", {}); +await consumeRequest(true); +await finish("The batched user answer"); +if (visibleFinals().at(-1) !== "The batched user answer") throw new Error("processing suppression hid a user batched before the custom message"); +const latestSeq = unprocessedSeqs().at(-1); +await processed.execute("ack-busy", { through: latestSeq }, undefined, undefined, {}); +await fire("agent_end", {}); +await fire("agent_settled", {}); + +// A first reply can be empty or unrelated: a retry that finally handles the +// outcome must stay visible. Reusing the same response across new sequence +// sets must not make it a duplicate, and only the tool closes each outcome. +for (const initial of ["", "Unrelated prior acknowledgment"]) { + await report2.execute("new-set", { task: "task-new", verdict: "captain", summary: "Another decision" }, undefined, undefined, {}); + const newSeq = unprocessedSeqs().at(-1); + const startCount = visibleFinals().length; + await runOf(async () => { + const firstReply = await finish(initial); + if ((firstReply.content[0]?.text ?? "") !== initial) throw new Error("first presentation changed its output"); + }); + const newAnswer = "Handled the new outcome."; + await runOf(async () => { + await finish(newAnswer); + if (visibleFinals().at(-1) !== newAnswer) throw new Error("the first real handling on a retry was hidden"); + const repeat = await finish(newAnswer); + if (repeat.content.length) throw new Error("a repeated retry final was retained"); + }); + if (visibleFinals().length !== startCount + (initial ? 2 : 1)) throw new Error("sequence comparison lost or duplicated a final"); + if (!unprocessedSeqs().includes(newSeq) || requests().at(-1).options.deliverAs !== "nextTurn") { + throw new Error("an empty, unrelated, or differing reply closed the durable obligation"); + } + await processed.execute("ack-new-set", { through: newSeq }, undefined, undefined, {}); + if (unprocessedSeqs().length) throw new Error("acknowledgement did not close the new set"); + await fire("agent_settled", {}); +} + // A session that does not own the fleet lock cannot acknowledge anything. writeFileSync(`${home}/state/.lock`, "1\n"); const foreign = await processed.execute("ack-foreign", { through: seqF }, undefined, undefined, {}); diff --git a/tests/fm-pi-branch-live-e2e.test.sh b/tests/fm-pi-branch-live-e2e.test.sh index 76a49a4995d..376c8b0ee52 100644 --- a/tests/fm-pi-branch-live-e2e.test.sh +++ b/tests/fm-pi-branch-live-e2e.test.sh @@ -991,3 +991,112 @@ if [ "$status" -ne 0 ] || [ "$out" != "STREAM_OK" ]; then fail "real-SDK streaming-time watcher delivery guard failed against pi-coding-agent $PI_VERSION: $out" fi pass "real Pi SDK $PI_VERSION queues a streaming-time watcher wake without before_agent_start, keeps the successor chain, and surfaces consumption of both follow-ups" + +# The first processing presentation keeps its visible response, while the +# hidden retry drops only empty or exact-repeat finals through the real event +# runner, stock assistant renderer, and persistence, including after reopen. +for retry_case in repeated differing empty first-empty; do +retryhome="$TMP_ROOT/retry-home-$retry_case" +retrydir="$TMP_ROOT/retry-agent-dir-$retry_case" +mkdir -p "$retryhome/state" "$retryhome/config" "$retrydir" +cp "$streamdir/models.json" "$retrydir/models.json" +BRANCH_PLUGIN="$repo/.pi/extensions/fm-branch-supervision.ts" \ + FM_HOME="$retryhome" FM_ROOT_OVERRIDE="$ROOT" RETRY_CASE="$retry_case" \ + PI_CODING_AGENT_DIR="$retrydir" PI_PACKAGE_DIR="$PI_PACKAGE_DIR" \ + node --input-type=module > "$TMP_ROOT/retry-output" 2>&1 <<'EOF' +import { readFileSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +const home = process.env.FM_HOME; +const pkg = resolve(process.env.PI_PACKAGE_DIR); +const { DefaultResourceLoader, ModelRegistry, ModelRuntime, SessionManager, SettingsManager, createAgentSession, initTheme } = + await import(pathToFileURL(`${pkg}/dist/index.js`).href); +const { AssistantMessageComponent } = await import(pathToFileURL(`${pkg}/dist/modes/interactive/components/assistant-message.js`).href); +initTheme("dark"); +writeFileSync(`${home}/state/.lock`, `${process.pid}\n`); +const outcome = (...args) => { + const result = spawnSync("bash", [`${process.env.FM_ROOT_OVERRIDE}/bin/fm-branch-outcome.sh`, ...args], { encoding: "utf8" }); + if (result.status !== 0) throw new Error(result.stderr); + return result.stdout.trim(); +}; +outcome("processed-init"); +const seq = Number(outcome("append", "--task", "example", "--verdict", "captain", "--summary", "A decision is needed")); +const original = "The requested result is complete and verified."; +const handled = process.env.RETRY_CASE === "first-empty" ? "" : "The example task needs a decision."; +const retryReply = process.env.RETRY_CASE === "repeated" ? handled : process.env.RETRY_CASE === "empty" ? "" : original; +const expectedFinals = [original, ...(handled ? [handled] : []), ...(retryReply && retryReply !== handled ? [retryReply] : [])]; +let completions = 0; +let settled = 0; +const failures = []; +const chunk = (text, finish = null) => `data: ${JSON.stringify({ + id: "local-retry-probe", object: "chat.completion.chunk", created: 1, model: "fm-live-stream-model", + choices: [{ index: 0, delta: text ? { role: "assistant", content: text } : {}, finish_reason: finish }], +})}\n\n`; +globalThis.fetch = async (input) => { + const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + if (!url.startsWith("https://fm-live-stream.invalid/")) throw new Error(`unexpected network request: ${url}`); + completions += 1; + const text = completions === 1 ? original : completions === 2 ? handled : completions === 3 ? retryReply : "The new user answer."; + return new Response(chunk(text) + chunk(null, "stop") + "data: [DONE]\n\n", { + headers: { "content-type": "text/event-stream" }, + }); +}; +const agentDir = process.env.PI_CODING_AGENT_DIR; +const settings = SettingsManager.create(home, agentDir); +const loader = new DefaultResourceLoader({ + cwd: home, agentDir, settingsManager: settings, + additionalExtensionPaths: [process.env.BRANCH_PLUGIN], + extensionFactories: [{ name: "retry-probe", factory: (pi) => { + pi.on("agent_settled", () => { settled += 1; }); + } }], + noSkills: true, noPromptTemplates: true, noThemes: true, noContextFiles: true, +}); +await loader.reload(); +const runtime = await ModelRuntime.create({ authPath: `${agentDir}/auth.json`, modelsPath: `${agentDir}/models.json` }); +const registry = new ModelRegistry(runtime); +await registry.refresh(); +const manager = SessionManager.create(home, `${home}/sessions`); +const { session } = await createAgentSession({ + cwd: home, sessionManager: manager, settingsManager: settings, resourceLoader: loader, + modelRuntime: runtime, model: registry.find("fm-live-stream", "fm-live-stream-model"), noTools: "builtin", +}); +let streamedRetries = 0; +const unsubscribe = session.subscribe((event) => { + if (event.type !== "message_update" || completions !== 3) return; + streamedRetries += 1; + const component = new AssistantMessageComponent(undefined, false, undefined, undefined, 0, session.extensionRunner.getMarkdownTransformers()); + component.updateContent(event.message, true); + const rendered = component.render(120).join("\n"); + if (retryReply && rendered.includes(retryReply)) failures.push("retry prose leaked from the streaming renderer"); +}); +await session.prompt("Finish the requested work."); +for (let i = 0; i < 600 && settled < 3; i += 1) await new Promise((done) => setTimeout(done, 50)); +if (settled !== 3 || completions !== 3) throw new Error(`retry chain did not settle: ${settled} settlements, ${completions} completions`); +if ((retryReply && streamedRetries === 0) || failures.length) throw new Error(`streaming suppression failed: ${streamedRetries} updates, ${failures}`); +const assistantText = (messages) => messages.filter((message) => message.role === "assistant") + .flatMap((message) => message.content.filter((part) => part.type === "text").map((part) => part.text)); +if (JSON.stringify(assistantText(session.messages)) !== JSON.stringify(expectedFinals)) throw new Error(`agent state lost new prose or retained an exact repeat: ${JSON.stringify(assistantText(session.messages))}`); +const reopened = SessionManager.open(manager.getSessionFile(), `${home}/sessions`); +if (JSON.stringify(assistantText(reopened.buildSessionContext().messages)) !== JSON.stringify(expectedFinals)) throw new Error("reopened session lost new prose or retained an exact repeat"); +if (!outcome("unprocessed").includes(`"seq":${seq}`)) throw new Error("silent retries advanced the processed marker"); +await session.prompt("A new user question."); +if (assistantText(session.messages).at(-1) !== "The new user answer.") throw new Error("a nextTurn retry hid the new user answer"); +const acknowledged = await session.getToolDefinition("fm_branch_processed").execute("ack", { through: seq }, undefined, undefined, {}); +if (acknowledged.isError || outcome("unprocessed")) throw new Error("the outcome could not be acknowledged after silent retries"); +const entries = readFileSync(manager.getSessionFile(), "utf8").split("\n").filter(Boolean).map((line) => JSON.parse(line)); +if (assistantText(entries.filter((entry) => entry.type === "message").map((entry) => entry.message)).length !== expectedFinals.length + 1) { + throw new Error("persisted finals do not match the retained handling outcomes"); +} +unsubscribe(); +session.dispose(); +console.log("RETRY_OK"); +process.exit(0); +EOF +status=$? +out=$(cat "$TMP_ROOT/retry-output") +if [ "$status" -ne 0 ] || [ "$out" != "RETRY_OK" ]; then + fail "real-SDK processing retry visibility guard ($retry_case) failed against pi-coding-agent $PI_VERSION: $out" +fi +done +pass "real Pi SDK $PI_VERSION suppresses only empty or exact-repeat retry finals, retains first and differing replies after reopen, buffers retry streaming, and keeps outcomes retryable" diff --git a/tests/fm-pi-seeded-home-trust-live-e2e.test.sh b/tests/fm-pi-seeded-home-trust-live-e2e.test.sh new file mode 100755 index 00000000000..93714ce2d0b --- /dev/null +++ b/tests/fm-pi-seeded-home-trust-live-e2e.test.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# Live guard for fm-spawn's Pi seeded-secondmate --approve preflight. +# +# Reproduces the Pi "Trust project folder?" stall on a freshly seeded +# secondmate-shaped home (tracked .pi/extensions + .fm-secondmate-home) under a +# disposable PI_CODING_AGENT_DIR, then proves the spawn-side --approve flag +# clears that stall without rewriting the disposable trust store. An unseeded +# path without --approve still prompts. +# +# Token-free: never submits a prompt and never answers the dialog with Enter. +# Uses Escape / kill-server only. Never touches ~/.pi. +# +# Policy: default-on wherever pi and tmux are installed (fm_live_gate). +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +REAL_TMUX=$(command -v tmux 2>/dev/null || true) +SOCKET="fm-pi-seeded-trust-$$" +LAB= +CHECKED=0 + +note() { printf '# %s\n' "$1"; } +pass() { printf 'ok - %s\n' "$1"; } + +cleanup() { + [ -z "${REAL_TMUX:-}" ] || "$REAL_TMUX" -L "$SOCKET" kill-server >/dev/null 2>&1 || true + [ -z "${LAB:-}" ] || rm -rf -- "$LAB" +} +trap cleanup EXIT + +fail() { printf 'not ok - %s\n' "$1" >&2; exit 1; } + +fm_live_gate default-on FM_PI_SEEDED_HOME_TRUST_LIVE pi tmux + +PI_BIN=$(command -v pi) || fail "pi missing after live gate" +VERSION_OUT=$("$PI_BIN" --version 2>&1) || fail "pi --version failed: $VERSION_OUT" +note "live pi version: $VERSION_OUT" + +if ! "$PI_BIN" --help 2>&1 | grep -Eq -- '(^|[[:space:]])--approve([^[:alnum:]_-]|$)'; then + note "installed pi does not advertise --approve; spawn omits the flag and this guard has nothing to prove" + echo "# fm-pi-seeded-home-trust-live-e2e: skipped (no --approve on installed pi)" + exit 0 +fi + +LAB=$(mktemp -d "${TMPDIR:-/tmp}/fm-pi-seeded-trust.XXXXXX") || fail "could not create disposable lab" +PI_DIR="$LAB/pi-agent" +mkdir -p "$PI_DIR" +printf '{}\n' > "$PI_DIR/trust.json" +TRUST_BEFORE=$(cat "$PI_DIR/trust.json") + +seed_home() { # + local dir=$1 id=$2 + mkdir -p "$dir/.pi/extensions" "$dir/data" "$dir/state" "$dir/config" + printf '%s\n' "$id" > "$dir/.fm-secondmate-home" + printf 'export default function () {}\n' > "$dir/.pi/extensions/fm-primary-turnend-guard.ts" + printf 'export default function () {}\n' > "$dir/.pi/extensions/fm-primary-pi-watch.ts" + printf '# test charter\n' > "$dir/data/charter.md" +} + +capture_until() { # + local session=$1 expect=$2 seconds=$3 out=$4 + local target="$session:w" tail='' i limit + limit=$((seconds * 5)) + for ((i = 0; i < limit; i++)); do + tail=$("$REAL_TMUX" -L "$SOCKET" capture-pane -p -t "$target" -S -80 2>/dev/null) || true + if printf '%s' "$tail" | grep -qiE "$expect"; then + printf '%s' "$tail" > "$out" + return 0 + fi + sleep 0.2 + done + printf '%s' "$tail" > "$out" + return 1 +} + +# --- 1. Fresh seeded home WITHOUT --approve stalls on the trust dialog ------ +SEED="$LAB/seeded-stall" +seed_home "$SEED" lab-sm-stall +"$REAL_TMUX" -L "$SOCKET" new-session -d -s stall -n w -c "$SEED" -- \ + env HOME="$LAB/home-stall" PI_CODING_AGENT_DIR="$PI_DIR" PI_OFFLINE=1 \ + "$PI_BIN" --no-session --no-skills --no-prompt-templates \ + || fail "could not launch pi without --approve" +if ! capture_until stall 'Trust project folder' 15 "$LAB/pane-stall.txt"; then + fail "seeded home without --approve never showed Trust project folder? within 15s: +$(cat "$LAB/pane-stall.txt")" +fi +"$REAL_TMUX" -L "$SOCKET" send-keys -t stall:w Escape >/dev/null 2>&1 || true +"$REAL_TMUX" -L "$SOCKET" kill-session -t stall >/dev/null 2>&1 || true +CHECKED=$((CHECKED + 1)) +pass "fresh seeded Pi secondmate-shaped home stalls on Trust project folder? without --approve" + +# --- 2. Same shape WITH --approve starts past the dialog; trust.json intact - +SEED2="$LAB/seeded-approve" +seed_home "$SEED2" lab-sm-approve +printf '{}\n' > "$PI_DIR/trust.json" +"$REAL_TMUX" -L "$SOCKET" new-session -d -s approve -n w -c "$SEED2" -- \ + env HOME="$LAB/home-approve" PI_CODING_AGENT_DIR="$PI_DIR" PI_OFFLINE=1 \ + "$PI_BIN" --approve --no-session --no-skills --no-prompt-templates \ + || fail "could not launch pi with --approve" +if ! capture_until approve 'fm-primary-turnend-guard|fm-primary-pi-watch|No models available|escape interrupt' 15 \ + "$LAB/pane-approve.txt"; then + fail "seeded home with --approve never reached a post-trust TUI within 15s: +$(cat "$LAB/pane-approve.txt")" +fi +if printf '%s' "$(cat "$LAB/pane-approve.txt")" | grep -qiE 'Trust project folder'; then + fail "seeded home with --approve still showed Trust project folder?: +$(cat "$LAB/pane-approve.txt")" +fi +"$REAL_TMUX" -L "$SOCKET" send-keys -t approve:w Escape >/dev/null 2>&1 || true +"$REAL_TMUX" -L "$SOCKET" kill-session -t approve >/dev/null 2>&1 || true +TRUST_AFTER=$(cat "$PI_DIR/trust.json") +[ "$TRUST_AFTER" = "$TRUST_BEFORE" ] || [ "$TRUST_AFTER" = '{}' ] \ + || fail " --approve rewrote the disposable trust store: before=$TRUST_BEFORE after=$TRUST_AFTER" +CHECKED=$((CHECKED + 1)) +pass "seeded home with --approve starts past the trust dialog without rewriting trust.json" + +# --- 3. Unseeded path without --approve still prompts ----------------------- +UNSEEDED="$LAB/unseeded" +mkdir -p "$UNSEEDED/.pi/extensions" +printf 'export default function () {}\n' > "$UNSEEDED/.pi/extensions/dummy.ts" +printf '{}\n' > "$PI_DIR/trust.json" +"$REAL_TMUX" -L "$SOCKET" new-session -d -s unseeded -n w -c "$UNSEEDED" -- \ + env HOME="$LAB/home-unseeded" PI_CODING_AGENT_DIR="$PI_DIR" PI_OFFLINE=1 \ + "$PI_BIN" --no-session --no-skills --no-prompt-templates \ + || fail "could not launch pi on an unseeded path" +if ! capture_until unseeded 'Trust project folder' 15 "$LAB/pane-unseeded.txt"; then + fail "unseeded path without --approve never showed Trust project folder? within 15s: +$(cat "$LAB/pane-unseeded.txt")" +fi +"$REAL_TMUX" -L "$SOCKET" send-keys -t unseeded:w Escape >/dev/null 2>&1 || true +"$REAL_TMUX" -L "$SOCKET" kill-session -t unseeded >/dev/null 2>&1 || true +CHECKED=$((CHECKED + 1)) +pass "unseeded path without --approve still prompts on Trust project folder?" + +[ "$CHECKED" -ge 3 ] || fail "guard checked nothing useful (checked=$CHECKED)" +echo "# all fm-pi-seeded-home-trust-live-e2e checks passed ($CHECKED)" diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index b9199ad24a7..b59fe6dad8d 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -1660,6 +1660,781 @@ EOF pass "Pi refused handling handshake is classified and not swallowed" } +test_pi_confirm_failure_retires_arm_with_distinct_watcher_pid() { + local repo home plugin log stop retired out status + repo="$TMP_ROOT/pi-confirm-distinct-root" + home="$TMP_ROOT/pi-confirm-distinct-home" + log="$TMP_ROOT/pi-confirm-distinct.log" + stop="$TMP_ROOT/pi-confirm-distinct.stop" + retired="$TMP_ROOT/pi-confirm-distinct.retired" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'refused generation=%s watcher=%s\n' "$2" "$4" >> "${FM_ARM_LOG:?}" + exit 1 +fi +printf 'arm=%s predecessor=%s\n' "$$" "${FM_WATCH_PREDECESSOR_ARM_PID:-none}" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: synthetic actionable close\n' + exit 0 +fi +sleep 0.02 & dead=$!; wait "$dead" 2>/dev/null || true +if [ "$dead" = "$$" ]; then dead=1; fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-distinct\n' "$dead" +trap 'printf "retired\n" > "${FM_RETIRED_FILE:?}"; exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" FM_RETIRED_FILE="$retired" node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +let prompt = ""; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + prompt += message; + }, +}; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +await tool.execute("tool-call-confirm-distinct", {}, undefined, undefined, {}); +for (let i = 0; i < 250 && !prompt.includes("handling delivery confirmation was rejected"); i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); +} +if (!prompt.includes("handling delivery confirmation was rejected")) { + throw new Error(`failed handshake was swallowed: ${prompt}`); +} +let retired = false; +for (let i = 0; i < 250 && !retired; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); + retired = existsSync(process.env.FM_RETIRED_FILE); +} +if (!retired) { + const rows = existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n") + : []; + throw new Error(`broken arm survived a failed confirmation with a distinct watcher pid: ${rows.join(" | ")}`); +} +const rows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); +const armRows = rows.filter((row) => row.startsWith("arm=")); +if (armRows.length !== 2) throw new Error(`expected one successor arm, got ${armRows.length}: ${rows.join(" | ")}`); +const watcherPid = rows.find((row) => row.startsWith("refused "))?.split("watcher=")[1]; +const armPid = armRows[1].split(" ")[0].slice("arm=".length); +if (!watcherPid || watcherPid === armPid) { + throw new Error(`fixture did not use distinct arm and watcher pids: ${rows.join(" | ")}`); +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi must retire the failed arm when the watcher pid differs from the arm pid: $out" + [ -z "$out" ] || fail "Pi confirm-distinct test printed output: $out" + pass "Pi confirm failure retires the named arm with distinct watcher pid" +} + +# A failed confirmation retires only the arm its own recovery token names. +# Here the restored successor has already exited (its stdout still held open, +# so its close never fires) and a manual repair has started a newer arm before +# the successor reports ready; the confirmation then fails with the +# successor's dead watcher pid, and the newer healthy arm must survive. +test_pi_confirm_failure_spares_a_newer_arm() { + local repo home plugin log stop go retired out status + repo="$TMP_ROOT/pi-confirm-newer-root" + home="$TMP_ROOT/pi-confirm-newer-home" + log="$TMP_ROOT/pi-confirm-newer.log" + stop="$TMP_ROOT/pi-confirm-newer.stop" + go="$TMP_ROOT/pi-confirm-newer.go" + retired="$TMP_ROOT/pi-confirm-newer.retired" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'refused generation=%s watcher=%s\n' "$2" "$4" >> "${FM_ARM_LOG:?}" + exit 1 +fi +printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: synthetic actionable close\n' + exit 0 +fi +if [ "$count" -eq 2 ]; then + sleep 0 & dead=$!; wait "$dead" 2>/dev/null || true + ( + while [ ! -e "${FM_GO_FILE:?}" ]; do sleep 0.02; done + printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-stale\n' "$dead" + while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.05; done + ) & + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-newer\n' "$$" +trap 'printf "retired\n" > "${FM_RETIRED_FILE:?}"; exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" FM_GO_FILE="$go" FM_RETIRED_FILE="$retired" node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +let prompt = ""; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + prompt += message; + }, +}; +const armRows = () => existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n").filter((row) => row.startsWith("arm=")) + : []; +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +await tool.execute("tool-call-confirm-newer-first", {}, undefined, undefined, {}); +for (let i = 0; i < 250 && armRows().length < 2; i += 1) await sleep(20); +if (armRows().length < 2) throw new Error("restoration never started a successor"); +const successorPid = Number(armRows()[1].slice("arm=".length)); +let dead = false; +for (let i = 0; i < 250 && !dead; i += 1) { + try { + process.kill(successorPid, 0); + await sleep(20); + } catch { + dead = true; + } +} +if (!dead) throw new Error(`successor pid ${successorPid} never exited`); +const repair = await tool.execute("tool-call-confirm-newer-repair", {}, undefined, undefined, {}); +if (!repair.content[0].text.includes("started Pi extension arm child")) { + throw new Error(`repair did not start a newer arm: ${repair.content[0].text}`); +} +for (let i = 0; i < 250 && armRows().length < 3; i += 1) await sleep(20); +if (armRows().length !== 3) throw new Error(`expected a newer third arm: ${armRows().join(" | ")}`); +const newerPid = Number(armRows()[2].slice("arm=".length)); +writeFileSync(process.env.FM_GO_FILE, "go\n"); +for (let i = 0; i < 250 && !prompt.includes("handling delivery confirmation was rejected"); i += 1) await sleep(20); +if (!prompt.includes("handling delivery confirmation was rejected")) { + throw new Error(`the stale successor's confirmation never failed: ${prompt}`); +} +await sleep(200); +if (existsSync(process.env.FM_RETIRED_FILE)) { + throw new Error("a failed confirmation for a stale successor retired the newer healthy arm"); +} +process.kill(newerPid, 0); +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi must not retire a newer arm when a stale successor's confirmation fails: $out" + [ -z "$out" ] || fail "Pi confirm-newer test printed output: $out" + pass "Pi confirm failure for a stale successor spares the newer arm" +} + +# A marker that advanced mid-restore supersedes the in-flight delivery: the +# shell reports a generation mismatch (status 3), so the wake must be +# delivered with no rejection appendix, nothing may be retired, and the +# attempt plus the confirm result must land in the bounded extension log. +# The log assertions run opted in (FM_WATCH_EXTENSION_LOG_KEEP_LINES=50 +# below); the default-off contract lives in +# test_pi_extension_log_stays_off_unless_opted_in. +test_pi_superseded_delivery_has_no_rejection_appendix() { + local repo home plugin log stop out status extension_log + repo="$TMP_ROOT/pi-handling-superseded-root" + home="$TMP_ROOT/pi-handling-superseded-home" + log="$TMP_ROOT/pi-handling-superseded.log" + stop="$TMP_ROOT/pi-handling-superseded.stop" + extension_log="$home/state/.watch-extension.log" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'superseded generation=%s watcher=%s\n' "$2" "$4" >> "${FM_ARM_LOG:?}" + exit 3 +fi +printf 'arm=%s predecessor=%s\n' "$$" "${FM_WATCH_PREDECESSOR_ARM_PID:-none}" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: synthetic actionable close\n' + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" FM_WATCH_EXTENSION_LOG_KEEP_LINES=50 node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +let prompt = ""; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + prompt += message; + }, +}; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +await tool.execute("tool-call-handling-superseded", {}, undefined, undefined, {}); +for (let i = 0; i < 250 && !prompt.includes("FIRSTMATE WATCHER WAKE"); i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); +} +if (!prompt.includes("FIRSTMATE WATCHER WAKE")) throw new Error(`missing follow-up: ${prompt}`); +if (prompt.includes("handling delivery confirmation was rejected")) { + throw new Error(`a superseded delivery carried a rejection appendix: ${prompt}`); +} +if ((prompt.match(/FIRSTMATE WATCHER WAKE/g) || []).length !== 1) { + throw new Error(`a superseded delivery was not a single plain message: ${prompt}`); +} +const rows = existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n") + : []; +if (rows.filter((row) => row.startsWith("superseded ")).length < 1) { + throw new Error(`handling-delivered was never attempted: ${rows.join(" | ")}`); +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi must deliver a superseded wake with no rejection appendix: $out" + [ -z "$out" ] || fail "Pi superseded-delivery test printed output: $out" + [ -f "$extension_log" ] || fail "Pi extension recorded no bounded restore/confirm log" + grep -qF "restore attempt=" "$extension_log" \ + || fail "extension log has no restore attempt: $(cat "$extension_log")" + grep -qF "result=superseded" "$extension_log" \ + || fail "extension log has no superseded confirm result: $(cat "$extension_log")" + pass "Pi superseded handling delivery carries no rejection appendix and is logged" +} + +# A superseded confirmation is not a failure, so the wake routes exactly like +# a confirmed delivery: an accepting supervision branch owns it and main gets +# no follow-up. The same fixture with the confirmation succeeding is the +# control, so the case cannot go vacuous. +test_pi_superseded_delivery_is_offered_to_branch() { + local repo home plugin log stop out status + repo="$TMP_ROOT/pi-superseded-branch-root" + home="$TMP_ROOT/pi-superseded-branch-home" + log="$TMP_ROOT/pi-superseded-branch.log" + stop="$TMP_ROOT/pi-superseded-branch.stop" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'confirm generation=%s watcher=%s status=%s\n' "$2" "$4" "${FM_CONFIRM_STATUS:?}" >> "${FM_ARM_LOG:?}" + exit "$FM_CONFIRM_STATUS" +fi +printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: superseded synthetic wake\n' + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" node --input-type=module 2>&1 <<'EOF' +import { readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +async function runScenario(confirmStatus) { + process.env.FM_CONFIRM_STATUS = String(confirmStatus); + writeFileSync(process.env.FM_ARM_LOG, ""); + const offers = []; + let mainPrompt = ""; + let tool = null; + const handlers = new Map(); + const bus = { + on(channel, handler) { + handlers.set(channel, [...(handlers.get(channel) ?? []), handler]); + return () => {}; + }, + emit(channel, data) { + for (const handler of handlers.get(channel) ?? []) handler(data); + }, + }; + bus.on("fm-branch-supervision:dispatch", (offer) => { + offers.push(offer.message); + offer.accept(); + }); + const pi = { + on() {}, + events: bus, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + mainPrompt += message; + }, + }; + const mod = await import(`${pathToFileURL(process.env.PLUGIN).href}?confirm=${confirmStatus}`); + mod.default(pi); + await tool.execute(`tool-call-superseded-branch-${confirmStatus}`, {}, undefined, undefined, {}); + for (let i = 0; i < 250 && offers.length === 0 && mainPrompt === ""; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); + } + const rows = readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n"); + return { offers, mainPrompt, rows }; +} + +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +writeFileSync(`${process.env.FM_HOME}/state/superseded-branch.meta`, "project=/projects/approved\nwindow=fm-superseded-branch\n"); +writeFileSync(`${process.env.FM_HOME}/state/.wake-queue`, "1\t1\tsignal\tsuperseded-branch.status\tsignal: superseded synthetic wake\n"); +for (const confirmStatus of [0, 3]) { + const result = await runScenario(confirmStatus); + if (!result.rows.some((row) => row.startsWith("confirm ") && row.endsWith(`status=${confirmStatus}`))) { + throw new Error(`confirm status ${confirmStatus} was never exercised: ${result.rows.join(" | ")}`); + } + if (result.offers.length !== 1) { + throw new Error(`confirm status ${confirmStatus}: expected one branch offer, got ${result.offers.length}; main got: ${result.mainPrompt}`); + } + if (!result.offers[0].includes("signal: superseded synthetic wake")) { + throw new Error(`confirm status ${confirmStatus}: offer missed the wake reason: ${result.offers[0]}`); + } + if (result.offers[0].includes("handling delivery confirmation was rejected")) { + throw new Error(`confirm status ${confirmStatus}: offer carried a rejection appendix: ${result.offers[0]}`); + } + if (result.mainPrompt !== "") { + throw new Error(`confirm status ${confirmStatus}: accepted offer still reached main: ${result.mainPrompt}`); + } +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi must offer a superseded wake to the branch exactly like a confirmed one: $out" + [ -z "$out" ] || fail "Pi superseded branch-offer test printed output: $out" + pass "Pi superseded handling delivery is offered to the branch like a confirmed one" +} + +# The extension diagnostic log is opt-in and default-off: the same +# mid-restore supersession that logs when opted in must create no +# state/.watch-extension.log file with the knob unset, zero, or +# non-numeric, while the wake is still delivered with no rejection +# appendix. Each knob value runs in a fresh home because the extension +# reads the knob once at module load. +test_pi_extension_log_stays_off_unless_opted_in() { + local repo driver mode home log stop extension_log knob_value out status + repo="$TMP_ROOT/pi-extension-log-off-root" + driver="$TMP_ROOT/pi-extension-log-off-driver.mjs" + mkdir -p "$repo/bin" + install_pi_watch_extension_fixture "$repo" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + printf 'superseded generation=%s watcher=%s\n' "$2" "$4" >> "${FM_ARM_LOG:?}" + exit 3 +fi +printf 'arm=%s predecessor=%s\n' "$$" "${FM_WATCH_PREDECESSOR_ARM_PID:-none}" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: synthetic actionable close\n' + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + cat > "$driver" <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +let prompt = ""; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async (message) => { + prompt += message; + }, +}; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +await tool.execute("tool-call-handling-superseded", {}, undefined, undefined, {}); +for (let i = 0; i < 250 && !prompt.includes("FIRSTMATE WATCHER WAKE"); i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); +} +if (!prompt.includes("FIRSTMATE WATCHER WAKE")) throw new Error(`missing follow-up: ${prompt}`); +if (prompt.includes("handling delivery confirmation was rejected")) { + throw new Error(`a superseded delivery carried a rejection appendix: ${prompt}`); +} +if ((prompt.match(/FIRSTMATE WATCHER WAKE/g) || []).length !== 1) { + throw new Error(`a superseded delivery was not a single plain message: ${prompt}`); +} +const rows = existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n") + : []; +if (rows.filter((row) => row.startsWith("superseded ")).length < 1) { + throw new Error(`handling-delivered was never attempted: ${rows.join(" | ")}`); +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF + for mode in unset zero bogus; do + home="$TMP_ROOT/pi-extension-log-off-home-$mode" + log="$TMP_ROOT/pi-extension-log-off-$mode.log" + stop="$TMP_ROOT/pi-extension-log-off-$mode.stop" + extension_log="$home/state/.watch-extension.log" + mkdir -p "$home/state" "$home/config" + if [ "$mode" = unset ]; then + out=$(PLUGIN="$repo/.pi/extensions/fm-primary-pi-watch.ts" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" node "$driver" 2>&1) + else + if [ "$mode" = zero ]; then knob_value=0; else knob_value="not-a-number"; fi + out=$(PLUGIN="$repo/.pi/extensions/fm-primary-pi-watch.ts" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" FM_WATCH_EXTENSION_LOG_KEEP_LINES="$knob_value" node "$driver" 2>&1) + fi + status=$? + expect_code 0 "$status" "Pi superseded delivery must still succeed with the log $mode: $out" + [ -z "$out" ] || fail "Pi log-off ($mode) run printed output: $out" + [ ! -e "$extension_log" ] || fail "Pi extension wrote its diagnostic log with the knob $mode: $(cat "$extension_log")" + done + pass "Pi extension diagnostic log stays off unless opted in" +} + +# A repair call must not no-op on an arm child whose process is already dead +# while its close event is still pending (stdio pipe held): the first arm +# below exits at once but leaves a pipe holder behind, so the extension still +# holds the handle with no close fired. The repair must start a fresh arm +# rather than answer unchanged. +test_pi_repair_starts_fresh_arm_over_dead_child() { + local repo home plugin log stop holder out status + repo="$TMP_ROOT/pi-stale-child-root" + home="$TMP_ROOT/pi-stale-child-home" + log="$TMP_ROOT/pi-stale-child.log" + stop="$TMP_ROOT/pi-stale-child.stop" + holder="$TMP_ROOT/pi-stale-child-holder.sh" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$holder" <<'SH' +#!/usr/bin/env bash +while [ ! -e "${1:?}" ]; do sleep 0.05; done +SH + chmod +x "$holder" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +printf 'arm=%s predecessor=%s\n' "$$" "${FM_WATCH_PREDECESSOR_ARM_PID:-none}" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" + ("${FM_HOLDER:?}" "${FM_STOP_FILE:?}" >&1 2>/dev/null &) + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" FM_HOLDER="$holder" node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async () => {}, +}; +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +const first = await tool.execute("tool-call-first-arm", {}, undefined, undefined, {}); +if (!first.content[0].text.includes("started Pi extension arm child")) { + throw new Error(`first arm did not start: ${first.content[0].text}`); +} +const armRows = () => existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n") + : []; +let armPid = ""; +for (let i = 0; i < 250 && !armPid; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); + const row = armRows().find((line) => line.startsWith("arm=")); + if (row) armPid = row.split(" ")[0].slice("arm=".length); +} +if (!armPid) throw new Error("first arm never logged its pid"); +let dead = false; +for (let i = 0; i < 250 && !dead; i += 1) { + try { + process.kill(Number(armPid), 0); + } catch { + dead = true; + } + if (!dead) await new Promise((resolve) => setTimeout(resolve, 20)); +} +if (!dead) throw new Error(`first arm pid ${armPid} never exited`); +const second = await tool.execute("tool-call-repair", {}, undefined, undefined, {}); +const text = second.content[0].text; +if (!text.includes("started Pi extension arm child")) { + throw new Error(`repair did not start a fresh arm: ${text}`); +} +if (text.includes("unchanged")) { + throw new Error(`repair no-opped on a dead child: ${text}`); +} +let rearmed = false; +for (let i = 0; i < 250 && !rearmed; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 20)); + rearmed = armRows().filter((line) => line.startsWith("arm=")).length >= 2; +} +if (!rearmed) { + throw new Error(`repair started no second arm: ${armRows().join(" | ")}`); +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi repair must start a fresh arm over a dead child handle: $out" + [ -z "$out" ] || fail "Pi stale-child repair test printed output: $out" + pass "Pi repair starts a fresh arm instead of no-opping on a dead child" +} + +# A scheduled continuity retry must not stall behind a dead-but-unclosed arm +# child. The first arm exits with its stdout held open, a manual repair starts +# a second arm that does the same, and then the first arm's close finally +# fires: its retry must see the dead second arm as an empty slot and start a +# fresh arm. +test_pi_scheduled_retry_starts_fresh_arm_over_dead_child() { + local repo home plugin log stop release out status + repo="$TMP_ROOT/pi-retry-dead-root" + home="$TMP_ROOT/pi-retry-dead-home" + log="$TMP_ROOT/pi-retry-dead.log" + stop="$TMP_ROOT/pi-retry-dead.stop" + release="$TMP_ROOT/pi-retry-dead.release" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +if [ "$count" -eq 1 ]; then + (while [ ! -e "${FM_RELEASE_FILE:?}" ]; do sleep 0.02; done) & + exit 0 +fi +if [ "$count" -eq 2 ]; then + (while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.05; done) & + exit 0 +fi +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" FM_RELEASE_FILE="$release" FM_WATCH_REARM_RETRY_BASE_MS=20 node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async () => {}, +}; +const armRows = () => existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n").filter((row) => row.startsWith("arm=")) + : []; +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); +async function waitForArms(count) { + for (let i = 0; i < 250 && armRows().length < count; i += 1) await sleep(20); + if (armRows().length < count) throw new Error(`expected ${count} arms: ${armRows().join(" | ")}`); + return Number(armRows()[count - 1].slice("arm=".length)); +} +async function waitForExit(pid) { + for (let i = 0; i < 250; i += 1) { + try { + process.kill(pid, 0); + } catch { + return; + } + await sleep(20); + } + throw new Error(`arm pid ${pid} never exited`); +} +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +await tool.execute("tool-call-retry-dead-first", {}, undefined, undefined, {}); +await waitForExit(await waitForArms(1)); +const repair = await tool.execute("tool-call-retry-dead-repair", {}, undefined, undefined, {}); +if (!repair.content[0].text.includes("started Pi extension arm child")) { + throw new Error(`repair did not start a second arm: ${repair.content[0].text}`); +} +await waitForExit(await waitForArms(2)); +writeFileSync(process.env.FM_RELEASE_FILE, "release\n"); +for (let i = 0; i < 250 && armRows().length < 3; i += 1) await sleep(20); +if (armRows().length !== 3) { + throw new Error(`the first arm's close scheduled no retry over the dead second arm: ${armRows().join(" | ")}`); +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi scheduled retry must start a fresh arm over a dead child handle: $out" + [ -z "$out" ] || fail "Pi scheduled-retry dead-child test printed output: $out" + pass "Pi scheduled retry starts a fresh arm instead of stalling on a dead child" +} + +# A verified successor that closes while its wake is still being delivered +# defers its retry to the end of that delivery. If a manual repair meanwhile +# left a dead-but-unclosed arm in the slot, the deferred retry must still +# start a fresh arm. +test_pi_deferred_close_starts_fresh_arm_over_dead_child() { + local repo home plugin log stop release out status + repo="$TMP_ROOT/pi-deferred-dead-root" + home="$TMP_ROOT/pi-deferred-dead-home" + log="$TMP_ROOT/pi-deferred-dead.log" + stop="$TMP_ROOT/pi-deferred-dead.stop" + release="$TMP_ROOT/pi-deferred-dead.release" + mkdir -p "$repo/bin" "$home/state" "$home/config" + install_pi_watch_extension_fixture "$repo" + plugin="$repo/.pi/extensions/fm-primary-pi-watch.ts" + cat > "$repo/bin/fm-watch-arm.sh" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = --handling-delivered ]; then + exit 0 +fi +printf 'arm=%s\n' "$$" >> "${FM_ARM_LOG:?}" +count=$(grep -c '^arm=' "$FM_ARM_LOG") +if [ "$count" -eq 1 ]; then + printf 'watcher: started pid=%s (beacon fresh)\n' "$$" + printf 'signal: synthetic actionable close\n' + exit 0 +fi +if [ "$count" -eq 2 ]; then + printf 'watcher: started pid=%s (beacon fresh) recovery-generation=fixture-generation\n' "$$" + while [ ! -e "${FM_RELEASE_FILE:?}" ]; do sleep 0.02; done + exit 0 +fi +printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +if [ "$count" -eq 3 ]; then + (while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.05; done) & + exit 0 +fi +trap 'exit 0' TERM INT +while [ ! -e "$FM_STOP_FILE" ]; do sleep 0.02; done +SH + chmod +x "$repo/bin/fm-watch-arm.sh" + out=$(PLUGIN="$plugin" FM_HOME="$home" FM_ROOT_OVERRIDE="$repo" FM_ARM_LOG="$log" FM_STOP_FILE="$stop" FM_RELEASE_FILE="$release" FM_WATCH_REARM_RETRY_BASE_MS=20 node --input-type=module 2>&1 <<'EOF' +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +let tool = null; +let deliveryStarted = false; +let finishDelivery = () => {}; +const deliveryHeld = new Promise((resolve) => { + finishDelivery = resolve; +}); +const pi = { + on() {}, + registerCommand() {}, + registerTool(candidate) { + if (candidate.name === "fm_watch_arm_pi") tool = candidate; + }, + sendUserMessage: async () => { + deliveryStarted = true; + await deliveryHeld; + }, +}; +const armRows = () => existsSync(process.env.FM_ARM_LOG) + ? readFileSync(process.env.FM_ARM_LOG, "utf8").trim().split("\n").filter((row) => row.startsWith("arm=")) + : []; +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); +async function waitForArms(count) { + for (let i = 0; i < 250 && armRows().length < count; i += 1) await sleep(20); + if (armRows().length < count) throw new Error(`expected ${count} arms: ${armRows().join(" | ")}`); + return Number(armRows()[count - 1].slice("arm=".length)); +} +async function waitForExit(pid) { + for (let i = 0; i < 250; i += 1) { + try { + process.kill(pid, 0); + } catch { + return; + } + await sleep(20); + } + throw new Error(`arm pid ${pid} never exited`); +} +writeFileSync(`${process.env.FM_HOME}/state/.lock`, `${process.pid}\n`); +const mod = await import(pathToFileURL(process.env.PLUGIN).href); +mod.default(pi); +await tool.execute("tool-call-deferred-dead-first", {}, undefined, undefined, {}); +const successorPid = await waitForArms(2); +for (let i = 0; i < 250 && !deliveryStarted; i += 1) await sleep(20); +if (!deliveryStarted) throw new Error("the restored wake was never delivered"); +writeFileSync(process.env.FM_RELEASE_FILE, "release\n"); +await waitForExit(successorPid); +await sleep(100); +const repair = await tool.execute("tool-call-deferred-dead-repair", {}, undefined, undefined, {}); +if (!repair.content[0].text.includes("started Pi extension arm child")) { + throw new Error(`repair did not start an arm after the successor closed: ${repair.content[0].text}`); +} +await waitForExit(await waitForArms(3)); +finishDelivery(); +for (let i = 0; i < 250 && armRows().length < 4; i += 1) await sleep(20); +if (armRows().length !== 4) { + throw new Error(`the deferred close started no retry over the dead repair arm: ${armRows().join(" | ")}`); +} +writeFileSync(process.env.FM_STOP_FILE, "stop\n"); +process.exit(0); +EOF +) + status=$? + expect_code 0 "$status" "Pi deferred close must start a fresh arm over a dead child handle: $out" + [ -z "$out" ] || fail "Pi deferred-close dead-child test printed output: $out" + pass "Pi deferred close starts a fresh arm instead of stalling on a dead child" +} + test_pi_hung_successor_falls_back_to_typed_wake() { local repo home plugin log out status repo="$TMP_ROOT/pi-hung-successor-root" @@ -4426,6 +5201,14 @@ test_pi_heartbeat_restoration_failure_stays_on_main test_pi_watcher_failure_never_offered_to_branch test_pi_away_record_collapses_eligibility_and_keeps_vetoes_on_main test_pi_handling_delivery_failure_is_typed_once +test_pi_confirm_failure_retires_arm_with_distinct_watcher_pid +test_pi_confirm_failure_spares_a_newer_arm +test_pi_superseded_delivery_has_no_rejection_appendix +test_pi_superseded_delivery_is_offered_to_branch +test_pi_extension_log_stays_off_unless_opted_in +test_pi_repair_starts_fresh_arm_over_dead_child +test_pi_scheduled_retry_starts_fresh_arm_over_dead_child +test_pi_deferred_close_starts_fresh_arm_over_dead_child test_pi_hung_successor_falls_back_to_typed_wake test_pi_unretired_successor_falls_back_without_retry test_pi_late_unretired_close_resumes_supervision diff --git a/tests/fm-remote-delta-read.test.sh b/tests/fm-remote-delta-read.test.sh new file mode 100755 index 00000000000..49c37fe920a --- /dev/null +++ b/tests/fm-remote-delta-read.test.sh @@ -0,0 +1,220 @@ +#!/usr/bin/env bash +# Behavior tests for bin/fm-remote-delta-read.sh, the append-only reply-log +# reader a remote lane runs as its preemptible long poll. +# +# Pins, through the executable interface: +# * the delta schema: offsets, prefix and payload hashes, and payload bytes +# * every continuity-break reason: truncated, prefix-changed, missing, and +# line-exceeds-bound, plus an unsafe symlink or traversal target +# * an incomplete tail line is withheld until a newline completes it +# * exit 75 when the wait window closes with nothing appended +# * the per-poll executable boundary: an unchanged log costs one stat per +# sample, and the bounded capture/hashing path runs only when the file's +# stat identity changed - a same-size in-place rewrite still breaks the +# continuity hash, so statting cheaper never hides a change. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P) +TMP_ROOT=$(fm_test_tmproot fm-remote-delta-read) +mkdir -p "$TMP_ROOT" +TMP_ROOT=$(cd "$TMP_ROOT" && pwd -P) +DELTA_HOME="$TMP_ROOT/home" +DELTA_LOG_REL=state/replies.status +mkdir -p "$DELTA_HOME/state" +READER="$ROOT/bin/fm-remote-delta-read.sh" + +EMPTY_SHA=$(: | shasum -a 256 | awk '{print $1}') +sha() { printf '%b' "$1" | shasum -a 256 | awk '{print $1}'; } + +run_reader() { # [rel] + FM_HOME="$DELTA_HOME" FM_REMOTE_DELTA_POLL_SECONDS=0.05 \ + "$READER" "${4:-$DELTA_LOG_REL}" "$1" "$2" "$3" +} + +# A growing log returns the complete appended lines with exact boundaries. +: > "$DELTA_HOME/$DELTA_LOG_REL" +run_reader 0 "$EMPTY_SHA" 4 > "$TMP_ROOT/growth.out" & +READER_PID=$! +sleep 0.3 +printf 'first line\n' >> "$DELTA_HOME/$DELTA_LOG_REL" +wait "$READER_PID" || fail "a delta on growth did not exit 0" +OUT=$(<"$TMP_ROOT/growth.out") +assert_contains "$OUT" 'status=delta' 'the grown log did not produce a delta' +assert_contains "$OUT" 'from_offset=0' 'the delta did not start at the caller cursor' +assert_contains "$OUT" 'to_offset=11' 'the delta did not stop at the complete line' +assert_contains "$OUT" "from_prefix_sha256=$EMPTY_SHA" 'the delta did not echo the caller prefix hash' +assert_contains "$OUT" 'payload_sha256='"$(sha 'first line\n')" 'the payload hash is not the appended bytes' +assert_contains "$OUT" 'payload_bytes=11' 'the payload byte count is wrong' +[ "$(tail -n 1 "$TMP_ROOT/growth.out")" = 'first line' ] || fail 'the delta did not carry the appended line' +pass 'an appended line produces a delta with exact offsets, hashes, and payload' + +# An unchanged log closes the window with 75 and never runs the snapshot path: +# one stat per sample is the whole per-poll cost. +DELTA_SHIM="$TMP_ROOT/delta-shim" +EXEC_LOG="$TMP_ROOT/delta-execs" +mkdir -p "$DELTA_SHIM" +for TOOL in perl shasum sha256sum od tail head wc tr date stat dirname basename; do + REAL=$(PATH=/usr/bin:/bin command -v "$TOOL" 2>/dev/null || true) + [ -n "$REAL" ] || continue + cat > "$DELTA_SHIM/$TOOL" <> "\$FM_TEST_EXEC_LOG" +exec $REAL "\$@" +SH + chmod +x "$DELTA_SHIM/$TOOL" +done +: > "$EXEC_LOG" +: > "$DELTA_HOME/$DELTA_LOG_REL" +FM_TEST_EXEC_LOG="$EXEC_LOG" PATH="$DELTA_SHIM:/usr/bin:/bin" run_reader 0 "$EMPTY_SHA" 2 > /dev/null && \ + fail "an unchanged log did not exit 75" || RC=$? +[ "${RC:-0}" -eq 75 ] || fail "an unchanged log closed its window with $RC instead of 75" +perl_execs=$(grep -cx perl "$EXEC_LOG" || true) +stat_execs=$(grep -cx stat "$EXEC_LOG" || true) +# The first poll always takes one snapshot: it must validate the caller's +# cursor prefix before waiting. The gate only suppresses the repeats. +[ "$perl_execs" -eq 1 ] || fail "an unchanged log ran the bounded capture $perl_execs times" +for TOOL in od tail head wc date; do + hits=$(grep -cx "$TOOL" "$EXEC_LOG" || true) + [ "$hits" -eq 0 ] || fail "an unchanged log ran $TOOL $hits times in the poll loop" +done +[ "$stat_execs" -ge 5 ] || fail "the unchanged window did not keep polling stat ($stat_execs)" +pass 'an unchanged log costs one stat per poll and exits 75 at the window' + +# Growth still pays the capture and hashing tools exactly when bytes appear. +: > "$EXEC_LOG" +FM_TEST_EXEC_LOG="$EXEC_LOG" PATH="$DELTA_SHIM:/usr/bin:/bin" run_reader 0 "$EMPTY_SHA" 4 > "$TMP_ROOT/growth2.out" & +READER_PID=$! +sleep 0.3 +printf 'counted change\n' >> "$DELTA_HOME/$DELTA_LOG_REL" +wait "$READER_PID" || fail 'the shimmed growth run did not exit 0' +assert_contains "$(<"$TMP_ROOT/growth2.out")" 'status=delta' 'the shimmed run lost the delta' +[ "$(grep -cx perl "$EXEC_LOG" || true)" -ge 1 ] || fail 'growth did not run the bounded capture' +[ "$(grep -cx shasum "$EXEC_LOG" || true)" -ge 2 ] || fail 'growth did not hash prefix and payload' +pass 'the capture and hashing path runs exactly once a real change lands' + +# A shrunk file reports the truncation with the hash of what actually remains. +printf 'alpha\nbeta\n' > "$DELTA_HOME/$DELTA_LOG_REL" +PREFIX_SHA=$(sha 'alpha\nbeta\n') +run_reader 11 "$PREFIX_SHA" 4 > "$TMP_ROOT/truncated.out" & +READER_PID=$! +sleep 0.3 +printf 'a\n' > "$DELTA_HOME/$DELTA_LOG_REL" +wait "$READER_PID" || fail 'the truncated read did not exit 0' +OUT=$(<"$TMP_ROOT/truncated.out") +assert_contains "$OUT" 'status=continuity-broken' 'truncation did not produce a break' +assert_contains "$OUT" 'reason=truncated' 'truncation was not named' +assert_contains "$OUT" 'to_offset=2' 'the break did not report the shrunk size' +assert_contains "$OUT" "to_prefix_sha256=$(sha 'a\n')" 'the break did not hash the remaining prefix' +pass 'a shrunk log breaks continuity as truncated with the remaining hash' + +# A same-size in-place rewrite changes only mtime/ctime: the stat gate must +# still take the snapshot, where the prefix hash catches the changed bytes. +# This rewrite lands in a later epoch second. +printf 'alpha\nbeta\n' > "$DELTA_HOME/$DELTA_LOG_REL" +run_reader 11 "$PREFIX_SHA" 4 > "$TMP_ROOT/rewrite.out" & +READER_PID=$! +sleep 1.1 +printf 'OMEGA\nbeta\n' > "$DELTA_HOME/$DELTA_LOG_REL" +wait "$READER_PID" || fail 'the rewritten read did not exit 0' +OUT=$(<"$TMP_ROOT/rewrite.out") +assert_contains "$OUT" 'status=continuity-broken' 'a same-size rewrite did not produce a break' +assert_contains "$OUT" 'reason=prefix-changed' 'the same-size rewrite was not named prefix-changed' +assert_contains "$OUT" 'to_offset=11' 'the break did not report the current size' +pass 'a same-size in-place rewrite breaks continuity as prefix-changed' + +# Model a same-second same-size rewrite at the stat executable boundary: +# size, inode, device, and whole-second timestamps stay fixed; only the +# fractions change. Rewrite the real log after the initial snapshot's prefix +# has been hashed, so scheduler load cannot move the test across a second. +SUBSECOND_SHIM="$TMP_ROOT/subsecond-shim" +mkdir -p "$SUBSECOND_SHIM" +cat > "$SUBSECOND_SHIM/stat" <<'SH' +#!/bin/sh +fraction=111111111 +[ ! -e "$FM_TEST_REWRITE_DONE" ] || fraction=222222222 +printf '11:100.%s:100.%s:123:456\n' "$fraction" "$fraction" +SH +REAL_SHASUM=$(PATH=/usr/bin:/bin command -v shasum) +cat > "$SUBSECOND_SHIM/shasum" < "\$FM_TEST_REWRITE_LOG" + fi + : > "\$FM_TEST_REWRITE_DONE" + fi + ;; +esac +SH +chmod +x "$SUBSECOND_SHIM/stat" "$SUBSECOND_SHIM/shasum" +printf 'alpha\nbeta\n' > "$DELTA_HOME/$DELTA_LOG_REL" +FM_TEST_REWRITE_LOG="$DELTA_HOME/$DELTA_LOG_REL" \ + FM_TEST_REWRITE_DONE="$TMP_ROOT/rewrite-done" \ + PATH="$SUBSECOND_SHIM:/usr/bin:/bin" \ + run_reader 11 "$PREFIX_SHA" 10 > "$TMP_ROOT/same-second.out" \ + || fail 'the same-second rewrite read did not exit 0' +[ -e "$TMP_ROOT/rewrite-done" ] || fail 'the initial prefix hash did not trigger the rewrite' +OUT=$(<"$TMP_ROOT/same-second.out") +assert_contains "$OUT" 'status=continuity-broken' 'the subsecond change did not break continuity' +assert_contains "$OUT" 'reason=prefix-changed' 'a same-second same-size rewrite was not detected' +pass 'a same-second same-size rewrite of the same inode breaks continuity' + +# A log that disappears mid-wait breaks as missing only for a nonzero cursor. +printf 'alpha\nbeta\n' > "$DELTA_HOME/$DELTA_LOG_REL" +FM_TEST_REWRITE_LOG="$DELTA_HOME/$DELTA_LOG_REL" \ + FM_TEST_REWRITE_DONE="$TMP_ROOT/remove-done" FM_TEST_REMOVE_LOG=1 \ + PATH="$SUBSECOND_SHIM:/usr/bin:/bin" \ + run_reader 11 "$PREFIX_SHA" 10 > "$TMP_ROOT/missing.out" \ + || fail 'the missing-file read did not exit 0' +OUT=$(<"$TMP_ROOT/missing.out") +assert_contains "$OUT" 'status=continuity-broken' 'a removed log did not produce a break' +assert_contains "$OUT" 'reason=missing' 'the removed log was not named missing' +pass 'a removed log breaks continuity as missing' + +# A removed log is not a break for a cursor at the origin: it keeps waiting, +# which is what a first poll against a not-yet-created log relies on. +run_reader 0 "$EMPTY_SHA" 1 > /dev/null && fail 'a missing log at offset 0 did not wait' || RC=$? +[ "${RC:-0}" -eq 75 ] || fail "a missing log at offset 0 exited $RC instead of 75" +pass 'a missing log at the origin cursor keeps waiting until the window closes' + +# An incomplete tail line is withheld until its newline lands, then delivered +# whole rather than as a fragment. +printf 'whole\n' > "$DELTA_HOME/$DELTA_LOG_REL" +run_reader 6 "$(sha 'whole\n')" 4 > "$TMP_ROOT/partial.out" & +READER_PID=$! +sleep 0.3 +printf 'frag' >> "$DELTA_HOME/$DELTA_LOG_REL" +sleep 0.4 +printf -- '-ment\n' >> "$DELTA_HOME/$DELTA_LOG_REL" +wait "$READER_PID" || fail 'the completed line did not exit 0' +OUT=$(<"$TMP_ROOT/partial.out") +assert_contains "$OUT" 'status=delta' 'the completed line did not produce a delta' +assert_contains "$OUT" 'to_offset=16' 'the delta did not stop at the completed line' +assert_contains "$OUT" 'payload_bytes=10' 'the payload did not carry the whole line' +[ "$(tail -n 1 "$TMP_ROOT/partial.out")" = 'frag-ment' ] || fail 'the payload did not join the fragment' +pass 'an unterminated tail is withheld until the newline completes it' + +# The same continuity rules apply to the schema's other break and refusal +# surfaces, with the wait window never entered. +printf 'past-bound tail' > "$DELTA_HOME/$DELTA_LOG_REL" +FM_HOME="$DELTA_HOME" FM_REMOTE_DELTA_MAX_BYTES=8 \ + run_reader 0 "$EMPTY_SHA" 1 > "$TMP_ROOT/bound.out" || fail 'the bound break did not exit 0' +assert_contains "$(<"$TMP_ROOT/bound.out")" 'reason=line-exceeds-bound' \ + 'a tail line longer than the payload bound did not break' +run_reader 0 "$EMPTY_SHA" 1 '../outside' > /dev/null 2>&1 && \ + fail 'a traversing path was accepted' || true +printf 'real\n' > "$DELTA_HOME/state/real.status" +ln -sfn real.status "$DELTA_HOME/state/link.status" +run_reader 0 "$EMPTY_SHA" 1 'state/link.status' > /dev/null 2>&1 && \ + fail 'a symlinked log was accepted' || true +pass 'the reader refuses traversal, symlinks, and oversized tail lines' + +printf 'delta-read contract tests complete\n' diff --git a/tests/fm-remote-job-orphan-reap.test.sh b/tests/fm-remote-job-orphan-reap.test.sh index f999ccd5b31..2935039ba72 100755 --- a/tests/fm-remote-job-orphan-reap.test.sh +++ b/tests/fm-remote-job-orphan-reap.test.sh @@ -127,7 +127,8 @@ start_worker() { export FM_REMOTE_JOB_STATE_ROOT="$state_root" export FM_REMOTE_JOB_PLATFORM_OVERRIDE=Linux export FM_REMOTE_JOB_ORPHAN_GRACE_SECONDS=1 - # shellcheck source=bin/fm-remote-job-lib.sh + # Production libraries are linted independently by fm-lint.sh. + # shellcheck source=/dev/null . "$ROOT/bin/fm-remote-job-lib.sh" fm_remote_job_start_linux_worker "$root" "$account_home" >&2 || exit 1 deadline=$(( $(date +%s) + 10 )) diff --git a/tests/fm-remote-job.test.sh b/tests/fm-remote-job.test.sh index d1e4399a41d..3fd53270ed2 100755 --- a/tests/fm-remote-job.test.sh +++ b/tests/fm-remote-job.test.sh @@ -50,6 +50,7 @@ STALL_WORKER_PID= STALL_REPLACEMENT_PID= STALL_JOB_GROUP= QUIET_WORKER_PID= +SCAN_LANE_PID= mkdir -p "$REMOTE_ROOT/bin" "$REMOTE_HOME" "$ACCOUNT_HOME" "$RUNTIME_BIN" # worker.pid records the serving child, not its restart supervisor, so stopping # that pid alone leaves the supervisor to respawn - the leak @@ -75,6 +76,7 @@ cleanup_remote_job_fixture() { [ -z "$LOST_TERM_PID" ] || kill -KILL "$LOST_TERM_PID" 2>/dev/null || true [ -z "$REPLACEMENT_OWNER_PID" ] || kill -KILL "$REPLACEMENT_OWNER_PID" 2>/dev/null || true [ -z "$QUIET_WORKER_PID" ] || kill -KILL "$QUIET_WORKER_PID" 2>/dev/null || true + [ -z "$SCAN_LANE_PID" ] || kill -KILL "$SCAN_LANE_PID" 2>/dev/null || true local stall_pid for stall_pid in "$STALL_WORKER_PID" "$STALL_REPLACEMENT_PID"; do [ -n "$stall_pid" ] || continue @@ -144,6 +146,85 @@ git -C "$REMOTE_ROOT" config user.name Test git -C "$REMOTE_ROOT" add AGENTS.md bin git -C "$REMOTE_ROOT" commit -qm 'remote job fixture' +# Observe the actual sleep executable boundary for the result consumer, a +# top-level command lane, and the dispatcher. Re-source the public library as +# callers may do; its own dispatcher default must not become a legacy override. +poll_cadence_case() ( + local label=$1 legacy=$2 active=$3 expected=$4 dispatch=$5 poll_dir pid='' i + poll_dir="$TMP_ROOT/poll-$label" + mkdir -p "$poll_dir/bin" + cat > "$poll_dir/bin/sleep" <<'SH' +#!/bin/bash +printf '%s\n' "$1" >> "$FM_POLL_SLEEP_LOG" +exec /bin/sleep "$@" +SH + chmod +x "$poll_dir/bin/sleep" + trap '[ -z "$pid" ] || { kill -TERM "$pid" 2>/dev/null || true; wait "$pid" 2>/dev/null || true; }' EXIT + unset FM_REMOTE_JOB_POLL_SECONDS FM_REMOTE_JOB_ACTIVE_POLL_SECONDS + # shellcheck disable=SC2030 # The legacy override is local to this cadence fixture. + [ -z "$legacy" ] || export FM_REMOTE_JOB_POLL_SECONDS="$legacy" + # shellcheck disable=SC2030 # The active override is local to this cadence fixture. + [ -z "$active" ] || export FM_REMOTE_JOB_ACTIVE_POLL_SECONDS="$active" + export FM_REMOTE_JOB_STATE_ROOT="$poll_dir/state" FM_ROOT_OVERRIDE="$REMOTE_ROOT" + # shellcheck disable=SC2030 # Each cadence fixture owns its subshell's bounds. + export FM_REMOTE_JOB_QUEUE_TIMEOUT=60 FM_REMOTE_JOB_TIMEOUT=30 + # shellcheck disable=SC2030 # The recording executable is local to this fixture. + export PATH="$poll_dir/bin:$PATH" FM_POLL_SLEEP_LOG="$poll_dir/sleeps" + # shellcheck source=bin/fm-remote-job-lib.sh + . "$ROOT/bin/fm-remote-job-lib.sh" + # shellcheck source=bin/fm-remote-job-lib.sh + . "$ROOT/bin/fm-remote-job-lib.sh" + fm_remote_job_stage "$ACCOUNT_HOME" "$REMOTE_ROOT" "$REMOTE_HOME" \ + fm-delay-job.sh 0.8 "$poll_dir/ran" /dev/null || fail "$FM_REMOTE_JOB_ERROR" + # Publish a real bounded result after the caller has entered its wait, without + # a lane's own samples contaminating this consumer-only executable log. + ( + /bin/sleep 0.8 + : > "$FM_REMOTE_JOB_JOBS/$FM_REMOTE_JOB_ID/stdout" + : > "$FM_REMOTE_JOB_JOBS/$FM_REMOTE_JOB_ID/stderr" + printf '0\n' > "$FM_REMOTE_JOB_JOBS/$FM_REMOTE_JOB_ID/exit" + fm_remote_job_write_state "$FM_REMOTE_JOB_JOBS/$FM_REMOTE_JOB_ID" 'done' + ) & + pid=$! + fm_remote_job_wait "$ACCOUNT_HOME" "$FM_REMOTE_JOB_ID" || fail "$FM_REMOTE_JOB_ERROR" + wait "$pid" || fail "$label result producer failed" + pid='' + [ "$FM_REMOTE_JOB_EXIT" -eq 0 ] || fail "$label result consumer lost the exit status" + grep -qx "$expected" "$FM_POLL_SLEEP_LOG" || fail "$label consumer never sampled at $expected seconds" + [ "$(sort -u "$FM_POLL_SLEEP_LOG")" = "$expected" ] || fail "$label consumer used another cadence" + + : > "$FM_POLL_SLEEP_LOG" + fm_remote_job_stage "$ACCOUNT_HOME" "$REMOTE_ROOT" "$REMOTE_HOME" \ + fm-delay-job.sh 0.8 "$poll_dir/ran" /dev/null || fail "$FM_REMOTE_JOB_ERROR" + HOME="$ACCOUNT_HOME" "$BASH" "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" --lane "$FM_REMOTE_JOB_ID" & + pid=$! + wait "$pid" || fail "$label command lane failed" + pid='' + [ -e "$poll_dir/ran" ] || fail "$label lane did not execute its command" + [ "$(fm_remote_job_read_state "$FM_REMOTE_JOB_JOBS/$FM_REMOTE_JOB_ID")" = 'done' ] || fail "$label lane did not publish completion" + grep -qx "$expected" "$FM_POLL_SLEEP_LOG" || fail "$label lane never sampled at $expected seconds" + if [ "$expected" != 0.05 ]; then + ! grep -qx 0.05 "$FM_POLL_SLEEP_LOG" || fail "$label lane still sampled at the dispatcher default" + fi + + : > "$FM_POLL_SLEEP_LOG" + HOME="$ACCOUNT_HOME" "$BASH" "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" > "$poll_dir/worker.log" 2>&1 & + pid=$! + for ((i = 0; i < 200; i++)); do + grep -qx 1 "$FM_POLL_SLEEP_LOG" && break + /bin/sleep 0.05 + done + grep -qx 1 "$FM_POLL_SLEEP_LOG" || fail "$label dispatcher never reached its one-second quiet wait" + [ "$(grep -cx "$dispatch" "$FM_POLL_SLEEP_LOG")" -eq 4 ] || fail "$label dispatcher did not limit its fast burst to four $dispatch-second waits" + kill -TERM "$pid" || fail "$label dispatcher stopped unexpectedly" + wait "$pid" 2>/dev/null || true + pid='' + pass "$label: result and command samples use $expected seconds; dispatcher uses four $dispatch-second waits then one second" +) +poll_cadence_case default '' '' 0.25 0.05 || exit 1 +poll_cadence_case legacy 0.07 '' 0.07 0.07 || exit 1 +poll_cadence_case active 0.07 0.12 0.12 0.07 || exit 1 + DEFAULT_STATE="$TMP_ROOT/default-timeout-jobs" DEFAULT_BOUNDS=$( unset FM_REMOTE_JOB_QUEUE_TIMEOUT @@ -1002,6 +1083,7 @@ fi exec '$(command -v sleep)' "\$@" SH chmod +x "$STALL_BIN/sleep" +# shellcheck disable=SC2031 # Cadence fixture PATH changes stayed in their subshells. HOME="$STALL_HOME" FM_ROOT_OVERRIDE="$REMOTE_ROOT" FM_REMOTE_JOB_STATE_ROOT="$STALL_STATE" \ FM_REMOTE_JOB_PLATFORM_OVERRIDE=Linux PATH="$STALL_BIN:$PATH" \ "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" --serve \ @@ -1220,6 +1302,209 @@ quiet_stop "$QUIET_WORKER_PID" QUIET_WORKER_PID= pass "an idle worker still repairs queue permissions and stops promptly on TERM" +# fm_remote_job_read_state is the per-sample read of the result consumers and +# the lane preemption scan, so it is built from builtins and must keep the +# published contract: a regular non-symlink file of at most 64 bytes, one +# newline-terminated line, and a value in the published set. An unterminated +# trailing fragment inside the size bound is still tolerated, matching the +# former tail -n +2 check. +STATE_CORPUS="$TMP_ROOT/state-corpus" +mkdir -p "$STATE_CORPUS/job-x" +state_accepts() { #