fix(bin): prevent false pipeline blocks after drive timeouts

[kunchenguid]

Intent

Ship the firstmate-side half of the no-mistakes daemon-timeout prevention work: clearer timeout and reattach rules - poll-instead-of-hold handling and crew guidance.
This must ship independently of the no-mistakes client half; there must be no cross-dependency in either direction, and it must not rely on any new no-mistakes flag (such as --wait, --no-wait, or an error_kind field) existing.
Use only the mitigation that works with the no-mistakes client as it is today: background the drive call and poll no-mistakes axi status.

The substance to implement, in the captain's terms (drawn from the daemon-timeout audit, section D plus its "shippable now" firstmate items e and f):

• A no-mistakes drive-call error or a harness command timeout is NOT evidence the daemon died.
  The daemon accepts respond immediately and runs the fix round in the background, so a client call that times out or is killed by the harness leaves the pipeline run alive and still working; the client was only ever waiting for a read.
• Before a crew reports the pipeline blocked, it must check no-mistakes daemon status and no-mistakes axi status.
  If the run is still running or fixing, reattach and keep going.
  A crew reports the pipeline blocked only when the daemon socket refuses connections or the run record is failed with a daemon error.
• The spawned harness has a command time limit (Claude Code kills a command at ten minutes maximum), which is far shorter than a fix round (a round is capped around thirty minutes and up to three rounds chain, so one drive call routinely holds far longer than ten minutes).
  Crews must background the drive call and poll status rather than sit in one blocking hold that the harness will kill.
• Firstmate itself must stop treating a crew's timeout-driven "pipeline is dead" claim as real.
  When a crew's blocked report mentions the daemon, a timeout, or unreachability while the pipeline run is still running or fixing with recent activity, it must be read as superseded because the run is alive, not as a genuine block, so a mistaken crew claim never reaches the captain.
• On a crew "pipeline unreachable" claim, firstmate reads the daemon status and the run status itself, steers the crew to reattach, and never restarts the shared daemon on a crew's claim.
  Only a genuine daemon-down finding - the socket refusing connections - is escalated.

What Changed

• Guide crews to background no-mistakes drive calls, poll run status, and reattach after client or harness timeouts instead of restarting the shared daemon.
• Reconcile pipeline-death claims against live run activity while preserving refused or missing daemon sockets as genuine blockers.
• Extend recovery guidance, architecture documentation, and crew-state regression coverage for timeout, reattachment, and socket-failure cases.

Risk Assessment

✅ Low: The changes are bounded guidance and classification updates that now preserve genuine socket failures while superseding timeout claims only with active-run evidence, with no new client dependency.

Testing

The configured changed-test baseline had already passed; targeted crew-state, brief-generation, and settle-loop tests passed, and end-user CLI/brief evidence confirms timeout claims reattach while genuine socket failures remain blocked across full and coarse run attribution. Manual evidence exposed two exact guidance phrasings that were initially missed; these were fixed and revalidated.

Evidence: Crew-state CLI transcript showing timeout reattachment and socket-down escalation

$ bin/fm-crew-state.sh evidence  # timed-out read, run has fresh activity
state: working · source: run-step · validating (fixing) · status-log superseded: run alive, not a daemon failure (steer reattach)

$ bin/fm-crew-state.sh evidence  # connection refused, persisted fixing record is stale
state: blocked · source: status-log · no-mistakes daemon connection refused · daemon socket down despite active run record

$ bin/fm-crew-state.sh evidence  # socket is missing, persisted fixing record is stale
state: blocked · source: status-log · no-mistakes daemon socket is missing · daemon socket down despite active run record

$ bin/fm-crew-state.sh evidence  # connection refused through runs-ledger coarse attribution
state: blocked · source: status-log · no-mistakes daemon connection refused · daemon socket down despite active run record

$ bin/fm-crew-state.sh evidence  # exact generated wording: socket refuses connections
state: blocked · source: status-log · no-mistakes daemon socket refuses connections · daemon socket down despite active run record

Evidence: Generated no-mistakes ship brief with background-and-poll and ordered daemon guidance

You are a crewmate: an autonomous worker agent managed by firstmate. Work on your own; do not wait for a human.

# Task
## Captain's intent
{TASK}

## Firstmate spec
{FIRSTMATE_SPEC}

# Herdr lifecycle declaration - NOT ENABLED
**HARD SAFETY GATE:** this scaffold cannot inspect the task text filled in above.
If the task will start, stop, delete, restart, profile, or otherwise drive Herdr lifecycle behavior, stop and regenerate the brief with `--herdr-lab` before dispatch.
Do not add Herdr lifecycle commands to this unguarded brief by hand.

# Setup
You are in a disposable git worktree of sample-project, at a detached HEAD on a clean default branch.

**Verify isolation before anything else.** Run `pwd -P` and `git rev-parse --show-toplevel`; both must resolve to the disposable task worktree you were launched in, such as a treehouse pool path or an Orca-managed worktree, not the primary checkout firstmate operates from.
The path check is authoritative: `git rev-parse --git-dir` and `git rev-parse --git-common-dir` can help inspect the repo, but they do not prove you are outside the primary checkout.
If the top-level path is the primary checkout or not the worktree you were launched in, STOP - do not branch or commit here - append `blocked: launched in primary checkout, not an isolated worktree` to the status file and stop.

1. First action: create your branch: `git checkout -b fm/demo-ship`
2. Run `no-mistakes doctor`; if it reports the repo is not initialized here, run `no-mistakes init`.

# Rules
1. Never push to the default branch. Never merge a PR.
2. Stay inside this worktree; modify nothing outside it.
3. Use gh-axi for GitHub operations and chrome-devtools-axi for browser operations.
4. Report status by appending one line:
   `echo "{state}: {one short line}" >> '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-ship.status'`
   States: working, needs-decision, blocked, paused, done, failed.
   Each append wakes firstmate, so report sparingly: only phase changes a supervisor
   would act on (setup done, bug reproduced, fix implemented, validation passed) and the
   needs-decision/blocked/paused/done/failed states. No step-by-step FYI progress lines;
   firstmate reads your pane for that.
   Whenever you mention a PR anywhere - a status line, your terminal, a summary - write its full
   https:// URL exactly as the forge printed it, never a bare number such as "PR 108"; firstmate
   copies that URL from your line rather than assembling one.
   A mid-task `working:` line (including setup complete) is nonterminal: do not end the
   turn after it; continue the same stage until a defined `done:` gate under Definition of done.
   Use `paused: {why}` - distinct from `blocked:` - ONLY when you are deliberately idling on a
   known external wait you expect to clear on its own (an upstream release, a rate-limit reset,
   a scheduled window): firstmate then leaves your idle pane alone and rechecks it on a long
   cadence instead of treating it as a possible wedge. Use `blocked:` when you are stuck and need help.
5. If you hit the same obstacle twice, append `blocked: {why}` and stop; firstmate will help.
6. If a decision belongs above the implementation worker (product choices, destructive actions),
   append `needs-decision: {summary of options}` and stop. Firstmate will reply with the decision.
   For a no-mistakes ask-user gate specifically, escalate all ask-user findings as one event plus one snapshot file, using that same shape even when the gate holds only a single ask-user finding: write only the ask-user findings, verbatim and unparaphrased (id, severity, file, line, description, authority), to `~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/data/demo-ship/nm-<run>-findings.txt`, then report the gate with
   `needs-decision [key=nm-<run>-<step>]: ask-user findings=<id1>,<id2>,... file=~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/data/demo-ship/nm-<run>-findings.txt`
   naming every ask-user finding id from that gate. The status line only points at the file; it never restates or summarizes a finding's content.
   A decision or blocker you opened stays open until a `resolved` line carrying its exact key lands; a later `done:` or `working:` line never closes it, even when the answer is what started that work.
   Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append `resolved: {how it cleared}` yourself (same `[key=<slug>]` if you opened it with one) as you resume.
7. Never stop, restart, or update the shared `no-mistakes` daemon - it is one instance serving
   every lane/home, so restarting it kills other lanes' in-flight pipeline runs; only firstmate
   manages the daemon.
   Before you append `blocked:` about the pipeline, run `no-mistakes daemon status` and
   `no-mistakes axi status`. If the daemon socket refuses connections or is missing, append
   `blocked: {the daemon error}` and stop even when the local run record still says running or
   fixing, because that record can be stale after the daemon exits. A run record failed with a
   daemon error is also a real block.
   Only after ruling out socket refusal, if the run is still running or fixing, reattach and keep
   going. A drive-call error, timeout, slow read, or generic unreachability is NOT a daemon error:
   the daemon accepts `respond` immediately and runs the round in the background, so a killed or
   timed-out call was only waiting for a read while the run kept working.

# Firstmate instruction inbox
Firstmate steers you through durable message files in '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-ship.inbox'.
When a terminal message says an instruction is waiting there - and at any natural checkpoint when you are unsure - list '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-ship.inbox'/*.msg, read and act on each message in numeric order, then acknowledge each handled message by moving it: `mv '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-ship.inbox'/NNN.msg '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-ship.inbox'/handled/`.
The move IS the acknowledgement: without it firstmate rings again and eventually treats you as stuck. An empty or absent inbox needs no action.

# Project memory
If `AGENTS.md` or `CLAUDE.md` already exists, or if this task produced durable project-intrinsic knowledge, run `~/.no-mistakes/worktrees/52b07e9083e7/01M1ST9CGC83HZ7HM22JSM5SHJ/bin/fm-ensure-agents-md.sh .` in the worktree.
Record only project knowledge useful to almost every future session.
For anything the codebase already shows, prefer a pointer to the authoritative file, command, or doc over copying the detail.
If you touch a project `AGENTS.md`, follow `~/.no-mistakes/worktrees/52b07e9083e7/01M1ST9CGC83HZ7HM22JSM5SHJ/bin/fm-ensure-agents-md.sh`'s self-governance contract in the same pass.
Keep it proportionate: skip `AGENTS.md` edits for trivial tasks that produced no durable project knowledge.

# Definition of done
Delivery contract: mode=no-mistakes
The task is complete only when committed on your branch.
When you believe it is complete, append `done: {summary}` to the status file and stop.
Firstmate will then instruct you to run /no-mistakes to validate and ship a PR.

You drive no-mistakes by responding to its gates, not by implementing fixes.
Follow the guidance no-mistakes itself provides for the mechanics: it loads when you invoke /no-mistakes, and `no-mistakes axi run --help` plus the `help` lines in each `axi` response are authoritative and version-matched to the installed binary.
When starting no-mistakes, pass `--intent` as only this brief's `## Captain's intent` subsection plus any later words the captain actually said.
For a legacy brief with no such subsection, include only words explicitly labeled `Captain:`, `Captain's words:`, `Captain's ask:`, or `Captain's intent:`; never copy its mixed `# Task` wholesale. If it has no provenance-marked captain words, stop and ask firstmate instead of starting no-mistakes.
Do not include `## Firstmate spec`, later Firstmate build constraints, or your own decisions and tradeoffs.
The `--intent` string you pass must be self-sufficient: that string plus the codebase must let a reader reconstruct roughly the same specification, without depending on a separate report, a PR, or context that lives only in this conversation.
When the captain's intent refers to a report, decision, or PR ("do items 1, 2, 3, and 7 of the report"), write the substance of the referenced items into `--intent` in the captain's terms, not only the pointer; that substance is the captain's ask by reference, while Firstmate's build instructions and your own decisions still stay out.
This replaces the no-mistakes skill's advice to enrich `--intent` with decisions and tradeoffs; that advice does not apply to Firstmate-dispatched work.
Do not hand-edit, commit, or fix findings yourself while a run is active - the pipeline applies every fix.

One drive call blocks until the next gate or outcome, which routinely outlives what your harness lets a single command run: Claude Code kills a command at ten minutes maximum, while one fix round is capped around thirty minutes and up to three rounds chain.
So background the drive call and poll `no-mistakes axi status` from a separate call instead of sitting in one blocking hold your harness will kill.
Where a harness's own command limit is not established, assume it bounds commands and use that same background-and-poll shape.
A killed or timed-out call is never evidence the daemon died: the daemon accepts your response immediately and runs the round in the background, so the call was only ever waiting for a read while the run kept working.
Reattach and keep going rather than reporting the pipeline blocked; rule 7 owns the checks that decide when a pipeline block is real.

Two firstmate-specific rules layer on top of that guidance:
- ask-user findings are never yours to answer: escalate to firstmate using rule 6's ask-user format and stop.
  Firstmate applies `ask-user-authority` and obtains any required captain decision.
  When the decision comes back, feed it to the gate with `no-mistakes axi respond` and let the pipeline apply it - do not route the question to "the user" or implement the fix yourself.
- NEVER pass `--yes` (or `-y`) to `no-mistakes axi run` or `no-mistakes axi respond`. It is banned fleet-wide.
  It auto-resolves every gate including ask-user findings with no escalation, and answering your own ask-user finding is a hard rule violation.

After /no-mistakes reports CI green (the CI-ready return point - do not wait for it to keep monitoring in the background until merge), append `done: PR {url} checks green` and stop. You are finished.

Evidence: Generated scout brief with ordered socket-refusal guidance

You are a crewmate: an autonomous worker agent managed by firstmate. Work on your own; do not wait for a human.

# Task
## Captain's intent
{TASK}

## Firstmate spec
{FIRSTMATE_SPEC}

# Herdr lifecycle declaration - NOT ENABLED
**HARD SAFETY GATE:** this scaffold cannot inspect the task text filled in above.
If the task will start, stop, delete, restart, profile, or otherwise drive Herdr lifecycle behavior, stop and regenerate the brief with `--herdr-lab` before dispatch.
Do not add Herdr lifecycle commands to this unguarded brief by hand.

# Setup
You are in a disposable git worktree of sample-project, at a detached HEAD on a clean default branch.
This is a SCOUT task: the deliverable is a written report, not a PR.
The worktree is your laboratory - install, run, edit, and make scratch commits freely; all of it is discarded at teardown.
The report is the only thing that survives, so anything worth keeping must be in it.

# Rules
1. Never push to any remote and never open a PR.
2. Stay inside this worktree; the only files you may write outside it are the report and the status file below.
3. Use gh-axi for GitHub operations and chrome-devtools-axi for browser operations.
4. Report status by appending one line:
   `echo "{state}: {one short line}" >> '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-scout.status'`
   States: working, needs-decision, blocked, paused, done, failed.
   Each append wakes firstmate, so report sparingly: only phase changes a supervisor
   would act on and the needs-decision/blocked/paused/done/failed states. No step-by-step
   FYI progress lines; firstmate reads your pane for that.
   Whenever you mention a PR anywhere - a status line, your terminal, a summary - write its full
   https:// URL exactly as the forge printed it, never a bare number such as "PR 108"; firstmate
   copies that URL from your line rather than assembling one.
   Use `paused: {why}` - distinct from `blocked:` - ONLY when you are deliberately idling on a
   known external wait you expect to clear on its own (an upstream release, a rate-limit reset):
   firstmate then leaves your idle pane alone and rechecks it on a long cadence instead of
   treating it as a possible wedge. Use `blocked:` when you are stuck and need help.
5. If you hit the same obstacle twice, append `blocked: {why}` and stop; firstmate will help.
6. If a decision belongs to a human (product choices, destructive actions),
   append `needs-decision: {summary of options}` and stop. Firstmate will reply with the decision.
   A decision or blocker you opened stays open until a `resolved` line carrying its exact key lands; a later `done:` or `working:` line never closes it, even when the answer is what started that work.
   Firstmate's reply normally writes that closing line at answer time; when a blocker or wait clears WITHOUT a firstmate reply, append `resolved: {how it cleared}` yourself (same `[key=<slug>]` if you opened it with one) as you resume.
7. Never stop, restart, or update the shared `no-mistakes` daemon - it is one instance serving
   every lane/home, so restarting it kills other lanes' in-flight pipeline runs; only firstmate
   manages the daemon.
   Before you append `blocked:` about the pipeline, run `no-mistakes daemon status` and
   `no-mistakes axi status`. If the daemon socket refuses connections or is missing, append
   `blocked: {the daemon error}` and stop even when the local run record still says running or
   fixing, because that record can be stale after the daemon exits. A run record failed with a
   daemon error is also a real block.
   Only after ruling out socket refusal, if the run is still running or fixing, reattach and keep
   going. A drive-call error, timeout, slow read, or generic unreachability is NOT a daemon error:
   the daemon accepts `respond` immediately and runs the round in the background, so a killed or
   timed-out call was only waiting for a read while the run kept working.

# Firstmate instruction inbox
Firstmate steers you through durable message files in '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-scout.inbox'.
When a terminal message says an instruction is waiting there - and at any natural checkpoint when you are unsure - list '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-scout.inbox'/*.msg, read and act on each message in numeric order, then acknowledge each handled message by moving it: `mv '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-scout.inbox'/NNN.msg '~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/state/demo-scout.inbox'/handled/`.
The move IS the acknowledgement: without it firstmate rings again and eventually treats you as stuck. An empty or absent inbox needs no action.

# Definition of done
Write your findings to `~/.no-mistakes/evidence/01M1ST9CGC83HZ7HM22JSM5SHJ/generated-briefs/data/demo-scout/report.md`.
The report must stand alone: what you did, what you found, the evidence (commands run, output, file:line references), and what you recommend.
If your deliverable is a visual artifact the captain will review and iterate on, you may host the Lavish review loop yourself (poll, revise, re-serve, staying alive) instead of handing it back to firstmate.
Before reporting done, read and follow `~/.no-mistakes/worktrees/52b07e9083e7/01M1ST9CGC83HZ7HM22JSM5SHJ/.agents/skills/captain-hold-lifecycle/SKILL.md` and pass its shared completion gate for the report and any visual review.
When the report is complete, append `done: {one-line conclusion}` to the status file and stop.
If your findings reveal work that should ship (e.g. you reproduced a bug and the fix is clear), say so in the report; firstmate may promote this task in place, and you would then receive mode-specific ship instructions as a follow-up message.

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (35m24s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed (2) ✅

• 🚨 .agents/skills/stuck-crewmate-recovery/SKILL.md:54 - The required criterion says only a genuine daemon-down finding—socket refusal—is escalated, but this rule unconditionally declares any running/fixing record proof that the crew is wrong. A concrete stale-record sequence remains reachable: daemon socket refuses connections, the persisted run still says fixing but its activity is quiet, and line 54 suppresses escalation. bin/fm-crew-state.sh:620 reinforces this by returning state: working; the new test at tests/fm-crew-state.test.sh:497 explicitly expects that suppression. Make verified socket refusal outrank a stale running/fixing record across both paths.
• ⚠️ bin/fm-crew-state.sh:331 - The new broken pipe matching alias is not required by the stated daemon/timeout/unreachability criteria and can misclassify an unrelated blocker such as blocked: database upload failed with broken pipe as a daemon failure while a run is active. Remove this extra matching path rather than hardening or documenting it.

🔧 Fix: Prioritize daemon socket failure and narrow unreachable matching
2 errors still open:

• 🚨 bin/fm-crew-state.sh:622 - Socket failure still loses to an active record on the supported coarse fallback path. When axi status returns another branch, runs can attribute this crew's stale active row as COARSE_STATUS=running, but RUN_STATUS remains empty, so this condition fails and the socket-refusal block is reported as working. Treat coarse running as an active stale record for this override too.
• 🚨 bin/fm-brief.sh:396 - The generated crew rule still says an active running/fixing record requires reattachment before considering the socket result. In the authorized sequence—daemon status reports connection refused while the persisted run says fixing—lines 396-398 issue conflicting actions. This contradicts the required rule that verified socket refusal outranks a stale active record. State the socket-refusal exception first in both generated brief variants.

🔧 Fix: Honor socket refusal across coarse status and crew guidance
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 tests failed with exit code 1
• bin/fm-test-run.sh --changed --exclude-family real-herdr-gated

🔧 Fix: Replace flaky settle timing assertion with pane-read count
✅ Re-checked - no issues remain.

• bin/fm-test-run.sh --changed --exclude-family real-herdr-gated
• Baseline supplied by outer executor: bin/fm-test-run.sh --changed --exclude-family real-herdr-gated
• tests/fm-crew-state.test.sh after extending regressions for connection refusal, missing sockets, generated “socket refuses connections” wording, recent timeout activity, broken-pipe exclusion, and coarse attribution
• tests/fm-brief.test.sh
• tests/fm-spawn-worktree-settle.test.sh
• Generated no-mistakes ship and scout briefs with FM_HOME=<evidence>/generated-briefs bin/fm-brief.sh ... and reviewed their emitted crew-facing guidance and ordering
• Invoked bin/fm-crew-state.sh evidence against realistic fake daemon/run responses for fresh timeout activity, stale fixing records with refused or missing sockets, and coarse runs-ledger attribution
• Checked generated briefs for forbidden dependencies using rg -n -- '--wait|--no-wait|error_kind'

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previous terminal-run socket-precedence issue is resolved because matched socket failures now emit blocked and terminate before attributed run state can override them.

_{Reviews (2): Last reviewed commit: "no-mistakes(ci): Fixed daemon socket fai..." | Re-trigger Greptile}