feat: add crash and reboot resilience for firstmate

[tommy230]

What Changed

• Adds crash/reboot autostart support with install/status/uninstall helpers, a systemd unit, and a resume watchdog that recreates the firstmate tmux session when needed.
• Hardens runtime coordination by moving wake locking to atomic O_EXCL file locks, preserving legacy lock compatibility, and tightening spawn/teardown safety checks.
• Documents the new recovery/autostart flow and adds focused coverage for autostart rendering, resume behavior, lock exclusivity, spawn worktree validation, teardown guards, and wake queue locking.

Risk Assessment

⚠️ Medium: Captain, the review found no material merge blockers, but the change touches systemd startup, tmux recovery, and custom shell locking, so residual cross-environment risk is higher than a small bounded patch.

Testing

Captain, I ran the targeted regression tests for lock atomicity, resume/autostart behavior, spawn worktree detection, teardown safety, and wake-queue supervision; then manually demonstrated the operator-facing CLI flow for autostart rendering and fm-resume session resurrection. All checks passed, with no UI surface involved, so no screenshot evidence was applicable.

Evidence: Relevant shell test transcript

ok - wake-queue lock is mutually exclusive under contention (4 workers)
ok - lock reclaims a dead holder but never a live one
ok - lock preserves fresh legacy empty-pid directories
ok - fm-resume creates the session and launches firstmate with the right context
ok - fm-resume is idempotent (no duplicate session)
ok - fm-resume --watch does not create an empty session after launch inference failure
ok - fm-resume --watch does not create an empty session after harness binary lookup failure
ok - fm-install-autostart renders firstmate.service from the active checkout
ok - spawn accepts a registered worktree outside the project tree
ok - local-only worktree with HEAD on a fork remote is torn down (fix holds)
ok - local-only worktree with truly unpushed work is refused (safety preserved)
ok - local-only worktree with work merged into local main is torn down (no regression)
ok - no-mistakes worktree with HEAD on origin is torn down (no regression)
ok - no-mistakes worktree with truly unpushed work is refused (no regression)
ok - local-only worktree with unpushed work is torn down under --force (escape hatch)
ok - untracked generated hook files do not block teardown
ok - tracked .claude/settings.local.json changes block teardown
ok - supervise daemon state root is scoped by FM_HOME
ok - concurrent append plus drain preserves queue records
ok - signal written while no watcher runs is caught on next run
ok - stale wake is queued before suppressor state is advanced
ok - check output is queued before cadence suppression
ok - simultaneous watcher starts leave exactly one live process
ok - two atomic drains cannot consume the same records twice
ok - drain collapses obvious duplicate heartbeat and signal records
ok - killed watcher stale lock is reclaimed
ok - live watcher lock with stale heartbeat is actionable
ok - guard warns when queued wakes are pending
ok - guard orders watcher re-arm after queued wake drain
ok - routine signal self-handles
ok - captain-relevant status verbs escalate
ok - check + unknown escalate; heartbeat self-handles
ok - transient stale self-handles and records a persistence marker
ok - stale + terminal status escalates immediately
ok - persistent stale escalates after threshold and clears its marker
ok - resumed (busy) stale clears its marker without escalating
ok - multiple escalations flush as a single batched digest
ok - batch flush measures max-delay from the first append, not the last
ok - catch-all scan escalates a missed terminal once, not twice
ok - handle_wake routes routine->self and captain->escalate
ok - INJECT_SKIP forces self-handle, bypassing captain-relevant classification
ok - is_wake_reason distinguishes watcher wake reasons from singleton-status stdout
ok - terminal-stale escalate removes its marker so housekeeping does not re-escalate
ok - captain signal escalate marks seen so the catch-all scan does not re-fire
ok - _collapse_newlines replaces newlines with literal separator
ok - afk flag absent: daemon does not inject, buffer preserved
ok - afk flag present: daemon injects with sentinel marker prefix
ok - injected digest is single-line (no embedded newlines)
ok - busy-guard defers injection when supervisor pane is busy
ok - marker detection: marker -> stay afk, no marker -> exit afk
ok - /afk invocation is exempt from afk exit (no self-cancel)
ok - should_exit_afk returns false when afk is not active
ok - strip_injection_marker removes the sentinel marker cleanly
ok - pane_input_pending detects partial input on the cursor line
ok - pane_input_pending: blank cursor line is not pending
ok - pane_input_pending: bare prompts are not pending (idle)
ok - composer guard defers injection when pane has pending input
ok - swallowed Enter: type-once + Enter-retry, no concatenation
ok - normal inject: exactly one digest, one Enter, no duplicates
ok - classify_signal dedupes against the catch-all scan seen marker
ok - classify_stale dedupes against the signal path seen marker

Evidence: Rendered firstmate.service

[Unit]
Description=Firstmate persistent supervisor session (crash/reboot resilience)
Documentation=https://github.com/kunchenguid/firstmate
# Wait for the Windows drive mounts; the orchestrator account config lives under
# /mnt/c. Non-fatal if the mount unit name differs - the resume script retries.
After=local-fs.target

[Service]
Type=simple
User=captain
WorkingDirectory="/Users/thomascarney/.no-mistakes/worktrees/73d6a10257e9/01KVTJ77EVGGTJX83VHV0SGHAN"
Environment="HOME=/var/folders/kg/vqcvwwlx3xs4wblm4wpvpkz00000gn/T/no-mistakes-evidence/01KVTJ77EVGGTJX83VHV0SGHAN/home"
Environment="FM_FIRSTMATE_COMMAND=exec /var/folders/kg/vqcvwwlx3xs4wblm4wpvpkz00000gn/T/no-mistakes-evidence/01KVTJ77EVGGTJX83VHV0SGHAN/fake-codex --dangerously-bypass-approvals-and-sandbox"
# The watchdog: ensure the firstmate tmux session exists, re-checking forever.
# On VM boot this recreates the session; if the loop itself ever dies, Restart
# brings it back. The loop re-execs nothing destructive - it no-ops when the
# session is already live.
ExecStart="/Users/thomascarney/.no-mistakes/worktrees/73d6a10257e9/01KVTJ77EVGGTJX83VHV0SGHAN/bin/fm-resume.sh" --watch
Restart=always
RestartSec=10
# Kill ONLY the watchdog loop on stop/restart, never the tmux server it guards.
# This makes firstmate survive `systemctl restart firstmate` and decouples the
# supervisor session's lifetime from this unit: the service is a guardian, not
# the owner. To fully stop firstmate, kill its tmux session directly.
KillMode=process

[Install]
WantedBy=multi-user.target

Evidence: Autostart installer CLI transcript

installed and enabled firstmate.service firstmate will now auto-resurrect on every boot and self-heal if it dies. --- unit --- enabled: no active: no --- session --- firstmate tmux session: not present

installed and enabled firstmate.service
firstmate will now auto-resurrect on every boot and self-heal if it dies.
--- unit ---
enabled: no
active: no
--- session ---
firstmate tmux session: not present

Evidence: fm-resume create transcript

fm-resume: created session 'evidence-firstmate' running firstmate

fm-resume: created session 'evidence-firstmate' running firstmate

Evidence: fm-resume idempotent transcript

fm-resume: session 'evidence-firstmate' already live

fm-resume: session 'evidence-firstmate' already live

Evidence: Firstmate launch evidence

fake codex launched: cwd=/Users/thomascarney/.no-mistakes/worktrees/73d6a10257e9/01KVTJ77EVGGTJX83VHV0SGHAN args=--dangerously-bypass-approvals-and-sandbox

fake codex launched: cwd=/Users/thomascarney/.no-mistakes/worktrees/73d6a10257e9/01KVTJ77EVGGTJX83VHV0SGHAN args=--dangerously-bypass-approvals-and-sandbox

Pipeline

Updates from git push no-mistakes

⏭️ **intent** - skipped

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed (8) ✅

• ⚠️ bin/fm-wake-lib.sh:98 - Legacy directory locks with an empty or not-yet-written pid are reclaimed immediately. During a rolling update, an old mkdir-based holder can create the directory before writing pid; this branch will remove that fresh live lock and allow a second holder. Apply the same fresh-empty stale grace used for file locks before rm -rf.
• ⚠️ bin/fm-resume.sh:32 - Autostart defaults are machine- and harness-specific: the service launches Claude with /mnt/c/Users/Owenz/.claude-orchestrator. In this shared repo, a default install on another user, another Windows profile, or a Codex/opencode firstmate will resurrect the wrong command or fail. Decide whether to render configured harness/bin/config into the unit or keep this intentionally single-machine.

🔧 Fix: Captain, fix autostart and legacy lock races
1 error still open:

• 🚨 tests/fm-lock-exclusivity.test.sh:1 - CI executes every tests/*.test.sh directly, but this new test and tests/fm-resume.test.sh were added with mode 100644, so the behavior-test job will hit Permission denied before running them. Commit both with executable mode 100755.

🔧 Fix: Captain, make new tests executable
2 issues (1 error, 1 warning) still open:

• 🚨 systemd/firstmate.service:10 - The systemd unit always runs the resumed session as root. A normal captain running firstmate as their own user will get a separate root-owned tmux server, root HOME/config/auth, and tmux attach -t firstmate from their account will not see the resurrected session. Render the intended user into the unit or make this a user service.
• ⚠️ bin/fm-teardown.sh:398 - The dirty check now ignores any status line containing .claude/settings.local.json, including tracked or modified files. If a project tracks that path, teardown can proceed and then rm -f it, discarding unlanded work. Restrict the exclusion to the generated untracked file, or rely on the git-info exclude path instead of suppressing tracked changes.

🔧 Fix: Fix autostart user and teardown dirty check
3 issues (2 warnings, 1 info) still open:

• ⚠️ bin/fm-install-autostart.sh:118 - firstmate_command failures are hidden by the nested command substitution, so an unknown harness or missing binary can still install a unit with FM_FIRSTMATE_COMMAND= blank. Capture the raw command with command=$(firstmate_command) || return 1 before quoting, so install fails instead of writing a broken autostart unit, captain.
• ⚠️ bin/fm-resume.sh:93 - In --watch mode, ensure_session || true disables errexit inside the function; if firstmate_command fails here, the function continues, creates the tmux session, sends only cd <root> &&, and future iterations no-op because the empty session now exists. Explicitly return on launch-command failure before tmux new-session.
• ℹ️ bin/fm-install-autostart.sh:173 - The status fallback never runs when systemctl is-enabled or is-active fails because the failing command is piped into sed without pipefail; a missing unit prints neither enabled: no nor active: no. Use an if or enable pipefail for these checks.

🔧 Fix: Captain, fix autostart failure handling
1 warning still open:

• ⚠️ bin/fm-install-autostart.sh:152 - The installer needs system privileges for /etc/systemd/system, but the unit is rendered in the same process that performs the privileged install. A normal user run can infer the active harness but fails at install; a sudo run can write the unit but commonly loses the harness ancestry, user PATH, and nvm-installed binary paths, so default install can fail or require undocumented env overrides. Render the unit as the invoking user before privilege escalation, or document/enforce the required sudo -E/FM_FIRSTMATE_COMMAND flow.

🔧 Fix: Captain, fix autostart privilege split
1 error still open:

• 🚨 bin/fm-spawn.sh:333 - fm-spawn.sh now accepts only pane cwd paths containing /.treehouse/, but treehouse get can return worktrees outside the project, for example the installed treehouse status reports paths under ~/.treehouse/.... Those valid spawns will wait 60s and fail even though the subshell entered a worktree. Validate the cwd as a git worktree for $PROJ_ABS instead of hardcoding the path substring.

🔧 Fix: Captain, validate spawn worktrees by git registration
2 warnings still open:

• ⚠️ bin/fm-install-autostart.sh:69 - If FM_FIRSTMATE_HARNESS names a supported harness but the binary is not on PATH and no FM_<HARNESS>_BIN override is set, firstmate_bin returns nonzero but the caller continues with an empty bin because it is invoked under raw_command=$(firstmate_command) || return 1. This can install a unit with a broken exec --... launch command; make each bin=$(firstmate_bin ...) explicitly || return 1, captain.
• ⚠️ bin/fm-resume.sh:60 - The same missing-binary path exists in direct fm-resume.sh use: a supported-but-unavailable harness can produce an empty exec command, create the tmux session, and then future watchdog iterations no-op because the session exists. Explicitly return when firstmate_bin fails before building the launch command.

🔧 Fix: Captain, guard missing harness binaries
1 warning still open:

• ⚠️ bin/fm-wake-lib.sh:102 - Legacy stale directory cleanup now uses rm -rf "$lockfile"; if two new processes race on the same stale legacy lock, one can remove the directory while another creates the new O_EXCL file lock, and the delayed rm -rf can delete that live file lock. Move the legacy directory aside first, or remove it with pid-file plus rmdir, so cleanup cannot unlink a replacement lock.

🔧 Fix: Captain, harden legacy lock cleanup
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• tests/fm-lock-exclusivity.test.sh
• tests/fm-resume.test.sh
• tests/fm-install-autostart.test.sh
• tests/fm-spawn-worktree.test.sh
• tests/fm-teardown.test.sh
• tests/fm-wake-queue.test.sh
• Manual bin/fm-install-autostart.sh install using fake systemd tools with FM_UNIT_DST writing rendered-firstmate.service into the evidence directory
• Manual bin/fm-resume.sh create/no-op run against an isolated tmux socket, verifying the fake firstmate process launched from the worktree with the expected codex bypass argument
• git status --short to confirm no working-tree artifacts remained

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[ki-za]

Scout plan (huddle-app second mate, 2026-08-29)

Proposed Options / Plan / Blocking / Lead decisions for this ticket, from ha-settings-plan scout report.

Current state

• app/src/lib/app-menu/AppMenu.svelte:36-118 is the existing keyed menu. It exposes font, one combined sent-reply layout, a read-only waiting queue, and open-in-browser. The queue is intentionally not a Huddle switcher (app/README.md:7-10).
• app/src/App.svelte:20-43, 144-153, 289-315 selects only the first queued Huddle and persists font and the combined layout in browser localStorage. app/src/lib/huddle/view-state.ts:3-18, 38-47 has no settings model, members editor, member selection, links, or toggle registry.
• The app-facing contract contains only queue/message/reply/attachment/Accept operations (app/src/lib/contract/client.ts:30-36; app/src/lib/contract/http.ts:44-123). docs/contract.md:29-47 contains no settings route, and src/cli.ts:59-148 contains no settings command.
• The only validated schemas are the four ledger-record schemas (schema/; src/schema.ts:1-5, 92-120). The server is currently the sole home writer (docs/contract.md:6-15).
• The current theme file exposes Lume variables rather than the later role-token set (app/src/theme/lume.css:1-20). #49 remains the theme seam.

Missing: one validated settings schema and one server-owned saved source read identically by the Huddle app and command-line tool; an app editor for the whole tree; and the settled member-selection, member-configuration, module, and toggle contracts.

Options

• Tree editor — makes links spatially visible; connector interaction becomes crowded and inaccessible in the bounded Huddle app column.
• List and inspector — keeps member order and links explicit, gives one selected member a focused editor, and fits the existing menu entrance. Recommended.
• Generated form — covers every registered toggle cheaply; it needs a separate presentation decision before it makes member relationships legible.

Recommended Plan

Ten years on, one schema prevents a browser preference, a command-line tool value, and a server value from becoming three contradictory settings. The plain name is settings because it already names the user entrance and the saved choices.

Modules

• schema/settings.v1.json and its TypeScript validation/loading companion in src/: huddle-infra owns these. This is the single settings schema that the server, command-line tool, and Huddle app read; it is not a ledger record or a second prompt source.
• Huddle-infra server persistence plus documented GET/PUT /api/settings and the matching command-line tool read/write surface in docs/contract.md and src/cli.ts. The server validates before saving and returns the normalized settings.
• app/src/lib/settings/ holds the Huddle app client adapter and editor state. app/src/lib/app-menu/AppMenu.svelte remains the menu entrance; its settings control opens a list-and-inspector view rather than expanding the compact menu into a graph.
• The list shows members and links. The inspector edits one member, its member configuration reference, member selection, and registered toggles. The toggle registry comes from feat: add opencode server backend #44; the Huddle app does not copy module text or define module identity.
• Move saved presentation choices into the same settings model only where their owning tickets make them saved settings: font; replyAlignment; replyMarker; theme; shortcuts; and registered toggles. Keep menu-open state, queue-open state, selected member, and drafts transient.

Seams

• The schema top level contains members, member_selection, links, and toggles. A member references a member configuration. member_selection accepts only none, deterministic, or agent; Watcher silently dies on Linux (stat -f portability crash) — misses all supervision wakes #26 defines what each choice does. feat(bin): recurring automated review sweep across the fleet #39 defines member-configuration contents; feat: add opencode server backend #44 defines module identity, provenance, persistence, precedence, and the toggle registry. This ticket renders and edits those contracts only.
• Huddle-infra owns schema location, validation, home persistence, server transport, command-line tool transport, and docs/contract.md. The Huddle app consumes the validated result through app/src/lib/contract/; it must remove localStorage as authority rather than layering another store on it.
• Settings changes never rewrite a Huddle ledger record, alter a message, alter queue order, or grant a member authority. The server remains the sole writer for the home.
• #45 separates replyAlignment from replyMarker; #47 keeps settings behind the menu; #49 supplies role tokens. Components consume role tokens, never theme-local swatches.

Validation criteria

• A shared settings fixture with two members, member configurations, links, every registered toggle, and each of none, deterministic, and agent validates in schema/settings.v1.json; server, command-line tool, and Huddle app read the same normalized result.
• Edit a member and toggle through the Huddle app; reopen through the command-line tool and after a server restart; the saved tree is identical. Perform the inverse command-line tool edit and confirm the app reloads it.
• The server and command-line tool reject an unknown module, unknown toggle, malformed settings, and a broken link with an actionable validation result; the prior saved settings remain unchanged.
• The Huddle app exposes every registered toggle exactly once, does not expose a module that the registry omits, and preserves the read-only queue/no-switching rule.
• Capture regular and 30rem-width browser smoke evidence: the list-and-inspector remains one bounded, scrollable column; keyboard focus reaches every control; the screenshot shows no theme-local swatch name. Run bun run check.

Blocking?

blocked by #26, #39, and #44: #26 owns member-selection behavior; #39 owns member-configuration shape; #44 owns the module source model, identity, persistence, precedence, and toggle registry. Huddle-infra must also accept the named schema/server/command-line-tool seam before the editor can ship. The menu shell can be prepared independently, but not the complete settings tree.

Lead decisions

None. The requested single schema, the three member-selection values, the menu entrance, and the dependent-contract ownership are already recorded. The list-and-inspector is a reversible implementation choice.