feat: add crash and reboot resilience for firstmate

AGENTS.md

@@ -67,7 +67,8 @@ README.md            public overview and development notes
 .github/workflows/   shared CI and PR enforcement, committed
 .agents/skills/      shared skills, committed
 .claude/skills       symlink to .agents/skills for claude compatibility
-bin/                 helper scripts, committed, including fm-fleet-sync.sh for clean default-branch refreshes and gone-branch pruning; read each script's header before first use
+systemd/             firstmate.service unit for crash/reboot autostart (section 5); install via bin/fm-install-autostart.sh
+bin/                 helper scripts, committed, including fm-fleet-sync.sh for clean default-branch refreshes and gone-branch pruning, fm-resume.sh (autostart watchdog) and fm-install-autostart.sh; read each script's header before first use
 config/crew-harness  crewmate harness override; LOCAL, gitignored; absent or "default" = same as firstmate
 data/                personal fleet records; LOCAL, gitignored as a whole
   backlog.md         task queue, dependencies, history
@@ -223,6 +224,8 @@ Reconcile reality with your records before doing anything else:
 A firstmate restart must be a non-event.
 All truth lives in tmux, state files, data/backlog.md, data/secondmates.md, persistent secondmate homes, and treehouse; your conversation memory is a cache.
 
+**Autostart (crash/reboot resilience).** A WSL VM teardown (host sleep, idle timeout, or a Windows Update reboot) kills tmux and every crewmate at once, and nothing relaunched firstmate after the VM came back. `systemd/firstmate.service` closes that gap: a watchdog (`bin/fm-resume.sh --watch`) recreates the persistent `firstmate` tmux session on boot and self-heals if it dies, so the captain re-attaches (`tmux attach -t firstmate`) to a live, state-intact firstmate instead of a cold start. Install/remove with `bin/fm-install-autostart.sh [install|status|uninstall]`; the unit's `KillMode=process` means stopping the service never kills a running firstmate. Pair it with `~/.wslconfig` `vmIdleTimeout=-1` (prevents the idle teardown in the first place; needs `wsl --shutdown` to apply). The watchdog does NOT relaunch crewmates - resuming in-flight work is recovery's job once the captain is back.
+
 ## 6. Project management
 
 All projects live flat under `projects/`.
@@ -579,7 +582,7 @@ This is why fewer, cheaper firstmate turns handle the same fleet.
   That empty composer is the acknowledgement that the submit landed, using the same border-aware detector so a bordered-empty claude composer counts as submitted rather than a false "swallowed Enter".
   `fm-send.sh` shares this primitive and exits non-zero on a positively-confirmed swallow, so firstmate learns a steer did not land instead of leaving it unsubmitted.
 - **Marker strip** - `strip_injection_marker` removes the sentinel prefix before classification/relay, so the digest text firstmate sees is clean.
-- **Portable singleton lock** - the daemon uses the repo's mkdir-based lock helper (`fm-wake-lib.sh`) instead of `flock`, which is absent on macOS.
+- **Portable singleton lock** - the daemon uses the repo's O_EXCL file-lock helper (`fm-wake-lib.sh`) instead of `flock`, which is absent on macOS.
 - **Dedupe across signal/stale/scan** - `classify_signal` and `classify_stale` both check the seen-status marker before escalating, so a status escalated by one path is not re-escalated by another in the same digest.
 - **Auto-discovered supervisor pane** - the daemon resolves its injection target from `FM_SUPERVISOR_TARGET`, then `$TMUX_PANE` (inherited from the pane that launched it), then a `firstmate:0` fallback with a warning; the resolution source is logged at startup so a wrong-but-resolving fallback is detectable.
 

README.md

@@ -137,8 +137,9 @@ firstmate works from any terminal - outside tmux, crewmates land in a detached `
 - **Project memory belongs to projects** - durable project-intrinsic agent knowledge lives in each project's committed `AGENTS.md`, with `CLAUDE.md` as a symlink.
   Ship briefs prompt crewmates to create or update those files through the normal delivery path; `data/projects.md` stays a thin private registry.
 - **Local clones stay fresh** - bootstrap and PR-based teardown refresh remote-backed project clones with clean default-branch fast-forwards when the clone is on the default branch and has no local work, and prune local branches whose remote is gone and that no worktree still needs.
-- **Restart-proof** - all state lives in tmux, status files, local markdown under `data/`, `data/secondmates.md`, and persistent secondmate homes.
+- **Restart-proof, with optional autostart** - all state lives in tmux, status files, local markdown under `data/`, `data/secondmates.md`, and persistent secondmate homes.
   Kill the first mate session anytime; the next one reconciles and carries on.
+  On WSL with systemd, `bin/fm-install-autostart.sh` installs `systemd/firstmate.service`, whose `bin/fm-resume.sh --watch` loop recreates the persistent `firstmate` tmux session after VM teardown, host reboot, or watchdog death.
 
 ## The bin/ toolbelt
 
@@ -148,6 +149,8 @@ The first mate drives these; you rarely need to, but they work by hand too.
 | ------------------------ | ------------------------------------------------------------------------------------------------------------------- |
 | `fm-bootstrap.sh`        | Detect missing or outdated toolchain pieces; refresh clones best-effort; install tools only after consent           |
 | `fm-fleet-sync.sh`       | Fetch clones, clean-fast-forward their checked-out default branches, and safely prune branches whose remote is gone |
+| `fm-install-autostart.sh` | Install, inspect, or remove the WSL systemd unit that keeps the persistent firstmate session alive across VM teardown |
+| `fm-resume.sh`           | Idempotently ensure the persistent firstmate tmux session exists; `--watch` repeats as the systemd watchdog loop     |
 | `fm-backlog-handoff.sh`  | Move already-judged in-scope queued backlog items from the main home into a seeded secondmate home                 |
 | `fm-brief.sh`            | Scaffold a ship brief, a report-only scout brief with `--scout`, or a secondmate charter with `--secondmate`      |
 | `fm-ensure-agents-md.sh` | Ensure project `AGENTS.md` is the real memory file and `CLAUDE.md` symlinks to it                                   |
@@ -204,6 +207,20 @@ FM_BUSY_REGEX='esc (to )?interrupt|Working\.\.\.'   # busy-pane signatures, shar
 FM_COMPOSER_IDLE_RE=    # optional empty-composer regex, applied after border stripping
 FM_SEND_RETRIES=3       # fm-send Enter-retry attempts after typing the line once
 FM_SEND_SLEEP=0.4       # seconds between fm-send submit checks
+FM_LOCK_STALE_AFTER=10  # seconds before a fresh empty lock file may be reclaimed as stale
+# crash/reboot autostart (bin/fm-resume.sh and systemd/firstmate.service)
+FM_SESSION=firstmate    # persistent supervisor tmux session name
+FM_FIRSTMATE_HARNESS=   # optional explicit harness: claude, codex, opencode, or pi
+FM_FIRSTMATE_COMMAND=   # optional full launch command rendered into systemd autostart
+FM_CLAUDE_BIN=          # optional Claude binary path for fm-resume.sh / autostart rendering
+FM_CODEX_BIN=           # optional Codex binary path for fm-resume.sh / autostart rendering
+FM_OPENCODE_BIN=        # optional opencode binary path for fm-resume.sh / autostart rendering
+FM_PI_BIN=              # optional pi binary path for fm-resume.sh / autostart rendering
+FM_CONFIG_DIR=          # optional Claude config dir for fm-resume.sh
+FM_RESUME_INTERVAL=60   # seconds between watchdog checks in fm-resume.sh --watch
+FM_AUTOSTART_USER=      # optional systemd User= override for fm-install-autostart.sh
+FM_AUTOSTART_HOME=      # optional HOME= override for the rendered systemd unit
+FM_UNIT_DST=/etc/systemd/system/firstmate.service  # optional unit destination, mainly for tests
 # sub-supervisor (bin/fm-supervise-daemon.sh); presence-gated via /afk
 FM_SUPERVISOR_TARGET=firstmate:0   # supervisor tmux target (override; auto-discovers from $TMUX_PANE)
 FM_INJECT_SKIP=heartbeat           # |-prefixes force-self-handled bypassing classification; empty disables
@@ -224,16 +241,20 @@ Human-authored pull requests targeting `main` must be raised through `git push n
 Local `.no-mistakes/` state and test evidence stay out of this repo; `.no-mistakes.yaml` keeps evidence in a temp directory instead.
 The current watcher reliability work keeps the one-shot process model and adds a durable queue plus singleton lock.
 The presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) provides proactive wake routing for walk-away supervision via the `/afk` skill; a blocking-waiter split remains a deferred follow-up phase.
+The crash/reboot resilience work adds a WSL systemd watchdog (`systemd/firstmate.service`) plus `fm-resume.sh`, and replaces wake-queue lock grants with an atomic `O_EXCL` file create so concurrent watchers cannot double-acquire the lock on WSL filesystems.
 
 ```sh
 bash -n bin/*.sh                          # syntax-check the toolbelt
 shellcheck bin/*.sh tests/*.sh            # lint the toolbelt and behavior tests; CI enforces this
 for test_script in tests/*.test.sh; do "$test_script"; done   # behavior tests, matching CI
+tests/fm-install-autostart.test.sh        # systemd unit rendering for the active checkout
+tests/fm-resume.test.sh                   # tmux session resurrection and idempotence under a private tmux server
+tests/fm-lock-exclusivity.test.sh         # wake-queue lock mutual exclusion, dead-holder reclaim, and live-holder protection
 tests/fm-wake-queue.test.sh               # durable wake queue, singleton behavior, sub-supervisor classifier, /afk presence-gating, border-aware composer, max-defer, and fm-send submit tests
 tests/fm-afk-inject-e2e.test.sh           # private-socket end-to-end test of the afk injection path (partial-input deferral, swallowed-Enter retry)
 tests/fm-bootstrap.test.sh                # bootstrap dependency and feature-probe tests
 tests/fm-secondmate.test.sh               # persistent secondmate routing, seeding, idle charter, backlog handoff, spawn, recovery, teardown, and FM_HOME tests
-tests/fm-teardown.test.sh                 # fm-teardown.sh unpushed-work safety check: local-only fork-remote allow, truly-unpushed refuse, merged-to-main allow, no-mistakes regression, --force override
+tests/fm-teardown.test.sh                 # fm-teardown.sh unpushed-work safety check, generated-hook cleanup, and --force override
 [ "$(readlink CLAUDE.md)" = "AGENTS.md" ]
 [ "$(readlink .claude/skills)" = "../.agents/skills" ]
 FM_HEARTBEAT=2 FM_POLL=1 bin/fm-watch.sh  # watcher smoke test (prints "heartbeat")

bin/fm-install-autostart.sh

@@ -0,0 +1,227 @@
+#!/usr/bin/env bash
+# Install (or remove) the systemd unit that makes firstmate survive a WSL VM
+# teardown - the failure that took the fleet out overnight (AGENTS.md section 5).
+#
+# systemd runs as PID 1 in this distro ([boot] systemd=true in /etc/wsl.conf),
+# so a system unit started at multi-user.target is the native, reboot-proof way
+# to auto-resurrect firstmate. The unit runs bin/fm-resume.sh as a watchdog.
+#
+# Usage:
+#   fm-install-autostart.sh            install, enable, and start the unit
+#   fm-install-autostart.sh status     show unit + session state
+#   fm-install-autostart.sh uninstall  stop, disable, and remove the unit
+#
+# Useful overrides:
+#   FM_FIRSTMATE_HARNESS / FM_FIRSTMATE_COMMAND choose the launch command.
+#   FM_CLAUDE_BIN, FM_CODEX_BIN, FM_OPENCODE_BIN, or FM_PI_BIN pin a binary.
+#   FM_AUTOSTART_USER / FM_AUTOSTART_HOME choose the systemd User and HOME.
+#   FM_UNIT_DST changes the rendered unit destination, mostly for tests.
+#
+# Reversible: `uninstall` leaves no trace and never touches a running firstmate
+# session (KillMode=process in the unit), so removing autostart cannot discard work.
+set -eu
+
+FM_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+UNIT_SRC="$FM_ROOT/systemd/firstmate.service"
+UNIT_NAME="firstmate.service"
+UNIT_DST="${FM_UNIT_DST:-/etc/systemd/system/$UNIT_NAME}"
+SYSTEM_UNIT_DST="/etc/systemd/system/$UNIT_NAME"
+
+shell_quote() {
+  local out
+  printf -v out '%q' "$1"
+  printf '%s' "$out"
+}
+
+systemd_quote() {
+  local value=$1
+  value=${value//\\/\\\\}
+  value=${value//\"/\\\"}
+  printf '%s' "$value"
+}
+
+firstmate_bin() {
+  local harness=$1 var value
+  var="FM_$(printf '%s' "$harness" | tr '[:lower:]' '[:upper:]')_BIN"
+  eval "value=\${$var:-}"
+  if [ -n "$value" ]; then
+    printf '%s\n' "$value"
+    return 0
+  fi
+  command -v "$harness"
+}
+
+firstmate_command() {
+  local harness bin config
+  if [ -n "${FM_FIRSTMATE_COMMAND:-}" ]; then
+    printf '%s\n' "$FM_FIRSTMATE_COMMAND"
+    return 0
+  fi
+  harness="${FM_FIRSTMATE_HARNESS:-}"
+  if [ -z "$harness" ]; then
+    harness=$("$FM_ROOT/bin/fm-harness.sh" 2>/dev/null || echo unknown)
+  fi
+  case "$harness" in
+    claude)
+      bin=$(firstmate_bin claude) || return 1
+      config="${FM_CONFIG_DIR:-${CLAUDE_CONFIG_DIR:-}}"
+      if [ -n "$config" ]; then
+        printf 'CLAUDE_CONFIG_DIR=%s IS_SANDBOX=1 exec %s --dangerously-skip-permissions\n' "$(shell_quote "$config")" "$(shell_quote "$bin")"
+      else
+        printf 'IS_SANDBOX=1 exec %s --dangerously-skip-permissions\n' "$(shell_quote "$bin")"
+      fi
+      ;;
+    codex)
+      bin=$(firstmate_bin codex) || return 1
+      printf 'exec %s --dangerously-bypass-approvals-and-sandbox\n' "$(shell_quote "$bin")"
+      ;;
+    opencode)
+      bin=$(firstmate_bin opencode) || return 1
+      printf 'OPENCODE_CONFIG_CONTENT=%s exec %s\n' "$(shell_quote '{"permission":{"*":"allow"}}')" "$(shell_quote "$bin")"
+      ;;
+    pi)
+      bin=$(firstmate_bin pi) || return 1
+      printf 'exec %s\n' "$(shell_quote "$bin")"
+      ;;
+    *)
+      echo "error: cannot infer firstmate launch command for harness '$harness'; set FM_FIRSTMATE_COMMAND" >&2
+      return 1
+      ;;
+  esac
+}
+
+autostart_user() {
+  if [ -n "${FM_AUTOSTART_USER:-}" ]; then
+    printf '%s\n' "$FM_AUTOSTART_USER"
+  elif [ "$(id -u)" -eq 0 ] && [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != root ]; then
+    printf '%s\n' "$SUDO_USER"
+  else
+    id -un
+  fi
+}
+
+autostart_home() {
+  local user=$1 home
+  if [ -n "${FM_AUTOSTART_HOME:-}" ]; then
+    printf '%s\n' "$FM_AUTOSTART_HOME"
+    return 0
+  fi
+  home=$(getent passwd "$user" 2>/dev/null | cut -d: -f6 || true)
+  if [ -n "$home" ]; then
+    printf '%s\n' "$home"
+    return 0
+  fi
+  if [ "$user" = "$(id -un)" ]; then
+    printf '%s\n' "$HOME"
+    return 0
+  fi
+  echo "error: cannot determine home for autostart user '$user'; set FM_AUTOSTART_HOME" >&2
+  return 1
+}
+
+render_unit() {
+  local root raw_command command raw_user user raw_home home line
+  root=$(systemd_quote "$FM_ROOT")
+  raw_command=$(firstmate_command) || return 1
+  command=$(systemd_quote "$raw_command")
+  raw_user=$(autostart_user)
+  user=$(systemd_quote "$raw_user")
+  raw_home=$(autostart_home "$raw_user") || return 1
+  home=$(systemd_quote "$raw_home")
+  while IFS= read -r line || [ -n "$line" ]; do
+    line=${line//@FM_ROOT@/$root}
+    line=${line//@FM_FIRSTMATE_COMMAND@/$command}
+    line=${line//@FM_USER@/$user}
+    printf '%s\n' "${line//@FM_HOME@/$home}"
+  done < "$UNIT_SRC"
+}
+
+require_systemd() {
+  if ! command -v systemctl >/dev/null 2>&1; then
+    echo "error: systemctl not found; this distro is not running systemd" >&2
+    exit 1
+  fi
+  if [ "$(ps -p 1 -o comm= 2>/dev/null)" != systemd ]; then
+    echo "error: PID 1 is not systemd; enable '[boot] systemd=true' in /etc/wsl.conf, then 'wsl --shutdown'" >&2
+    exit 1
+  fi
+}
+
+requires_privilege() {
+  [ "$UNIT_DST" = "$SYSTEM_UNIT_DST" ] || return 1
+  [ "$(id -u)" -ne 0 ]
+}
+
+run_privileged() {
+  if requires_privilege; then
+    sudo "$@"
+  else
+    "$@"
+  fi
+}
+
+refuse_ambiguous_sudo_render() {
+  if [ "$(id -u)" -eq 0 ] && [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != root ] && [ -z "${FM_FIRSTMATE_COMMAND:-}" ]; then
+    echo "error: run without sudo so the firstmate launch command is captured from your user session" >&2
+    echo "       or set FM_FIRSTMATE_COMMAND explicitly when invoking through sudo" >&2
+    exit 1
+  fi
+}
+
+cmd_install() {
+  local rendered
+  require_systemd
+  [ -f "$UNIT_SRC" ] || { echo "error: unit not found at $UNIT_SRC" >&2; exit 1; }
+  refuse_ambiguous_sudo_render
+  rendered=$(mktemp "${TMPDIR:-/tmp}/firstmate.service.XXXXXX")
+  trap 'rm -f "$rendered"' EXIT
+  render_unit > "$rendered"
+  # Install a copy (not a symlink): systemd does not follow symlinks under
+  # /mnt/c reliably, and the repo path is not guaranteed mounted at early boot.
+  run_privileged install -m 0644 "$rendered" "$UNIT_DST"
+  rm -f "$rendered"
+  trap - EXIT
+  run_privileged systemctl daemon-reload
+  run_privileged systemctl enable "$UNIT_NAME"
+  run_privileged systemctl restart "$UNIT_NAME"
+  echo "installed and enabled $UNIT_NAME"
+  echo "firstmate will now auto-resurrect on every boot and self-heal if it dies."
+  cmd_status
+}
+
+cmd_uninstall() {
+  require_systemd
+  run_privileged systemctl disable "$UNIT_NAME" 2>/dev/null || true
+  run_privileged systemctl stop "$UNIT_NAME" 2>/dev/null || true
+  run_privileged rm -f "$UNIT_DST"
+  run_privileged systemctl daemon-reload
+  echo "removed $UNIT_NAME (any running firstmate session was left untouched)"
+}
+
+cmd_status() {
+  require_systemd
+  echo "--- unit ---"
+  if enabled=$(systemctl is-enabled "$UNIT_NAME" 2>/dev/null); then
+    printf 'enabled: %s\n' "$enabled"
+  else
+    echo "enabled: no"
+  fi
+  if active=$(systemctl is-active "$UNIT_NAME" 2>/dev/null); then
+    printf 'active: %s\n' "$active"
+  else
+    echo "active: no"
+  fi
+  echo "--- session ---"
+  if tmux has-session -t "${FM_SESSION:-firstmate}" 2>/dev/null; then
+    echo "firstmate tmux session: LIVE (attach with: tmux attach -t ${FM_SESSION:-firstmate})"
+  else
+    echo "firstmate tmux session: not present"
+  fi
+}
+
+case "${1:-install}" in
+  install)   cmd_install ;;
+  uninstall) cmd_uninstall ;;
+  status)    cmd_status ;;
+  *) echo "usage: $(basename "$0") [install|status|uninstall]" >&2; exit 2 ;;
+esac