Skip to content

feat(bin): launchd watchdog backstop for watcher continuity - #2545

Closed
hnvivek wants to merge 12 commits into
kunchenguid:mainfrom
hnvivek:sync/upstream-20260817
Closed

hnvivek wants to merge 12 commits into
kunchenguid:mainfrom
hnvivek:sync/upstream-20260817

Conversation

@hnvivek

@hnvivek hnvivek commented Aug 17, 2026

Copy link
Copy Markdown

Intent

Adopt three weeks of upstream firstmate development (132 commits since merge-base a5fe1bc) into hnvivek/firstmate as one reviewed merge, resolving all overlap in favor of upstream where upstream already landed our local fixes (session-lock ancestry, CLAUDE_CONFIG_DIR spawn forwarding, Bash 3.2 brief scaffolding), while preserving the local launchd watcher-watchdog backstop feature and its tests/docs/lane wiring.

What Changed

  • Merged three weeks of upstream firstmate development (132 commits) into this branch, resolving overlap in favor of upstream where it had already landed the local fixes for session-lock Claude bg-spare ancestry, CLAUDE_CONFIG_DIR forwarding to claude crewmates, and Bash 3.2 parsing of fm-brief.sh.
  • Added a launchd watchdog backstop for watcher continuity: bin/fm-watchdog-check.sh re-arms the watcher when the continuity beacon is missing or stale past FM_GUARD_GRACE (skipping AFK, no-work, child-worktree, and held-owner-lock cases), and bin/fm-watchdog-install.sh installs/uninstalls/audits the com.firstmate.watcher-watchdog LaunchAgent, stamping FM_HOME and the user's PATH into the plist at install.
  • Documented the watchdog in docs/watcher-watchdog.md and cross-referenced it from the watcher-continuity, turnend-guard, and configuration docs; added tests/fm-watchdog-check.test.sh covering arm/no-arm behavior, installer verbs, and the plist template contract.

Risk Assessment

✅ Low: The only new change since the reviewed head is a well-bounded fix for the round-1 PATH finding — correctly sed-escaped, wired into the template contract test, and documented — leaving just one rare, fail-loud XML-escaping edge as an informational note.

Testing

Ran the watchdog backstop's own test suite plus the suites covering the three upstream-resolved local fixes (all pass), then demonstrated the feature end-to-end outside the test harness: in a hermetic fixture home the checker armed exactly when the beacon was stale and a live claude session owned the lock, stayed quiet while the beacon was fresh or the owner was dead, and never exited nonzero; the install-time PATH stamping was verified to yield a valid launchd plist under the real user PATH, and the installer's status verb ran cleanly. No failures; working tree left clean and fixtures removed.

Evidence: End-to-end watchdog checker transcript (4-tick lifecycle in a hermetic fixture home)
firstmate watcher-watchdog end-to-end check
fixture home : /tmp/fm-wd-e2e.C5cndS/home (plain git checkout, AGENTS.md, bin/, state/)
session lock : pid 50174 -> /tmp/fm-wd-e2e.C5cndS/fakebin/claude (claude-named, live)
in-flight    : state/inflight.meta present
grace window : FM_GUARD_GRACE=2s

== tick 1: beacon absent (watcher died silently) ==
checker exit=0
beacon after : fresh(0.1s)
arms fired   : 1

== tick 2: beacon fresh (a watcher is now supervising) ==
checker exit=0
new arms     : 0 (expected 0)

== tick 3: beacon stale again after idle gap ==
checker exit=0
new arms     : 1 (expected 1)
beacon after : fresh(0.1s)

== tick 4: session owner exited (home no longer owned) — stale beacon must NOT arm ==
checker exit=0
new arms     : 0 (expected 0)

total arms fired across all ticks: 2
Evidence: Stamped watchdog LaunchAgent plist as launchd would load it (real user PATH, per-home label)
<?xml version="1.0" encoding="UTF-8"?>
<!--
  launchd per-user agent template for the firstmate watcher watchdog.

  This is a TEMPLATE, not a loadable plist: bin/fm-watchdog-install.sh stamps the
  com.firstmate.watcher-watchdog.Users-vivek-Documents-firstmate, /Users/vivek/Documents/firstmate/bin/fm-watchdog-check.sh, /Users/vivek/Documents/firstmate, and /Users/vivek/.kimi-code/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/opt/homebrew/bin:/Users/vivek/.local/bin:/Users/vivek/go/bin:/Users/vivek/.cargo/bin:/Users/vivek/bin:/opt/homebrew/sbin:/usr/local/sbin:/Users/vivek/.lmstudio/bin:/Users/vivek/.claude/plugins/cache/ponytail/ponytail/4.8.3/bin placeholders with this home's
  real absolute paths, the installing user's PATH, and a per-home unique label,
  then loads the result from ~/Library/LaunchAgents. One stamped copy per firstmate home keeps several homes
  under one user from colliding. See docs/watcher-watchdog.md.

  RunAtLoad arms once at login/reboot so a home with in-flight work is never left
  blind across a restart; StartInterval re-checks the beacon on a short timer so a
  watcher that dies silently mid-gap is restored within the grace window,
  independent of any harness turn cadence.
-->
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.firstmate.watcher-watchdog.Users-vivek-Documents-firstmate</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/bash</string>
    <string>/Users/vivek/Documents/firstmate/bin/fm-watchdog-check.sh</string>
  </array>
  <key>EnvironmentVariables</key>
  <dict>
    <key>FM_HOME</key>
    <string>/Users/vivek/Documents/firstmate</string>
    <key>PATH</key>
    <string>/Users/vivek/.kimi-code/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/Users/vivek/.antigravity-ide/antigravity-ide/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/opt/homebrew/bin:/Users/vivek/.local/bin:/Users/vivek/go/bin:/Users/vivek/.cargo/bin:/Users/vivek/bin:/opt/homebrew/sbin:/usr/local/sbin:/Users/vivek/.lmstudio/bin:/Users/vivek/.claude/plugins/cache/ponytail/ponytail/4.8.3/bin</string>
  </dict>
  <key>RunAtLoad</key>
  <true/>
  <key>StartInterval</key>
  <integer>90</integer>
  <key>StandardOutPath</key>
  <string>/Users/vivek/Documents/firstmate/state/watchdog.out.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/vivek/Documents/firstmate/state/watchdog.err.log</string>
</dict>
</plist>
Evidence: Installer status CLI transcript
$ FM_HOME=/Users/vivek/.no-mistakes/worktrees/052c8ba410de/01M08FK27FPSFN55NY6FPJP7KF bin/fm-watchdog-install.sh status
watchdog home:  /Users/vivek/.no-mistakes/worktrees/052c8ba410de/01M08FK27FPSFN55NY6FPJP7KF
watchdog label: com.firstmate.watcher-watchdog.Users-vivek-no-mistakes-worktrees-052c8ba410de-01M08FK27FPSFN55NY6FPJP7KF
watchdog plist: /Users/vivek/Library/LaunchAgents/com.firstmate.watcher-watchdog.Users-vivek-no-mistakes-worktrees-052c8ba410de-01M08FK27FPSFN55NY6FPJP7KF.plist (absent)
watchdog loaded: no
exit=0

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

⚠️ **Rebase** - 1 warning

Push main to origin, or rebase your branch onto origin/main, before gating.

⚠️ **Review** - 1 info
  • ⚠️ launchd/com.firstmate.watcher-watchdog.plist:26 - The agent sets only FM_HOME in EnvironmentVariables, so the checker and the watcher it arms run under launchd's minimal PATH (/usr/bin:/bin:/usr/sbin:/sbin), not the user's shell PATH. fm-x-poll.sh requires jq (line 103), which macOS does not ship at /usr/bin — on a Homebrew jq install the X-mode poll degrades to a rate-limited 'x-mode-error missing jq' line, which the watcher classifies as an actionable check: wake. The watchdog then re-arms on every grace window with no session to drain, accumulating spurious wake-queue entries instead of providing supervision. The same minimal PATH can also make the scope gate's git -C (bin/fm-primary-scope-lib.sh:26) fail on machines without CLT's /usr/bin/git, leaving the watchdog silently inert.
  • ℹ️ bin/fm-watchdog-install.sh:43 - home_slug collapses all non-alphanumeric characters to a single dash, so two distinct homes whose paths differ only in such characters (e.g. /Users/x/my.home vs /Users/x/my-home) map to the same launchd label; installing the second silently repoints the shared plist to the second FM_HOME, and uninstalling either removes both homes' agent. The comment promises the slug keeps homes distinct, but it only makes collisions unlikely, not impossible.
  • ℹ️ bin/fm-watchdog-check.sh:105 - session_lock_live exits 0 whenever state/.lock names a dead harness pid and never reclaims it (unlike the Stop auto-arm, which recovers stale locks via fm-lock.sh). So if the harness itself crashes mid-idle-gap — not just the watcher — the home stays unsupervised until a new session starts, even with the watchdog installed. This is the documented deliberate boundary (header lines 41-46), noted here only so the backstop's coverage ceiling is visible during review.

🔧 Fix: Stamp user PATH into watchdog LaunchAgent plist at install
1 info still open:

  • ℹ️ bin/fm-watchdog-install.sh:59 - sed_escape covers the sed replacement side but not XML: a PATH or home path containing & (or <, >) is stamped verbatim into the plist, producing invalid XML that launchd refuses to load. The failure is loud (do_install dies with the launchctl diagnostic pointing at the plist), and such PATHs are rare, so this is a hardening note, not a merge blocker. XML-escaping stamped values (or running plutil -lint before load) would close it.
✅ **Test** - passed

✅ No issues found.

  • bash tests/fm-watchdog-check.test.sh — full watchdog behavior suite (12 checks: arm on missing/stale beacon, no-op on fresh beacon/no work/AFK/no live lock/child worktree/held owner lock, no double-arm, installer verbs, plist template contract)
  • bash tests/fm-session-lock-ancestry.test.sh — upstream resolution of the local session-lock ancestry fix
  • bash tests/fm-spawn-dispatch-profile.test.sh — upstream resolution of the local CLAUDE_CONFIG_DIR spawn forwarding fix (explicit 'claude forwards firstmate's CLAUDE_CONFIG_DIR' checks)
  • bash tests/fm-brief.test.sh — upstream resolution of the local Bash 3.2 brief scaffolding fix
  • bash tests/fm-watch-arm.test.sh — shared single-flight arm path the watchdog checker foregrounds
  • Manual e2e: hermetic fixture primary home (git checkout + AGENTS.md + bin/state) with fake fm-watch-arm.sh and a claude-named live session pid in state/.lock; ran bin/fm-watchdog-check.sh across four ticks (beacon absent → armed; fresh → no arm; stale after grace → armed; session owner killed → no arm) at FM_GUARD_GRACE=2
  • Manual verification of commit 11f13b6: replicated install-time stamp_plist against the real user PATH, validated the stamped plist with plutil, and extracted Label/ProgramArguments/RunAtLoad/StartInterval as launchd reads them
  • FM_HOME=$(pwd) bin/fm-watchdog-install.sh status — real installer CLI verb, read-only, exit 0
⚠️ **Document** - 1 info
  • ℹ️ docs/fm-test-portable-shards.md:79 - The portable-serial shard table's script counts (15/18/17/19 of 69 scripts) predate this change and now differ further from actual lane membership (25/30/28/28 of 111, confirmed via bin/fm-test-run.sh --list). The change added tests/fm-watchdog-check.test.sh to portable-serial without a weight hint, so it lands under PORTABLE_SERIAL_DEFAULT_WEIGHT_MS. Not edited: the table is a dated maintainer-verification record of the 2026-08-02 CI measurement and its prose scopes it to that run, and the doc's own refresh procedure requires downloading fresh CI timing artifacts rather than hand-authoring counts. Pre-existing drift, marginally extended by this change.
✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

hnvivek and others added 11 commits July 26, 2026 18:02
The three ship-mode Definition-of-done blocks built the brief text with
`DOD=$(cat <<EOF ... EOF)` nested inside `case`. Bash 3.2's lexer loses
quote state across a heredoc-in-$(...) inside `case`, so a single
apostrophe anywhere in the body (reintroduced in kunchenguid#945) aborted the whole
script with "unexpected EOF while looking for matching ')'". The scout
and secondmate paths use a plain `cat > file <<EOF` redirect (no $(...))
and were unaffected.

Rewrite each DOD block as `IFS= read -r -d '' DOD <<EOF || true` plus a
single `${DOD%$'\n'}` strip. Byte-for-byte identical brief output proven
against the bash-5 baseline for no-mistakes, direct-PR, local-only, and
scout scaffolds, under both /bin/bash 3.2 and bash 5.

Also extend the existing macos-stock-bash CI job to loop /bin/bash -n
over every bin/*.sh and bin/backends/*.sh, so any future bash-3.2-only
syntax regression in any tracked shell script fails before merge.
…claude pid (kunchenguid#1206)

* fix(session-lock): resolve Claude bg-spare ancestry to the outermost claude pid

fm_harness_ancestry_pid() previously returned the first ancestor process
whose command matched a verified harness name. Claude Code's Stop hook
fires as a bg-spare worker several levels below the session's actual
lock-owning claude process (hook shell -> claude bg-spare ->
claude bg-pty-host -> claude -> claude(lock)), so the first match was
the bg-spare worker, not the lock owner. fm_session_lock_owned_by_self()
then never matched state/.lock, and the Claude Stop auto-arm silently
treated its own primary session as an unrelated live owner and never
armed the watcher.

The walk now keeps going past a claude-named match, looking for a still
more ancestral claude-named match, and stops the instant a non-match
follows an already-found match (bounding it to a contiguous run rather
than the literal ancestry top, so an unrelated claude-named process
further up the real process tree is never mistaken for part of this
session's own nested chain). Every other harness keeps the original
first-match-wins behavior, since e.g. Pi's shared signed-wrapper
ancestry actually holds the session at the inner engine pid, not an
outer wrapper pid. Hop limit raised from 8 to 16 to cover the deeper
bg-spare chain.

* no-mistakes(review): Add nested-claude-ancestry regression test; fix nudge doc depth claim

* no-mistakes: apply CI fixes
…d#1195)

* fix(spawn): forward firstmate's CLAUDE_CONFIG_DIR to claude crewmates

Crewmate panes are created by a long-lived tmux/herdr daemon that does not
inherit firstmate's current environment. When firstmate runs under a non-default
CLAUDE_CONFIG_DIR (for example a work-vs-personal subscription split), a bare
`claude` in the crewmate pane fell back to the default ~/.claude store and
launched unauthenticated, blocking the crewmate before it could do any work.

fm-spawn now prefixes the claude launch with firstmate's own resolved
CLAUDE_CONFIG_DIR when set, so the crewmate uses the same credential/config
store firstmate is authenticated with. An unset value is the single-store
default and adds no prefix; non-claude harnesses are unaffected.

Adds three tests in fm-spawn-dispatch-profile.test.sh (forwarded-when-set,
omitted-when-unset, non-claude-ignored) and pins CLAUDE_CONFIG_DIR in the test
helper so launch assertions no longer depend on the developer's environment.

* no-mistakes: apply CI fixes
Add a turn-independent continuity layer that re-arms a silently-dead
watcher on a launchd timer, closing the gap where a watcher dies during
an idle gap with no Stop hook to re-arm it and state/.last-watcher-beat
goes stale for hours.

- bin/fm-watchdog-check.sh: checker reusing the Stop auto-arm's gates
  (primary scope, supervision need, not-AFK, live session lock) plus the
  shared single-flight owner lock (state/.claude-autoarm.lock), so it
  never double-arms with the hook and only acts when the beacon is past
  grace. It foregrounds the real arm wrapper (never shell &) and never
  writes outcome=rewake, since a launchd arm produces no Claude
  continuation and that outcome would falsely suppress one. A launchd
  process tree is not a harness ancestry, so it gates on a live lock
  holder rather than owned-by-self.
- launchd/com.firstmate.watcher-watchdog.plist: per-user agent template
  (RunAtLoad, 90s StartInterval) the installer stamps per home.
- bin/fm-watchdog-install.sh: stamp/load/unload + status; per-home label
  so homes under one user do not collide; idempotent reload; fails loud
  off-macOS.
- tests/fm-watchdog-check.test.sh: re-arm vs no-op (fresh beacon, each
  gate, held single-flight) and never-double-arm concurrency, plus the
  installer and plist-template contract.
- docs/watcher-watchdog.md with a cross-reference from
  watcher-continuity.md; scripts.md, documentation-audiences.json, and
  the watcher-wake-lock test family wiring.
# Conflicts:
#	.github/workflows/ci.yml
#	CONTRIBUTING.md
#	bin/backends/herdr.sh
#	bin/fm-brief.sh
#	bin/fm-session-lock-lib.sh
#	bin/fm-spawn.sh
#	bin/fm-test-run.sh
#	docs/configuration.md
#	docs/scripts.md
#	docs/sessionstart-nudge.md
#	docs/verification/supervision.md
#	tests/fm-spawn-dispatch-profile.test.sh
The scenarios that pin single-flight, lease hand-off, and stale-lock
refusals deliberately leave run workers alive when their assertion is
made; those workers are orphaned to init and keep polling after the
suite exits, and on the Linux CI shard their concurrent sleep loops
reliably blew the sub-second deadlines of the timing-sensitive suite
scheduled next (observed: five orphaned workers made
tests/fm-pi-watch-extension.test.sh fail four shard runs in a row while
passing five-for-five in isolation). Reap every process still
referencing this run's unique TMP_ROOT at exit, collapsing the double
slash a trailing-slash TMPDIR (macOS) leaves in TMP_ROOT but never in
the workers' own argv.
@hnvivek hnvivek closed this Aug 17, 2026
@hnvivek
hnvivek deleted the sync/upstream-20260817 branch August 17, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants