fix(watcher): identify processes by kernel start ticks, not drifting lstart

[kirangathani]

Intent

Make firstmate's watcher process-identity check immune to WSL2 wall-clock drift, so bin/fm-turnend-guard.sh stops falsely reporting a healthy live watcher as dead.

The bug: fm_pid_identity in bin/fm-wake-lib.sh identified a process with 'ps -p -o lstart= -o command='. lstart is not stored by the kernel; ps computes it as boot time plus the process's start ticks. WSL2 continually re-syncs its boot-time estimate against the Windows host clock, so the SAME live process yields a DIFFERENT lstart string on successive reads. fm_watcher_lock_matches_pid then rejected the live watcher and the turn-end guard printed 'TURN WOULD END BLIND' on every turn while supervision was actually running, making a genuine supervision failure indistinguishable from constant noise. Verified empirically on this machine against a real bin/fm-watch.sh sampled 40 times over 60s: the old form produced 3 distinct identities for one never-restarted process, the new form produced 1, and fm_watcher_lock_matches_pid now reports LIVE WATCHER RECOGNISED.

Deliberate decisions a reviewer reading only the diff would not know:

1. Identity is keyed on field 22 (starttime, clock ticks since boot) of /proc//stat. Ticks since boot cannot drift however the wall clock moves. The parse deliberately does NOT use a positional awk '$22': field 2 (comm) is parenthesized and may itself contain spaces or parentheses, which shifts every positional field. It splits on the LAST ')' and takes the 20th whitespace-separated token of the remainder. This was an explicit requirement and is covered by tests with a comm containing a space and one containing a ')', both synthetic stat lines and a real process.

2. macOS has no /proc and firstmate supports macOS, so the existing 'ps -o lstart=' path is preserved as the fallback, along with its LC_ALL=C pinning and the locale-invariance reason recorded in the original comment. A test simulates an unreadable /proc//stat by overriding the reader.

3. Back-compat was a required design decision with two acceptable options. I chose exact dual-format acceptance in a new fm_pid_identity_matches: if the recorded string is not in the new format, the legacy lstart form is re-derived and compared exactly. Rationale: both branches remain EXACT comparisons, so the check is never more permissive than the one it replaces (dead pids, recycled pids and changed commands are all still rejected), while a pre-upgrade lock on a non-drifting host keeps matching perfectly. On WSL2 a legacy record may still fail to re-derive, but that host was already broken before this change, so it is not a regression there, and the record self-heals at the owner's next restart. I deliberately did NOT have readers rewrite another process's lock file, since that would be a write from a non-owner into someone else's lock.

4. The three call sites that compare a PERSISTED identity now route through fm_pid_identity_matches rather than bare string equality (fm_watcher_lock_matches_pid in the lib, fm_afk_launch_lock_owned in bin/fm-afk-launch.sh, daemon_pid_matches in bin/fm-afk-start.sh). This is centralisation in the primitive's owner, not scattering per-caller logic. The site in fm-afk-launch.sh around line 578 that compares two identities both freshly computed in the same run was deliberately left as plain equality: there is no persisted record there, so no format-skew is possible.

5. The command half of the identity is read from /proc//cmdline rather than forking ps. This is deliberate, not incidental. bin/fm-afk-launch.sh:605 computes an identity inside its lock-acquire window, BEFORE its cleanup trap is installed at line 606 - a pre-existing race that is out of scope for this task. An added ps fork there widened that window enough to make tests/fm-afk-launch.test.sh fail 2 runs in 5 (baseline main: 0 in 3). Fork-free it is 0 in 8, and cheaper than the single ps fork it replaces. The underlying trap-ordering gap is left untouched on purpose: the task scope explicitly said not to change the lock protocol or anything else about supervision.

6. Two errexit hazards were closed deliberately: fm_pid_identity_matches spells its early return as a full 'if' rather than '[ ... ] && return 0', and fm_pid_start_ticks appends '|| true', because bin/fm-afk-start.sh runs under 'set -e'. The stderr redirect in the /proc reads is placed BEFORE the input redirect so a dead pid's missing stat file fails silently (redirections apply left to right).

Scope boundaries honoured: the lock protocol, the beacon mechanism and everything else about supervision are unchanged. Upstream firstmate has its own fix for this (PR 752 / issue 433) but this fork is 24 commits behind and that sync is a separate stalled task, so this was implemented directly rather than ported; any later conflict is the sync task's problem.

AGENTS.md was deliberately NOT edited: per the binding firstmate-coding-guidelines decision tree this is mechanics (script header comment) plus an incident record (docs/), and adding it to AGENTS.md would violate the size-discipline rule. The incident evidence was recorded in docs/turnend-guard.md with date, exact commands and exact output, as that document owns the consuming contract.

Test status: bin/fm-lint.sh clean. Full suite green except two failures I verified on a clean checkout of main before attributing them - tests/fm-session-start.test.sh ('MISSING diagnostic did not appear at all') fails 3/3 on baseline main, and tests/fm-watch-checkpoint.test.sh is flaky at 1/3 on baseline main. Neither is caused by this change.

What Changed

• fm_pid_identity in bin/fm-wake-lib.sh now derives a process identity from field 22 of /proc/<pid>/stat (start time in clock ticks since boot) plus /proc/<pid>/cmdline, instead of ps -o lstart= -o command=. lstart is computed as boot time plus start ticks, and WSL2 re-syncs its boot-time estimate against the Windows host clock, so the same live watcher yielded different identity strings seconds apart and bin/fm-turnend-guard.sh reported a healthy watcher as dead. The stat parse splits on the last ) rather than using a positional field, so a comm containing spaces or parentheses cannot shift the read; ps -o lstart= is preserved (with its LC_ALL=C pin) as the fallback for hosts without /proc, such as macOS.
• Added fm_pid_identity_matches, which compares a live pid against a persisted identity and, for a record still in the legacy lstart format, re-derives and compares that form exactly. Both branches stay exact comparisons, so dead, recycled, and changed-command pids are still rejected. The three call sites that read a persisted identity route through it: fm_watcher_lock_matches_pid, fm_afk_launch_lock_owned in bin/fm-afk-launch.sh, and daemon_pid_matches in bin/fm-afk-start.sh.
• tests/fm-watcher-lock.test.sh gains coverage for identity stability across repeated reads of one live pid, the odd-comm stat parse (synthetic lines and a real process named a b)c), the lstart fallback when /proc/<pid>/stat is unreadable, legacy-record matching, and rejection of dead/recycled/start-marker-mismatched pids; the existing locale-invariance test now stubs the /proc reader so it actually exercises the LC_ALL=C pin on Linux. docs/turnend-guard.md records the incident with the measured 40-sample evidence and points its Tests section at the new regression.

Risk Assessment

✅ Low: The follow-up commit is a test-only change that correctly closes the coverage regression flagged in round 1 (forcing the lstart fallback so the LC_ALL=C pin is actually exercised, plus a guard that fails loudly if the stub stops working), and the underlying identity change remains well-bounded, centralised in its owning library, and conformant with every stated acceptance criterion.

Testing

Completed 1 recorded test check.

• Outcome: ⚠️ 1 error across 1 run (13m59s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed ✅

• ⚠️ tests/fm-watcher-lock.test.sh:683 - test_pid_identity_is_locale_invariant no longer exercises the LC_ALL=C pin on Linux: fm_pid_identity now takes the /proc path there and never calls ps, so all three sampled identities are the locale-independent 'proc-starttime:...' form and the assertions hold trivially. Dropping the pin from fm_pid_identity_lstart would regress the ko_KR watcher-rejection bug undetected on Linux CI. The test comment is also stale ('the fix pins LC_ALL=C inside fm_pid_identity' - the pin is now in fm_pid_identity_lstart). Fix by forcing the fallback the way test_pid_identity_falls_back_without_proc_stat does (stub fm_pid_start_ticks to return 1), or by sampling fm_pid_identity_lstart directly.
• ℹ️ bin/fm-wake-lib.sh:115 - The new identity has no boot-unique component, so a persisted record is only meaningful within the boot that wrote it; lstart encoded an absolute wall-clock instant and did not have this property. Intra-boot it is airtight (a recycled pid necessarily has larger start ticks). Across a reboot, pid + ticks + cmdline can in principle repeat. Heavily gated in practice: a colliding process must have an identical cmdline, so it would be a legitimate same-role process that has already rewritten its own lock, and the watcher path also requires a fresh beacon. fm_afk_launch_lock_owned and daemon_pid_matches have no second gate. Noted as a property of the new format; adding /proc/sys/kernel/random/boot_id would close it at the cost of another format change.

🔧 Fix: force lstart fallback in locale-invariance identity test
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 error

• 🚨 tests failed with exit code 1
• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅

• ⚠️ linter found issues (exit code 1)

🔧 Fix: silence SC2016 on intentional inner-shell script body
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.