fix: stabilize Herdr pane process identity checks

[cloud-practitioner]

Intent

Fix a regression from PR 41 (commit e2b9778): the herdr pane process identity check is flaky, so live panes read as missing and get duplicated.

fm_backend_herdr_pane_process_identity in bin/backends/herdr.sh records ps:<shell_pid>:<ps -o lstart>.
ps lstart is derived from boot time plus starttime/HZ and drifts by plus or minus one second across calls.
Observed example: a pane shell pid 1408408 recorded Wed Oct 7 02:03:32 2026, while the same live process later read 02:03:31; /proc/1408408/stat field 22 stayed stable at 43800329.
fm_backend_herdr_endpoint_foreign then treats the live owning pane as foreign, fm_backend_agent_state reports missing, and the secondmate liveness sweep relaunched several second mates into new panes beside their still-running, lock-holding agents (duplicates; one hit the relaunch bound).
Existing worker and secondmate metas in every home still carry the flaky ps:<pid>:<lstart> identity line.

Requested fix:

• Derive the identity from a stable source (for example /proc/<pid>/stat starttime ticks, or pid plus boot id plus starttime ticks).
• Keep compatibility with existing ps:<pid>:<lstart> records (accept a plus-or-minus one second lstart match, or re-derive).
• Make fm-control relaunch and fm-secondmate-restart refresh the recorded identity when the pane process changes.
• Add regression tests for the drift.
• Priority over other queued work; deliver a firstmate PR for the merge call.

What Changed

• Derive Herdr pane identities from /proc start ticks and boot ID, retaining the portable ps fallback when start ticks are unavailable.
• Accept plus-or-minus one second of start-time drift for legacy ps: records so live panes are not incorrectly reported as missing and relaunched.
• Add regression coverage for drift, process replacement, and relaunch identity refresh; document identity formats and compatibility guarantees.

Risk Assessment

✅ Low: The change is narrowly scoped, preserves legacy identity compatibility and ownership controls, and refreshes identities through the existing shared launch boundary without introducing substantiated material defects.

Testing

Five focused regressions and all live scenarios passed after resolving fixture setup issues; the old adapter reproduced the drift failure, while current ownership checks and recovery commands behaved correctly. Product logs and persisted metadata were retained, and disposable labs were removed. The secondmate persistence acknowledgment was supplied manually; this was runtime identity validation, not an autonomous LLM evaluation or a UI change.

• Live validation: ✅ go - 6 of 6 scenarios driven live against the product

Scenario	Result	Live	Evidence
Repeatedly inspect a live pane and retain one stable kernel process identity	✅ pass	live	live-identity.log: twelve identical proc identities and real Pi agent_state=alive.
Supervise legacy records with ±1-second timestamp drift without authorizing duplicate recovery	✅ pass	live	live-identity.log: legacy -1/0/+1-second records remained alive, including the real-ps fallback; secondmate probe returned alive with kill=0. The base adapter returned missing.
Attempt active input with foreign or unreadable ownership and leave the real agent untouched	✅ pass	live	live-identity.log: mismatched identities returned missing, unavailable readers returned unreadable, and every attempted Ctrl+C was refused.
Relaunch a worker through fm-control and upgrade its legacy identity	✅ pass	live	live-recovery.log and worker-before.meta / worker-after-legacy.meta.
Recover a worker after a real Herdr restart and record the replacement process	✅ pass	live	live-recovery.log and worker-after-process-change.meta.
Restart a secondmate through fm-secondmate-restart and refresh its recorded identity	✅ pass	live	live-recovery.log, secondmate-restart-output.log, and secondmate-before.meta / secondmate-after.meta.

Evidence: Validation report with persisted identity comparisons and test limitations

Source: Validation report with persisted identity comparisons and test limitations

# Live process-identity validation

Runtime: Linux, real Herdr 0.9.3 and real Pi 1.0.4. No canned Herdr or harness responses were used for the live scenarios. All Herdr traffic used `bin/fm-herdr-lab.sh` through named `fm-lab-*` sessions. Disposable marked homes, an unchanged archived product checkout, and linked task worktrees stayed inside the gate worktree. Lab teardown verified the default-session fleet tripwire.

## Drift and safety evidence

[`live-identity.log`](live-identity.log) shows repeated identical kernel identities, live Pi classification under legacy timestamps offset by -1/0/+1 seconds, and the pre-fix adapter reporting `missing` for the same real pane and +1-second record. The real secondmate sweep probe returned `alive` with `kill=0`. The no-/proc route used the actual `ps` utility and also accepted the offsets. Removing both identity readers returned `unreadable` and refused active Ctrl+C. Mismatched identities returned `missing`, refused Ctrl+C, and left the unrelated real agent intact.

The legacy offsets were deliberate persisted-record fixtures; this does not claim the host's clock happened to jitter naturally during the run. The no-/proc and missing-ps cases used disposable process-reader configuration/PATH isolation, not a BSD host or a fake ps executable.

## Public recovery evidence

[`live-recovery.log`](live-recovery.log) records the real `fm-control.sh` worker relaunch, named-session stop/reprovision, worker rebind with a different root identity, and real `fm-secondmate-restart.sh` completion. The secondmate restart was invoked from a real sanitized Pi primary using Pi's shell-command interface in the disposable unchanged product checkout. Its persist request was acknowledged manually through the actual `fm-secondmate-report.sh` parent-channel protocol for the empty fixture; this proves restart/identity publication, not autonomous LLM interpretation of /stow.

| Persisted state | Endpoint | Root-process identity |
| --- | --- | --- |
| Worker before | `fm-lab-recovery-2971997-3994:w1:p1` | `ps:2973522:Wed Oct  7 02:48:46 2026` |
| Worker after legacy refresh | `fm-lab-recovery-2971997-3994:w1:p1` | `proc:2973522:df641d85-7c77-431d-8f5f-c3a804278bd8:44071834` |
| Worker after actual process replacement/rebind | `fm-lab-recovery-2971997-3994:w1:p2` | `proc:2989582:df641d85-7c77-431d-8f5f-c3a804278bd8:44073352` |
| Secondmate before | `fm-lab-recovery-2971997-3994:w2:p1` | `ps:2998471:Wed Oct  7 02:49:10 2026` |
| Secondmate after public restart | `fm-lab-recovery-2971997-3994:w2:p1` | `proc:2998471:df641d85-7c77-431d-8f5f-c3a804278bd8:44074226` |

The generated `.meta` files are retained alongside this report. Each refreshed record has exactly one identity line; every published identity equaled a fresh read of its real pane root. Worker recovery preserved the linked worktree. CLI pane captures and raw Herdr responses are also retained.

## Targeted regression checks

Only these executable test functions from `tests/fm-backend-herdr.test.sh` were run, by loading the suite definitions without its full invocation list:

- `test_process_identity_is_portable_and_distinguishes_pid_reuse`
- `test_process_identity_is_stable_across_ps_lstart_drift`
- `test_lstart_drift_does_not_make_a_live_pane_foreign`
- `test_relaunch_refreshes_the_recorded_process_identity`
- `test_spawn_and_relaunch_continue_without_process_identity`

[`targeted-regressions.log`](targeted-regressions.log) is supplemental fixture-based evidence, not live product evidence. No full suite, lint, formatter, static-analysis, push, PR, or CI phase was run.

## Reproduction drivers

The executed temporary drivers are preserved as `live-identity-driver.sh` and `live-recovery-driver.sh`; run from the gate worktree. Before invoking the identity driver, create `.identity-validation/live` and populate its `base-herdr.sh` using `git show e2b97788aa9a750d0cb51a3e2cd4eded8a6cea00:bin/backends/herdr.sh`. Both drivers tear down their labs in the same run. Initial setup retries resolved duplicate prepare/provision tripwires, inherited FM_* overrides, the restart subprocess's gate environment, and read-only generated hook-directory cleanup. Final live runs succeeded.

No UI/layout/copy change is involved: persisted identities, CLI outputs, and process/agent API responses are the acceptance surface; no screenshots were needed.

Evidence: Live drift reproduction, fallback compatibility, and ownership refusals

Source: Live drift reproduction, fallback compatibility, and ownership refusals

~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/live/tmp/fm-lab.ZwvsYI
{"id":"cli:workspace:create","result":{"root_pane":{"agent_status":"unknown","cwd":"~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation","focused":true,"foreground_cwd":"~/.no-mistakes/worktrees/450411b3e67c","pane_id":"w1:p1","revision":0,"scroll":{"max_offset_from_bottom":0,"offset_from_bottom":0,"viewport_rows":40},"tab_id":"w1:t1","terminal_id":"term_65d372ebd7d2c1","workspace_id":"w1"},"tab":{"agent_status":"unknown","focused":true,"label":"1","number":1,"pane_count":1,"tab_id":"w1:t1","workspace_id":"w1"},"type":"workspace_created","workspace":{"active_tab_id":"w1:t1","agent_status":"unknown","focused":true,"label":"identity-lab","number":1,"pane_count":1,"tab_count":1,"workspace_id":"w1"}}}
{"id":"cli:pane:process_info","result":{"process_info":{"foreground_process_group_id":3079617,"foreground_processes":[{"argv":["/bin/zsh"],"cmdline":"/bin/zsh","cwd":"~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/live/tmp/fm-lab.ZwvsYI","name":"zsh","pid":3079617}],"pane_id":"w1:p1","shell_pid":3079617},"type":"pane_process_info"}}
Real pane root identity: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 01: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 02: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 03: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 04: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 05: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 06: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 07: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 08: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 09: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 10: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 11: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Read 12: proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:44084072
Starting the real Pi CLI in the real Herdr pane.
{"id":"cli:agent:get","result":{"agent":{"agent":"pi","agent_status":"idle","cwd":"~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/live/tmp/fm-lab.ZwvsYI","focused":true,"foreground_cwd":"~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/live/tmp/fm-lab.ZwvsYI","pane_id":"w1:p1","revision":0,"state_change_seq":1,"tab_id":"w1:t1","terminal_id":"term_65d372ebd7d2c1","workspace_id":"w1"},"type":"agent_info"}}
Real Pi process liveness: alive
Legacy -1 second record: ps:3079617:Wed Oct  7 02:50:47 2026 -> agent_state=alive
Legacy +0 second record: ps:3079617:Wed Oct  7 02:50:48 2026 -> agent_state=alive
Legacy +1 second record: ps:3079617:Wed Oct  7 02:50:49 2026 -> agent_state=alive
Pre-fix liveness against the SAME real pane and +1-second legacy record: missing
Secondmate sweep: status=alive state=alive kill=0
No-/proc fallback identity from real ps: ps:3079617:Wed Oct  7 02:50:48 2026
Real ps fallback, legacy -1 second record -> alive
Real ps fallback, legacy +0 second record -> alive
Real ps fallback, legacy +1 second record -> alive
Both identity readers unavailable (no proc root, ps absent from PATH): unreadable
Unreadable identity refused active Ctrl+C; duplicate recovery is not authorized.
Adversarial identity ps:3079617:Wed Oct  7 02:03:32 2020 -> missing
Foreign ownership refused active Ctrl+C.
Unrelated real agent retained: pi on fm-lab-identity-3079362-29979:w1:p1
Adversarial identity proc:3079617:df641d85-7c77-431d-8f5f-c3a804278bd8:0 -> missing
Foreign ownership refused active Ctrl+C.
Unrelated real agent retained: pi on fm-lab-identity-3079362-29979:w1:p1
Adversarial identity proc:2:bad-boot:0 -> missing
Foreign ownership refused active Ctrl+C.
Unrelated real agent retained: pi on fm-lab-identity-3079362-29979:w1:p1
Final real pane liveness: alive
Lab removed; default-session fleet tripwire unchanged.

Evidence: Public worker recovery and secondmate restart identity refresh

Source: Public worker recovery and secondmate restart identity refresh

~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/tmp/fm-lab.V6zxr2
scaffolded: ~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/tmp/fm-lab.V6zxr2/data/identity-worker/brief.md (ship, mode=local-only; replace {TASK} and {FIRSTMATE_SPEC})
Public worker recovery: fm-control.sh identity-worker relaunch (legacy record, agent-free real pane)
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/tmp/fm-lab.V6zxr2
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/tmp/fm-lab.V6zxr2
notice: identity-worker ships mode=local-only while the standing posture for project is no-mistakes - less rigor than the captain's standing posture; proceed only on a current explicit captain instruction or an intake judgment you can state
relaunched identity-worker harness=pi from=pi model=default effort=default backend=herdr endpoint=fm-lab-recovery-2971997-3994:w1:p1 worktree=~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/worker
Worker identity refresh: ps:2973522:Wed Oct  7 02:48:46 2026 -> proc:2973522:df641d85-7c77-431d-8f5f-c3a804278bd8:44071834; observed root=proc:2973522:df641d85-7c77-431d-8f5f-c3a804278bd8:44071834
{"session":{"default":false,"name":"fm-lab-recovery-2971997-3994","running":false,"session_dir":"~/.config/herdr/sessions/fm-lab-recovery-2971997-3994","socket_path":"~/.config/herdr/sessions/fm-lab-recovery-2971997-3994/herdr.sock"},"stopped":true}
After real lab restart: recorded=proc:2973522:df641d85-7c77-431d-8f5f-c3a804278bd8:44071834 current=proc:2986547:df641d85-7c77-431d-8f5f-c3a804278bd8:44073058
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/tmp/fm-lab.V6zxr2
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/tmp/fm-lab.V6zxr2
notice: identity-worker ships mode=local-only while the standing posture for project is no-mistakes - less rigor than the captain's standing posture; proceed only on a current explicit captain instruction or an intake judgment you can state
relaunched identity-worker harness=pi from=pi model=default effort=default backend=herdr endpoint=fm-lab-recovery-2971997-3994:w1:p2 worktree=~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/worker
Worker rebind: old_endpoint=fm-lab-recovery-2971997-3994:w1:p1 new_endpoint=fm-lab-recovery-2971997-3994:w1:p2 identity=proc:2989582:df641d85-7c77-431d-8f5f-c3a804278bd8:44073352 observed=proc:2989582:df641d85-7c77-431d-8f5f-c3a804278bd8:44073352
scaffolded: ~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/tmp/fm-lab.V6zxr2/data/identity-mate/brief.md (secondmate charter)
home=~/.no-mistakes/worktrees/450411b3e67c/01M4A3BG1QYX4E134JQ4C4GJSF/.identity-validation/recovery/mate
Public secondmate restart: real fm-secondmate-restart.sh; manually acknowledge the empty fixture through the public parent-report protocol.
Fixture acknowledgement: corr=66524289fb9d7ce5; no open records exist in this empty secondmate.
restarted: identity-mate (pi)
summary: 1 of 1 restarted, 0 nudged, 0 unreached
Secondmate restart identity refresh: ps:2998471:Wed Oct  7 02:49:10 2026 -> proc:2998471:df641d85-7c77-431d-8f5f-c3a804278bd8:44074226; actual=proc:2998471:df641d85-7c77-431d-8f5f-c3a804278bd8:44074226
All public recovery identity postconditions verified against real pane roots.
Recovery lab removed; default-session fleet tripwire unchanged.

Evidence: Supplemental targeted regression results

Source: Supplemental targeted regression results

ok - portable ps identities work without /proc and distinguish process PIDs and start times
ok - the Herdr process identity derives from /proc start ticks and boot id, ignoring ps lstart drift but changing with the process
ok - a plus or minus one second ps lstart drift (legacy record, with or without /proc) no longer reads a live pane foreign or missing, while a different process still does
ok - relaunch (the fm-control and fm-secondmate-restart launch path) replaces a legacy or stale recorded identity with the current process's
ok - Herdr spawn and relaunch deliver their agent launch with legacy ownership fallback when ps fails

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅

• ⚠️ tests/fm-backend-herdr.test.sh:5676 - The legacy-record setup uses GNU-style sed -i, which fails with stock macOS/BSD sed because it requires a backup-suffix argument. That failure is unchecked, leaving the original proc: record intact; the assertion at tests/fm-backend-herdr.test.sh:5679 can then pass without exercising legacy-identity refresh. Use a portable, checked rewrite and verify that the persisted identity is the intended ps: value before relaunching.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• Live validation: ✅ go - 6 of 6 scenarios driven live against the product

Scenario	Result	Live	Evidence
Repeatedly inspect a live pane and retain one stable kernel process identity	✅ pass	live	live-identity.log: twelve identical proc identities and real Pi agent_state=alive.
Supervise legacy records with ±1-second timestamp drift without authorizing duplicate recovery	✅ pass	live	live-identity.log: legacy -1/0/+1-second records remained alive, including the real-ps fallback; secondmate probe returned alive with kill=0. The base adapter returned missing.
Attempt active input with foreign or unreadable ownership and leave the real agent untouched	✅ pass	live	live-identity.log: mismatched identities returned missing, unavailable readers returned unreadable, and every attempted Ctrl+C was refused.
Relaunch a worker through fm-control and upgrade its legacy identity	✅ pass	live	live-recovery.log and worker-before.meta / worker-after-legacy.meta.
Recover a worker after a real Herdr restart and record the replacement process	✅ pass	live	live-recovery.log and worker-after-process-change.meta.
Restart a secondmate through fm-secondmate-restart and refresh its recorded identity	✅ pass	live	live-recovery.log, secondmate-restart-output.log, and secondmate-before.meta / secondmate-after.meta.

• TMPDIR="$PWD/.identity-validation/tmp" bash tests/.identity-targeted.tmp.sh: executed the five targeted identity, drift, refresh, and unavailable-identity regression functions without running the full suite.
• bash .identity-validation/live-identity.sh: exercised real Herdr/Pi process ownership, legacy timestamp offsets, real-ps fallback, unreadable identities, and refused foreign input.
• Loaded the base-commit Herdr adapter against the same running pane and legacy +1-second record; reproduced its incorrect missing verdict.
• bash .identity-validation/live-recovery.sh: exercised public recovery commands against disposable homes and an unchanged archived product checkout.
• $CODE/bin/fm-control.sh identity-worker relaunch --note 'Disposable fixture; no open work. Remain idle; do not invoke any pipeline.'
• Stopped and reprovisioned the named lab using bin/fm-herdr-lab.sh, then repeated worker relaunch and verified a new endpoint and current process identity.
• $CODE/bin/fm-secondmate-restart.sh identity-mate, invoked through a real sanitized Pi primary; supplied the empty fixture's persistence acknowledgment through fm-secondmate-report.sh.
• Verified published worker and secondmate metadata against fresh real-pane process identities.
• Tore down every lab, removed transient worktree fixtures, and confirmed only the default Herdr session remained.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.