fix(bin): prevent stale crew liveness reports

[SimTet]

Intent

Stop firstmate from reporting stale liveness for a crew whose harness process has died but whose pane or shell still answers, so a dead worker surfaces as unknown instead of reading 'working' forever (upstream issue #3402: the session-start fleet digest printed 'endpoint: alive' for tmux windows that did not exist).

In bin/fm-crew-state.sh's no-run fallback, consult the recovery-grade fm_backend_agent_state classifier (owned by bin/fm-backend.sh) ahead of both the busy-hook read and the status-log fallback. Only a CONFIDENT dead/missing verdict changes behavior, reporting 'unknown / agent-state'; unverified, unreadable, and ambiguous verdicts must fall through byte-identical to the previous behavior, because a false 'crew is dead' would trigger recovery against a live worker, which is worse than the silent-working gap being closed. Reuse the existing single-owner classifier rather than forking a second one.

A TERMINAL done/failed status-log record is exempt from that dead verdict and is still reported as that terminal state, because a recorded outcome is not a liveness claim a dead harness can withdraw. Without the exemption, a finished single-owner task whose harness exited while its pane survived would lose its completion signal and have its already-answered keyed decision re-opened by bin/fm-fleet-snapshot.sh's lifecycle reconciliation. Only non-terminal readings (working/needs-decision/blocked/paused) are the stale-liveness case this check exists to suppress.

Also populate endpoint.agent_alive for every task with a recorded target in the fleet snapshot, and stop the herdr push path from waking firstmate for retired panes that no task currently records, committing that transition so the dedupe marker still advances and then absorbing it rather than misrouting it to the wrong crew.

Delivery constraints: this finishes the EXISTING open PR #2501 from its preserved, custody-returned branch fm/fm-liveness. Preserve every commit already on this branch, including the earlier accepted review corrections and the two recovered review fixes (the terminal-record exemption and its crew-state contract-header documentation); never reset, rebase away, replace, squash, or drop any preserved commit. Rebase onto exactly upstream kunchenguid/firstmate main 4ad8cba, which the primary local default branch and the SimTet fork default branch have both already been fast-forwarded to. Two files overlap that upstream range and need conflict resolution that keeps BOTH sides' intent rather than dropping preserved fixes: docs/architecture.md and tests/fm-session-start.test.sh. Update ONLY the existing PR 2501 and its existing fork branch. Never merge the PR, never force-push, never push to the kunchenguid upstream, never create a duplicate PR, and never discard preserved work.

Repo conventions that apply: bin/*.sh must pass bin/fm-lint.sh (pinned shellcheck), tracked Markdown uses one full sentence per line with plain dashes and no em dash, tests are colocated in tests/ as .test.sh and must exercise behavior through an executable interface rather than asserting implementation source bytes, and no agent name may be added as a commit co-author. Note that tests/fm-session-start.test.sh has one pre-existing environment-dependent failure ('MISSING diagnostic did not appear at all') that also fails on the base commit and is not caused by this branch.

What Changed

• Report confidently dead or missing harnesses as unknown during no-run crew-state fallback while preserving terminal outcomes and inconclusive behavior.
• Verify labeled tmux targets against exact window membership and expose agent liveness for every targeted task in fleet snapshots and views.
• Commit and absorb Herdr transitions from unrecorded panes so retired endpoints do not wake or misroute to active crew.

Risk Assessment

✅ Low: The behavior change is narrowly gated - only a confident dead/missing verdict acts, every other verdict falls through untouched, and the tmux primitive's stricter path is opt-in via an expected label that matches the window convention fm-spawn.sh enforces - with direct regression coverage including byte-identical fallthrough proofs, and the only findings are informational.

Testing

Baseline checks reproduced the stale working verdict and previously isolated the session-start fixture mismatch; all focused crew-state, fleet, Bearings, supervision-event, and node-free session-start tests now pass, and the CLI transcript confirms dead workers become unknown, terminal completion survives, endpoint liveness is populated, and retired Herdr panes are absorbed without waking firstmate while advancing dedupe.

Evidence: End-to-end firstmate liveness CLI transcript

Source: End-to-end firstmate liveness CLI transcript

FIRSTMATE LIVENESS END-TO-END TRANSCRIPT

$ fm-crew-state.sh dead-shell
state: unknown · source: agent-state · harness process confirmed dead: firstmate:fm-dead-shell

$ fm-crew-state.sh finished
state: done · source: status-log · report at data/finished/report.md

$ fm-fleet-view.sh (Under Way section)
## Under Way
| ID | Current | Kind | Repo/Project | Backend | Endpoint | Artifact | Path | Watch / return channel |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| alive | working / status-log | secondmate | - | tmux | present / alive | - | /home/simon/.no-mistakes/worktrees/a63d177202a2/01M1CKSZGB5NA68JE346CBB1ZN/.liveness-evidence-fixture/wt-alive | bin/fm-send.sh fm-alive '<request>' - read status/doc return channel; do not routinely fm-peek a secondmate for answers |
| dead-shell | unknown / agent-state | scout | - | tmux | present / dead | - | /home/simon/.no-mistakes/worktrees/a63d177202a2/01M1CKSZGB5NA68JE346CBB1ZN/.liveness-evidence-fixture/wt-dead-shell | bin/fm-peek.sh fm-dead-shell |
| finished | done / status-log | scout | - | tmux | present / dead | - | /home/simon/.no-mistakes/worktrees/a63d177202a2/01M1CKSZGB5NA68JE346CBB1ZN/.liveness-evidence-fixture/wt-finished | bin/fm-peek.sh fm-finished |
| missing-worker | unknown / agent-state | scout | - | tmux | absent / dead | - | /home/simon/.no-mistakes/worktrees/a63d177202a2/01M1CKSZGB5NA68JE346CBB1ZN/.liveness-evidence-fixture/wt-missing-worker | bin/fm-peek.sh fm-missing-worker |


$ fm-fleet-snapshot.sh --json | jq endpoint summary
[{"id":"alive","exists":true,"agent_alive":"alive"},{"id":"dead-shell","exists":true,"agent_alive":"dead"},{"id":"finished","exists":true,"agent_alive":"dead"},{"id":"missing-worker","exists":false,"agent_alive":"dead"}]

$ retired Herdr push transition (no task metadata records default:wZ:pR)
wake_queue=absent
wake_calls=absent
dedupe_marker=committed
triage=[2026-08-31T22:20:18+0200] absorbed push blocked (no recorded task for this pane - retired or unknown endpoint): default:wZ:pR

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (58m55s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 3 infos

• ℹ️ bin/fm-backend.sh:831 - fm_backend_target_exists' contract header still describes the tmux arm as a display-message probe that "Mirrors fm-crew-state.sh's pane_readable check". That is now only true for the label-less form: with an expected label the tmux arm reads the session inventory (list-windows -F '#{window_name}') and, when the target's window part is not exactly the label, returns absent without querying tmux at all. Since this header is the shared primitive's contract for callers that pass a label (fm-session-start.sh:825, fm-fleet-snapshot.sh:556, fm-control.sh:418) and for those that deliberately do not (fm-supervise-daemon.sh, fm-send.sh), document the label-gated rule and the tmux active-window fallback it exists to defeat.
• ℹ️ bin/fm-fleet-snapshot.sh:561 - Every task with a target now pays two endpoint probes per snapshot: fm_backend_target_exists plus fm_backend_agent_alive. For tmux that is two list-windows inventory reads; for herdr it is three CLI round-trips (pane get, then pane get + agent get) instead of one. fm_backend_agent_state's missing verdict already subsumes presence, but collapsing the two is not behavior-preserving (agent_state's unreadable would have to map to endpoint_exists=null, not false), so this is a cost note rather than a required change. Note also that, unlike the remote-secondmate arm above, this call is not wrapped in fm_run_timed - though fm_backend_target_exists already carried the same unbounded exposure for every task.
• ℹ️ bin/fm-push-transition-lib.sh:158 - The retired-pane absorb commits the herdr dedupe marker, so a blocked edge that arrives in the window between endpoint creation (fm-spawn.sh:2248) and the state/<id>.meta write (fm-spawn.sh:2856) is absorbed and never re-delivered on the push path; the marker clears only on a later working edge. The poll loop is unaffected (it uses .stale-<key>, not .herdr-escalated-<key>) and remains the documented permanent backstop, and committing the transition is exactly what the stated intent requires, so this is recorded as a tradeoff rather than a defect.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 tests/fm-session-start.test.sh:1320 - tests/fm-session-start.test.sh::test_endpoint_liveness_tmux failed on the branch (base 4ad8cba passes it). The fixture recorded window=fm-sess:live-window for task id task-live, but the branch's new label form of fm_backend_target_exists requires the tmux window name to be the expected fm-<id> label, so the live endpoint was reported dead. It was masked locally because this machine has /usr/bin/node, so the file aborts earlier at the pre-existing environment-dependent MISSING: node assertion; run with the suite's own FM_TEST_BASE_PATH knob pointed at a node-free PATH (the shape CI has), base reaches the test and passes 49/49 while the branch failed. Fixed by giving the fixture production naming (fm-sess:fm-task-live / fm-sess:fm-task-dead), matching bin/fm-spawn.sh's W="fm-$ID" and fm_backend_validate_task_endpoint's stated tmux record rule; assertions otherwise unchanged. The file now passes 49/49 on the branch, and the same fixed fixture also passes 49/49 against base code.
• bin/fm-test-run.sh tests/fm-crew-state.test.sh tests/fm-supervision-events.test.sh tests/fm-fleet-snapshot-view.test.sh (all pass, including the seven new agent-state cases and the terminal-record exemption)
• bin/fm-test-run.sh tests/fm-backend.test.sh tests/fm-tmux-agent-liveness.test.sh tests/fm-bearings-snapshot.test.sh tests/fm-secondmate-liveness.test.sh tests/fm-session-start.test.sh
• bin/fm-test-run.sh tests/fm-busy-state.test.sh tests/fm-backend-orca.test.sh tests/fm-teardown-endpoint-safety.test.sh tests/fm-daemon.test.sh (the other fm_backend_target_exists callers)
• FM_TEST_BASE_PATH=<node-free mirror of /usr/bin:/bin:/usr/sbin:/sbin> bash tests/fm-session-start.test.sh on base 4ad8cba (49 ok), on branch 379b683 (failed at test_endpoint_liveness_tmux), on branch after the fixture fix (49 ok), and base code with the fixed fixture (49 ok)
• Confirmed the MISSING diagnostic did not appear at all failure is pre-existing by running tests/fm-session-start.test.sh from a pristine git archive 4ad8cba checkout (same failure, same line)
• Manual end-to-end: real tmux 3.2a on an isolated socket with windows fm-live-crew (real verified-harness process), fm-dead-crew and fm-done-crew (husk shell only), plus ghost-crew whose recorded window does not exist; ran bin/fm-crew-state.sh <id> for all four on base and branch
• Manual end-to-end: bin/fm-fleet-view.sh against the same FM_HOME on base and branch (Under Way table, Current and Endpoint columns)
• Manual end-to-end: full bin/fm-session-start.sh run on base and branch against the same FM_HOME and live tmux server, comparing the FLEET STATE endpoint: lines (issue #3402 reproduction)
• Manual: drove handle_push_transition herdr default <blocked record> for a pane with no recorded task on base and branch, plus a live-recorded-task control, capturing the wake queue, watcher triage log and herdr dedupe marker
• Probed fm_backend_target_exists tmux fm-sess:live-window fm-task-live on base vs branch to isolate the label-strictness behaviour change

🔧 Fix: use production fm-<id> window names in session-start tmux fixture
✅ Re-checked - no issues remain.

• Against base 4ad8cba, extracted the tree with git archive and ran only test_fast_path_confirmed_dead_agent_overrides_frozen_busy_hook; it reproduced the stale state: working · source: pane regression.
• bash tests/fm-crew-state.test.sh
• bash tests/fm-fleet-snapshot-view.test.sh
• bash tests/fm-supervision-events.test.sh
• bash tests/fm-bearings-snapshot.test.sh
• FM_TEST_BASE_PATH="$PWD/.no-mistakes-test-bin" bash tests/fm-session-start.test.sh using a node-free PATH to avoid the documented environment-dependent failure.
• Manually invoked fm-crew-state.sh, fm-fleet-view.sh, and fm-fleet-snapshot.sh --json, then exercised a retired Herdr push transition and recorded its wake, dedupe, and triage state.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the eligible follow-up review scope.

No blocking failure remains.

_{Reviews (2): Last reviewed commit: "no-mistakes(document): Document liveness..." | Re-trigger Greptile}

[SimTet]

Hi maintainers, this PR has been refreshed onto the current upstream main. GitHub is holding its workflow runs for maintainer approval. When convenient, could you please approve them so CI can run?\n\n- CI: https://github.com/kunchenguid/firstmate/actions/runs/32524342928\n- Required validation: https://github.com/kunchenguid/firstmate/actions/runs/32524342946

[kunchenguid]

Speaking as Kun's firstmate:

VISION verdict: align. A confirmed-dead harness must not keep reporting working from a frozen busy hook or a stale status line. That is honest interface under load, and it reuses the existing recovery-grade classifier instead of mixing a new heuristic into scripts.

Class: corrective.

Reviewed the full diff vs base. fm-crew-state.sh consults fm_backend_agent_state on the no-run path only; dead/missing emit unknown · agent-state; unverified/unreadable/ambiguous fall through byte-identical. fm-fleet-snapshot.sh now fills endpoint.agent_alive for every recorded target. handle_push_transition commits then absorbs a pane with no recorded task, and still wakes once that pane is live-recorded. Tests cover both directions and the inconclusive no-ops.

Security: no.
Fork CI: approved on this HEAD (CI + Require no-mistakes). Checks are not yet green.
Ahead 6 / behind 1. mergeable MERGEABLE, mergeStateStatus UNSTABLE.
Overlap: bin/fm-crew-state.sh also in #2757 (comment-only there). docs/herdr-backend.md also in the held pair #2637/#2692 (this PR is one sentence on the retired-pane exemption). Not spawn-freshen.

Waiting on CI including no-mistakes, not on the captain. Not merge-eligible yet.