test(backend): add opt-in Herdr real-lab E2E coverage

[JTInventory]

Intent

Add opt-in real Herdr lab e2e: workspace-per-home, respawn-idem, prune-safety, secondmate marker. Never touch default session. Hermetic suite green without live e2e.

What Changed

• Added four opt-in real Herdr lab E2E suites covering per-home workspace isolation, respawn-idempotent husk replacement, default-tab prune safety, and secondmate marker delivery.
• Scoped destructive lab cleanup to private fm-lab-* sessions and added assertions that the live default session remains untouched.
• Documented FM_HERDR_E2E=1 coverage and the standalone marker-test opt-in; the no-mistakes pipeline completed successfully after review and lint auto-fixes.

Risk Assessment

✅ Low: The follow-up hardens cleanup status propagation, explicit lab-session version checks, duplicate-submit detection, and exact workspace inventories while preserving opt-in execution and default-session isolation.

Testing

The hermetic suite passed, and the normal-clone live run demonstrated workspace-per-home isolation, respawn idempotency, prune safety, and exactly-once marked/unmarked secondmate messaging. Evidence is stored under /tmp/no-mistakes-evidence/01KXXH5VFD15RHKF9KS74JM8CE/.

Evidence: Acceptance summary

16:skip: set FM_HERDR_SMOKE=1 to opt into the real Herdr AFK e2e
55:skip: set FM_HERDR_SMOKE=1 to opt into the real Herdr smoke
58:ok - real herdr E2E: a primary-shaped home spawns a crewmate on the herdr backend
59:ok - real herdr E2E: the primary-shaped home's crewmate landed in the 'firstmate' workspace
60:ok - real herdr E2E: the primary spawns a --secondmate task on the herdr backend
61:ok - real herdr E2E: a --secondmate spawn by the PRIMARY lands in the SECONDMATE's own labeled workspace, distinct from the primary's
62:ok - real herdr E2E: a crewmate spawns successfully FROM a secondmate-shaped home's own fm-spawn.sh process
63:ok - real herdr E2E: a crewmate spawned FROM the secondmate-shaped home lands in the secondmate's OWN workspace - falls out of per-home resolution, no glue needed
64:ok - real herdr E2E: the primary workspace inventory contains only the primary home's task
65:ok - real herdr E2E: the secondmate workspace inventory contains its own task and its spawned crewmate only
66:ok - real herdr E2E: tearing down cm1 closes only its own tab - the secondmate's and cm2's tabs survive untouched
67:ok - real herdr E2E: tearing down cm2 closes only its own tab - the secondmate's own tab (same workspace) survives untouched
554:ok - real Herdr lab: secondmate fm-send delivers exactly one from-firstmate marker
555:ok - real Herdr lab: direct captain input remains unmarked
922:All 68 behavior tests passed

Evidence: Full live Herdr suite transcript

ok: PR target repo jtinventory/firstmate verified
tmux 3.4
ok - gate refusal helper covers marker, empty marker, path backstop, and normal session
ok - spawn, send, and teardown refuse both gate signals before lifecycle work
ok - tracked gate-refusal wiring and trusted no-mistakes config are present
# all fm-gate-refuse tests passed
Running 68 behavior tests with 1 parallel job(s)
START: tests/fm-afk-inject-e2e.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-afk-inject-e2e/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-afk-inject-e2e/gotmp)
PASS: tests/fm-afk-inject-e2e.test.sh
ok - Scenario A: partial input defers injection; digest arrives clean after idle
ok - Scenario B: swallowed Enter produces exactly one clean digest
ok - Scenario C: a normal captain status injects exactly one clean single-line sentinel digest
all e2e injection tests passed
START: tests/fm-afk-inject-herdr-e2e.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-afk-inject-herdr-e2e/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-afk-inject-herdr-e2e/gotmp)
PASS: tests/fm-afk-inject-herdr-e2e.test.sh
skip: set FM_HERDR_SMOKE=1 to opt into the real Herdr AFK e2e
START: tests/fm-afk-launch.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-afk-launch/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-afk-launch/gotmp)
PASS: tests/fm-afk-launch.test.sh
ok - AFK daemon survives harness process-group reap and stops on return
ok - AFK launch fails closed when the durable away flag cannot be created
ok - AFK launch is idempotent and return clears the away flag
ok - AFK launch forwards the resolved Herdr supervisor target and backend
ok - AFK refresh fails closed when the durable away flag cannot be written
ok - AFK return retains the away flag when identity verification fails
ok - AFK return retains the away flag when a live daemon record is missing
ok - AFK return retains the away flag when a stale daemon record remains live
ok - AFK status keeps a live unverified daemon visible
ok - AFK return fails closed when the durable away flag cannot be removed
ok - AFK transition lock rejects an invalid state path promptly
ok - AFK transition lock distinguishes I/O failure from contention
ok - AFK transition lock handoff preserves contention return code
ok - AFK transition lock bounds contention retries
ok - AFK transition lock preserves a replacement daemon and away flag
ok - AFK return clears the away flag after confirmed daemon absence
ok - AFK return retains the away flag when TERM does not stop the daemon
all fm-afk-launch tests passed
START: tests/fm-backend-herdr-prune-safety-e2e.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-prune-safety-e2e/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-prune-safety-e2e/gotmp)
PASS: tests/fm-backend-herdr-prune-safety-e2e.test.sh
ok - repro setup: a pre-existing workspace labeled 'firstmate' collides with the primary home's own label
ok - repro setup: a live long-running process is running in the startup workspace's single tab (label '1'), heartbeating to a marker file
ok - fixed: container_ensure adopts the label-colliding startup workspace and reports NO seeded default tab (never a prune candidate)
ok - fixed: the live pane (and its live process) survived create_task untouched - the exact 2026-07-02 self-kill incident does not reproduce
ok - fixed: the startup workspace's original live tab is still present in tab list after the spawn
ok - happy path: a genuinely fresh workspace's seeded default tab is still pruned, leaving exactly one clean fm-<id> task tab
START: tests/fm-backend-herdr-respawn-idem-e2e.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-respawn-idem-e2e/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-respawn-idem-e2e/gotmp)
PASS: tests/fm-backend-herdr-respawn-idem-e2e.test.sh
ok - repro setup: two real fm-<id> task tabs exist (crewmate-shaped and secondmate-shaped), neither with a registered agent
ok - repro confirmed: after a real session restart, both task panes survive alive but with no registered agent - the restored-layout husk
ok - fixed: create_task closes and replaces the crewmate-shaped restored husk instead of refusing - no manual pane close needed
ok - fixed: create_task closes and replaces the secondmate-shaped restored husk instead of refusing - same fix, same function, both spawn shapes
ok - fixed: the workspace holds exactly the 2 replacement tabs after both respawns - no leaked husk tabs, no destroyed workspace
ok - fixed: a genuinely live duplicate (a real registered agent) still refuses exactly as before - the husk fix never closes a live pane
START: tests/fm-backend-herdr-smoke.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-smoke/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-smoke/gotmp)
PASS: tests/fm-backend-herdr-smoke.test.sh
skip: set FM_HERDR_SMOKE=1 to opt into the real Herdr smoke
START: tests/fm-backend-herdr-workspace-per-home-e2e.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-workspace-per-home-e2e/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr-workspace-per-home-e2e/gotmp)
PASS: tests/fm-backend-herdr-workspace-per-home-e2e.test.sh
ok - real herdr E2E: a primary-shaped home spawns a crewmate on the herdr backend
ok - real herdr E2E: the primary-shaped home's crewmate landed in the 'firstmate' workspace
ok - real herdr E2E: the primary spawns a --secondmate task on the herdr backend
ok - real herdr E2E: a --secondmate spawn by the PRIMARY lands in the SECONDMATE's own labeled workspace, distinct from the primary's
ok - real herdr E2E: a crewmate spawns successfully FROM a secondmate-shaped home's own fm-spawn.sh process
ok - real herdr E2E: a crewmate spawned FROM the secondmate-shaped home lands in the secondmate's OWN workspace - falls out of per-home resolution, no glue needed
ok - real herdr E2E: the primary workspace inventory contains only the primary home's task
ok - real herdr E2E: the secondmate workspace inventory contains its own task and its spawned crewmate only
ok - real herdr E2E: tearing down cm1 closes only its own tab - the secondmate's and cm2's tabs survive untouched
ok - real herdr E2E: tearing down cm2 closes only its own tab - the secondmate's own tab (same workspace) survives untouched
START: tests/fm-backend-herdr.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend-herdr/gotmp)
PASS: tests/fm-backend-herdr.test.sh
NOTICE: auto-detected herdr runtime (HERDR_ENV=1) - spawning into the EXPERIMENTAL herdr backend. Set config/backend or pass --backend tmux to opt out.
ok - Herdr selection honors explicit config/env and nested TMUX precedence
ok - bootstrap backend tool gating keeps Herdr dependencies opt-in
ok - Herdr version gate enforces 0.7.x and protocol 14+
ok - Herdr container ensure is version-gated and workspace-per-home
ok - Herdr workspace bind failures verify cleanup
ok - Herdr malformed workspace creates clean up new workspaces
ok - Herdr task creation, duplicate protection, target parsing, and key mapping work
ok - Herdr kill propagates close failures and avoids server creation
ok - Herdr capture uses safe over-fetch and native busy state
ok - Herdr text submit uses native confirmation and safe composer fallback
ok - Herdr wait_for_working samples native status and classifies blocked submits
ok - Herdr replaces a confirmed husk only after creating the replacement
ok - Herdr create cleanup closes tabs after post-create failures
ok - Herdr missing-id cleanup fails closed when unobserved
ok - Herdr pre-create failures preserve existing tabs
ok - Herdr seed pruning closes and verifies the exact seeded tab
ok - Herdr event capability probes use the explicit session
ok - Herdr eventwait returns and records a fresh blocked transition
ok - backend dispatch accepts Herdr and preserves opaque session:pane targets
START: tests/fm-backend.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-backend/gotmp)
PASS: tests/fm-backend.test.sh
ok - backend selection precedence, metadata default, and known/unknown validation
ok - fm-spawn refuses unknown backends fr

... [54207 bytes truncated] ...

l stale suppressors repair missing or corrupt stale-since timers
ok - triage log capping handles wc byte counts with leading spaces
ok - a heartbeat with no captain-relevant change is absorbed and backs off the cadence
ok - heartbeat backstop fail-safe surfaces a captain-relevant status the per-wake path missed
ok - the liveness beacon stays fresh while the watcher absorbs benign wakes (fm-guard never false-alarms)
ok - with .afk present the watcher reverts to one-shot so the daemon owns triage (no double-triage)
START: tests/fm-watcher-lock.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-watcher-lock/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-watcher-lock/gotmp)
PASS: tests/fm-watcher-lock.test.sh
ok - simultaneous watcher starts leave exactly one live process
ok - killed watcher stale lock is reclaimed
ok - live watcher lock with stale heartbeat is actionable
ok - guard banner leads when down with pending wakes (re-arm-after-drain) and stays silent when fresh+live
ok - guard requires a fresh beacon plus a live matching watcher lock
ok - concurrent fm_lock_try_acquire yields exactly one winner
ok - dead-pid stale lock is reclaimed by a single acquirer
ok - concurrent stale-lock steal yields exactly one winner
ok - live steal mutex is not reclaimed
ok - live-held lock is not stolen
ok - live-held lock with matching pid identity is not stolen
ok - live-held lock with mismatched pid identity is reclaimed
ok - live-held legacy identity remains protected during migration
ok - expired live-held legacy identity is reclaimed
ok - matching expired legacy watcher identity is migrated before expiry recovery
ok - live-held lock without pid identity remains live-held
ok - zombie follower lock is reclaimed using its stored process identity
ok - legacy zombie follower lock is reclaimed without new metadata
ok - fallback process identity includes stable process-group and command data
ok - fallback start identity accepts and distinguishes prior formats
ok - detach cleanup rejects legacy start tokens
ok - detached spawn waits for the target after exec
ok - detached spawn cleans the launcher after pid-file timeout
ok - detached spawn cleans the target after exec timeout
ok - arm reclaims a legacy follower lock whose pid was reused
ok - legacy follower locks without home scope fail closed
ok - watcher health rejects a zombie lock owner
ok - watcher health rejects an unpinned legacy lock
ok - Grok treats an existing follower as a live cycle
ok - empty mid-acquire lock keeps a minimum grace
ok - late original claimant cannot claim a recreated lock
ok - paused mid-acquire claimant backs off to active stealer
ok - watch restart refuses to signal a reused pid
ok - plain arm recovers from a reused-pid stale watcher lock
ok - watcher self-evicts when the lock pid no longer names it
ok - arm attaches to a live fresh watcher and exits only when that cycle ends
ok - arm migrates and attaches to a live legacy watcher lock
ok - arm rejects an unverified legacy watcher lock
ok - arm starts+confirms a fresh watcher on a clean lock and self-heals a dead-pid lock (never healthy off a dead pid)
ok - arm stands down on HUP while the detached watcher keeps its lock and beacon
Terminated
ok - watcher survives SIGTERM of the arm's entire process group
ok - a healthy cycle keeps one attach waiter and duplicate arms exit without stacking
ok - restart hands off the follower slot without stacking waiters
ok - process start identity distinguishes same-second processes
ok - arm propagates an immediate watcher wake before confirmation
ok - arm attaches to a peer watcher after child stands down and exits when peer dies
ok - arm reports FAILED and exits non-zero when no fresh watcher can be confirmed
START: tests/fm-x-mode.test.sh (TMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-x-mode/tmp GOTMPDIR=/tmp/fm-behavior-tests.EkGMR2/fm-x-mode/gotmp)
PASS: tests/fm-x-mode.test.sh
ok - fm-x-poll is a hard no-op without a token (inert default)
ok - fm-x-poll treats an explicitly empty env token as configured
ok - fm-x-poll stays silent on HTTP 204 (the common case)
ok - fm-x-poll lets an explicitly empty relay env override .env
ok - fm-x-poll surfaces auth/config errors once and clears on recovery
ok - fm-x-poll stashes the question and prints the compact marker
ok - fm-x-poll preserves in_reply_to conversation context in the inbox
ok - fm-x-poll reports inbox commit failures without emitting a mention wake
ok - fm-x-poll requires a non-empty question before waking
ok - fm-x-poll rejects an unsafe request_id (path-traversal guard)
ok - fm-x-reply posts a request-bound answer and echoes only the request_id
ok - fm-x-reply accepts the reply via --text-file and stdin (safe, unexpanded)
ok - fm-x-reply exits non-zero on a non-2xx relay response
ok - fm-x-reply cleans up auth header temp files on interrupted posts
ok - fm-x-reply rejects missing arguments with a usage error
ok - fm-x-reply --help makes image support discoverable
ok - fm-x-reply rejects whitespace-only reply text
ok - fm-x-reply dry-run records the would-be reply and never posts
ok - fm-x-reply dry-run works without a token
ok - fm-x-reply honors FMX_DRY_RUN from .env
ok - fm-x-reply lets an explicitly empty dry-run env override .env
ok - fm-x-reply dry-run fails when it cannot record the preview
ok - fmx_split_thread: word-boundary, within-limit, numbered, lossless, capped
ok - fm-x-reply keeps a concise reply as a single unnumbered tweet
ok - fm-x-reply auto-splits a long reply into a numbered thread (texts[])
ok - fm-x-reply clamps a below-floor max to 50 characters
ok - fm-x-reply posts a thread payload (texts[]) to the relay
ok - fm-x-reply --image posts an image object on answer
ok - fm-x-reply streams large image payloads outside curl argv
ok - fm-x-reply dry-run records compact image metadata for threaded replies
ok - fm-x-reply cleans image and payload temp files
ok - fm-x-reply --image rejects missing and unsupported image paths clearly
ok - fm-x-reply --image rejects oversized files before encoding
ok - fm-x-reply --followup posts to /connector/followup with the same request-bound body
ok - fm-x-reply --followup --image posts an image object
ok - fm-x-reply --followup is accepted in any position and leaves the answer path default
ok - fm-x-reply --followup dry-run marks the endpoint without changing the answer path
ok - fm-x-reply --followup auto-splits a long follow-up into a marked thread
ok - fm-x-reply followup dry-run keeps endpoint marker and compact image metadata
ok - fm-x-dismiss posts a request-bound dismiss and echoes only the request_id
ok - fm-x-dismiss dry-run records the would-be body and never posts
ok - fm-x-dismiss dry-run works without a token
ok - fm-x-dismiss exits non-zero on a non-2xx relay response
ok - fm-x-dismiss exits non-zero on a transport failure
ok - fm-x-dismiss rejects an unsafe request_id (path-traversal guard)
ok - fm-x-dismiss rejects missing or extra arguments with a usage error
ok - fm-x-link records and refreshes the X-request link without disturbing meta
ok - meta rewrites are independent of TMPDIR
ok - fm-x-link rejects unsafe ids, missing meta, and missing arguments
ok - fm-x-followup --check reports postable / not-linked correctly
ok - fm-x-followup --check prunes a link past the 24h window
ok - fm-x-followup posts the follow-up and clears the link on success
ok - fm-x-followup --image forwards the attachment through fm-x-reply --followup
ok - fm-x-followup keeps the link when the post fails
ok - fm-x-followup skips silently and clears the link past the 24h window
ok - fm-x-followup is a no-op for a task with no X link
ok - fm-x-followup dry-run records the follow-up and clears the link
ok - fm-x-followup rejects malformed invocations
ok - bootstrap activates X mode from an .env token, idempotently
ok - bootstrap reports missing X-mode dependencies before arming
ok - bootstrap does not report X mode on when activation artifacts cannot be written
ok - bootstrap is inert without a non-empty .env token (non-X users unaffected)
ok - bootstrap cleans up X artifacts on opt-out and is silent once off
ok - bootstrap reports failed X artifact cleanup on opt-out
All 68 behavior tests passed

Evidence: Post-test Herdr session inventory

--- Herdr sessions after all lab tests ---
{"sessions":[{"default":true,"name":"default","running":true,"session_dir":"/root/.config/herdr","socket_path":"/root/.config/herdr/herdr.sock"},{"default":false,"name":"firstmate","running":false,"session_dir":"/root/.config/herdr/sessions/firstmate","socket_path":"/root/.config/herdr/sessions/firstmate/herdr.sock"}]}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 5 issues found → auto-fixed ✅

• 🚨 tests/fm-backend-herdr-prune-safety-e2e.test.sh:45 - The cleanup ignores a failed herdr_safe_stop_and_delete; the following rm -rf returns success, so a tripwire failure can leave the lab session behind while the test still passes. Propagate the teardown status and fail the test when cleanup safety verification fails. The same pattern exists in the respawn and workspace-per-home suites.
• 🚨 tests/fm-send-secondmate-marker-herdr-e2e.test.sh:45 - fail prints an error before calling cleanup_all, but cleanup_all captures that successful print status and exits with 0, so any assertion failure in this suite is reported as a passing process. Capture the failure status before cleanup or pass it explicitly.
• ⚠️ tests/fm-backend-herdr-prune-safety-e2e.test.sh:55 - The required criterion is “Never touch default session.” This no-argument call invokes fm_backend_herdr_version_check, whose no-session branch runs herdr status --json against the default session; the respawn suite has the same call. Confirm that read-only default-session access is allowed, or pass the isolated $SESSION explicitly.
• ⚠️ tests/fm-send-secondmate-marker-herdr-e2e.test.sh:130 - The marker assertion checks only the first matching log line, so a regression that submits the same marked request multiple times still passes. Assert that exactly one matching submission exists and that no additional submissions were recorded.
• ⚠️ tests/fm-backend-herdr-workspace-per-home-e2e.test.sh:196 - The workspace-isolation checks only require expected labels and exclude three known labels; they do not reject duplicate or unrelated tabs. A leaked or misrouted task could remain while the test passes. Compare the complete tab-label sets/counts for both workspaces.

🔧 Fix: Hardened Herdr E2E cleanup, isolation, and inventory assertions
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• bash bin/fm-run-behavior-tests.sh
• FM_TEST_JOBS=1 bash bin/fm-run-behavior-tests.sh
• env -u NO_MISTAKES_GATE FM_HERDR_E2E=1 FM_TEST_JOBS=1 bash bin/fm-run-behavior-tests.sh
• Read-only herdr session list --json before and after testing
• git status --short

✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅

• ⚠️ linter found issues (exit code 1)

🔧 Fix: Captain: quote Herdr lab ID to clear SC2100
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.