test: back Herdr live-duplicate agent registrations with a running process

[cloud-practitioner]

Intent

Fix test failures found along the way: the duplicate-live-label case in tests/fm-backend-herdr-smoke.test.sh fails against the installed Herdr 0.9.x, on main as well as on branches, apparently because Herdr 0.9.x ignores the agent report call (report-agent). Make that case pass against the installed Herdr 0.9.x: first establish whether Herdr 0.9.x changed report-agent behaviour in a way firstmate's Herdr backend must handle (a real product bug) or whether only the test's assumption is stale, and fix the right side.

What Changed

• tests/fm-backend-herdr-smoke.test.sh and tests/fm-backend-herdr-respawn-idem-e2e.test.sh: in the live-duplicate case, the test now starts a claude symlink to sleep in the pane and waits until fm_backend_herdr_pane_process_state reads agent before it calls pane report-agent. Herdr 0.9.3 drops a registration within about a second when the pane's foreground is its own top shell, so before this change the registration was gone before the duplicate check ran. The smoke test also removes its new scratch dir on exit.
• Both tests now check that fm_backend_herdr_pane_agent_state returns live before testing that create_task refuses the duplicate label. Without that check, the case could pass on a stale or unreadable registration, which the husk check also refuses.
• docs/verification/runtime-backends.md: added a measurement from Herdr 0.9.3 with the probe commands and their output. A registration on a pane idling at its top shell is released, while a nested foreground shell keeps it. This confirms the backend's release handling still matches Herdr and that only the tests' assumption was out of date, so no product code changed.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change only touches tests and docs. Each live-duplicate fixture now runs an agent-named foreground process (a claude symlink to sleep). It waits until fm_backend_herdr_pane_process_state reads agent, then registers, then asserts fm_backend_herdr_pane_agent_state reads live before checking the refusal. I traced this against the classifier and the agent-state logic and it is correct. The new doc measurement backs the conclusion that only the test's assumption was stale: a real crew pane keeps its agent in the foreground, so it is not a product bug. The respawn-idem change matches a finding the pipeline declined earlier, but the author committed it themselves, so it does not conflict with a recorded decision.

Testing

I ran the two changed real-Herdr tests (smoke and respawn-idem) against the installed Herdr 0.9.3 on both the branch and a copy of base. Both passed on base by winning a race, so I added a 2s delay after report-agent in throwaway copies. With the delay, base fails exactly as reported and the branch still passes. A guarded fm-herdr-lab probe showed the cause: Herdr releases the registration within about 280ms on an idle top shell, but keeps it under a nested shell or a claude-named process. Firstmate's classifier reads these three shapes as no-agent, stale-agent and live, which is the correct product behaviour, so only the test assumption needed fixing. Everything passed; the lab session and temp copies are torn down.

• Live validation: ✅ go - 5 of 5 scenarios driven live against the product

Scenario	Result	Live	Evidence
Root cause: on Herdr 0.9.3, report-agent on a pane idling at its own top shell is dropped within about a second, but kept while a foreground process runs	✅ pass	live	probe-report-agent-release.txt: (A) idle zsh reads agent_not_found from t+282ms; (B) nested bash and (C) claude->sleep stay registered as idle for 3s
No product bug: the backend reads a released idle-shell report as a no-agent husk, a nested-shell leftover as stale-agent, and a running agent-named process as live	✅ pass	live	probe-backend-classification.txt: no-agent / stale-agent / live
Smoke test's duplicate-live-label case passes against the installed Herdr 0.9.3	✅ pass	live	smoke-branch.log: 'create_task refuses a same-labeled tab whose pane hosts a genuinely live registered agent', exit=0
Adversarial: a 2s delay after report-agent fails the base smoke test (the reported failure) while the branch still refuses	✅ pass	live	smoke-base-with-2s-delay.log (not ok REGRESSION, exit=1) vs smoke-branch-with-2s-delay.log (ok, exit=0)
Respawn-idem e2e still closes restored husks and refuses a genuinely live duplicate against Herdr 0.9.3	✅ pass	live	respawn-idem-branch.log exit=0; respawn-idem-base-with-2s-delay.log fails, respawn-idem-branch-with-2s-delay.log passes

Evidence: Herdr 0.9.3 report-agent registration lifetime probe (idle shell vs nested shell vs claude-named process)

Source: Herdr 0.9.3 report-agent registration lifetime probe (idle shell vs nested shell vs claude-named process)

# herdr 0.9.3; lab session fm-lab-probe-3913023-4000; 2026-10-01T02:18:36Z
## A. report-agent on w2:p1, foreground = its own top shell: [{"name":"zsh","argv0":"/bin/zsh"}]
  t+   14ms {"agent":"probe-agent","agent_status":"idle"}
  t+  282ms "agent_not_found"
  t+  544ms "agent_not_found"
  t+  809ms "agent_not_found"
  t+ 1072ms "agent_not_found"
  t+ 1337ms "agent_not_found"
  t+ 1601ms "agent_not_found"
  t+ 1868ms "agent_not_found"
  t+ 2141ms "agent_not_found"
  t+ 2409ms "agent_not_found"
  t+ 2678ms "agent_not_found"
  t+ 2946ms "agent_not_found"
## B. report-agent on w3:p1, foreground = nested bash: [{"name":"bash","argv0":"bash"}]
  t+   14ms {"agent":"probe-agent","agent_status":"idle"}
  t+  279ms {"agent":"probe-agent","agent_status":"idle"}
  t+  542ms {"agent":"probe-agent","agent_status":"idle"}
  t+  806ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1069ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1336ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1599ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1864ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2126ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2388ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2649ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2911ms {"agent":"probe-agent","agent_status":"idle"}
## C. report-agent on w4:p1, foreground = claude->sleep symlink (the fixture's backing process): [{"name":"claude","argv0":null}]
  t+   12ms {"agent":"probe-agent","agent_status":"idle"}
  t+  275ms {"agent":"probe-agent","agent_status":"idle"}
  t+  538ms {"agent":"probe-agent","agent_status":"idle"}
  t+  801ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1063ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1327ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1588ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 1850ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2112ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2374ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2636ms {"agent":"probe-agent","agent_status":"idle"}
  t+ 2898ms {"agent":"probe-agent","agent_status":"idle"}

Evidence: Firstmate backend classification of the probe panes

Source: Firstmate backend classification of the probe panes

w2:p1 process_state=shell agent_state=no-agent w3:p1 process_state=shell agent_state=stale-agent w4:p1 process_state=agent agent_state=live

w2:p1 process_state=shell agent_state=no-agent
w3:p1 process_state=shell agent_state=stale-agent
w4:p1 process_state=agent agent_state=live

Evidence: Smoke test on branch

Source: Smoke test on branch

ok - real herdr: version_check accepts the installed binary's protocol
ok - real herdr: container_ensure starts the isolated session's server, creates the firstmate workspace (fm-lab-backend-smoke-3901326:w1), and reports its seeded default tab id (w1:t1)
ok - real herdr: session status normalizes running and compatible
ok - real herdr: container_ensure is idempotent (reuses/adopts the existing firstmate workspace, reports no seeded default tab on adoption)
ok - real herdr: create_task prunes the freshly-created workspace's seeded default tab, leaving exactly one clean fm-<id> task tab
ok - real herdr: create_task refuses a same-labeled tab whose pane hosts a genuinely live registered agent (unchanged behavior)
ok - real herdr: create_task closes and replaces a same-labeled tab whose pane hosts no registered agent (the restored-husk shape), leaving the workspace intact
ok - real herdr: a secondmate-shaped home (.fm-secondmate-home) gets its OWN herdr workspace, distinct from the primary's, in the SAME session
ok - real herdr: the secondmate-shaped home's workspace is labeled 2ndmate-<secondmate-id> in herdr itself
ok - real herdr: a task spawned into the secondmate-shaped home lands as a tab inside the secondmate's OWN workspace
ok - real herdr: list_live stays scoped to each home's own workspace - neither home sees the other's tasks
ok - real herdr: BOTH workspace ids/labels AND both tasks' pane ids survive a session stop + fresh server restart (multi-workspace shape)
ok - real herdr: send_text_line runs a command atomically (pane run) and its output is capturable
ok - real herdr: send_literal + send_key Enter submit as two separate steps (verified: send-text does NOT auto-submit)
ok - real herdr: current_path reads the pane's live cwd
note: FM_HERDR_SMOKE_REAL_CLAUDE=1 not set; skipping the real-agent busy_state check
ok - real herdr: kill removes the pane and is idempotent/best-effort
ok - real herdr: list_live discovers a live task tab by fm-<id> label
exit=0

Evidence: Respawn-idem e2e on branch

Source: Respawn-idem e2e on branch

ok - repro setup: two real fm-<id> task tabs exist (crewmate-shaped and secondmate-shaped), neither with a registered agent
ok - repro confirmed: after a real session restart, both task panes survive alive but with no registered agent - the restored-layout husk
ok - fixed: create_task closes and replaces the crewmate-shaped restored husk instead of refusing - no manual pane close needed
ok - fixed: create_task closes and replaces the secondmate-shaped restored husk instead of refusing - same fix, same function, both spawn shapes
ok - fixed: the workspace holds exactly the 2 replacement tabs after both respawns - no leaked husk tabs, no destroyed workspace
ok - fixed: a genuinely live duplicate (a real registered agent) still refuses exactly as before - the husk fix never closes a live pane
exit=0

Evidence: Smoke test at base with 2s delay (reproduces the reported failure)

Source: Smoke test at base with 2s delay (reproduces the reported failure)

not ok - REGRESSION: create_task should refuse a duplicate label whose pane hosts a genuinely live registered agent (idle counts as live) exit=1

ok - real herdr: version_check accepts the installed binary's protocol
ok - real herdr: container_ensure starts the isolated session's server, creates the firstmate workspace (fm-lab-backend-smoke-3919723:w1), and reports its seeded default tab id (w1:t1)
ok - real herdr: session status normalizes running and compatible
ok - real herdr: container_ensure is idempotent (reuses/adopts the existing firstmate workspace, reports no seeded default tab on adoption)
ok - real herdr: create_task prunes the freshly-created workspace's seeded default tab, leaving exactly one clean fm-<id> task tab
not ok - REGRESSION: create_task should refuse a duplicate label whose pane hosts a genuinely live registered agent (idle counts as live)
fm-herdr-lab: missing fleet-state tripwire for 'fm-lab-backend-smoke-3919723'; refusing destructive calls
exit=1

Evidence: Smoke test on branch with the same 2s delay

Source: Smoke test on branch with the same 2s delay

ok - real herdr: version_check accepts the installed binary's protocol
ok - real herdr: container_ensure starts the isolated session's server, creates the firstmate workspace (fm-lab-backend-smoke-3920456:w1), and reports its seeded default tab id (w1:t1)
ok - real herdr: session status normalizes running and compatible
ok - real herdr: container_ensure is idempotent (reuses/adopts the existing firstmate workspace, reports no seeded default tab on adoption)
ok - real herdr: create_task prunes the freshly-created workspace's seeded default tab, leaving exactly one clean fm-<id> task tab
ok - real herdr: create_task refuses a same-labeled tab whose pane hosts a genuinely live registered agent (unchanged behavior)
ok - real herdr: create_task closes and replaces a same-labeled tab whose pane hosts no registered agent (the restored-husk shape), leaving the workspace intact
ok - real herdr: a secondmate-shaped home (.fm-secondmate-home) gets its OWN herdr workspace, distinct from the primary's, in the SAME session
ok - real herdr: the secondmate-shaped home's workspace is labeled 2ndmate-<secondmate-id> in herdr itself
ok - real herdr: a task spawned into the secondmate-shaped home lands as a tab inside the secondmate's OWN workspace
ok - real herdr: list_live stays scoped to each home's own workspace - neither home sees the other's tasks
ok - real herdr: BOTH workspace ids/labels AND both tasks' pane ids survive a session stop + fresh server restart (multi-workspace shape)
ok - real herdr: send_text_line runs a command atomically (pane run) and its output is capturable
ok - real herdr: send_literal + send_key Enter submit as two separate steps (verified: send-text does NOT auto-submit)
ok - real herdr: current_path reads the pane's live cwd
note: FM_HERDR_SMOKE_REAL_CLAUDE=1 not set; skipping the real-agent busy_state check
ok - real herdr: kill removes the pane and is idempotent/best-effort
ok - real herdr: list_live discovers a live task tab by fm-<id> label
exit=0

Evidence: Respawn-idem at base with 2s delay (fails)

Source: Respawn-idem at base with 2s delay (fails)

ok - repro setup: two real fm-<id> task tabs exist (crewmate-shaped and secondmate-shaped), neither with a registered agent
ok - repro confirmed: after a real session restart, both task panes survive alive but with no registered agent - the restored-layout husk
ok - fixed: create_task closes and replaces the crewmate-shaped restored husk instead of refusing - no manual pane close needed
ok - fixed: create_task closes and replaces the secondmate-shaped restored husk instead of refusing - same fix, same function, both spawn shapes
ok - fixed: the workspace holds exactly the 2 replacement tabs after both respawns - no leaked husk tabs, no destroyed workspace
not ok - REGRESSION: create_task should refuse a same-labeled tab whose pane hosts a genuinely live registered agent
fm-herdr-lab: missing fleet-state tripwire for 'fm-lab-respawn-idem-e2e-3924196'; refusing destructive calls
exit=1

Evidence: Respawn-idem on branch with 2s delay (passes)

Source: Respawn-idem on branch with 2s delay (passes)

ok - repro setup: two real fm-<id> task tabs exist (crewmate-shaped and secondmate-shaped), neither with a registered agent
ok - repro confirmed: after a real session restart, both task panes survive alive but with no registered agent - the restored-layout husk
ok - fixed: create_task closes and replaces the crewmate-shaped restored husk instead of refusing - no manual pane close needed
ok - fixed: create_task closes and replaces the secondmate-shaped restored husk instead of refusing - same fix, same function, both spawn shapes
ok - fixed: the workspace holds exactly the 2 replacement tabs after both respawns - no leaked husk tabs, no destroyed workspace
ok - fixed: a genuinely live duplicate (a real registered agent) still refuses exactly as before - the husk fix never closes a live pane
exit=0

Evidence: Smoke test at base, no delay (passes by winning the race)

Source: Smoke test at base, no delay (passes by winning the race)

ok - real herdr: version_check accepts the installed binary's protocol
ok - real herdr: container_ensure starts the isolated session's server, creates the firstmate workspace (fm-lab-backend-smoke-3907397:w1), and reports its seeded default tab id (w1:t1)
ok - real herdr: session status normalizes running and compatible
ok - real herdr: container_ensure is idempotent (reuses/adopts the existing firstmate workspace, reports no seeded default tab on adoption)
ok - real herdr: create_task prunes the freshly-created workspace's seeded default tab, leaving exactly one clean fm-<id> task tab
ok - real herdr: create_task refuses a same-labeled tab whose pane hosts a genuinely live registered agent (unchanged behavior)
ok - real herdr: create_task closes and replaces a same-labeled tab whose pane hosts no registered agent (the restored-husk shape), leaving the workspace intact
ok - real herdr: a secondmate-shaped home (.fm-secondmate-home) gets its OWN herdr workspace, distinct from the primary's, in the SAME session
ok - real herdr: the secondmate-shaped home's workspace is labeled 2ndmate-<secondmate-id> in herdr itself
ok - real herdr: a task spawned into the secondmate-shaped home lands as a tab inside the secondmate's OWN workspace
ok - real herdr: list_live stays scoped to each home's own workspace - neither home sees the other's tasks
ok - real herdr: BOTH workspace ids/labels AND both tasks' pane ids survive a session stop + fresh server restart (multi-workspace shape)
ok - real herdr: send_text_line runs a command atomically (pane run) and its output is capturable
ok - real herdr: send_literal + send_key Enter submit as two separate steps (verified: send-text does NOT auto-submit)
ok - real herdr: current_path reads the pane's live cwd
note: FM_HERDR_SMOKE_REAL_CLAUDE=1 not set; skipping the real-agent busy_state check
ok - real herdr: kill removes the pane and is idempotent/best-effort
ok - real herdr: list_live discovers a live task tab by fm-<id> label
exit=0

Evidence: Respawn-idem at base, no delay (passes by winning the race)

Source: Respawn-idem at base, no delay (passes by winning the race)

ok - repro setup: two real fm-<id> task tabs exist (crewmate-shaped and secondmate-shaped), neither with a registered agent
ok - repro confirmed: after a real session restart, both task panes survive alive but with no registered agent - the restored-layout husk
ok - fixed: create_task closes and replaces the crewmate-shaped restored husk instead of refusing - no manual pane close needed
ok - fixed: create_task closes and replaces the secondmate-shaped restored husk instead of refusing - same fix, same function, both spawn shapes
ok - fixed: the workspace holds exactly the 2 replacement tabs after both respawns - no leaked husk tabs, no destroyed workspace
ok - fixed: a genuinely live duplicate (a real registered agent) still refuses exactly as before - the husk fix never closes a live pane
exit=0

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• Live validation: ✅ go - 5 of 5 scenarios driven live against the product

Scenario	Result	Live	Evidence
Root cause: on Herdr 0.9.3, report-agent on a pane idling at its own top shell is dropped within about a second, but kept while a foreground process runs	✅ pass	live	probe-report-agent-release.txt: (A) idle zsh reads agent_not_found from t+282ms; (B) nested bash and (C) claude->sleep stay registered as idle for 3s
No product bug: the backend reads a released idle-shell report as a no-agent husk, a nested-shell leftover as stale-agent, and a running agent-named process as live	✅ pass	live	probe-backend-classification.txt: no-agent / stale-agent / live
Smoke test's duplicate-live-label case passes against the installed Herdr 0.9.3	✅ pass	live	smoke-branch.log: 'create_task refuses a same-labeled tab whose pane hosts a genuinely live registered agent', exit=0
Adversarial: a 2s delay after report-agent fails the base smoke test (the reported failure) while the branch still refuses	✅ pass	live	smoke-base-with-2s-delay.log (not ok REGRESSION, exit=1) vs smoke-branch-with-2s-delay.log (ok, exit=0)
Respawn-idem e2e still closes restored husks and refuses a genuinely live duplicate against Herdr 0.9.3	✅ pass	live	respawn-idem-branch.log exit=0; respawn-idem-base-with-2s-delay.log fails, respawn-idem-branch-with-2s-delay.log passes

• bash tests/fm-backend-herdr-smoke.test.sh on the branch (real Herdr 0.9.3, own fm-lab-* session): exit 0, live-duplicate refusal ok
• bash tests/fm-backend-herdr-respawn-idem-e2e.test.sh on the branch: exit 0, live-duplicate refusal ok
• Both tests from a git archive bbc8d9a copy of base: both passed, so the base failure is a timing race
• Base and branch copies with sleep 2 added after herdr pane report-agent: base smoke and base respawn-idem both fail with not ok - REGRESSION: create_task should refuse ...; branch smoke and branch respawn-idem both pass
• Lab probe via bin/fm-herdr-lab.sh name/provision/run/teardown: pane report-agent then agent get polled every 250ms on (A) an idle top shell, (B) a nested bash in the foreground, (C) a claude->sleep symlink in the foreground
• fm_backend_herdr_pane_process_state / fm_backend_herdr_pane_agent_state run on the three probe panes in the lab
• bin/fm-herdr-lab.sh teardown, then confirmed only the default session remains and the worktree is clean

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.