feat(bin): add a disposable live supervision lab builder - #6037
Merged
Merged
Conversation
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… root before looking up the harness executable. The affected behavioral test and shell syntax check pass; the refusal also works with Claude absent from PATH
…nly recorded lab processes and their descendants; the worker gate is in its granted task directory and its path is exposed; readiness uses current crew state; and mate and worker IDs use 12 nonce hex digits. The CLI behavior tests pass, as do shell syntax, ShellCheck, and diff checks. The Claude no-host path is unchanged
…e binary by supplying a test-local stub. The full fm-live-lab test, shell syntax check, and diff check pass
….sh: down waits for recorded processes and escalates before cleanup, PID roots are checked against recorded start times, and Claude primary trust is rechecked after mate/worker readiness. Added behavioral tests in tests/fm-live-lab.test.sh. bin/fm-lint.sh and tests/fm-live-lab.test.sh pass
…uctions, unused retry variable, and teardown PID revalidation in bin/fm-live-lab.sh. Added behavioral tests in tests/fm-live-lab.test.sh. Both requested commands pass: tests/fm-live-lab.test.sh and bin/fm-lint.sh
…D and start time, including children orphaned when a root exits. Up now rejects an empty pane PID before calling ps. Added regression tests and a Linux-safe worker fixture. bin/fm-lint.sh and tests/fm-live-lab.test.sh pass
…shutdown and made the worker fixture verify its exact window with a Linux-available shell. Both requested checks pass. The lab test takes about 66 seconds locally, so the under-one-minute target remains unmet
…fm-live-lab.test.sh. Teardown now tracks identity-checked members of captured lab process groups, including children orphaned during shutdown, without signaling the caller’s group or unrelated processes. Lint passed, and the lab test passed four times
… permanently dropped, preventing a reused group ID from signalling unrelated work. Added a ps-shim regression test. The lab test, lint, and diff checks pass
…ab.test.sh. The TERM-born-child fixture now waits until its handler is installed before calling down. Down sends SIGKILL to identity-valid survivors on every pass from pass 20 onward and includes survivor process details if it must refuse cleanup. bin/fm-lint.sh and tests/fm-live-lab.test.sh pass locally; Linux CI remains to be verified
…tity-checked scans separated by 0.5 seconds, and removed the unused test loop variable without changing the TERM-born-child test. The lab test, lint, and diff check pass locally
andrewesweet
pushed a commit
to andrewesweet/firstmate
that referenced
this pull request
Sep 30, 2026
…#6037) * feat(bin): add fm-live-lab.sh, a one-command live supervision lab builder * fix(bin): exact lab windows, per-lab task ids, self-safe teardown * fix(bin): target lab windows by id, stop lab descendants, add readiness tests * fix(bin): keep Claude's auto-updater off in live labs; list fm-live-lab.sh * fix(bin): start the lab tmux server without user config * no-mistakes(review): Scope lab teardown to its store, root, and task ids * no-mistakes(review): Record selected user stores at up for check and down * no-mistakes(document): Clarify live lab documentation and remove stale narratives * no-mistakes(ci): Fixed the CI failure by checking for an existing lab root before looking up the harness executable. The affected behavioral test and shell syntax check pass; the refusal also works with Claude absent from PATH * no-mistakes(ci): Fixed all four Greptile findings: teardown signals only recorded lab processes and their descendants; the worker gate is in its granted task directory and its path is exposed; readiness uses current crew state; and mate and worker IDs use 12 nonce hex digits. The CLI behavior tests pass, as do shell syntax, ShellCheck, and diff checks. The Claude no-host path is unchanged * no-mistakes(ci): Fixed the CI test’s dependence on an installed Claude binary by supplying a test-local stub. The full fm-live-lab test, shell syntax check, and diff check pass * no-mistakes(ci): Fixed all three selected findings in bin/fm-live-lab.sh: down waits for recorded processes and escalates before cleanup, PID roots are checked against recorded start times, and Claude primary trust is rechecked after mate/worker readiness. Added behavioral tests in tests/fm-live-lab.test.sh. bin/fm-lint.sh and tests/fm-live-lab.test.sh pass * no-mistakes(ci): Fixed the pre-primary settle wait, worker gate instructions, unused retry variable, and teardown PID revalidation in bin/fm-live-lab.sh. Added behavioral tests in tests/fm-live-lab.test.sh. Both requested commands pass: tests/fm-live-lab.test.sh and bin/fm-lint.sh * no-mistakes(ci): Fixed teardown to track pre-kill lab processes by PID and start time, including children orphaned when a root exits. Up now rejects an empty pane PID before calling ps. Added regression tests and a Linux-safe worker fixture. bin/fm-lint.sh and tests/fm-live-lab.test.sh pass * no-mistakes(ci): Fixed teardown tracking for children spawned during shutdown and made the worker fixture verify its exact window with a Linux-available shell. Both requested checks pass. The lab test takes about 66 seconds locally, so the under-one-minute target remains unmet * no-mistakes(ci): Fixed ci-2 and ci-4 in bin/fm-live-lab.sh and tests/fm-live-lab.test.sh. Teardown now tracks identity-checked members of captured lab process groups, including children orphaned during shutdown, without signaling the caller’s group or unrelated processes. Lint passed, and the lab test passed four times * no-mistakes(ci): Fixed teardown so an observed-empty process group is permanently dropped, preventing a reused group ID from signalling unrelated work. Added a ps-shim regression test. The lab test, lint, and diff checks pass * no-mistakes(ci): Fixed ci-1 in bin/fm-live-lab.sh and tests/fm-live-lab.test.sh. The TERM-born-child fixture now waits until its handler is installed before calling down. Down sends SIGKILL to identity-valid survivors on every pass from pass 20 onward and includes survivor process details if it must refuse cleanup. bin/fm-lint.sh and tests/fm-live-lab.test.sh pass locally; Linux CI remains to be verified * no-mistakes(ci): Fixed down’s teardown wait to require two empty identity-checked scans separated by 0.5 seconds, and removed the unused test loop variable without changing the TERM-born-child test. The lab test, lint, and diff check pass locally
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
convert decided plan to an implementation plan in md file and pass that back to firstmate for implementation
Context: that closed the captain's review of the AFK revamp audit, in which the captain selected "Sequence and 3d scope: approve-claude-only-3d1" (the audit's proposed execution sequence approved; the first default-on flip is Claude-only), "3e non-Claude scope: build-3d2-first", all twelve backlog dispositions, and "Sailboat and anchor on Claude: build-now-always-on". The resulting plan is data/fm-afk-revamp-audit-s1/implementation-plan.md in the Firstmate home ~/fm-homes/fmdev-f1, with its evidence in report.md beside it. The AFK revamp's standing words also apply: "2 should be done by opus crewmates" and "this is a major architectural revamp so i want it to do very careful live validation including regression in isolated live environments with some real complex sessions before calling it done. it's ok to use my real llm tokens here".
This task is L in that plan (an accelerator the Claude default-on flip's lab validation uses): a one-command tested lab builder for live supervision validation. Its earlier approval ("yes", via main) rested on this evidence: the signal-span lane spent about 5 hours, of which the code change was about 20 minutes; the rest was hand-built lab misses (Claude workspace-trust prompt in the lab copy, attended mirror feed not wired, a preflight only on an unmerged branch, Pi lab missing the branch extension, a Codex model the account rejects), each forcing a rebuild. Scope as approved: one command standing up a disposable lab main session on Claude or Pi with the supervision host or branch fully wired (hooks, mirror feed, all extensions, lab-only trust), optionally a real seeded local second mate and a gated worker, verifying readiness itself before returning, and tearing everything down cleanly (tmux servers, lab dirs, lab ~/.claude.json entries by atomic replace, no treehouse residue). Addendum, exact words: "(and make sure it references what this long session had done, not literally redoing the whole thing all over again)".
What Changed
fm-live-lab.shto build, check, interact with, and tear down isolated Claude or Pi supervision labs, with optional seeded second mate and gated worker.Risk Assessment
🚨 High: Teardown can kill another lab's processes and break a supported Claude configuration layout, so this should not merge without correction or explicit approval.
Testing
Targeted behavioral tests passed. Real Pi and Claude primaries launched in disposable labs, but neither reached a probe-ready turn because the isolated homes lacked login setup; both labs were torn down successfully. CLI transcripts were captured as evidence.
bash tests/fm-live-lab.test.shdrives the real CLI against private tmux servers and processes; both real-primary attempts also ended with successfuldownEvidence: Pi launch, readiness refusal, and teardown
Source: Pi launch, readiness refusal, and teardown
Evidence: Claude first-run dialog and teardown
Source: Claude first-run dialog and teardown
Live validation with real logins
The pipeline's own live attempts above ran in a credential-less sandbox (Pi had no Codex key; Claude stopped at its first-run theme picker), so the implementer drove the real harnesses with real logins on macOS arm64 (Claude Code 2.1.283 sonnet, Pi 0.82.0
openai-codex/gpt-6-luna).4f23477c--mate --workercount_notes(), and reported done; the supervision host took the combined wake (handled ... posture=attended reports=2), recorded the worker's finish as captain outcome 1 and a staged second-mate row as routine outcome 2, and the lab main woke through the real Stop-hook rewake and drainedBRANCH OUTCOMES.530ff2e4--materoutinewithin about a minute.17c4c9bf--mateand Pi, concurrentlycaa0e0a4(pipeline head, after the review fixes)--mate --workerand Pi--mate, concurrently, builder and lab tree both from this headready;downexit 0 for both. Against snapshots taken just before: no lab process left,~/.treehouseidentical, Pi trust store byte-identical,~/.claude.jsonproject keys added none and lost none, no private tmux dir left, the lab tasks'/tmp/fm-<id>and/tmp/fm-<id>+<home-sha>dirs removed.e2dc0e56--mate --worker~/.claude.jsonand dropped the primary's new trust entry, so the primary hit the trust dialog;downdid not wait for exiting Claude processes, so the mate re-added its entry. Both fixed in later rounds.89023211--mate --workerdownwas clean.eeb406a3--mate --workerdownexit 0 with no residue, including the detached supervision-host chain.c7a65940--mate --workerdownexit 0, every pre-down lab process gone, no residue.95d7b2b9(final pipeline head)--mate --worker, builder and lab tree both from this headready;downexit 0: none of the 7 pre-down lab processes alive afterwards, 4 Claude entries removed,~/.claude.jsonproject keys added none and lost none, Pi trust store identical, no new~/.treehouseentry, no new/tmp/fm-*or/tmp/fml.*.Earlier runs also found and fixed, before review: vacuous window checks (tmux resolves a missing window name, even
=name, to the current window), teardown matching its own subshells, orphaned descendants of lab processes, per-spawn/tmpdirs outside the lab, Claude's auto-updater replacing the shared binary under a lab, and the user's tmux plugins running in the lab server.A marker-based teardown (every lab process carrying a lab-unique environment variable) was evaluated in a live lab and rejected: macOS
psdoes not expose the environment of Apple platform binaries such as thecaffeinateevery Claude process starts or the watcher'ssleep, so those lab children would be invisible to it. Teardown instead tracks pre-kill pid+start-time pairs, their descendants, and live members of recorded process groups while those groups stay non-empty.Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-live-lab.sh:159- With CLAUDE_CONFIG_DIR set, lab_run drops it, so Claude trust registration and launches use $HOME/.claude.json, while check_trust (line 237) and down (line 571) inspect the configured store. Readiness fails and teardown leaves the lab's trust entries behind. Preserve one selected store across up, check, and down, including invocations from a later shell.bin/fm-live-lab.sh:597- fm-claude-trust.sh explicitly supports a symlinked .claude.json by writing its resolved target, but down renames over the symlink. For a user with that supported layout, teardown replaces their link and leaves the target containing lab entries. Resolve and validate the target before the atomic replacement, as trust registration does.bin/fm-live-lab.sh:560- lab_pids matches the lab root as an unbounded substring of any process command. If explicit lab roots are /tmp/lab and /tmp/lab2, down /tmp/lab selects and kills processes for /tmp/lab2 at line 620. Restrict process ownership to this lab's endpoints or path-bounded arguments before sending signals.bin/fm-live-lab.sh:630- A spawn can create /tmp/fm-<id> at fm-spawn.sh:4369 and fail before publishing state/<id>.meta. Because down discovers task temp directories only through metadata, a failed --mate or --worker up leaves that directory behind. Clean the recorded lab IDs even when their metadata was never published.bin/fm-live-lab.sh:447- The new --env option is not required by the stated lab-builder scope. At line 519 it can override FM_HOME or HOME after the lab values, making the primary operate on a non-lab home before readiness detects the mismatch. Remove this unrequired arbitrary environment-override component rather than hardening it.bin/fm-live-lab.sh:458- The Claude --supervision-host none / --expect-host no path introduced at lines 443–444 and 501, with checks skipped at line 332, builds a lab without the supervision host. No stated requirement needs a hostless Claude mode; the requested lab has the host fully wired. Remove this opt-out path while retaining selection of a host line.🔧 Fix applied.
7 issues (3 errors, 4 warnings) still open:
bin/fm-live-lab.sh:159- With CLAUDE_CONFIG_DIR set, lab_run drops it, so Claude trust registration and launches use $HOME/.claude.json, while check_trust (line 237) and down (line 571) inspect the configured store. Readiness fails and teardown leaves the lab's trust entries behind. Preserve one selected store across up, check, and down, including invocations from a later shell.bin/fm-live-lab.sh:597- fm-claude-trust.sh explicitly supports a symlinked .claude.json by writing its resolved target, but down renames over the symlink. For a user with that supported layout, teardown replaces their link and leaves the target containing lab entries. Resolve and validate the target before the atomic replacement, as trust registration does.bin/fm-live-lab.sh:560- lab_pids matches the lab root as an unbounded substring of any process command. If explicit lab roots are /tmp/lab and /tmp/lab2, down /tmp/lab selects and kills processes for /tmp/lab2 at line 620. Restrict process ownership to this lab's endpoints or path-bounded arguments before sending signals.bin/fm-live-lab.sh:630- A spawn can create /tmp/fm-<id> at fm-spawn.sh:4369 and fail before publishing state/<id>.meta. Because down discovers task temp directories only through metadata, a failed --mate or --worker up leaves that directory behind. Clean the recorded lab IDs even when their metadata was never published.bin/fm-live-lab.sh:447- The new --env option is not required by the stated lab-builder scope. At line 519 it can override FM_HOME or HOME after the lab values, making the primary operate on a non-lab home before readiness detects the mismatch. Remove this unrequired arbitrary environment-override component rather than hardening it.bin/fm-live-lab.sh:458- The Claude --supervision-host none / --expect-host no path introduced at lines 443–444 and 501, with checks skipped at line 332, builds a lab without the supervision host. No stated requirement needs a hostless Claude mode; the requested lab has the host fully wired. Remove this opt-out path while retaining selection of a host line.bin/fm-live-lab.sh:497- The round-1 trust-store fix records an empty claude_config_dir when CLAUDE_CONFIG_DIR is unset. Ifupruns with HOME=A anddownruns later with HOME=B, claude_store (line 159) checks B/.claude.json, leaving the lab trust entries in A/.claude.json;checkhas the same sibling failure at line 247. Record the absolute store selected at up, including the default, without changing the primary's default Claude environment.🔧 Fix applied.
1 warning still open:
bin/fm-live-lab.sh:458- The Claude --supervision-host none / --expect-host no path introduced at lines 443–444 and 501, with checks skipped at line 332, builds a lab without the supervision host. No stated requirement needs a hostless Claude mode; the requested lab has the host fully wired. Remove this opt-out path while retaining selection of a host line.bash tests/fm-live-lab.test.shdrives the real CLI against private tmux servers and processes; both real-primary attempts also ended with successfuldownbash tests/fm-live-lab.test.shbin/fm-live-lab.sh up --harness pi --timeout 45 <isolated-worktree-lab>; inspected its pane and randownbin/fm-live-lab.sh up --harness claude --supervision-host none --expect-host no --timeout 15 <isolated-worktree-lab>; inspected its pane and randown✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.