test(tests): guard fixture cleanup from deleting the working directory - #29
Merged
zeeshaanahmad merged 3 commits intoAug 26, 2026
Merged
Conversation
A test file copied out of tests/ cannot resolve `. "$(dirname
"${BASH_SOURCE[0]}")/lib.sh"`, and bash treats a failed `.` as an ordinary
non-zero return. The file kept running with every helper undefined, so
`TMP_ROOT=$(fm_test_tmproot prefix)` became the empty string, the widespread
`TMP_ROOT=$(cd "$TMP_ROOT" && pwd -P)` idiom turned that into $PWD because
`cd ""` succeeds as a no-op, and the file's own EXIT trap ran `rm -rf` on it.
That deleted a live task worktree.
Three guards, because they fail independently:
- tests/lib.sh gains fm_test_rmtree, the single guarded removal path for a
fixture root. It refuses an empty path, and separately refuses any path that
does not resolve strictly inside the temp root fm_test_tmproot allocates
from - non-empty is not the same as safe, since $PWD is non-empty. The
registry reaper, the orphan sweep, and the 16 files that own their teardown
trap all route through it, so the guard is written once rather than at 112
call sites.
- Every relative source of a tests/-local helper now carries `|| exit 1`, so a
file that cannot load its library aborts at that line instead of continuing
with undefined helpers.
- tests/fm-control-herdr-smoke.test.sh checks its own mktemp before resolving
it through `cd`, which is the same degradation with a different trigger.
tests/fm-test-fixture-cleanup.test.sh pins all three: three cases drive
fm_test_rmtree directly (empty, outside-root, and the incident's exact
unset-root-resolves-to-$PWD chain), and a sweep copies all 125 library-dependent
files out of tests/ and asserts each one refuses and leaves its working
directory byte-identical.
… refused The temp-root containment rule was too strict for one real case. The source-boundary parity check in tests/fm-lint.test.sh feeds fm-lint.sh a REPO-RELATIVE source path, so its fixture has to be created under $ROOT rather than the temp root. The guard refused that teardown, failing the test and leaking the fixture into the working tree. Rather than widen the containment rule - which would have readmitted the original hazard, since the deleted worktree was itself under a repo root - tests/lib.sh gains fm_test_own_fixture: an explicit declaration that records a root in the same registry fm_test_tmproot uses, so fm_test_rmtree permits it. The declaration is itself guarded and refuses the repo root, the working directory, and any ancestor of it, which are exactly the shapes a degraded TMP_ROOT takes. The escape hatch therefore cannot authorize the deletion the guard exists to prevent. tests/fm-lint.test.sh declares its parity root through that helper, and also checks its own mktemp instead of leaving the result unverified. Two regression cases cover the new path: a declared root outside the temp root is removable, and declaring the working directory, an ancestor of it, or the repo root is refused. Removing the declaration guard fails the second.
…fused
tests/fm-session-start.test.sh registers two fixture roots at the literal path
/tmp/fm-<id>, mirroring firstmate's own per-task temp root convention because
that is what the code under test creates. On macOS TMPDIR is under /var/folders,
so those roots sit outside the fixture temp root and the guard refused their
teardown, failing the file and leaking both into /tmp.
Widening the containment rule to trust /tmp wholesale is NOT the fix: the working
directory can itself be under /tmp, which is exactly how the original incident
reproduced in a sandbox. Declaring keeps the cwd and repo-root guard in force
while permitting the teardown, and is the same mechanism tests/fm-lint.test.sh
already uses for its repo-rooted parity fixture.
A regression case pins both directions - undeclared stays refused, declared is
removed - and making the declaration a no-op fails it.
A sweep of every removal target in the suite confirms these were the last two
outside the fixture temp root: the remaining FM_TEST_CLEANUP_DIRS entries and
every fm_test_rmtree call site resolve under ${TMPDIR:-/tmp}.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Make the firstmate test suite's temp-directory cleanup incapable of deleting the working directory.
THE INCIDENT THIS FIXES (real, already happened): a crewmate copied a test out of tests/ to instrument it. Outside tests/, the
. "$(dirname "${BASH_SOURCE[0]}")/lib.sh"source failed; bash treats a failed.as an ordinary non-zero return, so the file kept running with every helper undefined. fm_test_tmproot was undefined, so TMP_ROOT became the empty string; the widespreadTMP_ROOT=$(cd "$TMP_ROOT" && pwd -P)idiom then turned that into $PWD becausecd ""succeeds as a no-op; and the file's own EXIT trap ranrm -rfon it, deleting a live task worktree. 112 test files use fm_test_tmproot. The bug was reproduced end to end in a throwaway sandbox before any fix, and the same action was re-run afterwards to prove it now aborts and deletes nothing.THREE REQUIREMENTS, all required because they fail independently:
rm -rf.SHAPE CHOSEN AND WHY (deliberate, not accidental): one shared guarded helper rather than 112 edited call sites. tests/lib.sh gains fm_test_rmtree as the single removal path for a fixture root, so the guard is written once. The 112 files that merely call fm_test_tmproot were left untouched; their fixtures are removed by fm_test_cleanup, which now routes through the guard. Only the 16 files that own their teardown trap and therefore bypass fm_test_cleanup needed a call-site edit, each a one-token substitution.
DELIBERATE DECISIONS A REVIEWER WOULD NOT SEE FROM THE DIFF:
|| exit 1is appended to every relative source of a tests/-local helper (130 lines across 107+ files, one token each). This is requirement 3 and is load-bearing, not style. The uniform mechanical shape was chosen precisely so the diff stays reviewable.2>/dev/nullon tests/fm-remote-backlog-handoff.test.sh's retry loop was deliberately preserved; it mitigates a documented pre-existing CI flake and the guard still prevents the deletion regardless of diagnostic visibility.rm -rf "/tmp/fm-$id". Neither is a fixture-root teardown and neither can resolve to the working directory.A COMMENT OR CONVENTION NOTE WAS EXPLICITLY NOT ACCEPTABLE as a fix, because the defect is that this happens silently to someone doing an ordinary thing.
REGRESSION AND MUTATION-FALSIFICATION (required): tests/fm-test-fixture-cleanup.test.sh gains cases that drive fm_test_rmtree directly (empty path, path outside the temp root, and the incident's exact unset-root-resolves-to-$PWD chain), cases for the declaration path in both directions, and a sweep that copies all 125 library-dependent files out of tests/ and asserts each refuses and leaves its working directory byte-identical. Every guard was mutation-falsified individually: removing it fails the suite, restoring it passes. The first mutation of the load guard did NOT fail, because the second layer absorbed it; the sweep was sharpened to assert the working directory is unchanged in both directions so it detects the silent continuation itself rather than only its worst outcome.
TWO REGRESSIONS THIS WORK ITSELF CAUSED, FOUND AND FIXED: the guard's first shape refused legitimate teardowns in tests/fm-lint.test.sh and tests/fm-session-start.test.sh. Both were found only by running the full CI portable lanes, not by targeted verification, and both are fixed by explicit guarded declaration rather than by weakening the guard. After the second, every removal target in the suite was enumerated to confirm no third exists.
SCOPE BOUNDARY: fix the deletion hazard only. Do not restructure test fixtures, rename helpers, or tidy the 112 call sites beyond what the guard needs. A large diff is harder to trust and this is the change where trust matters most.
REBASE CONTEXT: this branch was rebased onto current origin/main 99c1d7e, which is PR 28 ("bound remote-job worker waits to fix CI hangs and leaks") and edits tests/lib.sh too. tests/lib.sh did not conflict; both sets of helpers are present. Two conflicts were resolved keeping BOTH changes: tests/fm-remote-job.test.sh keeps PR 28's worker teardown in full and ends on fm_test_rmtree instead of raw
rm -rf, and tests/fm-send-remote-delivery.test.sh keeps the upstream-added pending-reply-lib source plus the load guard.FIRSTMATE REPO CONSTRAINTS: this repo has real working GitHub Actions (13 checks) and --skip=ci must NOT be passed; the checks must actually run and go green, which is the whole point for a change to a library every test loads. Push to origin only, never upstream. Never add an agent name as a commit co-author. One full sentence per line in tracked Markdown, plain dash never an em dash. bin/.sh and bin/backends/.sh must pass bin/fm-lint.sh.
What Changed
fm_test_rmtreetotests/lib.shas the single guarded removal path for a fixture root: it aborts loudly instead of runningrm -rfwhen the target path is unset/empty or falls outside the expected temp root, closing theTMP_ROOT=""→cd "" && pwd -P→$PWDchain that previously let a teardown trap delete a live working directory. Addedfm_test_own_fixtureas a guarded escape hatch for declaring fixture roots that legitimately live outside the temp root (refusing the repo root, the working directory, or any of its ancestors), and routedfm_test_cleanupthroughfm_test_rmtreeso it now exits 1 if any removal was refused.|| exit 1to the relative. "$(dirname "${BASH_SOURCE[0]}")/lib.sh"-style sources across 107+ test files, so a failed source now aborts the script immediately instead of silently continuing with every helper undefined.fm_test_cleanup) to callfm_test_rmtreeat their removal call site instead of rawrm -rf, includingtests/fm-remote-job.test.sh(keeping its worker teardown) andtests/fm-remote-backlog-handoff.test.sh(keeping its existing2>/dev/nullflake mitigation).tests/fm-lint.test.sh's source-boundary parity fixture andtests/fm-session-start.test.sh's literal/tmp/fm-<id>roots — throughfm_test_own_fixture, and expandedtests/fm-test-fixture-cleanup.test.shwith regression and mutation-falsification coverage forfm_test_rmtree, the declaration path, and a sweep that copies all library-dependent test files out oftests/and asserts each refuses cleanup while leaving its working directory unchanged.Risk Assessment
✅ Low: The change is tightly scoped to tests/, implements a symlink-safe, containment-correct guard (fm_test_rmtree/fm_test_own_fixture) that verifiably satisfies all three required invariants (abort on empty root, abort on out-of-root path, abort on failed lib.sh load), and is mutation-falsified with real regression coverage. I traced every remaining un-migrated raw rm -rf in tests/ (LAB/SCRATCH e2e fixtures, /tmp/fm-id literals, sub-path removals) and confirmed none can reach the reported incident's collapse mechanism (a cd ""-driven resolution to $PWD), matching the user intent's explicit scope boundary.
Testing
Ran the targeted regression suite (tests/fm-test-fixture-cleanup.test.sh) plus the two dependent files that exercise the new fm_test_own_fixture escape hatch (fm-lint.test.sh, fm-session-start.test.sh); all pass, directly proving the guard blocks the empty-TMP_ROOT, outside-temp-root, and incident-exact unset-root-resolves-to-$PWD deletion paths while still permitting the two legitimate out-of-root fixtures. Cross-checked the full 127-file diff against every specific claim in the stated intent (guard implementation, mechanical edit scope, intentionally unrouted call sites, rebase-merge resolution) and found no discrepancies. Working tree left clean.
Evidence: fm-test-fixture-cleanup.test.sh output — guard refusing each unsafe path and the 125-file outside-tests/ sweep
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
bash bin/fm-test-run.sh tests/fm-test-fixture-cleanup.test.sh— 12/12 pass, including new cases that drive fm_test_rmtree directly (empty path, path outside the temp root, and the incident's exact unset-TMP_ROOT-resolves-to-$PWD chain), the fm_test_own_fixture declaration path in both directions, and the 125-file sweep that copies every lib.sh-dependent test file outside tests/ and asserts each refuses with its working directory byte-identicalbash bin/fm-test-run.sh tests/fm-lint.test.sh tests/fm-session-start.test.sh— both suites fully green (4/4 and 49/49 respectively); these are the two files whose own teardown traps needed the fm_test_own_fixture escape hatch (fm-lint.test.sh's repo-relative $ROOT-based parity fixture, fm-session-start.test.sh's two /tmp/fm-<id> secondmate fixtures) and were the two regressions this branch itself found and fixed during CImanual diff review of all 127 changed test files against the stated intent: confirmed the|| exit 1mechanical edit on every relative tests/-local source line, confirmed the 15-16 files with an owned teardown trap were switched from rawrm -rftofm_test_rmtree, confirmed the two intentionally-unrouted removals (tests/fm-herdr-session-cleanup.test.sh's multi-arg glob cleanup and tests/fm-backend-orca.test.sh's rawrm -rf "/tmp/fm-$id") remain raw as scoped, confirmed the rebase-conflict resolutions in tests/fm-remote-job.test.sh and tests/fm-send-remote-delivery.test.sh, and confirmed the 2>/dev/null preservation in tests/fm-remote-backlog-handoff.test.sh's retry loopgit status --shortbefore and after the test runs — clean, no stray fixtures or artifacts left in the working tree✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.