Repository navigation
fix(bin): prevent stale Treehouse slot reuse - #6735
DiodatoNocita wants to merge 11 commits into
Conversation
|
| path=$(fm_meta_get "$meta" "$field") | ||
| [ -n "$path" ] || return 2 |
There was a problem hiding this comment.
Orca cleanup blocks Treehouse workers
spawn_record_names_treehouse_slot rejects an empty worktree= in every scanned record, including valid Orca cleanup records. If Orca creates a worktree but cannot return its path, and removal keeps failing, the existing abort path saves cleanup_recovery=orca with an empty worktree=. The new scan then refuses every Treehouse spawn in that home, even for unrelated projects and free slots.
Exclude records explicitly owned by Orca from the Treehouse ownership scan, while keeping ambiguous Treehouse evidence fail-closed. Add a test with this recovery record beside a free Treehouse slot.
|
Speaking as Kun's firstmate: thanks, @DiodatoNocita. I reviewed this against main What's needed from you: fix the Greptile P1 on Worth knowing too: your own pipeline log shows the Test step hit its 30-minute budget and Lint reported a warning (exit 1). CI is the real check, so please make sure it is green on the new head. What I checked: head Contract-class: new-default. The VISION.md, rule by rule
|
|
Speaking as Kun's firstmate: thanks for the new push, @DiodatoNocita. I re-reviewed head What changed since What's still needed from you: the Greptile P1 on What I checked: the no-mistakes attestation matches head Contract-class: new-default (no change). VISION.md, rule by rule
Next step: you (the Orca P1 fix), then CI. |
|
Speaking as Kun's firstmate: a quick update, @DiodatoNocita. #5343 (project-capacity admission) just landed on main as |
|
Speaking as Kun's firstmate: CI finished on Behavior tests (Herdr) failed in (Behavior portable serial 2 was also cancelled at its 30-minute limit. That one is a shard budget issue main has right now too, so I'm not holding it against you.) Still waiting on you, and still new-default, so it goes to Kun once it's otherwise ready. |
…laim repair and executable regression in bin/fm-spawn.sh and tests/fm-spawn-pool-base-freshen.test.sh. Verified with bash -n, git diff --check, and the full focused spawn-pool suite (passed). Removed abandoned .ci-herdr artifacts; no disposable artifacts remain. Intentional changes remain uncommitted for the outer executor
4108f6a to
bc7b57c
Compare
Intent
Implement a generic, explicit, force-gated Firstmate recovery that safely retires an obsolete local task record only when it collides with a specifically named current owner on the same managed Treehouse slot. Preserve the shared checkout, branch, current-owner record and endpoint, current and unrelated processes, reports, and unrelated project working directories; keep ordinary collision refusal unchanged; reject incorrect, ambiguous, remote, cross-home, cross-project, non-Treehouse, Orca, third-owner, path-drifted, live-obsolete-agent, unsafe-claim, unsafe-task-temp, process-present, or changing evidence. Use the existing crash-safe backlog closure and leave no broad destructive bypass. Investigate the host-crash/process-lease slot-reuse cause and prevent future reuse before spawn can replace a claim or reset/freshen the checkout. Add comprehensive executable success and refusal tests, maintainer documentation and help, do not touch live CAS records, validate fully, commit, and ship through no-mistakes.
What Changed
Risk Assessment
Testing
Used the completed disposable real-Herdr E2E result as directed, inspected the fixture’s exact slot-release ordering, and ran short syntax and patch-integrity checks. No UI artifact applies because this is a CLI/runtime backend scenario; the disposable captured logs were intentionally removed.
tests/fm-backend-herdr-presentation-e2e.test.shrun exited 0 with all 31 real-Herdr assertions passing; its required disposable.herdr-e2e-test.logand `.herdr-e2e-…Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ No issues found.
bin/fm-spawn.sh:3504- The same-task/same-home exception accepts a fresh spawn without proving the prior endpoint is agent-free. Concrete sequence: a task’s Treehouse process lease expires while its prior harness/terminal agent remains live; a freshfm-spawn.sh <same-id> <project>obtains that slot, line 3504 skips the durable-owner refusal solely because the record name matches, then the path claims/freshens the shared checkout and republishes the record for a second agent. Require the existing record’s validated endpoint to be recovery-grade and conclusively dead/gone before this exception; otherwise refuse and direct callers to relaunch/recovery. This was introduced by the prior CI repair commit 8788f31. The matching regression at tests/fm-spawn-pool-base-freshen.test.sh:841 repeats the spawn but never establishes that the first endpoint is absent, so it permits this unsafe case.🔧 Fix applied.
2 errors still open:
bin/fm-spawn.sh:3504- The same-task/same-home exception accepts a fresh spawn without proving the prior endpoint is agent-free. Concrete sequence: a task’s Treehouse process lease expires while its prior harness/terminal agent remains live; a freshfm-spawn.sh <same-id> <project>obtains that slot, line 3504 skips the durable-owner refusal solely because the record name matches, then the path claims/freshens the shared checkout and republishes the record for a second agent. Require the existing record’s validated endpoint to be recovery-grade and conclusively dead/gone before this exception; otherwise refuse and direct callers to relaunch/recovery. This was introduced by the prior CI repair commit 8788f31. The matching regression at tests/fm-spawn-pool-base-freshen.test.sh:841 repeats the spawn but never establishes that the first endpoint is absent, so it permits this unsafe case.bin/fm-spawn.sh:3564- A syntactically valid slot-owner claim can still be overwritten when its claimed task record is absent. Concrete path: Treehouse returns a slot whose.fm-slot-ownersaystask=foreignandhome=<existing local home>/; if<home>/state/foreign.metais absent, line 3564 skips inspection and spawn then writes its own claim/freshens the checkout. That defeats the durable-claim guard for a slot reassigned to a task whose record is not reachable, contrary to the intended reuse prevention. The same invariant is violated for a claim pointing into an already-scanned local state directory:known=1at bin/fm-spawn.sh:3560-3564 suppresses any claim-record check, so an absent claimed record is also silently overwritten. Refuse a different-task claim unless ownership has been conclusively released, and cover both absent-record paths through the spawn interface.🔧 Fix applied.
1 error still open:
bin/fm-teardown.sh:3681- Stale-owner retirement treats rawmissingas proof that the obsolete agent is gone. For a tmux task,fm_backend_agent_statereturnsmissingwhen the currently addressed server lacks the window, butfm_control_endpoint_absence_verdictdocuments that this cannot prove absence because the record has no tmux socket identity. Thus a live obsolete agent on another tmux server can pass this check, have its durable record/backlog retired, and lose recoverability—contrary to the required live-obsolete-agent refusal. Require a validated recovery-grade endpoint and acceptmissingonly after the shared absence verdict proves it gone; refuse the windowless shortcut too. Same invariant:bin/fm-teardown.sh:3676manufacturesmissingfor a record without any endpoint, which cannot establish agent absence. The new runtime test attests/fm-teardown-endpoint-safety.test.sh:796covers a visible agent but not this missing/unprovable-tmux path.🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Test** - 2 issues found → auto-fixed (2) → no changes applied → fix attempted; result not reported (3) ✅
git -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY statusandgit -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY diff). Respond with fix to validate it, or abort.🔧 No changes applied.
1 warning still open:
PATH=.fm-test-tools.S7z1Yz/node_modules/.bin:$PATH FM_TARGET_TEST=test_stale_owner_retirement_stages_crash_safe_backlog_closure bash tests/fm-backlog-atomicity.test.shFM_TARGET_TEST=test_pool_slot_recorded_owner_refuses_reuse_before_mutation bash tests/fm-spawn-pool-base-freshen.test.shPATH=.fm-test-tools.S7z1Yz/node_modules/.bin:$PATH FM_TEST_EVIDENCE=1 bash tests/fm-spawn-pool-base-freshen.test.shgit diff --checkLive validation:⚠️ inconclusive - 1 of 4 scenarios driven live against the product
bin/fm-herdr-lab.sh provision,run ... workspace list, andteardownreturned the recorded workspace-list JSONnpm install --prefix .fm-test-tools-48866 --no-save --ignore-scripts tasks-axi@0.2.6(disposable tool path)Focused executabletest_stale_owner_retirement_stages_crash_safe_backlog_closureFocused executable spawn slot-owner regression functionsFocused executable stale-owner endpoint-safety regression functionsbin/fm-herdr-lab.sh provision/run/teardownin an isolatedfm-lab-validation-livesession🔧 Fix applied.
2 warnings still open:
PATH=<isolated npm tasks-axi@0.2.6>/node_modules/.bin:$PATHtargeted execution oftest_stale_owner_retirement_stages_crash_safe_backlog_closurefromtests/fm-backlog-atomicity.test.shTargeted execution oftest_pool_slot_recorded_owner_refuses_reuse_before_mutation,test_pool_slot_claimed_owner_requires_a_record,test_pool_slot_exact_local_owner_can_reclaim, andtest_pool_slot_exact_local_owner_refuses_live_endpointfromtests/fm-spawn-pool-base-freshen.test.shTargeted execution oftest_explicit_stale_owner_retirement_preserves_the_shared_owner,test_stale_owner_runtime_evidence_refuses, andtest_stale_owner_structural_evidence_refuses_ambiguityfromtests/fm-teardown-endpoint-safety.test.shbin/fm-herdr-lab.sh provision <named fm-lab session>plus status inspection and guarded teardown🔧 Fix attempted; result not reported.
4 issues (1 error, 3 warnings) still open:
git -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY statusandgit -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY diff). Respond with fix to validate it, or abort.PATH=<isolated npm tasks-axi@0.2.6>/node_modules/.bin:$PATHtargeted execution oftest_stale_owner_retirement_stages_crash_safe_backlog_closurefromtests/fm-backlog-atomicity.test.shTargeted execution oftest_pool_slot_recorded_owner_refuses_reuse_before_mutation,test_pool_slot_claimed_owner_requires_a_record,test_pool_slot_exact_local_owner_can_reclaim, andtest_pool_slot_exact_local_owner_refuses_live_endpointfromtests/fm-spawn-pool-base-freshen.test.shTargeted execution oftest_explicit_stale_owner_retirement_preserves_the_shared_owner,test_stale_owner_runtime_evidence_refuses, andtest_stale_owner_structural_evidence_refuses_ambiguityfromtests/fm-teardown-endpoint-safety.test.shbin/fm-herdr-lab.sh provision <named fm-lab session>plus status inspection and guarded teardown🔧 Fix attempted; result not reported.
4 issues (1 error, 3 warnings) still open:
git -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY statusandgit -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY diff). Respond with fix to validate it, or abort.PATH=<isolated npm tasks-axi@0.2.6>/node_modules/.bin:$PATHtargeted execution oftest_stale_owner_retirement_stages_crash_safe_backlog_closurefromtests/fm-backlog-atomicity.test.shTargeted execution oftest_pool_slot_recorded_owner_refuses_reuse_before_mutation,test_pool_slot_claimed_owner_requires_a_record,test_pool_slot_exact_local_owner_can_reclaim, andtest_pool_slot_exact_local_owner_refuses_live_endpointfromtests/fm-spawn-pool-base-freshen.test.shTargeted execution oftest_explicit_stale_owner_retirement_preserves_the_shared_owner,test_stale_owner_runtime_evidence_refuses, andtest_stale_owner_structural_evidence_refuses_ambiguityfromtests/fm-teardown-endpoint-safety.test.shbin/fm-herdr-lab.sh provision <named fm-lab session>plus status inspection and guarded teardown🔧 Fix attempted; result not reported.
5 issues (2 errors, 3 warnings) still open:
git -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY statusandgit -C ~/.no-mistakes/worktrees/0cd3991fdb26/01M494AZT00KH1NZEDRMA0T6MY diff). Respond with fix to validate it, or abort.tests/fm-backend-herdr-presentation-e2e.test.sh- The completed captured real-Herdr run exited 1 after 21 passing scenarios because the post-legacy spawn reached a Treehouse slot still claimed by anchor after anchor.meta was removed. Preserve the production refusal; make the disposable fixture release the exact anchor slot before deleting its record, then run one status-captured E2E pass.PATH=<isolated npm tasks-axi@0.2.6>/node_modules/.bin:$PATHtargeted execution oftest_stale_owner_retirement_stages_crash_safe_backlog_closurefromtests/fm-backlog-atomicity.test.shTargeted execution oftest_pool_slot_recorded_owner_refuses_reuse_before_mutation,test_pool_slot_claimed_owner_requires_a_record,test_pool_slot_exact_local_owner_can_reclaim, andtest_pool_slot_exact_local_owner_refuses_live_endpointfromtests/fm-spawn-pool-base-freshen.test.shTargeted execution oftest_explicit_stale_owner_retirement_preserves_the_shared_owner,test_stale_owner_runtime_evidence_refuses, andtest_stale_owner_structural_evidence_refuses_ambiguityfromtests/fm-teardown-endpoint-safety.test.shbin/fm-herdr-lab.sh provision <named fm-lab session>plus status inspection and guarded teardown🔧 Fix applied.
✅ Re-checked - no issues remain.
tests/fm-backend-herdr-presentation-e2e.test.shrun exited 0 with all 31 real-Herdr assertions passing; its required disposable.herdr-e2e-test.logand `.herdr-e2e-…Operator-confirmed capturedtests/fm-backend-herdr-presentation-e2e.test.shreal-Herdr E2E run (status 0; 31 assertions; not rerun per instruction)bash -n tests/fm-backend-herdr-presentation-e2e.test.shgit diff --check ac0811c4c820d82009d7a3b8f09555cbf38340c8..HEADInspectedHEADfixture diff and verified.herdr-e2e-test.log/.herdr-e2e-test.statusare absent as required✅ **Document** - passed
✅ No issues found.
✅ No issues found.
🔧 No changes applied.
1 warning still open:
✅ **Push** - passed
✅ No issues found.
✅ No issues found.