Repository navigation
feat(bin): expand worker launch controls and watcher reliability - #21
Merged
Merged
Conversation
…id#6649) * fix(herdr): make exact-resume presentation-lock wait instead of a bounded timeout The exact-resume path in bin/fm-spawn.sh used the same 50-attempt-then- give-up lock acquire as the new-task-create path, but the two paths are not equivalent on contention: a create has no prior state to strand and can safely fall back to a flat layout, while a resume is recovering a specific existing identity that a concurrent recovery may legitimately be holding the lock for. Giving up there does not degrade gracefully, it hard-fails the resume outright. The suite's own concurrent cross-home recoveries test already asserts both concurrent recoveries succeed with a genuine reclaim, and the file's header comment already (inaccurately) claimed lock contention falls back to the ordinary flat layout for both paths alike, so the intended contract was always that recoveries serialize and both succeed, not that either one refuses under a short bound. Give spawn_herdr_presentation_order_lock_acquire a wait mode that uses this file's own established fm_lock_acquire_wait idiom (already used for its other fleet-shared locks) instead of the bounded loop, and use it only at the exact-resume call site. The new-task-create call site is unchanged and keeps its bounded-then-flat-fallback behavior, which is already covered by its own passing test. Dead-owner PID-liveness reclaim inside fm_lock_try_acquire still bounds the wait against a holder that crashed mid-hold. Adds a deterministic regression test that holds the shared session lock from an unrelated process for well past the old bound, then asserts the resume succeeds with a genuine reclaim and took close to the full hold duration, so a fix that merely widens the bound rather than genuinely waiting is still caught. The existing concurrent cross-home recovery test exercises this under real timing but does not reliably outlast a fixed bound on its own. Corrects the header comment's claim that create and resume share one bounded-then-flat-fallback behavior on lock contention; they no longer do. * no-mistakes(document): Document Herdr recovery waiting for presentation lock * no-mistakes(document): Update stale hard-refusal claim in verification log * no-mistakes(ci): Fixed the Greptile finding on tests/fm-backend-herdr-presentation-e2e.test.sh:1389 by bounding the resume lock-wait regression's spawn_task call. Added an optional 4th `deadline_seconds` arg to the `spawn_task` helper (defaults to empty, so all ~20 other existing call sites are unaffected and unwrapped by `timeout`). The lock-wait test now passes `LOCK_WAIT_HOLD_SECONDS + 60` (90s) as the deadline, and a dedicated check for exit code 124 emits a clear "hung for over Xs instead of waiting out a Ys lock hold" diagnostic before falling through to the existing pass/fail assertions, which are unchanged. No product code was touched. Verified with `bash -n`, `shellcheck -x` (no warnings), a standalone reproduction of the timeout/no-timeout/success paths, the project's `bin/fm-lint.sh --fast` on the file (clean), and the full `tests/fm-lint.test.sh` suite (all 46 assertions pass) * no-mistakes(ci): Replaced the direct `timeout "$deadline_seconds"` call in `spawn_task()` (tests/fm-backend-herdr-presentation-e2e.test.sh) with the repo's portable bounded-execution helper: sourced `bin/fm-timeout-lib.sh` at the top of the file and changed `deadline_cmd=(timeout "$deadline_seconds")` to `deadline_cmd=(fm_run_timed "$deadline_seconds")`. This removes the GNU/BSD `timeout` dependency that would fail with exit 127 on a stock macOS host without coreutils, while preserving identical semantics (exit 124 on bound-hit, command's own exit otherwise), which the existing `[ "$LOCK_WAIT_STATUS" -eq 124 ]` diagnostic check already relies on. Verified: `bash -n` syntax check, `bin/fm-lint.sh --fast` clean, full `tests/fm-lint.test.sh` suite (46/46 pass), and a standalone repro confirming `fm_run_timed` returns 124 on timeout and 0 on success identically to the prior `timeout` call. No other direct `timeout` calls exist in this file or elsewhere in the PR's diff, so no sibling sites remain * fix(herdr): gate exact-resume lock wait behind --herdr-resume-lock-wait Keep refuse-by-default on presentation-order lock contention for Herdr exact resume. Callers that need concurrent recoveries to serialize must pass --herdr-resume-lock-wait; unbounded blocking on a third-party session lock is never the default. Update docs and the real-Herdr e2e suite so the default path asserts the refusal and the opt-in path asserts the wait. * no-mistakes(test): Fix e2e test's lost exit status after if/fi with no else branch * docs(herdr): stop advertising --herdr-resume-lock-wait on --relaunch The relaunch path reuses the recorded endpoint and never takes the presentation-order lock, so the flag is inert there. Drop it from the --relaunch usage line and state where the flag applies. * no-mistakes(review): Clarify lock-wait docs; simplify bash-3.2-safe spawn_task helper * no-mistakes(ci): Fixed ci-1 (Greptile P2). In tests/fm-backend-herdr-presentation-e2e.test.sh, the failure cleanup `cleanup_all` stopped only `LOCK_CONTENTION_OWNER_PID`. It now also stops `LOCK_REFUSE_HOLDER_PID` and `LOCK_WAIT_HOLDER_PID`, the holders of the two new contention cases, so a `fail` before their explicit `wait` no longer leaves them running. Both new PIDs are initialised empty next to the existing one, and each is cleared right after its successful `wait` so cleanup never touches a finished PID. I changed nothing else. `bash -n` passes. The real Herdr e2e run passed both new cases ("default resumed identity refuses session lock contention" and "--herdr-resume-lock-wait waits out session lock contention instead of refusing"). The full run hit my 550s timeout in a later, unrelated case, after the new cases passed
….2 (kunchenguid#6762) * fix(bin): let TERM stop a watcher blocked in a pane capture on bash 3.2 Stock macOS bash 3.2 holds a HUP or TERM until a running command substitution's child exits, and the watcher read every pane through $(fm_backend_capture ...). A blocked backend read therefore held the watcher's stop for as long as the read lasted, and a stopped watcher left the hung read orphaned. tests/fm-watch-triage.test.sh test_term_stops_a_watcher_blocked_inside_a_poll failed on /bin/bash 3.2 for this reason while passing on bash 5. Pane captures now go through watcher_capture, which runs the read as a waited background process group recorded like a check's, so the stop is honored at once and watcher_cleanup stops a read still in flight along with its per-call output file. * no-mistakes(review): Run drain-ring idle capture in watcher shell, add regression test * no-mistakes(document): Document watcher TERM handling for blocked checks and captures * no-mistakes(test): Silence bash 3.2 setpgid race noise from watcher captures * fix(bin): verify the capture group and scope the stop claim to pane reads watcher_capture now confirms its background read leads its own process group, as run_check_capture already does, so watcher_cleanup never relies on a group that set -m failed to create. The comment and continuity doc now say only fm_backend_capture pane reads go through watcher_capture; agent-state and composer-state reads still run inside command substitutions.
* feat(spawn): add per-home worker tool exclusions Add an optional per-home config/crew-exclude-tools file listing tool names to hide from workers, one per line, with blank lines and # comments allowed. It applies to every ship and scout launch and relaunch in that home, is never inherited by another home, and does not affect secondmate agents. Pi and pi-signed apply it through --exclude-tools, which also covers MCP tool names. Any other runtime, and a raw launch command, refuses the launch when the list is non-empty rather than ignoring it. Malformed entries are refused before provisioning, and before a relaunch stops a running worker. Exclusions that match no tool in the worker's loaded registry are reported as unverified warnings in its status record instead of refusing the worker. Closes kunchenguid#6744 * no-mistakes(review): Preserve UTF-8 exclusion paths and verify Pi lifecycle behavior * no-mistakes(document): Clarify worker tool exclusion documentation * no-mistakes(ci): Fixed ci-1 in bin/fm-exclude-tools-lib.sh: a failed read now returns an error before printing names, so all shared launch and relaunch callers refuse rather than silently dropping exclusions. Added deterministic regression coverage for a file disappearing after readability checks across Pi/pi-signed ship and scout launches. Reproduced the original failure; verified 83 spawn checks, 77 relaunch checks, direct parser/runtime failure cases, full targeted lint, Bash syntax, and git diff --check. Relaunch tests passed with existing fixture-cleanup permission warnings. ci-2 remains unchanged per the user's decision; the outer executor owns the fresh CI run
kunchenguid#5343) * refactor(bin): share the local Firstmate home walk from the wake library Teardown's walk over the root home and its registered local secondmate homes moves into bin/fm-wake-lib.sh as fm_local_firstmate_state_dirs, next to fm_firstmate_root_home, so a second consumer can count task records across this machine's homes without a copy. Teardown keeps its exact refusal wording through a thin wrapper. * feat(bin): defer spawns beyond a project's declared machine capacity A project whose machine-local resource only serves a few workers at once had no way to tell Firstmate so: every queued item was launched, and the surplus workers spent full-context turns retrying the resource. config/project-capacity in the root home now declares how many workers each named project admits at once on this machine. bin/fm-spawn.sh counts the ship and scout records on the same project origin across the root and its local secondmate homes, skipping ones whose ready PR is recorded, while holding the shared project lock through publication. A spawn with every place held exits 75 before any brief render, endpoint, worktree, record, or backlog move, so the item stays queued; batches report it as deferred. Undeclared projects keep today's uncapped dispatch, and an unreadable declaration refuses rather than guessing the limit. Refs kunchenguid#4237 * no-mistakes(review): Document that capacity matches the clone directory name * no-mistakes(document): Rewrap stale fm-wake-lib root-home doc comment * no-mistakes(review): Dedupe local state dirs by identity to avoid double-counting * no-mistakes(document): Rewrap fm_local_firstmate_state_dirs error doc comment * no-mistakes(ci): I fixed all four Greptile findings. All 14 tests in tests/fm-project-capacity.test.sh pass, and shellcheck at warning level is clean on the changed files. Each new test failed against the old code and passes now. - **ci-1 (spaced names):** a declaration line must give a name its capacity whenever the name is a valid clone directory name. `fm_project_capacity_lookup` now trims each line, skips blank lines and lines whose first non-blank character is `#`, and takes the last field as the capacity. Everything before that field is the name, so it may contain spaces. The old error cases still refuse: a single field is rejected, and trailing text leaves a last field that is not an integer. The library header and docs/configuration.md now say a name starting with `#` cannot be declared. New test `test_spaced_project_name_is_declared` declares `my heavy project 1` next to an indented comment line and gets a deferral. - **ci-2 (unreadable records):** the holder count must never silently leave out a holder. `fm_project_capacity_occupants` now refuses when a local home's state directory exists but cannot be read or listed, or when a `.meta` file cannot be read. The error names the path, and `fm-spawn.sh` shows it in its existing refusal message. New test `test_unreadable_holders_refuse_admission` covers an unreadable record in the root home and an unreadable state directory in a registered local secondmate home, then checks that the spawn is admitted once both are readable. The test is skipped when run as root. - **ci-3 (Orca lock):** any spawn that can become a holder for a capped project must take that project's lock. The lookup now also reports whether the declaration caps any project at all, and an Orca spawn takes the per-origin lock whenever it does. This covers every capped same-origin clone. It also covers some cases where no same-origin clone is capped, because a spawn cannot find clones under other directory names without searching for them. With no declaration file, Orca still skips the lock. The comments in the library and in the `fm-spawn.sh` header are updated. The Orca test now clones the origin as `project-2`, which has no declaration, and checks that its Orca spawn refuses while the lock is held and publishes no record. - **ci-4 (worktrees):** `assert_nothing_created` now also compares the project's `git worktree list` from before and after a deferred spawn. Both tests that call it take that snapshot first. Files changed: bin/fm-project-capacity-lib.sh, bin/fm-spawn.sh, docs/configuration.md, tests/fm-project-capacity.test.sh * fix(bin): declare capacity for a project name that begins with # A clone directory whose name begins with # was skipped as a comment, so that project stayed uncapped. A line is a declaration when the # is written against the rest of the name and the line ends with a capacity; a # followed by whitespace stays a comment. * no-mistakes(document): Rewrap project-capacity library header comment * no-mistakes(ci): Lint 2 fails because this PR's code pushes ShellCheck past its memory cap. ShellCheck ran out of memory analyzing bin/fm-teardown.sh in CI (reason=memory, rc=251, peak about 8.39 GB). On current main the same file passes at about 7.29 GB. **Cause:** the new `fm_local_firstmate_state_dirs` function in bin/fm-wake-lib.sh had a conditional `. fm-secondmate-registry-lib.sh` with a `# shellcheck source=` directive inside the function. ShellCheck followed that source again, inside a function scope, wherever fm-wake-lib.sh is sourced, and bin/fm-teardown.sh is the heaviest root that sources it. Measured locally with `shellcheck --norc --external-sources bin/fm-teardown.sh`: - current main (fd325b1): 7.29 GB - main merged with this PR: 7.86 GB - the same merge without the in-function source: 7.27 GB **Rule this restores:** this change must not make any lint root heavier than it is on main. That function holds the only new nested source in the change. **Fix:** I removed the in-function source, which no caller needs. Both callers already load the registry library at top level before calling the function: - bin/fm-teardown.sh sources it directly. - bin/fm-spawn.sh, the only user of bin/fm-project-capacity-lib.sh, gets it through bin/fm-ff-lib.sh. I also documented the requirement in the function's comment and in the "Requires" note in bin/fm-project-capacity-lib.sh. No behaviour changes. **Verification:** - ShellCheck on head: bin/fm-teardown.sh peaks at 7.12 GB and bin/fm-spawn.sh at 6.68 GB, both with rc=0. bin/fm-wake-lib.sh and bin/fm-project-capacity-lib.sh lint clean. - tests/fm-project-capacity.test.sh, tests/fm-teardown.test.sh (102 ok) and tests/fm-teardown-endpoint-safety.test.sh all pass. Files changed: bin/fm-wake-lib.sh, bin/fm-project-capacity-lib.sh * fix(bin): release the Herdr session lock when reclaim finishes A concurrent resume in another home waits five seconds for that lock. Reclaim is the last presentation change on the recovery path, so holding the lock through the launch tail made the waiter time out. The contributions arm check also freezes its one-second clock, the same way the budget tests do, because an unfrozen clock can tick past before the first forge read. * no-mistakes(review): Keep Herdr session lock through launch handoff after reclaim * no-mistakes(review): Skip the spawning task's own record in capacity count * no-mistakes(review): Restore release test comment above its test * docs: scope PR-ready re-evaluation to a declared project capacity A ready pull request frees a place only when that project declares capacity, so the always-loaded backlog contract should re-evaluate on that handoff only in that case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
go sync
Context: the scheduled sync that merges upstream/main (kunchenguid/firstmate) into the fork's main (RajeshRajendiran/firstmate) stopped with a merge conflict on 2026-10-07 at fork main 2268b71. Upstream added 47aff86 (per-home worker tool exclusions, kunchenguid#6750), 0f34fab (TERM stops a watcher blocked in a pane capture on bash 3.2, kunchenguid#6762) and 23e71b3 (opt-in --herdr-resume-lock-wait to fm-spawn, kunchenguid#6649). Conflicting files: bin/fm-spawn.sh and bin/fm-watch.sh. The fork still carries an older local pipeline fix (commit 5796796, "extended only Herdr recovery lock waits") touching these files; upstream kunchenguid#6649 now ships its own version of the same Herdr resume lock wait.
What Changed
Risk Assessment
✅ Low: The functional changes and prior fixes are well-bounded; the only remaining concern is a harmless duplicated test setup block.
Testing
Ran the focused real-Herdr presentation E2E suite on Herdr 0.9.3. Concurrent opted-in recoveries serialized successfully, default recovery refused lock contention, and
--herdr-resume-lock-waitwaited through a 30-second lock hold before reclaiming the exact husk without focus drift; guarded teardown preserved the default session.--herdr-resume-lock-waitwaits out sustained lock contention and then succeedsEvidence: Herdr resume-lock-wait live scenario evidence
Source: Herdr resume-lock-wait live scenario evidence
Merge requirement
This is a fork-sync merge and must be merged with a merge commit, never squashed.
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-project-capacity-lib.sh:159- Capacity admission can fail open on read errors.fm_project_capacity_lookupunconditionally returns success after the redirected loop, so ifproject-capacitybecomes unreadable or disappears after the checks at lines 105-110, the failed open leaves capacity empty and the spawn proceeds uncapped. The same invariant is vulnerable when occupant fieldskind,pr, andprojectare read through the error-suppressingfm_meta_getat lines 201-204, and when a registry read failure at bin/fm-wake-lib.sh:1506-1526 is overwritten by a later sibling-home iteration. A configured cap can therefore admit an extra worker without error. Capture each file into a validated snapshot or explicitly propagate every open/read failure before admission.bin/fm-watch.sh:2491- The merge regresses the fork's cleanup ordering by movingtrap watcher_cleanup EXITto line 2603, well after the singleton lock is acquired. A HUP or TERM during lines 2491-2602—particularly while the recovery-marker operations at lines 2496 and 2501 wait—exits with no cleanup trap, leaving.watch.lockbehind and skipping downtime publication. Install the cleanup trap immediately after successful singleton acquisition, retaining the new capture-output cleanup.bin/fm-project-capacity-lib.sh:150- Duplicate-name detection interpolates unrestricted project names into a shell glob over a|-delimited string. Valid clone names containing|,*,?, or[can collide or act as patterns; for example, declarations fora|bandbare incorrectly rejected as duplicates. Compare names as exact array elements instead of encoding them into a pattern string.🔧 Fix applied.
1 warning still open:
tests/fm-contributions.test.sh:1027- The merge introduced a second clock-freeze comment and/bin/datewrite immediately after the existing identical setup at lines 1023-1025. No sync requirement needs this parallel copy; remove the later block.🔧 Fix applied.
1 warning still open:
tests/fm-contributions.test.sh:1027- The merge introduced a second clock-freeze comment and/bin/datewrite immediately after the existing identical setup at lines 1023-1025. No sync requirement needs this parallel copy; remove the later block.🔧 **Test** - 2 issues found → no changes applied ✅
git -C ~/.no-mistakes/worktrees/37339719b87e/01M4C9XCG7KZEAXE8SHZG3FZ09 statusandgit -C ~/.no-mistakes/worktrees/37339719b87e/01M4C9XCG7KZEAXE8SHZG3FZ09 diff). Respond with fix to validate it, or abort.🔧 No changes applied.
✅ Re-checked - no issues remain.
--herdr-resume-lock-waitwaits out sustained lock contention and then succeedstests/fm-backend-herdr-presentation-e2e.test.sh 2>&1 | tee ~/.no-mistakes/evidence/01M4C9XCG7KZEAXE8SHZG3FZ09/herdr-presentation-e2e.logVerified guarded lab teardown reported the default-session tripwire intact and removed the transient lab directory.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.