Repository navigation
fix: reduce process creation in supervision loops - #69
Merged
Merged
Conversation
Read per-row, per-task and per-cycle values with shell builtins instead of command substitutions, and take everything a status signature needs from one stat. Measured with a fork counter on the live fleet: a crew-state read of the whole fleet starts 38% fewer processes, a quiet watcher scan over 41 status logs 94% fewer, and an owner-watchdog tick 66% fewer, with identical output.
…l name when a runner lacks the cached one
…valence, and host rates
…missing uname to the restricted-PATH bearings fixture, fixing the shared portable-serial and stock-Bash failure. Added narrowly documented SC2031 annotations at every affected immediate background-PID capture for ShellCheck 0.11.0’s dynamic printf -v false positive. Updated CI documentation; production logic and supervision measurement evidence remain unchanged. Verification passed: canonical full-analysis lint for all three affected test roots; stock Bash timeout and fleet snapshot/view suites; focused bearings scenarios covering Perl timeout, successful PR enrichment, custom branch mapping, and GitHub failure degradation. Removed the disposable harness. Limitations: the full bearings suite exceeded the local 900-second deadline before reaching the repaired fixture; macOS cannot enforce Linux CI’s required address-space limit, so lint passed with supported local bounds settings
MrGTV-love
force-pushed
the
fm/fm-supervision-fork-budget
branch
from
October 9, 2026 14:03
779ad34 to
eeb642c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I would like all bugs to be fixed so tomorrow can be focused entirely to Vernant and not problems preventning Vernant from getting built.
A recommended architecture should come with proof.
Context (the backlog record for this bug, as filed):
Cut process creation in the supervision loops (repo: firstmate) (kind: ship) (since 2026-10-08)
Cut process creation in fm-watch.sh, fm-crew-state.sh, fm-fleet-snapshot.sh, fm-procevent.sh (owner watchdog loop near bin/fm-procevent.sh:1530-1560) and fm-home-summary-refresh.sh: remove per-line subshells and repeated jq/grep/sed forks. Proof: processes per cycle before and after, byte-identical output, host process-creation rate before and after. Coordinate with in-flight fm-home-summary-refresh-deadline and fm-wake-drain-fast (do not redo their files' already-fixed parts).
Source: data/fm-sandbox-offload-analysis/report.md (2026-10-08). 21-minute sample at load 102 on 18 cores: 0.1% idle, 56% kernel time, ~61% of CPU in sub-second processes, ~2,850 new processes per second, firstmate supervision scripts start 61% of short-lived process sightings (sleeps excluded); 106 of 155 agent sessions idle the whole window holding ~33 GB. Target for the R4 set over one fleet day: 1-min load <= 36 for >= 90% of the day, process creation < 500/s, kernel time < 25%, no new timeout-class failures. Pattern: measure, fix the root, test, measure again (as fm-wake-drain-fast did).
What Changed
Risk Assessment
✅ Low: The hot-loop refactors are bounded, and no source-backed behavioral regression was found; the only finding concerns accidentally committed test scratch files.
Testing
After correcting disposable-fixture setup, live watcher transitions, crew/fleet byte equivalence, summary publication, and watchdog ownership guards passed, with product transcripts and JSON outputs retained. The target helper checks passed while four corresponding base checks failed on their expected subshell budgets, but those checks did not establish a live result for the helper-budget scenario. Previously supplied whole-process and paired host-rate measurements were not independently remeasured.
Evidence: Live watcher transitions and recovery acknowledgements
Source: Live watcher transitions and recovery acknowledgements
Evidence: Crew state and snapshot byte-equivalence transcript
Source: Crew state and snapshot byte-equivalence transcript
Evidence: Published home-summary ledger
Source: Published home-summary ledger
Evidence: Unsafe state-root refusal and runner cleanup
Source: Unsafe state-root refusal and runner cleanup
Evidence: Helper-level subshell-budget before/after comparison
Source: Helper-level subshell-budget before/after comparison
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
tests/fm-supervision-fork-budget.test.sh:64- The intent explicitly requires "processes per cycle before and after, byte-identical output, host process-creation rate before and after." The added counter measures subshell-depth transitions, not all process starts; its consumers at tests/fm-supervision-fork-budget.test.sh:88 (ledger), :322 (signal scan), :351 (window listing), and :378 (watchdog checks) provide helper-level budgets. Commit 3ee9a1a states per-cycle percentage reductions, but the reviewed history and files contain no host process-creation-rate before/after results or reference to that evidence. Supply the required measurement evidence with comparable workload and sampling intervals, or obtain explicit approval to omit that acceptance criterion.🔧 Fix applied.
2 issues (1 error, 1 warning) still open:
tests/fm-supervision-fork-budget.test.sh:64- The intent explicitly requires "processes per cycle before and after, byte-identical output, host process-creation rate before and after." The added counter measures subshell-depth transitions, not all process starts; its consumers at tests/fm-supervision-fork-budget.test.sh:88 (ledger), :322 (signal scan), :351 (window listing), and :378 (watchdog checks) provide helper-level budgets. Commit 3ee9a1a states per-cycle percentage reductions, but the reviewed history and files contain no host process-creation-rate before/after results or reference to that evidence. Supply the required measurement evidence with comparable workload and sampling intervals, or obtain explicit approval to omit that acceptance criterion.tests/fm-supervision-fork-budget.test.sh:6- Round 1's documentation-only fix correctly clarifies the counter and references R1-evidence.md, but commit d097d9d has no message body. This leaves the recorded instruction to "summarize it in the message body" unsatisfied. Update only that commit's message with a concise summary of the supplied before/after process counts, paired host-rate results, and live-home sampling limitation; do not change script logic.🔧 No changes applied.
3 issues (1 error, 2 warnings) still open:
tests/fm-supervision-fork-budget.test.sh:64- The intent explicitly requires "processes per cycle before and after, byte-identical output, host process-creation rate before and after." The added counter measures subshell-depth transitions, not all process starts; its consumers at tests/fm-supervision-fork-budget.test.sh:88 (ledger), :322 (signal scan), :351 (window listing), and :378 (watchdog checks) provide helper-level budgets. Commit 3ee9a1a states per-cycle percentage reductions, but the reviewed history and files contain no host process-creation-rate before/after results or reference to that evidence. Supply the required measurement evidence with comparable workload and sampling intervals, or obtain explicit approval to omit that acceptance criterion.tests/fm-supervision-fork-budget.test.sh:6- Round 1's documentation-only fix correctly clarifies the counter and references R1-evidence.md, but commit d097d9d has no message body. This leaves the recorded instruction to "summarize it in the message body" unsatisfied. Update only that commit's message with a concise summary of the supplied before/after process counts, paired host-rate results, and live-home sampling limitation; do not change script logic.tests/fm-supervision-fork-budget.test.sh:6- The selected Round 2 instruction to put the measured before/after evidence in the commit message body remains unsatisfied: current HEAD d097d9d references R1-evidence.md only in its subject and has no message body. The header clarification is correct, and no script changes are needed. Add the supplied process counts, output-equivalence results, paired host-rate measurements, and live-home sampling limitation to the commit message body. The later pipeline-owned PR-description update is not a blocker in this review phase.🔧 Fix applied.
✅ Re-checked - no issues remain.
fm-timeout-status.SJ3G9X:1- Simplification: uncertified fixer commit 31a39ef adds two generated test scratch files: fm-timeout-status.SJ3G9X:1 contains only an exit status, and .fm-test-process.47950.bL23DQ is an empty process registry. Neither is measurement evidence or a required project component. Remove these two additions, retaining the legitimate fixture and documentation fixes.🔧 Fix applied.
1 warning still open:
fm-timeout-status.SJ3G9X:1- Simplification: uncertified fixer commit 31a39ef adds two generated test scratch files: fm-timeout-status.SJ3G9X:1 contains only an exit status, and .fm-test-process.47950.bL23DQ is an empty process registry. Neither is measurement evidence or a required project component. Remove these two additions, retaining the legitimate fixture and documentation fixes.✅ No issues found.
git diff fd9b8b020201a029880b3980b545f352ae357a39 2ced3143dfe9fb593fbe84ede6dd810daffef0ecto scope the changed runtime behavior.bash tests/fm-supervision-fork-budget.test.shwith a workspace-local TMPDIR; exercised helper budgets, parser equivalence, file-kind signatures, alternate IFS, and multiline process identities.bash bin/fm-watch.shagainst disposable homes: delivered status changes, acknowledged them throughbin/fm-wake-drain.sh, restarted quietly, and detected a same-size status-file replacement.tmux -S .live-supervision/live.socket new-session -d -s fm-lab-supervision -x 120 -y 40to provide an isolated shell endpoint for reader scenarios; no primary harness or production endpoint was used.bash bin/fm-busy-event.sh arm ...and base/targetfm-crew-state.sh <id>across eight representative task records, including repeated metadata keys and an unterminated final line.Base/targetfm-fleet-snapshot.sh --jsonand--secondmate-home-summaryagainst the same frozen home, withFM_SNAPSHOT_NOW=2026-10-09T05:00:00Z.bin/fm-procevent.sh register,reconcile,list, andretirein two isolated homes; changed one state directory from 0700 to 0777 and verified runner/child termination without stopping the other home.dtrace -q -n 'syscall::fork:entry { @starts = count(); }' -c /usr/bin/true; unavailable because DTrace requires additional privileges.Built a workspace-local fork/posix_spawn observer withclang, ad-hoc signed a disposable Bash copy withcodesign, and passed a four-start nested calibration. A comparable eight-task crew-state workload recorded 553 calls on base versus 425 on target with identical output. Experimental snapshot instrumentation failed and was discarded.Confirmed private process-event claims were retired, no owned watchdogs remained, and the private tmux server had stopped; removed.live-supervision, including copied sources, fixtures, temporary outputs, and observer binaries.Live validation:⚠️ inconclusive - 7 of 8 scenarios driven live against the product
bash tests/fm-supervision-fork-budget.test.shgit archive b062a76a97fe371231bb35cc6d6631c7eaeef00c binfollowed by fourFM_TEST_ONLYhelper-budget checks against the extracted base scriptsbin/fm-lab-home.sh create <lab>for isolated watcher, snapshot, and process-event homesFM_HOME=<lab> bin/fm-watch.shwith twelve adopted status logs; appended a failure, acknowledged its recovery generation, and verified quiet rearmingReplaced a status file with equal-length contents at a new inode, then substituted a symlink whose target contained Unicode and a tab; observed and acknowledged both real watcher wakesbin/fm-herdr-lab.sh provision/run/teardown <named-fm-lab-session>for real isolated pane inventorybin/fm-busy-event.sh armand base/targetbin/fm-crew-state.sh <id>comparisons for working, paused, failed, done, missing metadata, and duplicate-key recordsBase/targetbin/fm-fleet-snapshot.sh --jsonand--secondmate-home-summarycomparisons with identical observation-time inputs;bin/fm-home-summary-refresh.shpublication and mode-0600 checksbin/fm-procevent.sh register/ensure-listening/list/retirewith real blocking children in two homes; exercised transient lease loss, physical state-root replacement, and a world-writable rootConfirmed owned runner PIDs were absent and removed lab homes, the baseline source snapshot, claim registry, and worktree scratch directory✅ **Document** - passed
✅ No issues found.
✅ No issues found.
🔧 **Lint** - 1 issue found ✅
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.