Repository navigation
fix: contain runaway test and lab process trees - #66
Merged
Merged
Conversation
MrGTV-love
force-pushed
the
fm/fm-proc-budget-wrapper
branch
from
October 9, 2026 12:31
5bc54a2 to
244c67c
Compare
MrGTV-love
changed the title
The kernel compares the process limit with the user's whole process count, so one runaway scratch tree (a shim whose basename is itself) filled the table and cost every lane its forks. bin/fm-proc-budget.sh sets the limit to the current count plus a headroom (default 1500) for one command tree only. bin/fm-test-run.sh starts each script under it, bin/fm-live-lab.sh starts every lab process under it, and the crewmate brief scaffolds carry a rule that routes ad-hoc shim, lab, and measurement scripts through it.
…rial 7. Empty host output incorrectly triggered a crash retry before an explicit hand-back failure was classified, allowing the replacement host to park indefinitely. Removed that retry condition while preserving bounded retries for signal exits. Added behavioral coverage for empty/nonempty output with and without a live successor, and updated the supervision documentation. The regression failed before the fix. Afterward, both fm-claude-stop-autoarm and fm-supervision-host-hook suites passed through the budgeted runner with no skips. A targeted real-host smoke also confirmed notification without retry when readiness output was absent. ShellCheck with sourced dependencies and Bash syntax checks passed. Temporary fixtures were removed
MrGTV-love
force-pushed
the
fm/fm-proc-budget-wrapper
branch
from
October 9, 2026 14:37
244c67c to
dab28e7
Compare
MrGTV-love
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I would like all bugs to be fixed so tomorrow can be focused entirely to Vernant and not problems preventning Vernant from getting built.
A recommended architecture should come with proof.
Context (the backlog record for this bug, as filed):
Per-tree process budget for tests, labs and measurement scripts (repo: firstmate) (kind: ship) (since 2026-10-08)
Report section 8.1 (+8.4 optional): add bin/fm-proc-budget.sh -- <command...> that sets ulimit -u to (current user process count + extra, default 1500) for the child tree only. Use it in bin/fm-test-run.sh where each test script command is built, in the lab creators/runners (coordinate with in-flight fm-lab-orphan-reaper, which touches them), and add a one-line rule to the worker brief scaffold: run any shim, lab or measurement script through fm-proc-budget.sh. Proof: the report's leashed reproduction (bomb stopped at +376, no failure outside the tree) as a behavioral test. Optional 8.4: rerun a drain once on exit 128 + fork EAGAIN text + no WAKE_ACK line.
Source: data/fm-fork-eagain-diagnosis/report.md (2026-10-08). Cause of the fleet-wide fork EAGAIN 17:14-17:23: worker fm-board-listener-retire built a self-recursive test shim (basename linked to a shim that calls basename) that piled ~10,300 processes against kern.maxprocperuid 10,666; lanes d3/d5, Main and workers lost drains (macOS bash 3.2 exits 128 on one failed fork). Rate fixes (fork-budget, lint bound, drain-fast) do not stop a pile-up.
What Changed
fm-proc-budget.shto cap child-tree process limits at the current user count plus configurable headroom (default 1500), without widening inherited limits; refuse execution when a budget cannot be set.Risk Assessment
✅ Low: The changes are narrowly scoped, preserve tighter inherited budgets and existing ownership controls, and introduce no substantiated source defect or intent contradiction.
Testing
Live wrapper, recursive-shim, isolated test-runner, generated-brief, and real tmux launch checks passed. The runaway stopped at depth 352 while the outside probe completed 1,059 forks without stderr. Targeted wrapper, fake-Herdr, and auto-arm regressions passed; transcripts, generated-output evidence, and incident diagnostics were retained. No complete repository suite, lint, static analysis, or real Herdr lifecycle ran, and all disposable fixtures were removed.
Evidence: Wrapper limits, propagation, nesting, and refusals
Source: Wrapper limits, propagation, nesting, and refusals
Evidence: Leashed recursive-shim incident and unaffected outside probe
Source: Leashed recursive-shim incident and unaffected outside probe
Evidence: Complete runaway-tree fork diagnostics
Source: Complete runaway-tree fork diagnostics
Evidence: Real test-runner measurement limits
Source: Real test-runner measurement limits
Evidence: Test runner refuses unbudgeted execution
Source: Test runner refuses unbudgeted execution
Evidence: Generated brief rules and execution of their quoted commands
Source: Generated brief rules and execution of their quoted commands
Evidence: Real private tmux budget inheritance and teardown
Source: Real private tmux budget inheritance and teardown
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 7 issues found → auto-fixed ✅
bin/fm-brief.sh- merge conflict rebasing onto origin/mainbin/fm-proc-budget.sh- merge conflict rebasing onto origin/mainbin/fm-test-run.sh- merge conflict rebasing onto origin/maindocs/scripts.md- merge conflict rebasing onto origin/maintests/fm-brief.test.sh- merge conflict rebasing onto origin/maintests/fm-proc-budget.test.sh- merge conflict rebasing onto origin/maintests/fm-test-run.test.sh- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Review** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
Scoped the submitted change withgit diff --statand the targeted base-to-target diff.Measured parent limits usingulimit -S -uandulimit -H -u, and counted real user processes.Drovebin/fm-proc-budget.sh 300 -- bash -c ...to observe sealed limits, preserved PID, intact arguments, and child exit status.Exercised default and nested budgets, attempted limit escalation, rejected malformed extras, and injected a failingpsdependency.Ranbash tests/fm-proc-budget.test.sh, including the real leashed recursive-shim reproduction and concurrent unbudgeted fork probe.Ranbin/fm-test-run.sh --jobs 1 --per-script-timeout-secs 20 --json <fixture-output> <disposable-script>with an isolated real CPU pool; repeated with failingps.Generated briefs usingbin/fm-brief.shfor no-mistakes, direct-PR, local-only, and scout modes; executed their emitted wrapper commands from normal and space/apostrophe-containing roots.Created/tmp/fm-lab-01M4FNQD-u6x17fgrusingbin/fm-lab-home.sh create; loaded the actuallab_runfunction and launched private tmux with-L fm-lab -f /dev/null -x 120 -y 40.Captured two real tmux panes, inspected server/pane process identities, killed only the private server, ranbin/fm-lab-home.sh teardown, removed the lab, and confirmed its processes and directories were gone.Ranbash tests/fm-herdr-lab.test.shagainst its stateful fake Herdr only.Saved command transcripts and generated briefs, then removed all disposable worktree fixtures.✅ No issues found.
command -v bash python3 tmux claude codex omp; caller soft/hard limit baseline and post-execution checks.Directbin/fm-proc-budget.shexecutions covering default and explicit budgets, measured count arithmetic, nested limits, argument quoting, exit status, exec-preserved PID, malformed calls, and denied process-count access./bin/bash tests/fm-proc-budget.test.sh./bin/bash .fm-budget-validation/leashed-reproduction.sh: evidence-producing recursive-shim containment with a concurrent unbudgeted fork probe.Isolated realfm-test-run.shexecutions: serial workload; two concurrent workloads with--jobs 2 --per-script-timeout-secs 30; denied process-count access.test_each_script_runs_under_a_process_budgetfromtests/fm-test-run.test.sh.Realfm-brief.shgeneration and execution of emitted wrapper commands for no-mistakes, direct-PR, local-only, and scout modes, including a helper path containing spaces and an apostrophe.test_crewmate_scaffolds_route_scratch_scripts_through_the_process_budgetandtest_process_budget_command_quotes_foreign_firstmate_pathfromtests/fm-brief.test.sh./bin/bash .fm-budget-validation/tmux-lab-proof.sh: helper-created authorized lab, shippedlab_runfunction, private tmux socket, pane/descendant observations, helper teardown, and removal proof./bin/bash tests/fm-herdr-lab.test.sh: existing fake-Herdr coverage only.test_host_failed_handback_is_not_retriedandtest_host_crash_is_retried_then_reportedfromtests/fm-claude-stop-autoarm.test.sh: mocked regression coverage only.Removed.fm-budget-validationand confirmed the authorized tmux lab path was absent.✅ No issues found.
bash .gate-proc-budget-validation/wrapper-scenarios.sh: default and explicit budgets, descendant inheritance, unchanged parent limits, exec PID preservation, arguments, exit status, nesting, invalid inputs, and unreadable process counts.bash tests/fm-proc-budget.test.sh: targeted wrapper regression, including the leashed recursive shim.bash .gate-proc-budget-validation/incident-live.sh: recursive basename shim alongside an unbudgeted fork probe, with complete diagnostics retained.Ran an unchanged disposable copy ofbin/fm-test-run.sh --jobs 1 --per-script-timeout-secs 15 --json <evidence-path> tests/fm-budget-live.test.shthrough a workspace-local CPU-pass pool; repeated with a failingpsdependency.Executedbin/fm-brief.shfor no-mistakes, direct-PR, local-only, and scout modes; executed each emitted budget command, including paths containing spaces and an apostrophe.bash .gate-proc-budget-validation/tmux-lab-live.sh: minted the authorized lab withfm-lab-home.sh create, launched real tmux through the shippedfm-live-lab.shlaunch function, observed pane and descendant limits, killed the private server, ran helper teardown, and confirmed removal.bash tests/fm-herdr-lab.test.sh: fake-client server/viewer budget and guarded-cleanup regression coverage; no real Herdr calls.bash tests/fm-claude-stop-autoarm.test.sh: supplemental fixture-driven regression coverage, including silent failed hand-backs and crash retries.Confirmed legacy bypass environment variables do not disable budgeting and obsolete--countand--checkmodes exit 2.Removed all newly created worktree fixtures and confirmed the authorized external lab was absent.✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.