Skip to content

fix(ci): slice-4 ejections — kernel mock-proc memory leak + O(n^2) launchctl pattern (t_b303235e) - #1183

Merged
Kyzcreig merged 2 commits into
mainfrom
fix/slice4-kernel-mock-leak-launchctl-quadratic
Sep 26, 2026
Merged

Kyzcreig merged 2 commits into
mainfrom
fix/slice4-kernel-mock-leak-launchctl-quadratic

Conversation

@Kyzcreig

@Kyzcreig Kyzcreig commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Card t_b303235e. Both causes of the 3 MQ ejections on 'Run tests slice 4/8' (36179091722, 36180608091, 36204054240).

  1. Runs 1-2 (runner shutdown signal, exit 143): the only unfinished file was tests/tools/test_local_env_blocklist.py. Its execute_code test handed a MagicMock proc to the session kernel. _stderr_reader then spins forever, because MagicMock read1() is never b"". With the reader race forced in a linux py3.11 container, RSS reached 45 GB in 9 min, so hosted runners hit OOM. The test now aborts the fake spawn right after capturing the env, the same way TestPythonPathComposition in test_code_execution_modes does.
  2. Run 3 (per-file ceiling): test_benign_segment_scaling_benchmark took 170s on an idle container. The unanchored double-lookahead launchctl DANGEROUS_PATTERN was O(n^2). Anchoring it with \A leaves detection unchanged, since a match at any offset implies one at offset 0. Runtime is now 1.36s.

Verified in the container (uv sync --locked, py3.11): both files ran 20/20 green in isolation, 10 of those with the reader race forced. test_approval + test_execute_code_surfaces_blocks + test_execution_flag_detection: 197 passed. launchctl positive/negative probes behave the same as before.

Not a regression from the 18:11-18:35Z landings. Both defects predate them (#643, 2026-08-31, and 3c8d3ec, 2026-06-26). The 3700d36 LPT change only moved which files ran together.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…unchctl pattern (t_b303235e)

Two independent causes of 'Run tests slice 4/8' failing in MQ groups:

1. r-1060-139 / r-1078-139 (runner "shutdown signal", exit 143): the only
   unfinished file was tests/tools/test_local_env_blocklist.py. Its
   execute_code composition test returned a MagicMock proc into the
   always-on session kernel; _stderr_reader's read1() never returns b"",
   so the daemon thread spins forever recording mock calls. Reproduced in
   a linux py3.11 container with the stderr reader delayed 2s: 45 GB RSS
   after 9 min, main thread parked until the 300s cell deadline -> OOM on
   hosted runners. Fix: abort the fake spawn after env capture (same
   pattern as test_code_execution_modes.TestPythonPathComposition).

2. r-1078-19e (per-file ceiling): test_benign_segment_scaling_benchmark
   took 170s on an idle container. cProfile: 98% in one DANGEROUS_PATTERNS
   entry, the unanchored double-lookahead launchctl pattern, re-scanned at
   every offset (O(n^2)). Anchor with \A — a match at any offset implies
   one at offset 0, so detection is unchanged.

Verified (container, py3.11, uv sync --locked): both files 20/20 green in
isolation (10 with forced reader race); benchmark 170s -> 1.36s;
test_approval + test_execute_code_surfaces_blocks + flag detection
197 passed; launchctl positive/negative cases unchanged.
@ang-fleet-lander

Copy link
Copy Markdown

🤖 merged-by: daedalus · lane: unspecified · gate: BYPASS: FleetReview paused (FR-paused bypass per card t_b303235e) · why: CI green on 504aeaa (all 16 slices + All required checks). Test-only change plus \A anchor on one regex. Fixes the slice-4 MQ ejections.

@ang-fleet-lander
ang-fleet-lander Bot added this pull request to the merge queue Sep 26, 2026
@Kyzcreig
Kyzcreig removed this pull request from the merge queue due to a manual request Sep 26, 2026
@Kyzcreig
Kyzcreig added this pull request to the merge queue Sep 26, 2026
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

Apollo 19:0x PT (Ace 19:02 'do all of it'): jumped to the front of the merge queue. #1183 fixes the slice-4/8 ejector class (kernel MagicMock leak OOM + O(n^2) launchctl regex) that ejected 3 groups in 5h; #1060/#918 are the last PRs of the 09-25 boil-ocean sweep. Batched in one jump so in-flight groups restart once.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 26, 2026
@Kyzcreig
Kyzcreig added this pull request to the merge queue Sep 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 26, 2026
@ang-fleet-lander

Copy link
Copy Markdown

🤖 merged-by: apollo · lane: boil-ocean · gate: BYPASS: FR PAUSED by Ace ruling 2026-09-22 (state/fleetreview-pause-20260922.md); Apollo-reviewed lands via bypass · why: re-enqueue #3 after update-branch (head a8c3c3d): twice removed from the queue with no group CI run and no red; slice-4/8 ejector fix (t_b303235e)

@blacksmith-sh

blacksmith-sh Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
TestPinTransition/test_cache_busting_signature_reflects_pin_peer_name View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

🤖 merged-by: apollo · lane: unspecified · gate: BYPASS: FleetReview daily budget exhausted (spent=600 paged=true), no terminal record possible today; Ace directed admin-merge · why: ROOT-CAUSE fix for merge-queue ejections: anchors O(n^2) launchctl regex in tools/approval.py; test_execution_flag_detection.py 104s->23s locally (main vs branch), 6/6 sampled MQ ejections in 12h were this file timing out. Ace: admin-merge the fixes (21:06 PT) · red-ci allowed [All required checks pass,Python tests / Tests complete,Python tests / Run tests slice 13/16]: slice 13/16 red = tests/honcho_plugin/test_pin_peer_name.py, not in diff (2 files: tools/approval.py, tests/tools/test_local_env_blocklist.py); 25/25 green 3x on branch + on clean fork/main hermetic HOME; slice-order pollution

@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

🤖 merged-by: apollo · lane: unspecified · gate: BYPASS: FleetReview daily budget exhausted (spent=600 paged=true), no terminal record possible today; Ace directed admin-merge · why: ROOT-CAUSE fix for merge-queue ejections: anchors O(n^2) launchctl regex in tools/approval.py; test_execution_flag_detection.py 104s->23s locally (main vs branch), 6/6 sampled MQ ejections in 12h were this file timing out. Ace: admin-merge the fixes (21:06 PT) · red-ci allowed [All required checks pass,Python tests / Tests complete,Python tests / Run tests slice 13/16]: slice 13/16 red = tests/honcho_plugin/test_pin_peer_name.py, not in diff; 25/25 green 3x on branch + on clean fork/main hermetic HOME; slice-order pollution

@Kyzcreig
Kyzcreig disabled auto-merge September 26, 2026 04:12
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

🤖 merged-by: apollo · lane: unspecified · gate: BYPASS: FleetReview daily budget exhausted (spent=600 paged=true), no terminal record possible today; Ace directed admin-merge · why: ROOT-CAUSE fix for merge-queue ejections: anchors O(n^2) launchctl regex in tools/approval.py; test_execution_flag_detection.py 104s->23s locally (main vs branch), 6/6 sampled MQ ejections in 12h were this file timing out. Ace: admin-merge the fixes (21:06 PT) · red-ci allowed [All required checks pass,Python tests / Tests complete,Python tests / Run tests slice 13/16]: slice 13/16 red = tests/honcho_plugin/test_pin_peer_name.py, not in diff; 25/25 green 3x on branch + on clean fork/main hermetic HOME; slice-order pollution

@Kyzcreig
Kyzcreig merged commit f94ff55 into main Sep 26, 2026
95 of 101 checks passed
@Kyzcreig
Kyzcreig deleted the fix/slice4-kernel-mock-leak-launchctl-quadratic branch September 26, 2026 04:14
@Kyzcreig Kyzcreig added the fleetreview:post-merge Ask FleetReview to review this MERGED pull (merge commit vs first parent) label Sep 26, 2026
@ang-fleetreview

Copy link
Copy Markdown

FleetReview

Below the merit threshold, review skipped (28 changed lines < 150 and 2 changed files < 6, no hot path touched; skipped by policy).


FleetReview · reviewKind: skipped-by-policy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fleetreview:post-merge Ask FleetReview to review this MERGED pull (merge commit vs first parent)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant