Repository navigation
fix: recover stalled omp lane wakes and preserve supervision ownership - #48
Merged
Merged
Conversation
…e loaded markers on the session omp puts a queued user follow-up back into the composer when a run is interrupted, so a watcher wake queued behind a running turn sat there unsubmitted and the lane looked idle with a frozen wake queue. Separately, a working lane's box composer (spinner and elapsed time in the top border) read unknown, so a doorbell or restart request typed into a working lane could not be seen as unsubmitted, and a descendant omp that auto-discovered the same extensions overwrote the turn-end marker with its own soon-dead pid. - omp watch extension: after agent_end, find its own wake in the composer, remove only that text, and send it again (bounded), leaving operator text exactly as typed. - Stalled-loop check: submit a pending composer that holds nothing but watcher wakes with one bare Enter (fm_task_inbox_submit_held_wake), never typing, and keep the parent alarm for any other composer content. - Composer classifier: read omp's working box border (spinner plus elapsed time); the Herdr submit re-reads a pending verdict once before any retry so a stale frame cannot earn an Enter that aborts a running turn. - omp extensions: only the process named in state/.lock records its marker or arms a watcher, and the turn-end guard re-asserts its marker at every turn boundary. - Behavioral tests for each, an opt-in live Herdr guard, and the dated verification entry.
…nd recovery timing
…g contention" This reverts commit 330d855.
…est-run.sh, tests/fm-contributions.test.sh, and docs/fm-test-portable-shards.md. Parallel shard 1 and the coverage guard shared one cause: the new wake-restore test pushed unhinted serial scripts above 15%. Added its observed 51 ms timing from a zero-failure CI shard, explicitly documenting that it measured the opt-in capability skip, not live recovery. Serial shard 5 failed because its generated-check budget fixture raced wall-clock ticks and child startup. Both configured and inherited cases now use a controlled logical deadline and a fixture-local pass-through timeout, preserving assertions that a read occurred and durable records remained unchanged. Production timeout, wake-lock, and drain behavior are unchanged. Verification passed: executable --check-coverage reported complete coverage of 254 tests with 31/212 serial scripts unhinted; focused union/disjointness and parallel-balance regressions passed; configured/inherited budget regressions passed. A 20-second-budget negative control completed observation and failed the unchanged-record assertion as expected. Broader local verification remained limited by timeouts: the full runner suite and exhaustive shard-list regression reached 900 seconds, and a separate hint-bound regression reached 300 seconds. No full-suite green result is claimed. Temporary evidence and invocation files were removed; no pipeline control or push was performed
…d non-omp harnesses
…y hardening and housekeeping
…y-boundary coverage
…air marker ownership
…dentified omp panes
…cumented coverage
…resh-turn retries
Owner
Author
|
This PR supersedes #40. Why it replaces PR 40:
The old branch is left in place. |
This was referenced Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
vs hold up work??!!
why did you hold up the work vs switching models?
Context: the captain said this on 2026-10-05 about the fleet sitting idle; the standing expectation is that work never sits stalled. On 2026-10-06 (19:50-20:55 CDT) Vernant lane supervisors (omp secondmates) d1, d4 (twice), d6 and d7 raised "secondmate wake-loop stalled". Firstmate found the injected watcher wake text ("FIRSTMATE_OP: v1 watcher: FIRSTMATE WATCHER WAKE: stale: ... Run bin/fm-wake-drain.sh first ...") sitting UNSUBMITTED inside each lane's omp box composer;
fm-send <lane> --key Entersubmitted it and each lane then processed the wake. Separately, state/.omp-turnend-extension-loaded pointed at dead omp PIDs in lane homes ~/.treehouse/firstmate-7bab20/{1,4,5,7}/firstmate (and home 2 until it was restarted), while state/.omp-watch-extension-loaded pointed at the live omp, so fm_omp_extension_owns_supervision returned 1. fm-secondmate-restart rewrote both markers for d1 and d3. Related merged work: PR 33 (omp box composer recognition, root cause: omp live-reloads overlay files).What Changed
Risk Assessment
Testing
Installed omp 18.8.1 passed live restoration, draft-preservation, marker-ownership, and submission scenarios on isolated Herdr and tmux setups; the adversarial run correctly rejected ordinary queued delivery as restoration proof. Portable regressions passed after ancestry isolation, while the initial Herdr invocation and later wake-suite portion hit command deadlines. Fixture setup problems were corrected, rendered terminal evidence and logs were retained, and disposable resources were removed.
operator draft kept, followed by the unchanged draft after recovery.TMUX_DOORBELL_HANDLEDresponse after the running command completed.Evidence: Direct wake restoration and unchanged operator draft
Source: Direct wake restoration and unchanged operator draft
Evidence: Successful live Herdr wake guard
Source: Successful live Herdr wake guard
Evidence: Real tmux message processing response
Source: Real tmux message processing response
Evidence: Live tmux composer and submission transcript
Source: Live tmux composer and submission transcript
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
tests/fm-omp-wake-restore-live-e2e.test.sh:253- Both live recovery probes can report success without exercising restored-editor recovery. Concrete sequence: wake_is_queued confirms a busy turn and fresh queued wake at lines 223–242, but the turn completes between that check and Escape. Ordinary follow-up delivery drains the queue and leaves the editor empty, satisfying the clean-wake assertions at lines 250–257 even with recovery disabled. Ordinary delivery also preserves the operator draft, satisfying the sibling probe at lines 261–271. The previous run's Round 2 finding was selected for correction, but this proof gap remains in the target. Before either success result, require observation of the fresh wake in a pending composer after Escape, or equivalent direct recovery evidence excluding ordinary queued delivery. If restoration was missed, retry from a fresh busy turn a bounded number of times and then fail. This is a source-derived race, not a live reproduction; the remedy is confined to the live guard.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
operator draft kept, followed by the unchanged draft after recovery.TMUX_DOORBELL_HANDLEDresponse after the running command completed.bash tests/fm-tmux-submit-busy.test.sh— passed submission-confirmation, unavailable-identity, retry, and non-omp compatibility regressions.bash tests/fm-composer-lib.test.sh— passed composer classification and busy-state regressions.bash tests/fm-omp-harness.test.sh— initially encountered contamination from the runner’s real omp ancestry; reran through the disposable ancestry-isolated runnerpython3 .nm-live/omp-regression.py, and the complete suite passed, including preparation-cancellation retention and marker ownership.bash tests/fm-backend-herdr.test.sh— first invocation exceeded its deadline; the subsequent invocation through.nm-live/herdr-selected.shexecuted the full Herdr suite and completed successfully because the attempted selector was ineffective.bash tests/fm-wake-queue.test.sh— exercised the changed secondmate stall and no-input boundaries, but the later portion was interrupted by the shared command deadline; not reported as a complete-suite pass.FM_OMP_WAKE_RESTORE_LIVE=1 FM_OMP_WAKE_RESTORE_LIVE_MODEL=openai-codex/gpt-6.1-sol bash tests/fm-omp-wake-restore-live-e2e.test.sh— ran through the evidence relay with workspace-local disposable fixtures; passed after correcting fixture setup.python3 .nm-live/tmux.py— drove installed omp through a marked lab home and private tmux socket with a 120×40 truecolor terminal; observed busy draft classification, confirmed submission, and the actualTMUX_DOORBELL_HANDLEDresponse.env LIVE_ADVERSARY=1 python3 .nm-live/run.py— delayed each real Escape until ordinary queued delivery finished; verified three fresh attempts followed by the expected no-restoration rejection.Captured and rendered actual terminal frames, retained CLI transcripts and late-Escape agent-state responses, tore down the named labs and private tmux server, and removed disposable drivers and worktree test resources.✅ **Document** - passed
✅ No issues found.
🔧 **Lint** - 1 issue found → auto-fixed ✅
🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Push** - passed
✅ No issues found.