feat: add durable local steering inbox - #68
Merged
Merged
Conversation
Port upstream PR kunchenguid#2856's local steering-inbox design onto the fork while preserving its typed OMP/Hermes delivery semantics, RunPod guards, and local composer API. Add metadata lifecycle serialization, worker acknowledgement scaffolds, watcher re-ring escalation, portable regression coverage, and live Codex/OMP evidence. Keep the remote secondmate inbox leg out of scope for PR B.
dnth
force-pushed
the
fm/fm-adopt-steer-inbox-reframe
branch
from
August 28, 2026 09:18
0ffa21c to
aa5445e
Compare
Document the captain-approved exclusions for the remote secondmate and Orca abort-recovery metadata writers without changing behavior. Point both sites to the dedicated race-verification follow-up.
Document the captain-approved omission of Hermes doorbell serialization from the faithful upstream port and point the ring boundary to its dedicated follow-up.
Revert the over-broad all-installed harness sweep after it exceeded the validation budget and exposed separate adapter compatibility issues. Keep the acceptance guard explicitly scoped to the required live Codex and OMP record-to-ack proof.
This was referenced Aug 30, 2026
This was referenced Aug 31, 2026
This was referenced Sep 1, 2026
dnth
added a commit
that referenced
this pull request
Sep 5, 2026
* fix(spawn): lock the Orca abort-recovery metadata publication The #68 steering-inbox port put ordinary metadata publication, inbox delivery, and teardown under the per-task metadata lifecycle lock but left two writers outside it. This closes the one real gap and pins the other as inert. Orca abort-recovery writer: it runs from spawn's EXIT trap when the Orca worktree was created but its removal failed. An abort before the ordinary publication never took the lifecycle lock, so a relaunch abort could land its recovery record inside a concurrent teardown of the previous incarnation, where teardown's removal destroys the record, or after it, resurrecting a task just retired. The writer now acquires the lifecycle lock when it is not already held and releases it with the ordinary path. Remote-secondmate writer: it publishes under the secondmate registry lock, which the remote teardown path holds across removing the route and retiring the metadata, and its registry re-read under that lock refuses a retired route. Publication therefore precedes retirement or never happens. The code comment and docs/architecture.md now state that ordering instead of deferring it. Tests: tests/fm-backend-orca.test.sh holds the lifecycle lock from a live process and proves the abort-recovery record waits for release, lands, and leaves the lock released; the case fails against the previous fm-spawn.sh. tests/fm-remote-secondmate-lifecycle-e2e.test.sh now asserts a launch after retirement refuses without republishing metadata or reaching the remote host. Claude-Session: https://claude.ai/code/session_01J8TXxHmpLDhL4yoS3pCCoe * no-mistakes(review): Clarified remote publication retirement ordering documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Implement PR A, the LOCAL inbox leg of the durable steering-inbox reframe, as a direct port of upstream kunchenguid/firstmate PR kunchenguid#2856 at commit e46df1a onto this fork, adapting it to the grounded scout report and this fork. The captain-authorized pivot supersedes the abandoned from-scratch implementation: leave those commits behind, do not reapply the prior round-4 fixes, and inherit upstream e46df1a semantics rather than re-deriving them.
Scope is LOCAL leg only. Keep upstream PR kunchenguid#2901 and commit ddf74ef remote secondmate inbox transport as separate PR B. Do not carry bin/fm-omp-send.mjs, PR #60 native bridge, bin/fm-control.sh, or other fork-absent upstream control-plane files.
Add bin/fm-task-inbox-lib.sh as the single owner of durable sequenced state/.inbox records, the constant self-describing doorbell, worker acknowledgement by moving records to handled/, and the watcher re-ring/escalation ladder. Ordinary local task text must publish a record and type no payload bytes to the terminal. Adapt fm_task_inbox_ring to this fork composer API while preserving the captain-approved direct upstream boundary.
In bin/fm-send.sh, classify inbox versus typed planes before OMP or Hermes typed preparation. Close --resolve-key at inbox enqueue. Preserve typed carve-outs for slash commands, Codex dollar invocations, explicit backend targets, and --key. Preserve delivered-no-turn only for the typed plane. Preserve RunPod wake and delivery locks, lease guards, marked secondmate pending-reply behavior, and the current remote typed path.
Port the metadata lifecycle lock seam first: fm_meta_lock_path is shared by the enumerated normal local metadata publication path, inbox enqueue revalidation, and teardown so enqueue cannot race retirement. Include contention coverage. Every generated ship, scout, and secondmate brief must carry receive-and-ack instructions. Teardown must remove the inbox only after existing safety and landing gates.
The captain explicitly accepts the direct upstream kunchenguid#2856 transaction boundary: release the metadata lock immediately after durable inbox record publication. Do not extend that transaction through pending-reply confirmation or --resolve-key closure before teardown. Treat inbox-meta-lock-released-before-delivery-commit as intended accepted upstream behavior, not a code fix.
The captain explicitly excludes two metadata writers from fm_meta_lock_path in this PR: the out-of-scope remote-secondmate writer and the fork-only local Orca abort-recovery writer. Treat metadata-lock-seam-misses-publications as intended accepted-upstream-port behavior, not a code fix. Do not add lock coverage or lock-order surface to either path. Each writer carries a brief inline comment naming the intentional exclusion and follow-up fm-meta-lock-orca-remote-race-verify.
The captain explicitly defers Hermes inbox doorbell serialization: Hermes is a fork-only crewmate/scout adapter with no upstream kunchenguid#2856 equivalent, so adding .hermes-delivery.lock serialization at the shared ring boundary is fork-absent machinery beyond this faithful port. Treat inbox-hermes-doorbell-bypasses-delivery-lock as intended accepted risk, not a code fix. Do not add serialization here. The ring boundary carries a brief note naming the intentional deferral and follow-up fm-inbox-hermes-doorbell-serialize.
The captain-authorized upstream-port pivot supersedes the earlier round-4 decisions and explicitly says not to reapply those findings. Treat the cmux expected-label item as the previously superseded round-4 finding. Accept the upstream kunchenguid#2856 semantics for symlink handling, direct ladder bookkeeping, ring-before-bookkeeping, unbounded spawn/teardown lifecycle lock waits, ambiguous composer ringing, and wake-before-escalation-marker ordering; do not apply the abandoned round-4 fixes.
Add inbox_steer_check to bin/fm-watch.sh once per task per poll before the secondmate idle exemption, derive backend, harness, expected label, and OMP composer context from metadata, and preserve the 30-second PR validation-lock freshness guard already owned by origin/main commit 23dbd0f without reverting or duplicating it.
Keep documentation, skills, and test routing aligned. Required evidence includes focused portable fm-task-inbox and fm-send-inbox suites, adapted send, brief, teardown, pending-reply, secondmate, and watcher suites, pinned ShellCheck, and a live neutral-project record-to-doorbell-to-handled proof for Codex and OMP.
Retain the no-mistakes CI fixes already committed on top: documentation alignment, inbox-aware Hermes fixtures, Herdr acquisition-owned ready-file handoff with portable and real-Herdr coverage, the trace-context fixture adaptation, accepted-boundary documentation alignment, secondmate concurrency synchronization on durable record publication, bounded force-reaping in the AFK injection test, and final verification clarifications.
The captain requires genuine CI verdicts and forbids treating cancelled or incomplete checks as passing. Behavior portable serial 3 was retriggered after an infrastructure cancellation and passed genuinely; all 15 checks were green before this comment-only follow-up.
Open and validate the PR only against the fork repository dnth/firstmate, never kunchenguid/firstmate.
The captain requires the existing spawn and teardown metadata-lock contention regressions to be selected whenever bin/fm-spawn.sh or bin/fm-teardown.sh changes. Keep this minimal and in-scope: the single test-routing owner adds session-bootstrap only for those two source paths, and executable fm-test-run assertions prove both selections. Do not broaden routing or add new behavior.
The live steering-inbox acceptance guard is explicitly scoped to AC5's required Codex and OMP proof, not a universal harness compatibility matrix. FM_SEND_INBOX_LIVE_HARNESSES may narrow diagnostic reruns within those supported recipes. Adapter-specific compatibility remains owned by each harness's existing live guard and named follow-ups. Do not reintroduce the all-installed sweep: it exceeded the 30-minute validation budget and exposed separate Claude, Kimi, and Hermes compatibility issues outside this port, while the required Codex/OMP record-to-doorbell-to-handled proof passes.
Retain the test-quality cleanup already committed on top: trace-context timing, dependency, harness/backend/kind independence, and no-task-prose-read guarantees are proven through executable fixed-entropy, blocking-FIFO, controlled-PATH, and elapsed-time fixtures rather than source parsing; pinned ShellCheck passes.
The captain requires faithful upstream e46df1a immediate ringing after enqueue. Remove the fork-added busy-state precheck and always call fm_task_inbox_ring once immediately after durable publication, even while the task is busy. Do not defer the initial doorbell to the watcher; preserve the rest of the watcher ladder and fork adaptations.
The captain requires the best-effort fm_task_inbox_ring attempt immediately after durable inbox record publication and metadata-lock release, before fallible pending-reply confirmation and before fm_send_close_resolved_keys ... || exit 1. Exactly one immediate doorbell must be attempted for every successful enqueue; a resolve-key closure failure must occur only afterward. This is an in-scope ordering correction, not new machinery.
Firstmate-Validation-Generation: 8984cbcf8a1ae8ad0b45f3d723e8c013
What Changed
Risk Assessment
Testing
Both focused portable suites passed, including immediate busy-task ringing and ring-before-resolve-key-failure ordering. The initial live attempt correctly hit the gate-worktree safety guard; rerunning from a neutral project proved real Codex and OMP workers acted on durable records and moved them into
handled/. No baseline test commands were supplied as already run, and no transient worktree artifacts remained.Evidence: Live Codex and OMP inbox proof
ok - codex (codex-cli 0.149.1): real worker acted on and acknowledged the durable record ok - omp (omp/18.0.4): real worker acted on and acknowledged the durable record ok - live steering-inbox doorbell guard: 2 harnesses verifiedPipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-send-inbox.test.shbash tests/fm-task-inbox.test.shInitial live guard from the gate worktree, confirming the expected lifecycle safety refusalcd /tmp && FM_SEND_INBOX_LIVE_E2E=1 FM_SEND_INBOX_LIVE_TIMEOUT=240 bash <worktree>/tests/fm-send-inbox-doorbell-live-e2e.test.sh✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.