feat(bin): own Codex app-server worker lifecycle

[coreldh]

Intent

Make Firstmate the owning client of codex app-server so Codex worker turn state comes from the protocol rather than pane guessing. Own one foreground child process group with bidirectional protocol pipes and bounded stderr; drive initialize, thread, and turn lifecycle; require terminal protocol status and child process result to agree before success; use an explicit deadline as the only hang detector; signal only the exact owned process group; publish concrete fail-closed evidence through the single busy-state resolver; and clean up through teardown. Incorporate the independent adversarial gate finding bound to e0f7bac: observe the child exit/reap event, never signal a reaped or recycled numeric process-group id, and resolve without waiting for inherited pipe closure. Add a regression that safely audits and fails on any signal attempt after reap. Also make the version gate inspect FM_CODEX_BIN, call Codex's identifier a thread id rather than session id, and raise the suite default deadline above one second. Preserve maintainer merge authority, never broadly kill processes, and do not add or drive Herdr lifecycle behavior.

What Changed

• Add a Firstmate-owned Codex app-server client that drives validated initialize, thread, and turn protocol handshakes with bounded stderr.
• Wire Codex spawning, busy-state publication, terminal receipts, and teardown through the version-gated client, requiring protocol completion and clean child exit to agree while limiting escalation to the owned process group before reap.
• Document the lifecycle contract and add deterministic and live coverage for startup failures, interrupts, deadlines, protocol/process mismatches, and inherited-pipe signaling regressions.

Risk Assessment

⚠️ Medium: Captain, no remaining source-verifiable defect was found, but the change is a substantial protocol and process-group lifecycle replacement whose execution evidence remains deferred to the test phase.

Testing

Verified the exact target checkout, portable protocol/process lifecycle and post-reap signal audit, busy-state/version/wiring/control integrations, and real Codex 0.147.0 success, failed-terminal, interrupt, and deadline/reap paths; a manual turn produced matching streamed output, terminal receipt, clean child exit, thread terminology, and idle resolver state. The directly relevant teardown cleanup passed, while one unrelated pre-existing Herdr fixture case failed for the isolated test-harness reason reported above.

Evidence: Live app-server E2E guard

Real Codex 0.147.0 success, failed-terminal, interrupt, and timeout/reap controls.

codex-cli 0.147.0
ok - real app-server success joined completed terminal with clean process exit
ok - real app-server failed terminal remained failure after clean process exit
ok - real app-server interruption used turn/interrupt and observed its terminal
ok - real app-server deadline recorded timeout and reaped its owned process group
ok - codex-cli 0.147.0 live owning-client guard passed all individual controls

Evidence: Manual real-Codex success transcript

APPSERVER_PROTOCOL_EVIDENCE; Codex turn success: turn-completed; Codex thread id printed.

�[2K› 
�[2KAPPSERVER_PROTOCOL_EVIDENCE
[Codex turn success: turn-completed]

Codex thread: 01a00960-9442-7880-9526-171501392258

Evidence: Persisted joint protocol/process receipt

outcome=success, terminal=completed, child_exit=0, child_signal=none, with concrete thread and turn ids.

v1 gen=g1786863718.14681.16064 outcome=success event=turn-completed thread=01a00960-9442-7880-9526-171501392258 turn=01a00960-953c-77a0-ae67-37020f02ae98 terminal=completed terminal_error=none child_pid=14794 child_exit=0 child_signal=none started=1786863718 ended=1786863725 deadline=1786863779 stderr_log=codex-appserver.stderr.log

Evidence: Persisted busy-state resolver result

state=idle source=codex-appserver event=turn-completed deadline=none

v1 gen=g1786863718.14681.16064 seq=3 state=idle source=codex-appserver event=turn-completed ts=1786863725 deadline=none

- Outcome: ⚠️ 1 warning across 1 run (5m17s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (3) ✅

• 🚨 bin/fm-codex-appserver-client.mjs:627 - The required criterion says “use an explicit deadline as the only hang detector,” but this adds an independent 30-second startupTimer that terminates a child stalled during initialize/thread setup, while the explicit turn deadline is not created until after that setup. Captain, decide whether this separate startup timeout is authorized; otherwise establish the authoritative absolute deadline when the child is spawned and use its remaining budget across initialize, thread, and turn lifecycle.

🔧 Fix: Honor captain-authorized bounded startup timeout
1 error still open:

• 🚨 bin/fm-codex-appserver-client.mjs:441 - turnId is set by an unsolicited turn/started notification, so it is not proof that turn/start returned successfully. If that notification arrives and the response is withheld, this guard disables the startup timeout while no turn deadline has been installed, leaving the client hung indefinitely; if terminal notifications and a clean exit follow, the current success checks can also accept the turn without the required successful handshake. Track an explicit startup-complete state only after validating the turn/start response, gate busy publication/success on it, and extend the startup regression with an early notification plus withheld response.

🔧 Fix: Require validated handshake before turn activation
1 error still open:

• 🚨 bin/fm-codex-appserver-client.mjs:444 - startupTimeout() unconditionally replaces an earlier concrete failure. If malformed protocol input triggers protocolFailure() shortly before the startup deadline and the child remains alive through the grace period, this timer changes protocol-invalid-json into startup-timeout, so the receipt no longer reports the actual give-up path. Preserve the first recorded failure while still running startup escalation.

🔧 Fix: Preserve initial startup failure through timeout
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 warning

• ⚠️ tests/fm-teardown.test.sh:1565 - The broader teardown script has a pre-existing unrelated fixture defect: its missing-adapter case executes a copied teardown while setting FM_ROOT_OVERRIDE to the original repository, making the supposedly removed Herdr adapter available. The target only changes Codex receipt removal in teardown, and that directly relevant case passed. This unrelated Herdr test was left unchanged to preserve the explicit no-Herdr scope.
• git rev-parse HEAD, git status --porcelain=v1, and target diff inspection
• bash tests/fm-codex-appserver-client.test.sh
• bash tests/fm-busy-state.test.sh
• bash tests/fm-busy-adapter-wiring.test.sh
• bash tests/fm-control.test.sh
• bash tests/fm-teardown.test.sh
• FM_CODEX_LIVENESS_LIVE_E2E=1 bash tests/fm-codex-liveness-live-e2e.test.sh using installed codex-cli 0.147.0
• Manual real-Codex one-shot invocation of bin/fm-codex-appserver-client.mjs, followed by inspection of its streamed output, terminal receipt, busy-state record, and child exit result
• Verified /tmp/fm-codex-appserver-evidence-success was removed and git status --porcelain=v1 remained empty

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[coreldh]

Closing. This builds on #2441's hook-based approach, and on Codex versions outside its gate it makes crewmate spawn refuse outright, which is what the red portable shards show. The underlying Codex busy-state gap remains tracked in #2374 and #2474.