Skip to content

fix(bin): make Codex liveness deadline-aware - #2441

Closed
coreldh wants to merge 3 commits into
kunchenguid:mainfrom
coreldh:fm/c0815-fm-codex-liveness-fix
Closed

coreldh wants to merge 3 commits into
kunchenguid:mainfrom
coreldh:fm/c0815-fm-codex-liveness-fix

Conversation

@coreldh

@coreldh coreldh commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Intent

Repair Firstmate's Codex liveness so installed codex-cli 0.147.0 and strict later stable versions use the empirically verified lifecycle hooks, while older, unreadable, prerelease, or unexpected versions conservatively remain unknown. Pair every opened turn with an absolute deadline because API errors and manual interruptions emit no Stop: UserPromptSubmit opens deadline-bound busy, Stop alone proves successful idle, SessionEnd and deadline expiry surface unknown, and a late Stop must not rewrite an expired turn as success. Generate per-launch inline hooks that compose with existing project hooks and do not hand-edit live busy state. Prove fm-crew-state working, successful completion, and missing-terminal/error/interruption behavior against a real Codex worker, plus portable controls; the local enforcing population was 11 lines and all 11 mutations were killed. Do not build the separately filed app-server owning-client subsystem. The unrelated composer structural-edge assertion is confirmed to fail identically on clean base f1a4af4 and is being repaired separately; do not absorb, weaken, or repair it in this change.

What Changed

  • Enable lifecycle-based Codex liveness for canonical stable codex-cli versions starting at 0.147.0 while leaving older, prerelease, unreadable, and noncanonical versions unverified.
  • Generate composable per-launch UserPromptSubmit, Stop, and SessionEnd hooks, with deadline-bound busy records that surface missing terminals as unknown and prevent late stops from reporting success.
  • Add portable and live-worker coverage for working, successful, API-error, interruption, deadline-expiry, and version-gating behavior, and document the verified hook contract.

Risk Assessment

✅ Low: Captain, the corrective commit now rejects noncanonical numeric components while preserving canonical 0.147.0 and later stable versions, and the complete durable liveness change has no remaining material source-verifiable concerns.

Testing

After confirming the exact base and target, all three focused portable tests passed and a real Codex CLI 0.147.0 worker proved working-to-done success plus deadline-surfaced API-error and manual-interruption behavior; transcripts were captured, temporary state was removed, and the worktree remained clean.

Evidence: Real Codex liveness E2E transcript

codex-cli 0.147.0 ok - real Codex success moved fm-crew-state from working to done ok - real Codex API error omitted Stop and surfaced on its deadline ok - real Codex interruption omitted Stop and surfaced on its deadline

COMMAND: FM_CODEX_LIVENESS_LIVE_E2E=1 tests/fm-codex-liveness-live-e2e.test.sh
codex-cli 0.147.0
ok - real Codex success moved fm-crew-state from working to done
ok - real Codex API error omitted Stop and surfaced on its deadline
ok - real Codex interruption omitted Stop and surfaced on its deadline
ok - codex-cli 0.147.0 live Codex liveness guard passed success, API-error, and interrupt controls
Evidence: Portable liveness controls transcript
COMMAND: tests/fm-busy-state.test.sh
ok - arm mints a gen sidecar and seeds busy fm-spawn at seq=1
ok - apply advances seq under the armed gen and attributes the writing source
ok - firstmate-owned interrupt and recovery events bind to the current gen
ok - apply is refused for a task whose busy contract was never armed
ok - retire waits for the writer lock and cannot remove a new incarnation
ok - retire treats only an absent sidecar as already retired
ok - a late event from a previous incarnation is rejected, record unchanged
ok - a record from a stale incarnation classifies unknown, never idle
ok - a converted adapter with no record classifies unknown, never idle
ok - malformed records classify unknown malformed, never busy or idle
ok - a record with no armed gen sidecar classifies unknown
ok - a record is trusted only by the adapter whose source wrote it
ok - converted adapters never classify busy from rendered footer text
ok - the grok fallback is regex-scoped to grok and classifies only grok tasks
ok - Codex version gating is strict and every verified open turn is deadline-bound
ok - standalone kimi classifies unknown until the live verification gate opens
ok - cursor classifies only from its transcript fold, never rendered text or native state
ok - endpoint death is the only process-level override and yields dead, never busy
ok - herdr's native verdict is trusted for busy only, and records outrank it
ok - record parsing never clobbers the caller's positional parameters, glob setting, or fields
ok - the boolean view reports busy only on an exact busy verdict
all fm-busy-state tests passed

COMMAND: tests/fm-busy-adapter-wiring.test.sh
ok - pi extension reports agent_start busy, settles idle only via ctx.isIdle(), and keeps turn_end a notification
ok - pi extension awaits agent_settled before the next agent_start without a test delay
ok - pi extension events from a superseded incarnation are rejected as stale
ok - kimi and grok install no unverified semantic wiring and classify through their own gates
ok - opencode plugin classifies from session.status, scoped to the latched worker session
ok - claude hooks open on UserPromptSubmit and close on Stop, StopFailure, and SessionEnd
ok - claude hook events from a superseded incarnation are rejected without breaking the hook
ok - Codex wiring is version-gated, inline, generation-bound, and deadline-bound
all fm-busy-adapter-wiring tests passed

COMMAND: tests/fm-crew-state.test.sh
ok - active run-step is authoritative
ok - stale needs-decision over active run is superseded
ok - stale blocked over active run is superseded
ok - genuine parked run is not flagged superseded
ok - scalar gate parked run is not flagged superseded
ok - gate block parked run is not flagged superseded
ok - ci-ready status log beats monitoring run
ok - ci-monitoring run with checks already green surfaces done
ok - top-level ci status uses ci log green marker
ok - terminal no-checks ci-monitor marker surfaces done
ok - base-advance rearm after green stays working
ok - pending no-checks ci-monitor marker stays working
ok - ci-monitoring run with checks not yet green stays working
ok - a fresh issue after an earlier green reading is not masked
ok - stale checks-green status log does not mask CI relapse
ok - ci fixing is not overridden by an earlier green marker
ok - top-level fixing is not overridden by a stale ci running row
ok - top-level fixing is not overridden by a stale done log
ok - terminal passed run is authoritative
ok - terminal failed run is authoritative
ok - cross-branch run is attributed via the real runs list
ok - cross-branch attribution picks the branch's most recent row
ok - coarse run does not probe another branch's ci log
ok - another branch's run is ignored, falls back
ok - no run + a busy semantic record reads working, attributed to its source
ok - fm-crew-state distinguishes Codex working, successful, deadline-expired, and dead states
ok - a converted adapter never reads working from rendered footer text
ok - grok still reads working through its isolated rendered-tail fallback
ok - herdr's native busy verdict reads working with no record present
ok - a mid-tool-call crew stays working because its record outranks herdr's generation state
ok - an idle record with idle agent_status stays not-busy (no regression for a human-blocked agent)
ok - no run + idle pane uses the status-log verb
ok - no run + idle pane parses keyed status syntax
ok - no run + idle pane on a paused: status reports state: paused with its reason
ok - no run + idle pane honors the configured paused verb
ok - a trailing resolved: event does not corrupt state render (idle stays idle)
ok - dead window ignores stale status log
ok - closed pane still reports a terminal run-step
ok - closed pane still reports an active run-step
ok - no timeout command uses perl bound
ok - scout skips the run lookup
ok - torn-down worktree is handled gracefully
ok - missing meta is handled gracefully
ok - crew_is_provably_working absorbs a validating crew found only via the runs-list fallback
ok - crew_is_provably_working still surfaces a genuinely stopped crew (safety property preserved)
ok - usage error exits 2
ok - historical same-branch rewritten head is not attributed as current
ok - active run with valid descendant fix head remains current
ok - local work advanced past run head invalidates attribution
ok - missing run head falls back instead of matching by branch
all fm-crew-state tests passed

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • 🚨 bin/fm-busy-lib.sh:142 - Intent requires “unexpected versions conservatively remain unknown,” but this regex accepts non-canonical versions such as codex-cli 00.147.0 and codex-cli 0.147.00; numeric comparison then enables unverified lifecycle hooks. Require canonical components (0|[1-9][0-9]*) before applying the version floor.

🔧 Fix: Reject noncanonical Codex versions
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • pwd -P; git rev-parse HEAD; git rev-parse --verify f1a4af426d7199c1781bc91ccd143b8e1f732d10; git diff --stat/--name-status f1a4af4..b4b78fe
  • codex --version
  • tests/fm-busy-state.test.sh
  • tests/fm-busy-adapter-wiring.test.sh
  • tests/fm-crew-state.test.sh
  • CODEX_HOME=$PWD/.codex-live-home-gate FM_CODEX_LIVENESS_LIVE_E2E=1 tests/fm-codex-liveness-live-e2e.test.sh
  • Verified the isolated Codex home was removed and git status --short remained empty
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@coreldh

coreldh commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

Closing. The Codex busy-state gap is real and is tracked in #2374, but this approach depends on --dangerously-bypass-hook-trust, which conflicts with the direction settled in #4673 and shipped in #4689 (crewmates launch with --disable hooks). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant