feat: add opt-in codex app-server probe

[e-jung]

Intent

Implement a contained Codex app-server / Codex-managed-worktree spike for firstmate. Add the smallest safe, opt-in probe/scaffold that can inspect the installed Codex app-server command/schema and run stdio-only app-server basics without touching real captain projects or exposing unauthenticated non-loopback listeners. Document the evidence, GUI visibility uncertainty, metadata needed for codex_thread_id/active turn/worktree/archive/read semantics, Codex-managed worktree lifecycle differences, and recommend when a future backend=codex-app should start ship/scout-only before secondmates. Do not register codex-app as a spawn-capable backend or destabilize Herdr/tmux.

What Changed

• Adds bin/fm-codex-appserver-probe.sh, an opt-in, stdio-only probe that inspects the installed Codex app-server command/schema and exercises basic create/read/archive-thread lifecycle without registering codex-app as a spawn-capable backend or destabilizing the existing tmux/herdr/zellij/orca backends.
• Adds docs/codex-app-backend-spike.md recording the spike evidence and recommendations: metadata gaps for thread/turn/worktree/archive semantics, Codex-managed worktree lifecycle differences, and guidance that a future backend=codex-app should start ship/scout-only before secondmates.
• Adds tests/fm-codex-appserver-probe.test.sh covering dry-run, schema lifecycle, and live stdio paths, and fixes node-on-PATH flakiness in tests/fm-session-start.test.sh.

Risk Assessment

✅ Low: The change is a well-bounded, opt-in, standalone experimental probe with safe dry-run defaults, stdio-only live mode, explicit refusal of unauthenticated non-loopback listeners, matching tests, and no modification to any existing firstmate spawn/backend machinery; round-1 issues are fixed and no new material issues remain.

Testing

Ran the two affected test files (fm-codex-appserver-probe, fm-session-start) and the full safety/non-registration matrix end-to-end against a fake codex CLI: default dry-run, help, non-loopback WS refusal, loopback + stdio-only gating, unsupported-listen refusal, schema generation/summary incl. failure exit-code propagation and missing-method detection, live stdio create/read/archive incl. keep-thread and graceful no-rollout archive degradation, and the fm-spawn.sh --backend codex-app rejection proving codex-app is not registered while tmux/herdr/zellij/orca remain. All targeted tests pass; working tree left clean; no UI surface exists for this CLI-only spike so the captured CLI transcripts are the reviewer-facing evidence.

Evidence: Probe test results (fm-codex-appserver-probe.test.sh)

ok - fm-codex-appserver-probe: default invocation is safe dry-run ok - fm-codex-appserver-probe: refuses unauthenticated non-loopback WebSocket listener ok - fm-codex-appserver-probe: schema generation summarizes lifecycle methods ok - fm-codex-appserver-probe: live stdio probe drives create/read/archive protocol path

ok - fm-codex-appserver-probe: default invocation is safe dry-run
ok - fm-codex-appserver-probe: refuses unauthenticated non-loopback WebSocket listener
ok - fm-codex-appserver-probe: schema generation summarizes lifecycle methods
ok - fm-codex-appserver-probe: live stdio probe drives create/read/archive protocol path

Evidence: Default dry-run + help (end-user experience)

## default invocation mode: dry-run would_create_thread: no (pass --create-thread ...) safety: no listener opened; no Codex thread created; no project worktree modified

########## END-USER DEMO 1: default invocation (safe dry-run) ##########
mode: dry-run
would_check: codex --help
would_generate_schema: codex app-server generate-json-schema --out <dir>
would_live_handshake: codex app-server --stdio
would_create_thread: no (pass --create-thread to create/read a throwaway no-turn thread)
safety: no listener opened; no Codex thread created; no project worktree modified

########## DEMO 2: --help ##########
usage: fm-codex-appserver-probe.sh [options]

Safe defaults:
  With no options this is a dry run. It does not start app-server, create
  threads, expose listeners, or touch project worktrees.

Options:
  --dry-run                 Print the probe plan only (default)
  --schema-dir DIR          Generate the installed app-server JSON schema bundle
                            into DIR and summarize lifecycle methods
  --experimental-schema     Include experimental schema methods/fields
  --live-handshake          Start app-server over stdio and run initialize +
                            thread/list only
  --create-thread           With the live stdio probe, create a thread in a temp
                            cwd and read it; archive is best-effort because a
                            no-turn thread may not have a rollout yet
  --keep-thread             Do not archive a created thread; prints codex:// link
  --cwd DIR                 cwd for --create-thread (default: fresh temp dir)
  --model MODEL             model override for --create-thread
  --listen URL              Transport to validate. Live probe supports stdio://
                            only; ws:// non-loopback requires auth flags.
  --ws-auth MODE            capability-token or signed-bearer-token
  --ws-token-file PATH      capability token file for ws auth
  --ws-token-sha256 HEX     capability token verifier for ws auth
  --ws-shared-secret-file PATH
                            signed bearer shared secret file for ws auth
  -h, --help                Show this help

Examples:
  fm-codex-appserver-probe.sh
  fm-codex-appserver-probe.sh --schema-dir /tmp/fm-codex-schema
  fm-codex-appserver-probe.sh --live-handshake
  fm-codex-appserver-probe.sh --live-handshake --create-thread --keep-thread

Evidence: Safety guards: non-loopback WS refused, loopback+stdio-only, unsupported listen

refusing unauthenticated non-loopback WebSocket listener 'ws://0.0.0.0:4500' (exit 1) live probe currently supports stdio:// only (exit 1) unsupported --listen URL: tcp://example.com:1234 (exit 1)

########## DEMO 3: unsafe non-loopback WS refused (without auth) ##########
error: refusing unauthenticated non-loopback WebSocket listener 'ws://0.0.0.0:4500'
exit=1

########## DEMO 4: loopback WS allowed past the auth gate (still refuses live because stdio-only) ##########
error: live probe currently supports stdio:// only; use codex app-server directly for authenticated WebSocket testing
exit=1

########## DEMO 5: unsupported --listen URL refused ##########
error: unsupported --listen URL: tcp://example.com:1234
exit=1

Evidence: codex-app NOT a spawn-capable backend; Herdr/tmux untouched

error: unknown backend 'codex-app' (known: tmux herdr zellij orca) bin/fm-backend.sh:58:FM_BACKEND_SPAWN="tmux herdr zellij orca"

########## DEMO 6: codex-app is NOT a spawn-capable backend ##########
error: unknown backend 'codex-app' (known: tmux herdr zellij orca)
exit=1

########## DEMO 7: known backends list (should NOT include codex-app) ##########
(no codex-app references in backend dispatcher or backends dir)

########## DEMO 8: implemented/spawn-capable backends (should be tmux/herdr/zellij/orca only) ##########
bin/fm-backend.sh:58:FM_BACKEND_SPAWN="tmux herdr zellij orca"
bin/fm-backend.sh:148:  fm_backend_list_contains "$FM_BACKEND_SPAWN" "$name" && return 0
bin/fm-backend.sh:149:  echo "error: backend '$name' does not support task spawning yet (spawn-supported: $FM_BACKEND_SPAWN)" >&2

Evidence: Schema generation + live create/read/archive + keep-thread output

schema_required_lifecycle: present schema_turn_methods: turn/interrupt,turn/start,turn/steer live_initialize: ok created_thread_id: 00000000-0000-4000-8000-000000000001 codex_thread_url: codex://threads/00000000-0000-4000-8000-000000000001 archived_created_thread: yes

########## END-USER DEMO: schema generation + lifecycle summary ##########
codex_cli: codex-cli 0.142.5-test
schema_dir: /tmp/tmp.X1q19yTxA8/schema-out
schema_method_count: 8
schema_required_lifecycle: present
schema_thread_methods: thread/archive,thread/list,thread/read,thread/start
schema_turn_methods: turn/interrupt,turn/start,turn/steer
exit=0

########## END-USER DEMO: live stdio create/read/archive ##########
codex_cli: codex-cli 0.142.5-test
live_initialize: ok
codex_home: /tmp/fake-codex
live_thread_list: ok count=0
created_thread_id: 00000000-0000-4000-8000-000000000001
codex_thread_url: codex://threads/00000000-0000-4000-8000-000000000001
thread_read_after_create: ok includeTurns=true
archived_created_thread: yes
thread_cwd: /tmp/fm-codex-appserver-probe-QfxL2S
notifications_seen: 
exit=0

########## END-USER DEMO: keep-thread prints codex:// link ##########
codex_cli: codex-cli 0.142.5-test
live_initialize: ok
codex_home: /tmp/fake-codex
live_thread_list: ok count=0
created_thread_id: 00000000-0000-4000-8000-000000000001
codex_thread_url: codex://threads/00000000-0000-4000-8000-000000000001
thread_read_after_create: ok includeTurns=true
archived_created_thread: no
thread_cwd: /tmp/fm-codex-appserver-probe-ExwK3S
notifications_seen: 
exit=0

Evidence: Archive no-rollout graceful degradation

archived_created_thread: unavailable-no-rollout

########## archive fails with 'rollout' -> reported as unavailable-no-rollout ##########
codex_cli: codex-cli 0.142.5-test
live_initialize: ok
codex_home: /tmp/fake-codex
live_thread_list: ok count=0
created_thread_id: abc-rollout-test
codex_thread_url: codex://threads/abc-rollout-test
thread_read_after_create: ok includeTurns=true
archived_created_thread: unavailable-no-rollout
thread_cwd: /tmp/fm-codex-appserver-probe-7BljlQ
notifications_seen: 
exit=0

Evidence: Schema exit-code propagation (gen failure exit 3; missing-method exit 2)

schema gen failed -> exit=3 schema_required_lifecycle: missing thread/list,thread/read,thread/archive,... -> exit=2

########## schema generation failure -> nonzero exit propagated ##########
codex_cli: codex-cli 0.142.5-test
boom: schema gen failed
exit=3

########## schema present but missing a required lifecycle method -> summarize fails nonzero ##########
codex_cli: codex-cli 0.142.5-test
schema_dir: /tmp/tmp.FkZfiN4vM3/schema-out2
schema_method_count: 2
schema_required_lifecycle: missing thread/list,thread/read,thread/archive,turn/start,turn/steer,turn/interrupt
schema_thread_methods: thread/start
schema_turn_methods: 
exit=2

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (30m23s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 3 issues found → auto-fixed ✅

• ⚠️ bin/fm-codex-appserver-probe.sh:351 - Schema-generation failures are masked by exit 0. The script uses only set -u (no set -e), so when codex app-server generate-json-schema (line 351) or summarize_schema (line 352) fails, the exit codes are discarded and execution falls through to the trailing if [ "$LIVE_HANDSHAKE" -eq 1 ] (lines 355-357). When no live handshake runs (or it succeeds), the script exits 0 even though schema generation failed - I confirmed this with a trace: a failing codex+summarize pair yields exit=0. Automated callers/CI relying on the exit code would silently miss schema failures. Propagate the schema command/function exit codes (e.g. capture and return them, or gate the trailing block on success).
• ⚠️ bin/fm-codex-appserver-probe.sh:249 - Graceful-degradation paths hinge on substring-matching external Codex error text: includes("includeTurns") (line 249) and includes("no rollout") (line 260). The spike docs explicitly position this probe as the tool to re-run as Codex versions change, so if Codex rewords these messages the probe will throw instead of reporting the documented unavailable-no-rollout / includeTurns=false fallbacks. Unexpected errors are still surfaced loudly (good), but the known-state handling is version-fragile. Consider matching on error codes if Codex exposes them, lowercasing the comparison, or tolerating any failure for a no-turn thread rather than a literal phrase.
• ℹ️ bin/fm-codex-appserver-probe.sh:54 - The WebSocket loopback/auth validation is advisory only: the live probe unconditionally rejects any non-stdio transport (lines 323, 325), so is_loopback_ws (line 54, whose ws://127.* glob would also accept crafted hosts like ws://127.0.0.1.evil.com) and has_ws_auth_material (which treats a single capability-token field as sufficient) never gate an actual listener. No current security impact, but when a future backend=codex-app wires real ws these loose checks should be tightened (parse the host; require both token + verifier).

🔧 Fix: propagate schema exit codes and harden codex app-server probe error matching
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 tests failed with exit code 1
• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: fix node-on-PATH test flakiness in fm-session-start tests
✅ Re-checked - no issues remain.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• bash tests/fm-codex-appserver-probe.test.sh (4/4 assertions: dry-run default, non-loopback WS refusal, schema lifecycle summary, live stdio create/read/archive)
• bash tests/fm-session-start.test.sh (all session-start composition/ordering/liveness assertions, incl. the lavish-axi MISSING-line determinism fix)
• bash bin/fm-codex-appserver-probe.sh (no args) -> dry-run safety banner
• bash bin/fm-codex-appserver-probe.sh --help -> usage/options/examples
• bash bin/fm-codex-appserver-probe.sh --live-handshake --listen ws://0.0.0.0:4500 -> refuses unauthenticated non-loopback WS (exit 1)
• bash bin/fm-codex-appserver-probe.sh --live-handshake --listen ws://127.0.0.1:4500 -> loopback passes auth gate, still refuses live (stdio-only) (exit 1)
• bash bin/fm-codex-appserver-probe.sh --listen tcp://example.com:1234 -> unsupported listen refused (exit 1)
• schema generation + lifecycle summary against fake codex (8 methods, schema_required_lifecycle: present)
• live stdio create/read/archive against fake codex (live_initialize: ok, created_thread_id, codex_thread_url, archived_created_thread: yes)
• live stdio --create-thread --keep-thread against fake codex (archived_created_thread: no, temp cwd still cleaned up)
• archive-error-with-rollout matching -> archived_created_thread: unavailable-no-rollout (exit 0, graceful degradation)
• schema-generation failure propagates codex exit 3; schema-present-but-missing-lifecycle -> summarize exit 2
• FM_HOME=/tmp/... bash bin/fm-spawn.sh ... --backend codex-app -> rejected as unknown backend (known: tmux herdr zellij orca); grep confirms no codex-app references in bin/fm-backend.sh or bin/backends/
• post-test git status --porcelain clean (no transient artifacts left in worktree)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[kunchenguid]

The probe implementation looks safe to me: it is opt-in, defaults to dry-run, uses stdio for live probing, refuses unauthenticated non-loopback WS, and does not register codex-app as a backend.
Please revise the docs so #222 owns the Codex App backend contract:

1. In docs/codex-app-backend-spike.md, keep only dated app-server probe evidence and how to rerun bin/fm-codex-appserver-probe.sh. Add the exact Codex CLI/Desktop version, exact commands run, and exact relevant output for schema, stdio initialize/thread-list, thread/start/read, and archive/no-rollout behavior.
2. Replace the "Metadata A Backend Would Need", "Codex-Managed Worktrees In Firstmate Terms", and "Roadmap Recommendation" sections with short pointers to docs/codex-app-backend.md after #222 lands. Do not restate the production gate there.
3. Move the staged-adoption rule into docs/codex-app-backend.md in #222: ship/scout-only first; secondmates remain out of scope until ordinary task create/read/send-or-turn-start/interrupt/archive/status/teardown behavior is proven.
4. Keep README.md and docs/scripts.md as one-line pointers to the probe/spike only. They should not carry the backend contract or rollout details.

[kunchenguid]

Thanks for the PR! It looks like this branch has a merge conflict with the base branch right now. When you get a chance, could you rebase onto (or merge in) the latest base branch, resolve the conflict, and push? Once GitHub shows the PR as mergeable again, it'll be picked back up for review.

_{Noted for firstmate#243 at 51ffd763.}

[e-jung]

Closing as not planned because the Codex app-server experiment no longer matches my Orca-based workflow. The branch and investigation remain available as history. Thank you for considering it.