fix(codex): add progress-stall watchdog for keepalive-only Codex hangs

[Kyzcreig]

Problem

chatgpt.com/backend-api/codex has a failure mode where it keeps the
socket alive with periodic keepalive / response.in_progress frames but
never emits a real delta or completes. Both existing Codex watchdogs
are satisfied by these frames:

• the no-byte TTFB detector sees events arriving (_codex_stream_last_event_ts is set), so it never fires;
• the stream-idle detector sees the keepalive gap stay under its threshold, so it never fires.

So the call rides the socket all the way to the blunt wall-clock stale
timeout and burns it on every retry — observed in production: 90s × 6 retries ≈ 9 min of dead time before the fallback model kicks in. A fresh
reconnect to the same backend succeeds in ~1s (verified by direct probe).

This killed a scheduled digest run during a ~35-min Codex blip: 22 consecutive
Non-streaming API call stale for 90s kills, the 40s fast-reconnect cutoff
armed every retry but never won.

Root cause

run_codex_stream reset _codex_stream_last_progress_ts (chat_completion_helpers.py:378) but never set or read it anywhere — a dead, half-wired signal. The intended "forward progress" watchdog was stubbed and never finished.

Fix

Wire up the progress signal and add the watchdog:

1. codex_runtime._on_event now stamps _codex_stream_last_progress_ts only on real forward progress — text/reasoning deltas, function_call frames, response.output_item.done, terminal/error events — distinct from _codex_stream_last_event_ts, which any frame (including keepalives) refreshes.
2. A progress-stall watchdog in interruptible_api_call kills a stream that is emitting frames but making zero forward progress past the fast-reconnect cutoff (default = the no-byte TTFB cutoff, clamped strictly below the stale timer), so the retry loop reconnects in ~1s instead of waiting out the blunt stale timeout. Tunable via HERMES_CODEX_PROGRESS_STALE_TIMEOUT_SECONDS (0 disables). Close reason: codex_progress_stall_kill.

The no-byte (first-byte) and stream-idle paths are unchanged; the new branch only arms once at least one event has arrived, so it never races the no-byte detector.

Verification

• Reproduced the exact prod signature in a harness: a keepalive-only stall that burned 92s before now dies at ~42s with codex_progress_stall_kill.
• Two regression tests added next to the existing watchdog suite:
  • test_progress_stall_kills_keepalive_only_stream — keepalive-only stream is killed fast via the progress path, not the 90s stale path.
  • test_progress_stall_does_not_kill_streaming_progress — a stream making real progress for longer than the fast cutoff is not killed.
• Gold-standard proof: with the impl reverted, test_progress_stall_kills_keepalive_only_stream FAILS (rides to the ~90s stale timer, elapsed ≫ 20s); with the fix it passes at ~42s.
• Full tests/agent/test_codex_ttfb_watchdog.py suite: 13 passed (11 existing + 2 new), on both the fork base and upstream base.

[github-actions]

🔎 Lint report: fix/codex-progress-stall-watchdog vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9955 on HEAD, 9955 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 5159 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.