fix(codex): healthy app-server sessions survive long post-tool reasoning silence (#112928, salvage #112932)

[teknium1]

A Codex app-server session that goes quiet on the wire after a tool result — because the model is reasoning over a large context — is no longer interrupted and retired after 90s; it now completes normally and is retired only when the subprocess actually dies or the overall turn deadline is hit.

• agent/transports/codex_app_server_session.py::CodexAppServerSession._run_started_turn: the post-tool quiet timer is now observability only. Past post_tool_quiet_timeout it logs one warning per tool result (codex has emitted no events for 90s after a tool result; still waiting (turn deadline 600s)) and keeps polling; it no longer sends turn/interrupt or sets should_retire.
• Retirement keeps its two real signals, both already checked on every poll iteration: _subprocess_died (process/transport gone) and the turn_timeout deadline. A truly wedged codex is still bounded.
• Test: the old monotonic-clock watchdog test becomes the invariant test_post_tool_silence_warns_but_does_not_retire_a_healthy_turn (tool item → silence past the threshold → turn/completed: no interrupt, no retirement, warning logged). Red on base, green on this branch. test_post_tool_watchdog_resets_on_further_activity unchanged.
• No config key or user docs reference the watchdog (grep -rn post_tool_quiet website/docs docs = 0; no non-test caller passes it) — nothing to document.

Live repro (fake app-server client: tool completion, then 0.6s of silence with is_alive() true, post_tool_quiet_timeout=0.2, then turn/completed; plus two controls):

before (origin/main 9796235):

SYMPTOM (healthy, silent 0.6s > quiet 0.2s): interrupted=True should_retire=True error='codex went silent for 0s after a tool result; retiring app-server session.' turn_interrupt_sent=True
CONTROL dead-process: interrupted=True should_retire=True error='codex app-server subprocess exited unexpectedly'
CONTROL turn_timeout (0.20s): retired by the quiet watchdog before the deadline

after:

WARNING codex has emitted no events for 0s after a tool result; still waiting (turn deadline 5s)
SYMPTOM (healthy, silent 0.6s > quiet 0.2s): interrupted=False should_retire=False error=None turn_interrupt_sent=False tool_iterations=1
CONTROL dead-process: interrupted=True should_retire=True error='codex app-server subprocess exited unexpectedly'
CONTROL turn_timeout (0.50s): interrupted=True should_retire=True error='turn timed out after 0.5s'

Tests: scripts/run_tests.sh tests/agent/transports/test_codex_app_server_session.py → 35 passed; test_codex_app_server_runtime.py + test_codex_event_projector.py → 45 passed.

Root cause: the watchdog treated N seconds of post-tool wire silence as proof of a wedged process, but the app-server emits zero events during long reasoning phases while remaining fully responsive (the interrupt was acknowledged in ~25ms), so healthy turns were killed and surfaced as protocol_violation / crashed workers.

Fixes #112928
Salvages #112932 (@KoNit-K) — same direction (silence must not retire); this PR keeps the quiet threshold as a logged warning instead of deleting the timer and the before_poll hook, and keeps the existing reset-on-activity test.

Dropped hunks

• fix(codex): tolerate post-tool app-server silence #112932's removal of the post_tool_quiet_timeout kwarg, the before_poll hook on _drive_turn, and the quiet timer itself — retained as observability so operators can still see long post-tool silences in the log (the reporter's own recommendation).

Infographic

Review follow-up

• [MINOR] stale watchdog tripped comment at agent/codex_runtime.py:493 and the test_post_tool_watchdog_resets_on_further_activity name/docstring — fixed @54623b0: comment now lists only the real retirement causes and notes post-tool silence only warns (Codex app-server sessions are incorrectly retired after 90s of post-tool silence #112928); test renamed test_post_tool_activity_clears_the_quiet_timer_and_never_retires with a docstring on the warning semantics (assertions unchanged; 35/35 file tests pass).
• [MINOR] wedged-but-alive codex (fix(codex-runtime): retire wedged sessions + post-tool watchdog + OAuth refresh classify #25769) now burns the full hardcoded 600s turn_timeout → residual (below); larger than a cheap fix.

Known residuals

• agent/transports/codex_app_server_session.py::_run_started_turn.warn_if_quiet: The case fix(codex-runtime): retire wedged sessions + post-tool watchdog + OAuth refresh classify #25769 (12f755c) deliberately guarded — a codex subprocess that stays alive but never emits (CPU-spin/wedge) — is no longer fast-failed and now burns the full turn_timeout, which the only production caller (agent/codex_runtime.py run_turn(user_input=...)) leaves at the hardcoded, non-configurable 600s default; not fail-open forever (deadline still interrupts+retires), but 90s→600s per wedged turn with no operator knob. Suggested: make turn_timeout configurable/plumbed from codex_runtime, or replace pure-silence retirement with an RPC liveness ping so a genuinely wedged-alive process is still retired early.

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 54623b0 — fix(codex): drop the stale 'watchdog tripped' retirement cau}

⚠️ Warnings

CI timings · View report · View job

Wall time 12m36s vs 6m8s (+105.4%). 5 job(s) slower, 6 faster, 1 unchanged.

• Python tests / Run tests: +17.0s
• Check no case-colliding filenames / check-case-collisions: +7.0s
• OS-specific tests / macOS-only tests: -6.0s
• Check no committed infographics / check-no-committed-infographics: -6.0s
• Python tests / e2e: +4.0s