Skip to content

fix: update activity tracker during Codex Responses streaming - #8027

Closed
tomqiaozc wants to merge 1 commit into
NousResearch:mainfrom
tomqiaozc:fix/codex-stream-activity-tracking
Closed

tomqiaozc wants to merge 1 commit into
NousResearch:mainfrom
tomqiaozc:fix/codex-stream-activity-tracking

Conversation

@tomqiaozc

Copy link
Copy Markdown

Summary

  • Adds _touch_activity() calls in _run_codex_stream() and _run_codex_create_stream_fallback() so the cron inactivity tracker stays fresh during active Codex Responses streaming.
  • Without this fix, the activity description stays stale at "starting API call #N" even while stream events are flowing, causing the cron inactivity timeout to kill active jobs.
  • Touch points: stream entry, text deltas, reasoning deltas, and fallback stream loop.

Fixes #7794

Test plan

  • test_codex_stream_updates_activity_tracker — verifies _last_activity_ts and _last_activity_desc are updated during stream
  • test_codex_fallback_stream_updates_activity_tracker — same for the create(stream=True) fallback path
  • Full Codex Responses test suite: 42/42 passed

🤖 Generated with Claude Code

The Codex Responses stream paths (_run_codex_stream and
_run_codex_create_stream_fallback) never called _touch_activity(),
leaving the cron inactivity tracker stale at "starting API call #N"
even while stream events were actively flowing. This caused cron jobs
to be killed by the inactivity timeout during long Codex responses.

Add _touch_activity() calls on stream entry, text deltas, reasoning
deltas, and in the fallback stream loop.

Fixes NousResearch#7794

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@ratacat

ratacat commented Apr 25, 2026

Copy link
Copy Markdown

Supportive +1 — this is a useful fix for the Codex cron timeout family.

Even though our most recent incident looked more like a post-stream/finalization wedge than an active-stream inactivity timeout, keeping _touch_activity() fresh during Codex Responses streaming is still important. Without this, long healthy Codex responses can look idle to cron even while tokens/reasoning are flowing, which makes the watchdog semantics misleading.

Small elegance suggestions, non-blocking:

  1. The activity descriptions could be made a little more specific/distinct, e.g. receiving Codex text delta, receiving Codex reasoning delta, and receiving Codex fallback stream event. That would make future cron timeout logs more diagnostic.

  2. If there is an easy central stream-event handler or helper in this area later, it might be cleaner to put the _touch_activity() call in one place for all Codex stream events rather than at several event-specific branches. The current patch is small and clear, though.

  3. This pairs well with fix(codex): avoid custom keepalive transport on chatgpt backend #12953 but does not replace it: this PR keeps the cron watchdog honest during active streaming, while fix(codex): avoid custom keepalive transport on chatgpt backend #12953 avoids the custom transport path that appears to be causing Codex backend failures/wedges.

Overall: I think this is worth landing as part of the broader Codex cron reliability work.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cron Cron scheduler and job management labels Apr 25, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the focused Codex cron-reliability fix. This is an automated hermes-sweeper review; the requested behavior is already on main and shipped in v2026.4.13.

  • Commit 2773b18b5 (fix(run_agent): refresh activity during streaming responses) added per-event _touch_activity() coverage for the Codex stream and fallback paths.
  • Current main preserves this after the Codex runtime extraction: agent/codex_runtime.py:840-843 calls _touch_activity() from the shared event callback, and agent/codex_runtime.py:652-655 invokes that callback for every SSE event.
  • The former fallback now forwards to the same event-driven path (agent/codex_runtime.py:913-922), so it has identical coverage.
  • Regression coverage already exists in tests/run_agent/test_streaming.py:994-1031 and tests/run_agent/test_streaming.py:1073-1120.
  • The supportive review discussion correctly identified the importance of active-stream freshness; the current central event handler provides that guarantee without separate event-specific touchpoints.

@teknium1 teknium1 closed this Jul 12, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Cron inactivity timeout fires during active openai-codex streaming because Codex stream events do not update activity tracker

4 participants