Skip to content

fix: harden codex stale timeout and minimax overload retries - #11462

Open
Obrais-cloud wants to merge 1 commit into
NousResearch:mainfrom
Obrais-cloud:obrais/local-auth-fixes-20260414
Open

fix: harden codex stale timeout and minimax overload retries#11462
Obrais-cloud wants to merge 1 commit into
NousResearch:mainfrom
Obrais-cloud:obrais/local-auth-fixes-20260414

Conversation

@Obrais-cloud

Copy link
Copy Markdown

Summary

  • estimate Codex/Responses request size from input + instructions + tools, not just messages
  • prevent false 300s stale aborts on large openai-codex non-streaming requests
  • add retry wait-time selection so overloaded upstreams (503/529, e.g. MiniMax) back off longer and respect Retry-After
  • add regression tests for both behaviors

Test Plan

  • python -m pytest tests/test_retry_utils.py -q
  • python -m pytest tests/run_agent/test_run_agent.py -q -k "RequestContextTokenEstimate"
  • python -m pytest tests/run_agent/test_run_agent_codex_responses.py -q

@Obrais-cloud
Obrais-cloud force-pushed the obrais/local-auth-fixes-20260414 branch from 614fdf8 to 045644b Compare April 17, 2026 07:25
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/openai OpenAI / Codex Responses API provider/minimax MiniMax (Anthropic transport) labels Apr 24, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused retry and timeout work. The Codex half has since landed through a broader refactor, but the generic overload retry gap remains.

Problems

  • Current main already estimates Responses payloads from input, instructions, and tools in agent/chat_completion_helpers.py:66-112; it also applies the OpenAI-Codex large-payload stale floor at agent/chat_completion_helpers.py:131-147 and :387-393 (commit 2d422720b). The run_agent.py changes in this PR therefore need to be dropped during salvage.
  • The retry gap is now in agent/conversation_loop.py: 503/529 classify as overloaded (agent/error_classifier.py:1033-1044), but overloaded is excluded from is_rate_limited (agent/conversation_loop.py:3141-3149), so Retry-After is not read (:4094-4113).
  • The proposed 180-second cap conflicts with current main's documented 600-second Retry-After ceiling at agent/conversation_loop.py:4102-4107.

Suggested changes

  • Re-scope the salvage to the current retry loop, preserve the 600-second cap, and add a conversation-loop test for a 503/529 overload with Retry-After.

Automated hermes-sweeper review.

Comment thread agent/retry_utils.py
upstream cluster has time to recover. Retry-After, when present, wins.
"""

normalized_reason = getattr(reason, "value", reason) or ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not introduce a separate 180s cap when salvaging this. Current main intentionally caps Retry-After at 600s because a 120s cap retried before known reset windows (agent/conversation_loop.py:4102-4107). Reuse or centralize that policy and cover an overload response with a Retry-After header.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 12, 2026

@GottZ GottZ left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

Summary

Five PRs address overlapping retry and provider-handling failures: #47205/#52333 add named custom-provider matching and Z.AI-specific adaptive backoff, #59523/#60034 make that backoff reachable on the overloaded path, and #11462 combines an already-landed Codex timeout change with a still-distinct generic 503/529 Retry-After gap.

Related pull requests

  • #11462 related — (+128/-10) — keep open with a salvage path: drop the Codex request-size/stale-timeout changes because commit 2d422720b already implements them in agent/chat_completion_helpers.py:66-112, :131-147, and :387-393; retain and rework the generic overload handling in agent/conversation_loop.py so 503/529 overloads read Retry-After while preserving the existing 600-second ceiling. This follows the contributor keep_open review rather than treating the current mixed diff as mergeable.
  • #47205 [closed] related — (+237/-5) — already implemented on main: this closed PR added custom:<provider> extra_body matching and narrow adaptive backoff for Z.AI Coding Plan 429/1305 overloads; the discussion identifies equivalent upstream commit 6f2b2a1f34, and #52333 is the merged salvage/reference implementation.
  • #52333 [merged] related — (+238/-5) — merged reference implementation: it salvaged #47205 by adding named custom-provider extra_body matching and the Z.AI-specific 30/60/90/120-second overload policy, but its original gate left that policy unreachable for errors classified as overloaded and under the default retry ceiling.
  • #59523 [closed] related — (+87/-4) — canonical twin, merged through #60034: it identified and fixed both reachability defects by running Z.AI 429/1305 handling on the overloaded path and raising the retry ceiling so the long tier executes. The contributor discussion confirms its commits were cherry-picked into #60034 with authorship preserved.
  • #60034 [merged] related — (+96/-5) — merged reference for the #59523 fix: its diff carries the same overloaded-path routing and retry-ceiling mechanism, plus a shared short-attempt constant. The visible contributor triage review calls #59523 the canonical twin by creation time; #60034 is retained here only as the merged implementation, not as a replacement for that canonical designation.

Duplicates

#47205 and #52333 are substantially the same provider-matching and Z.AI adaptive-backoff change, with #52333 the merged salvage. #59523 and #60034 are twins; #59523 remains the contributor-designated canonical PR, while #60034 is the merged vehicle containing its commits plus the shared-constant refinement.

Suggested consolidation

Keep #11462 open with a salvage path: remove the Codex half already covered by commit 2d422720b, retarget the remaining generic 503/529 Retry-After fix to agent/conversation_loop.py, preserve the documented 600-second ceiling, and add a conversation-loop regression test. Treat closed #47205 as superseded by merged #52333, and closed #59523 as the canonical source twin whose implementation landed through merged #60034; no reopen or merge action is needed for those completed duplicate chains.

Cross-PR triage: Reviewed 5 pull requests and 0 issues in this complex. Each diff was read against this issue; Assessment working set: 54 kB of PR diffs, 9 kB of issue/PR text, 6 kB of discussion (7 comments), 0 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/minimax MiniMax (Anthropic transport) provider/openai OpenAI / Codex Responses API sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants