Skip to content

feat(agent): make overload (503/529) backoff configurable - #56037

Open
Ddhjx-code wants to merge 2 commits into
NousResearch:mainfrom
Ddhjx-code:feat/configurable-overload-backoff
Open

feat(agent): make overload (503/529) backoff configurable#56037
Ddhjx-code wants to merge 2 commits into
NousResearch:mainfrom
Ddhjx-code:feat/configurable-overload-backoff

Conversation

@Ddhjx-code

Copy link
Copy Markdown

Summary

  • Exposes three config keys (overload_max_retries, overload_base_delay, overload_max_delay) under agent: so users can tune how aggressively the retry loop backs off on provider 503/529 before falling back or surfacing the error.
  • Splits the previously coupled overloaded/timeout fallback threshold so each can evolve independently.
  • Defaults match prior hardcoded behavior (2 retries, 2.0s base, 60.0s max) — zero behavioral change when unconfigured.

Closes #55540

Changes

File What
agent/agent_init.py Read 3 new config keys with type-safe parsing and floor clamping
agent/conversation_loop.py Use configured values for overload fallback threshold and backoff delays
cli-config.yaml.example Document the new keys with usage guidance
tests/agent/test_overload_backoff_config.py 15 tests covering defaults, custom values, clamping, invalid input, threshold logic, and delay behavior

Test plan

  • New unit tests pass (pytest tests/agent/test_overload_backoff_config.py — 15 passed)
  • Existing output-cap and retry tests unaffected
  • Manual: set overload_max_retries: 0 and verify immediate fallback on 503
  • Manual: set overload_base_delay: 30.0 and verify longer waits on overload retry

…ch#55540)

Expose three config keys under `agent:` so users can tune how
aggressively the retry loop backs off on provider overload before
falling back or surfacing the error:

- overload_max_retries (default 2)
- overload_base_delay (default 2.0s)
- overload_max_delay (default 60.0s)

Defaults match the prior hardcoded behavior — zero behavioral change
when unconfigured.
@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 1, 2026
…-by-one

Address two issues found during review:

1. overload_max_retries was not upper-bounded against api_max_retries,
   causing the overload-specific fallback to never trigger when the
   configured value exceeded the outer retry loop ceiling.

2. The >= comparison in the fallback guard made values 0 and 1 produce
   identical behavior (both = zero retries). Changed to > so N means
   exactly N retries before fallback.
@Ddhjx-code

Copy link
Copy Markdown
Author

Addressed both findings from the review bot on the mirror PR (hashbender#129):

  1. Upper-bound clampoverload_max_retries is now clamped to api_max_retries at parse time (min(overload, api)), so the overload fallback threshold can never exceed the outer retry loop ceiling.

  2. Off-by-one fix — Changed >= to > in the fallback guard so that N means exactly N retries before fallback. Values 0 and 1 are now distinguishable.

Added 3 new test cases covering the clamp and boundary behavior. All 18 tests pass.

See commit 9b169e1.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fallback/backoff controls explicit.

Problems

  • The PR predicates on FailoverReason.overloaded, but current main uses that same reason for the narrow Z.AI Coding HTTP-429 path and intentionally extends its retry ceiling for the adaptive 30/60/90/120-second schedule (agent/conversation_loop.py:3177-3188, agent/retry_utils.py:108-154; commit 1c702aa73). The proposal says 503/529, so the salvage needs to decide and test whether the new settings should affect this 429 path.
  • tests/agent/test_overload_backoff_config.py:15-40 copies the production parser, and :115-159 copies the fallback predicate. Those tests do not execute agent/agent_init.py or agent/conversation_loop.py, so they cannot detect a wiring regression. tests/run_agent/test_api_max_retries_config.py:12-28 provides the existing AIAgent-based pattern.
  • The new documented settings are not represented in hermes_cli/config.py:990-1021 or the public retry configuration section at website/docs/user-guide/configuration.md:832-840.

Suggested changes

  • Rebase the behavior on the current Z.AI-aware retry path, preserve its narrow adaptive policy, and add production-path tests for generic 503/529 plus the Z.AI 429 boundary.
  • Register and document the final configuration surface alongside agent.api_max_retries.

Automated hermes-sweeper review.

return overrides


def _apply_overload_config(agent, agent_section, api_max_retries=3):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper duplicates the implementation under test, so every parsing assertion can pass even if agent/agent_init.py stops assigning these attributes. Please construct AIAgent through the config loader (as tests/run_agent/test_api_max_retries_config.py does) and assert the real initialized fields.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Longer/configurable backoff for HTTP 503/529 provider overload (parity with Z.AI path)

3 participants