feat(agent): make overload (503/529) backoff configurable - #56037
feat(agent): make overload (503/529) backoff configurable#56037Ddhjx-code wants to merge 2 commits into
Conversation
…ch#55540) Expose three config keys under `agent:` so users can tune how aggressively the retry loop backs off on provider overload before falling back or surfacing the error: - overload_max_retries (default 2) - overload_base_delay (default 2.0s) - overload_max_delay (default 60.0s) Defaults match the prior hardcoded behavior — zero behavioral change when unconfigured.
…-by-one Address two issues found during review: 1. overload_max_retries was not upper-bounded against api_max_retries, causing the overload-specific fallback to never trigger when the configured value exceeded the outer retry loop ceiling. 2. The >= comparison in the fallback guard made values 0 and 1 produce identical behavior (both = zero retries). Changed to > so N means exactly N retries before fallback.
|
Addressed both findings from the review bot on the mirror PR (hashbender#129):
Added 3 new test cases covering the clamp and boundary behavior. All 18 tests pass. See commit 9b169e1. |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for making the fallback/backoff controls explicit.
Problems
- The PR predicates on
FailoverReason.overloaded, but current main uses that same reason for the narrow Z.AI Coding HTTP-429 path and intentionally extends its retry ceiling for the adaptive 30/60/90/120-second schedule (agent/conversation_loop.py:3177-3188,agent/retry_utils.py:108-154; commit1c702aa73). The proposal says 503/529, so the salvage needs to decide and test whether the new settings should affect this 429 path. tests/agent/test_overload_backoff_config.py:15-40copies the production parser, and:115-159copies the fallback predicate. Those tests do not executeagent/agent_init.pyoragent/conversation_loop.py, so they cannot detect a wiring regression.tests/run_agent/test_api_max_retries_config.py:12-28provides the existing AIAgent-based pattern.- The new documented settings are not represented in
hermes_cli/config.py:990-1021or the public retry configuration section atwebsite/docs/user-guide/configuration.md:832-840.
Suggested changes
- Rebase the behavior on the current Z.AI-aware retry path, preserve its narrow adaptive policy, and add production-path tests for generic 503/529 plus the Z.AI 429 boundary.
- Register and document the final configuration surface alongside
agent.api_max_retries.
Automated hermes-sweeper review.
| return overrides | ||
|
|
||
|
|
||
| def _apply_overload_config(agent, agent_section, api_max_retries=3): |
There was a problem hiding this comment.
This helper duplicates the implementation under test, so every parsing assertion can pass even if agent/agent_init.py stops assigning these attributes. Please construct AIAgent through the config loader (as tests/run_agent/test_api_max_retries_config.py does) and assert the real initialized fields.
Summary
overload_max_retries,overload_base_delay,overload_max_delay) underagent:so users can tune how aggressively the retry loop backs off on provider 503/529 before falling back or surfacing the error.overloaded/timeoutfallback threshold so each can evolve independently.Closes #55540
Changes
agent/agent_init.pyagent/conversation_loop.pycli-config.yaml.exampletests/agent/test_overload_backoff_config.pyTest plan
pytest tests/agent/test_overload_backoff_config.py— 15 passed)overload_max_retries: 0and verify immediate fallback on 503overload_base_delay: 30.0and verify longer waits on overload retry