Skip to content

fix(agent): classify 429 'overloaded' bodies as overloaded, not rate_limit (#14038) - #53578

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-97a90860
Jun 27, 2026
Merged

fix(agent): classify 429 'overloaded' bodies as overloaded, not rate_limit (#14038)#53578
teknium1 merged 1 commit into
mainfrom
hermes/hermes-97a90860

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

A provider returning HTTP 429 with an "overloaded" body no longer exhausts the credential pool. The 429 status path classified every 429 as rate_limit with should_rotate_credential=True, so a busy (but valid) provider got its single key marked exhausted after two errors — locking out users with one key, who have nothing to rotate to.

Closes #14038.

Root cause

_classify_by_status returned rate_limit unconditionally for HTTP 429 — it never inspected the body. Z.AI / Zhipu reuse 429 for server-wide overload. The credential is valid; the server is just busy, so rotation is both useless and destructive (it burns the pool while the endpoint is still busy).

This is the credential-pool-exhaustion vector the issue describes ("after 2 errors the single API key is marked exhausted"). The companion message-only path was fixed separately in #52890 (salvage of #14261 by @ms-alan); this PR covers the 429-body vector that #52890 didn't touch. Approach informed by the closed #15414 by @briandevans, which first identified this half.

Changes

  • agent/error_classifier.py: 429 path now matches the body against _OVERLOADED_PATTERNS and routes overload language to FailoverReason.overloaded (retryable, no credential rotation), matching the existing 503/529 and message-only paths. Genuine rate limits (no overload language) still rotate.
  • Extracted the inline overloaded tuple fix(agent): classify message-only 'overloaded' as server overload (salvage of #14261 by @ms-alan) #52890 added into a shared _OVERLOADED_PATTERNS module constant so the status-code and message paths use one list.
  • Tests: added a 429-overloaded regression test and a guard that a genuine 429 rate limit still rotates.

Validation

Error Before After
HTTP 429 + "temporarily overloaded" rate_limit + rotate (exhausts pool) overloaded · backoff+retry · no rotation
HTTP 429 "rate limit exceeded" rate_limit + rotate rate_limit + rotate (unchanged)
HTTP 503/529 overloaded overloaded overloaded (unchanged)
message-only overloaded overloaded (#52890) overloaded (unchanged)

scripts/run_tests.sh tests/agent/test_error_classifier.py → 164 passed. tests/agent/test_credential_pool_routing.py → 10 passed.

Infographic

429-overloaded-misclassification-fix

@github-actions

github-actions Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

🔎 Lint report: hermes/hermes-97a90860 vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11502 on HEAD, 11500 on base (🆕 +2)

🆕 New issues (2):

Rule Count
unresolved-attribute 2
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`
run_agent.py:3014: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`

✅ Fixed issues (1):

Rule Count
invalid-assignment 1
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`

Unchanged: 6049 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

…limit

Z.AI / Zhipu reuse HTTP 429 for server-wide overload. The 429 status
path classified these unconditionally as rate_limit with
should_rotate_credential=True, so an overloaded provider exhausted the
credential pool after two errors — fatal for a single-key user, who has
nothing to rotate to.

The credential is valid; the server is just busy. Disambiguate the 429
body against a shared _OVERLOADED_PATTERNS list and route overload
language to FailoverReason.overloaded (retryable, no rotation), matching
the existing 503/529 path and the message-only path (#52890). Genuine
rate limits (no overload language) still rotate.

Extracted the inline overloaded tuple #52890 added into the shared
_OVERLOADED_PATTERNS constant so the status-code and message paths use
one list.

Closes #14038.
@teknium1
teknium1 force-pushed the hermes/hermes-97a90860 branch from 157199c to 7f98b31 Compare June 27, 2026 11:07
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Jun 27, 2026
@teknium1
teknium1 merged commit 38e7bd8 into main Jun 27, 2026
30 checks passed
@teknium1
teknium1 deleted the hermes/hermes-97a90860 branch June 27, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"overloaded" server errors classified as rate_limit, exhausting credential pool

2 participants