fix(agent): classify 429 'overloaded' bodies as overloaded, not rate_limit (#14038) - #53578
Merged
Conversation
This was referenced Jun 27, 2026
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-attribute |
2 |
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`
run_agent.py:3014: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
✅ Fixed issues (1):
| Rule | Count |
|---|---|
invalid-assignment |
1 |
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`
Unchanged: 6049 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
…limit Z.AI / Zhipu reuse HTTP 429 for server-wide overload. The 429 status path classified these unconditionally as rate_limit with should_rotate_credential=True, so an overloaded provider exhausted the credential pool after two errors — fatal for a single-key user, who has nothing to rotate to. The credential is valid; the server is just busy. Disambiguate the 429 body against a shared _OVERLOADED_PATTERNS list and route overload language to FailoverReason.overloaded (retryable, no rotation), matching the existing 503/529 path and the message-only path (#52890). Genuine rate limits (no overload language) still rotate. Extracted the inline overloaded tuple #52890 added into the shared _OVERLOADED_PATTERNS constant so the status-code and message paths use one list. Closes #14038.
teknium1
force-pushed
the
hermes/hermes-97a90860
branch
from
June 27, 2026 11:07
157199c to
7f98b31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A provider returning HTTP 429 with an "overloaded" body no longer exhausts the credential pool. The 429 status path classified every 429 as
rate_limitwithshould_rotate_credential=True, so a busy (but valid) provider got its single key marked exhausted after two errors — locking out users with one key, who have nothing to rotate to.Closes #14038.
Root cause
_classify_by_statusreturnedrate_limitunconditionally for HTTP 429 — it never inspected the body. Z.AI / Zhipu reuse 429 for server-wide overload. The credential is valid; the server is just busy, so rotation is both useless and destructive (it burns the pool while the endpoint is still busy).This is the credential-pool-exhaustion vector the issue describes ("after 2 errors the single API key is marked exhausted"). The companion message-only path was fixed separately in #52890 (salvage of #14261 by @ms-alan); this PR covers the 429-body vector that #52890 didn't touch. Approach informed by the closed #15414 by @briandevans, which first identified this half.
Changes
agent/error_classifier.py: 429 path now matches the body against_OVERLOADED_PATTERNSand routes overload language toFailoverReason.overloaded(retryable, no credential rotation), matching the existing 503/529 and message-only paths. Genuine rate limits (no overload language) still rotate._OVERLOADED_PATTERNSmodule constant so the status-code and message paths use one list.Validation
rate_limit+ rotate (exhausts pool)overloaded· backoff+retry · no rotationrate_limit+ rotaterate_limit+ rotate (unchanged)overloadedoverloaded(unchanged)overloaded(#52890)overloaded(unchanged)scripts/run_tests.sh tests/agent/test_error_classifier.py→ 164 passed.tests/agent/test_credential_pool_routing.py→ 10 passed.Infographic