Skip to content

fix(error_classifier): treat Z.AI 1305 overload as overloaded, not rate_limit - #77771

Open
Rams-Lab-01 wants to merge 1 commit into
NousResearch:mainfrom
Rams-Lab-01:fix/zai-1305-localized-overload
Open

Rams-Lab-01 wants to merge 1 commit into
NousResearch:mainfrom
Rams-Lab-01:fix/zai-1305-localized-overload

Conversation

@Rams-Lab-01

Copy link
Copy Markdown

Problem

#53578 added _OVERLOADED_PATTERNS so a 429 carrying overload language takes the transient-overload path instead of rotating the credential.

The pattern list is English-only, but Z.AI / Zhipu return the code-1305 overload message localized depending on account locale:

429 - {'error': {'code': '1305', 'message': '该模型当前访问量过大,请您稍后再试'}}

None of the English patterns match that text, so the provider that motivated #53578 still falls through to rate_limit and exhausts the pool after two errors — fatal for a single-key user, who has nothing to rotate to.

Per Z.AI's error reference, 1305 is "the service may be temporarily overloaded, please try again later" — a server-side capacity condition, not a per-credential rate limit.

Production impact

A transient Z.AI overload on glm-4.6v-flash marked a barely-used GLM_API_KEY exhausted, collapsed the pool to no available entries, and fell back to a text-only model that rejects image_url:

credential pool: marking GLM_API_KEY exhausted (status=429), rotating
credential pool: no available entries (all exhausted or empty)
Auxiliary vision: rate limit on zai — falling back to main agent model (deepseek-v4-flash-free)
... 400 unknown variant `image_url`, expected `text`

The user-visible result was vision silently degrading to OCR, on a key that had barely been used.

Fix

1. error_classifier — classify on the structured error code first.

The code is locale-independent, so classification no longer depends on the account's message language. Only 1305 is treated as overload; the other Z.AI 429 codes keep their existing behavior:

Code Meaning Behavior
1305 service temporarily overloaded overload — retry, keep credential
1302, 1313 rate limit / fair-usage throttle rate_limit — rotate (unchanged)
1308, 1310, 13161321 usage & spend limits quota/billing (unchanged)

The zh-CN overload phrasings are also added to _OVERLOADED_PATTERNS as a secondary net. This deliberately excludes 请求过于频繁 ("requests too frequent") and the generic 请稍后再试 ("try again later"), which also appear on genuine rate limits and must keep rotating.

2. auxiliary_client — the auxiliary path never consulted the classifier.

Vision / web_extract / compression / cron go through _is_rate_limit_error(), which returns True for any openai RateLimitError before inspecting the message:

if type(exc).__name__ == "RateLimitError":
    return True

So a 1305 burned the pool there regardless of fix (1) — this is the path that actually broke vision in the report above. _recover_provider_pool, the single choke point for auxiliary pool marking, now leaves the credential intact on an overload and lets the caller retry / fall back. It shares the pattern and code tables with error_classifier so both layers classify identically.

Tests

5 regression tests added covering:

  • the localized 1305 body (the real-world failure)
  • code-based classification when the message text is unrecognizable
  • localized overload text with no structured code
  • guards: 1302 and 请求过于频繁 must still classify as rate_limit and still rotate
tests/agent/test_error_classifier.py .......... 74 passed

test_credential_pool_routing.py reports 2 failures (test_resolve_turn_includes_pool, test_unmatched_key_does_not_retry_only_pool_entry) — these are pre-existing on main and unrelated to this change; verified identical with and without it by stashing.

The change is purely additive (+151 lines, no deletions, no behavior removed).

Refs #14038

…te_limit

NousResearch#53578 added _OVERLOADED_PATTERNS so a 429 carrying overload language takes
the transient-overload path instead of rotating the credential. The pattern
list is English-only, but Z.AI/Zhipu return the code-1305 overload message
localized ("该模型当前访问量过大,请您稍后再试") depending on account locale.
The motivating provider therefore still fell through to rate_limit and
exhausted the pool after two errors — fatal for a single-key user, who has
nothing to rotate to.

Two gaps are closed:

1. error_classifier now checks the structured error code before the message
   patterns. The code is locale-independent, so classification no longer
   depends on the account's message language. Only 1305 is treated as
   overload; 1302 and 1313 (genuine rate limits) and 1308, 1310, 1316-1321
   (usage/spend limits) keep their existing behavior. The zh-CN overload
   phrasings are added to _OVERLOADED_PATTERNS as a secondary net,
   deliberately excluding "请求过于频繁" and the generic "请稍后再试", which
   also appear on real rate limits.

2. The auxiliary path (vision, web_extract, compression, cron) never
   consulted the classifier: _is_rate_limit_error() returns True for any
   openai RateLimitError before inspecting the message, so a 1305 burned the
   pool there regardless of the first fix. _recover_provider_pool — the
   single choke point for auxiliary pool marking — now leaves the credential
   intact on an overload and lets the caller retry or fall back. It shares
   the pattern and code tables with error_classifier so both layers agree.

Observed in production: a transient Z.AI overload on glm-4.6v-flash marked a
barely-used GLM_API_KEY exhausted, collapsed the pool to "no available
entries", and fell back to a text-only model that cannot accept image_url,
degrading vision to OCR for the end user.

Adds regression tests for the localized 1305 body, code-based classification
without recognizable message text, and guards asserting that 1302 and
"请求过于频繁" still classify as rate_limit and still rotate.

Refs NousResearch#14038
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation provider/zai ZAI provider P2 Medium — degraded but workaround exists labels Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/zai ZAI provider tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants