fix(agent): run Z.AI Coding overload adaptive backoff on the overloaded path - #60034
Merged
kshitijk4poor merged 3 commits intoJul 7, 2026
Merged
Conversation
Z.AI Coding Plan GLM-5.2 reports server overload as HTTP 429 code 1305
("temporarily overloaded"). classify_api_error routes that to
FailoverReason.overloaded (so a valid credential pool isn't burned), but
the adaptive Z.AI backoff was gated on is_rate_limited — which excludes
overloaded — so it never ran (policy=default) and the request failed after
a few quick short retries.
Two compounding causes, both fixed here:
1. Detect the Z.AI overload 429 directly and let its adaptive backoff run
on the overloaded path, not only the rate_limit path.
2. Raise the retry ceiling for this narrow case via
zai_coding_overload_retry_ceiling(). The long-backoff tier
(30/60/90/120s) starts after short_attempts (3) retries, but the default
api_max_retries is also 3, so the loop always gave up before the long
tier could run — leaving the whole long-backoff schedule as dead code.
Scope is limited to the existing narrow is_zai_coding_overload_error match,
so other providers' 429/503/529 handling is unchanged.
Assert the invariant that the Z.AI overload retry ceiling exceeds the short-retry threshold (the original bug had them equal, so the long tier was dead code), and walk the attempt range the retry loop actually traverses to prove the full 30/60/90/120s long-backoff schedule now runs.
…ange-detector assert Follow-up on the salvage of NousResearch#59523. Two low-risk cleanups surfaced by review: - Extract _ZAI_CODING_OVERLOAD_SHORT_ATTEMPTS as a module constant so adaptive_rate_limit_backoff() and zai_coding_overload_retry_ceiling() share one source of truth. Previously both hardcoded short_attempts=3 independently; tuning one without the other would silently desync the retry ceiling from the backoff schedule. - Replace the tautological formula-mirroring assert in test_zai_overload_retry_ceiling_exceeds_short_attempts with a behavior invariant (ceiling leaves headroom for every long-backoff entry), per the repo's contracts-over-snapshots testing rule.
alt-glitch
added
type/bug
Collaborator
Duplicate of #59523 (earliest of this twin pair, opened 2026-07-06). Diff comparison confirms the same mechanism: both introduce |
2 tasks
DeamonDev888
pushed a commit
to DeamonDev888/hermes-agent
that referenced
this pull request
Jul 11, 2026
End-to-end fix for the Z.AI credential pool cascade bug discovered
during a live audit of 8 production keys. Five coordinated layers
ensure the right endpoint is selected, the wrong error is not treated
as a payment error, and the user-configured overrides work end-to-end.
== Problem statement ==
Z.AI Coding Plan subscriptions (id.secret.* format) authenticate on
/api/coding/paas/v4 but the metered /api/paas/v4 endpoint also returns
HTTP 200 for glm-5. Without explicit routing, the metered endpoint
gets cached in auth.json and Coding Plan traffic silently draws from
the metered billing pool instead of the subscription.
A second symptom: Z.AI Coding Plan error codes 1113 (Insufficient
balance or no resource package) and 1308 (Usage limit reached for 5
hour) are per-key rolling quotas. They were misclassified as payment
errors via substring traps intended for Vertex AI (resource exhausted)
and Nous Portal (reached your session usage limit). A single exhausted
key cascade-marked every other key in the pool, taking the whole
provider offline.
A third gap: keys on the Anthropic Messages wire (/api/anthropic) were
not in ZAI_ENDPOINTS at all, so detect_zai_endpoint() returned the
wrong URL for any Anthropic-wire Coding Plan key.
== Solution overview ==
Five layers, all coordinated, all tested end-to-end:
Layer 1 - _is_payment_error() exemption for Z.AI Coding Plan
Detects api.z.ai / z.ai/api/coding / zhipuai+coding context plus
codes 1113, 1308 and message patterns. Returns False so the
credential pool rotation layer handles per-key 5h rolling quotas
correctly. Real Z.AI payment errors (e.g. code 1311 plan-block)
continue to flow through the normal payment-fallback path.
Layer 2 - Vision auto-detect zai_openai_urls ordering
The vision helper list now probes Coding Plan endpoints (global +
China) before the metered paas/v4 fallbacks. Coding Plan keys
authenticate on first vision call. Corrects an indentation bug in
the original PR NousResearch#55116.
Layer 3 - Runtime pool base_url re-resolution
_resolve_api_key_provider() now re-invokes _resolve_zai_base_url()
for provider_id == zai so manual-pool entries added via
her mes auth add honor the cached detected_endpoint state in
auth.json and the GLM_BASE_URL env override. Brings manual-pool
parity with the env-seeded path fixed in commit 9e84416.
Probe failures are caught and logged so they never break pool
selection.
Layer 4 - config.yaml model.base_url precedence
New _configured_zai_base_url() helper reads model.base_url from
config.yaml when model.provider is a Z.AI alias (zai / glm / z-ai /
z.ai / zhipu). The precedence chain in _resolve_zai_base_url() is:
1. GLM_BASE_URL env var (highest)
2. model.base_url from config.yaml (when provider is Z.AI)
3. cached detected_endpoint in auth.json
4. live probe of all candidate endpoints
5. registry default
The provider alias guard prevents leakage from non-Z.AI configs.
Layer 5 - Anthropic-wire endpoints in ZAI_ENDPOINTS
Extended from 4 to 6 candidates with anthropic-global and
anthropic-cn. coding-global is probed FIRST (99% of keys that
accept coding endpoint also accept metered, so probing coding
first caches the right URL on first try). anthropic-global is
position 2 (fallthrough for pure Anthropic-wire keys).
New _zai_probe_path() and _zai_probe_body() helpers dispatch the
right HTTP body shape: Anthropic Messages for anthropic-* ids
(/v1/messages, no stream field, anthropic-version header) and
OpenAI chat completions for everything else.
== Audit results ==
Validated against 8 real production keys (see
tests/agent/test_zai_8keys_audit.py):
- 6/8 keys: pure Anthropic-wire subscribers (Anthropic Messages)
- 1/8 keys: pure OpenAI-wire (standard paas/v4)
- 1/8 keys: full-stack (works on all 3 endpoints)
Live test audit (tests/agent/test_zai_live_audit.py) covers every
Z.AI error category against the real api.z.ai API: 200, 1305, 1308,
1113, 401, 402 - all classified correctly.
Before fix: detect_zai_endpoint() returned the wrong endpoint for
6/8 keys. After fix: 9/9 keys correctly routed and verified working.
== Test coverage ==
249 unit + integration tests, 21 live tests (opt-in via env var):
tests/hermes_cli/test_zai_5gateway_extension.py (29 tests)
ZAI_ENDPOINTS structure, dispatch helpers, signature stability,
backward compat, probe order pinning
tests/hermes_cli/test_zai_config_yaml_precedence.py (8 tests)
GLM_BASE_URL > model.base_url > cached > probe > default
tests/agent/test_zai_manual_pool_routing.py (7 tests)
Runtime re-resolution, GLM_BASE_URL forwarding, probe failure
tolerance, non-zai provider leak guard
tests/agent/test_auxiliary_client_zai_payment_classification.py
(14 tests)
_is_payment_error Z.AI Coding Plan exemption, verbatim Z.AI
response bodies, negative cases for Vertex/Bedrock/OpenRouter
tests/agent/test_zai_e2e_pool.py + test_zai_e2e_pool_rotation.py
(8 tests)
Real HTTP round-trip via local mock Z.AI server, 5-key
round_robin, revoked key, probe failure recovery
tests/agent/test_zai_live.py + test_zai_live_audit.py +
test_zai_8keys_audit.py (21 tests)
Live against api.z.ai, opt-in via HERMES_RUN_LIVE=1 or
GLM_AUDIT_KEYS env var. Keys masked to 8-char prefix in all
output. File is safe to commit.
== Related work ==
PR NousResearch#61492 - _is_payment_error() Z.AI exemption (cloned in Layer 1)
PR NousResearch#55116 - vision helper coding endpoint (Layer 2)
PR NousResearch#58088 - config.yaml base_url precedence (Layer 4)
PR NousResearch#24915 - 4-variant provider split (intentionally NOT adopted:
orthogonal refactor, deferred)
PR NousResearch#54643 - /api/anthropic to /api/coding/paas/v4 rewrite
(orthogonal; this PR probes the right wire, NousResearch#54643 rewrites at
runtime if a user forces the OpenAI wire via config)
PR NousResearch#55007 - stream probes without reading bodies (orthogonal;
this PR can refactor _zai_probe_path to use httpx.stream() once
NousResearch#55007 lands)
PR NousResearch#61333 - skip /anthropic to /v1 rewrite (complementary)
PR NousResearch#60753 - preserve /anthropic for custom vision (complementary)
PR NousResearch#32174 - add zai-coding provider (different design choice)
PR NousResearch#60034 - Z.AI Coding overload adaptive backoff (already merged)
== Related issues ==
NousResearch#61487 - cascade _is_payment_error (Layer 1 closes)
NousResearch#61563 - manual-pool routing audit (this PR implements)
NousResearch#47970 - GLM-5.2 context_length fallback (Layer 5 helps)
NousResearch#55112 - auxiliary vision hardcoded zai (Layer 5 helps)
NousResearch#47685 - Hermes Agent prompt block on Z.ai (orthogonal)
== Test commands ==
Unit + integration (no network):
pytest tests/hermes_cli/test_api_key_providers.py \
tests/hermes_cli/test_zai_config_yaml_precedence.py \
tests/hermes_cli/test_zai_5gateway_extension.py \
tests/agent/test_auxiliary_client.py::TestIsPaymentError \
tests/agent/test_auxiliary_client_zai_payment_classification.py \
tests/agent/test_zai_manual_pool_routing.py \
tests/agent/test_zai_e2e_pool.py \
tests/agent/test_zai_e2e_pool_rotation.py -v
Live (consumes Z.AI quota, requires HERMES_RUN_LIVE=1):
pytest tests/agent/test_zai_live.py \
tests/agent/test_zai_live_audit.py \
tests/agent/test_zai_8keys_audit.py -v --runlive
Cross-platform check:
scripts/check-windows-footguns.py --diff upstream/main
== Security ==
- All live test keys read from env var (GLM_TEST_KEYS, GLM_AUDIT_KEYS,
GLM_WORKING_KEY) NEVER from files. Files are safe to commit
publicly.
- All keys masked to 8-char prefix in any printed output.
- No hardcoded credentials, no path leaks, no LAN IPs.
- Privacy scan: 0 secrets, 0 private paths in diff.
Co-authored-by: Hermes triage bot <bot@nousresearch.com>
Refs: NousResearch#61487, NousResearch#61563, PR NousResearch#61492, PR NousResearch#55116, PR NousResearch#58088, PR NousResearch#24915,
PR NousResearch#54643, PR NousResearch#55007, PR NousResearch#61333, PR NousResearch#60753, PR NousResearch#32174,
PR NousResearch#60034, NousResearch#47970, NousResearch#55112, NousResearch#47685
santhreal
pushed a commit
to santhreal/hermes-agent
that referenced
this pull request
Jul 13, 2026
…3-zai-overload-backoff fix(agent): run Z.AI Coding overload adaptive backoff on the overloaded path
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…3-zai-overload-backoff fix(agent): run Z.AI Coding overload adaptive backoff on the overloaded path
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…3-zai-overload-backoff fix(agent): run Z.AI Coding overload adaptive backoff on the overloaded path
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…3-zai-overload-backoff fix(agent): run Z.AI Coding overload adaptive backoff on the overloaded path
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Z.AI Coding Plan GLM-5.2 overload 429s (code 1305, "temporarily overloaded") now run the adaptive long-backoff schedule (3 short retries → 30/60/90/120s) instead of giving up after ~12s.
Two compounding bugs on current
main, both confirmed:classify_api_errorroutes the overload 429 toFailoverReason.overloaded(so a valid credential pool isn't burned), but the Z.AI adaptive backoff (added alongside the overload classification in6f2b2a1f3) is gated onis_rate_limited, which excludesoverloaded. The overload 429 fell through to the generic short backoff — the adaptive path was dead for the exact error it was written for.adaptive_rate_limit_backoffkeeps the firstshort_attempts(3) retries short, then switches to the 30/60/90/120s tier. But the defaultapi_max_retriesis also 3, soretry_count >= max_retriesfired before any long-tier attempt — the whole long-backoff schedule was dead code under the shipped config.Changes
agent/conversation_loop.py: detect the overload 429 directly (_is_zai_coding_overload) so its adaptive backoff runs on theoverloadedpath, and raise the retry ceiling viazai_coding_overload_retry_ceiling()so the long tier is reachable. Status line distinguishes "Provider overloaded" from "Rate limited".agent/retry_utils.py: addzai_coding_overload_retry_ceiling(); extract_ZAI_CODING_OVERLOAD_SHORT_ATTEMPTSas a shared constant so the ceiling and the backoff walker can't desync (follow-up).tests/test_retry_utils.py: ceiling-vs-short-attempts invariant + an end-to-end reachability test walking the loop's real attempt range and asserting the full 30/60/90/120s schedule runs.Scope stays inside the existing narrow
is_zai_coding_overload_errormatch (base URLapi.z.ai/api/coding/paas/v4+glm-5.2+ code 1305 / "temporarily overloaded"). Other providers' 429/503/529 handling is unchanged; the credential-pool-preservingoverloadedclassification is kept intact.Validation
policy=default, ~2s/4s, give up at 3 (~12s)tests/test_retry_utils.py+ 6run_agentretry/fallback files).Credit
Based on #59523 by @xxxigm — cherry-picked to preserve authorship. Follow-up commit (shared short-attempts constant, behavior-invariant test, comment trim) by @kshitijk4poor.
Closes #59523