Skip to content

fix(zai): adaptive backoff for Coding Plan GLM-5.2 overload (1305) + named custom-provider extra_body - #52333

Merged
teknium1 merged 2 commits into
mainfrom
hermes/hermes-5d84034f
Jun 25, 2026
Merged

fix(zai): adaptive backoff for Coding Plan GLM-5.2 overload (1305) + named custom-provider extra_body#52333
teknium1 merged 2 commits into
mainfrom
hermes/hermes-5d84034f

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Z.AI Coding Plan GLM-5.2 transient overload 429s (code 1305) now back off adaptively instead of failing fast after 3 short retries, and custom:<key> named providers correctly merge their extra_body.

Treats 1305 strictly as transient capacity overload — short retries first, then a widening 30→60→90→120s schedule (capped, jittered) — with long waits surfaced immediately so the TUI doesn't look frozen. No prompt rewriting.

Changes

  • agent/retry_utils.py: is_zai_coding_overload_error() (narrow: 429 + coding endpoint + glm-5.2 + 1305/"overloaded") + adaptive_rate_limit_backoff().
  • agent/conversation_loop.py: wire adaptive backoff into the rate-limit retry path; Retry-After headers still win; long waits use _emit_status, short ones stay buffered.
  • agent/agent_init.py: match custom:<provider_key> / custom:<name> when merging custom-provider extra_body.
  • Tests for both.

Validation

Result
tests/test_retry_utils.py + tests/agent/test_custom_provider_extra_body.py 17/17 pass

Salvages #47205 (@benbenlijie) onto current main, authorship preserved. Refs #50663.

Note: the prompt-phrase workaround (#48124 / #47685) is intentionally NOT included — we treat Z.AI's behavior as capacity overload, not something Hermes should silently rewrite around.

Infographic

zai-adaptive-overload-backoff

@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

🔎 Lint report: hermes/hermes-5d84034f vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11268 on HEAD, 11270 on base (✅ -2)

🆕 New issues (1):

Rule Count
invalid-assignment 1
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`

✅ Fixed issues (2):

Rule Count
unresolved-attribute 2
First entries
run_agent.py:2989: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`

Unchanged: 5941 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/zai ZAI provider P2 Medium — degraded but workaround exists labels Jun 25, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

This PR adds adaptive backoff for Z.AI Coding Plan GLM-5.2 overload errors (1305) and supports named custom providers with extra_body. The retry logic now uses adaptive_rate_limit_backoff() which provides provider-specific backoff policies.

Key observations

  • adaptive_rate_limit_backoff() returns both wait time and policy name
  • Z.AI Coding overload gets longer backoff (can last minutes)
  • Long waits surface progress immediately instead of buffering
  • Named custom providers (custom:key) now filter by provider key/name
  • Good test coverage for the custom provider extra_body matching

Looks Good

  • Clean integration of adaptive backoff into the retry loop
  • Proper handling of long waits with immediate status updates
  • Named custom provider support is well-tested

Reviewed by Hermes Agent (cron)

@davidgut1982

Copy link
Copy Markdown
Contributor

Plugin solution for the brand-word root cause

This PR (now merged) addresses the retry/backoff behavior for 1305 errors, but the root cause — Z.AI's brand-word content filter on "Hermes Agent" in the system prompt — is still unfixed on main. Users continue to hit 429/1305 even with adaptive backoff because the filter is deterministic, not transient.

I've built a drop-in plugin that fixes the root cause without core modifications:

Repo: https://github.com/davidgut1982/zai-brandfix

Uses ProviderProfile.prepare_messages() to rewrite "Hermes Agent""ZCode" in system messages, and build_api_kwargs_extras() to inject ZCode Desktop 3.1.8 fingerprint headers. No source patches, survives hermes update.

Install:

mkdir -p ~/.hermes/plugins/model-providers/zai/
curl -o ~/.hermes/plugins/model-providers/zai/__init__.py \
  https://raw.githubusercontent.com/davidgut1982/zai-brandfix/main/__init__.py
curl -o ~/.hermes/plugins/model-providers/zai/plugin.yaml \
  https://raw.githubusercontent.com/davidgut1982/zai-brandfix/main/plugin.yaml

See also: #47685

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/zai ZAI provider type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants