fix(auxiliary): exclude Z.AI Coding Plan rolling quotas from payment error detection

[liuhao1024]

What does this PR do?

Fixes a cascade-failure bug in the Z.AI credential pool where a single key hitting its 5-hour rolling quota causes all other keys in the pool to be marked as exhausted, taking the entire provider offline instead of rotating to the next healthy key.

Root cause: _is_payment_error() matched Z.AI Coding Plan error codes (1113 and 1308) via substring traps intended for other providers:

• 1113 "Insufficient balance or no resource package" matched "resource exhausted" (Vertex AI pattern)
• 1308 "Usage limit reached for 5 hour" matched "reached your session usage limit" (Nous Portal pattern)

Fix: Added an early-return block that detects Z.AI Coding Plan endpoint patterns and these specific error codes, returning False to enable credential pool rotation. Real Z.AI payment errors (e.g. code 1311 plan-block) continue to flow through the normal payment-error path.

Related Issue

Fixes #61487

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• agent/auxiliary_client.py: Added early-return block in _is_payment_error() to detect Z.AI Coding Plan errors (api.z.ai domain, z.ai/api/coding endpoint, or zhipuai+coding keywords) and error codes 1113, 1308, and message patterns "no resource package" and "usage limit reached". Updated docstring to document this exception.
• tests/agent/test_auxiliary_client.py: Added 6 regression tests:
  • test_zai_coding_plan_1113_no_resource_package: Code 1113 is NOT a payment error
  • test_zai_coding_plan_1308_usage_limit_reached: Code 1308 is NOT a payment error
  • test_zai_coding_plan_api_zai_domain: api.z.ai domain errors are NOT payment errors
  • test_zai_coding_plan_coding_endpoint: Coding endpoint errors are NOT payment errors
  • test_zai_metered_endpoint_payment_error_still_detected: Real Z.AI payment errors (402) still work
  • test_vertex_ai_resource_exhausted_still_detected: Vertex AI "resource exhausted" still detected

How to Test

1. Unit tests: Run pytest tests/agent/test_auxiliary_client.py::TestIsPaymentError -v — all 23 tests pass (17 existing + 6 new).

2. Integration verification (if you have a Z.AI Coding Plan subscription with multiple keys):

   • Configure a multi-key pool with hermes config set credential_pool_strategies.zai round_robin
   • Drive one key into its 5-hour rolling quota (30 sequential glm-4-air calls with max_tokens=256 is enough)
   • Make another request through the auxiliary client
   • Before fix: All keys in the pool are marked exhausted, provider goes offline
   • After fix: Only the exhausted key is marked; the next healthy key in the pool is tried and succeeds

3. Negative case verification: The fix does not affect real Z.AI payment errors:

   • Code 1311 ("subscription plan does not yet include access to model") is a permanent block and correctly triggers payment-fallback behavior
   • Unit test test_zai_metered_endpoint_payment_error_still_detected confirms this

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 15.2

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Test run output:

============================= test session starts =============================
collected 23 items

tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_402_status_code PASSED [  4%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_402_with_credits_message PASSED [  8%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_with_credits_message PASSED [ 13%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_404_free_tier_model_block_is_payment PASSED [ 17%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_403_subscription_required_is_payment PASSED [ 21%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_session_usage_limit_is_payment PASSED [ 26%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_404_generic_not_found_is_not_payment PASSED [ 30%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_without_credits_message_is_not_payment PASSED [ 34%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_generic_500_is_not_payment PASSED [ 39%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_no_status_code_with_billing_message PASSED [ 43%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_no_status_code_no_message PASSED [ 47%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_quota_exceeded PASSED [ 52%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_too_many_tokens_per_day PASSED [ 56%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_daily_limit_phrase PASSED [ 60%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_resource_exhausted_grpc PASSED [ 65%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_daily_quota_phrase PASSED [ 69%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_429_transient_rate_limit_not_quota PASSED [ 73%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_zai_coding_plan_1113_no_resource_package PASSED [ 78%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_zai_coding_plan_1308_usage_limit_reached PASSED [ 82%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_zai_coding_plan_api_zai_domain PASSED [ 86%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_zai_coding_plan_coding_endpoint PASSED [ 91%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_zai_metered_endpoint_payment_error_still_detected PASSED [ 95%]
tests/agent/test_auxiliary_client.py::TestIsPaymentError::test_vertex_ai_resource_exhausted_still_detected PASSED [100%]

============================== 23 passed in 0.65s =============================

[tonydwb]

Code Review Summary

Verdict: Comment

Overview

• Excludes Z.AI Coding Plan rolling quotas from payment error detection
• Small, targeted fix (+62/0)

Looks Good

• Clean, focused
• No security concerns

---

Reviewed by Hermes Agent

[DeamonDev888]

🟢 Superseded by #61575 — please close

@liuhao1024 thanks for the fast turnaround on this fix. PR #61575 includes your _is_payment_error() logic verbatim (Layer 1), plus 4 additional layers that close the full Z.AI routing story:

• Vision helper probes coding endpoints first (corrects the indentation bug in fix(auxiliary): prepend zai coding-plan vision endpoint before metered paas/v4 (fixes #55112) #55116)
• Runtime pool re-resolves _resolve_zai_base_url() for manual-pool entries (the original bug)
• config.yaml model.base_url precedence (adapts PR fix(zai): honor configured base url #58088)
• ZAI_ENDPOINTS extended with anthropic-global/anthropic-cn (closes the wire-format detection gap for 6 of 8 audited production keys)

I don't have maintainer permissions to close this, but if a maintainer agrees this PR is now redundant, please close in favor of #61575. Credit remains in the git history either way.

Live audit: 9/9 production keys correctly routed, 0 regression, 0 Windows footguns.

[liuhao1024]

Closing — superseded by #61575

Thanks @DeamonDev888 for coordinating this. PR #61575 includes the _is_payment_error() Z.AI Coding Plan exemption logic from this PR (as Layer 1 of a broader fix), plus 4 additional layers that cover the full Z.AI routing story:

• Vision helper probes coding endpoints first
• Runtime pool re-resolution for manual-pool entries
• config.yaml model.base_url precedence
• Extended ZAI_ENDPOINTS with anthropic-global/anthropic-cn

Closing in favor of #61575 to avoid fragmenting the Z.AI routing story across multiple PRs. Both fixes are functionally equivalent on the _is_payment_error front.

Fixes #61487

[DeamonDev888]

Superseded by #62467. Your Z.AI payment-error classification fix is subsumed by the unified resolver, which handles 1113/1308 correctly across all surfaces. Closing recommendation: close in favor of #62467.