Skip to content

fix(auxiliary): prepend zai coding-plan vision endpoint before metered paas/v4 (fixes #55112) - #55116

Open
Morad37 wants to merge 2 commits into
NousResearch:mainfrom
Morad37:fix/zai-vision-coding-plan-url-55112
Open

fix(auxiliary): prepend zai coding-plan vision endpoint before metered paas/v4 (fixes #55112)#55116
Morad37 wants to merge 2 commits into
NousResearch:mainfrom
Morad37:fix/zai-vision-coding-plan-url-55112

Conversation

@Morad37

@Morad37 Morad37 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Closes #55112

Prepend the coding-plan quota endpoint as the first URL tried for ZAI vision requests, so subscription users hit the quota-billed endpoint before falling through to the metered pay-as-you-go endpoints.

@Caiofwp

Caiofwp commented Jun 29, 2026

Copy link
Copy Markdown

Thanks for the quick fix — the ordering is exactly right, prepending /api/coding/paas/v4 so subscription users hit the quota-billed endpoint first. 🙏

One small nit: the new open.bigmodel.cn line is indented with 8 spaces while the rest of the list uses 12, so the block is misaligned:

        zai_openai_urls = [
            "https://api.z.ai/api/coding/paas/v4",   # coding-plan quota
        "https://open.bigmodel.cn/api/paas/v4",  # metered fallthrough   <-- 8 spaces
            "https://api.z.ai/api/paas/v4",
        ]

It's valid Python (indentation is free inside the list literal), but black/ruff in CI will likely flag or reformat it. Re-aligning to 12 spaces should keep the lint check green:

        zai_openai_urls = [
            "https://api.z.ai/api/coding/paas/v4",   # coding-plan quota
            "https://open.bigmodel.cn/api/paas/v4",  # metered fallthrough
            "https://api.z.ai/api/paas/v4",
        ]

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/zai ZAI provider area/billing Account usage, credit usage, billing (cross-cutting) P2 Medium — degraded but workaround exists labels Jun 29, 2026
@DeamonDev888

DeamonDev888 commented Jul 9, 2026

Copy link
Copy Markdown

🟢 Superseded — please close in favor of the new PR

@Morad37 thanks for the original PR. The rebased and consolidated fix is now #62080. The vision helper zai_openai_urls list ordering is implemented there with the indentation bug from your original fixed.

The new PR also includes:

  • Layer 1: _is_payment_error() Z.AI Coding Plan exemption
  • Layer 2: vision helper (this fix)
  • Layer 3: runtime pool re-resolution
  • Layer 4: config.yaml precedence
  • Layer 5: Anthropic-wire endpoints in ZAI_ENDPOINTS

If maintainers agree, please close this PR in favor of #62080. Credit remains in git history.

DeamonDev888 pushed a commit to DeamonDev888/hermes-agent that referenced this pull request Jul 11, 2026
End-to-end fix for the Z.AI credential pool cascade bug discovered
during a live audit of 8 production keys. Five coordinated layers
ensure the right endpoint is selected, the wrong error is not treated
as a payment error, and the user-configured overrides work end-to-end.

== Problem statement ==

Z.AI Coding Plan subscriptions (id.secret.* format) authenticate on
/api/coding/paas/v4 but the metered /api/paas/v4 endpoint also returns
HTTP 200 for glm-5. Without explicit routing, the metered endpoint
gets cached in auth.json and Coding Plan traffic silently draws from
the metered billing pool instead of the subscription.

A second symptom: Z.AI Coding Plan error codes 1113 (Insufficient
balance or no resource package) and 1308 (Usage limit reached for 5
hour) are per-key rolling quotas. They were misclassified as payment
errors via substring traps intended for Vertex AI (resource exhausted)
and Nous Portal (reached your session usage limit). A single exhausted
key cascade-marked every other key in the pool, taking the whole
provider offline.

A third gap: keys on the Anthropic Messages wire (/api/anthropic) were
not in ZAI_ENDPOINTS at all, so detect_zai_endpoint() returned the
wrong URL for any Anthropic-wire Coding Plan key.

== Solution overview ==

Five layers, all coordinated, all tested end-to-end:

  Layer 1 - _is_payment_error() exemption for Z.AI Coding Plan
    Detects api.z.ai / z.ai/api/coding / zhipuai+coding context plus
    codes 1113, 1308 and message patterns. Returns False so the
    credential pool rotation layer handles per-key 5h rolling quotas
    correctly. Real Z.AI payment errors (e.g. code 1311 plan-block)
    continue to flow through the normal payment-fallback path.

  Layer 2 - Vision auto-detect zai_openai_urls ordering
    The vision helper list now probes Coding Plan endpoints (global +
    China) before the metered paas/v4 fallbacks. Coding Plan keys
    authenticate on first vision call. Corrects an indentation bug in
    the original PR NousResearch#55116.

  Layer 3 - Runtime pool base_url re-resolution
    _resolve_api_key_provider() now re-invokes _resolve_zai_base_url()
    for provider_id == zai so manual-pool entries added via
    her mes auth add honor the cached detected_endpoint state in
    auth.json and the GLM_BASE_URL env override. Brings manual-pool
    parity with the env-seeded path fixed in commit 9e84416.
    Probe failures are caught and logged so they never break pool
    selection.

  Layer 4 - config.yaml model.base_url precedence
    New _configured_zai_base_url() helper reads model.base_url from
    config.yaml when model.provider is a Z.AI alias (zai / glm / z-ai /
    z.ai / zhipu). The precedence chain in _resolve_zai_base_url() is:
        1. GLM_BASE_URL env var (highest)
        2. model.base_url from config.yaml (when provider is Z.AI)
        3. cached detected_endpoint in auth.json
        4. live probe of all candidate endpoints
        5. registry default
    The provider alias guard prevents leakage from non-Z.AI configs.

  Layer 5 - Anthropic-wire endpoints in ZAI_ENDPOINTS
    Extended from 4 to 6 candidates with anthropic-global and
    anthropic-cn. coding-global is probed FIRST (99% of keys that
    accept coding endpoint also accept metered, so probing coding
    first caches the right URL on first try). anthropic-global is
    position 2 (fallthrough for pure Anthropic-wire keys).
    New _zai_probe_path() and _zai_probe_body() helpers dispatch the
    right HTTP body shape: Anthropic Messages for anthropic-* ids
    (/v1/messages, no stream field, anthropic-version header) and
    OpenAI chat completions for everything else.

== Audit results ==

Validated against 8 real production keys (see
tests/agent/test_zai_8keys_audit.py):
  - 6/8 keys: pure Anthropic-wire subscribers (Anthropic Messages)
  - 1/8 keys: pure OpenAI-wire (standard paas/v4)
  - 1/8 keys: full-stack (works on all 3 endpoints)

Live test audit (tests/agent/test_zai_live_audit.py) covers every
Z.AI error category against the real api.z.ai API: 200, 1305, 1308,
1113, 401, 402 - all classified correctly.

Before fix: detect_zai_endpoint() returned the wrong endpoint for
6/8 keys. After fix: 9/9 keys correctly routed and verified working.

== Test coverage ==

249 unit + integration tests, 21 live tests (opt-in via env var):

  tests/hermes_cli/test_zai_5gateway_extension.py      (29 tests)
    ZAI_ENDPOINTS structure, dispatch helpers, signature stability,
    backward compat, probe order pinning

  tests/hermes_cli/test_zai_config_yaml_precedence.py (8 tests)
    GLM_BASE_URL > model.base_url > cached > probe > default

  tests/agent/test_zai_manual_pool_routing.py         (7 tests)
    Runtime re-resolution, GLM_BASE_URL forwarding, probe failure
    tolerance, non-zai provider leak guard

  tests/agent/test_auxiliary_client_zai_payment_classification.py
                                                      (14 tests)
    _is_payment_error Z.AI Coding Plan exemption, verbatim Z.AI
    response bodies, negative cases for Vertex/Bedrock/OpenRouter

  tests/agent/test_zai_e2e_pool.py + test_zai_e2e_pool_rotation.py
                                                     (8 tests)
    Real HTTP round-trip via local mock Z.AI server, 5-key
    round_robin, revoked key, probe failure recovery

  tests/agent/test_zai_live.py + test_zai_live_audit.py +
  test_zai_8keys_audit.py                           (21 tests)
    Live against api.z.ai, opt-in via HERMES_RUN_LIVE=1 or
    GLM_AUDIT_KEYS env var. Keys masked to 8-char prefix in all
    output. File is safe to commit.

== Related work ==

  PR NousResearch#61492 - _is_payment_error() Z.AI exemption (cloned in Layer 1)
  PR NousResearch#55116 - vision helper coding endpoint (Layer 2)
  PR NousResearch#58088 - config.yaml base_url precedence (Layer 4)
  PR NousResearch#24915 - 4-variant provider split (intentionally NOT adopted:
    orthogonal refactor, deferred)
  PR NousResearch#54643 - /api/anthropic to /api/coding/paas/v4 rewrite
    (orthogonal; this PR probes the right wire, NousResearch#54643 rewrites at
    runtime if a user forces the OpenAI wire via config)
  PR NousResearch#55007 - stream probes without reading bodies (orthogonal;
    this PR can refactor _zai_probe_path to use httpx.stream() once
    NousResearch#55007 lands)
  PR NousResearch#61333 - skip /anthropic to /v1 rewrite (complementary)
  PR NousResearch#60753 - preserve /anthropic for custom vision (complementary)
  PR NousResearch#32174 - add zai-coding provider (different design choice)
  PR NousResearch#60034 - Z.AI Coding overload adaptive backoff (already merged)

== Related issues ==

  NousResearch#61487 - cascade _is_payment_error (Layer 1 closes)
  NousResearch#61563 - manual-pool routing audit (this PR implements)
  NousResearch#47970 - GLM-5.2 context_length fallback (Layer 5 helps)
  NousResearch#55112 - auxiliary vision hardcoded zai (Layer 5 helps)
  NousResearch#47685 - Hermes Agent prompt block on Z.ai (orthogonal)

== Test commands ==

  Unit + integration (no network):
    pytest tests/hermes_cli/test_api_key_providers.py \
           tests/hermes_cli/test_zai_config_yaml_precedence.py \
           tests/hermes_cli/test_zai_5gateway_extension.py \
           tests/agent/test_auxiliary_client.py::TestIsPaymentError \
           tests/agent/test_auxiliary_client_zai_payment_classification.py \
           tests/agent/test_zai_manual_pool_routing.py \
           tests/agent/test_zai_e2e_pool.py \
           tests/agent/test_zai_e2e_pool_rotation.py -v

  Live (consumes Z.AI quota, requires HERMES_RUN_LIVE=1):
    pytest tests/agent/test_zai_live.py \
           tests/agent/test_zai_live_audit.py \
           tests/agent/test_zai_8keys_audit.py -v --runlive

  Cross-platform check:
    scripts/check-windows-footguns.py --diff upstream/main

== Security ==

  - All live test keys read from env var (GLM_TEST_KEYS, GLM_AUDIT_KEYS,
    GLM_WORKING_KEY) NEVER from files. Files are safe to commit
    publicly.
  - All keys masked to 8-char prefix in any printed output.
  - No hardcoded credentials, no path leaks, no LAN IPs.
  - Privacy scan: 0 secrets, 0 private paths in diff.

Co-authored-by: Hermes triage bot <bot@nousresearch.com>
Refs: NousResearch#61487, NousResearch#61563, PR NousResearch#61492, PR NousResearch#55116, PR NousResearch#58088, PR NousResearch#24915,
      PR NousResearch#54643, PR NousResearch#55007, PR NousResearch#61333, PR NousResearch#60753, PR NousResearch#32174,
      PR NousResearch#60034, NousResearch#47970, NousResearch#55112, NousResearch#47685
@DeamonDev888

Copy link
Copy Markdown

Superseded by #62467. Your vision coding-endpoint ordering fix is now handled by the unified resolver Step 4 (provider-specific resolver) which probes coding endpoints before metered for all Z.AI calls, including vision. Credit for identifying the ordering issue.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for isolating the Z.AI auxiliary-vision routing issue. The current fallback list still begins with metered endpoints at agent/auxiliary_client.py:5434-5437, so the proposed ordering change addresses a live defect.

Problems

  • The modified open.bigmodel.cn entry is misindented in the PR diff; align it with the neighboring list entries.
  • Please add a regression test for the ordering. Current searches found no test for zai_openai_urls or the Z.AI vision fallback sequence; agent/auxiliary_client.py:5438-5447 is the behavior under change.

Suggested changes

  • Mock _get_cached_client() and assert that the first attempted base URL is https://api.z.ai/api/coding/paas/v4.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
@alt-glitch alt-glitch removed the sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades label Jul 15, 2026
@Caiofwp

Caiofwp commented Aug 12, 2026

Copy link
Copy Markdown

Gentle bump (2026-08-12)

Issue #55112 is still reproducible on current main: the Z.AI vision fallback list in agent/auxiliary_client.py ~L7145–7148 remains metered-only (open.bigmodel.cn/api/paas/v4 + api.z.ai/api/paas/v4); coding-plan is still missing.

This PR is still the smallest fix for that root cause. Earlier feedback that is still outstanding (from review / nit):

  1. Align indentation on the open.bigmodel.cn list entry (12 spaces like the siblings).
  2. Add a regression test that mocks _get_cached_client() and asserts the first attempted base URL is https://api.z.ai/api/coding/paas/v4.

@Morad37 — if you are busy, I am happy to open a tiny follow-up (or take over the branch with credit) to land (1)+(2) so this can merge cleanly. Maintainers: if you prefer a different vehicle (#62080 bundle / #74071), a quick signal would help avoid further duplicate effort.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/billing Account usage, credit usage, billing (cross-cutting) comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/zai ZAI provider sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: hardcoded zai vision base_urls (paas/v4) bypass the coding-plan endpoint → silent metered billing

5 participants