Skip to content

fix(zai): stream endpoint probes without reading bodies - #55007

Open
ooiuuii wants to merge 1 commit into
NousResearch:mainfrom
ooiuuii:fix/zai-endpoint-probe-stream
Open

fix(zai): stream endpoint probes without reading bodies#55007
ooiuuii wants to merge 1 commit into
NousResearch:mainfrom
ooiuuii:fix/zai-endpoint-probe-stream

Conversation

@ooiuuii

@ooiuuii ooiuuii commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Closes #55006

Summary

  • Switch detect_zai_endpoint() from non-streaming httpx.post(...) to httpx.stream(...).
  • Keep the existing status-only endpoint detection behavior while closing probe responses without reading bodies.
  • Add focused tests proving successful and non-200 probes do not consume response bodies.

Context

This mirrors the endpoint-probe boundary fixed in OpenClaw openclaw/openclaw#97540, adapted to Hermes' simpler Z.AI detection path.

Hermes only needs status_code for this probe. It does not parse error JSON, so the safest and smallest fix is to avoid response-body reads entirely rather than add a body parser/cap.

Duplicate Audit

Live GitHub searches found no matching open Hermes issue or PR for Z.AI endpoint-probe response-body reads:

  • ZAI endpoint probe response
  • Z.AI endpoint probe body
  • broad open PR/issue scans containing zai|z.ai|probe|endpoint|response|body|read|bound|cap

Related response-cap PRs exist for other providers and surfaces; #54723/#54671 is xAI OAuth fallback activation and unrelated to Z.AI endpoint probing.

Tests

  • uv run --extra dev python -m pytest tests\hermes_cli\test_zai_endpoint_probe.py -q --basetemp .pytest-tmp-zai-endpoint-probe-stream
  • uv run --extra dev python -m ruff check hermes_cli\auth.py tests\hermes_cli\test_zai_endpoint_probe.py
  • git diff --check

Autoreview was not run because this worktree does not contain .agents/skills/autoreview/scripts/autoreview.

@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard provider/zai ZAI provider area/auth Authentication, OAuth, credential pools P3 Low — cosmetic, nice to have labels Jun 29, 2026
@ooiuuii
ooiuuii force-pushed the fix/zai-endpoint-probe-stream branch from 87b2a30 to 2d09123 Compare June 29, 2026 22:24
@DeamonDev888

DeamonDev888 commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown

👋 Superseded — please see the new PR

@ooiuuii thanks for the perf optimization. #62080 is the new rebased PR that closes PR #61575.

Your stream-probe optimization is orthogonal to #62080: #62080 adds body-shape dispatch helpers (_zai_probe_path / _zai_probe_body) that can be easily refactored to use httpx.stream() once #55007 lands. Both PRs are good and they compose cleanly.

This was referenced Jul 9, 2026
Closed
Open
Open
DeamonDev888 pushed a commit to DeamonDev888/hermes-agent that referenced this pull request Jul 11, 2026
DeamonDev888 Hermes triage bot
feat(agent+auth): comprehensive Z.AI Coding Plan routing fix
ae1ea05 
End-to-end fix for the Z.AI credential pool cascade bug discovered
during a live audit of 8 production keys. Five coordinated layers
ensure the right endpoint is selected, the wrong error is not treated
as a payment error, and the user-configured overrides work end-to-end.

== Problem statement ==

Z.AI Coding Plan subscriptions (id.secret.* format) authenticate on
/api/coding/paas/v4 but the metered /api/paas/v4 endpoint also returns
HTTP 200 for glm-5. Without explicit routing, the metered endpoint
gets cached in auth.json and Coding Plan traffic silently draws from
the metered billing pool instead of the subscription.

A second symptom: Z.AI Coding Plan error codes 1113 (Insufficient
balance or no resource package) and 1308 (Usage limit reached for 5
hour) are per-key rolling quotas. They were misclassified as payment
errors via substring traps intended for Vertex AI (resource exhausted)
and Nous Portal (reached your session usage limit). A single exhausted
key cascade-marked every other key in the pool, taking the whole
provider offline.

A third gap: keys on the Anthropic Messages wire (/api/anthropic) were
not in ZAI_ENDPOINTS at all, so detect_zai_endpoint() returned the
wrong URL for any Anthropic-wire Coding Plan key.

== Solution overview ==

Five layers, all coordinated, all tested end-to-end:

  Layer 1 - _is_payment_error() exemption for Z.AI Coding Plan
    Detects api.z.ai / z.ai/api/coding / zhipuai+coding context plus
    codes 1113, 1308 and message patterns. Returns False so the
    credential pool rotation layer handles per-key 5h rolling quotas
    correctly. Real Z.AI payment errors (e.g. code 1311 plan-block)
    continue to flow through the normal payment-fallback path.

  Layer 2 - Vision auto-detect zai_openai_urls ordering
    The vision helper list now probes Coding Plan endpoints (global +
    China) before the metered paas/v4 fallbacks. Coding Plan keys
    authenticate on first vision call. Corrects an indentation bug in
    the original PR NousResearch#55116.

  Layer 3 - Runtime pool base_url re-resolution
    _resolve_api_key_provider() now re-invokes _resolve_zai_base_url()
    for provider_id == zai so manual-pool entries added via
    her mes auth add honor the cached detected_endpoint state in
    auth.json and the GLM_BASE_URL env override. Brings manual-pool
    parity with the env-seeded path fixed in commit 9e84416.
    Probe failures are caught and logged so they never break pool
    selection.

  Layer 4 - config.yaml model.base_url precedence
    New _configured_zai_base_url() helper reads model.base_url from
    config.yaml when model.provider is a Z.AI alias (zai / glm / z-ai /
    z.ai / zhipu). The precedence chain in _resolve_zai_base_url() is:
        1. GLM_BASE_URL env var (highest)
        2. model.base_url from config.yaml (when provider is Z.AI)
        3. cached detected_endpoint in auth.json
        4. live probe of all candidate endpoints
        5. registry default
    The provider alias guard prevents leakage from non-Z.AI configs.

  Layer 5 - Anthropic-wire endpoints in ZAI_ENDPOINTS
    Extended from 4 to 6 candidates with anthropic-global and
    anthropic-cn. coding-global is probed FIRST (99% of keys that
    accept coding endpoint also accept metered, so probing coding
    first caches the right URL on first try). anthropic-global is
    position 2 (fallthrough for pure Anthropic-wire keys).
    New _zai_probe_path() and _zai_probe_body() helpers dispatch the
    right HTTP body shape: Anthropic Messages for anthropic-* ids
    (/v1/messages, no stream field, anthropic-version header) and
    OpenAI chat completions for everything else.

== Audit results ==

Validated against 8 real production keys (see
tests/agent/test_zai_8keys_audit.py):
  - 6/8 keys: pure Anthropic-wire subscribers (Anthropic Messages)
  - 1/8 keys: pure OpenAI-wire (standard paas/v4)
  - 1/8 keys: full-stack (works on all 3 endpoints)

Live test audit (tests/agent/test_zai_live_audit.py) covers every
Z.AI error category against the real api.z.ai API: 200, 1305, 1308,
1113, 401, 402 - all classified correctly.

Before fix: detect_zai_endpoint() returned the wrong endpoint for
6/8 keys. After fix: 9/9 keys correctly routed and verified working.

== Test coverage ==

249 unit + integration tests, 21 live tests (opt-in via env var):

  tests/hermes_cli/test_zai_5gateway_extension.py      (29 tests)
    ZAI_ENDPOINTS structure, dispatch helpers, signature stability,
    backward compat, probe order pinning

  tests/hermes_cli/test_zai_config_yaml_precedence.py (8 tests)
    GLM_BASE_URL > model.base_url > cached > probe > default

  tests/agent/test_zai_manual_pool_routing.py         (7 tests)
    Runtime re-resolution, GLM_BASE_URL forwarding, probe failure
    tolerance, non-zai provider leak guard

  tests/agent/test_auxiliary_client_zai_payment_classification.py
                                                      (14 tests)
    _is_payment_error Z.AI Coding Plan exemption, verbatim Z.AI
    response bodies, negative cases for Vertex/Bedrock/OpenRouter

  tests/agent/test_zai_e2e_pool.py + test_zai_e2e_pool_rotation.py
                                                     (8 tests)
    Real HTTP round-trip via local mock Z.AI server, 5-key
    round_robin, revoked key, probe failure recovery

  tests/agent/test_zai_live.py + test_zai_live_audit.py +
  test_zai_8keys_audit.py                           (21 tests)
    Live against api.z.ai, opt-in via HERMES_RUN_LIVE=1 or
    GLM_AUDIT_KEYS env var. Keys masked to 8-char prefix in all
    output. File is safe to commit.

== Related work ==

  PR NousResearch#61492 - _is_payment_error() Z.AI exemption (cloned in Layer 1)
  PR NousResearch#55116 - vision helper coding endpoint (Layer 2)
  PR NousResearch#58088 - config.yaml base_url precedence (Layer 4)
  PR NousResearch#24915 - 4-variant provider split (intentionally NOT adopted:
    orthogonal refactor, deferred)
  PR NousResearch#54643 - /api/anthropic to /api/coding/paas/v4 rewrite
    (orthogonal; this PR probes the right wire, NousResearch#54643 rewrites at
    runtime if a user forces the OpenAI wire via config)
  PR NousResearch#55007 - stream probes without reading bodies (orthogonal;
    this PR can refactor _zai_probe_path to use httpx.stream() once
    NousResearch#55007 lands)
  PR NousResearch#61333 - skip /anthropic to /v1 rewrite (complementary)
  PR NousResearch#60753 - preserve /anthropic for custom vision (complementary)
  PR NousResearch#32174 - add zai-coding provider (different design choice)
  PR NousResearch#60034 - Z.AI Coding overload adaptive backoff (already merged)

== Related issues ==

  NousResearch#61487 - cascade _is_payment_error (Layer 1 closes)
  NousResearch#61563 - manual-pool routing audit (this PR implements)
  NousResearch#47970 - GLM-5.2 context_length fallback (Layer 5 helps)
  NousResearch#55112 - auxiliary vision hardcoded zai (Layer 5 helps)
  NousResearch#47685 - Hermes Agent prompt block on Z.ai (orthogonal)

== Test commands ==

  Unit + integration (no network):
    pytest tests/hermes_cli/test_api_key_providers.py \
           tests/hermes_cli/test_zai_config_yaml_precedence.py \
           tests/hermes_cli/test_zai_5gateway_extension.py \
           tests/agent/test_auxiliary_client.py::TestIsPaymentError \
           tests/agent/test_auxiliary_client_zai_payment_classification.py \
           tests/agent/test_zai_manual_pool_routing.py \
           tests/agent/test_zai_e2e_pool.py \
           tests/agent/test_zai_e2e_pool_rotation.py -v

  Live (consumes Z.AI quota, requires HERMES_RUN_LIVE=1):
    pytest tests/agent/test_zai_live.py \
           tests/agent/test_zai_live_audit.py \
           tests/agent/test_zai_8keys_audit.py -v --runlive

  Cross-platform check:
    scripts/check-windows-footguns.py --diff upstream/main

== Security ==

  - All live test keys read from env var (GLM_TEST_KEYS, GLM_AUDIT_KEYS,
    GLM_WORKING_KEY) NEVER from files. Files are safe to commit
    publicly.
  - All keys masked to 8-char prefix in any printed output.
  - No hardcoded credentials, no path leaks, no LAN IPs.
  - Privacy scan: 0 secrets, 0 private paths in diff.

Co-authored-by: Hermes triage bot <bot@nousresearch.com>
Refs: NousResearch#61487, NousResearch#61563, PR NousResearch#61492, PR NousResearch#55116, PR NousResearch#58088, PR NousResearch#24915,
      PR NousResearch#54643, PR NousResearch#55007, PR NousResearch#61333, PR NousResearch#60753, PR NousResearch#32174,
      PR NousResearch#60034, NousResearch#47970, NousResearch#55112, NousResearch#47685
@DeamonDev888 DeamonDev888 mentioned this pull request Jul 11, 2026
Open
@DeamonDev888

DeamonDev888 commented Jul 11, 2026

Copy link
Copy Markdown

Orthogonal optimization: our PR #62467 does not change probe streaming behavior. Your optimization (stream probes without reading bodies) can stack on top of our resolver. We reference this PR in our related work section.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users area/streaming Streaming responses: gateway delivery, provider wire labels Jul 15, 2026
@GottZ GottZ mentioned this pull request Jul 28, 2026
Open
@ooiuuii
ooiuuii force-pushed the fix/zai-endpoint-probe-stream branch from 2d09123 to 7b78469 Compare August 13, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/auth Authentication, OAuth, credential pools area/streaming Streaming responses: gateway delivery, provider wire comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have provider/zai ZAI provider sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Z.AI endpoint probes load response bodies even though only status is needed

4 participants

@ooiuuii @DeamonDev888 @alt-glitch @teknium1