Skip to content

perf(zai): parallelize endpoint detection probes - #7821

Closed
light-merlin-dark wants to merge 1 commit into
NousResearch:mainfrom
light-merlin-dark:perf/parallel-zai-endpoint-probe
Closed

perf(zai): parallelize endpoint detection probes#7821
light-merlin-dark wants to merge 1 commit into
NousResearch:mainfrom
light-merlin-dark:perf/parallel-zai-endpoint-probe

Conversation

@light-merlin-dark

@light-merlin-dark light-merlin-dark commented Apr 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Z.AI has separate billing for general vs coding plans and global vs China endpoints. On startup, detect_zai_endpoint() probes up to 4 endpoints sequentially with 8s timeout each.

When the first endpoints return non-200 (e.g. rate-limited 429), the probe walks through all of them before finding a working one. This adds 8-9 seconds to every cold startup where the endpoint cache doesn't exist.

Solution

Replace the sequential for loop with concurrent.futures.ThreadPoolExecutor to probe all 4 endpoints in parallel. Results are collected and returned in ZAI_ENDPOINTS priority order so the preference chain (global → cn → coding-global → coding-cn) is preserved.

Metrics

Measured on macOS M4 Max, Python 3.11, Hermes v0.8.0:

Endpoint Sequential timing Status
global 0.9s 429 (rate limited)
cn 1.6s 429 (rate limited)
coding-global 4.3s 200 ✓
coding-cn 2.0s 200 ✓
Before After
Total probe time 8.8s ~4.5s

The parallel version is bounded by the slowest single endpoint rather than the sum of all endpoints.

Files changed

  • hermes_cli/auth.pydetect_zai_endpoint() now uses ThreadPoolExecutor; extracted _probe_single_zai_endpoint() helper.

Merlin (@EnchantedRobot on X) & GLM 5.1 via OpenCode

Z.AI has separate billing for general vs coding plans and global vs
China endpoints. On startup, detect_zai_endpoint() probes up to 4
endpoints sequentially with 8s timeout each, taking 8-9 seconds when
the first endpoints return non-200 (rate limited) before a working one
is found.

Replace the sequential loop with concurrent.futures.ThreadPoolExecutor
to probe all 4 endpoints in parallel. Results are returned in
ZAI_ENDPOINTS priority order so the preference chain is preserved.

Benchmark on macOS M4 Max, Python 3.11, Hermes v0.8.0:
  Before: 8.8s (sequential: global=0.9s/429, cn=1.6s/429,
           coding-global=4.3s/200, coding-cn=2.0s/200)
  After:  ~4.5s (single round-trip, bounded by slowest endpoint)

Signed-off-by: Merlin <merlin@merlin.me>
@alt-glitch alt-glitch added type/perf Performance improvement or optimization P3 Low — cosmetic, nice to have provider/zai ZAI provider labels Apr 29, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for targeting a real cold-start latency path: current main still probes synchronously in hermes_cli/auth.py:639-667.

Problems

  • Current main changed ZAI_ENDPOINTS so its third field is probe_models, a list (hermes_cli/auth.py:623-628). The proposed helper treats that field as one scalar model (hermes_cli/auth.py:392) and posts it directly (hermes_cli/auth.py:401), which would send a list and lose the current fallback across candidate models.
  • There is no direct probe test. Existing Z.AI tests monkeypatch detect_zai_endpoint() at the credential-resolution boundary (tests/hermes_cli/test_api_key_providers.py:963-1007).

Suggested changes

  • Salvage as one worker per endpoint that preserves the current per-endpoint candidate-model loop, then select successful endpoint results in ZAI_ENDPOINTS priority order.
  • Add mocked coverage for candidate fallback and deterministic endpoint-priority selection.

Automated hermes-sweeper review.

Comment thread hermes_cli/auth.py
api_key: str, endpoint: tuple, timeout: float,
) -> Optional[Dict[str, str]]:
"""Probe a single Z.AI endpoint. Returns endpoint info dict or None."""
ep_id, base_url, model, label = endpoint

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current main's third tuple element is probe_models, a list (hermes_cli/auth.py:623-628), not a scalar. A salvage of this helper would send that list as json["model"] and skip the existing candidate-model fallback. Please make the worker iterate each endpoint's candidate models and return the first successful scalar model.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low — cosmetic, nice to have provider/zai ZAI provider sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/perf Performance improvement or optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants