perf(zai): parallelize endpoint detection probes - #7821
Closed
light-merlin-dark wants to merge 1 commit into
Closed
Conversation
Z.AI has separate billing for general vs coding plans and global vs
China endpoints. On startup, detect_zai_endpoint() probes up to 4
endpoints sequentially with 8s timeout each, taking 8-9 seconds when
the first endpoints return non-200 (rate limited) before a working one
is found.
Replace the sequential loop with concurrent.futures.ThreadPoolExecutor
to probe all 4 endpoints in parallel. Results are returned in
ZAI_ENDPOINTS priority order so the preference chain is preserved.
Benchmark on macOS M4 Max, Python 3.11, Hermes v0.8.0:
Before: 8.8s (sequential: global=0.9s/429, cn=1.6s/429,
coding-global=4.3s/200, coding-cn=2.0s/200)
After: ~4.5s (single round-trip, bounded by slowest endpoint)
Signed-off-by: Merlin <merlin@merlin.me>
teknium1
reviewed
Jul 12, 2026
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for targeting a real cold-start latency path: current main still probes synchronously in hermes_cli/auth.py:639-667.
Problems
- Current main changed
ZAI_ENDPOINTSso its third field isprobe_models, a list (hermes_cli/auth.py:623-628). The proposed helper treats that field as one scalarmodel(hermes_cli/auth.py:392) and posts it directly (hermes_cli/auth.py:401), which would send a list and lose the current fallback across candidate models. - There is no direct probe test. Existing Z.AI tests monkeypatch
detect_zai_endpoint()at the credential-resolution boundary (tests/hermes_cli/test_api_key_providers.py:963-1007).
Suggested changes
- Salvage as one worker per endpoint that preserves the current per-endpoint candidate-model loop, then select successful endpoint results in
ZAI_ENDPOINTSpriority order. - Add mocked coverage for candidate fallback and deterministic endpoint-priority selection.
Automated hermes-sweeper review.
| api_key: str, endpoint: tuple, timeout: float, | ||
| ) -> Optional[Dict[str, str]]: | ||
| """Probe a single Z.AI endpoint. Returns endpoint info dict or None.""" | ||
| ep_id, base_url, model, label = endpoint |
Contributor
There was a problem hiding this comment.
Current main's third tuple element is probe_models, a list (hermes_cli/auth.py:623-628), not a scalar. A salvage of this helper would send that list as json["model"] and skip the existing candidate-model fallback. Please make the worker iterate each endpoint's candidate models and return the first successful scalar model.
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Z.AI has separate billing for general vs coding plans and global vs China endpoints. On startup,
detect_zai_endpoint()probes up to 4 endpoints sequentially with 8s timeout each.When the first endpoints return non-200 (e.g. rate-limited 429), the probe walks through all of them before finding a working one. This adds 8-9 seconds to every cold startup where the endpoint cache doesn't exist.
Solution
Replace the sequential
forloop withconcurrent.futures.ThreadPoolExecutorto probe all 4 endpoints in parallel. Results are collected and returned inZAI_ENDPOINTSpriority order so the preference chain (global → cn → coding-global → coding-cn) is preserved.Metrics
Measured on macOS M4 Max, Python 3.11, Hermes v0.8.0:
The parallel version is bounded by the slowest single endpoint rather than the sum of all endpoints.
Files changed
hermes_cli/auth.py—detect_zai_endpoint()now usesThreadPoolExecutor; extracted_probe_single_zai_endpoint()helper.Merlin (@EnchantedRobot on X) & GLM 5.1 via OpenCode