perf(zai): parallelize endpoint detection probes (salvage #7821) - #77602
Merged
kshitijk4poor merged 3 commits intoAug 3, 2026
Merged
Conversation
Z.AI has separate billing for general vs coding plans and global vs
China endpoints. On startup, detect_zai_endpoint() probes up to 4
endpoints sequentially with 8s timeout each, taking 8-9 seconds when
the first endpoints return non-200 (rate limited) before a working one
is found.
Replace the sequential loop with concurrent.futures.ThreadPoolExecutor
to probe all 4 endpoints in parallel. Results are returned in
ZAI_ENDPOINTS priority order so the preference chain is preserved.
Benchmark on macOS M4 Max, Python 3.11, Hermes v0.8.0:
Before: 8.8s (sequential: global=0.9s/429, cn=1.6s/429,
coding-global=4.3s/200, coding-cn=2.0s/200)
After: ~4.5s (single round-trip, bounded by slowest endpoint)
Signed-off-by: Merlin <merlin@merlin.me>
Rebase fold: the original PR predates ZAI_ENDPOINTS growing per-endpoint probe_models lists; the parallel worker now preserves that candidate-model fallback loop (was: scalar model). Tests (both mutation-checked): - candidate-model fallback within one endpoint worker - ZAI_ENDPOINTS priority order wins over completion order - all-fail returns None
kshitijk4poor
enabled auto-merge (rebase)
August 3, 2026 11:34
kshitijk4poor
disabled auto-merge
August 3, 2026 11:34
alt-glitch
added
type/perf
…fy finding) The as_completed drain + `with` join made the parallel version WORSE than sequential main in the common case (first endpoint succeeds fast, others slow/unreachable): main returned at first success, the parallel version waited for every straggler. Now: after each completion, walk endpoints in priority order and return as soon as a success is unbeatable (all higher-priority probes already finished); pool uses shutdown(wait=False) so losers drain in the background. Mutation-checked: removing the early exit makes the new timing test fail (8.2s vs <1.5s).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Salvage of #7821 by @light-merlin-dark — original commit cherry-picked (authorship preserved) + a rebase-fold commit adapting it to current main.
What this does for users
hermes setup/ first-start with a Z.AI key probes up to 4 endpoints (global/cn × standard/coding-plan) to find which one accepts the key — sequentially on main, so a key that only works on the LAST endpoint waits through every earlier probe (and each probe can burn up to the 8s timeout on slow/unreachable regional endpoints). This parallelizes the probes with one worker per endpoint while keeping selection DETERMINISTIC: first match in ZAI_ENDPOINTS priority order wins, not completion order.Rebase fold (the sweeper's review point, addressed)
Current main's
ZAI_ENDPOINTSthird field is aprobe_modelsLIST (candidate-model fallback per endpoint); the original PR predates that and treated it as one scalar model. The fold restores the per-endpoint candidate loop inside each worker — behavior-identical to main's fallback semantics, just parallel across endpoints.Measured impact
Mocked httpx.post with 200ms per probe, only the last-priority endpoint's first model succeeding (worst case for sequential), median of 3:
Honest caveat: mocked RTT, and the win scales with per-probe latency (real WAN probes to bigmodel.cn from outside China regularly hit multi-second latencies or the full 8s timeout, where parallel ≈ max(probe) instead of sum(probes) — up to ~4× there); with a fast key on the FIRST endpoint, sequential was already optimal and the parallel version is a wash. The probe only runs when the endpoint isn't already cached in auth.json (keyed on key hash), so this is a first-run/setup-path win, not per-turn.
Verification
tests/hermes_cli/test_api_key_providers.py), incl. 3 new testsCloses #7821.
Post-review fold (simplify pass)
The efficiency reviewer caught a real regression vs main in the COMMON case: the as_completed drain +
with-join waited for ALL probes even after the highest-priority endpoint had already won, while sequential main returned at first success. Folded an early exit (return as soon as a success is unbeatable in priority order; pool shutdown(wait=False) so losers drain in the background). Mutation-checked: removing the early exit fails the new timing test (8.2s vs <1.5s). Worst case (last-priority endpoint wins) re-measured unchanged: 0.81s vs main's 1.42s.