perf: parallelize provider model-list fetches in model picker - #80415
Closed
lepetitprince716-prog wants to merge 1 commit into
Closed
lepetitprince716-prog wants to merge 1 commit into
lepetitprince716-prog wants to merge 1 commit into
Conversation
When the 1h provider_models_cache.json TTL lapses, the model picker serially fetches /v1/models for each authenticated provider. With 10+ providers this stacks to 15-30s of blocking before the picker renders. Add a parallel prefetch step before the serial picker build loops: - _collect_authed_provider_slugs(): lightweight credential pre-scan that mirrors sections 1/2/2b without fetching model lists - _prefetch_provider_models_parallel(): ThreadPoolExecutor-based concurrent fetch of stale/missing cache entries (max 8 workers) - update_provider_cache_entry(): thread-safe single-entry cache writer with threading.Lock to prevent concurrent write races Guardrails: - Skipped when <=3 authed providers (overhead not worth it) - Skipped when refresh=True (serial path force-refreshes) - Exception-isolated (falls back to serial path on any failure) - No behavioral change (same model lists, same picker output) Closes NousResearch#80413
Collaborator
|
Thanks for this fix! It was salvaged into #86764 (cherry-picked onto current main with your authorship preserved in the commit history) and is now merged. Closing since the work has landed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the 1h
provider_models_cache.jsonTTL lapses,list_authenticated_providers()fetches each authed provider's/v1/modelsendpoint serially. With 10+ providers this stacks to 15-30s of blocking before the picker renders. This PR adds a parallel prefetch step that warms stale cache entries concurrently before the serial picker loops start.Closes #80413.
Complementary to #72762 (which fixes credential-pool memoization and Copilot token exchange). Even with #72762 fixed, the serial
/v1/modelsround-trips remain when the cache is stale.Changes
hermes_cli/models.pyupdate_provider_cache_entry()— thread-safe single-entry cache writer usingthreading.Lock. Prevents concurrent read-modify-write races on the sharedprovider_models_cache.jsonwhen multiple prefetch workers persist their results simultaneously._cache_write_lock— module-level lock guarding all cache entry updates.hermes_cli/model_switch.py_collect_authed_provider_slugs()— lightweight credential pre-scan mirroring the credential-check logic from sections 1/2/2b oflist_authenticated_providers(), but never fetches model lists. Returns provider slugs that have credentials._prefetch_provider_models_parallel()— for each stale/missing provider, callscached_provider_model_ids(slug, force_refresh=True)concurrently viaThreadPoolExecutor(max 8 workers). Fresh entries are skipped. Each worker usesupdate_provider_cache_entry()for thread-safe persistence.list_authenticated_providers()— afterdataandcuratedare built but before the serial section-1 loop, the prefetch runs if >3 providers are authed andrefresh=False.tests/hermes_cli/test_model_cache_parallel_prefetch.py(new, 12 tests)TestUpdateProviderCacheEntry— thread-safe write, no-clobber, concurrent writes (4 tests)TestPrefetchProviderModelsParallel— skip fresh, fetch stale, parallelism verification, exception swallowing, empty list (5 tests)TestPrefetchIntegration— prefetch called with >3 providers, skipped with ≤3, skipped on refresh (3 tests)Guardrails
refresh=True— the serial path already force-refreshesTest Results
Expected Impact
For a 13-provider setup with expired cache: