Skip to content

perf: parallelize provider model-list fetches in model picker - #80415

Closed
lepetitprince716-prog wants to merge 1 commit into
NousResearch:mainfrom
lepetitprince716-prog:perf/parallel-model-picker
Closed

lepetitprince716-prog wants to merge 1 commit into
NousResearch:mainfrom
lepetitprince716-prog:perf/parallel-model-picker

Conversation

@lepetitprince716-prog

Copy link
Copy Markdown
Contributor

Summary

When the 1h provider_models_cache.json TTL lapses, list_authenticated_providers() fetches each authed provider's /v1/models endpoint serially. With 10+ providers this stacks to 15-30s of blocking before the picker renders. This PR adds a parallel prefetch step that warms stale cache entries concurrently before the serial picker loops start.

Closes #80413.

Complementary to #72762 (which fixes credential-pool memoization and Copilot token exchange). Even with #72762 fixed, the serial /v1/models round-trips remain when the cache is stale.

Changes

hermes_cli/models.py

  • update_provider_cache_entry() — thread-safe single-entry cache writer using threading.Lock. Prevents concurrent read-modify-write races on the shared provider_models_cache.json when multiple prefetch workers persist their results simultaneously.
  • _cache_write_lock — module-level lock guarding all cache entry updates.

hermes_cli/model_switch.py

  • _collect_authed_provider_slugs() — lightweight credential pre-scan mirroring the credential-check logic from sections 1/2/2b of list_authenticated_providers(), but never fetches model lists. Returns provider slugs that have credentials.
  • _prefetch_provider_models_parallel() — for each stale/missing provider, calls cached_provider_model_ids(slug, force_refresh=True) concurrently via ThreadPoolExecutor (max 8 workers). Fresh entries are skipped. Each worker uses update_provider_cache_entry() for thread-safe persistence.
  • Integration in list_authenticated_providers() — after data and curated are built but before the serial section-1 loop, the prefetch runs if >3 providers are authed and refresh=False.

tests/hermes_cli/test_model_cache_parallel_prefetch.py (new, 12 tests)

  • TestUpdateProviderCacheEntry — thread-safe write, no-clobber, concurrent writes (4 tests)
  • TestPrefetchProviderModelsParallel — skip fresh, fetch stale, parallelism verification, exception swallowing, empty list (5 tests)
  • TestPrefetchIntegration — prefetch called with >3 providers, skipped with ≤3, skipped on refresh (3 tests)

Guardrails

  • Skipped when ≤3 authed providers — thread-pool overhead not worth it for the common 1-2 provider case
  • Skipped when refresh=True — the serial path already force-refreshes
  • Exception-isolated — any prefetch failure silently falls back to the existing serial path
  • No behavioral change — same model lists, same picker output, just faster

Test Results

# New tests + existing SWR tests
21 passed in 28.00s

# Existing model_switch + picker tests (no regressions)
45 passed in 42.54s
74 passed in 102.76s

# Total: 140 tests, 0 failures

Expected Impact

For a 13-provider setup with expired cache:

  • Before: ~20-30s serial blocking (sum of all provider latencies)
  • After: ~8s parallel (max single provider latency), rest served from warm cache

When the 1h provider_models_cache.json TTL lapses, the model picker
serially fetches /v1/models for each authenticated provider. With 10+
providers this stacks to 15-30s of blocking before the picker renders.

Add a parallel prefetch step before the serial picker build loops:
- _collect_authed_provider_slugs(): lightweight credential pre-scan
  that mirrors sections 1/2/2b without fetching model lists
- _prefetch_provider_models_parallel(): ThreadPoolExecutor-based
  concurrent fetch of stale/missing cache entries (max 8 workers)
- update_provider_cache_entry(): thread-safe single-entry cache writer
  with threading.Lock to prevent concurrent write races

Guardrails:
- Skipped when <=3 authed providers (overhead not worth it)
- Skipped when refresh=True (serial path force-refreshes)
- Exception-isolated (falls back to serial path on any failure)
- No behavioral change (same model lists, same picker output)

Closes NousResearch#80413
@alt-glitch alt-glitch added type/perf Performance improvement or optimization comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have labels Aug 6, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for this fix! It was salvaged into #86764 (cherry-picked onto current main with your authorship preserved in the commit history) and is now merged. Closing since the work has landed.

@teknium1 teknium1 closed this Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/perf Performance improvement or optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: parallelize provider model-list fetches in model picker (serial /v1/models bottleneck)

3 participants