perf: salvage query/serialization cluster — session-search I/O, adaptive hydration, Anthropic client reuse, parallel model fetches - #86764
Merged
Conversation
Every search route in _search_messages_impl (FTS, CJK bigram, trigram, LIKE fallback, rebuild-gap supplement) selected m.content, then the result tail popped it unread. On DBs with multi-MB tool rows, each search read and materialized up to `limit` full rows only to discard them. Snippets come from snippet()/substr() in SQL and the context window is re-fetched by id, so no code path ever read the column. Drop the column from all six SELECT lists. Returned dicts are unchanged: content was never part of the public result (the pop ran before return), and tests/test_hermes_state.py already documents that contract. (cherry picked from commit d0c3af1)
(cherry picked from commit 60a3530)
(cherry picked from commit 5f6de98)
(cherry picked from commit c9b1286)
(cherry picked from commit 7359220)
…rebuilding it per call _create_request_anthropic_client() built a fresh anthropic.Anthropic client (and httpx pool) on every single LLM call, and _close_request_anthropic_client() always fully closed it right after - unlike the OpenAI-wire path, which caches and reuses one warm client across sequential calls via a single-slot cache keyed on the effective client kwargs. Add the same single-slot cache to the Anthropic-wire path: keyed on credentials, base URL/Bedrock region, per-model timeout, and the 1M-beta flag; in_use guards concurrent calls from sharing one pool's close/abort lifecycle; poisoned marks a cross-thread-aborted slot so the owner-thread close discards it; reuse only on request_complete / stream_request_complete (the same _REQUEST_CLIENT_REUSE_REASONS the OpenAI path already uses). Wires a teardown hook into release_clients()/close() mirroring _close_cached_request_openai_client. Fixes #HPA-02 (cherry picked from commit 37f90df)
When the 1h provider_models_cache.json TTL lapses, the model picker serially fetches /v1/models for each authenticated provider. With 10+ providers this stacks to 15-30s of blocking before the picker renders. Add a parallel prefetch step before the serial picker build loops: - _collect_authed_provider_slugs(): lightweight credential pre-scan that mirrors sections 1/2/2b without fetching model lists - _prefetch_provider_models_parallel(): ThreadPoolExecutor-based concurrent fetch of stale/missing cache entries (max 8 workers) - update_provider_cache_entry(): thread-safe single-entry cache writer with threading.Lock to prevent concurrent write races Guardrails: - Skipped when <=3 authed providers (overhead not worth it) - Skipped when refresh=True (serial path force-refreshes) - Exception-isolated (falls back to serial path on any failure) - No behavioral change (same model lists, same picker output) Closes #80413 (cherry picked from commit 89dddd6)
… wrapper Main extracted a session_search() wrapper (owned-DB lifecycle) around _session_search_impl after #82595 was opened; the cherry-picked detail parameter landed on the impl only. Append it to the wrapper with the same positional-compatibility contract and pass it through.
alt-glitch
added
type/perf
Contributor
૮ >ﻌ< ა ci reviewran on 9f2b8af — chore(contributors): map lepetitprince716@gmail.com -> lepet
|
This was referenced Aug 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Salvages the wave-2 query/serialization hot-path cluster: four contributor perf PRs covering session-search I/O, discovery hydration, Anthropic client reuse, and parallel model-list fetches.
Changes
m.contentonly for the result tail to pop it unread; on DBs with multi-MB tool rows every search materialized up tolimitfull rows for nothing. Reapplied onto main's refactored_finalize_search_matches(7 SELECT sites); the defensivepop("content")stays as a guard. Public results are unchanged —contentwas never a returned field.detail="full"restores full hydration everywhere. Thedetailparameter is appended after all existing parameters (positional-compatibility test included), and a follow-up commit forwards it through main's new owned-DBsession_search()wrapper.anthropic.Anthropicwas rebuilt per call (fresh httpx pool + TCP/TLS handshake). Now a single-slot cache mirrors the OpenAI-wire path: keyed on credentials, base URL/region, timeout, and the 1M-context beta flag, within_use/poisoned/closed guards so credential rotation, model switches, or cross-thread aborts always force a fresh client./v1/modelsserially (15–30s with 10+ providers). A pre-scan collects credentialed provider slugs without fetching, then warms stale cache entries via an 8-worker pool with a thread-safe single-entry cache writer; fresh entries and ≤3-provider setups skip the pool entirely. During rebase the pre-scan's rawos.environcredential probes were routed through_scoped_key_env()to respect main's per-profile secret-scope sweep (854007d).Excluded
pyproject.tomlhas no orjson entry); the task brief said to skip in that case rather than add a new dependency.Validation
tests/tools/test_session_search.pytests/test_hermes_state.pytests/run_agent/test_token_persistence_non_cli.pytests/agent/test_anthropic_request_client_reuse.pytests/hermes_cli/test_model_cache_parallel_prefetch.pyPrompt-cache/invariant review: none of these touch conversation context, message alternation, or the system prompt. The Anthropic client cache key covers every input that must force a rebuild (credential rotation, base URL/region, timeout, 1M-beta flag).
Credits
Covers #77034, #82595, #77144, #80415. Fixes #80413.
Infographic