fix(model_metadata): eliminate idle Ollama API probes from gateway processes by stevenau21 · Pull Request #37905 · NousResearch/hermes-agent

stevenau21 · 2026-06-03T05:19:40Z

Problem

When Hermes gateways are idle (no active conversations), the gateway process still makes repeated HTTP calls to the Ollama API endpoint. This happens because:

probes Ollama via before checking hardcoded
has no in-memory cache — every call is a live HTTP POST to
has no cache — every call makes 1-4 live HTTP requests to the endpoint to determine server type

For users running Ollama on remote hosts (Tailscale, LAN), this produces ~3 API calls per 5 minutes even when the gateway is completely idle. The persistent disk cache () also fails when the user's config uses a different than what was originally cached (e.g., vs ).

Fix (3 commits)

Commit 1 — Stop idle Ollama API calls ()

Reorder to check before any live Ollama probe
Add in-memory TTL cache () for with 1-hour TTL
Cache 404/405 failures as so repeated probes against non-Ollama endpoints are suppressed

Commit 2 — Normalize cache base_url keys ()

Add to strip trailing slashes from cache keys
Integrate at 4 points: load, save, lookup, and in-memory cache
Prevents cache misses when config and cached keys differ by a trailing

Commit 3 — Cache ()

Add dict with seconds
Cache key = normalized
Server type never changes during a single process lifetime, so this is safe
Drops repeated server detection from 4 HTTP requests per call to zero on cache hit

Verification

Runtime test confirms:

Known models (, , ) → resolved from defaults, zero network calls
Unknown model → one probe, cached, subsequent calls hit cache
→ 138ms on first call, 0ms on cache hits

Impact

Idle gateways make zero Ollama API calls instead of ~3 per 5 minutes
Users with remote Ollama endpoints (Tailscale, VPN, LAN) see immediate benefit
No behavior change for models that require live Ollama detection
Cache TTL is defensive (1 hour); could be process lifetime since Ollama configs are static

- Reorder resolution: check DEFAULT_CONTEXT_LENGTHS before any live Ollama probe - Add in-memory TTL cache to _query_ollama_api_show (1 hour per model+URL) - Cache 404/405 failures too, so non-Ollama servers only get probed once - This stops idle gateways from making repeated /api/show calls to Ollama even when no conversation is active and all models are known families

…slash mismatches The persistent context_length cache and in-memory Ollama probe cache keyed entries as model@base_url. When base_url had a trailing slash (e.g. from Ollama's /api/tags response or manual config), the key never matched the config's no-slash URL, causing a cache miss and repeated idle probes. Changes: - Add _normalize_cache_base_url() to strip trailing slashes - Normalize on read in _load_context_cache() so old caches self-heal - Normalize on write in save_context_length() and get_cached_context_length() - Normalize in _query_ollama_api_show() in-memory cache key

…ma probes Adds in-memory TTL cache for detect_local_server_type() which was making 1-4 HTTP requests every call with zero caching. Called repeatedly during idle gateway operation. Cache key=base_url, TTL=1hr. Refs previous fix for _query_ollama_api_show cache.

@Morad37

…t-cache keys Follow-up hunks completing the probe-cache cluster: 1. _query_ollama_api_show now goes through the existing _LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) — it was the one remaining per-resolution POST not covered by the #56431-era wrapper. Failures are never memoized so a server that comes up mid-startup is re-probed. Idea credit: #42081 (@Morad37), reworked to comply with the positive-only rule. 2. Persistent context-cache keys are normalized through _context_cache_key (trailing-slash strip) so http://host/v1 and http://host/v1/ share one entry; reads and invalidation honor legacy un-normalized rows. Idea credit: #37905 (@stevenau21). Tests: TTL hit collapses to one POST, failure-not-memoized (mutation-verified: unconditional caching makes it fail), namespace no-collision vs the sibling probe, slash-variant dedup, legacy-row read, dual-shape invalidation.

kshitijk4poor · 2026-07-09T08:42:00Z

Thanks @stevenau21 — the idea from this PR landed! The probe-cache cluster has now landed on main via PR #61368 (merge commit f556edc), which salvaged this cluster of PRs onto current main with structured review, live smoke tests, and full test gates.

The key-normalization kernel from this PR was reimplemented against current main (the file had drifted substantially since this was opened): persistent context-cache keys are now normalized through a single _context_cache_key() chokepoint so /v1 and /v1/ share one row, with backward-compatible reads and invalidation for legacy un-normalized rows in both migration directions. You're credited in the commit message of 2bfc12b43.

Closing as superseded by the merged salvage.

@Morad37

…t-cache keys Follow-up hunks completing the probe-cache cluster: 1. _query_ollama_api_show now goes through the existing _LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) — it was the one remaining per-resolution POST not covered by the NousResearch#56431-era wrapper. Failures are never memoized so a server that comes up mid-startup is re-probed. Idea credit: NousResearch#42081 (@Morad37), reworked to comply with the positive-only rule. 2. Persistent context-cache keys are normalized through _context_cache_key (trailing-slash strip) so http://host/v1 and http://host/v1/ share one entry; reads and invalidation honor legacy un-normalized rows. Idea credit: NousResearch#37905 (@stevenau21). Tests: TTL hit collapses to one POST, failure-not-memoized (mutation-verified: unconditional caching makes it fail), namespace no-collision vs the sibling probe, slash-variant dedup, legacy-row read, dual-shape invalidation.

@Morad37

…t-cache keys Follow-up hunks completing the probe-cache cluster: 1. _query_ollama_api_show now goes through the existing _LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) — it was the one remaining per-resolution POST not covered by the NousResearch#56431-era wrapper. Failures are never memoized so a server that comes up mid-startup is re-probed. Idea credit: NousResearch#42081 (@Morad37), reworked to comply with the positive-only rule. 2. Persistent context-cache keys are normalized through _context_cache_key (trailing-slash strip) so http://host/v1 and http://host/v1/ share one entry; reads and invalidation honor legacy un-normalized rows. Idea credit: NousResearch#37905 (@stevenau21). Tests: TTL hit collapses to one POST, failure-not-memoized (mutation-verified: unconditional caching makes it fail), namespace no-collision vs the sibling probe, slash-variant dedup, legacy-row read, dual-shape invalidation.

@Morad37

…t-cache keys Follow-up hunks completing the probe-cache cluster: 1. _query_ollama_api_show now goes through the existing _LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) — it was the one remaining per-resolution POST not covered by the NousResearch#56431-era wrapper. Failures are never memoized so a server that comes up mid-startup is re-probed. Idea credit: NousResearch#42081 (@Morad37), reworked to comply with the positive-only rule. 2. Persistent context-cache keys are normalized through _context_cache_key (trailing-slash strip) so http://host/v1 and http://host/v1/ share one entry; reads and invalidation honor legacy un-normalized rows. Idea credit: NousResearch#37905 (@stevenau21). Tests: TTL hit collapses to one POST, failure-not-memoized (mutation-verified: unconditional caching makes it fail), namespace no-collision vs the sibling probe, slash-variant dedup, legacy-row read, dual-shape invalidation.

@Morad37

…t-cache keys Follow-up hunks completing the probe-cache cluster: 1. _query_ollama_api_show now goes through the existing _LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) — it was the one remaining per-resolution POST not covered by the NousResearch#56431-era wrapper. Failures are never memoized so a server that comes up mid-startup is re-probed. Idea credit: NousResearch#42081 (@Morad37), reworked to comply with the positive-only rule. 2. Persistent context-cache keys are normalized through _context_cache_key (trailing-slash strip) so http://host/v1 and http://host/v1/ share one entry; reads and invalidation honor legacy un-normalized rows. Idea credit: NousResearch#37905 (@stevenau21). Tests: TTL hit collapses to one POST, failure-not-memoized (mutation-verified: unconditional caching makes it fail), namespace no-collision vs the sibling probe, slash-variant dedup, legacy-row read, dual-shape invalidation.

@Morad37

…t-cache keys Follow-up hunks completing the probe-cache cluster: 1. _query_ollama_api_show now goes through the existing _LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) — it was the one remaining per-resolution POST not covered by the NousResearch#56431-era wrapper. Failures are never memoized so a server that comes up mid-startup is re-probed. Idea credit: NousResearch#42081 (@Morad37), reworked to comply with the positive-only rule. 2. Persistent context-cache keys are normalized through _context_cache_key (trailing-slash strip) so http://host/v1 and http://host/v1/ share one entry; reads and invalidation honor legacy un-normalized rows. Idea credit: NousResearch#37905 (@stevenau21). Tests: TTL hit collapses to one POST, failure-not-memoized (mutation-verified: unconditional caching makes it fail), namespace no-collision vs the sibling probe, slash-variant dedup, legacy-row read, dual-shape invalidation.

stevenau21 added 3 commits June 3, 2026 10:28

alt-glitch added type/perf Performance improvement or optimization comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/ollama Ollama / local models P3 Low — cosmetic, nice to have labels Jun 3, 2026

alt-glitch mentioned this pull request Jul 1, 2026

perf: cache detect_local_server_type() results per process lifetime #56139

Closed

jcjc81 mentioned this pull request Jul 7, 2026

fix(agent): also send chat_template_kwargs.enable_thinking=false for llama.cpp/vLLM #12427

Open

9 tasks

kshitijk4poor mentioned this pull request Jul 9, 2026

perf(model_metadata): salvage probe-cache cluster #61368

Merged

kshitijk4poor closed this Jul 9, 2026

Navigation Menu

Uh oh!

fix(model_metadata): eliminate idle Ollama API probes from gateway processes - #37905

fix(model_metadata): eliminate idle Ollama API probes from gateway processes#37905
stevenau21 wants to merge 3 commits into
NousResearch:mainfrom
stevenau21:fix/ollama-idle-probe-caching

stevenau21 commented Jun 3, 2026

Uh oh!

kshitijk4poor commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

stevenau21 commented Jun 3, 2026

Problem

Fix (3 commits)

Commit 1 — Stop idle Ollama API calls ()

Commit 2 — Normalize cache base_url keys ()

Commit 3 — Cache ()

Verification

Impact

Uh oh!

kshitijk4poor commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants