Skip to content

fix(model_metadata): cache detect_local_server_type result for process lifetime - #29986

Closed
uzaylisak wants to merge 2 commits into
NousResearch:mainfrom
uzaylisak:fix/endpoint-probe-path-cache
Closed

uzaylisak wants to merge 2 commits into
NousResearch:mainfrom
uzaylisak:fix/endpoint-probe-path-cache

Conversation

@uzaylisak

Copy link
Copy Markdown
Contributor

Problem

Every 5 minutes fetch_endpoint_model_metadata() is called (TTL = _ENDPOINT_MODEL_CACHE_TTL = 300). Each time it calls detect_local_server_type(), which blindly re-runs the full probe waterfall:

  1. GET /api/v1/models → 404 (LM Studio check)
  2. GET /api/tags → 404 (Ollama check)
  3. GET /v1/props → 404 (llama.cpp check)
  4. GET /version → 200 (vLLM — found)

On a vllm backend the first three always 404, polluting server logs every 5 minutes. Reported in #29971.

Fix

Add _endpoint_probe_path_cache: Dict[str, str] = {} at module level. detect_local_server_type() checks this cache on entry; on a hit it returns immediately without making any HTTP requests. On the first call the waterfall runs as before, and if a server type is identified it is stored for the lifetime of the process.

The probe stages also use if result is None guards so each step is skipped once an earlier one succeeds — reducing requests even on the very first call.

Impact

  • vllm / any server that is not LM Studio or Ollama: only 1 probe round-trip ever, vs. 4 per 5-minute refresh cycle previously.
  • LM Studio, Ollama, llama.cpp: one extra dict lookup per call; behaviour is identical.
  • Cache is intentionally process-scoped (not TTL-based) because a server type does not change during a session.

Closes #29971.

uzaylisak added 2 commits May 21, 2026 22:29
…s lifetime

Every 5 minutes the endpoint metadata refresh re-runs the full server-type
waterfall (LM Studio -> Ollama -> llama.cpp -> vLLM), spraying 404s at
endpoints the server never exposes (e.g. /api/v1/models and /api/tags on a
vllm backend).

Add _endpoint_probe_path_cache (base_url -> server type) so the first
successful probe's result is reused for the lifetime of the process.
Subsequent refreshes skip straight to the known-good path.

Fixes NousResearch#29971.
@uzaylisak
uzaylisak force-pushed the fix/endpoint-probe-path-cache branch from 29a402a to 869272c Compare May 21, 2026 19:30
@uzaylisak uzaylisak closed this May 21, 2026
@uzaylisak
uzaylisak deleted the fix/endpoint-probe-path-cache branch May 21, 2026 19:30
@alt-glitch alt-glitch added type/perf Performance improvement or optimization P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have type/perf Performance improvement or optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Hermes makes bogus calls to unknown endpoints every 5 minutes

2 participants