feat(agent): first-class llama.cpp provider with zero-config dashboard surfacing

[Abd0r]

What does this PR do?

Two coherent pieces of work landing together so a llama.cpp user has an end-to-end zero-config path:

A. First-class llama-cpp provider. Today users can already point Hermes at llama-server via --provider custom, but that path requires hand-setting OPENAI_BASE_URL, the model picker doesn't list it, and the dashboard has no way to surface a running server. This PR adds a proper llama-cpp provider plugin so the picker, the dashboard, --provider flag, HERMES_INFERENCE_PROVIDER, and config.yaml all know about it natively. Crucially, it includes a probe-and-surface block in list_authenticated_providers that picks up a running llama-server automatically — no env vars required.

B. scripts/start-llama-server.sh — explicit resubmission per @teknium1's invitation in #19796 (comment):

scripts/start-llama-server.sh was out of scope for the web-search story so it didn't land — happy to review that as a separate PR (or skill) if you want to resubmit.

The launcher's defaults (port 8088) line up with the new provider's default base URL, so the end-to-end flow becomes:

./scripts/start-llama-server.sh ~/models/your-model.gguf
hermes chat --provider llama-cpp

Related Issues / PRs

• Refs Feature: Local Model Setup Skill — Ollama, llama.cpp & vLLM Configuration Guide with Model Recommendations (inspired by Liquid AI LocalCowork) #523 — partially delivers Phase 2 (llama.cpp setup guide) and Phase 3 (auto-detection of running local servers, scoped to llama.cpp). Ollama and vLLM coverage from that issue's roadmap remain out of scope here.
• Refs [Bug]: Custom endpoint 400 error bug llama.cpp #10375 — sidesteps the custom-endpoint 400-error path by giving llama.cpp users a dedicated provider that doesn't go through the generic custom flow. Underlying bug in the custom-endpoint code is unchanged.
• Follow-up to feat(tools/web): add searxng, brave-free, ddgs as fallback backends #19796 (closed) — the search-backend portion shipped via #21337 (merged, authorship preserved on 04193cf71); SearXNG shipped via cd2cbc73b / 5c906d702. The launcher was the one piece deferred there, and it's resubmitted here per teknium's invitation. No content collision with tools/web_providers/ — that package is web-search only; this PR adds a model-provider plugin under plugins/model-providers/.
• Supersedes feat(tools): local_web_tools — free-tier web search/extract + llama-server launcher #19607 (closed). That PR's local_web_tools.py is obsolete (replaced by tools/web_providers/ post-merge); only the launcher needed to land, which this PR carries forward — stripped of model-family-specific detection so it's pure llama.cpp.

Not claiming Closes on any of these — feature requests partially delivered should stay open for maintainers to retitle/scope.

Type of Change

• ✨ New feature (non-breaking change that adds functionality)
• 📝 Documentation update
• ✅ Tests (adding test coverage)

Changes Made

Provider plugin

• plugins/model-providers/llama-cpp/ (new) — LlamaCppProfile with default base_url http://127.0.0.1:8088/v1, aliases llamacpp / llama.cpp / llama_cpp / llama-server, and an offline-tolerant fetch_models override that returns None instead of raising when the local server is down.
• plugins/model-providers/custom/__init__.py — drops the llamacpp / llama.cpp / llama-cpp aliases from the generic custom profile so the dedicated provider's aliases win.

Auth + CLI alias resolution

• hermes_cli/auth.py — adds llama-cpp to PROVIDER_REGISTRY (modeled on the lmstudio entry: auth_type="api_key" with optional LLAMA_CPP_API_KEY + LLAMA_CPP_BASE_URL env vars). Removes the old hardcoded llama.cpp/llamacpp/llama-cpp → custom alias mappings so the plugin's auto-extend wins.
• hermes_cli/models.py — adds the same alias mappings to _PROVIDER_ALIASES so --provider llama.cpp resolves correctly through the CLI parser path.

Dashboard / picker probe

• hermes_cli/model_switch.py — adds a probe-and-surface block in list_authenticated_providers, mirroring the existing lmstudio pattern. Three surfacing modes:

  1. Live probe — ${LLAMA_CPP_BASE_URL}/models with a 300 ms cold-discovery timeout. If llama-server responds, the row appears with the loaded model. This is the "magical" path.
  2. Hint mode — LLAMA_CPP_API_KEY/LLAMA_CPP_BASE_URL set, or current provider matches one of the aliases → 1.5 s timeout.
  3. Sticky current — when llama-cpp is the user's selected provider but the server is offline, the row still appears (uses current_model) so users don't lose access after a server restart.

  When no env vars, no current selection, and no live server, the row is not injected — keeps the picker tidy for non-llama.cpp users.

Launcher (resubmit per @teknium1's invitation)

• scripts/start-llama-server.sh (new) — turnkey llama-server launcher whose default port (8088) lines up with the plugin's default base_url. Prints an alignment hint when PORT/HOST diverge from the default. Generic — no model-family-specific detection.

Tests + docs

• tests/providers/test_llama_cpp_profile.py (new) — 12 tests covering plugin registration, alias resolution end-to-end through hermes_cli.auth.resolve_provider, CANONICAL_PROVIDERS auto-injection, PROVIDER_REGISTRY entry shape, picker surfacing in three modes (current+offline, no-clutter, alias resolution), and the offline-graceful fetch_models override.
• tests/providers/test_plugin_discovery.py — bumped expected profile count 33 → 34.
• website/docs/guides/local-llamacpp-setup.md (new) — user-facing setup guide modeled on the existing local-ollama-setup.md.
• website/docs/reference/environment-variables.md — documents LLAMA_CPP_API_KEY / LLAMA_CPP_BASE_URL and adds llama-cpp to the HERMES_INFERENCE_PROVIDER accepted-values list.

How to Test

1. Targeted suites (no venv setup needed for these):

   pytest tests/providers/ -o "addopts=-m 'not integration'" -q
   # → 90 passed
   pytest tests/providers/test_llama_cpp_profile.py -v
   # → 12 passed
   pytest tests/hermes_cli/test_model_switch_custom_providers.py \
          tests/hermes_cli/test_user_providers_model_switch.py \
          tests/hermes_cli/test_custom_provider_model_switch.py \
          tests/hermes_cli/test_api_key_providers.py \
          tests/hermes_cli/test_auth_provider_gate.py \
          -o "addopts=-m 'not integration'" -q
   # → 221 passed

2. End-to-end UX (requires a GGUF model and llama-server on PATH):

   chmod +x scripts/start-llama-server.sh
   ./scripts/start-llama-server.sh ~/models/your-model.gguf

   Then in another shell:

   hermes model       # llama.cpp row should appear with the loaded model
   hermes chat --provider llama-cpp

3. Picker UI without running server:

   • Open the dashboard with no LLAMA_CPP_* env vars and no llama-cpp selection in config.yaml → llama.cpp should NOT appear (no clutter).
   • Set model.provider: llama-cpp in config.yaml → row appears with current_model even when the server is offline.

4. Alias resolution:

   hermes chat --provider llamacpp -q "hi"
   hermes chat --provider llama.cpp -q "hi"
   hermes chat --provider llama-server -q "hi"

   All should resolve to the same llama-cpp provider.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (feat(agent):)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to first-class llama.cpp support + the explicitly-invited launcher resubmit
• I've run targeted test suites (tests/providers/ + relevant tests/hermes_cli/) — 316 tests across the affected modules pass. The full suite via scripts/run_tests.sh requires a uv venv I haven't set up locally; happy to defer to CI as the source of truth.
• I've added tests for my changes (12 new tests in tests/providers/test_llama_cpp_profile.py)
• I've tested on macOS 26.4 (arm64). The launcher uses nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 8 so it works on Linux + macOS + WSL2; not exercised on native Windows.

Documentation & Housekeeping

• Updated website/docs/guides/local-llamacpp-setup.md (new) and website/docs/reference/environment-variables.md
• N/A — no cli-config.yaml.example keys added; new env vars are optional and documented in the env-vars reference
• N/A — no architectural / workflow changes
• Considered cross-platform: launcher is shell-only (Linux + macOS + WSL2). Native Windows users can run python -m llama_cpp.server and set LLAMA_CPP_BASE_URL to point Hermes at it; the provider plugin itself is pure Python and platform-neutral.
• N/A — no tool schema changes

Why "first-class" instead of just --provider custom

The custom flow requires users to set OPENAI_BASE_URL, doesn't show in the model picker, and provides no dashboard auto-detection. For a backend as common as llama.cpp, the registration boilerplate is a small price for: discoverability in hermes model / hermes setup, dashboard surfacing, alias resolution, and (most importantly) zero-config UX once the launcher is running.

[teknium1]

Thanks for the cohesive provider, launcher, documentation, and test work. The generic custom-provider path and the launcher-focused prior discussion are useful context, but this change cannot land in this repository as an in-tree provider integration.

• This automated hermes-sweeper review found that the PR adds plugins/model-providers/llama-cpp/__init__.py, registering a new third-party llama.cpp provider (f82089fe8745, plugins/model-providers/llama-cpp/__init__.py:48).
• Maintainer policy requires third-party product integrations, including niche in-tree LLM-provider plugins, to ship as standalone plugins installed under ~/.hermes/plugins/ or via a pip entry point (AGENTS.md:797-813). This is a coupling and maintenance policy, not a judgment on the implementation quality.
• Current main already supports llama.cpp through the generic custom provider (plugins/model-providers/custom/__init__.py:3-5,75-92) and documents that configuration path (website/docs/integrations/providers.md:776-803).

Please publish the provider as a standalone plugin repository using the existing model-provider discovery surface; it can then be shared through the Nous Research Discord #plugins-skills-and-skins channel.

---

Closed as not-planned per standing maintainer policy (in-tree-provider-integration). This is a design-direction decision, not a code-quality judgment — see the Contribution Rubric in AGENTS.md for what the project is looking for. If you believe this policy was misapplied to your change, comment here and a maintainer will take a look.