From e10dbb372527dc687caafdec049e670e0c4c8e7a Mon Sep 17 00:00:00 2001 From: KrazyKraut Date: Fri, 8 May 2026 11:30:37 +0000 Subject: [PATCH 1/2] feat(provider): add vMLX local inference backend for Apple Silicon --- plugins/model-providers/vmlx/CHANGELOG.md | 36 ++++++ plugins/model-providers/vmlx/README.md | 142 ++++++++++++++++++++++ plugins/model-providers/vmlx/__init__.py | 55 +++++++++ plugins/model-providers/vmlx/plugin.yaml | 5 + tests/test_vmlx_provider.py | 91 ++++++++++++++ 5 files changed, 329 insertions(+) create mode 100644 plugins/model-providers/vmlx/CHANGELOG.md create mode 100644 plugins/model-providers/vmlx/README.md create mode 100644 plugins/model-providers/vmlx/__init__.py create mode 100644 plugins/model-providers/vmlx/plugin.yaml create mode 100644 tests/test_vmlx_provider.py diff --git a/plugins/model-providers/vmlx/CHANGELOG.md b/plugins/model-providers/vmlx/CHANGELOG.md new file mode 100644 index 0000000000000..55c2642509106 --- /dev/null +++ b/plugins/model-providers/vmlx/CHANGELOG.md @@ -0,0 +1,36 @@ +# Changelog + +All notable changes to the `vmlx` provider plugin are documented here. +Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); +versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.1.0] — Initial release + +### Added +- Two `ProviderProfile` registrations under `plugins/model-providers/vmlx/`: + - `vmlx` — primary, `http://localhost:8000/v1`, aliases + `mlx`, `mlx-server`, `apple-mlx`, `vmlx-primary`. + - `vmlx-janitor` — auxiliary, `http://localhost:8001/v1`, aliases + `vmlx-aux`, `mlx-janitor`. +- Both profiles use `env_vars=()` and `fallback_models=()` — no API key, + no cloud fallback (airgap by construction). +- Apple Silicon platform gate via `ImportError` on non-Darwin so the plugin + is invisible to Linux/Windows contributors. +- README with hardware sizing, dual-port serving instructions, Hermes config, + and 5-row troubleshooting table. +- pytest suite covering profile registration, default base URLs, and the + non-Darwin import guard. + +### Compatibility +- Built against the v0.13.0 plugin contract introduced in + [#20324](https://github.com/NousResearch/hermes-agent/pull/20324) + (ProviderProfile ABC + `plugins/model-providers/`). +- Follows the canonical pattern documented in + [#20749](https://github.com/NousResearch/hermes-agent/pull/20749) + (`website/docs/developer-guide/model-provider-plugin.md`). +- Apple Silicon (M1 / M2 / M3 / M4 / M5 family) only. + +### Stability +- Pre-1.0: provider names, aliases, and default base URLs may shift in + response to PR-review feedback. Once the plugin ships in an upstream + release, semver applies and graduation to 1.0 freezes the surface. diff --git a/plugins/model-providers/vmlx/README.md b/plugins/model-providers/vmlx/README.md new file mode 100644 index 0000000000000..e80793a0ba312 --- /dev/null +++ b/plugins/model-providers/vmlx/README.md @@ -0,0 +1,142 @@ +# vMLX provider — local Apple Silicon inference + +`vmlx` is a Hermes model-provider plugin that runs MLX-format LLMs on Apple +Silicon and serves them through an OpenAI-compatible chat-completions endpoint. +It is the recommended backend for fully airgapped Hermes operation on macOS. + +> **Status:** v0.1.0 — initial release. The public surface (provider names, +> base URLs, aliases) may evolve in response to PR-review feedback before +> the plugin graduates to 1.0. See [`CHANGELOG.md`](./CHANGELOG.md). + +## Why this exists + +The bundled [`custom`](../custom/) profile already covers generic local +OpenAI-compatible servers (Ollama, vLLM, llama.cpp). `vmlx` is a sibling +that adds: + +- **Sane localhost defaults** — `base_url` is pre-filled, no config needed. +- **Apple Silicon platform gate** — `ImportError` on non-Darwin so the + plugin is invisible to Linux/Windows contributors. +- **Primary/janitor split out of the box** — two profiles registered side + by side so auxiliary work (compression, memory writes, summarization, + skill curation) can run on a separate `vmlx serve` instance and stop + consuming the primary model's context window. + +## What gets registered + +The plugin's `__init__.py` calls `register_provider()` twice: + +| Profile name | Default base URL | Suggested role | +|----------------|-------------------------------|-------------------------| +| `vmlx` | `http://localhost:8000/v1` | Primary agent loop | +| `vmlx-janitor` | `http://localhost:8001/v1` | Auxiliary / aux model | + +Aliases: `mlx`, `mlx-server`, `apple-mlx`, `vmlx-primary` resolve to `vmlx`; +`vmlx-aux`, `mlx-janitor` resolve to `vmlx-janitor`. + +## Hardware requirements + +| Model class | Quantization | Recommended unified memory | +|-------------|--------------|----------------------------| +| 3–8 B | 4-bit | 16 GB | +| 13–14 B | 4-bit | 24 GB | +| 30–34 B | 4-bit | 36 GB | +| 70 B | 4-bit | 64 GB | + +Apple Silicon (M1 / M2 / M3 / M4 / M5 family) only. Intel Macs are not +supported by MLX itself. + +## Installation + +```bash +pip install vmlx +``` + +Discovery is automatic — `providers/__init__.py._discover_providers()` picks +up the directory the next time `get_provider_profile()` is called. + +## Model acquisition + +Suggestions; any MLX-quantized chat model in the same size class works. + +```bash +mkdir -p ~/models +huggingface-cli download mlx-community/Qwen2.5-32B-Instruct-4bit \ + --local-dir ~/models/primary +huggingface-cli download mlx-community/gemma-3-4b-it-4bit \ + --local-dir ~/models/janitor +``` + +## Serving both models + +```bash +vmlx serve --model ~/models/primary --port 8000 --ctx-size 65536 +vmlx serve --model ~/models/janitor --port 8001 --ctx-size 16384 +``` + +For permanent setup with `launchd`, see the [macOS airgap guide](../../../website/docs/guides/macos-airgap.md). + +## Hermes config + +Minimal — picks up the primary profile's defaults: + +```yaml +model: + provider: vmlx + name: primary + context_length: 65536 + temperature: 0.2 + +fallback_providers: [] +``` + +`fallback_providers: []` is the explicit airgap declaration: Hermes will +never reach for a cloud provider on inference failure. + +For the primary/janitor split, point auxiliary tasks at the `vmlx-janitor` +profile via your Hermes version's auxiliary-routing config (see the +provider-runtime developer guide for resolution precedence). A common +pattern: + +```yaml +auxiliary_routes: + compression: { provider: vmlx-janitor, name: janitor } + memory_write: { provider: vmlx-janitor, name: janitor } + skill_curation: { provider: vmlx-janitor, name: janitor } + context_summary:{ provider: vmlx-janitor, name: janitor } +``` + +If your Hermes release does not support per-task `provider` overrides yet, +serve both models on `:8000` and use `default_aux_model` instead. + +## Verification + +```bash +hermes doctor +``` + +Expected (relevant lines — the `/models` probe is automatic for any +`api_key`/empty-`env_vars` profile): + +``` +[ok] provider 'vmlx' registered +[ok] provider 'vmlx-janitor' registered +[ok] http://localhost:8000/v1/models reachable +[ok] http://localhost:8001/v1/models reachable +``` + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---------|--------------|-----| +| `vmlx` not in `hermes doctor` output | Plugin import failed (likely non-Darwin) | This plugin is Apple Silicon only; check `python -c "import platform; print(platform.system())"` | +| `connection refused` on /models probe | `vmlx serve` not listening on the expected port | `lsof -nP -iTCP:8000 -sTCP:LISTEN`; restart the server | +| `context length exceeded` mid-loop | `--ctx-size` lower than `model.context_length` | Restart server with matching `--ctx-size` | +| Janitor crashes on first request | OOM (model + primary both loaded) | Smaller janitor model or 4-bit quantization | +| Override the bundled defaults | Plugin lives at `/plugins/model-providers/vmlx/` (bundled) | Drop a same-named directory under `$HERMES_HOME/plugins/model-providers/vmlx/` — user plugins win because `register_provider()` is last-writer-wins | + +## Contributing + +Issues and PRs welcome — see [CONTRIBUTING.md](../../../CONTRIBUTING.md). The +plugin's macOS gate lives in `__init__.py` (`ImportError` on non-Darwin), so +non-Mac contributors can hack on Hermes without it loading. diff --git a/plugins/model-providers/vmlx/__init__.py b/plugins/model-providers/vmlx/__init__.py new file mode 100644 index 0000000000000..21cc8dcbd5d67 --- /dev/null +++ b/plugins/model-providers/vmlx/__init__.py @@ -0,0 +1,55 @@ +"""vMLX provider profiles — Apple Silicon local inference. + +Registers two profiles against the v0.13.0 ``providers/`` plugin contract: + + * ``vmlx`` — primary, http://localhost:8000/v1, large context. + * ``vmlx-janitor`` — auxiliary (compression / summarization / memory writes / + skill curation), http://localhost:8001/v1, short context. + +Both speak the OpenAI chat-completions wire protocol against a local +``vmlx serve`` instance. Apple Silicon only; the ``platform.system()`` guard +below makes the import fail cleanly on Linux/Windows so the discovery loop +in ``providers/__init__.py`` skips the directory. + +Sibling of the bundled ``custom``/Ollama profile, but with sane localhost +defaults baked in for vMLX so airgap setups need zero configuration. +""" +from __future__ import annotations + +import platform + +from providers import register_provider +from providers.base import ProviderProfile + +if platform.system() != "Darwin": + raise ImportError("vmlx provider requires macOS (Apple Silicon)") + +vmlx_primary = ProviderProfile( + name="vmlx", + aliases=("mlx", "mlx-server", "apple-mlx", "vmlx-primary"), + display_name="vMLX (Apple Silicon local inference)", + description=( + "MLX-format LLMs served locally via `vmlx serve` on Apple Silicon. " + "Airgap-friendly: no API key, no cloud fallback." + ), + env_vars=(), + base_url="http://localhost:8000/v1", + fallback_models=(), +) + +vmlx_janitor = ProviderProfile( + name="vmlx-janitor", + aliases=("vmlx-aux", "mlx-janitor"), + display_name="vMLX janitor (Apple Silicon, auxiliary tasks)", + description=( + "Smaller MLX model on a separate `vmlx serve` instance — drives " + "compression, summarization, memory writes, and skill curation so the " + "primary model's context window stays free for the agent loop." + ), + env_vars=(), + base_url="http://localhost:8001/v1", + fallback_models=(), +) + +register_provider(vmlx_primary) +register_provider(vmlx_janitor) diff --git a/plugins/model-providers/vmlx/plugin.yaml b/plugins/model-providers/vmlx/plugin.yaml new file mode 100644 index 0000000000000..10bd08a71c5c2 --- /dev/null +++ b/plugins/model-providers/vmlx/plugin.yaml @@ -0,0 +1,5 @@ +name: vmlx +kind: model-provider +version: 0.1.0 +description: vMLX (Apple Silicon local inference) — OpenAI-compatible MLX server with primary/janitor split +author: "maurice-jobst" diff --git a/tests/test_vmlx_provider.py b/tests/test_vmlx_provider.py new file mode 100644 index 0000000000000..e7c00e616c644 --- /dev/null +++ b/tests/test_vmlx_provider.py @@ -0,0 +1,91 @@ +"""Tests for the vMLX model-provider plugin. + +Validates the v0.13.0 ``providers/`` contract: two ``ProviderProfile`` +instances are registered at module import (vmlx + vmlx-janitor), and the +plugin's ``platform.system()`` guard fails the import cleanly on non-Darwin. + +The plugin lives at ``plugins/model-providers/vmlx/`` (hyphen — not a valid +Python identifier) so we load it via ``importlib.util.spec_from_file_location`` +the same way ``providers/__init__.py._discover_providers()`` does. +""" +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path +from typing import Any +from unittest import mock + +import pytest + +PLUGIN_INIT = ( + Path(__file__).resolve().parents[1] + / "plugins" + / "model-providers" + / "vmlx" + / "__init__.py" +) + + +def _load_plugin(register_provider_mock: Any) -> Any: + sys.modules.pop("vmlx_plugin_under_test", None) + spec = importlib.util.spec_from_file_location( + "vmlx_plugin_under_test", PLUGIN_INIT + ) + assert spec is not None and spec.loader is not None, ( + f"plugin not found at {PLUGIN_INIT}" + ) + module = importlib.util.module_from_spec(spec) + with mock.patch("providers.register_provider", register_provider_mock): + spec.loader.exec_module(module) + return module + + +@pytest.mark.skipif(sys.platform != "darwin", reason="vMLX is macOS-only") +def test_plugin_registers_two_profiles() -> None: + register_provider = mock.MagicMock() + _load_plugin(register_provider) + + assert register_provider.call_count == 2 + profiles = [c.args[0] for c in register_provider.call_args_list] + names = {p.name for p in profiles} + assert names == {"vmlx", "vmlx-janitor"} + + +@pytest.mark.skipif(sys.platform != "darwin", reason="vMLX is macOS-only") +def test_primary_profile_has_correct_defaults() -> None: + register_provider = mock.MagicMock() + _load_plugin(register_provider) + + primary = next( + c.args[0] + for c in register_provider.call_args_list + if c.args[0].name == "vmlx" + ) + assert primary.base_url == "http://localhost:8000/v1" + assert primary.env_vars == () + assert primary.fallback_models == () + assert "mlx" in primary.aliases + + +@pytest.mark.skipif(sys.platform != "darwin", reason="vMLX is macOS-only") +def test_janitor_profile_on_separate_port() -> None: + register_provider = mock.MagicMock() + _load_plugin(register_provider) + + janitor = next( + c.args[0] + for c in register_provider.call_args_list + if c.args[0].name == "vmlx-janitor" + ) + assert janitor.base_url == "http://localhost:8001/v1" + assert janitor.env_vars == () + assert janitor.fallback_models == () + + +def test_plugin_raises_importerror_on_non_darwin() -> None: + register_provider = mock.MagicMock() + with mock.patch("platform.system", return_value="Linux"): + with pytest.raises(ImportError, match="macOS"): + _load_plugin(register_provider) + register_provider.assert_not_called() From 7394c2aadea6ca87db13b3ad947fd5086acb816b Mon Sep 17 00:00:00 2001 From: KrazyKraut Date: Fri, 8 May 2026 11:30:39 +0000 Subject: [PATCH 2/2] feat(docs): add macOS airgap setup guide --- website/docs/guides/macos-airgap.md | 252 ++++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 website/docs/guides/macos-airgap.md diff --git a/website/docs/guides/macos-airgap.md b/website/docs/guides/macos-airgap.md new file mode 100644 index 0000000000000..6a9ede00efe11 --- /dev/null +++ b/website/docs/guides/macos-airgap.md @@ -0,0 +1,252 @@ +--- +sidebar_label: macOS airgap mode +sidebar_position: 7 +description: >- + Run Hermes with zero cloud calls on Apple Silicon using the vMLX provider + plugin. Suitable for privacy-sensitive, regulated, and offline workloads. +--- + +# macOS airgap mode + +This guide walks you from a clean macOS install to a fully airgapped Hermes — +no cloud API calls, no telemetry, all inference on-device — using the +**vMLX** model-provider plugin (Hermes v0.13.0+). + +For provider-only reference docs, see +[`plugins/model-providers/vmlx/README.md`](https://github.com/NousResearch/hermes-agent/tree/main/plugins/model-providers/vmlx). +For the underlying plugin contract, see +[Model Provider Plugins](/docs/developer-guide/model-provider-plugin). + +:::info Who is this for? +- You handle data that cannot leave your machine (legal, medical, KRITIS, + regulated, classified-equivalent workloads). +- You want predictable cost: $0 per token, forever. +- You want Hermes to keep working on a plane, in a SCIF, or anywhere the WAN + is down. +::: + +## Prerequisites + +- Apple Silicon Mac (M1 / M2 / M3 / M4 / M5 family). Intel is not supported. +- macOS 14 or newer. +- 24 GB unified memory minimum; 36 GB+ for a primary model in the 30 B class. +- Python 3.11 (managed via `uv` recommended). +- Hermes Agent **v0.13.0 or later** — earlier versions don't have the + `ProviderProfile` plugin contract that `vmlx` registers against. +- `hermes doctor` passing on a non-airgap config. + +## 1. Install vMLX + +```bash +pip install vmlx +``` + +The plugin loads automatically — discovery happens at first +`get_provider_profile()` call. + +## 2. Download MLX models + +You need **two** models: a *primary* for the main agent loop and a smaller +*janitor* for auxiliary tasks (compression, memory writes, summarization, +skill curation). + +```bash +mkdir -p ~/models + +huggingface-cli download mlx-community/Qwen2.5-32B-Instruct-4bit \ + --local-dir ~/models/primary + +huggingface-cli download mlx-community/gemma-3-4b-it-4bit \ + --local-dir ~/models/janitor +``` + +The names above are suggestions — any MLX-quantized chat model in those +size classes will work. + +## 3. Start both vMLX servers + +For a manual smoke test: + +```bash +vmlx serve --model ~/models/primary --port 8000 --ctx-size 65536 & +vmlx serve --model ~/models/janitor --port 8001 --ctx-size 16384 & +``` + +For permanent setup, see the [launchd section](#launchd-auto-start) below. + +## 4. Configure Hermes + +The `vmlx` plugin registers two profiles with sane defaults — primary on +`:8000`, janitor on `:8001`. You only need to wire them into your +`config.yaml`: + +```yaml +model: + provider: vmlx + name: primary + context_length: 65536 + temperature: 0.2 + +auxiliary_routes: + compression: { provider: vmlx-janitor, name: janitor } + memory_write: { provider: vmlx-janitor, name: janitor } + skill_curation: { provider: vmlx-janitor, name: janitor } + context_summary: { provider: vmlx-janitor, name: janitor } + +fallback_providers: [] +``` + +`fallback_providers: []` is the explicit airgap declaration — Hermes will +never reach for a cloud provider on inference failure. + +If your Hermes release does not yet support per-task `provider` overrides +in `auxiliary_routes`, drop the block and use `default_aux_model` to point +at a smaller model on the primary endpoint instead. + +And the matching `.env` — local-only variables, no cloud keys: + +```bash +# Local endpoints only. Do NOT add OPENAI_API_KEY, ANTHROPIC_API_KEY, or any +# other cloud credential here — adding one defeats the airgap guarantee +# because plugins may opportunistically use cloud creds as fallbacks. +HERMES_LOG_LEVEL=info +HERMES_TELEMETRY=off +``` + +## launchd auto-start + +`launchd` runs with an empty `PATH`, so the plists need an **absolute** path +to the `vmlx` binary. The location depends on how you installed it: + +```bash +# Find the absolute path to substitute below. +which vmlx +# Common locations on Apple Silicon: +# /opt/homebrew/bin/vmlx (Homebrew on Apple Silicon) +# ~/Library/Python/3.11/bin/vmlx (pip --user) +# /bin/vmlx (virtualenv) +``` + +Capture both substitutions before writing the plists: + +```bash +export VMLX_BIN="$(which vmlx)" +export USERNAME="$(whoami)" +``` + +`~/Library/LaunchAgents/dev.hermes.vmlx-primary.plist` — `VMLX_BIN` and +`USERNAME` are placeholders to substitute: + +```xml + + + + + Labeldev.hermes.vmlx-primary + ProgramArguments + + VMLX_BIN + serve + --model/Users/USERNAME/models/primary + --port8000 + --ctx-size65536 + + RunAtLoad + KeepAlive + StandardOutPath + /Users/USERNAME/Library/Logs/vmlx-primary.log + StandardErrorPath + /Users/USERNAME/Library/Logs/vmlx-primary.log + + +``` + +`~/Library/LaunchAgents/dev.hermes.vmlx-janitor.plist`: + +```xml + + + + + Labeldev.hermes.vmlx-janitor + ProgramArguments + + VMLX_BIN + serve + --model/Users/USERNAME/models/janitor + --port8001 + --ctx-size16384 + + RunAtLoad + KeepAlive + StandardOutPath + /Users/USERNAME/Library/Logs/vmlx-janitor.log + StandardErrorPath + /Users/USERNAME/Library/Logs/vmlx-janitor.log + + +``` + +Substitute the placeholders in-place and load both: + +```bash +sed -i '' "s|VMLX_BIN|$VMLX_BIN|g; s|USERNAME|$USERNAME|g" \ + ~/Library/LaunchAgents/dev.hermes.vmlx-primary.plist \ + ~/Library/LaunchAgents/dev.hermes.vmlx-janitor.plist + +launchctl load ~/Library/LaunchAgents/dev.hermes.vmlx-primary.plist +launchctl load ~/Library/LaunchAgents/dev.hermes.vmlx-janitor.plist +launchctl list | grep vmlx +``` + +## 5. Verify + +```bash +hermes doctor +``` + +Expected (relevant lines — the `/models` probe is automatic per the +v0.13.0 plugin contract): + +``` +[ok] provider 'vmlx' registered +[ok] provider 'vmlx-janitor' registered +[ok] http://localhost:8000/v1/models reachable +[ok] http://localhost:8001/v1/models reachable +[ok] fallback_providers is empty (airgap mode) +``` + +A quick end-to-end loop: + +```bash +hermes run "summarize the contents of ~/Documents/notes.md in three bullets" +``` + +If the primary model produces a response and `tcpdump` shows no off-host +traffic from the Hermes process, the airgap is working: + +```bash +sudo tcpdump -i any -nn 'host not 127.0.0.1 and host not ::1' & +hermes run "..." +``` + +## Troubleshooting + +| Symptom | Cause | Fix | +|---------|-------|-----| +| `vmlx` not registered in `hermes doctor` | Plugin import failed (often non-Darwin) | Apple Silicon only; check `python -c "import platform; print(platform.system())"` | +| `/models` probe fails on :8000 or :8001 | `vmlx serve` not listening | `launchctl list \| grep vmlx`; tail `~/Library/Logs/vmlx-{primary,janitor}.log` | +| `context length exceeded` mid-loop | `--ctx-size` lower than `config.yaml` `context_length` | Restart the affected server with matching `--ctx-size` | +| `connection refused` on :8001 only | Janitor crashed (often OOM with both models loaded) | Smaller janitor model or raise quantization to 4-bit | +| All requests slow on first call only | MLX warming the model into unified memory | Expected; subsequent calls hit the warm model | +| `address already in use` on launch | Another process owns 8000/8001 | `lsof -nP -iTCP:8000 -sTCP:LISTEN` then kill or change port | +| `auxiliary_routes` config rejected | Hermes release pre-dates per-task provider override | Drop the block; use `default_aux_model` on the primary profile instead | + +:::tip Contributing +Found a gap or a bug in this guide? See +[CONTRIBUTING.md](https://github.com/NousResearch/hermes-agent/blob/main/CONTRIBUTING.md). +The plugin's macOS gate lives in `__init__.py` (`ImportError` on non-Darwin), +so non-Mac contributors can hack on Hermes without it loading. +:::