From 8cf368cfe293bcbf6be70bbcf8cd8374fba5f905 Mon Sep 17 00:00:00 2001 From: Michael Neale Date: Fri, 12 Jun 2026 16:35:25 +1000 Subject: [PATCH 1/3] updating to reflect reality --- .skills/README.md | 26 +++++ .skills/connect-agents/SKILL.md | 97 ++++++++++++++++++ .skills/deploy-linux-gpu/SKILL.md | 20 +++- .skills/deploy-macos/SKILL.md | 163 ++++++++++++++++++++++++++++++ .skills/deploy/SKILL.md | 122 ---------------------- .skills/mesh-join/SKILL.md | 138 +++++++++++++++++++++++++ AGENTS.md | 7 +- 7 files changed, 444 insertions(+), 129 deletions(-) create mode 100644 .skills/README.md create mode 100644 .skills/connect-agents/SKILL.md create mode 100644 .skills/deploy-macos/SKILL.md delete mode 100644 .skills/deploy/SKILL.md create mode 100644 .skills/mesh-join/SKILL.md diff --git a/.skills/README.md b/.skills/README.md new file mode 100644 index 0000000000..98bca108c5 --- /dev/null +++ b/.skills/README.md @@ -0,0 +1,26 @@ +# Repo Agent Skills + +Skills under `.skills/` are auto-picked-up by agents working in this repo. +Each is a focused, current how-to; deeper reference lives in `docs/`. + +| Skill | Use when | +|---|---| +| [deploy-macos](deploy-macos/SKILL.md) | Install/launch mesh-llm on a macOS node (release install or dev-build bundle, codesign/quarantine, verify serving) | +| [deploy-linux-gpu](deploy-linux-gpu/SKILL.md) | Install/launch mesh-llm on a remote Linux GPU node (Vast.ai/RunPod/self-managed CUDA, supervisor/systemd, verify serving) | +| [mesh-join](mesh-join/SKILL.md) | Create/join/publish meshes: invite tokens, `--auto`, named meshes, client-only nodes, NAT/bind issues, multi-node verification | +| [connect-agents](connect-agents/SKILL.md) | Point Goose/Claude Code/OpenCode/Pi or any OpenAI client at a running mesh; tool-call validation; blackboard | + +Ground rules baked into all of these: + +- The bundle/release is a **single `mesh-llm` binary** with the embedded staged + runtime. No `rpc-server`, no `llama-server`, no `.dylib` set. +- mesh-llm downloads models itself — pass `--model `, never pre-download. +- `--headless` only hides the web UI; it is not a backgrounding mechanism. +- Prefer `mesh-llm stop` over `pkill`. + +Related docs: `docs/USAGE.md` (install/service/storage), `docs/CLI.md` +(commands and model refs), `docs/MESHES.md` (mesh workflows), +`docs/AGENTS.md` (agent clients), `docs/SKIPPY_SPLITS.md` (big-model splits). + +Maintainer-facing skills (skippy internals, patch queues, benchmarks, lab) live +in `.agents/skills/`; plugin-shipped skills install via `mesh-llm skills install`. diff --git a/.skills/connect-agents/SKILL.md b/.skills/connect-agents/SKILL.md new file mode 100644 index 0000000000..c1bb87ba37 --- /dev/null +++ b/.skills/connect-agents/SKILL.md @@ -0,0 +1,97 @@ +--- +name: connect-agents +description: Use this skill when connecting agent tools or OpenAI clients to mesh-llm — launching or configuring Goose, Claude Code, OpenCode, Pi, curl, or any OpenAI-compatible client against a local or remote mesh, picking a model, or validating tool-call reliability. +metadata: + short-description: Connect agents and OpenAI clients to mesh-llm +--- + +# connect-agents + +Use this when pointing an agent harness or any OpenAI client at a running +mesh-llm node. Full reference: `docs/AGENTS.md`. + +## Mental model + +- Every node serves an OpenAI-compatible API at `http://:9337/v1`. +- `GET /v1/models` lists everything reachable (local + mesh peers); requests + route by the `model` field. +- Special model ids: `auto` lets the mesh pick; `mesh` engages the + mixture-of-agents path. Otherwise use an exact id from `/v1/models`. +- For coding agents, pick a tool-capable model. If `--model` is omitted, the + built-in launchers pick the strongest tool-capable model available. + +## Built-in launchers (preferred) + +mesh-llm launches the major agent CLIs with config injected for you: + +```bash +mesh-llm goose [--model ] # writes ~/.config/goose/custom_providers/mesh.json +mesh-llm claude [--model ] +mesh-llm opencode [--model ] [--host ] # injects OPENCODE_CONFIG_CONTENT (no file edits) +mesh-llm pi [--model ] [--host ] # writes ~/.pi/agent/models.json +``` + +- `goose`/`claude` reuse a local mesh on the chosen `--port`. +- `opencode`/`pi` target `--host` (default `127.0.0.1:9337`) and auto-start a + local client only for loopback targets; the auto-started node is cleaned up + when the harness exits. +- `mesh-llm pi --write` / `mesh-llm opencode --write` update config without + launching (use `--host` for remote meshes). +- Agent launch commands also install available plugin skills for that agent + (`mesh-llm skills install` does it standalone). + +## Manual config (any OpenAI client) + +Base URL `http://:9337/v1`, any non-empty API key: + +```bash +export GOOSE_PROVIDER=openai GOOSE_MODEL="" +export OPENAI_HOST="http://127.0.0.1:9337" OPENAI_API_KEY="mesh" +``` + +```bash +curl -s http://localhost:9337/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}' +``` + +Exact manual provider JSON for OpenCode and Pi is in `docs/AGENTS.md`. + +## Validating agent behavior + +Direct API contract probe (tool-call forcing, streaming reconstruction): + +```bash +scripts/qa-agent-tool-call-reliability.py \ + --base-url http://127.0.0.1:9337/v1 --models auto,mesh --attempts 3 \ + --output target/agent-tool-call-reliability/results.jsonl +``` + +Broader harness (models, chat, streaming, plus optional Goose/OpenCode/Pi +smokes): `scripts/qa-nightly-stability.py` — see `docs/AGENTS.md`. Use +`--print-plan` on either script for a side-effect-free preview. + +## Blackboard (cross-mesh agent coordination) + +Agents can share status/questions across the mesh via the blackboard plugin — +even from a client-only node: + +```bash +mesh-llm plugins install blackboard +mesh-llm blackboard "STATUS: [org/repo branch:main] refactoring billing module" +mesh-llm blackboard --search "QUESTION" +``` + +MCP access: the management endpoint `http://127.0.0.1:3131/mcp` exposes +`blackboard_post`, `blackboard_search`, `blackboard_feed`. Posts are visible to +every peer — never post secrets, credentials, private paths, or customer data. + +## Gotchas + +- Use a base URL ending in `/v1`; prefer chat-completions over the Responses + API unless the client documents Responses support. +- Model ids must match `/v1/models` exactly (they can contain spaces — quote + them). +- An empty `/v1/models` usually means the model is still loading or no mesh was + joined yet — check `/api/status` on `:3131` (see `mesh-join`). +- The response `"model"` field tells you which node/model actually answered. diff --git a/.skills/deploy-linux-gpu/SKILL.md b/.skills/deploy-linux-gpu/SKILL.md index f859b5cb7e..b311372307 100644 --- a/.skills/deploy-linux-gpu/SKILL.md +++ b/.skills/deploy-linux-gpu/SKILL.md @@ -11,12 +11,18 @@ Use this when standing up mesh-llm on a remote Linux GPU box (rented GPU like Vast.ai / RunPod, or your own server) to serve a specific model and join the mesh. -This is the Linux/CUDA counterpart to the macOS `deploy` skill. It does NOT use +This is the Linux/CUDA counterpart to the `deploy-macos` skill. It does NOT use the old `llama-server`/`rpc-server` lane — the current binary embeds the staged runtime. There are no `.dylib`/`codesign`/quarantine steps on Linux. note --auto flag tells it to join the public mesh. serve command with --model tells it to run a specific model. -Example here are for solo serving, don't read this in isolation without other docs and skills. +Examples here are for solo serving — don't read this in isolation: + +- `mesh-join` — creating/joining private and public meshes (tokens, NAT, multi-node) +- `connect-agents` — pointing Goose/Claude Code/OpenCode/Pi at a running mesh +- `docs/USAGE.md` — install details, service mode, model storage +- `docs/CLI.md` — full command and model-ref reference +- `docs/SKIPPY_SPLITS.md` — splitting big models across nodes ## The one rule that matters most @@ -109,9 +115,13 @@ supervisorctl reread && supervisorctl update && supervisorctl start mesh-llm supervisorctl status mesh-llm ``` -If there is no supervisor, run it under `systemd --user`, `tmux new -d`, or as a -foreground process in a held SSH session for first-run debugging (allocate a TTY -with `ssh -tt host 'bash -lc "..."'`). +If there is no supervisor, the installer can set up a `systemd --user` service +for you (`curl -fsSL .../install.sh | sh -s -- --service` installs +`~/.config/systemd/user/mesh-llm.service`; startup models go in +`~/.mesh-llm/config.toml`, and `sudo loginctl enable-linger "$USER"` makes it +survive reboot before login). Otherwise use `tmux new -d`, or a foreground +process in a held SSH session for first-run debugging (allocate a TTY with +`ssh -tt host 'bash -lc "..."'`). ## Verify it's actually serving diff --git a/.skills/deploy-macos/SKILL.md b/.skills/deploy-macos/SKILL.md new file mode 100644 index 0000000000..758f0a7c22 --- /dev/null +++ b/.skills/deploy-macos/SKILL.md @@ -0,0 +1,163 @@ +--- +name: deploy-macos +description: Use this skill when deploying, installing, launching, or serving mesh-llm on a macOS machine (local or remote over SSH), including installing a release, shipping a dev build bundle, codesign/quarantine fixes, choosing a model, and verifying it serves. +metadata: + short-description: Deploy mesh-llm on a macOS node +--- + +# deploy-macos + +Use this when standing up mesh-llm on a macOS machine — either installing a +release or shipping a locally built dev binary to a remote Mac for testing. + +This is the macOS counterpart to `deploy-linux-gpu`. The current binary embeds +the staged llama.cpp runtime: the bundle is a **single `mesh-llm` binary**. +There is no `rpc-server`, no `llama-server`, and no `.dylib` set anymore — if +you see instructions mentioning those, they are outdated. + +Related skills/docs: + +- `deploy-linux-gpu` — remote Linux/CUDA nodes +- `mesh-join` — creating/joining private and public meshes (tokens, NAT, multi-node) +- `connect-agents` — pointing Goose/Claude Code/OpenCode/Pi at a running mesh +- `docs/USAGE.md` — install details, service mode, model storage +- `docs/CLI.md` — full command and model-ref reference + +## The one rule that matters most + +**mesh-llm resolves and downloads the model itself.** Pass `--model ` and +it fetches the GGUF into the standard Hugging Face cache on first use. Do NOT +pre-download with `hf`/`huggingface-cli`, do NOT scp GGUFs around. (Only +`--gguf` takes a local file path you manage yourself.) + +## Install path A: official release (most cases) + +```bash +curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash +``` + +The binary lands at `~/.local/bin/mesh-llm` (may not be on a non-interactive +SSH `PATH` — use the full path or `bash -lc`). Metal is the macOS backend; the +installer picks it automatically. + +To install as a per-user background service (launchd agent) in the same step: + +```bash +curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash -s -- --service +``` + +Service files: `~/Library/LaunchAgents/com.mesh-llm.mesh-llm.plist`, shared env +in `~/.config/mesh-llm/service.env`, startup models in `~/.mesh-llm/config.toml`. + +## Install path B: dev build to a remote Mac + +Build and bundle locally (from the repo): + +```bash +just release-build # serious testing must use the release binary +just bundle # /tmp/mesh-llm-bundle.tar.gz (single mesh-llm binary) +``` + +Ship and unpack: + +```bash +scp -P /tmp/mesh-llm-bundle.tar.gz user@host: +ssh -p user@host 'mkdir -p ~/bin && tar xzf mesh-llm-bundle.tar.gz -C ~/bin --strip-components=1' +``` + +### Fix macOS quarantine — ALWAYS after scp + +Files transferred via scp get provenance/quarantine xattrs that make macOS +SIGKILL the binary on launch (exit 137). After every scp: + +```bash +codesign -s - ~/bin/mesh-llm +xattr -cr ~/bin/ +``` + +Verify: `xattr ~/bin/mesh-llm` should print nothing. Note codesign changes the +file hash — don't compare local vs remote hashes after signing. + +Verify the version on the remote matches what you built: + +```bash +~/bin/mesh-llm --version +``` + +## Launch + +Serve a model and join the public mesh: + +```bash +mesh-llm serve --model unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL --auto +``` + +- `--auto` discovers and joins the community mesh; local serving and mesh + joining happen together. +- Without `--auto` (and without `--join`/`--discover`) you create a private + mesh and an invite token is emitted — see the `mesh-join` skill. +- `--model` accepts catalog names, `repo:QUANT`, `repo/file.gguf`, or a full HF + URL. `--gguf /path/file.gguf` serves a local file directly. +- API on `:9337`, management console on `:3131` (override with `--port` / + `--console`). + +Notes / gotchas: + +- **Do NOT use `--headless` to "go quiet"** — it only disables the embedded web + UI and does nothing for backgrounding. For machine-readable output use + `--log-format json`. +- **Model load takes time.** Poll `/v1/models` until your model appears before + concluding anything is broken. +- For background test runs from an agent: + `bash -c 'nohup mesh-llm serve --model --auto > /tmp/mesh.log 2>&1 & disown'`. + For persistence across reboots, prefer the `--service` install. + +## Verify it's actually serving + +```bash +# Ports bound +lsof -nP -iTCP:9337 -iTCP:3131 -sTCP:LISTEN + +# Models (union of local + mesh peers) +curl -s http://localhost:9337/v1/models | python3 -m json.tool + +# Status / peers +curl -s http://localhost:3131/api/status | python3 -m json.tool + +# Inference — the returned "model" field tells you which node/model answered +curl -s http://localhost:9337/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"auto","messages":[{"role":"user","content":"hi"}],"max_tokens":16}' +``` + +To force your local model specifically, pass its exact id from `/v1/models` +instead of `auto`. + +## Logs and state + +- `~/.mesh-llm/runtime//logs/skippy-native.log` — embedded llama.cpp/skippy + native logs. Check here first if a model fails to load. +- `~/.mesh-llm/key` — persistent node identity. +- `~/.mesh-llm/config.toml` — startup models and defaults for bare `mesh-llm serve`. +- HF cache (`~/.cache/huggingface/...`) — downloaded GGUFs; you generally never + need to touch this. + +## Stop / clean up + +```bash +mesh-llm stop # scoped stop of tracked instances (preferred) +# emergency only: +pkill -9 -f mesh-llm +``` + +A clean stop removes the instance runtime dir under `~/.mesh-llm/runtime/`. + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| Exit 137 immediately after scp | macOS quarantine/provenance xattr | `codesign -s - ; xattr -cr ` | +| `mesh-llm: command not found` over SSH | `~/.local/bin` not on non-interactive PATH | Full path or `bash -lc` | +| Empty `/v1/models` | Model still downloading/loading | Wait; watch skippy-native.log | +| "No inference server available" | Election in progress or load failed | Check stderr + skippy-native.log | +| Stale runtime dir after crash | Unclean exit | `rm -rf ~/.mesh-llm/runtime//` (auto-GC'd after 1h too) | diff --git a/.skills/deploy/SKILL.md b/.skills/deploy/SKILL.md deleted file mode 100644 index ae857bf262..0000000000 --- a/.skills/deploy/SKILL.md +++ /dev/null @@ -1,122 +0,0 @@ -# Deploy mesh-llm to a remote macOS node - -## Build the bundle locally - -```bash -cd /path/to/deez -just bundle # creates /tmp/mesh-bundle.tar.gz -``` - -## Copy to remote - -```bash -scp -P /tmp/mesh-bundle.tar.gz user@host: -``` - -## Install on remote - -```bash -ssh -p user@host -mkdir -p ~/bin && tar xzf mesh-bundle.tar.gz -C ~/bin --strip-components=1 -``` - -The bundle contains: `mesh-llm`, `rpc-server`, `llama-server`, `*.dylib`. - -## Fix macOS quarantine - -Files transferred via scp get `com.apple.provenance` xattr which causes macOS to SIGKILL (exit 137) on launch. **Always run after scp:** - -```bash -codesign -s - ~/bin/mesh-llm -codesign -s - ~/bin/rpc-server -codesign -s - ~/bin/llama-server -xattr -cr ~/bin/ -``` - -To verify: `xattr ~/bin/mesh-llm` should return nothing. If you see `com.apple.provenance` or `com.apple.quarantine`, the binary will be killed on launch. - -## Download a model - -```bash -~/bin/mesh-llm download 32b --draft # downloads to ~/.models/ -``` - -Or list all available models: -```bash -~/bin/mesh-llm download -``` - -Models go in `~/.models/` by convention. Both nodes need the same GGUF file for distributed inference. - -## Start the node - -### As first node (creates mesh) -```bash -nohup ~/bin/mesh-llm --model Qwen2.5-32B --bind-port 7842 > /tmp/mesh.log 2>&1 & -``` - -- `--bind-port` pins QUIC to a fixed UDP port for NAT port forwarding -- The invite token is printed to stderr (captured in the log) - -Get the token: -```bash -grep "Invite token:" /tmp/mesh.log | tail -1 | sed "s/Invite token: //" -``` - -### As joining node -```bash -nohup ~/bin/mesh-llm --model Qwen2.5-32B --join > /tmp/mesh.log 2>&1 & -``` - -### As lite client (no GPU, no model, API access only) -```bash -nohup ~/bin/mesh-llm --client --join > /tmp/mesh.log 2>&1 & -``` - -## Networking - -- **Only one side needs port forwarding.** Forward the `--bind-port` UDP port on the router of whichever node creates the mesh. -- The joining side does not need port forwarding. -- Check connectivity: the invite token embeds the creator's addresses. If the joiner can reach any of them over UDP, it works. -- If iroh relays are blocked on the remote network (DNS sinkhole), use `--relay ` to specify a reachable relay, or rely on direct UDP with port forwarding. - -## Verifying it works - -```bash -# Check processes are running -pgrep -la "mesh-llm|rpc-server|llama-server" - -# Check API -curl -s http://localhost:9337/v1/models - -# Test inference -curl -s http://localhost:9337/v1/chat/completions \ - -H 'Content-Type: application/json' \ - -d '{"model":"test","messages":[{"role":"user","content":"hi"}],"max_tokens":5}' -``` - -## Stopping - -```bash -pkill -f mesh-llm; pkill -f rpc-server; pkill -f llama-server -``` - -rpc-server and llama-server are child processes of mesh-llm, but killing the parent doesn't always kill them (they can become orphans with ppid=1). Always kill all three explicitly. - -## Troubleshooting - -| Symptom | Cause | Fix | -|---------|-------|-----| -| Exit 137 immediately | macOS quarantine xattr | `codesign -s - ~/bin/*; xattr -cr ~/bin/` | -| Empty reply from API | llama-server still loading | Wait. Check `/tmp/mesh-llm-llama-server.log` | -| "No inference server available" | Election in progress or llama-server crashed | Check `/tmp/mesh.log` for errors | -| Timeout waiting for tunnel maps | Peer disconnected during model load | Will auto-recover on next mesh change | -| Orphan rpc-server holding GPU memory | Parent mesh-llm was killed | `pkill -f rpc-server` | -| `*.n0.iroh-canary.iroh.link` DNS fails | Network has DNS sinkhole | Use `--bind-port` + UDP port forwarding instead of relays | - -## Log locations - -- `~/.mesh-llm/key` — persistent node identity -- `/tmp/mesh.log` — main process output (if started with `> /tmp/mesh.log 2>&1`) -- `/tmp/mesh-llm-llama-server.log` — llama-server stdout/stderr -- `/tmp/mesh-llm-rpc-.log` — rpc-server stdout/stderr diff --git a/.skills/mesh-join/SKILL.md b/.skills/mesh-join/SKILL.md new file mode 100644 index 0000000000..04aff0f53c --- /dev/null +++ b/.skills/mesh-join/SKILL.md @@ -0,0 +1,138 @@ +--- +name: mesh-join +description: Use this skill when creating, joining, publishing, or connecting mesh-llm nodes into a mesh — private meshes with invite tokens, the public mesh via --auto, named/published meshes, client-only nodes, NAT/firewall/bind issues, or verifying multi-node setups. +metadata: + short-description: Create and join mesh-llm meshes +--- + +# mesh-join + +Use this when wiring two or more mesh-llm nodes together, or attaching a +client-only node to an existing mesh. Per-platform install/serve steps live in +`deploy-macos` and `deploy-linux-gpu`; this skill covers the mesh topology +itself. Full reference: `docs/MESHES.md`. + +## Mental model + +- A node can **serve** models (`serve`), be an **API-only client** (`client`), + or both at once. +- Starting `serve` with no `--join`/`--discover`/`--auto` **creates a private + mesh** and emits an invite token. +- `--auto` discovers published meshes (Nostr by default) and joins the best + one — the public community mesh in practice. +- `--publish` makes your mesh discoverable; without it the mesh is private and + joinable only via the invite token. +- Every node exposes the same OpenAI API on `:9337`; `/v1/models` returns the + union of local + peer models and requests route by the `model` field. + +## Public mesh (the easy path) + +```bash +mesh-llm serve --auto # serve hardware + join the public mesh +mesh-llm serve --model --auto # serve a specific model + join +mesh-llm client --auto # API-only client, no GPU needed +``` + +Confirm joining via `discovery_joined` in the log (use `--log-format json` for +machine-readable events) or `peers` in `/api/status`. + +## Private mesh: create + join + +```bash +# Node A — creates the mesh, prints an invite token +mesh-llm serve --model Qwen3-8B-Q4_K_M +``` + +Grab the token: with `--log-format json` it is the `invite_token` event +(`token` field). In pretty mode it is printed to the terminal at startup. + +```bash +# Node B — another serving node +mesh-llm serve --join + +# Or an API-only client +mesh-llm client --join +``` + +`--join` is repeatable. Requirement-aware meshes (version/attestation policy) +use signed bootstrap tokens; legacy/private meshes use the older unsigned +token. Either way, the flow above is the same. + +## Published / named meshes + +```bash +# Publish for discovery, with a friendly name +mesh-llm serve --model Qwen3-8B-Q4_K_M --publish --mesh-name "lab-a" + +# Join by name from anywhere +mesh-llm serve --discover "lab-a" +mesh-llm client --discover "lab-a" + +# Browse what's out there +mesh-llm discover +mesh-llm discover --name "lab-a" +mesh-llm discover --model qwen --min-vram 24 +mesh-llm discover --auto # prints the best invite token (script-friendly) +``` + +`--mesh-name` without `--publish` is only a local label — the mesh stays +private. + +## LAN-only discovery + +`--mesh-discovery-mode mdns` keeps discovery and transport startup LAN-only: +no Nostr relays, no public iroh relays, no public STUN. Joins still require a +supplied matching invite token (mDNS advertisements only carry fingerprints). + +## NAT, firewalls, multi-interface hosts + +- Default Nostr mode uses managed iroh relays when direct UDP paths fail — + usually no port forwarding is needed. +- For direct connectivity, pin QUIC with `--bind-port ` on the + mesh-creating node and forward that UDP port. **Only the creator side needs + forwarding**; joiners don't. +- On multi-interface Linux/Docker hosts (`--network host`), iroh may advertise + bridge addresses like `172.17.0.1` that collide across machines. Pin the real + interface: `--bind-ip --bind-port `. +- `--listen-all` only affects the local HTTP API/console listener, not mesh + QUIC. + +## Verify a multi-node mesh + +```bash +# Peers on each node (expect N-1) +curl -s http://localhost:3131/api/status | python3 -m json.tool + +# Union of models across the mesh +curl -s http://localhost:9337/v1/models | python3 -m json.tool + +# Route to a specific peer's model — the response "model" field +# confirms which node answered +curl -s http://localhost:9337/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"","messages":[{"role":"user","content":"hi"}],"max_tokens":16}' +``` + +`/api/status` also reports the publication state (`private`, `public`, +`publish_failed`). + +## Splitting big models across nodes + +When one node cannot fit the model, use Skippy layer splits — same mesh +mechanics plus `--split` and a layer-package model on every serving node. See +`docs/SKIPPY_SPLITS.md`; diagnose readiness with `mesh-llm doctor split`. + +## Ownership / trust (private deployments) + +For owner-attested meshes: `mesh-llm auth init`, then start nodes with +`--owner-key`, `--node-label`, `--trust-policy`, `--trust-owner`. Details in +`docs/MESHES.md` ("Private ownership and trust"). + +## Gotchas + +- Two instances on one machine need distinct ports: `--port` (API, default + 9337) and `--console` (management, default 3131). +- Model load after join takes time — poll `/v1/models`, don't assume failure. +- Clients are zero-state on the host side: a `client` node doesn't appear in + the host's peer list. That's expected, not a bug. +- `--headless` only hides the web UI; the management API stays on `--console`. diff --git a/AGENTS.md b/AGENTS.md index 2cc47dd5ff..9c70211b5a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -461,10 +461,13 @@ Pull request titles and descriptions should be user-focused by default. ### Deploy to Remote ```bash -just bundle -# scp bundle to remote, tar xzf, codesign -s - the three binaries +just bundle # /tmp/mesh-llm-bundle.tar.gz — single mesh-llm binary +# scp bundle to remote, tar xzf, then on macOS: codesign -s - mesh-llm && xattr -cr ``` +For the full per-platform deploy flows, see the repo skills `.skills/deploy-macos/` +and `.skills/deploy-linux-gpu/`. + ### Cleanup Clean shutdown removes the instance's runtime directory automatically. Prefer the scoped runtime-aware commands first: From 96507ea991db9d97979344d32029d404afe733a8 Mon Sep 17 00:00:00 2001 From: Michael Neale Date: Fri, 12 Jun 2026 17:53:54 +1000 Subject: [PATCH 2/3] Refresh agent docs: fix stale repo map, add Windows deploy skill, correct design docs AGENTS.md repo map updates: - Remove phantom docs (MoE_PLAN, MoE_DEPLOY_DESIGN, LLAMA_CPP_FORK, docs/moe/) and phantom crates (mesh-api, mesh-api-ffi, mesh-host-core) from the map - Document the real CLI split: mesh-llm-cli (Clap), mesh-llm-commands (handlers), crates/mesh-llm/src/commands/ (dispatch); the host-runtime src/cli/ directory referenced by three rules no longer exists - Expand the workspace crate list to cover the ~20 unlisted crates, grouped by responsibility - Fix 'just clean-ui' -> 'just ui-clean' (also in mesh-llm-ui AGENTS.md), version bump location (workspace Cargo.toml), and in-tree plugin list (only blobstore remains; others are external plugins) - Release flow now documents the workflow_dispatch path as primary RELEASE.md: lead with the GitHub Actions dispatch path. New .skills/deploy-windows skill covering install.ps1 (irm | iex), flavor selection, the PR #828 PowerShell 5.1 gotchas, contrib/windows helpers, and verification; cross-linked from the other deploy skills. Design doc corrections (stale-as-fact content only): - DESIGN.md: architecture tree updated to the host-runtime module layout (no src/cli/, no rewrite.rs); NodeRole comments, stream table, bootstrap proxy, and B2B transfer sections no longer claim external llama-server/rpc-server processes - METRICS.md: /api/runtime/llama producer is the embedded runtime's lane status, not an external llama-server /metrics+/slots poller - message_protocol.md: add missing 0x0a PLUGIN_MESH_STREAM and 0x0d SUBPROTOCOL rows to the stream table - VIRTUAL_LLM.md, MODEL_ROUTER.md, LLAMA_STAGE_INTEGRATION_PLAN.md: add status framing so historical llama-server-era content reads as historical, with pointers to the current implementation paths --- .skills/README.md | 1 + .skills/deploy-linux-gpu/SKILL.md | 3 +- .skills/deploy-macos/SKILL.md | 1 + .skills/deploy-windows/SKILL.md | 145 ++++++++++++++++++++ AGENTS.md | 119 ++++++++++------ RELEASE.md | 15 ++ crates/mesh-llm-ui/AGENTS.md | 2 +- docs/design/DESIGN.md | 43 +++--- docs/design/LLAMA_STAGE_INTEGRATION_PLAN.md | 10 ++ docs/design/METRICS.md | 39 +++--- docs/design/MODEL_ROUTER.md | 9 ++ docs/design/VIRTUAL_LLM.md | 15 +- docs/design/message_protocol.md | 2 + 13 files changed, 321 insertions(+), 83 deletions(-) create mode 100644 .skills/deploy-windows/SKILL.md diff --git a/.skills/README.md b/.skills/README.md index 98bca108c5..1f71d2412e 100644 --- a/.skills/README.md +++ b/.skills/README.md @@ -7,6 +7,7 @@ Each is a focused, current how-to; deeper reference lives in `docs/`. |---|---| | [deploy-macos](deploy-macos/SKILL.md) | Install/launch mesh-llm on a macOS node (release install or dev-build bundle, codesign/quarantine, verify serving) | | [deploy-linux-gpu](deploy-linux-gpu/SKILL.md) | Install/launch mesh-llm on a remote Linux GPU node (Vast.ai/RunPod/self-managed CUDA, supervisor/systemd, verify serving) | +| [deploy-windows](deploy-windows/SKILL.md) | Install/launch mesh-llm on Windows (install.ps1 via `irm \| iex`, flavor selection CUDA/ROCm/Vulkan/CPU, contrib helper scripts, PowerShell gotchas) | | [mesh-join](mesh-join/SKILL.md) | Create/join/publish meshes: invite tokens, `--auto`, named meshes, client-only nodes, NAT/bind issues, multi-node verification | | [connect-agents](connect-agents/SKILL.md) | Point Goose/Claude Code/OpenCode/Pi or any OpenAI client at a running mesh; tool-call validation; blackboard | diff --git a/.skills/deploy-linux-gpu/SKILL.md b/.skills/deploy-linux-gpu/SKILL.md index b311372307..74dbf5aa76 100644 --- a/.skills/deploy-linux-gpu/SKILL.md +++ b/.skills/deploy-linux-gpu/SKILL.md @@ -1,6 +1,6 @@ --- name: deploy-linux-gpu -description: Use this skill when deploying, installing, launching, or serving mesh-llm on a remote Linux GPU node (rented GPUs like Vast.ai or RunPod, or a self-managed CUDA server), including installing the CUDA build, choosing a model and running +description: Use this skill when deploying, installing, launching, or serving mesh-llm on a remote Linux GPU node (rented GPUs like Vast.ai or RunPod, or a self-managed CUDA server), including installing the CUDA build, choosing a model, keeping it alive under a supervisor, and verifying it serves. metadata: short-description: Deploy mesh-llm on a remote Linux GPU node --- @@ -18,6 +18,7 @@ runtime. There are no `.dylib`/`codesign`/quarantine steps on Linux. note --auto flag tells it to join the public mesh. serve command with --model tells it to run a specific model. Examples here are for solo serving — don't read this in isolation: +- `deploy-macos` / `deploy-windows` — other platforms - `mesh-join` — creating/joining private and public meshes (tokens, NAT, multi-node) - `connect-agents` — pointing Goose/Claude Code/OpenCode/Pi at a running mesh - `docs/USAGE.md` — install details, service mode, model storage diff --git a/.skills/deploy-macos/SKILL.md b/.skills/deploy-macos/SKILL.md index 758f0a7c22..5c0680a599 100644 --- a/.skills/deploy-macos/SKILL.md +++ b/.skills/deploy-macos/SKILL.md @@ -18,6 +18,7 @@ you see instructions mentioning those, they are outdated. Related skills/docs: - `deploy-linux-gpu` — remote Linux/CUDA nodes +- `deploy-windows` — Windows nodes - `mesh-join` — creating/joining private and public meshes (tokens, NAT, multi-node) - `connect-agents` — pointing Goose/Claude Code/OpenCode/Pi at a running mesh - `docs/USAGE.md` — install details, service mode, model storage diff --git a/.skills/deploy-windows/SKILL.md b/.skills/deploy-windows/SKILL.md new file mode 100644 index 0000000000..adbbe4f7be --- /dev/null +++ b/.skills/deploy-windows/SKILL.md @@ -0,0 +1,145 @@ +--- +name: deploy-windows +description: Use this skill when installing, deploying, launching, serving, or troubleshooting mesh-llm on a Windows machine — PowerShell install via install.ps1, flavor selection (CUDA/ROCm/Vulkan/CPU), source builds, the contrib helper scripts, and verifying it serves. +metadata: + short-description: Deploy mesh-llm on a Windows node +--- + +# deploy-windows + +Use this when standing up mesh-llm on Windows. Counterpart to `deploy-macos` +and `deploy-linux-gpu`. Same single-binary embedded-runtime architecture; the +binary is `mesh-llm.exe` and release archives are `.zip` +(`mesh-llm-x86_64-pc-windows-msvc[-].zip`). + +Related skills/docs: + +- `mesh-join` — creating/joining meshes (tokens, NAT, multi-node) +- `connect-agents` — pointing Goose/Claude Code/OpenCode/Pi at a running mesh +- `docs/USAGE.md` — install details; `docs/CLI.md` — full command reference +- `contrib/windows/README.md` — local PowerShell helper scripts + +## The one rule that matters most + +**mesh-llm resolves and downloads the model itself.** Pass `--model ` and +it fetches the GGUF on first use. Do NOT pre-download with `hf` CLI tools. +(Only `--gguf` takes a local file path you manage yourself.) + +## Install (PowerShell) + +```powershell +irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex +``` + +Force a flavor non-interactively: + +```powershell +$env:MESH_LLM_INSTALL_FLAVOR = "vulkan" +irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex +``` + +Facts: + +- Flavors: `cuda-blackwell`, `cuda`, `rocm`, `vulkan`, `cpu`. **No Metal on + Windows.** The installer probes `nvidia-smi` (incl. compute capability for + Blackwell), ROCm tooling, and `vulkaninfo`, then recommends; when + input/output is redirected (scripted/SSH) it takes the recommendation + without prompting. +- Installs to `%LOCALAPPDATA%\mesh-llm\bin` (override: + `-InstallDir` / `MESH_LLM_INSTALL_DIR`) and prepends it to the **user** + `Path` unless `-NoPathUpdate`. Open a new shell, or use the full path, after + install. +- CUDA bundles ship their CUDA DLLs alongside `mesh-llm.exe` — no system CUDA + toolkit install is required to run. +- Other knobs: `-PreRelease` / `MESH_LLM_INSTALL_PRERELEASE=1`, + `MESH_LLM_REQUIRE_CHECKSUM=1` (makes a missing `.sha256` sidecar fatal; + default is warn-and-continue). + +### `irm | iex` gotchas (learned the hard way — PR #828) + +- A missing checksum sidecar on older releases **warns and continues**; that + is expected, not a failure. +- On Windows PowerShell 5.1, network errors during sidecar download can + surface as vague response-less `WebException`s rather than clean 404s. The + installer handles this; if you're debugging a fork/older script, know that + `iex` also breaks `[ValidateSet]` params (param init to `""` fails + validation before the script body runs). When `irm | iex` misbehaves, + fall back to downloading the script and running it as a file: + `irm -OutFile install.ps1; .\install.ps1 -Flavor vulkan`. + +## Build from source (dev) + +From a repo checkout (needs Rust, CMake + MSVC, Node, `just`): + +```powershell +just build # auto-detects cuda / rocm / vulkan / cpu +just build backend=vulkan # override backend +``` + +Output: `target\release\mesh-llm.exe` (for `just release-build`) or +`target\debug\` for `just build`. Windows release archives use the dedicated +`release-build-*-windows` / `release-bundle-*-windows` recipes. On native +Windows, `just check-release` skips the Bash-only parity checks. + +## Launch + +```powershell +mesh-llm serve --model unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL --auto +``` + +- Same surface as other platforms: `--auto` joins the public mesh; bare + `serve --model` creates a private mesh and prints an invite token + (see `mesh-join`). API on `:9337`, console on `:3131`. +- Pin a specific GPU with `--device` (e.g. `--device Vulkan1`, + `--device cuda:0`); list devices with `mesh-llm gpus`. +- There is **no service install on Windows** (no launchd/systemd equivalent + in `install.ps1`). Run it in a terminal, or wrap it yourself (Task + Scheduler / NSSM) — startup models go in `%USERPROFILE%\.mesh-llm\config.toml` + and bare `mesh-llm serve` reads them. + +### Repo helper scripts (dev checkouts) + +`contrib\windows\` wraps a local build (falls back to `mesh-llm` on `Path`): + +```powershell +.\contrib\windows\StartMeshServer.ps1 -Model Qwen2.5-3B-Instruct-Q4_K_M -Device Vulkan1 +.\contrib\windows\StartChat.ps1 -Model Qwen2.5-3B-Instruct-Q4_K_M +.\contrib\windows\CollectSplitDiagnostics.ps1 -Model -ConsoleUrls http://127.0.0.1:3131 -ApiUrls http://127.0.0.1:9337/v1 +``` + +The diagnostics collector attaches to running nodes and zips redacted API +payloads, GPU/process facts, and `skippy-native.log` tails — useful when +filing split/runtime issues from a Windows box. + +## Verify + +```powershell +mesh-llm --version +curl.exe -s http://localhost:9337/v1/models +curl.exe -s http://localhost:3131/api/status +curl.exe -s http://localhost:9337/v1/chat/completions -H "Content-Type: application/json" -d '{\"model\":\"auto\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":16}' +``` + +Use `curl.exe` explicitly — bare `curl` in PowerShell aliases to +`Invoke-WebRequest` with different argument semantics. Model load takes time; +poll `/v1/models` before concluding failure. + +## Logs, state, stop + +- Runtime/instance state: `%USERPROFILE%\.mesh-llm\runtime\\` — embedded + native logs at `logs\skippy-native.log`. +- Config: `%USERPROFILE%\.mesh-llm\config.toml`; identity: `.mesh-llm\key`. +- Stop: `mesh-llm stop` (preferred). Emergency: + `Stop-Process -Name mesh-llm -Force`. + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `irm \| iex` dies before any output | Old script / `[ValidateSet]`-style param bug | Update; or download script to a file and run it | +| "could not download checksum sidecar" hard failure | Old installer + release without `.sha256` | Update installer; missing sidecar should warn-and-continue | +| `mesh-llm` not found after install | New `Path` not in current shell | Open a new terminal or use `%LOCALAPPDATA%\mesh-llm\bin\mesh-llm.exe` | +| CUDA flavor won't start on new GPUs | Blackwell needs its own bundle | Install `cuda-blackwell` flavor (or let detection pick it) | +| GPU not used | Wrong flavor or device | `mesh-llm gpus`; reinstall correct flavor; `--device ` | +| curl JSON errors in PowerShell | `curl` is an IWR alias | Use `curl.exe`, or `Invoke-RestMethod` | +| Empty `/v1/models` | Model still downloading/loading | Wait; check `skippy-native.log` | diff --git a/AGENTS.md b/AGENTS.md index 9c70211b5a..99f252700d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ This repo (`mesh-llm`) contains mesh-llm — a Rust binary that pools GPUs over QUIC for distributed LLM inference using llama.cpp. -The workspace is split across many crates under `crates/`. The shipped binary `mesh-llm` is a thin shim (`crates/mesh-llm/`) that re-exports `mesh-llm-host-runtime`, where the bulk of host-side logic lives. A lighter parallel crate `mesh-client` (`mesh-llm-client`) carries the same domain shape for client-only usage. Embedded llama.cpp staged-runtime support lives in the `skippy-*` crates. +The workspace is split across many crates under `crates/`. The shipped binary `mesh-llm` (`crates/mesh-llm/`) is a thin entry point: it builds the Tokio runtime, parses the CLI via `mesh-llm-cli`, dispatches one-shot commands (via its `commands/` module and `mesh-llm-commands`), and hands the runtime surfaces (`serve` / `client`) to `mesh-llm-host-runtime`, where the bulk of host-side logic lives. A lighter parallel crate `mesh-client` (`mesh-llm-client`) carries the same domain shape for client-only usage. Embedded llama.cpp staged-runtime support lives in the `skippy-*` crates. ## Key Docs @@ -24,11 +24,9 @@ The workspace is split across many crates under `crates/`. The shipped binary `m | `docs/design/DESIGN.md` | Architecture, protocols, features | | `docs/design/TESTING.md` | Test playbook, scenarios, remote deploy | | `docs/design/MULTI_MODAL.md` | Multimodal design: capability model, blob plugin, console, routing | -| `docs/design/MoE_PLAN.md` | MoE expert sharding design | -| `docs/design/MoE_DEPLOY_DESIGN.md` | MoE auto-deploy UX | | `docs/design/VIRTUAL_LLM.md` | Virtual LLM engine (inter-model collaboration) | -| `docs/design/LLAMA_CPP_FORK.md` | llama.cpp fork: what's patched, how to update, how to sync | -| `docs/moe/README.md` | MoE analyzer, placement, and CLI planning notes | +| `docs/design/LLAMA_STAGE_INTEGRATION_PLAN.md` | llama.cpp staged-runtime integration and patch-queue background | +| `docs/SKIPPY.md` | Skippy integration readiness and parity notes | | `docs/plugins/README.md` | Plugin architecture and plugin development | | `fly/README.md` | Fly.io deployment (console + API apps) | | `tools/relay-fly-legacy/README.md` | Archived self-hosted iroh relay reference; production uses services.iroh.computer | @@ -74,7 +72,7 @@ just auto # build + stop + start with --auto just ui-dev # vite dev server with HMR just website-build # build website/ into docs/ for static hosting just website-dev # Eleventy dev server on :8765 -just clean-ui # nuke node_modules + dist (fixes stale npm state) +just ui-clean # nuke node_modules + dist (fixes stale npm state) ``` **Which build to use:** @@ -104,7 +102,7 @@ then copy `./target/release/mesh-llm`. If `just build` fails on the UI step with `npm error Exit handler never called!`, run: ```bash -just clean-ui +just ui-clean just build ``` @@ -129,34 +127,73 @@ libraries. The only durable llama.cpp patch queue is The workspace lives under `crates/`. The most important crates: -- `mesh-llm/` — shipped binary; thin shim with `main.rs` building the Tokio runtime and `lib.rs` re-exporting `mesh-llm-host-runtime`. Almost no domain code here. -- `mesh-llm-host-runtime/` — the host-side monolith. Owns runtime orchestration, mesh, inference, networking, API, CLI, plugins, models, system integration. This is where most changes land. +Shipped binary and CLI surface: + +- `mesh-llm/` — shipped binary; `main.rs` builds the Tokio runtime, `lib.rs` owns `run_main` (CLI parse → one-shot command dispatch via its `commands/` module → runtime handoff), and re-exports `mesh-llm-host-runtime` as a transitional shim. No domain logic here. +- `mesh-llm-cli/` — Clap types, argument parsing, serve/client surface normalization. No handlers. +- `mesh-llm-commands/` — user-facing command handlers (auth, gpus, update, skills, agent launchers like goose/pi/opencode/claude, plugin, benchmark, model packaging). +- `mesh-llm-tui/` — terminal UI and progress output surface. +- `mesh-llm-events/` — shared runtime event and output contracts (`OutputEvent`, log formats). + +Host and client runtimes: + +- `mesh-llm-host-runtime/` — the host-side monolith. Owns runtime orchestration, mesh, inference, networking, management API, plugins, models, system integration. This is where most changes land. - `mesh-client/` (`mesh-llm-client`) — lighter parallel client surface with its own `inference/`, `network/`, `models/`, `mesh/` modules. Used as a dev/test surface and for client-only deployments. +- `mesh-llm-node/`, `mesh-llm-embedded-runtime/` — embeddable node primitives and in-process full-node embedding API. +- `mesh-llm-config/` — configuration parsing and validation (`~/.mesh-llm/config.toml`). - `mesh-llm-ui/` — React web console and embedded asset crate (shadcn/ui patterns, see https://ui.shadcn.com/llms.txt). +- `mesh-llm-console-server/` — static file server for embedded console assets. + +Shared foundations: + - `mesh-llm-types/` — shared model/capability types used across crates. - `mesh-llm-protocol/` — wire protocol types and protobuf bindings. - `mesh-llm-routing/` — routing primitives shared across host and client. - `mesh-llm-system/` — machine-local hardware, benchmark, autoupdate, process helpers. +- `mesh-llm-identity/` — owner identity and envelope crypto primitives. +- `mesh-llm-guardrails/` — guardrail and compaction primitives for OpenAI-compatible paths. +- `mesh-llm-hardware-profile/`, `mesh-llm-native-runtime/`, `mesh-llm-runtime-install/` — hardware profile detection, native runtime manifest/selection, runtime download/install/cache. - `mesh-llm-plugin/` — plugin runtime/DSL primitives. -- `mesh-llm-identity/` — identity primitives. -- `mesh-api/`, `mesh-api-ffi/` — management API surface and FFI bindings. -- `mesh-host-core/` — minimal shared host core. +- `mesh-llm-plugin-manager/` — plugin package management (catalog, install, store). +- `mesh-llm-skills/` — agent skill data model and installer primitives. + +SDK and API surface: + +- `mesh-llm-sdk/` — Rust SDK facade for clients and embedded serving. +- `mesh-llm-api-server/`, `mesh-llm-api-client/` — public Rust SDK APIs for embedding nodes / client-only use. +- `mesh-llm-ffi/`, `mesh-llm-nodejs/` — FFI bindings and Node.js native addon. - `openai-frontend/` — OpenAI-compatible HTTP frontend (chat, completions, responses, models). +- `mesh-mixture-of-agents/` — Mixture-of-Agents fan-out/arbitration engine. + +Models: + - `model-artifact/`, `model-hf/`, `model-package/`, `model-ref/`, `model-resolver/` — model catalog, HuggingFace download, packaging, reference resolution. + +Embedded staged runtime (skippy): + - `skippy-ffi/` — Rust ABI bindings to the patched llama.cpp staged runtime. - `skippy-runtime/` — Rust-side staged runtime, package materialization, model info. - `skippy-server/` — embedded staged-runtime serving (frontend, binary transport, runtime state, embedded HTTP). - `skippy-protocol/`, `skippy-topology/`, `skippy-coordinator/`, `skippy-cache/`, `skippy-prompt/`, `skippy-metrics/`, `skippy-bench/`, `skippy-correctness/`, `skippy-model-package/` — supporting skippy infrastructure. + +Tools and benchmarks: + - `metrics-server/` — standalone metrics collector binary. - `mesh-llm-gpu-bench/`, `llama-spec-bench/`, `mesh-llm-test-harness/` — benchmarking and test harness binaries. +This list covers the crates you are most likely to touch; check `crates/` and each crate's `Cargo.toml` description for anything not listed. + Other top-level directories: -- `docs/` — Project docs, grouped by topic. +- `docs/` — Project docs, grouped by topic (see `docs/README.md` for the map). - `website/` — Eleventy source for the public website; builds into `docs/`. - `docs/design/` — Architecture, protocol, and testing docs. -- `docs/moe/` — MoE ranking, placement, and CLI plans. +- `docs/skippy/` — Skippy family certification, configuration, benchmarks, parity. - `docs/plugins/` — Plugin architecture docs and plans. +- `docs/specs/` — Focused behavior specs for individual features. +- `.skills/` — Repo agent skills (per-platform deploy, mesh-join, connect-agents); auto-picked-up by agents. +- `.agents/skills/` — Maintainer-facing agent skills (skippy internals, patch queues, benchmarks). +- `sdk/` — SDK packaging for Node, Swift, Kotlin. - `fly/` — Fly.io deployment (console + API client apps). - `tools/relay-fly-legacy/` — Archived self-hosted iroh relay reference; production uses services.iroh.computer. - `evals/` — Benchmarking and evaluation scripts. @@ -173,7 +210,6 @@ The host-runtime crate root should stay minimal. Use semantic ownership for module placement. Inside `crates/mesh-llm-host-runtime/src/`: -- `cli/` — Clap types, command parsing, command dispatch, and user-facing command handlers. - `runtime/` — top-level process orchestration, startup/runtime coordination, runtime instance, capacity, split planning, proxy lifecycle. - `network/` — request routing, proxying, tunneling, relay/discovery networking, request-affinity logic, endpoint rewrite, target health, OpenAI transport glue. - `inference/` — model-serving logic, election, launch, pipeline, MoE behavior, embedded skippy integration. @@ -181,7 +217,7 @@ Use semantic ownership for module placement. Inside `crates/mesh-llm-host-runtim - `models/` — model catalog, resolution, downloads, local model storage, model metadata. - `mesh/` — peer membership, gossip, heartbeats, identity, peer state, mesh node behavior. - `plugin/` — plugin host, plugin runtime, transport, config, MCP bridge support. -- `plugins/` — concrete plugins (blobstore, flash_moe, openai_endpoint, telemetry, blackboard). +- `plugins/` — concrete in-tree plugins (currently `blobstore/`; most plugins like blackboard, openai-endpoint, and flash-moe/ln are external packages installed via `mesh-llm plugins install`). - `api/` — management API surface and route handling. - `protocol/` — wire protocol types, encoding/decoding, conversions. - `runtime_data/` — runtime data collection, API views, status snapshots. @@ -189,9 +225,10 @@ Use semantic ownership for module placement. Inside `crates/mesh-llm-host-runtim CLI ownership rule. -- All command handlers belong under `crates/mesh-llm-host-runtime/src/cli/`, usually `cli/commands/`. -- Domain modules should not own Clap parsing or top-level command dispatch. -- Domain modules may expose reusable functions that CLI handlers call. +- Clap types, argument parsing, and surface normalization belong in `crates/mesh-llm-cli/`. +- User-facing command handlers belong in `crates/mesh-llm-commands/` (or the shipped binary's `crates/mesh-llm/src/commands/` dispatch layer for wiring). +- Domain modules in `mesh-llm-host-runtime` should not own Clap parsing or top-level command dispatch. +- Domain modules may expose reusable functions that command handlers call. Do not introduce generic buckets. @@ -244,8 +281,8 @@ Current structure notes. - Request-affinity code belongs with networking/routing behavior (`network/affinity.rs`), not `system/`. - Plugin MCP support belongs inside `mesh-llm-host-runtime/src/plugin/`, not as a separate root module. -- Model command handlers belong in `mesh-llm-host-runtime/src/cli/commands/`; `models/` should stay domain-focused. -- The shipped binary crate (`crates/mesh-llm/`) should remain a thin shim; do not move domain logic into it. +- Model command handlers belong in `mesh-llm-commands/` (or `crates/mesh-llm/src/commands/` for dispatch wiring); host-runtime `models/` should stay domain-focused. +- The shipped binary crate (`crates/mesh-llm/`) carries CLI dispatch wiring only; do not move domain logic into it. ## Code Quality Rules for New Code @@ -266,7 +303,7 @@ Current structure notes. Host runtime (main monolith — `crates/mesh-llm-host-runtime/src/`): -- `lib.rs` — crate entry; exposes `run_main` (called from `crates/mesh-llm/src/main.rs`). +- `lib.rs` — crate entry; exposes the runtime entrypoints (`run_runtime_initialized`, `initialize_host_runtime`) called from `crates/mesh-llm/src/lib.rs`. - `runtime/mod.rs` — top-level startup flows, runtime orchestration, command dispatch. - `runtime/instance.rs` — per-instance runtime directory management: `InstanceRuntime`, pidfiles, flock liveness, scoped orphan reaping, local instance scanning. - `runtime/local.rs` — local model startup loop. @@ -292,13 +329,15 @@ Host runtime (main monolith — `crates/mesh-llm-host-runtime/src/`): - `models/capabilities.rs` — multimodal/vision/audio/reasoning capability inference. - `models/resolve/` — model reference resolution. - `plugins/blobstore/mod.rs` — request-scoped media object storage for multimodal. -- `plugins/flash_moe/`, `plugins/openai_endpoint/`, `plugins/telemetry/`, `plugins/blackboard/` — other in-tree plugins. -- `cli/mod.rs`, `cli/commands/` — Clap command surface and dispatch. +- `plugin/` — plugin host, runtime, transport, config, MCP bridge (external plugins install via `mesh-llm plugins install`). -Shipped binary (`crates/mesh-llm/src/`): +Shipped binary and CLI (`crates/mesh-llm/src/`, `crates/mesh-llm-cli/src/`, `crates/mesh-llm-commands/src/`): -- `main.rs` — builds the Tokio runtime (custom stack size via `MESH_TOKIO_STACK_SIZE`) and calls `mesh_llm::run_main()`. -- `lib.rs` — `pub use mesh_llm_host_runtime::*;` (transitional re-export). +- `mesh-llm/src/main.rs` — builds the Tokio runtime (custom stack size via `MESH_TOKIO_STACK_SIZE`) and calls `mesh_llm::run_main()`. +- `mesh-llm/src/lib.rs` — `run_main`: CLI parse, one-shot command dispatch, runtime handoff; plus a transitional `pub use mesh_llm_host_runtime::*;` re-export. +- `mesh-llm/src/commands/` — dispatch wiring from parsed `Command` values to handlers. +- `mesh-llm-cli/src/parser.rs` — Clap surface, serve/client arg normalization, advanced help. +- `mesh-llm-commands/src/` — user-facing handlers (auth, gpus, update, skills, agent launchers, plugin, benchmark). Embedded staged runtime (`crates/skippy-*`): @@ -430,7 +469,7 @@ Before committing, run the local checks most likely to fail in CI for the files ### UI changes - Use the repo's supported workflow and run `just build`. -- If `just build` fails on the UI step with `npm error Exit handler never called!`, run `just clean-ui` and then rerun `just build`. +- If `just build` fails on the UI step with `npm error Exit handler never called!`, run `just ui-clean` and then rerun `just build`. ### Commit standard @@ -465,8 +504,8 @@ just bundle # /tmp/mesh-llm-bundle.tar.gz — single mesh-llm binary # scp bundle to remote, tar xzf, then on macOS: codesign -s - mesh-llm && xattr -cr ``` -For the full per-platform deploy flows, see the repo skills `.skills/deploy-macos/` -and `.skills/deploy-linux-gpu/`. +For the full per-platform deploy flows, see the repo skills `.skills/deploy-macos/`, +`.skills/deploy-linux-gpu/`, and `.skills/deploy-windows/`. ### Cleanup @@ -503,7 +542,7 @@ bash -c './target/debug/mesh-llm serve --model "..." --auto > /tmp/mesh.log 2>&1 **Every deploy to test machines MUST follow this checklist.** ### Before starting nodes -1. **Bump VERSION** in `crates/mesh-llm/Cargo.toml` (the shipped binary crate) so you can verify the running binary is new code. +1. **Bump VERSION** in the root `Cargo.toml` (`[workspace.package] version`; crates inherit it via `version.workspace = true`) so you can verify the running binary is new code. 2. `just build && just bundle` 3. Kill ALL processes on ALL nodes — `pkill -9 -f mesh-llm` 4. Verify clean — `ps -eo pid,args | grep -E 'mesh-llm' | grep -v grep` must be empty. @@ -548,19 +587,11 @@ For stale instances (crashed mesh-llm leaving behind a runtime dir): See `RELEASE.md` for the full process. -Current release flow: +Current release flow: kick off the **Release** workflow (`.github/workflows/release.yml`) from the GitHub Actions UI via `workflow_dispatch` with the version input (e.g. `v0.X.Y`). + +The dispatched workflow handles everything: it bumps versions via `scripts/release-version.sh`, generates and patches the SwiftPM manifest, packages SDK console assets, creates and pushes the release tag at a release-prep commit, builds the full artifact matrix (macOS, Linux CPU/ARM64/CUDA/CUDA-Blackwell/ROCm/Vulkan, Windows CPU/CUDA/ROCm/Vulkan), and publishes the GitHub release. Dispatch inputs include `skip_gpu_bundles` and `canary` (dry-run: build + smoke without publishing). -1. Build and verify locally: - ```bash - just build - just bundle - ``` -2. Release from a clean local `main` branch: - ```bash - just release v0.X.Y - ``` - This bumps the version, refreshes `Cargo.lock` without upgrading dependencies, commits as `v0.X.Y: release`, pushes `main`, and then pushes only the new release tag. -3. Pushing a `v*` tag triggers `.github/workflows/release.yml`, which builds the release artifacts on Linux CPU, Linux CUDA, and macOS and creates the GitHub release automatically. +Pushing a `v*` tag manually also triggers the workflow, but that path requires preparing `Package.swift` and SDK console assets in the tag commit yourself — see `RELEASE.md`. Prefer the dispatch path. ### Installer checksum sidecars @@ -589,5 +620,5 @@ Test machine IPs, SSH details, and passwords are in `~/Documents/private-note.tx - **No `api_key_token` feature** — explicitly rejected, removed in v0.26.0. - **No credentials in tracked files** — IPs, passwords, SSH commands belong in `~/Documents/private-note.txt` only. -- **No domain logic in `crates/mesh-llm/src/`** — that crate is a thin shim over `mesh-llm-host-runtime`; put new code in the host-runtime crate (or a more specific peer crate). +- **No domain logic in `crates/mesh-llm/src/`** — that crate is CLI dispatch wiring over `mesh-llm-cli` / `mesh-llm-commands` / `mesh-llm-host-runtime`; put new domain code in the host-runtime crate (or a more specific peer crate). - **No external `llama-server` / `rpc-server` runtime lane** — the embedded staged runtime via patched llama.cpp is the only supported path. diff --git a/RELEASE.md b/RELEASE.md index 82bf70b447..b23e0ced2e 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,5 +1,20 @@ # Releasing mesh-llm +## Preferred path: dispatch from GitHub + +Releases are normally cut by running the **Release** workflow +(`.github/workflows/release.yml`) from the GitHub Actions UI via +`workflow_dispatch` with the version input (for example `v0.31.0`). The +dispatched workflow bumps versions, generates and patches the SwiftPM +manifest, packages SDK console assets, creates and pushes the release tag, +builds all platform bundles, and publishes the GitHub release. Dispatch inputs +include `skip_gpu_bundles` and `canary` (dry-run: build and smoke everything +without publishing). + +The sections below document the underlying steps. They matter when releasing +manually via a tag push, debugging the workflow, or validating bundles +locally. + ## Prerequisites - `just` installed diff --git a/crates/mesh-llm-ui/AGENTS.md b/crates/mesh-llm-ui/AGENTS.md index f52741e0ed..d86933294c 100644 --- a/crates/mesh-llm-ui/AGENTS.md +++ b/crates/mesh-llm-ui/AGENTS.md @@ -14,7 +14,7 @@ From repo root: - `just ui-dev` - `just ui-test` -- `just clean-ui` +- `just ui-clean` - `scripts/build-ui.sh crates/mesh-llm-ui` Package-local scripts: diff --git a/docs/design/DESIGN.md b/docs/design/DESIGN.md index 3fec81edea..5489703dc8 100644 --- a/docs/design/DESIGN.md +++ b/docs/design/DESIGN.md @@ -12,30 +12,36 @@ need to be split across peers. ## Architecture +The workspace is split across many crates (see the root `AGENTS.md` for the +full map). The shipped `mesh-llm` binary is a thin entry point: Clap parsing +lives in `mesh-llm-cli`, one-shot command handlers in `mesh-llm-commands`, and +the host-side runtime in `mesh-llm-host-runtime`, whose module layout carries +most of the behavior described in this document: + ``` -src/ -├── main.rs CLI args, orchestration (auto, idle, passive) -├── lib.rs Crate root re-exports +crates/mesh-llm-host-runtime/src/ +├── lib.rs Crate entry; runtime entrypoints called by the binary ├── api/ Management API (:3131): status, models, search, events, discover -├── cli/ Clap types, command parsing, command handlers ├── crypto/ Key management, envelope encryption, keychain ├── inference/ │ ├── election.rs Per-model host election and split planning │ ├── skippy/ Embedded staged runtime integration +│ ├── virtual_llm.rs Inter-model collaboration hooks │ └── pipeline.rs Inference pipeline coordination ├── mesh/mod.rs Node struct, QUIC endpoint, gossip, peer management, mesh identity ├── models/ │ ├── capabilities.rs Vision/audio/multimodal/reasoning capability inference │ ├── catalog.rs Model catalog and HuggingFace downloads -│ ├── resolve.rs Model path resolution, mmproj lookup +│ ├── resolve/ Model reference resolution, mmproj lookup │ └── ... GGUF parsing, inventory, search, topology ├── network/ │ ├── proxy.rs HTTP proxy: request parsing, model routing, response helpers │ ├── router.rs Request classification, model scoring, multimodal routing -│ ├── tunnel.rs TCP ↔ QUIC relay (RPC + HTTP), B2B rewrite map +│ ├── tunnel.rs TCP ↔ QUIC relay, B2B tunnel map │ ├── nostr.rs Nostr discovery, score_mesh(), smart_auto() │ ├── affinity.rs Prefix-affinity request routing -│ └── rewrite.rs REGISTER_PEER interception and endpoint rewriting +│ └── openai/ OpenAI transport glue +├── plugin/ Plugin host, runtime, transport, MCP bridge ├── plugins/ │ └── blobstore/ Request-scoped media object storage for multimodal ├── protocol/ Wire protocol types, protobuf encoding/decoding @@ -48,8 +54,8 @@ src/ ```rust enum NodeRole { - Worker, // rpc-server, provides GPU compute - Host { http_port: u16 }, // llama-server + rpc-server, serves HTTP API + Worker, // provides staged GPU compute for a model + Host { http_port: u16 }, // runs the local serving runtime, serves HTTP API Client, // no compute, just API access via tunnel } ``` @@ -72,9 +78,9 @@ Single QUIC connection per peer, multiplexed by 1-byte prefix: | Byte | Type | Purpose | Format | |------|------|---------|--------| | 0x01 | GOSSIP | Peer announcements (role, serving, VRAM, models, explicit interest, demand, mesh_id) | protobuf `GossipFrame` | -| 0x02 | TUNNEL_RPC | TCP relay to remote rpc-server | raw TCP relay | +| 0x02 | TUNNEL | TCP relay for remote runtime compute traffic | raw TCP relay | | 0x03 | TUNNEL_MAP | B2B tunnel port map exchange | protobuf `TunnelMap` | -| 0x04 | TUNNEL_HTTP | TCP relay to remote llama-server HTTP | raw TCP relay | +| 0x04 | TUNNEL_HTTP | TCP relay to a remote node's HTTP API | raw TCP relay | | 0x05 | ROUTE_REQUEST | Routing table for passive nodes (hosts + models) | protobuf `RouteTableRequest` / `RouteTable` | | 0x06 | PEER_DOWN | Death broadcast (immediate, from any node that detects a death) | protobuf `PeerDown` | | 0x07 | PEER_LEAVING | Clean shutdown broadcast (ctrl-c) | protobuf `PeerLeaving` | @@ -158,7 +164,7 @@ Changing mesh requirements creates a new mesh. ## Bootstrap Proxy When joining an existing mesh, a tunnel-only API proxy starts immediately on the -local port — before rpc-server or llama-server are ready. Requests are tunneled to +local port — before the local serving runtime is ready. Requests are tunneled to mesh hosts via QUIC. When the real `api_proxy` is ready, it takes over the listener. This gives instant API access (within seconds of `mesh-llm serve --join`) while the local @@ -241,11 +247,14 @@ When a model requires splitting across nodes: ## B2B Direct Transfer -When the model is split across workers, activation tensors flow directly -between workers (1 hop) instead of through the host (2 hops): -1. Each node broadcasts `{EndpointId → tunnel_port}` via `STREAM_TUNNEL_MAP` -2. `rewrite.rs` intercepts `REGISTER_PEER` and rewrites ports for local tunnels -3. llama.cpp's `PUSH_TENSOR_TO_PEER` goes directly between workers +When the model is split across workers, activation data flows directly +between workers (1 hop) instead of through the host (2 hops). Each node +broadcasts `{EndpointId → tunnel_port}` via `STREAM_TUNNEL_MAP` so peers can +open direct worker-to-worker tunnels. In the current embedded staged runtime, +stage-to-stage activation traffic uses the Skippy binary stage transport over +these direct paths; the legacy llama.cpp RPC rewrite path (`rewrite.rs` +intercepting `REGISTER_PEER`) survives only in the `mesh-client` compatibility +surface. ## Management API (port 3131) diff --git a/docs/design/LLAMA_STAGE_INTEGRATION_PLAN.md b/docs/design/LLAMA_STAGE_INTEGRATION_PLAN.md index f90528a0a8..9a5ea682eb 100644 --- a/docs/design/LLAMA_STAGE_INTEGRATION_PLAN.md +++ b/docs/design/LLAMA_STAGE_INTEGRATION_PLAN.md @@ -1,5 +1,15 @@ # llama-stage-runtime Integration Plan +## Status: completed — historical plan + +This migration has shipped. mesh-llm now embeds the staged (Skippy) runtime +behind the C ABI, built from the upstream llama.cpp pin plus the patch queue in +`third_party/llama.cpp/patches` (pinned by `third_party/llama.cpp/upstream.txt`). +The external `llama-server` / `rpc-server` runtime lane described below as +"Current State" no longer exists and must not be reintroduced. Keep this +document as background for why the patch queue and embedded ABI are shaped the +way they are; do not treat its "current state" sections as current. + ## Purpose Track the planned integration of `/Users/jdumay/code/llama-stage-runtime` into diff --git a/docs/design/METRICS.md b/docs/design/METRICS.md index 5c63f94506..c9fb186926 100644 --- a/docs/design/METRICS.md +++ b/docs/design/METRICS.md @@ -52,43 +52,46 @@ All currently implemented metrics are `local-only`. None of them are new gossip Layer: **Information** | Scope: **local-only** -Local llama.cpp runtime diagnostics for the current node. These values are produced next to -the llama-server process, stored in the runtime-data collector, and exposed to the local API/UI -so an operator can see whether llama.cpp is reporting metrics and which slots are currently busy. -They are not gossiped, not protocol fields, and not routing inputs. +Local serving-runtime diagnostics for the current node. These values are produced +by the embedded staged (Skippy) runtime, stored in the runtime-data collector, and +exposed to the local API/UI so an operator can see whether the runtime is serving +and which slots/lanes are currently busy. They are not gossiped, not protocol +fields, and not routing inputs. Producer flow: ```text -llama-server /metrics + /slots - -> inference runtime poller +embedded runtime lane/slot status + -> LocalRuntimeModelHandle::llama_slots_snapshot -> RuntimeDataProducer -> RuntimeDataCollector -> GET /api/runtime/llama -> local UI ``` -The `/metrics` endpoint is Prometheus text owned by llama.cpp, so mesh-llm treats it as a -local diagnostic source rather than as a stable mesh metric registry. Only explicitly permitted -metric names and labels are retained as structured `items.metrics`; unknown series are ignored. -The `/slots` endpoint is normalized into `items.slots` with the original slot array index preserved, -because index order is how operators can correlate busy/idle state with llama.cpp slot position. -Both llama.cpp responses are read with explicit byte limits. Raw slot diagnostics are capped to a -bounded number of entries and large per-slot JSON fragments are replaced with a truncation marker. +Slot snapshots are normalized into `items.slots` with the original slot/lane +index preserved, because index order is how operators correlate busy/idle state +with runtime lane position. The payload shape (and the `llama` naming) is kept +from the earlier external-llama-server era for API stability: the `items.metrics` +rows and Prometheus-derived fields remain supported as bounded, permitted +diagnostic items, but in the embedded runtime the primary producer is the +runtime's own lane status. Raw slot diagnostics are capped to a bounded number +of entries and large per-slot JSON fragments are replaced with a truncation +marker. | Field(s) | Meaning | | --- | --- | -| `metrics.status`, `slots.status` | Whether the local llama.cpp `/metrics` and `/slots` endpoints are ready, unavailable, or returning errors | -| `items.metrics[]` | Bounded, permitted metric items derived from llama.cpp Prometheus samples | -| `items.slots[]` | Slot items preserving original `/slots` array index and busy state | +| `metrics.status`, `slots.status` | Whether local runtime metrics and slot diagnostics are ready, unavailable, or returning errors | +| `items.metrics[]` | Bounded, permitted metric items (legacy Prometheus-derived shape) | +| `items.slots[]` | Slot items preserving original slot/lane index and busy state | | `items.slots_total`, `items.slots_busy` | Bounded local slot activity counts for operator display | | `metrics.raw_text` | Truncated local diagnostic text for debugging, not a stable API for routing or peer behavior | | `slots.slots[]` | Bounded raw slot diagnostics retained for local debugging; consumers should prefer `items.slots[]` | Routing clarification: these diagnostics are routing-adjacent only in the sense that a human can use them to debug serving health. They are not consumed by routing today. If future routing logic needs -llama.cpp health, it should consume a separate bounded runtime signal with explicit freshness and -error semantics, not raw Prometheus samples or slot internals. +runtime health, it should consume a separate bounded runtime signal with explicit freshness and +error semantics, not raw diagnostic samples or slot internals. ### `/api/status` `routing_metrics` diff --git a/docs/design/MODEL_ROUTER.md b/docs/design/MODEL_ROUTER.md index 805115a8a8..9c4ed7d689 100644 --- a/docs/design/MODEL_ROUTER.md +++ b/docs/design/MODEL_ROUTER.md @@ -1,5 +1,14 @@ # Model Router Design +## Status: Phase 1 implemented — historical design + +The heuristic router shipped: request classification (`Category`) and model +scoring live in `crates/mesh-llm-host-runtime/src/network/router.rs`. Later +phases remain partly aspirational; see `ROUTER_V2.md` for the follow-on design. +References below to per-model `llama-server` processes predate the embedded +staged runtime — today multiple local models run inside the single `mesh-llm` +process. + ## Overview mesh-llm becomes a smart routing layer: multiple models across the mesh, requests classified and routed to the best model for the job. diff --git a/docs/design/VIRTUAL_LLM.md b/docs/design/VIRTUAL_LLM.md index 3a568ac157..7cef95cd2d 100644 --- a/docs/design/VIRTUAL_LLM.md +++ b/docs/design/VIRTUAL_LLM.md @@ -1,14 +1,25 @@ # Virtual LLM Engine -Callback hooks from llama-server into mesh-llm during inference. +Callback hooks from the serving runtime into mesh-llm during inference. Related: [#183](https://github.com/michaelneale/mesh-llm/issues/183), [#165](https://github.com/michaelneale/mesh-llm/issues/165), [PR #225](https://github.com/michaelneale/mesh-llm/pull/225) +> **Note:** This design predates the embedded staged runtime. Where it says +> "llama-server", read "the serving runtime". The hook concept and the Rust-side +> handlers (`inference/virtual_llm.rs`, `inference/consult.rs`, +> `api/routes/mesh_hook.rs`, `openai-frontend/src/hooks.rs`) are current; the +> embedded Skippy path applies the same hooks in-process +> (`inference/skippy/hooks.rs`, `skippy-server` before-chat hooks) rather than +> over a localhost HTTP callback from an external process. The +> "Temporary co-iteration setup" section is historical — `llama-patches/` and +> `sync.sh` no longer exist; C++ changes live in the +> `third_party/llama.cpp/patches` queue. + --- ## What it does -llama-server detects when it might need help and calls mesh-llm on localhost. mesh-llm consults other models in the mesh and replies with context to inject. The caller sees one seamless response. +The serving runtime detects when it might need help and calls mesh-llm. mesh-llm consults other models in the mesh and replies with context to inject. The caller sees one seamless response. Three hook points (Hook 1, Hook 2, Hook 2b), all synchronous — each is a blocking POST to `http://localhost:{mesh_port}/mesh/hook`. diff --git a/docs/design/message_protocol.md b/docs/design/message_protocol.md index 389d048afb..c9d79f7e87 100644 --- a/docs/design/message_protocol.md +++ b/docs/design/message_protocol.md @@ -23,8 +23,10 @@ Each QUIC connection carries multiple logical streams, distinguished by a 1-byte | 0x07 | PEER_LEAVING | send | protobuf `PeerLeaving` | | 0x08 | PLUGIN_CHANNEL | bidirectional | plugin protocol | | 0x09 | PLUGIN_BULK_TRANSFER | send | plugin protocol bulk data | +| 0x0a | PLUGIN_MESH_STREAM | bidirectional | plugin protocol mesh stream | | 0x0b | CONFIG_SUBSCRIBE | reserved | legacy mesh-plane config stream ID; do not reuse | | 0x0c | CONFIG_PUSH | reserved | legacy mesh-plane config stream ID; do not reuse | +| 0x0d | SUBPROTOCOL | bidirectional | protobuf `MeshSubprotocolOpen`, then subprotocol-owned framing | Streams 0x02 and 0x04 are raw TCP relay tunnels. They carry llama.cpp RPC and HTTP traffic respectively and are not subject to protobuf framing or generation validation. From 2e04ce4a76e3e5ce69301bec792d59db88fe2066 Mon Sep 17 00:00:00 2001 From: Michael Neale Date: Fri, 12 Jun 2026 18:11:33 +1000 Subject: [PATCH 3/3] Add language specifier to DESIGN.md architecture code fence --- docs/design/DESIGN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/DESIGN.md b/docs/design/DESIGN.md index 5489703dc8..e437cd1240 100644 --- a/docs/design/DESIGN.md +++ b/docs/design/DESIGN.md @@ -18,7 +18,7 @@ lives in `mesh-llm-cli`, one-shot command handlers in `mesh-llm-commands`, and the host-side runtime in `mesh-llm-host-runtime`, whose module layout carries most of the behavior described in this document: -``` +```text crates/mesh-llm-host-runtime/src/ ├── lib.rs Crate entry; runtime entrypoints called by the binary ├── api/ Management API (:3131): status, models, search, events, discover