Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions tools/server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,58 @@ The server includes a set of built-in tools that enable the LLM to access the lo

To use this feature, start the server with `--tools all`. You can also enable only specific tools by passing a comma-separated list: `--tools name1,name2,...`. Run `--help` for the full list of available tool names.

### MCP servers

Besides the built-in tools, the server can expose tools coming from MCP servers, added in [#26062](https://github.com/ggml-org/llama.cpp/pull/26062). Only the stdio transport is supported: such a server is a child process reading JSON-RPC messages on its stdin and writing replies on its stdout, so nothing has to be started or maintained outside `llama-server`.

Servers are declared in a Cursor-compatible JSON file:

```json
{
"mcpServers": {
"example": { "command": "/path/to/server", "args": [] }
}
}
```

```sh
llama-server -m model.gguf --mcp-servers-config mcp.json
```

The same JSON can be passed inline with `--mcp-servers-json`. Each entry under `mcpServers` accepts:

| Key | Explanation |
| --- | ----------- |
| `command` | executable to spawn, required, entries without it are skipped |
| `args` | array of arguments |
| `env` | object merged over the parent environment |
| `cwd` | working directory of the child process |
| `timeout_ms` | per-tool-call timeout (default: 30000) |

Every server is spawned once at startup to list its tools, then stopped, and respawned on demand when one of its tools is called. Tools are exposed as `<server>_<tool>` alongside the built-in ones: they show up in the Web UI and in `GET /tools`, and the model calls them like any other tool. A name colliding with an already registered tool is skipped. This is independent of `--tools`, MCP servers can be the only tools available.

The child process runs with the same privileges as the server, so only declare commands you trust. As with `--tools`, `--cors-origins` then defaults to `localhost`.

Note: `--ui-mcp-proxy` is unrelated, it only lets the Web UI reach remote MCP servers from the browser.

Any server written against the [MCP specification](https://modelcontextprotocol.io) works as is, whether it uses an official SDK or not: the transport is one JSON-RPC message per line on stdio, so a script wrapping an existing program is a valid server too.

### CORS

By default the server reflects any `Origin` header back with credentials allowed. This matches the old, always-on `*` behavior and is fine as long as the server only exposes stateless, read-only endpoints.

Enabling `--tools` or `--agent` exposes file read/write over the API, so in that case `--cors-origins` defaults to `localhost` instead: only pages served from localhost can reach the server. Pass `--cors-origins` explicitly to override either default.

Recommended `--cors-origins` setting, depending on where the server runs:

| Deployment | Recommendation |
| ---------- | --------------- |
| Public | set an API key, put the server behind a reverse proxy, `--cors-origins` optional |
| Local network | set `--cors-origins` to your frontend's origin |
| Same machine | `--cors-origins localhost` (default once `--agent` is set) |

Related flags: `--cors-origins`, `--cors-methods`, `--cors-headers`, `--cors-credentials` / `--no-cors-credentials`. Background and rationale: [#25655](https://github.com/ggml-org/llama.cpp/pull/25655).

## Build

`llama-server` is built alongside everything else from the root of the project
Expand Down