feat: localai assistant chat modality

.agents/api-endpoints-and-auth.md

@@ -330,3 +330,16 @@ When adding a new endpoint:
 - [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
 - [ ] Tests cover both authenticated and unauthenticated access
 - [ ] Swagger regenerated (`make swagger`) if you changed any `@Router`/`@Tags`/`@Param` annotation
+
+## Companion: MCP admin tool surface
+
+**Required for admin endpoints.** Every new admin endpoint MUST be considered for the MCP admin tool surface — the REST API and the MCP tool catalog can drift silently otherwise, and both the LocalAI Assistant chat modality and the standalone `local-ai mcp-server` rely on `pkg/mcp/localaitools/` to mirror REST.
+
+Two outcomes are acceptable; one is not:
+
+- **Tool added.** The new endpoint is something an admin would manage conversationally (install, list, edit, toggle, upgrade). Follow the full checklist in [.agents/localai-assistant-mcp.md](localai-assistant-mcp.md): add a `LocalAIClient` interface method, implement it in both `inproc` and `httpapi`, register the tool with a `Tool*` constant, update the skill prompts, **and add the route to `toolToHTTPRoute` in `pkg/mcp/localaitools/coverage_test.go`**.
+- **Tool deliberately skipped.** The endpoint is internal/diagnostic and adding a chat path would be misleading. Document the decision in the PR description; no code action.
+- **Forgot.** This breaks the contract. The `TestToolHTTPRouteMappingComplete` test in `pkg/mcp/localaitools` is a partial guard (it checks every `Tool*` has a route mapping), but it does NOT detect new REST endpoints without a tool — that's still a process check on the PR author.
+
+**Add to the bottom of the checklist below**:
+- [ ] If admin: decided whether MCP coverage is needed; if yes, tool registered + map updated; if no, skip-reason in PR description.

.agents/localai-assistant-mcp.md

@@ -0,0 +1,97 @@
+# LocalAI Assistant — admin MCP server
+
+This document is the contract for **anyone** (human or AI agent) touching LocalAI's admin REST surface, the in-process MCP server that wraps it, or the embedded skill prompts that teach the assistant how to use it. Read this before adding/removing/renaming admin endpoints, MCP tools, or skill recipes.
+
+## What this feature is
+
+`pkg/mcp/localaitools/` is a public Go package that exposes LocalAI's admin/management surface as an MCP server. It is used in two ways:
+
+1. **In-process**: when an admin opens a chat with `metadata.localai_assistant=true`, the chat handler injects the in-memory MCP server (paired `net.Pipe()` transport, no HTTP loopback) so the LLM can install models, manage backends and edit configs by chatting.
+2. **Standalone**: the `local-ai mcp-server --target=…` subcommand serves the same MCP server over stdio, talking HTTP to a remote LocalAI instance.
+
+The two modes share **all** tool definitions and skill prompts. They differ only in their `LocalAIClient` implementation (`inproc/` calls services directly; `httpapi/` calls REST).
+
+## The three things you must keep in sync
+
+When you change LocalAI's admin surface, three layers must stay aligned:
+
+1. **REST endpoint** in `core/http/endpoints/localai/*.go`.
+2. **MCP tool registration** in `pkg/mcp/localaitools/tools_*.go`, plus a method on `LocalAIClient` (in `client.go`) and implementations in both `inproc/client.go` **and** `httpapi/client.go`.
+3. **Skill prompt** under `pkg/mcp/localaitools/prompts/skills/*.md` — the markdown that teaches the LLM how to use the new tool. If the new tool fits an existing recipe, update that recipe; otherwise add a new file.
+
+If you ship a REST endpoint without (2) and (3), conversational admins won't see the feature.
+
+## Checklist for adding a new admin endpoint
+
+- [ ] REST endpoint exists in `core/http/endpoints/localai/*.go` and is gated by `auth.RequireAdmin()` in `core/http/routes/localai.go`.
+- [ ] `LocalAIClient` interface in `pkg/mcp/localaitools/client.go` has a method covering the new operation.
+- [ ] DTOs added/updated in `pkg/mcp/localaitools/dto.go` (JSON-tagged; never expose raw service types).
+- [ ] `inproc/client.go` implements the new method by calling the service directly (not via HTTP loopback).
+- [ ] `httpapi/client.go` implements the new method by calling the REST endpoint.
+- [ ] Tool registration added in the appropriate `pkg/mcp/localaitools/tools_*.go`. Mutating tools must reference safety rule 1 in the description.
+- [ ] If the tool is mutating, ensure `Options{DisableMutating: true}` skips it (mirror the pattern in `tools_models.go`).
+- [ ] Skill prompt added or updated under `pkg/mcp/localaitools/prompts/skills/`. The prompt must instruct the LLM when to call the tool, what to ask the user first, and what to do on error.
+- [ ] Tests:
+   - `pkg/mcp/localaitools/server_test.go` adds the tool name to `expectedFullCatalog` and `expectedReadOnlyCatalog` (if read-only).
+   - Tool dispatch is added to `TestEachToolDispatchesToClient`.
+   - `pkg/mcp/localaitools/httpapi/client_test.go` covers the new HTTP path.
+
+## Adding a new skill recipe (no new tool)
+
+Sometimes you want to teach the LLM a new pattern that uses existing tools. Drop a markdown file under `pkg/mcp/localaitools/prompts/skills/<verb>_<noun>.md`. The file is automatically embedded by `//go:embed` and assembled into the system prompt in lexicographic order. No Go changes needed.
+
+Conventions:
+- Filename: `<verb>_<noun>.md` (e.g. `install_chat_model.md`, `upgrade_backend.md`).
+- First line: `# Skill: <Title Case description>`.
+- Number the steps. Reference exact tool names in backticks.
+- If the skill mutates state, remind the LLM to confirm with the user.
+
+## Code conventions
+
+These rules guard against the magic-literal drift that surfaced in the first audit. Do not re-introduce bare strings.
+
+- **Tool names** always come from the `Tool*` constants in `pkg/mcp/localaitools/tools.go`. Tool registrations, the test catalog (`server_test.go`'s `expectedFullCatalog` / `expectedReadOnlyCatalog`), and dispatch tables reference the constants. The embedded skill prompts under `prompts/` keep bare strings — that's the one allowed exception, and `TestPromptsContainSafetyAnchors` enforces alignment.
+- **Toggle/pin actions** use the `modeladmin.Action` type (`pkg/mcp/localaitools` and `core/services/modeladmin`). Use `ActionEnable`/`ActionDisable`/`ActionPin`/`ActionUnpin`; never bare `"enable"`/`"pin"` strings.
+- **Capability tags** for `list_installed_models` use the `localaitools.Capability` type (`capability.go`). The `LocalAIClient.ListInstalledModels` interface takes a typed `Capability`, and the `inproc` switch only accepts canonical values (`"embed"`/`"embedding"` are not aliases — only `CapabilityEmbeddings`).
+- **HTTP error checks** in `httpapi.Client` use `errors.Is(err, ErrHTTPNotFound)`, not substring matches on `err.Error()`. The typed `*HTTPError` carries `StatusCode` and `Body`; add new sentinel errors as needed rather than re-introducing string matching.
+- **Channel sends** to `GalleryService.ModelGalleryChannel` / `BackendGalleryChannel` from inproc clients MUST select on `ctx.Done()` so a cancelled chat completion releases the goroutine. See `inproc.sendModelOp` / `sendBackendOp`.
+- **Disk writes** of model config YAML go through `modeladmin.writeFileAtomic` (temp file + `os.Rename`). `os.WriteFile` truncates on crash and corrupts the model.
+- **MCP server lifecycle**: every initialised holder MUST register `Close()` with `signals.RegisterGracefulTerminationHandler`. The standalone `mcp-server` CLI uses `signal.NotifyContext` to honour SIGINT/SIGTERM.
+
+## File map (where to look)
+
+```
+pkg/mcp/localaitools/
+  client.go              # LocalAIClient interface + DTO registry
+  dto.go                 # JSON-tagged DTOs shared by both client impls
+  server.go              # NewServer(client, opts) — registers tools
+  tools.go               # Tool* name constants (single source of truth)
+  capability.go          # Capability type + constants
+  tools_models.go        # gallery_search, install_model, import_model_uri, ...
+  tools_backends.go
+  tools_config.go
+  tools_system.go
+  tools_state.go
+  prompts.go             # //go:embed loader + SystemPrompt(opts)
+  prompts/00_role.md
+  prompts/10_safety.md   # SAFETY RULES — change with care
+  prompts/20_tools.md    # curated tool catalog with one-liners
+  prompts/skills/*.md
+  inproc/client.go       # in-process LocalAIClient (services-direct)
+  httpapi/client.go      # REST LocalAIClient (for standalone CLI / remote)
+core/http/endpoints/mcp/
+  localai_assistant.go   # process-wide holder + LocalToolExecutor
+core/cli/mcp_server.go   # local-ai mcp-server subcommand
+```
+
+## Why two clients
+
+The in-process MCP server runs inside the same LocalAI binary that serves chat. Going over HTTP loopback would (a) require minting a synthetic admin API key for the server to authenticate against itself, (b) double-marshal every tool dispatch, and (c) lose access to in-process channels (e.g. `GalleryService.ModelGalleryChannel` for streaming install progress). So in-process uses `inproc.Client`. The standalone stdio CLI talks to a *remote* LocalAI; HTTP is the only option, so it uses `httpapi.Client`. Both implement the same `LocalAIClient` interface, and the parity test in `pkg/mcp/localaitools/parity_test.go` (when present) keeps their output equivalent.
+
+## Why prompt-enforced confirmation, not code gates
+
+The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.md` rule 1) that requires the LLM to summarise the action and wait for explicit user confirmation before calling it. There is no `plan_*`/`apply_*` two-step in code. If you add a mutating tool, do **not** add per-tool confirmation logic in Go — instead, list the new tool name in `prompts/10_safety.md` so the LLM knows it falls under the confirmation rule.
+
+## Distributed mode
+
+The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period.

AGENTS.md

@@ -28,6 +28,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 | [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
 | [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |
 | [.agents/adding-gallery-models.md](.agents/adding-gallery-models.md) | Adding GGUF models from HuggingFace to the model gallery |
+| [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) | LocalAI Assistant chat modality — adding admin tools to the in-process MCP server, editing skill prompts, keeping REST + MCP + skills in sync |
 
 ## Quick Reference
 
@@ -36,5 +37,6 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 - **Comments**: Explain *why*, not *what*
 - **Docs**: Update `docs/content/` when adding features or changing config
 - **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
+- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
 - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
 - **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI

core/application/application.go

@@ -17,7 +17,10 @@ import (
 	"github.com/mudler/LocalAI/core/services/voicerecognition"
 	"github.com/mudler/LocalAI/core/templates"
 	pkggrpc "github.com/mudler/LocalAI/pkg/grpc"
+	localaitools "github.com/mudler/LocalAI/pkg/mcp/localaitools"
+	localaiInproc "github.com/mudler/LocalAI/pkg/mcp/localaitools/inproc"
 	"github.com/mudler/LocalAI/pkg/model"
+	"github.com/mudler/LocalAI/pkg/signals"
 	"github.com/mudler/xlog"
 	"gorm.io/gorm"
 )
@@ -60,6 +63,10 @@ type Application struct {
 
 	// Upgrade checker (background service for detecting backend upgrades)
 	upgradeChecker *UpgradeChecker
+
+	// LocalAI Assistant in-process MCP server. nil when DisableLocalAIAssistant
+	// is set; otherwise initialised in start() after galleryService.
+	localAIAssistant *mcpTools.LocalAIAssistantHolder
 }
 
 func newApplication(appConfig *config.ApplicationConfig) *Application {
@@ -137,6 +144,13 @@ func (a *Application) UpgradeChecker() *UpgradeChecker {
 	return a.upgradeChecker
 }
 
+// LocalAIAssistant returns the in-process MCP holder used by the chat handler
+// when an admin opts into the assistant modality. Returns nil when the feature
+// is disabled at startup.
+func (a *Application) LocalAIAssistant() *mcpTools.LocalAIAssistantHolder {
+	return a.localAIAssistant
+}
+
 // distributedDB returns the PostgreSQL database for distributed coordination,
 // or nil in standalone mode.
 func (a *Application) distributedDB() *gorm.DB {
@@ -230,6 +244,32 @@ func (a *Application) start() error {
 
 	a.galleryService = galleryService
 
+	// LocalAI Assistant: in-process MCP server exposing admin tools. Initialised
+	// once at startup and reused across chat sessions that opt in via metadata.
+	if !a.applicationConfig.DisableLocalAIAssistant {
+		holder := mcpTools.NewLocalAIAssistantHolder()
+		assistantClient := localaiInproc.New(
+			a.applicationConfig,
+			a.applicationConfig.SystemState,
+			a.backendLoader,
+			a.modelLoader,
+			a.galleryService,
+		)
+		if err := holder.Initialize(a.applicationConfig.Context, assistantClient, localaitools.Options{}); err != nil {
+			// Why log+continue instead of fail: the assistant is an optional
+			// feature; a failure here must not take down the whole server.
+			xlog.Warn("LocalAI Assistant initialisation failed; feature unavailable", "error", err)
+		} else {
+			a.localAIAssistant = holder
+			// Tear the in-memory transport pair down on SIGINT/SIGTERM so the
+			// goroutine ends cleanly. Mirrors how core/http/endpoints/mcp/tools.go
+			// closes its per-model MCP sessions on graceful termination.
+			signals.RegisterGracefulTerminationHandler(func() {
+				_ = holder.Close()
+			})
+		}
+	}
+
 	// Initialize agent job service (Start() is deferred to after distributed wiring)
 	agentJobService := agentpool.NewAgentJobService(
 		a.ApplicationConfig(),

core/cli/cli.go

@@ -20,6 +20,7 @@ var CLI struct {
 	AgentWorker     AgentWorkerCMD     `cmd:"" name:"agent-worker" help:"Start an agent worker for distributed mode (executes agent chats via NATS)"`
 	Util            UtilCMD            `cmd:"" help:"Utility commands"`
 	Agent           AgentCMD           `cmd:"" help:"Run agents standalone without the full LocalAI server"`
+	MCPServer       MCPServerCMD       `cmd:"" name:"mcp-server" help:"Run the LocalAI admin tool surface as a stdio MCP server (controls a remote LocalAI instance over HTTP)"`
 	Explorer        ExplorerCMD        `cmd:"" help:"Run p2p explorer"`
 	Completion      CompletionCMD      `cmd:"" help:"Generate shell completion scripts for bash, zsh, or fish"`
 }

core/cli/mcp_server.go

@@ -0,0 +1,47 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"os/signal"
+	"syscall"
+
+	"github.com/modelcontextprotocol/go-sdk/mcp"
+
+	cliContext "github.com/mudler/LocalAI/core/cli/context"
+	localaitools "github.com/mudler/LocalAI/pkg/mcp/localaitools"
+	"github.com/mudler/LocalAI/pkg/mcp/localaitools/httpapi"
+)
+
+// MCPServerCMD runs the LocalAI admin tool surface as a stdio MCP server,
+// targeting a remote LocalAI instance over its HTTP API. The same Go package
+// that powers the in-process LocalAI Assistant chat modality is used here —
+// only the LocalAIClient implementation differs (httpapi instead of inproc).
+type MCPServerCMD struct {
+	Target   string `env:"LOCALAI_MCP_TARGET" default:"http://localhost:8080" help:"LocalAI base URL"`
+	APIKey   string `env:"LOCALAI_API_KEY" help:"Bearer API key for the target LocalAI"`
+	ReadOnly bool   `help:"Skip registration of mutating tools (install/delete/edit/upgrade/etc.) so the assistant can browse without changing remote state"`
+}
+
+func (m *MCPServerCMD) Run(_ *cliContext.Context) error {
+	if m.Target == "" {
+		return fmt.Errorf("--target / LOCALAI_MCP_TARGET is required")
+	}
+
+	client := httpapi.New(m.Target, m.APIKey)
+	srv := localaitools.NewServer(client, localaitools.Options{
+		DisableMutating: m.ReadOnly,
+	})
+
+	// Stdio: the host (e.g. Claude Desktop, Cursor, mcphost) talks JSON-RPC
+	// over our stdin/stdout. There's nothing else this process should print —
+	// every other goroutine logging to stderr is fine, but stdout is sacred.
+	//
+	// Honour SIGINT/SIGTERM so a Ctrl-C from the host or `kill -TERM` from
+	// process supervision gives srv.Run a chance to drain in-flight calls
+	// before exiting.
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
+	defer stop()
+	return srv.Run(ctx, &mcp.StdioTransport{})
+}

core/cli/run.go

@@ -101,6 +101,9 @@ type RunCMD struct {
 	AgentJobRetentionDays              int      `env:"LOCALAI_AGENT_JOB_RETENTION_DAYS,AGENT_JOB_RETENTION_DAYS" default:"30" help:"Number of days to keep agent job history (default: 30)" group:"api"`
 	OpenResponsesStoreTTL              string   `env:"LOCALAI_OPEN_RESPONSES_STORE_TTL,OPEN_RESPONSES_STORE_TTL" default:"0" help:"TTL for Open Responses store (e.g., 1h, 30m, 0 = no expiration)" group:"api"`
 
+	// LocalAI Assistant chat modality (in-process admin MCP server)
+	DisableLocalAIAssistant bool `env:"LOCALAI_DISABLE_ASSISTANT" default:"false" help:"Disable the LocalAI Assistant chat modality (in-process admin MCP server)" group:"assistant"`
+
 	// Agent Pool (LocalAGI)
 	DisableAgents                  bool   `env:"LOCALAI_DISABLE_AGENTS" default:"false" help:"Disable the agent pool feature" group:"agents"`
 	AgentPoolAPIURL                string `env:"LOCALAI_AGENT_POOL_API_URL" help:"Default API URL for agents (defaults to self-referencing LocalAI)" group:"agents"`
@@ -323,6 +326,9 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	if r.AgentPoolDefaultModel != "" {
 		opts = append(opts, config.WithAgentPoolDefaultModel(r.AgentPoolDefaultModel))
 	}
+	if r.DisableLocalAIAssistant {
+		opts = append(opts, config.WithDisableLocalAIAssistant(true))
+	}
 	if r.AgentPoolMultimodalModel != "" {
 		opts = append(opts, config.WithAgentPoolMultimodalModel(r.AgentPoolMultimodalModel))
 	}