diff --git a/.github/workflows/syncNext.yml b/.github/workflows/syncNext.yml new file mode 100644 index 0000000..d4a7ea0 --- /dev/null +++ b/.github/workflows/syncNext.yml @@ -0,0 +1,16 @@ +name: Sync next + +# Carries every change that lands on the base branch onto the paired `next` +# iteration branch, so `next` never drifts behind what has already shipped. +# `next` is created on the first run if the repo does not have one yet. +on: + push: + branches: ['master'] + workflow_dispatch: + +jobs: + sync: + uses: Start9Labs/start-technologies/.github/workflows/syncNext.yml@master + permissions: + contents: write + pull-requests: write diff --git a/AGENTS.md b/AGENTS.md index b1cc431..46b7721 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,12 +6,11 @@ Develop it inside a StartOS packaging workspace created by `start-cli s9pk init- which provides the packaging guide and agent context one level up. If you're reading this in a bare clone with no workspace, the full guide is at . -Work this package's `TODO.md` from top to bottom. Keep `README.md` (architecture, for developers and LLMs) and `instructions.md` (end-user docs) in sync with your changes. +Work this package's `TODO.md` from top to bottom. Keep `README.md` (technical reference for an AI support or administering agent) and `instructions.md` (end-user docs) in sync with your changes. ## This repo -- **Package id is `llama-cpp`.** Ships four image variants — `generic` (CPU), `nvidia` (CUDA), `rocm` (AMD), and `vulkan` — selected at build time via `VARIANT=…`; the `Makefile` overrides `ARCHES`/`TARGETS` before the `s9pk.mk` include and fans out to per-variant targets. `rocm` is x86-only. Exposes one `ui` interface (`api`, port 8080): the OpenAI-compatible API plus built-in chat UI, gated by OS reverse-proxy basic auth (username `admin`, password set via the **Set UI Password** critical action). No dependencies. - -## Inspecting a running install - -To run a command inside the service's container (read its generated config, grep app logs), use `start-cli package attach llama-cpp -n llama-cpp-sub -- `. Select the subcontainer by **name** with `-n` (the name passed to `SubContainer.of` in `main.ts` — here `llama-cpp-sub`) or by image with `-i`. Note: `-s/--subcontainer` matches the internal **Guid**, not the name, so passing a name to `-s` fails with "no matching subcontainers". +- **One repo, four builds.** `VARIANT` (see the `Makefile` targets) selects the image, architectures, and `hardwareRequirements` for `generic`, `nvidia`, `rocm`, and `vulkan`. Bump `upstreamBuild` once in `startos/manifest/index.ts` and every variant follows. +- **The AMD matcher is a positive allowlist of discrete product names, not an iGPU exclusion.** StartOS's regex engine has no lookahead, and ROCm is unreliable on integrated Radeon — so a broad `amdgpu` match would route Ryzen APUs onto a build that does not work for them. +- **The one-hour grace period on the health check is the model download.** Don't shorten it to something that looks more like a health check. +- **Preset sizing reads VRAM where it can and system memory otherwise** (`startos/hardware.ts`), and the result is cached per process. `minMemoryGB` in `actions/presets.ts` is weights plus roughly 25% for the KV cache — keep that convention when adding a preset. diff --git a/README.md b/README.md index 08a4925..0e1628c 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,15 @@ # llama.cpp on StartOS -> **Upstream repo:** -> -> **Upstream `llama-server` docs:** -> -> Everything not listed here behaves the same as upstream `llama-server`. If a flag, endpoint, or behavior is not mentioned in this document, upstream documentation is accurate and fully applicable. +> Everything not listed in this document should behave the same as upstream +> llama.cpp. If a feature, setting, or behavior is not mentioned here, the +> upstream documentation is accurate and fully applicable — see the +> Documentation section of `instructions.md` for links. -[llama.cpp](https://github.com/ggml-org/llama.cpp) is a high-performance C/C++ runtime for large language models in GGUF format. This package wraps its built-in HTTP server (`llama-server`), which exposes an OpenAI-compatible API and a small in-browser chat UI on the same port. +[llama.cpp](https://github.com/ggml-org/llama.cpp) runs large language models locally and serves them over an OpenAI-compatible API. This package ships one build per accelerator, sizes its model presets to the hardware it finds, and puts authentication in front of a server that has none of its own. + +- **Upstream repo:** +- **Wrapper repo:** --- @@ -18,186 +20,178 @@ - [Image and Container Runtime](#image-and-container-runtime) - [Volume and Data Layout](#volume-and-data-layout) -- [Installation and First-Run Flow](#installation-and-first-run-flow) -- [Configuration Management](#configuration-management) -- [Network Access and Interfaces](#network-access-and-interfaces) -- [Actions (StartOS UI)](#actions-startos-ui) +- [File Models](#file-models) - [Dependencies](#dependencies) -- [Backups and Restore](#backups-and-restore) +- [Network Access and Interfaces](#network-access-and-interfaces) +- [Installation and First-Run Flow](#installation-and-first-run-flow) +- [Actions](#actions) +- [Tasks](#tasks) - [Health Checks](#health-checks) +- [Backups and Restore](#backups-and-restore) - [Limitations and Differences](#limitations-and-differences) -- [What Is Unchanged from Upstream](#what-is-unchanged-from-upstream) -- [Contributing](#contributing) - [Quick Reference for AI Consumers](#quick-reference-for-ai-consumers) --- ## Image and Container Runtime -The package ships four variants, selected at build time via the `VARIANT` env var (driven by the `Makefile`): +The upstream image is used unmodified, but **the package is built four times** — one variant per accelerator — and StartOS installs whichever matches your hardware. -| Variant | Image | Arches | Accelerator | Offered to GPU driver | -| --------- | ------------------------------------------ | --------------- | ------------- | ---------------------- | -| `generic` | `ghcr.io/ggml-org/llama.cpp:server` | x86_64, aarch64 | CPU only | — (universal fallback) | -| `nvidia` | `ghcr.io/ggml-org/llama.cpp:server-cuda` | x86_64, aarch64 | CUDA (NVIDIA) | `nvidia` | -| `rocm` | `ghcr.io/ggml-org/llama.cpp:server-rocm` | x86_64 | ROCm (AMD) | `amdgpu` | -| `vulkan` | `ghcr.io/ggml-org/llama.cpp:server-vulkan` | x86_64, aarch64 | Vulkan | `i915` (Intel) | +| Variant | Upstream image | Architectures | Selected when | +| --------- | -------------- | --------------- | -------------------------------------------- | +| `generic` | CPU server | x86_64, aarch64 | Nothing more specific matches — the fallback | +| `nvidia` | CUDA server | x86_64, aarch64 | An NVIDIA GPU on the `nvidia` driver | +| `rocm` | ROCm server | x86_64 | A **discrete** AMD GPU on `amdgpu` | +| `vulkan` | Vulkan server | x86_64, aarch64 | An Intel GPU on the `i915` driver | -All four variants publish under a single package version. Each declares a distinct `hardwareRequirements.device`, so StartOS serves each host the most specific variant its detected hardware satisfies — `nvidia`/`rocm`/`vulkan` for matching GPUs, and `generic` as the universal CPU fallback for everything else. Note that `vulkan` matches only Intel GPUs on the `i915` driver; newer Intel GPUs on the `xe` driver (and non-Intel Vulkan-only setups) fall back to `generic`. `rocm` matches the `amdgpu` driver but is narrowed by GPU product name to **discrete** AMD GPUs (Navi / Radeon RX / Instinct); integrated Radeon graphics (e.g. the Radeon 680M in Ryzen APUs), where ROCm is unreliable, fall back to `generic`. `nvidia` matches the `nvidia` driver, which is present only when StartOS is installed from a `-nvidia` platform flavor (`x86_64-nvidia` / `aarch64-nvidia`, bundling the NVIDIA driver and container toolkit); on the standard or `-nonfree` flavors an NVIDIA card isn't detected and falls back to `generic` (CPU), even with the card physically present. +Selection is StartOS's, from the hardware requirements each variant declares; the most specific compatible one wins, and `generic` is the only variant with no requirement. -| Property | Value | -| ------------ | ------------------- | -| Entrypoint | `/app/llama-server` | -| Working dir | `/app` | -| Default port | 8080 | +The AMD requirement matches discrete cards by product name rather than excluding integrated ones, because ROCm is unreliable on integrated Radeon graphics and the matcher has no way to express an exclusion. ---- +| Subcontainer | Purpose | +| -------------------------------------------- | ------------------------------------------------ | +| `llama-cpp-sub` | The `primary` daemon, and the one to `attach` to | +| `detect-nvidia`, `detect-rocm`, `detect-mem` | Temporary; used to size the model presets | +| `delete-cache` | Temporary; the Delete Model Cache action | ## Volume and Data Layout -| Volume | Mount Point | Purpose | -| ------ | ----------- | ---------------------------------------------------- | -| `main` | `/data` | `store.json` (serve args) and `models/` (GGUF cache) | - -The container runs with `LLAMA_CACHE=/data/models` and `HF_HOME=/data/huggingface`, so all `-hf ` downloads land on the persistent volume. +One volume, and most of it is downloaded models. ---- +| Volume | Mount Point | Purpose | +| ------ | ----------- | ------------------------------------------------------------------------------- | +| `main` | `/data` | `store.json`, the GGUF model cache under `models/`, and HuggingFace's own cache | -## Installation and First-Run Flow +Models are the bulk of it — a single quantized model runs from roughly one to forty gigabytes depending on size. -| Step | StartOS | -| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Install | Marketplace install or sideload `.s9pk` | -| First-run tasks | Two `critical` tasks: **Set UI Password** (created whenever no password is set) and **Set Model** (created whenever no model is selected). Both are created on install and re-surface if the underlying value is later cleared. | -| Start service | After **Set Model** has been run; until then the daemon idles | -| Pull the model | Automatic on first start (cached on the `main` volume) | +## File Models -Until **Set Model** has been run, the daemon stays in an idle (`sleep infinity`) state and the API port is closed — the health check reports "No model selected." Once a model is selected, llama-server is restarted with the chosen serve arguments. +One model. Two of its keys decide whether the service can run at all; the third only exists so a form can remember what you last told it. ---- +| File | Format | Modelled | Written by | +| ------------ | ------ | ----------------------- | ----------------------------------------- | +| `store.json` | JSON | Yes — `FileHelper.json` | The Set Model and Set UI Password actions | -## Configuration Management - -Serve configuration is stored at `/data/store.json` and managed via the **Set Model** action: - -```json -{ - "serveArgs": [ - "-hf", - "unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M", - "-c", - "8192", - "-ngl", - "999" - ] -} -``` +| Key | Notes | +| ---------------- | ----------------------------------------------------------------------------------------------- | +| `serveArgs` | The full argument list handed to `llama-server`, composed by the Set Model action | +| `uiPassword` | The password for the proxy's basic auth; the username is always `admin` | +| `modelSelection` | What the Set Model form last submitted, so it can be prefilled next time — read by nothing else | -`serveArgs` is the exact list of arguments appended after `/app/llama-server`. The daemon adds `--host 0.0.0.0` and `--port 8080` at runtime. +Nothing else writes the file, and neither `serveArgs` nor `uiPassword` is defaulted — both are absent until you run their action, and each absence raises a task. -`llama-server` itself runs **keyless** — no `--api-key`. Access is instead gated by HTTP **basic auth enforced at the StartOS reverse proxy** (`addSsl.auth`): the OS validates credentials before any request reaches the container. The username is hard-coded to `admin`; the password is generated by the **Set UI Password** action and stored as `uiPassword` in `store.json`. `setupInterfaces` reads it reactively, so rotating it via the action takes effect without a manual restart. **Set UI Password** is a `critical` task, which blocks the service from starting until a password is set — so the service never runs (and the gate never serves) without one. +`modelSelection` holds the chosen `selection` plus, when Custom was chosen, a `custom` object carrying that variant's fields; picking a preset clears `custom`. The daemon never reads it, so it cannot disagree with `serveArgs` about what is actually running — at worst it prefills a form with a stale answer. -Dependent StartOS services reach llama.cpp over the internal service mesh (`http://llama-cpp.startos:8080`), which is not behind the proxy gate, so they connect keyless. +**No configuration file reaches the application.** Two environment variables are set, both redirecting caches onto the volume so downloaded weights survive a container rebuild: -**Curated presets:** the Set Model action surfaces a hardware-tier-aware list of GGUF presets and disables ones too large for the detected memory: +| Variable | Value | +| ------------- | ------------------- | +| `LLAMA_CACHE` | `/data/models` | +| `HF_HOME` | `/data/huggingface` | -| Preset | Repo (`-hf`) | Min memory | -| ------------------------------ | --------------------------------------------------------- | ---------- | -| Llama 3.2 1B Instruct | `unsloth/Llama-3.2-1B-Instruct-GGUF:Q4_K_M` | 2 GB | -| Llama 3.2 3B Instruct | `unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M` | 4 GB | -| Qwen2.5 7B Instruct | `unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M` | 6 GB | -| Llama 3.1 8B Instruct | `unsloth/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M` | 8 GB | -| Qwen2.5 14B Instruct | `unsloth/Qwen2.5-14B-Instruct-GGUF:Q4_K_M` | 12 GB | -| Mistral Small 3.2 24B Instruct | `unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M` | 18 GB | -| Qwen3 30B-A3B Instruct | `unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M` | 22 GB | -| Qwen2.5 32B Instruct | `unsloth/Qwen2.5-32B-Instruct-GGUF:Q4_K_M` | 24 GB | -| Llama 3.3 70B Instruct | `unsloth/Llama-3.3-70B-Instruct-GGUF:Q4_K_M` | 48 GB | +Everything else about how the model is served is in `serveArgs`, and the package always appends the host and port itself so the server binds where the interface expects it. -The **Custom** variant accepts a HuggingFace repo, optional filename, context size, GPU layer count, and extra `llama-server` flags. For settings that can't be expressed cleanly via the form (quoted JSON, multi-word strings), edit `store.json` directly. +## Dependencies ---- +None. ## Network Access and Interfaces -| Interface | Port | Protocol | Type | Purpose | -| ---------------- | ---- | -------- | ---- | ---------------------------------------- | -| llama.cpp Server | 8080 | HTTP | `ui` | Built-in chat UI + OpenAI-compatible API | +One interface, serving both the OpenAI-compatible API and llama.cpp's built-in chat UI. -The chat UI and the API share a single port, gated by basic auth (`admin` + the generated password) at the proxy. Access methods (StartOS 0.4.x): LAN IP, `.local`, Tor `.onion`, and custom domains if configured. Browsers get a native login prompt. OpenAI-compatible clients hitting the public interface use base URL `/v1` and must supply the basic-auth credentials (e.g. `curl -u admin:`); other StartOS services use the keyless internal `http://llama-cpp.startos:8080/v1`. +| Interface | Id | Type | Port | Description | +| ---------------- | ----- | ---- | ---- | ------------------------------------ | +| llama.cpp Server | `api` | ui | 8080 | The API and the built-in chat client | -Selected upstream endpoints: +**Authentication is added by StartOS, not by llama.cpp.** The server itself runs keyless; the binding declares HTTP basic auth at the edge, with the username `admin` and the password from `store.json`. Until a password is set the binding is configured with an empty one — which never serves anything, because the service is blocked from starting by a `critical` task at the same time. -| Endpoint | Method | Purpose | -| ---------------------- | ------ | ------------------------------------------------ | -| `/v1/chat/completions` | POST | OpenAI-compatible chat | -| `/v1/completions` | POST | OpenAI-compatible text completion | -| `/v1/embeddings` | POST | Embeddings (when the loaded model supports them) | -| `/health` | GET | Health probe | -| `/props` | GET | Loaded model info | +An OpenAI-compatible client therefore needs those basic-auth credentials as well as whatever it would normally send. -The full surface area is documented in upstream `tools/server/README.md`. +## Installation and First-Run Flow ---- +Install writes nothing and the service starts idle: **two `critical` tasks** stand between a fresh install and a working one, and both must be cleared. -## Actions (StartOS UI) +1. **Set UI Password** — until then there is no credential in front of the API. +2. **Set Model** — until then there is nothing to serve. The daemon runs but does no work, and its health check says so by name. -| Action | Purpose | -| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Set Model** | Choose a curated preset (with hardware-tier-aware availability) or a custom HuggingFace GGUF. Writes `serveArgs` to `store.json` and restarts the daemon. | -| **Set UI Password** | Generate (or rotate) the web UI login password. Username is always `admin`. Returns the new credentials; the proxy gate picks them up automatically. | -| **Delete Model Cache** | Remove a specific filename from `/data/models` to reclaim disk space. | +Both are raised by a condition rather than at install time, so they reappear if either value is later cleared. ---- +The first start after choosing a model **downloads it**, which is why the health check allows an hour before reporting failure. A large model on a slow connection can take most of that. -## Dependencies +## Actions -None. +Three actions, all user-facing. ---- +### Set Model -## Backups and Restore +Chooses what the server runs — either a curated preset or a model of your own. -**Included in backup:** +- **What it changes:** `serveArgs` and `modelSelection` in `store.json`, replacing each entirely. +- **Cost:** seconds to write, then a restart — and, if the model is not already cached, a download that can take a long time. +- **Repeat safety:** safe to re-run. Switching back to a previously used model is fast, because the old one is still cached. +- **The form reopens on your current selection**, read back from `modelSelection`, so changing one setting does not mean re-entering the rest. With nothing chosen yet it falls back to the hardware-filtered default. +- **Presets are filtered to your hardware.** The form reads the accelerator's memory — VRAM on NVIDIA and ROCm, system memory otherwise — and disables any preset that would not fit, defaulting to the smallest that does. The estimate is the quantized weights plus roughly a quarter for the context cache. +- **Custom** takes a HuggingFace GGUF repo, optionally a specific file, a context size, a GPU-layer count, and extra server flags. Those extra flags are split on whitespace, so a quoted value with spaces will not survive. -- `main` volume — `store.json` _and_ all cached GGUF weights under `models/`. +### Set UI Password -**Restore behavior:** +Generates the password for the API and chat UI. -- Serve args and any locally cached models are restored verbatim. No reconfiguration needed. +- **What it changes:** `uiPassword` in `store.json`, and through it the binding's basic-auth credential. +- **Cost:** seconds, then a restart. +- **Repeat safety:** safe to re-run, but it **replaces** the existing password — every saved client login has to be updated. +- **Outputs:** the username `admin` and the new password. -Backups can be very large depending on how many models you've cached — a single 70B Q4 file is ~40 GB. +### Delete Model Cache ---- +Removes one downloaded model file to reclaim disk. + +- **What it changes:** deletes the named file from the model cache. Path separators are stripped from the input, so it cannot reach outside that directory. +- **Repeat safety:** idempotent; deleting a file that is not there succeeds. +- **Not reversible**, but not destructive either — the model is re-downloaded if selected again. + +## Tasks + +Two tasks, both raised by a condition rather than at install, and both blocking. + +| Task | Severity | Raised when | Cleared when | +| --------------- | ---------- | ----------------------------- | --------------- | +| Set UI Password | `critical` | Whenever no password is set | The action runs | +| Set Model | `critical` | Whenever no model is selected | The action runs | + +Because they are conditional, clearing either value later raises its task again rather than leaving the service running unauthenticated or idle. ## Health Checks -| Check | Method | Grace period | Messages | -| ------------- | ---------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -| llama.cpp API | Port listening on 8080 | 60 minutes (cold-cache model downloads) | "The llama.cpp API is ready" / "The llama.cpp API is not ready" or "No model selected. Run the \"Set Model\" action." | +One check, on the daemon. ---- +| Check | Method | Grace Period | +| ------------------------- | ---------------------- | ------------ | +| `primary` "llama.cpp API" | Port 8080 is listening | 1 hour | -## Limitations and Differences +**The hour-long grace is for the model download**, which happens on the first start after a selection and is bounded only by size and bandwidth. -1. **One model per process.** llama-server holds a single GGUF in memory. To switch models, run **Set Model** again — the service restarts with the new weights. -2. **Custom-action arg splitting.** The Custom variant's `Extra arguments` field is split on whitespace, so JSON values with quoted spaces will not survive — edit `store.json` directly for those. -3. **Hardware-tier detection is best-effort.** GPU memory is read from `nvidia-smi` / `rocm-smi`; on Vulkan and unsupported topologies, the preset filter falls back to total system RAM as a memory budget. -4. **Variants are independent installs.** Switching from e.g. `generic` to `nvidia` is an uninstall + reinstall, not an in-place change; cached models on the `main` volume can be restored from backup. +With no model selected the daemon idles rather than exiting, and the check's failure message names the action to run — so an unconfigured install reports what to do rather than looking broken. ---- +## Backups and Restore -## What Is Unchanged from Upstream +The `main` volume is copied wholesale — `sdk.Backups.ofVolumes('main')`. No dump step and nothing excluded. -- The full `llama-server` HTTP API and built-in chat UI. -- All `llama-server` CLI flags — anything not consumed by the package wrapper passes straight through (via the Custom variant's extra args). -- HuggingFace `-hf` model downloads and the `LLAMA_CACHE` layout. -- GGUF model support, embedding endpoints, OpenAI-compatible response shapes, and tool-call formats. +**That means the model cache is in the backup**, which is very likely the largest thing on the server. A backup of this service is dominated by weights that could be re-downloaded instead; [Delete Model Cache](#actions) is the way to trim what gets captured. ---- +- **Included:** `store.json` with the model selection and password, and every downloaded model. +- **Restore:** complete, and no tasks are raised — the selection and password come back, and the model is already cached, so the first start does not re-download. -## Contributing +## Limitations and Differences -Build and development workflow follow the StartOS packaging guide: . Keep `README.md`, `instructions.md`, and `AGENTS.md` in sync with any change to user-visible behavior or package structure. See [UPDATING.md](UPDATING.md) for the upstream-bump procedure. +1. **Two settings are required before the service does anything**, and each is enforced by a blocking task rather than defaulted. +2. **Authentication is the reverse proxy's, not llama.cpp's.** Every client, including API clients, must send basic-auth credentials. +3. **Which accelerator variant you get is decided by StartOS**, from the hardware present; it is not a setting. +4. **Integrated AMD graphics fall back to the generic CPU build.** ROCm is matched only for discrete cards. +5. **The Vulkan variant matches Intel GPUs only**, on the `i915` driver. +6. **Model presets are filtered by detected memory**, and the fit estimate is approximate — a preset that is enabled can still be tight at large context sizes. +7. **Extra server flags are split on whitespace**, so quoted arguments containing spaces do not survive. +8. **Models are included in backups.** Expect the backup to be as large as the cache. --- @@ -205,44 +199,33 @@ Build and development workflow follow the StartOS packaging guide: generic) - vulkan: - image: ghcr.io/ggml-org/llama.cpp:server-vulkan - arch: [x86_64, aarch64] - accel: vulkan - gpu_driver: i915 # Intel GPUs only +image: ghcr.io/ggml-org/llama.cpp # server, server-cuda, server-rocm, or server-vulkan per variant +architectures: + - x86_64 + - aarch64 # not for the rocm variant +subcontainers: + - llama-cpp-sub # the running daemon + - detect-nvidia # temporary; preset sizing + - detect-rocm # temporary; preset sizing + - detect-mem # temporary; preset sizing + - delete-cache # temporary; the Delete Model Cache action volumes: main: /data -ports: - api_and_ui: 8080 -env: - LLAMA_CACHE: /data/models - HF_HOME: /data/huggingface -dependencies: none -auth: # llama-server runs keyless; basic auth enforced at the OS reverse proxy - type: basic - username: admin # hard-coded - password: generated by set-ui-password, stored as store.json uiPassword - internal_mesh: keyless # http://llama-cpp.startos:8080 bypasses the proxy gate -startos_managed_args: ['--host 0.0.0.0', '--port 8080'] +file_models: + - store.json +startos_managed_env_vars: + - LLAMA_CACHE + - HF_HOME +dependencies: [] +interfaces: + api: { type: ui, port: 8080 } # basic auth enforced at the edge, username "admin" actions: - set-model - set-ui-password - delete-model-cache +tasks: + - { action: set-ui-password, severity: critical } + - { action: set-model, severity: critical } +health_checks: + - primary # displayed "llama.cpp API"; 1-hour grace covers the model download ``` diff --git a/UPDATING.md b/UPDATING.md index 1427d55..914a166 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -4,11 +4,11 @@ This package wraps [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) llama.cpp releases use monotonic build numbers of the form `bNNNN` (no semver), one per merged commit. Each release publishes four server image variants on `ghcr.io`: -| Variant | Image tag | Arches | -| --------- | ----------------------------------------- | ------------ | -| `generic` | `ghcr.io/ggml-org/llama.cpp:server-bNNNN` | amd64, arm64 | -| `nvidia` | `ghcr.io/ggml-org/llama.cpp:server-cuda-bNNNN` | amd64, arm64 | -| `rocm` | `ghcr.io/ggml-org/llama.cpp:server-rocm-bNNNN` | amd64 | +| Variant | Image tag | Arches | +| --------- | ------------------------------------------------ | ------------ | +| `generic` | `ghcr.io/ggml-org/llama.cpp:server-bNNNN` | amd64, arm64 | +| `nvidia` | `ghcr.io/ggml-org/llama.cpp:server-cuda-bNNNN` | amd64, arm64 | +| `rocm` | `ghcr.io/ggml-org/llama.cpp:server-rocm-bNNNN` | amd64 | | `vulkan` | `ghcr.io/ggml-org/llama.cpp:server-vulkan-bNNNN` | amd64, arm64 | All four variants are cut from the same upstream commit and bump together. diff --git a/assets/.gitkeep b/assets/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/assets/README.md b/assets/README.md deleted file mode 100644 index 4fbfc10..0000000 --- a/assets/README.md +++ /dev/null @@ -1 +0,0 @@ -Use the `/assets` directory to include additional files or scripts needed by your service. diff --git a/instructions.md b/instructions.md index 5852eae..351255d 100644 --- a/instructions.md +++ b/instructions.md @@ -44,7 +44,7 @@ curl -s -u admin: /v1/chat/completions \ ### Actions -- **Set Model** — switch to a different preset or custom GGUF. The service restarts with the new weights; uncached models download on this restart. +- **Set Model** — switch to a different preset or custom GGUF. The form opens with your current selection already filled in, so you can change one setting without re-entering the rest. The service restarts with the new weights; uncached models download on this restart. - **Set UI Password** — generate a new web UI password (username stays `admin`). Use it for first-time setup or to rotate the password later. - **Delete Model Cache** — remove a specific filename from the cache (e.g. `Qwen2.5-7B-Instruct-Q4_K_M.gguf`) to reclaim disk space. A deleted model will be re-downloaded if you select it again. diff --git a/startos/actions/setModel.ts b/startos/actions/setModel.ts index d2d94c1..70d9226 100644 --- a/startos/actions/setModel.ts +++ b/startos/actions/setModel.ts @@ -113,11 +113,29 @@ export const setModel = sdk.Action.withInput( inputSpec, - async ({ effects }) => ({}), + async ({ effects }) => { + const saved = await storeJson.read((s) => s?.modelSelection).const(effects) + if (!saved || !(saved.selection in allVariants)) return {} + return { + config: { selection: saved.selection, value: saved.custom ?? {} }, + } + }, async ({ effects, input }) => { const config = input.config let serveArgs: string[] + let modelSelection: { + selection: string + custom: + | { + hfRepo: string + hfFile?: string + ctx: number + ngl: number + extraArgs?: string + } + | undefined + } if (config.selection === 'custom') { const v = config.value serveArgs = ['-hf', v.hfRepo] @@ -129,6 +147,16 @@ export const setModel = sdk.Action.withInput( if (v.extraArgs && v.extraArgs.trim().length > 0) { serveArgs.push(...v.extraArgs.split(/\s+/).filter(Boolean)) } + modelSelection = { + selection: 'custom', + custom: { + hfRepo: v.hfRepo, + hfFile: v.hfFile?.trim() || undefined, + ctx: v.ctx, + ngl: v.ngl, + extraArgs: v.extraArgs?.trim() || undefined, + }, + } } else { const preset = models.find((m) => m.id === config.selection) if (!preset) { @@ -138,7 +166,8 @@ export const setModel = sdk.Action.withInput( if (preset.hfFile) serveArgs.push('-hff', preset.hfFile) serveArgs.push('-c', String(preset.defaultCtx)) if (isGpuVariant) serveArgs.push('-ngl', '999') + modelSelection = { selection: config.selection, custom: undefined } } - await storeJson.merge(effects, { serveArgs }) + await storeJson.merge(effects, { serveArgs, modelSelection }) }, ) diff --git a/startos/fileModels/.gitkeep b/startos/fileModels/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/startos/fileModels/store.json.ts b/startos/fileModels/store.json.ts index 23a43d6..beb4a0c 100644 --- a/startos/fileModels/store.json.ts +++ b/startos/fileModels/store.json.ts @@ -4,6 +4,21 @@ import { sdk } from '../sdk' const shape = z.object({ serveArgs: z.array(z.string()).optional().catch(undefined), uiPassword: z.string().optional().catch(undefined), + modelSelection: z + .object({ + selection: z.string(), + custom: z + .object({ + hfRepo: z.string(), + hfFile: z.string().optional(), + ctx: z.number(), + ngl: z.number(), + extraArgs: z.string().optional(), + }) + .optional(), + }) + .optional() + .catch(undefined), }) export const storeJson = FileHelper.json( diff --git a/startos/i18n/dictionaries/default.ts b/startos/i18n/dictionaries/default.ts index 69c2aa0..e176a3f 100644 --- a/startos/i18n/dictionaries/default.ts +++ b/startos/i18n/dictionaries/default.ts @@ -14,25 +14,19 @@ const dict = { // actions/setModel.ts 'Set Model': 7, - 'Pick a curated GGUF preset sized for your hardware, or supply a custom HuggingFace model. The model will be downloaded on first startup if not already cached.': - 8, - 'Changing the model will restart the service and may require downloading a new model.': - 9, + 'Pick a curated GGUF preset sized for your hardware, or supply a custom HuggingFace model. The model will be downloaded on first startup if not already cached.': 8, + 'Changing the model will restart the service and may require downloading a new model.': 9, Configuration: 10, 'HuggingFace repo': 11, - 'A HuggingFace GGUF repo, optionally with a quant tag (e.g. `unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M`).': - 12, + 'A HuggingFace GGUF repo, optionally with a quant tag (e.g. `unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M`).': 12, 'HuggingFace file (optional)': 13, - 'Specific GGUF filename inside the repo. Leave empty to let llama-server pick.': - 14, + 'Specific GGUF filename inside the repo. Leave empty to let llama-server pick.': 14, 'Context size': 15, 'Maximum context length in tokens. 0 uses the model default.': 16, 'GPU layers': 17, - 'Number of model layers to offload to GPU. Use a large value (e.g. 999) to offload everything; ignored on the generic CPU variant.': - 18, + 'Number of model layers to offload to GPU. Use a large value (e.g. 999) to offload everything; ignored on the generic CPU variant.': 18, 'Extra arguments': 19, - 'Additional `llama-server` flags, space-separated. Advanced — split on whitespace, so quoted values will not survive.': - 20, + 'Additional `llama-server` flags, space-separated. Advanced — split on whitespace, so quoted values will not survive.': 20, Custom: 21, // model preset labels @@ -50,17 +44,13 @@ const dict = { 'Delete Model Cache': 31, 'Remove a downloaded GGUF model from the cache to free up disk space': 32, 'Cached file': 33, - 'Filename inside `/data/models` to delete (e.g. `Qwen2.5-7B-Instruct-Q4_K_M.gguf`).': - 34, - 'This will permanently delete the cached file. The model will be re-downloaded if selected again.': - 35, + 'Filename inside `/data/models` to delete (e.g. `Qwen2.5-7B-Instruct-Q4_K_M.gguf`).': 34, + 'This will permanently delete the cached file. The model will be re-downloaded if selected again.': 35, // actions/setUiPassword.ts 'Set UI Password': 36, - 'Generate a new password for logging in to the llama.cpp web UI. The username is always "admin".': - 37, - 'This replaces any existing password. Update saved logins after running it.': - 38, + 'Generate a new password for logging in to the llama.cpp web UI. The username is always "admin".': 37, + 'This replaces any existing password. Update saved logins after running it.': 38, // init/initializeService.ts 'Generate a password to log in to the llama.cpp web UI': 39, diff --git a/startos/manifest/i18n.ts b/startos/manifest/i18n.ts index 577e71d..fc23749 100644 --- a/startos/manifest/i18n.ts +++ b/startos/manifest/i18n.ts @@ -16,5 +16,5 @@ export const long = { pl_PL: 'llama.cpp to wydajne środowisko uruchomieniowe C/C++ dla dużych modeli językowych w formacie GGUF. Ten pakiet otacza oficjalny plik binarny `llama-server`, udostępniając API HTTP zgodne z OpenAI i wbudowany interfejs czatu webowego. Wybierz wyselekcjonowany preset dopasowany do Twojego sprzętu lub podaj własny GGUF z HuggingFace — serwer pobierze wagi przy pierwszym uruchomieniu i będzie je obsługiwał lokalnie, bez żadnych połączeń z usługami zewnętrznymi.', fr_FR: - "llama.cpp est un runtime C/C++ haute performance pour les grands modèles de langage au format GGUF. Ce paquet enveloppe le binaire officiel `llama-server`, exposant une API HTTP compatible OpenAI et une interface de chat web intégrée. Choisissez un préréglage de modèle dimensionné pour votre matériel, ou fournissez un GGUF HuggingFace personnalisé — le serveur télécharge les poids au premier démarrage et les sert localement, sans appels à des tiers.", + 'llama.cpp est un runtime C/C++ haute performance pour les grands modèles de langage au format GGUF. Ce paquet enveloppe le binaire officiel `llama-server`, exposant une API HTTP compatible OpenAI et une interface de chat web intégrée. Choisissez un préréglage de modèle dimensionné pour votre matériel, ou fournissez un GGUF HuggingFace personnalisé — le serveur télécharge les poids au premier démarrage et les sert localement, sans appels à des tiers.', } diff --git a/startos/manifest/index.ts b/startos/manifest/index.ts index 887503b..890a304 100644 --- a/startos/manifest/index.ts +++ b/startos/manifest/index.ts @@ -6,7 +6,7 @@ const variant = process.env.VARIANT || 'generic' type Mutable = { -readonly [K in keyof T]: Mutable } const mutable = (value: T): Mutable => value as Mutable -const upstreamBuild = 'b10438' +const upstreamBuild = 'b10450' const imageConfigs = { generic: { diff --git a/startos/versions/current.ts b/startos/versions/current.ts index 370c28b..3a1f438 100644 --- a/startos/versions/current.ts +++ b/startos/versions/current.ts @@ -1,58 +1,93 @@ import { IMPOSSIBLE, VersionInfo } from '@start9labs/start-sdk' export const current = VersionInfo.of({ - version: '1.0.10438:0', + version: '1.0.10450:0', releaseNotes: { - en_US: `Updated llama.cpp to build b10438. - -- A routine maintenance bump — 40 builds since b10398. -- The server now answers \`/metrics\` and \`/slots\` while a request is being processed, so monitoring no longer stalls behind generation, and several metrics were corrected. -- The built-in chat UI is served with no-cache, so upgrades take effect without a manual browser refresh. -- Adds support for the MiniMax Text-01 and MiniMax-M1 model families. -- Fixes tool calling with LFM2 models and image handling with Granite 4 vision models. -- Speculative decoding now auto-detects the draft model type from the draft GGUF's metadata. - -llama.cpp publishes one build per merged commit and does not provide a per-build changelog. Full commit range: https://github.com/ggml-org/llama.cpp/compare/b10398...b10438`, - es_ES: `Actualiza llama.cpp a la compilación b10438. - -- Una actualización de mantenimiento rutinaria: 40 compilaciones desde la b10398. -- El servidor ahora responde a \`/metrics\` y \`/slots\` mientras procesa una petición, por lo que la supervisión ya no queda bloqueada tras la generación, y se han corregido varias métricas. -- La interfaz de chat integrada se sirve sin caché, de modo que las actualizaciones surten efecto sin recargar manualmente el navegador. -- Añade compatibilidad con las familias de modelos MiniMax Text-01 y MiniMax-M1. -- Corrige la llamada a herramientas con modelos LFM2 y el tratamiento de imágenes con los modelos de visión Granite 4. -- La decodificación especulativa detecta ahora automáticamente el tipo de modelo borrador a partir de los metadatos de su GGUF. - -llama.cpp publica una compilación por cada commit fusionado y no ofrece un registro de cambios por compilación. Rango completo de commits: https://github.com/ggml-org/llama.cpp/compare/b10398...b10438`, - de_DE: `Aktualisiert llama.cpp auf Build b10438. - -- Ein routinemäßiges Wartungsupdate – 40 Builds seit b10398. -- Der Server beantwortet \`/metrics\` und \`/slots\` jetzt auch während der Verarbeitung einer Anfrage, sodass die Überwachung nicht mehr hinter der Generierung hängen bleibt; zudem wurden mehrere Metriken korrigiert. -- Die integrierte Chat-Oberfläche wird ohne Caching ausgeliefert, sodass Aktualisierungen ohne manuelles Neuladen des Browsers wirksam werden. -- Ergänzt Unterstützung für die Modellfamilien MiniMax Text-01 und MiniMax-M1. -- Behebt Werkzeugaufrufe mit LFM2-Modellen sowie die Bildverarbeitung mit den Granite-4-Vision-Modellen. -- Die spekulative Dekodierung erkennt den Typ des Entwurfsmodells jetzt automatisch anhand der Metadaten der Entwurfs-GGUF. - -llama.cpp veröffentlicht einen Build pro zusammengeführtem Commit und stellt kein Änderungsprotokoll je Build bereit. Vollständiger Commit-Bereich: https://github.com/ggml-org/llama.cpp/compare/b10398...b10438`, - pl_PL: `Aktualizuje llama.cpp do kompilacji b10438. - -- Rutynowa aktualizacja konserwacyjna — 40 kompilacji od b10398. -- Serwer odpowiada teraz na \`/metrics\` i \`/slots\` również w trakcie przetwarzania żądania, więc monitorowanie nie czeka już na zakończenie generowania; poprawiono także kilka metryk. -- Wbudowany interfejs czatu jest serwowany bez pamięci podręcznej, dzięki czemu aktualizacje działają bez ręcznego odświeżania przeglądarki. -- Dodaje obsługę rodzin modeli MiniMax Text-01 i MiniMax-M1. -- Poprawia wywoływanie narzędzi w modelach LFM2 oraz obsługę obrazów w modelach wizyjnych Granite 4. -- Dekodowanie spekulacyjne automatycznie rozpoznaje typ modelu roboczego na podstawie metadanych jego pliku GGUF. - -llama.cpp publikuje jedną kompilację na scalony commit i nie udostępnia listy zmian dla poszczególnych kompilacji. Pełny zakres commitów: https://github.com/ggml-org/llama.cpp/compare/b10398...b10438`, - fr_FR: `Met à jour llama.cpp vers la version b10438. - -- Une mise à jour de maintenance de routine — 40 versions depuis la b10398. -- Le serveur répond désormais à \`/metrics\` et \`/slots\` pendant le traitement d'une requête : la supervision n'attend plus la fin de la génération, et plusieurs métriques ont été corrigées. -- L'interface de discussion intégrée est servie sans mise en cache, si bien que les mises à jour s'appliquent sans rechargement manuel du navigateur. -- Ajoute la prise en charge des familles de modèles MiniMax Text-01 et MiniMax-M1. -- Corrige l'appel d'outils avec les modèles LFM2 et le traitement des images avec les modèles de vision Granite 4. -- Le décodage spéculatif détecte maintenant automatiquement le type du modèle brouillon à partir des métadonnées de son GGUF. - -llama.cpp publie une version par commit fusionné et ne fournit pas de journal des modifications par version. Plage complète des commits : https://github.com/ggml-org/llama.cpp/compare/b10398...b10438`, + en_US: `Updated llama.cpp to build b10450, and the **Set Model** action now remembers what you picked. + +**This package** + +- The **Set Model** action now opens pre-filled with your current selection instead of resetting to the defaults, so you can change one setting — or just check which custom model is configured — without re-entering everything. + +**llama.cpp** + +- A small maintenance bump — 12 builds since b10438. +- Adds support for the Kimi-K3 text model family. +- The server's request queue was reworked, improving behaviour under concurrent requests. +- Vulkan: better performance on Intel Xe graphics, and a workaround for a problematic Intel driver version. +- The built-in chat UI now masks the API key field so browsers stop offering to save it. +- More robust GGUF parsing — malformed model metadata is rejected instead of misread. +- The \`--mmap\`/\`--no-mmap\`/\`--mlock\` flags are now deprecated in favour of \`--load-mode\`. They still work; if you pass them through **Set Model**'s extra arguments, switch when convenient. + +llama.cpp publishes one build per merged commit and does not provide a per-build changelog. Full commit range: https://github.com/ggml-org/llama.cpp/compare/b10438...b10450`, + es_ES: `Actualiza llama.cpp a la compilación b10450, y la acción **Establecer modelo** ahora recuerda lo que eligió. + +**Este paquete** + +- La acción **Establecer modelo** ahora se abre con su selección actual ya rellenada, en lugar de volver a los valores predeterminados, de modo que puede cambiar un solo ajuste —o simplemente comprobar qué modelo personalizado está configurado— sin volver a introducirlo todo. + +**llama.cpp** + +- Una pequeña actualización de mantenimiento: 12 compilaciones desde la b10438. +- Añade compatibilidad con la familia de modelos de texto Kimi-K3. +- Se ha rediseñado la cola de peticiones del servidor, mejorando su comportamiento con peticiones simultáneas. +- Vulkan: mejor rendimiento en gráficos Intel Xe y una solución alternativa para una versión problemática del controlador de Intel. +- La interfaz de chat integrada ahora enmascara el campo de la clave de API para que el navegador no ofrezca guardarla. +- Análisis de GGUF más robusto: los metadatos de modelo mal formados se rechazan en lugar de interpretarse mal. +- Las opciones \`--mmap\`/\`--no-mmap\`/\`--mlock\` quedan obsoletas en favor de \`--load-mode\`. Siguen funcionando; si las pasa como argumentos adicionales en **Establecer modelo**, cámbielas cuando le resulte cómodo. + +llama.cpp publica una compilación por cada commit fusionado y no ofrece un registro de cambios por compilación. Rango completo de commits: https://github.com/ggml-org/llama.cpp/compare/b10438...b10450`, + de_DE: `Aktualisiert llama.cpp auf Build b10450, und die Aktion **Modell festlegen** merkt sich jetzt Ihre Auswahl. + +**Dieses Paket** + +- Die Aktion **Modell festlegen** öffnet sich jetzt mit Ihrer aktuellen Auswahl vorausgefüllt, statt auf die Standardwerte zurückzuspringen. So können Sie eine einzelne Einstellung ändern – oder einfach nachsehen, welches benutzerdefinierte Modell konfiguriert ist – ohne alles neu einzugeben. + +**llama.cpp** + +- Ein kleines Wartungsupdate – 12 Builds seit b10438. +- Ergänzt Unterstützung für die Textmodellfamilie Kimi-K3. +- Die Anfrage-Warteschlange des Servers wurde überarbeitet, was das Verhalten bei gleichzeitigen Anfragen verbessert. +- Vulkan: bessere Leistung auf Intel-Xe-Grafik sowie eine Umgehung für eine problematische Intel-Treiberversion. +- Die integrierte Chat-Oberfläche maskiert das Feld für den API-Schlüssel, damit Browser dessen Speicherung nicht mehr anbieten. +- Robustere GGUF-Auswertung: fehlerhafte Modell-Metadaten werden abgelehnt statt falsch gelesen. +- Die Optionen \`--mmap\`/\`--no-mmap\`/\`--mlock\` gelten zugunsten von \`--load-mode\` als veraltet. Sie funktionieren weiterhin; wenn Sie sie über die zusätzlichen Argumente von **Modell festlegen** übergeben, stellen Sie bei Gelegenheit um. + +llama.cpp veröffentlicht einen Build pro zusammengeführtem Commit und stellt kein Änderungsprotokoll je Build bereit. Vollständiger Commit-Bereich: https://github.com/ggml-org/llama.cpp/compare/b10438...b10450`, + pl_PL: `Aktualizuje llama.cpp do kompilacji b10450, a akcja **Ustaw model** zapamiętuje teraz Twój wybór. + +**Ten pakiet** + +- Akcja **Ustaw model** otwiera się teraz wypełniona bieżącym wyborem, zamiast wracać do wartości domyślnych — możesz zmienić pojedyncze ustawienie lub po prostu sprawdzić, który model niestandardowy jest skonfigurowany, bez wpisywania wszystkiego od nowa. + +**llama.cpp** + +- Niewielka aktualizacja konserwacyjna — 12 kompilacji od b10438. +- Dodaje obsługę rodziny modeli tekstowych Kimi-K3. +- Przeprojektowano kolejkę żądań serwera, co poprawia zachowanie przy równoczesnych żądaniach. +- Vulkan: lepsza wydajność na układach graficznych Intel Xe oraz obejście problematycznej wersji sterownika Intela. +- Wbudowany interfejs czatu maskuje teraz pole klucza API, dzięki czemu przeglądarka nie proponuje jego zapisania. +- Solidniejsze przetwarzanie plików GGUF — błędne metadane modelu są odrzucane zamiast błędnie odczytywane. +- Opcje \`--mmap\`/\`--no-mmap\`/\`--mlock\` są przestarzałe na rzecz \`--load-mode\`. Nadal działają; jeśli przekazujesz je w dodatkowych argumentach akcji **Ustaw model**, zmień je przy okazji. + +llama.cpp publikuje jedną kompilację na scalony commit i nie udostępnia listy zmian dla poszczególnych kompilacji. Pełny zakres commitów: https://github.com/ggml-org/llama.cpp/compare/b10438...b10450`, + fr_FR: `Met à jour llama.cpp vers la version b10450, et l'action **Définir le modèle** mémorise désormais votre sélection. + +**Ce paquet** + +- L'action **Définir le modèle** s'ouvre désormais pré-remplie avec votre sélection actuelle au lieu de revenir aux valeurs par défaut : vous pouvez ainsi modifier un seul paramètre — ou simplement vérifier quel modèle personnalisé est configuré — sans tout ressaisir. + +**llama.cpp** + +- Une petite mise à jour de maintenance — 12 versions depuis la b10438. +- Ajoute la prise en charge de la famille de modèles de texte Kimi-K3. +- La file d'attente des requêtes du serveur a été repensée, ce qui améliore le comportement en cas de requêtes simultanées. +- Vulkan : meilleures performances sur les cartes graphiques Intel Xe et contournement d'une version problématique du pilote Intel. +- L'interface de discussion intégrée masque désormais le champ de la clé d'API, afin que le navigateur ne propose plus de l'enregistrer. +- Analyse des fichiers GGUF plus robuste : des métadonnées de modèle malformées sont rejetées au lieu d'être mal interprétées. +- Les options \`--mmap\`/\`--no-mmap\`/\`--mlock\` sont désormais obsolètes au profit de \`--load-mode\`. Elles fonctionnent toujours ; si vous les passez dans les arguments supplémentaires de **Définir le modèle**, changez-les à votre convenance. + +llama.cpp publie une version par commit fusionné et ne fournit pas de journal des modifications par version. Plage complète des commits : https://github.com/ggml-org/llama.cpp/compare/b10438...b10450`, }, migrations: { up: async ({ effects }) => {},