Add Muse-Glimmer-30B on a pinned mainline llama.cpp image

Muse-Glimmer-30B-GGUF is a dense 30B VLM whose architecture landed in mainline
llama.cpp on 2026-08-10 (ggml-org/llama.cpp#26841, build b10353). The bonsai image
builds PrismML's `prism` fork, whose branch head is 2026-07-31 — ten days older than
the architecture — so it cannot load this model at all. Hence a second llama.cpp
image rather than a new POOL entry.

The two images are not redundant and neither replaces the other: the fork carries the
Q2_0_g128 ternary kernels Ternary-Bonsai needs and mainline lacks; mainline carries
every architecture merged after the fork diverged. Both share docker/gguf-serve.sh as
their entrypoint, so a model moves between them by changing only `image:`.

Two entries, because KV turned out to be nearly free on this model (52 layers,
num_key_value_heads=2, and a 3:1 sliding/full attention split) — the full native
131072 context costs ~1.82 GiB, so weights, not context, are the constraint:

  muse-glimmer            one 3090, 17gb+vision,          131072 in 1 slot, f16 KV
  Muse-Glimmer-30B-split  both 3090s, dynamic+vision+draft, 131072 in 1 slot, f16 KV

Both are ungrouped (new UNGROUPED_GGUF list), so neither generates pairNN co-load
pairs: the standby model is not a co-load partner. config.yaml is purely additive —
104 -> 106 models with all pre-existing entries byte-identical and groups unchanged
at 45.

muse-glimmer is also the on-call standby: ttl 0 (never idle-unload) plus a poller that
wakes it once both 3090s have been quiet for an hour. It deliberately does NOT use
`persistent: true` — llama-swap defines that as "other groups can never unload this
group's members", which would pin the cards and starve every other model. Eviction is
already free via the existing exclusive groups, and the wakeup request itself displaces
whatever is squatting, so nothing calls the unload API.

Also fixes a pre-existing gap: bonsai-image.yml did not trigger on docker/gguf-serve.sh
even though Dockerfile.bonsai COPYs it, so a change there would have rebuilt only the
mainline image and left the bonsai one silently stale.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Tuomo Vendelin

.github/workflows/bonsai-image.yml

@@ -10,6 +10,9 @@ on:
     paths:
       - "Dockerfile.bonsai"
       - "docker/bonsai-serve.sh"
+      # Dockerfile.bonsai COPYs this too — without it here, a gguf-serve.sh change
+      # rebuilds only the mainline image and leaves this one silently stale.
+      - "docker/gguf-serve.sh"
       - ".github/workflows/bonsai-image.yml"
   workflow_dispatch:
 

.github/workflows/llamacpp-image.yml

@@ -0,0 +1,83 @@
+name: llamacpp-image
+
+# Builds the pinned mainline llama.cpp image (Muse-Glimmer-30B and any future model the
+# PrismML fork can't load) and pushes it to GHCR. This is a CUDA compile, so it only runs
+# when its own files change or on manual dispatch — NOT on every push.
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "Dockerfile.llamacpp"
+      - "docker/gguf-serve.sh"
+      - ".github/workflows/llamacpp-image.yml"
+  workflow_dispatch:
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository_owner }}/llamacpp-mainline   # separate package from bonsai
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      # The CUDA build produces GBs of object files; the runner's / is only ~14 GB.
+      # Free extra space AND move Docker's storage to the large /mnt volume (~70 GB)
+      # so buildkit doesn't fill / mid-build.
+      - name: Maximize build disk
+        run: |
+          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/.ghcup \
+                      /usr/local/lib/android /opt/hostedtoolcache /usr/share/swift \
+                      /usr/local/share/boost /usr/lib/jvm /usr/share/miniconda || true
+          sudo swapoff -a || true
+          sudo systemctl stop docker docker.socket || true
+          sudo mkdir -p /mnt/docker
+          echo '{"data-root":"/mnt/docker"}' | sudo tee /etc/docker/daemon.json
+          sudo systemctl start docker
+          echo "--- disk ---"; df -h / /mnt; docker info | grep -i "docker root" || true
+
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Buildx
+        # docker-container driver is required for the GHA layer cache (cache-to/from).
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Extract image metadata
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=raw,value=latest,enable={{is_default_branch}}
+            type=sha,format=short
+
+      - name: Build and push
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: Dockerfile.llamacpp
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Summary
+        run: |
+          echo "### Mainline llama.cpp image pushed :rocket:" >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "${{ steps.meta.outputs.tags }}" | tr ' ' '\n' >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "Make this package public too, then it's referenced by config.yaml." >> "$GITHUB_STEP_SUMMARY"

ARCHITECTURE.md

@@ -138,11 +138,29 @@ cache blocks" failure.
 | AWQ / safetensors (most models) | **vLLM** `v0.26.0` | `cmd: docker run … vllm/vllm-openai …` (DooD) |
 | GGUF, mainstream arch | llama.cpp | bundled `llama-server` child process |
 | GGUF, exotic (e.g. `Ternary-Bonsai-27B`) | **PrismML llama.cpp fork** | `cmd: docker run …` of a fork image (custom kernels) |
+| GGUF, newer arch (e.g. `Muse-Glimmer-30B`) | **mainline llama.cpp, pinned build** | `cmd: docker run …` of `Dockerfile.llamacpp` |
 
 `Ternary-Bonsai-27B` is a hybrid-attention, multimodal, ternary-quantized model built for
 a **PrismML fork of llama.cpp** — vLLM 0.25.1 cannot serve it. This is a concrete reason
 the backend-agnostic design matters.
 
+### Why there are TWO llama.cpp images
+
+This is the non-obvious bit. They are not redundant and neither can replace the other:
+
+- `Dockerfile.bonsai` builds **PrismML's `prism` fork**, which carries the Q2_0_g128 ternary
+  and hybrid-attention CUDA kernels `Ternary-Bonsai-27B` needs. Mainline does not have them.
+- `Dockerfile.llamacpp` builds **mainline at a pinned build tag**. The fork's branch head is
+  2026-07-31, so it predates any architecture merged after that — `Muse-Glimmer-30B` landed
+  in mainline on 2026-08-10 (`ggml-org/llama.cpp#26841`, build `b10353`) and fails on the
+  fork with an unknown-architecture error.
+
+Both images share `docker/gguf-serve.sh` as their entrypoint, so moving a model between them
+means changing only `image:` in the generated config. Pin the mainline tag rather than
+tracking a rolling one, for the same reason the vLLM image is pinned to `v0.26.0`. When
+adding a GGUF model, the question to answer first is *which image can actually load it* —
+check when its architecture was merged against the fork's branch date.
+
 ## Request lifecycle
 
 1. Client → `POST /v1/chat/completions` to llama-swap `:9292` with `"model": "<id>"`.

Dockerfile

@@ -21,5 +21,11 @@ RUN apt-get update \
 # models stays a git push (see .github/workflows/build-and-push.yml).
 COPY config.yaml /etc/llama-swap/config/config.yaml
 
+# On-call standby poller. Baked in for the same reason as config.yaml above — the
+# docker-compose `oncall-wakeup` service runs this image with an entrypoint override
+# rather than bind-mounting the script, so a Portainer Git stack can't mangle it.
+COPY scripts/oncall-wakeup.sh /usr/local/bin/oncall-wakeup.sh
+RUN chmod +x /usr/local/bin/oncall-wakeup.sh
+
 # Entrypoint/CMD are inherited from the base image; the compose file passes
 # --config and --listen.

Dockerfile.llamacpp

@@ -0,0 +1,69 @@
+# Mainline llama.cpp image (pinned build) for GGUF models the PrismML fork can't load.
+#
+# Why this exists ALONGSIDE Dockerfile.bonsai: the bonsai image builds PrismML's `prism`
+# fork, which carries the Q2_0_g128 ternary kernels Ternary-Bonsai-27B needs but that
+# mainline does not have. The fork's branch head is 2026-07-31, so it predates newer
+# architectures — Muse-Glimmer-30B landed in mainline on 2026-08-10 (ggml-org/llama.cpp
+# #26841) and needs build b10353+. Neither image can serve both models, hence two.
+#
+# Pin LLAMACPP_TAG to a build tag, never a rolling one — same policy as the vLLM image
+# (vllm/vllm-openai:v0.26.0). The official ghcr.io/ggml-org/llama.cpp:server-cuda package
+# only publishes a rolling tag for recent builds, which is why we compile instead.
+#
+# Built & pushed to GHCR by .github/workflows/llamacpp-image.yml.
+# NOTE: Dockerfile inline comments are only valid on their own line (never after an
+# instruction like ARG/ENV), so all comments here stand alone.
+
+# ---- build stage -----------------------------------------------------------
+FROM nvidia/cuda:12.6.2-devel-ubuntu22.04 AS build
+# RTX 3090 = sm_86 (this box's two LLM cards).
+ARG CUDA_ARCHS=86
+# b10362: 9 builds after Muse Glimmer support (b10353), to pick up immediate fixes.
+ARG LLAMACPP_TAG=b10362
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      git cmake ninja-build build-essential ca-certificates libcurl4-openssl-dev \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN git clone --depth 1 -b ${LLAMACPP_TAG} https://github.com/ggml-org/llama.cpp.git /src
+WORKDIR /src
+
+# The build box has no GPU/driver, so executables can't resolve CUDA driver-API
+# symbols (cuMemCreate, cuDeviceGet, ...). Link against the CUDA driver STUB the
+# devel image ships, exposed as libcuda.so.1 via LIBRARY_PATH. Build ONLY the
+# server target. Cap parallelism by RAM to avoid nvcc OOM on smaller runners.
+RUN export LIBRARY_PATH="/usr/local/cuda/lib64/stubs${LIBRARY_PATH:+:$LIBRARY_PATH}" \
+ && cmake -B build -G Ninja \
+      -DGGML_CUDA=ON \
+      -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCHS} \
+      -DCMAKE_BUILD_TYPE=Release \
+      -DLLAMA_BUILD_TESTS=OFF \
+      -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" \
+      -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" \
+ && MEM_GB=$(( $(awk '/MemTotal/{print $2}' /proc/meminfo) / 1048576 )) \
+ && JOBS=$([ "${MEM_GB:-0}" -lt 16 ] && echo 2 || echo 4) \
+ && echo "building llama.cpp ${LLAMACPP_TAG} with -j${JOBS} (host ~${MEM_GB} GB RAM)" \
+ && cmake --build build --target llama-server -j "${JOBS}"
+
+# ---- runtime stage ---------------------------------------------------------
+FROM nvidia/cuda:12.6.2-runtime-ubuntu22.04
+ENV DEBIAN_FRONTEND=noninteractive
+# hf CLI: gguf-serve.sh downloads weights with it (this build's llama-server has no
+# HTTPS). libcurl4/libgomp1: runtime deps of llama-server. curl: container health tooling.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      python3 python3-pip libgomp1 libcurl4 ca-certificates curl \
+ && pip3 install --no-cache-dir gguf "huggingface_hub[cli]" hf_xet \
+ && rm -rf /var/lib/apt/lists/*
+
+# Copy the whole build output dir so the versioned .so SONAME symlinks
+# (libggml-cuda.so -> .so.0 -> .so.0.13.1) are preserved for the runtime linker.
+COPY --from=build /src/build/bin/ /opt/llamacpp/bin/
+ENV PATH="/opt/llamacpp/bin:${PATH}"
+ENV LD_LIBRARY_PATH="/opt/llamacpp/bin"
+
+# Same entrypoint the bonsai image uses for its non-ternary GGUF models, so a model
+# moves between the two images by changing only the `image:` in config.yaml.
+COPY docker/gguf-serve.sh /usr/local/bin/gguf-serve.sh
+RUN chmod +x /usr/local/bin/gguf-serve.sh
+
+ENTRYPOINT ["/usr/local/bin/gguf-serve.sh"]

README.md

@@ -23,6 +23,10 @@ static GPU layout** — which is more reliable than dynamic VRAM packing for thi
 | `Dockerfile.bonsai` + `docker/bonsai-serve.sh` | PrismML llama.cpp fork image for Ternary-Bonsai-27B (built by CI → GHCR) |
 | `.github/workflows/build-and-push.yml` | CI: builds the Dockerfile and pushes the image to GHCR (Portainer can't build from a repo) |
 | `.github/workflows/bonsai-image.yml` | CI: builds the PrismML fork image → `ghcr.io/<owner>/bonsai-llama` |
+| `Dockerfile.llamacpp` | **Mainline** llama.cpp at a pinned build (`b10362`) — for models the PrismML fork is too old to load (built by CI → GHCR) |
+| `.github/workflows/llamacpp-image.yml` | CI: builds the mainline image → `ghcr.io/<owner>/llamacpp-mainline` |
+| `docker/gguf-serve.sh` | Shared GGUF entrypoint for **both** llama.cpp images (download-then-serve; vision / drafter / multi-GPU via `GGUF_*` env vars) |
+| `scripts/oncall-wakeup.sh` | On-call standby poller — keeps one model warm when the box is idle (see below) |
 | `docker-compose.yml` | Portainer stack definition (pulls the pre-built image) |
 | `config.yaml` | llama-swap model definitions — GENERATED by `gen_pairs_config.py` (do not hand-edit): standalone models + co-load pairs + solo big models |
 | `gen_pairs_config.py` | Generates `config.yaml`: `pairNN.<model>` co-load pairs (one model per 3090) + standalone `<model>` entries. Edit the `POOL`, run `python3 gen_pairs_config.py > config.yaml` |
@@ -198,6 +202,70 @@ repo builds the fork into its own image and points the model at it:
    `llama-server` with vision + DSpark speculative + tool calling, on one 3090. It speaks
    OpenAI `/v1` + `/health`, so llama-swap proxies it like any other model.
 
+## Muse-Glimmer-30B (mainline llama.cpp backend)
+
+A dense 30B vision-language model (Apache-2.0). It needs **mainline** llama.cpp `b10353+`
+— support landed 2026-08-10 — so it **cannot** use the bonsai image, whose `prism` fork
+branch is from 2026-07-31, ten days older than the architecture. That's why there are two
+llama.cpp images; see ARCHITECTURE → "Backend matrix".
+
+It's cheap on KV: 52 layers, `num_key_value_heads: 2`, and a 3:1 sliding/full attention
+split (39 sliding layers windowed at 2048, 13 full). The full layers cost 13 KiB/token and
+the sliding ones a flat 78 MiB, so the **entire native 131072 context costs under 2 GiB**.
+Weights, not context, are the constraint — which is why two entries exist:
+
+| Model ID | GPUs | Quant | Context | Notes |
+|---|---|---|---|---|
+| `muse-glimmer` | one 3090 | `17gb` + vision | 131072, 1 slot | the **on-call** model |
+| `Muse-Glimmer-30B-split` | both 3090s | `dynamic` + vision + drafter | 131072, 1 slot | best quality; owns both cards |
+
+Both run `-np 1` — the full native context in a single slot, with unquantized f16 KV. Neither
+is in `POOL`, so **neither generates `pairNN` co-load pairs**: the standby model is not a
+co-load partner, and pairing it would have added 10 pairs nothing would request. They live in
+`UNGROUPED_GGUF` in `gen_pairs_config.py`, and the pair count stays at 45.
+
+1. **Build the image** — the `llamacpp-image` workflow (`Dockerfile.llamacpp`) compiles
+   mainline at the pinned `LLAMACPP_TAG` and pushes `ghcr.io/<owner>/llamacpp-mainline:latest`.
+   Then **make that GHCR package public** (same as the other images).
+2. **Download the weights** on the host (~38 GB, so don't let the first cold start do it):
+
+   ```bash
+   HF_TOKEN=hf_... hf download meta-models/Muse-Glimmer-30B-GGUF \
+     muse-glimmer-30B-kquant-17gb.gguf muse-glimmer-30B-kquant-dynamic.gguf \
+     mmproj-kquant.gguf dflash-kquant.gguf \
+     --local-dir /models/hf-cache/gguf/meta-models_Muse-Glimmer-30B-GGUF
+   ```
+
+3. Request either model ID. `gguf-serve.sh` already passes `--jinja`, which the model
+   requires for its embedded chat template.
+
+The model supports `reasoning_strength` (`low`/`medium`/`high`/`xhigh`). It's left at the
+model's own default; to pin it, add `params=dict(reasoning_strength="low")` to the POOL
+entry in `gen_pairs_config.py` and regenerate — it emits a `filters.setParams` block.
+
+## On-call standby model
+
+`muse-glimmer` is kept resident whenever the box has stopped working, **without** ever
+blocking another model. Three parts:
+
+- Its standalone entry carries **`ttl: 0`** (never idle-unload). Its `pairNN.` members keep
+  the normal 5 h TTL.
+- Every pair/solo group is `exclusive: true`, so **a request to any other model evicts it
+  immediately**. Nothing has to unload it explicitly.
+- The `oncall-wakeup` compose service (`scripts/oncall-wakeup.sh`) polls `/metrics`; when
+  **both 3090s** sit below `IDLE_PCT` for `IDLE_SECONDS`, it sends a 1-token request, which
+  evicts whatever is squatting and loads `muse-glimmer`. It matches GPUs by **UUID**, not
+  index, so a reboot reordering the cards can't make it watch the A2000.
+
+> ⚠️ This is deliberately **not** `persistent: true` — llama-swap defines that as "other
+> groups can never unload this group's members", which would pin the cards and starve every
+> other model (same trap as the warning above).
+
+Tunable via the service's env vars (`ONCALL_MODEL`, `IDLE_SECONDS`, `IDLE_PCT`,
+`POLL_SECONDS`); set `IDLE_SECONDS` low to test, then put it back. **Consequence to accept:**
+with the poller running, an idle model gets replaced after ~1 h rather than surviving its
+full 5 h TTL. The TTLs still govern unloading; the poller governs replacement.
+
 ## Operational notes
 
 - **vLLM crash mitigation:** keep `--enforce-eager` on `Qwen3.6-35B-A3B` until the Xid 31

TODO.md

@@ -58,6 +58,47 @@ Status legend: `[ ]` todo · `[~]` in progress · `[x]` done
   unquantized for this): test SEPARATELY from prefix caching first — the combo has
   crashed during cudagraph profiling on hybrid Mamba models — then together.
 
+## Post-deploy verification (Muse-Glimmer-30B + on-call standby)
+
+Config-side checks already pass locally: 106 models / 45 groups, with all 104 pre-existing
+entries byte-identical and the two Muse-Glimmer entries purely additive. The rest needs the host.
+
+- [ ] Make the `llamacpp-mainline` GHCR package **public** (or add registry creds in Portainer).
+- [ ] Rebuild the **bonsai** image too — both images share `docker/gguf-serve.sh`, which
+  changed. Without it, `qwythos-v2` / `fablevibes` run an older script than config assumes.
+- [ ] Pre-download the ~38 GB of GGUFs (README → Muse-Glimmer) so the first cold start
+  isn't a multi-GB stall.
+- [ ] **Restart llama-swap** — config is read at startup only. `/v1/models` going 104 → 106
+  confirms the new config was actually picked up.
+- [ ] `muse-glimmer` cold start: watch `nvidia-smi` against the budget (16.76 weights +
+  1.40 mmproj + ~1.2 compute + ~1.82 KV ≈ 21.2 GiB, **~2.1 GiB spare**). This is the
+  thinnest number in the whole change. It assumes llama.cpp allocates SWA layers windowed,
+  not full — if it allocates full, KV jumps to ~6.5 GiB and it OOMs. Fallback ladder:
+  drop `ctx` 131072 → 65536, then `cache_type="q8_0"`.
+- [ ] **Coherence prompt** on first load — per the `qwen3.5-9b` precedent above, trust the
+  output, not a clean startup log.
+- [ ] **Vision probe**: send an image part and confirm a grounded description, proving
+  `--mmproj` actually attached rather than being silently ignored.
+- [ ] Context probe near 131k, plus an over-limit request (expect a clean 400).
+- [ ] **Stop tokens**: the model card warns never to stop on `<|eom|>` (only
+  `<|end_of_text|>` / `<|eot|>`). Confirm generations end cleanly and aren't truncated
+  mid-reasoning; if they are, add explicit EOG handling in `gguf-serve.sh`.
+- [ ] Confirm `-np 1` behaviour under load: a second concurrent request should QUEUE behind
+  the first (llama-server has one slot; `concurrencyLimit: 4` lets llama-swap admit 4), not
+  429 or share context. If queueing hurts in practice, that is the argument for raising
+  `par` — at the cost of KV, since each extra slot adds its own 2048 sliding window.
+- [ ] `Muse-Glimmer-30B-split`: confirm `-sm layer` spreads across both 3090s, and measure
+  decode speed — expect ~single-card, since layer-split is pipeline, not tensor, parallel.
+  If it's *slower* than the single-card entry, the split entry isn't earning its disk.
+- [ ] On-call: set `IDLE_SECONDS=120` on the `oncall-wakeup` service, confirm exactly one
+  wakeup fires and `/running` then shows `muse-glimmer`. Restore `3600`.
+- [ ] On-call **yield test**: with `muse-glimmer` resident, request `gemma-26b` and confirm
+  a clean swap. This is what `persistent: true` would have broken.
+- [ ] On-call **no-interrupt test**: start a long generation on another model, confirm the
+  poller doesn't fire mid-stream (GPU util stays above `IDLE_PCT`, resetting the counter).
+- [ ] Decide whether the ~1 h replacement delay is right in practice — it now supersedes
+  the 5 h TTL for *replacement* (TTLs still govern unloading).
+
 ## 1. Build the custom image
 
 - [ ] `Dockerfile` = `FROM ghcr.io/mostlygeek/llama-swap:unified-cuda` + `docker.io`.