Muse-Glimmer-30B + mainline llama.cpp image → main

.github/workflows/bonsai-image.yml

@@ -10,6 +10,9 @@ on:
     paths:
       - "Dockerfile.bonsai"
       - "docker/bonsai-serve.sh"
+      # Dockerfile.bonsai COPYs this too — without it here, a gguf-serve.sh change
+      # rebuilds only the mainline image and leaves this one silently stale.
+      - "docker/gguf-serve.sh"
       - ".github/workflows/bonsai-image.yml"
   workflow_dispatch:
 

.github/workflows/llamacpp-image.yml

@@ -0,0 +1,83 @@
+name: llamacpp-image
+
+# Builds the pinned mainline llama.cpp image (Muse-Glimmer-30B and any future model the
+# PrismML fork can't load) and pushes it to GHCR. This is a CUDA compile, so it only runs
+# when its own files change or on manual dispatch — NOT on every push.
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "Dockerfile.llamacpp"
+      - "docker/gguf-serve.sh"
+      - ".github/workflows/llamacpp-image.yml"
+  workflow_dispatch:
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository_owner }}/llamacpp-mainline   # separate package from bonsai
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      # The CUDA build produces GBs of object files; the runner's / is only ~14 GB.
+      # Free extra space AND move Docker's storage to the large /mnt volume (~70 GB)
+      # so buildkit doesn't fill / mid-build.
+      - name: Maximize build disk
+        run: |
+          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/.ghcup \
+                      /usr/local/lib/android /opt/hostedtoolcache /usr/share/swift \
+                      /usr/local/share/boost /usr/lib/jvm /usr/share/miniconda || true
+          sudo swapoff -a || true
+          sudo systemctl stop docker docker.socket || true
+          sudo mkdir -p /mnt/docker
+          echo '{"data-root":"/mnt/docker"}' | sudo tee /etc/docker/daemon.json
+          sudo systemctl start docker
+          echo "--- disk ---"; df -h / /mnt; docker info | grep -i "docker root" || true
+
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Buildx
+        # docker-container driver is required for the GHA layer cache (cache-to/from).
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Extract image metadata
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=raw,value=latest,enable={{is_default_branch}}
+            type=sha,format=short
+
+      - name: Build and push
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: Dockerfile.llamacpp
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Summary
+        run: |
+          echo "### Mainline llama.cpp image pushed :rocket:" >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "${{ steps.meta.outputs.tags }}" | tr ' ' '\n' >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          echo "Make this package public too, then it's referenced by config.yaml." >> "$GITHUB_STEP_SUMMARY"

ARCHITECTURE.md

@@ -138,11 +138,29 @@ cache blocks" failure.
 | AWQ / safetensors (most models) | **vLLM** `v0.26.0` | `cmd: docker run … vllm/vllm-openai …` (DooD) |
 | GGUF, mainstream arch | llama.cpp | bundled `llama-server` child process |
 | GGUF, exotic (e.g. `Ternary-Bonsai-27B`) | **PrismML llama.cpp fork** | `cmd: docker run …` of a fork image (custom kernels) |
+| GGUF, newer arch (e.g. `Muse-Glimmer-30B`) | **mainline llama.cpp, pinned build** | `cmd: docker run …` of `Dockerfile.llamacpp` |
 
 `Ternary-Bonsai-27B` is a hybrid-attention, multimodal, ternary-quantized model built for
 a **PrismML fork of llama.cpp** — vLLM 0.25.1 cannot serve it. This is a concrete reason
 the backend-agnostic design matters.
 
+### Why there are TWO llama.cpp images
+
+This is the non-obvious bit. They are not redundant and neither can replace the other:
+
+- `Dockerfile.bonsai` builds **PrismML's `prism` fork**, which carries the Q2_0_g128 ternary
+  and hybrid-attention CUDA kernels `Ternary-Bonsai-27B` needs. Mainline does not have them.
+- `Dockerfile.llamacpp` builds **mainline at a pinned build tag**. The fork's branch head is
+  2026-07-31, so it predates any architecture merged after that — `Muse-Glimmer-30B` landed
+  in mainline on 2026-08-10 (`ggml-org/llama.cpp#26841`, build `b10353`) and fails on the
+  fork with an unknown-architecture error.
+
+Both images share `docker/gguf-serve.sh` as their entrypoint, so moving a model between them
+means changing only `image:` in the generated config. Pin the mainline tag rather than
+tracking a rolling one, for the same reason the vLLM image is pinned to `v0.26.0`. When
+adding a GGUF model, the question to answer first is *which image can actually load it* —
+check when its architecture was merged against the fork's branch date.
+
 ## Request lifecycle
 
 1. Client → `POST /v1/chat/completions` to llama-swap `:9292` with `"model": "<id>"`.

Dockerfile

@@ -21,5 +21,11 @@ RUN apt-get update \
 # models stays a git push (see .github/workflows/build-and-push.yml).
 COPY config.yaml /etc/llama-swap/config/config.yaml
 
+# On-call standby poller. Baked in for the same reason as config.yaml above — the
+# docker-compose `oncall-wakeup` service runs this image with an entrypoint override
+# rather than bind-mounting the script, so a Portainer Git stack can't mangle it.
+COPY scripts/oncall-wakeup.sh /usr/local/bin/oncall-wakeup.sh
+RUN chmod +x /usr/local/bin/oncall-wakeup.sh
+
 # Entrypoint/CMD are inherited from the base image; the compose file passes
 # --config and --listen.

Dockerfile.llamacpp

@@ -0,0 +1,69 @@
+# Mainline llama.cpp image (pinned build) for GGUF models the PrismML fork can't load.
+#
+# Why this exists ALONGSIDE Dockerfile.bonsai: the bonsai image builds PrismML's `prism`
+# fork, which carries the Q2_0_g128 ternary kernels Ternary-Bonsai-27B needs but that
+# mainline does not have. The fork's branch head is 2026-07-31, so it predates newer
+# architectures — Muse-Glimmer-30B landed in mainline on 2026-08-10 (ggml-org/llama.cpp
+# #26841) and needs build b10353+. Neither image can serve both models, hence two.
+#
+# Pin LLAMACPP_TAG to a build tag, never a rolling one — same policy as the vLLM image
+# (vllm/vllm-openai:v0.26.0). The official ghcr.io/ggml-org/llama.cpp:server-cuda package
+# only publishes a rolling tag for recent builds, which is why we compile instead.
+#
+# Built & pushed to GHCR by .github/workflows/llamacpp-image.yml.
+# NOTE: Dockerfile inline comments are only valid on their own line (never after an
+# instruction like ARG/ENV), so all comments here stand alone.
+
+# ---- build stage -----------------------------------------------------------
+FROM nvidia/cuda:12.6.2-devel-ubuntu22.04 AS build
+# RTX 3090 = sm_86 (this box's two LLM cards).
+ARG CUDA_ARCHS=86
+# b10362: 9 builds after Muse Glimmer support (b10353), to pick up immediate fixes.
+ARG LLAMACPP_TAG=b10362
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      git cmake ninja-build build-essential ca-certificates libcurl4-openssl-dev \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN git clone --depth 1 -b ${LLAMACPP_TAG} https://github.com/ggml-org/llama.cpp.git /src
+WORKDIR /src
+
+# The build box has no GPU/driver, so executables can't resolve CUDA driver-API
+# symbols (cuMemCreate, cuDeviceGet, ...). Link against the CUDA driver STUB the
+# devel image ships, exposed as libcuda.so.1 via LIBRARY_PATH. Build ONLY the
+# server target. Cap parallelism by RAM to avoid nvcc OOM on smaller runners.
+RUN export LIBRARY_PATH="/usr/local/cuda/lib64/stubs${LIBRARY_PATH:+:$LIBRARY_PATH}" \
+ && cmake -B build -G Ninja \
+      -DGGML_CUDA=ON \
+      -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCHS} \
+      -DCMAKE_BUILD_TYPE=Release \
+      -DLLAMA_BUILD_TESTS=OFF \
+      -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" \
+      -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" \
+ && MEM_GB=$(( $(awk '/MemTotal/{print $2}' /proc/meminfo) / 1048576 )) \
+ && JOBS=$([ "${MEM_GB:-0}" -lt 16 ] && echo 2 || echo 4) \
+ && echo "building llama.cpp ${LLAMACPP_TAG} with -j${JOBS} (host ~${MEM_GB} GB RAM)" \
+ && cmake --build build --target llama-server -j "${JOBS}"
+
+# ---- runtime stage ---------------------------------------------------------
+FROM nvidia/cuda:12.6.2-runtime-ubuntu22.04
+ENV DEBIAN_FRONTEND=noninteractive
+# hf CLI: gguf-serve.sh downloads weights with it (this build's llama-server has no
+# HTTPS). libcurl4/libgomp1: runtime deps of llama-server. curl: container health tooling.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      python3 python3-pip libgomp1 libcurl4 ca-certificates curl \
+ && pip3 install --no-cache-dir gguf "huggingface_hub[cli]" hf_xet \
+ && rm -rf /var/lib/apt/lists/*
+
+# Copy the whole build output dir so the versioned .so SONAME symlinks
+# (libggml-cuda.so -> .so.0 -> .so.0.13.1) are preserved for the runtime linker.
+COPY --from=build /src/build/bin/ /opt/llamacpp/bin/
+ENV PATH="/opt/llamacpp/bin:${PATH}"
+ENV LD_LIBRARY_PATH="/opt/llamacpp/bin"
+
+# Same entrypoint the bonsai image uses for its non-ternary GGUF models, so a model
+# moves between the two images by changing only the `image:` in config.yaml.
+COPY docker/gguf-serve.sh /usr/local/bin/gguf-serve.sh
+RUN chmod +x /usr/local/bin/gguf-serve.sh
+
+ENTRYPOINT ["/usr/local/bin/gguf-serve.sh"]

README.md

@@ -23,6 +23,10 @@ static GPU layout** — which is more reliable than dynamic VRAM packing for thi
 | `Dockerfile.bonsai` + `docker/bonsai-serve.sh` | PrismML llama.cpp fork image for Ternary-Bonsai-27B (built by CI → GHCR) |
 | `.github/workflows/build-and-push.yml` | CI: builds the Dockerfile and pushes the image to GHCR (Portainer can't build from a repo) |
 | `.github/workflows/bonsai-image.yml` | CI: builds the PrismML fork image → `ghcr.io/<owner>/bonsai-llama` |
+| `Dockerfile.llamacpp` | **Mainline** llama.cpp at a pinned build (`b10362`) — for models the PrismML fork is too old to load (built by CI → GHCR) |
+| `.github/workflows/llamacpp-image.yml` | CI: builds the mainline image → `ghcr.io/<owner>/llamacpp-mainline` |
+| `docker/gguf-serve.sh` | Shared GGUF entrypoint for **both** llama.cpp images (download-then-serve; vision / drafter / multi-GPU via `GGUF_*` env vars) |
+| `scripts/oncall-wakeup.sh` | On-call standby poller — keeps one model warm when the box is idle (see below) |
 | `docker-compose.yml` | Portainer stack definition (pulls the pre-built image) |
 | `config.yaml` | llama-swap model definitions — GENERATED by `gen_pairs_config.py` (do not hand-edit): standalone models + co-load pairs + solo big models |
 | `gen_pairs_config.py` | Generates `config.yaml`: `pairNN.<model>` co-load pairs (one model per 3090) + standalone `<model>` entries. Edit the `POOL`, run `python3 gen_pairs_config.py > config.yaml` |
@@ -198,6 +202,70 @@ repo builds the fork into its own image and points the model at it:
    `llama-server` with vision + DSpark speculative + tool calling, on one 3090. It speaks
    OpenAI `/v1` + `/health`, so llama-swap proxies it like any other model.
 
+## Muse-Glimmer-30B (mainline llama.cpp backend)
+
+A dense 30B vision-language model (Apache-2.0). It needs **mainline** llama.cpp `b10353+`
+— support landed 2026-08-10 — so it **cannot** use the bonsai image, whose `prism` fork
+branch is from 2026-07-31, ten days older than the architecture. That's why there are two
+llama.cpp images; see ARCHITECTURE → "Backend matrix".
+
+It's cheap on KV: 52 layers, `num_key_value_heads: 2`, and a 3:1 sliding/full attention
+split (39 sliding layers windowed at 2048, 13 full). The full layers cost 13 KiB/token and
+the sliding ones a flat 78 MiB, so the **entire native 131072 context costs under 2 GiB**.
+Weights, not context, are the constraint — which is why two entries exist:
+
+| Model ID | GPUs | Quant | Context | Notes |
+|---|---|---|---|---|
+| `muse-glimmer` | one 3090 | `17gb` + vision | 131072, 1 slot | the **on-call** model |
+| `Muse-Glimmer-30B-split` | both 3090s | `dynamic` + vision + drafter | 131072, 1 slot | best quality; owns both cards |
+
+Both run `-np 1` — the full native context in a single slot, with unquantized f16 KV. Neither
+is in `POOL`, so **neither generates `pairNN` co-load pairs**: the standby model is not a
+co-load partner, and pairing it would have added 10 pairs nothing would request. They live in
+`UNGROUPED_GGUF` in `gen_pairs_config.py`, and the pair count stays at 45.
+
+1. **Build the image** — the `llamacpp-image` workflow (`Dockerfile.llamacpp`) compiles
+   mainline at the pinned `LLAMACPP_TAG` and pushes `ghcr.io/<owner>/llamacpp-mainline:latest`.
+   Then **make that GHCR package public** (same as the other images).
+2. **Download the weights** on the host (~38 GB, so don't let the first cold start do it):
+
+   ```bash
+   HF_TOKEN=hf_... hf download meta-models/Muse-Glimmer-30B-GGUF \
+     muse-glimmer-30B-kquant-17gb.gguf muse-glimmer-30B-kquant-dynamic.gguf \
+     mmproj-kquant.gguf dflash-kquant.gguf \
+     --local-dir /models/hf-cache/gguf/meta-models_Muse-Glimmer-30B-GGUF
+   ```
+
+3. Request either model ID. `gguf-serve.sh` already passes `--jinja`, which the model
+   requires for its embedded chat template.
+
+The model supports `reasoning_strength` (`low`/`medium`/`high`/`xhigh`). It's left at the
+model's own default; to pin it, add `params=dict(reasoning_strength="low")` to the POOL
+entry in `gen_pairs_config.py` and regenerate — it emits a `filters.setParams` block.
+
+## On-call standby model
+
+`muse-glimmer` is kept resident whenever the box has stopped working, **without** ever
+blocking another model. Three parts:
+
+- Its standalone entry carries **`ttl: 0`** (never idle-unload). Its `pairNN.` members keep
+  the normal 5 h TTL.
+- Every pair/solo group is `exclusive: true`, so **a request to any other model evicts it
+  immediately**. Nothing has to unload it explicitly.
+- The `oncall-wakeup` compose service (`scripts/oncall-wakeup.sh`) polls `/metrics`; when
+  **both 3090s** sit below `IDLE_PCT` for `IDLE_SECONDS`, it sends a 1-token request, which
+  evicts whatever is squatting and loads `muse-glimmer`. It matches GPUs by **UUID**, not
+  index, so a reboot reordering the cards can't make it watch the A2000.
+
+> ⚠️ This is deliberately **not** `persistent: true` — llama-swap defines that as "other
+> groups can never unload this group's members", which would pin the cards and starve every
+> other model (same trap as the warning above).
+
+Tunable via the service's env vars (`ONCALL_MODEL`, `IDLE_SECONDS`, `IDLE_PCT`,
+`POLL_SECONDS`); set `IDLE_SECONDS` low to test, then put it back. **Consequence to accept:**
+with the poller running, an idle model gets replaced after ~1 h rather than surviving its
+full 5 h TTL. The TTLs still govern unloading; the poller governs replacement.
+
 ## Operational notes
 
 - **vLLM crash mitigation:** keep `--enforce-eager` on `Qwen3.6-35B-A3B` until the Xid 31

TODO.md

@@ -58,6 +58,47 @@ Status legend: `[ ]` todo · `[~]` in progress · `[x]` done
   unquantized for this): test SEPARATELY from prefix caching first — the combo has
   crashed during cudagraph profiling on hybrid Mamba models — then together.
 
+## Post-deploy verification (Muse-Glimmer-30B + on-call standby)
+
+Config-side checks already pass locally: 106 models / 45 groups, with all 104 pre-existing
+entries byte-identical and the two Muse-Glimmer entries purely additive. The rest needs the host.
+
+- [ ] Make the `llamacpp-mainline` GHCR package **public** (or add registry creds in Portainer).
+- [ ] Rebuild the **bonsai** image too — both images share `docker/gguf-serve.sh`, which
+  changed. Without it, `qwythos-v2` / `fablevibes` run an older script than config assumes.
+- [ ] Pre-download the ~38 GB of GGUFs (README → Muse-Glimmer) so the first cold start
+  isn't a multi-GB stall.
+- [ ] **Restart llama-swap** — config is read at startup only. `/v1/models` going 104 → 106
+  confirms the new config was actually picked up.
+- [ ] `muse-glimmer` cold start: watch `nvidia-smi` against the budget (16.76 weights +
+  1.40 mmproj + ~1.2 compute + ~1.82 KV ≈ 21.2 GiB, **~2.1 GiB spare**). This is the
+  thinnest number in the whole change. It assumes llama.cpp allocates SWA layers windowed,
+  not full — if it allocates full, KV jumps to ~6.5 GiB and it OOMs. Fallback ladder:
+  drop `ctx` 131072 → 65536, then `cache_type="q8_0"`.
+- [ ] **Coherence prompt** on first load — per the `qwen3.5-9b` precedent above, trust the
+  output, not a clean startup log.
+- [ ] **Vision probe**: send an image part and confirm a grounded description, proving
+  `--mmproj` actually attached rather than being silently ignored.
+- [ ] Context probe near 131k, plus an over-limit request (expect a clean 400).
+- [ ] **Stop tokens**: the model card warns never to stop on `<|eom|>` (only
+  `<|end_of_text|>` / `<|eot|>`). Confirm generations end cleanly and aren't truncated
+  mid-reasoning; if they are, add explicit EOG handling in `gguf-serve.sh`.
+- [ ] Confirm `-np 1` behaviour under load: a second concurrent request should QUEUE behind
+  the first (llama-server has one slot; `concurrencyLimit: 4` lets llama-swap admit 4), not
+  429 or share context. If queueing hurts in practice, that is the argument for raising
+  `par` — at the cost of KV, since each extra slot adds its own 2048 sliding window.
+- [ ] `Muse-Glimmer-30B-split`: confirm `-sm layer` spreads across both 3090s, and measure
+  decode speed — expect ~single-card, since layer-split is pipeline, not tensor, parallel.
+  If it's *slower* than the single-card entry, the split entry isn't earning its disk.
+- [ ] On-call: set `IDLE_SECONDS=120` on the `oncall-wakeup` service, confirm exactly one
+  wakeup fires and `/running` then shows `muse-glimmer`. Restore `3600`.
+- [ ] On-call **yield test**: with `muse-glimmer` resident, request `gemma-26b` and confirm
+  a clean swap. This is what `persistent: true` would have broken.
+- [ ] On-call **no-interrupt test**: start a long generation on another model, confirm the
+  poller doesn't fire mid-stream (GPU util stays above `IDLE_PCT`, resetting the counter).
+- [ ] Decide whether the ~1 h replacement delay is right in practice — it now supersedes
+  the 5 h TTL for *replacement* (TTLs still govern unloading).
+
 ## 1. Build the custom image
 
 - [ ] `Dockerfile` = `FROM ghcr.io/mostlygeek/llama-swap:unified-cuda` + `docker.io`.