Add Muse-Glimmer-30B on a pinned mainline llama.cpp image

[TKontu]

Why a second llama.cpp image

meta-models/Muse-Glimmer-30B-GGUF is a dense 30B Apache-2.0 VLM. Its architecture landed in mainline llama.cpp on 2026-08-10 (#26841, build b10353). The bonsai image builds PrismML's prism fork, whose branch head is 2026-07-31 — ten days older than the architecture — so it cannot load this model at all.

The two images are not redundant and neither replaces the other:

• Dockerfile.bonsai — the fork's Q2_0_g128 ternary + hybrid-attention kernels, which Ternary-Bonsai-27B needs and mainline lacks.
• Dockerfile.llamacpp (new) — mainline pinned at b10362, carrying every architecture merged after the fork diverged.

Both share docker/gguf-serve.sh as their entrypoint, so a model moves between them by changing only image:. Pinned to a build tag rather than a rolling one, same policy as vllm/vllm-openai:v0.26.0.

Sizing

KV turned out to be unusually cheap here — 52 layers, num_key_value_heads: 2, head_dim: 128, and a 3:1 sliding/full attention split (39 sliding layers windowed at 2048, 13 full). Full layers cost 13 KiB/token, sliding layers a flat 78 MiB, so the entire native 131072 context costs ~1.82 GiB. Weights, not context, are the constraint:

Model ID	GPUs	Files	Context
muse-glimmer	one 3090	17gb + mmproj	131072, 1 slot, f16 KV
Muse-Glimmer-30B-split	both 3090s, -sm layer	dynamic + mmproj + dflash	131072, 1 slot, f16 KV

Both run -np 1, which is also marginally cheaper than parallel slots: llama.cpp gives each slot its own sliding window, so parallelism multiplies the 78 MiB while leaving the full-layer cost fixed.

Neither is in POOL (new UNGROUPED_GGUF list), so neither generates pairNN pairs — the standby model is not a co-load partner.

On-call standby

muse-glimmer stays resident when the box is idle, without ever blocking another model:

• ttl: 0 — never idle-unload.
• Existing exclusive: true groups already evict it on any other request. Free, no change.
• New oncall-wakeup compose service wakes it once both 3090s sit under 5% for an hour, matching GPUs by UUID (indices reorder across reboots; the A2000 must never count).

Deliberately not persistent: true — llama-swap defines that as "other groups can never unload this group's members", which would pin the cards and starve everything else. The wakeup request itself displaces squatters, so nothing calls the unload API.

Drive-by fix

bonsai-image.yml did not trigger on docker/gguf-serve.sh despite Dockerfile.bonsai copying it — a change there would have rebuilt only the mainline image and left the bonsai one silently stale.

Verification

Config-side, checked locally:

• config.yaml is purely additive: 104 → 106 models, all 104 pre-existing entries byte-identical, groups unchanged at 45.
• Generator is idempotent; all YAML parses; both shell scripts pass sh -n.
• With no optional vars set, gguf-serve.sh emits a byte-identical command line, so qwythos-v2 / fablevibes are unaffected.
• The awk metric parser was tested against the live /metrics on v247 and correctly ignores the A2000.

Everything requiring GPUs is unverified and listed in TODO.md. The thinnest number is muse-glimmer's ~2.1 GiB headroom, which assumes llama.cpp allocates the sliding-window layers windowed rather than full; if it allocates full, KV jumps to ~6.5 GiB and it OOMs. Fallback ladder is documented.

Before this works on the host

1. Merge — the llamacpp-mainline GHCR package does not exist yet (workflows only run on main), then make it public.
2. The bonsai image rebuilds automatically now, thanks to the trigger fix.
3. Pre-download ~38 GB of GGUFs.
4. Restart llama-swap — config is read at startup only; /v1/models going 104 → 106 confirms pickup.

🤖 Generated with Claude Code