Muse-Glimmer-30B + mainline llama.cpp image → main - #9
Merged
Merged
Conversation
Muse-Glimmer-30B-GGUF is a dense 30B VLM whose architecture landed in mainline llama.cpp on 2026-08-10 (ggml-org/llama.cpp#26841, build b10353). The bonsai image builds PrismML's `prism` fork, whose branch head is 2026-07-31 — ten days older than the architecture — so it cannot load this model at all. Hence a second llama.cpp image rather than a new POOL entry. The two images are not redundant and neither replaces the other: the fork carries the Q2_0_g128 ternary kernels Ternary-Bonsai needs and mainline lacks; mainline carries every architecture merged after the fork diverged. Both share docker/gguf-serve.sh as their entrypoint, so a model moves between them by changing only `image:`. Two entries, because KV turned out to be nearly free on this model (52 layers, num_key_value_heads=2, and a 3:1 sliding/full attention split) — the full native 131072 context costs ~1.82 GiB, so weights, not context, are the constraint: muse-glimmer one 3090, 17gb+vision, 131072 in 1 slot, f16 KV Muse-Glimmer-30B-split both 3090s, dynamic+vision+draft, 131072 in 1 slot, f16 KV Both are ungrouped (new UNGROUPED_GGUF list), so neither generates pairNN co-load pairs: the standby model is not a co-load partner. config.yaml is purely additive — 104 -> 106 models with all pre-existing entries byte-identical and groups unchanged at 45. muse-glimmer is also the on-call standby: ttl 0 (never idle-unload) plus a poller that wakes it once both 3090s have been quiet for an hour. It deliberately does NOT use `persistent: true` — llama-swap defines that as "other groups can never unload this group's members", which would pin the cards and starve every other model. Eviction is already free via the existing exclusive groups, and the wakeup request itself displaces whatever is squatting, so nothing calls the unload API. Also fixes a pre-existing gap: bonsai-image.yml did not trigger on docker/gguf-serve.sh even though Dockerfile.bonsai COPYs it, so a change there would have rebuilt only the mainline image and left the bonsai one silently stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add Muse-Glimmer-30B on a pinned mainline llama.cpp image
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the Muse-Glimmer work (#8) to
main. #7 mergedlonger-ttlintomainbefore #8 landed onlonger-ttl, so these two commits never reached the default branch — which means neither image workflow has run yet (both trigger only onmain).Contents:
bf6400d+ the #8 merge. Full rationale is in #8; the short version:Dockerfile.llamacpp— mainline llama.cpp pinned atb10362. The bonsai image'sprismfork (branch head 2026-07-31) predates the Muse Glimmer architecture (mainline 2026-08-10,b10353) and cannot load it.-np 1at the full 131072 context with f16 KV. NopairNNpairs.muse-glimmeras on-call standby:ttl: 0+ a wakeup poller, deliberately notpersistent: true.bonsai-image.ymlnot triggering ondocker/gguf-serve.sh.config.yamlis purely additive: 104 → 106 models, all pre-existing entries byte-identical, groups unchanged at 45.What happens on merge
llamacpp-imagebuilds for the first time → createsghcr.io/tkontu/llamacpp-mainline. It's a CUDA compile (120 min timeout).bonsai-imagerebuilds too, via the trigger fix — required, sincegguf-serve.shchanged andDockerfile.bonsaicopies it.build-and-pushrebuilds the main image (config.yaml+ the baked-in poller script changed).Still needed afterwards
llamacpp-mainlinepackage public — it will be private on first push./v1/modelsgoing 104 → 106 confirms pickup.TODO.md. Thinnest number ismuse-glimmer's ~2.1 GiB headroom.🤖 Generated with Claude Code