feat: GPU llama.cpp builds via FREELLAMA_BACKEND, resumable pulls, and a configurable ready timeout

[johnpangalos]

Born from getting poolside's freshly released Laguna S 2.1 (118B MoE, 8.5B active) running through freellama on a 16 GB AMD RX 9070 XT + 32 GB RAM machine. Three gaps surfaced, each fixed here:

FREELLAMA_BACKEND — GPU release builds

pickAsset always chose the plain CPU build and actively penalized every GPU variant, so a GPU machine ran inference on CPU unless the user built llama.cpp themselves. FREELLAMA_BACKEND=vulkan (or cuda, rocm, sycl, ...) now selects the matching prebuilt asset from the same llama.cpp release. Variants install to bin/<tag>-<backend> so they don't collide with the CPU build of the same tag, and the default behavior is unchanged. Verified live: vulkan fetched llama-b10088-bin-ubuntu-vulkan-x64.tar.gz and the default still resolves the plain build, ignoring the -vulkan dir.

Resumable pulls

An interrupted download restarted from zero — painful at Laguna size (39.7 GB for unsloth's UD-Q2_K_XL). downloadGguf now keeps the .partial file and resumes it with an HTTP Range request; a server that ignores the range falls back to a full restart, and a 416 clears the stale partial. Verified live: a 7.7 GB partial resumed at 20% instead of restarting.

FREELLAMA_READY_TIMEOUT

The 180 s llama-server readiness cap was hardcoded and SIGKILLs a model that loads slower — realistic for tens-of-GB models. Now overridable in seconds via env, same default.

Tests cover backend asset selection, the resume path (206 + range-ignored restart), and timeout parsing. deno task check and deno task test (25/25) pass.

Laguna S 2.1 benchmark results on the RX 9070 XT to follow in a comment.

🤖 Generated with Claude Code

[johnpangalos]

Laguna S 2.1 benchmark results (RX 9070 XT 16 GB + 32 GB RAM, Vulkan b10088)

Model: unsloth/Laguna-S-2.1-GGUF:UD-Q2_K_XL (39.7 GB, 48 layers, 256 experts / 10 active). llama-bench sweep of --n-cpu-moe (expert layers kept on CPU), with -ngl 99 -fa 1 -t 16:

n_cpu_moe	pp64 t/s	tg32 t/s
48 (all experts on CPU)	6.5	1.6
38	12.0	6.6
34	24.6	9.7
32	15.4	10.3
30	17.6	9.5

Generation plateaus ~10 t/s at --n-cpu-moe 32 (16 expert layers in VRAM); pushing further starts saturating the 16 GB card. End-to-end through the freellama CLI:

FREELLAMA_BACKEND=vulkan \
FREELLAMA_READY_TIMEOUT=600 \
FREELLAMA_CTX=8192 \
FREELLAMA_SERVER_ARGS="-ngl 99 --n-cpu-moe 32 -fa on -t 16" \
freellama run unsloth/Laguna-S-2.1-GGUF:UD-Q2_K_XL

→ 9.9 t/s generation, ~7 t/s prompt eval, vs 2.3 t/s untuned (all experts on CPU). A 118B-param model at usable speed on a gaming PC.

Note: Laguna support landed in llama.cpp master today (ggml-org/llama.cpp#25165, first release b10088) — the official prebuilt binaries work; poolside's fork is only needed for DFlash speculative decoding, which hasn't been upstreamed yet.