Add Laguna 2.1 models (XS + S) and DFlash vLLM serving mode to llm template

[EBro912]

Stacked on #76 (retarget to main once it merges). Supersedes #79, which will be closed as a duplicate once this merges.

What

Three additions to the python/llm template, aimed at Wendy on NVIDIA hardware:

1. Laguna 2.1 models in the picker — Poolside's open-weight agentic-coding MoEs, both pinned to the registry's q4_K_M tag so a scaffolded project gets a known quantisation (the other published tags are too large for the target devices or MLX-only):

Option	Total / active params	Context	Download	Device class
laguna-xs-2.1:q4_K_M	33B / 3B	256K	20 GB	AGX Orin 32GB or larger
laguna-s-2.1:q4_K_M	118B / 8B	256K	96 GB	Thor 128GB class only

#76 already made Laguna run on Thor (stock Ollama image for the laguna architecture, 20m load timeout); measured ~20.6 tok/s warm for the S model on the current 96GB blob. The per-option comments say explicitly which devices don't fit, so the picker never offers a model the device can't load.

2. DFlash speculative decoding for Laguna S — DFlash (lossless block-diffusion drafting, ~2-4x generation speedup) has no Ollama support; its backends are vLLM/SGLang/Transformers/MLX. So the laguna-s-2.1-dflash picker option swaps the backend at scaffold time: Go-template conditionals in docker-compose.yml/wendy.json replace the ollama service with a vllm one serving poolside/Laguna-S-2.1-NVFP4 + its paired 1B DFlash draft behind an OpenAI-compatible API, with Open WebUI pointed at that. Every other pick renders the exact same topology as today.

Key decisions

• Sentinel picker value (no colon) so it can never be mistaken for an Ollama pull tag.
• Single multi-arch NGC vLLM image (nvcr.io/nvidia/vllm:26.06-py3), no WENDY_GPU_VENDOR/JETPACK build-arg matrix — this path only targets NVIDIA 128GB-class hardware. Note the JetPack CUDA constraint documented in the README: NGC 26.06+ needs CUDA 13.3+, so on JetPack 7.2 the DFlash pick is blocked until a matching JetPack/NGC pairing exists.
• gpu-memory-utilization=0.70: vLLM's 0.9 default assumes discrete VRAM and would reserve ~110GB of Thor's 122.8GiB unified memory, starving the OS and WebUI.
• BF16 draft (~2GB) rather than NVFP4, keeping NVFP4-kernel risk off the draft path; swappable via DFLASH_DRAFT_MODEL.
• Probe-and-fallback: DFlash support is still landing in upstream vLLM (Add Laguna XS.2.1 DFlash drafter support vllm-project/vllm#46853). The entrypoint greps the installed package and, if support is missing (or a DFlash-enabled launch dies fast), demotes to plain vLLM serving with a loud WARNING — the app works either way, and enabling DFlash later is a one-line image-tag bump. DFLASH_DISABLE=1 forces plain serving.
• Registry-verified metadata (diverges from Add Laguna 2.1 models to the llm template picker #79 where the registry disagrees): both q4_K_M tags have a 256K context window — the 1M figure belongs to other laguna-s-2.1 tags, not the pinned one — and the S blob is 96GB since its upstream re-publish (was 75GB).

Verification

• Rendered docker-compose.yml + wendy.json through Go text/template for gemma4:e2b, laguna-xs-2.1:q4_K_M, laguna-s-2.1:q4_K_M, and laguna-s-2.1-dflash: all pass docker compose config and JSON validation; the dflash render has no ollama service/volume, all Ollama picks are byte-equivalent to today's output modulo the model value.
• bash -n on both entrypoints; template.schema.json validates.
• Tag sizes/context windows confirmed against ollama.com for both Laguna models; all three poolside HF repos confirmed public (no HF_TOKEN needed).
• On-device (Jetson AGX Thor, WendyOS 0.18.1): laguna-s-2.1:q4_K_M via Ollama loads in ~26s warm-cache and generates ~20.6 tok/s. The DFlash/vLLM path is ecosystem-blocked on JetPack 7.2 (CUDA 13.2 vs NGC 26.06's CUDA 13.3 torch; see README) — verified the probe-and-fallback logic and documented the constraint instead.
• Not yet done: on-device run of laguna-xs-2.1 on an AGX Orin; dflash end-to-end on CUDA 13.3+ hardware (DGX Spark).

🤖 Generated with Claude Code

https://claude.ai/code/session_01QFzB4dyWDCpxkmY78oEyRy

[martien-wdy]

Reviewed the picker changes and the new serving path. The model metadata here is more accurate than in my own pull request 79. I checked the registry and every tag for the larger Laguna model reports 256K context, not the 1M I had written, so this supersedes mine correctly. My own measurement on Thor came out at 20.73 tokens per second against the roughly 20.6 noted here, which is close agreement from two independent runs.

The template conditional approach looks clean. Verifying that every non DFlash choice renders byte equivalent output is exactly the right property to check when adding conditionals to a shared template, and giving each backend its own volume avoids stale data collisions. The memory limit reasoning for unified memory against the usual discrete card default is a genuinely non obvious trap, caught correctly.

One issue is worth blocking on. On the current JetPack the new backend cannot fail cleanly, it retries forever after a large download while the platform still reports the app as healthy. Details are on the line, along with two smaller robustness points in the same file.

Since this path cannot work on any hardware the team currently has, I would also gently question shipping it as an ordinary picker choice rather than landing the plumbing behind a clearly experimental label. That is a product call, not a code one.

[martien-wdy]

Benchmarked both Laguna picks from this PR on a Jetson AGX Thor, so the XS entry now has an on device number too. This covers the "not yet done" item for XS, though on Thor rather than an AGX Orin.

Setup

wendystudio-walter, 128 GB, sm_110, CUDA 13.2, Ollama 0.32.5, all other containers stopped. 200 decoded tokens, num_ctx 2048, temperature 0, seed 42, one discarded warm-up, median of five runs. Figures come from Ollama's own eval_count / eval_duration, measured in-process on the device.

Results

Model	Total / active params	Decode tok/s	Prefill tok/s	GPU resident	stdev
qwen2.5:3b (reference)	3B dense	62.80	3357	2.08 GB	0.10
laguna-xs-2.1:q4_K_M	33B / 3B	58.83	1249	20.35 GB	0.18
laguna-s-2.1:q4_K_M	118B / 8B	20.73	562	95.99 GB	0.02

100% GPU residency verified for both Laguna runs. The S figure agrees with the ~20.6 tok/s in the PR description, from an independent run.

Worth knowing for the picker text

XS is remarkably strong on Thor. It decodes within 7% of a 3B dense model (58.83 against 62.80) while carrying 11x the total parameters, because it activates about the same 3B per token. Together these three rows show decode throughput tracking active parameters while prefill tracks total ones (3357, 1249, 562). On Thor-class hardware XS arguably deserves a recommendation over S for most uses: near-identical decode speed to a tiny model, at a fifth of the download and a fifth of the memory.

Download and load costs, measured. S reported exactly 96.03 GB, confirming the figure in the picker; it took 68 minutes at ~23 MB/s over wireless. Cold load was 45 seconds, comfortably inside the existing 20 minute timeout, though helped by a page cache warm from the download.

One suggestion for both Ollama entries. Pinning num_ctx matters more here than for any other model in the picker. Left unpinned, Ollama sizes the key/value cache from free memory, and at 96 GB of weights on a 131.9 GB device that can tip into a partial CPU offload, which produces a plausible-looking but wrong throughput number. Might be worth a line in the README so people capacity-planning these models do not get caught by it.

[EBro912]

All six inline threads are addressed in ca09d07; replies with the specifics are on each thread. Summary of what changed beyond the line fixes:

Benchmarks folded in. The XS on-Thor number (~59 tok/s) is now in the picker comment and the README's Thor row, the active-vs-total parameter effect and the prefer-XS-on-Thor guidance are in the README's Laguna section, and the num_ctx pinning advisory is there too, framed exactly as a capacity-planning trap (unpinned KV sizing can tip 96GB of weights into partial CPU offload with plausible-looking throughput). The S decode figures now have two independent runs within 1% of each other (20.6 / 20.73), which is as settled as that number needs to be.

On the experimental question: implemented as a label — the picker option now reads laguna-s-2.1 (DFlash • vLLM, experimental) and its comment leads with "Experimental:". The plumbing ships, nothing hides behind a flag, but nobody picks it thinking it's a supported path on current hardware. If the team would rather gate it harder (or drop the label once a CUDA 13.3 JetPack lands), that's a one-line schema change either way — flagging it for Ethan as the final call.

One benchmark follow-up someone with an AGX Orin could close out: XS's numbers above are from Thor; the 32GB Orin class it's actually aimed at is still unmeasured.