Skip to content

Catalog Laguna XS 2.1 and S 2.1 - #27

Merged
rammsguns merged 1 commit into
mainfrom
laguna-catalog
Aug 12, 2026
Merged

Catalog Laguna XS 2.1 and S 2.1#27
rammsguns merged 1 commit into
mainfrom
laguna-catalog

Conversation

@rammsguns

Copy link
Copy Markdown
Owner

Closes #26.

Adds laguna-xs-2.1 and laguna-s-2.1 to the catalog with hf-api provenance dated 2026-08-12.

What was verified, and how

Every figure is one curl away from the fact_source in the row:

XS 2.1 S 2.1
createdAt 2026-06-20 2026-07-13
safetensors total 33,442,617,088 117,561,977,600
max_position_embeddings 262,144 1,048,576
experts / top-k 256 / 8 256 / 10
licence openmdw-1.1 openmdw-1.1

tools and reasoning come from the chat template, which carries tool_call, tools, reasoning_content and enable_thinking.

The estimator, checked against real bytes

This is the part I'd most like reviewed, because it is the number every fit decision runs on:

Row Quant Estimated Measured Δ
laguna-xs-2.1 Q4_K_M 20,165 MB 20,274 MB 0.5%
laguna-s-2.1 UD-Q4_K_XL 73,353 MB 73,395 MB 0.06%

Measured = summed GGUF file bytes from the published repos on 2026-08-12. A test holds both to 3%.

Behaviour change worth a decision

Laguna XS 2.1 becomes the top medium pick on this machine, ahead of qwen3-coder-30b (81 vs 73), and Laguna S 2.1 becomes the top large pick. Nothing was weighted to make that happen. With every coding rating still unknown the quality term does no discriminating work, so the ranking runs on freshness, hardware fit, speed and features — and a June 2026 MoE with a 262k context wins all four against a July 2025 one.

That is the existing design behaving as designed. It is also an argument for finishing the local benchmark rather than for hand-weighting the table. Say so if you'd rather the new rows not be default picks and I'll change it.

Three judgement calls, flag any of them

  1. CATALOG_MAX_ROWS 15 → 17 rather than retiring two rows. The comment now carries the date and reason, and says that past ~20 the answer is to retire instead. Retiring phi-4 and qwen3-1.7b was the alternative; that is a product call I didn't want to make silently.
  2. Ratings stay unknown despite Poolside publishing 70.9% SWE-bench Verified for XS. Reasoning in the table comment. This is the first time the rule has cost us a real number.
  3. SELECT_MAX_LISTED raised to match. Left at 15, the two lowest-scoring rows would have vanished from the menu with no message. A test now enforces that the caps agree.

Not done

Neither model has been downloaded or run here — that is 19 GB and 68 GB respectively, and the rows make no claim that depends on it. The architecture reached mainline llama.cpp in ggml-org/llama.cpp#25165 (merged 2026-07-22) and the binary on this machine is newer, so support is present; that is a version check, not a load test.

Tests

388 pass, 11 new (8 catalog, 1 score, 2 selector). Both runners green, including tests/isolated.sh.

Two existing score tests were changed rather than added to. They asserted a fixed list of three size classes, which was really an undocumented assertion that no catalogued model is unrunnable at 20 GB with zero offload. A 117.6B row makes that false, and it surfaced as a failure in the grouping tests rather than where the assumption lived. Both now assert the invariant score_rank documents — each class once, contiguous, unsupported last.

Two rows for Poolside's Laguna 2.1 family, the first models in the
catalog published after 2025-07-31. Each of them exercises part of the
schema that no existing row did.

Attribution. Unsloth mirrors S 2.1 as GGUF and is where most people meet
the model, so `unsloth/...` reads as the obvious repo. It is not:
canonical_repo names the original publisher, and Unsloth does not mirror
XS 2.1 at all, so the mirror is not even a consistent answer. Both rows
name Poolside, and a test rejects a fact_source pointing at the mirror.

Active parameters. The names say A3B and A8B; config.json says 2.7B and
7.8B. The speed score is built on the active count, so a row that copies
the name claims the model is slower than it is. The formula is written
above catalog_rows() and both figures reproduce the safetensors total to
within 1.5%.

UD quants. Unsloth's dynamic quants allocate bits per tensor and are not
the plain quant they are named after -- UD-Q2_K_XL is 2.70 bpw where
plain Q2_K is 3.35. Reusing the plain figure would misestimate Laguna S
by roughly 75 GB, in the direction that says it fits. The four values
this repo references are measured from the published files; an unlisted
UD quant still returns status 1, so catalog_est_size_mb fails loudly
rather than sizing a 70 GB download off an invented number.

Both estimates now agree with reality: 20,165 MB estimated against
20,274 measured for XS at Q4_K_M, 73,353 against 73,395 for S at
UD-Q4_K_XL. A test holds them to 3%.

XS 2.1 has no quant fallback because Poolside publishes exactly two
GGUFs of it. An invented IQ4_XS alternative would make select_quant_file
match some other file instead of failing.

Ratings stay unknown, and that is the interesting case rather than the
default one: Poolside ships .eval_results/swe-bench_verified.yaml
claiming 70.9% for XS 2.1. No other row here has a SWE-bench figure, so
recording it would rank "published a number" against "did not" while
looking like a quality judgement.

CATALOG_MAX_ROWS 15 -> 17, with the date and reason in the comment.
SELECT_MAX_LISTED moves with it, because selector_build drops everything
past its cap silently -- a catalogued model that never reaches the menu
is invisible in the one place a user would look. A test now enforces
that the two agree.

Two score tests were asserting a fixed list of three size classes, which
was really an assertion that no catalogued model is unrunnable at 20 GB
with no offload. That stopped being true, and it failed in the grouping
tests rather than where the assumption lived. Both now assert the
documented invariant -- each class once, contiguous, unsupported last --
so they hold whatever the catalog contains.

Closes #26

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@rammsguns rammsguns left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PM review: product decisions accepted.

  • Keep CATALOG_MAX_ROWS and SELECT_MAX_LISTED at 17. Adding the two current, relevant rows is preferable to silently retiring useful coverage; revisit retirement before the catalog exceeds roughly 20 rows.
  • Keep Laguna XS 2.1 and S 2.1 eligible for normal selection. Do not suppress or hand-penalize XS merely because metadata currently makes it the top medium pick. The unfinished local benchmark is the correct place to add comparable quality evidence.
  • Keep both ratings unknown; the isolated vendor SWE-bench disclosure is not comparable with the rest of this catalog.
  • No model download or runtime test is required for this issue. The PR clearly distinguishes sourced configuration/GGUF-byte facts from measurements not performed locally.

The acceptance criteria are covered and CI is green. One wording nit, not a blocker: the selector regression test enforces SELECT_MAX_LISTED >= CATALOG_MAX_ROWS, which is the safety invariant, rather than strict equality as the PR summary says.

Ready to merge from the PM side. Merge this before the TUNING.md queue because it does not participate in that conflict chain.

@rammsguns
rammsguns merged commit aa069c9 into main Aug 12, 2026
1 check passed
@rammsguns
rammsguns deleted the laguna-catalog branch August 12, 2026 21:35
rammsguns added a commit that referenced this pull request Aug 13, 2026
One conflict, in README.md, and it was an insertion collision rather than a
disagreement: this branch adds "Rating the models you serve" immediately
before "## Configuration", and #25 adds "The llama-swap binary is pinned and
verified" in the same place. Neither edits the other's text.

Both sections are kept, ratings first. The ratings section continues the
catalog thread the surrounding prose is already on -- it opens by answering
"a quarter of the score is a neutral placeholder", which is the sentence two
sections above it -- and the llama-swap section is about 40-serve.sh, so it
reads as the last thing before Configuration rather than an interruption.
Nothing was dropped from either side.

Everything else merged clean, including the two places most likely to have
collided: catalog_ratings() in lib/catalog.sh, where #27 added two Laguna
rows carrying unknown ratings while this branch changes how ratings are
produced, and tests/cases/catalog_test.sh.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Catalog Poolside Laguna XS 2.1 and S 2.1

1 participant