Local coding benchmark: measure the ratings instead of leaving them unknown - #21
Conversation
Every row in catalog_ratings() reads `unknown`, and a quarter of the recommendation has been resting on a neutral 50 as a result. This is the deferred local-benchmark work the README promised. 61-rate-models.sh runs a fixed, versioned suite against the models this machine actually serves, at the quant they are actually served at, and writes the evidence to ~/llm-rating-<date>.txt. Four constraints, all deliberate: Nothing the model produces is executed. Grading generated code by running it means running text from a model on the developer's own machine, as their own user, for a score. Every task is graded by reading the response -- an exact answer, a tool_use block, a parse -- and the task set is written around that rather than pretending the constraint is not there. It measures agent-shaped competence, not SWE-bench: read a snippet, pick the right tool with the right arguments, obey an output format. Tool tasks carry double weight and all three tools are offered on every one of them, so a model that always calls the first tool fails two of three. A model that describes a tool call in prose fails all three, which is the failure that makes a local model useless to Claude Code and the one a text-only grader would otherwise score as a pass. It is comparable across models on one machine and nowhere else, which is the comparison the ranking needs and why the method is `local-benchmark`. The confidence ceiling is `medium`. A single pass is `low`; two clean repeats are `medium`; `high` is not reachable from twelve text-graded tasks at one quant on one machine. Three consequences elsewhere: - catalog_validate now branches the source rule on the method. A local benchmark cites `file:<artifact>.txt`, a basename under $HOME; there is no https address for a file on this machine and inventing one would be a fabrication in the one column whose job is to say where a number came from. vendor-benchmark still requires https. - score_confidence counts a rating only at `medium` or better, so a hurried single pass cannot raise the confidence of the ranking it feeds. - The script never edits the catalog. It prints rows to paste. A table curated by hand is not improved by a script that rewrites its own evidence base. A run where any task errored is written down but not offered as a row: a partial run is a report, not evidence. 69 new tests. The grading is pure, so all of it is tested without a server; the nine that drive the script end to end go through the curl mock. 447 tests across 13 suites pass, ordinary and network-isolated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rammsguns
left a comment
rammsguns
left a comment
There was a problem hiding this comment.
Two benchmark-validity gaps need to be closed before this becomes the evidence behind 25% of model scoring.
-
The suite says it measures output-format compliance, but the format tasks do not enforce their prompts.
format-oneworduses the general answer normalizer, which accepts preceding prose, fences, quotes, and punctuation (lib/rate.sh:209-227).format-jsonexplicitly accepts fences and extracts an object from surrounding prose (lib/rate.sh:230-247), although the prompt requires a bare JSON object.format-diffpasses any response containing a line that matches^@@, including prose around a diff. This makes the README and issue #20 claim ("obey an output format") stronger than the score actually measures. Add a strict grading kind or strict flags for these tasks, with negative tests for prose, fences, and extra text. -
The artifact does not record enough runtime identity to reproduce or compare the result. It records the served alias and sampling settings, but not the resolved GGUF filename/quant, llama.cpp revision, effective context, or relevant serving flags. The same alias can therefore produce the same catalog row from materially different runtimes. Capture the available model/runtime identity in the artifact (and clearly mark unavailable fields) before calling it evidence "at the quant they are actually served at."
The Linux CI is green. I also attempted the full suite locally under Git Bash, but that compatibility run did not complete within the bounded review window; this review is based on the code paths and their existing tests.
Two review findings on #21, both about whether the number this produces is evidence. 1. The format tasks did not enforce their prompts. `format-oneword` went through the lenient answer normaliser, which strips fences, quotes, punctuation and preceding prose -- so "Let me think.\n\nbash." passed a task whose entire subject is formatting. `format-json` accepted a fence and dug an object out of surrounding prose although the prompt forbids both. `format-diff` passed anything containing a line matching ^@@, preamble included. The suite claimed to measure output-format compliance and measured nothing of the sort. There are now two grading regimes, and the split is the point. `answer` and `tool` stay lenient: the question there is whether the model knows the answer or can call the tool, and grading the wrapping would make every task partly a formatting test. `oneword`, `json-only` and `diff-only` grade the WHOLE response, forgiving nothing but surrounding whitespace -- a transport detail, not a formatting choice. `diff-only` also requires every non-empty line to be diff syntax, not just the presence of a hunk header: a model that cannot suppress its preamble cannot be trusted to emit a patch a tool will apply. And it requires the diff to actually make the change, so a well-formed diff that replaces foo with the wrong string fails. A strict kind may not call rate_normalise_answer, and a test asserts that structurally rather than by convention, so the bug cannot return quietly. The lenient `match` and `json` kinds are gone -- no task used them once the format tasks became strict, and an unused grading kind is untested code in the one place where untested code decides whether a model passed. A test now checks both directions: every task's kind is implemented, and every implemented kind is used. 2. The artifact did not record enough to reproduce or compare a result. It had the served alias and the sampling settings, but an alias is not a runtime: it fronts whatever GGUF 30-models.sh last downloaded, built by whatever 20-build-llamacpp.sh last compiled, at whatever context 40-serve.sh last configured. Two materially different runtimes could produce the same catalog row. Each run now records the llama.cpp revision, the weights and their quant read off the file on disk, the per-model serving flags including CUDA_VISIBLE_DEVICES, and the live n_ctx from /props. The quant is on the RESULT line too, so ratings taken at different quants cannot be compared by accident. Every field is read from something that exists or is written `unavailable`; none is inferred from another. The live context is matched to the model being rated by its weights path rather than taken from the first upstream that answers -- with several models loaded that is some other model's context, and recording it would be worse than recording nothing. The new /props probe goes through a single low-level HTTP function, so curl still lives in exactly one place and every other function in the file stays testable without a server. The test that enforced that now checks it by function rather than by counting lines. Both the status of an HTTP call and the error text cross a command substitution, so both are returned in band -- the status as the first line of stdout, the error on stderr. A variable set in a subshell does not survive it. 90 tests in the suite now, 461 across 13, ordinary and network-isolated both green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both findings are addressed in 031f660. CI is green. 1. The format tasks now enforce their promptsYou were right that this was the more serious of the two: the suite advertised Lenient, and deliberately so — Strict — the whole response, forgiving nothing but surrounding whitespace:
Surrounding whitespace is the one exception, and only because a trailing Two things beyond what you asked for, both because the review made them
The regression is now blocked structurally rather than by convention: a test The lenient 2. The artifact records the runtimeEach run now records:
The quant is read off the file rather than from the catalog's preference: the The live context is matched to the model being rated by its weights path, Nothing is inferred from anything else, and a missing field is written The new Tests22 new in the rating suite (90 total), 468 across 13 suites. Ordinary and One correction to the commit message on that commit: it says 461 across 13 Note on the Git Bash runUnderstood that the Windows compatibility pass did not finish in the review |
|
PM assignment — proceed with the #21 integration update. Scope:
Acceptance: PR is cleanly mergeable against Product decision for #29: keep the 48 GB threshold based on the weight budget after KV reserve. Do not thread sticker VRAM through a fourth argument. The recommendation should reflect usable fit, so a nominal 48 GB card that leaves roughly 38 GB for weights should receive the hedged wording. |
One conflict, in README.md, and it was an insertion collision rather than a disagreement: this branch adds "Rating the models you serve" immediately before "## Configuration", and #25 adds "The llama-swap binary is pinned and verified" in the same place. Neither edits the other's text. Both sections are kept, ratings first. The ratings section continues the catalog thread the surrounding prose is already on -- it opens by answering "a quarter of the score is a neutral placeholder", which is the sentence two sections above it -- and the llama-swap section is about 40-serve.sh, so it reads as the last thing before Configuration rather than an interruption. Nothing was dropped from either side. Everything else merged clean, including the two places most likely to have collided: catalog_ratings() in lib/catalog.sh, where #27 added two Laguna rows carrying unknown ratings while this branch changes how ratings are produced, and tests/cases/catalog_test.sh.
rammsguns
left a comment
rammsguns
left a comment
There was a problem hiding this comment.
Integration review: the merge resolution is correct and both sides are preserved. The benchmark implementation files remain unchanged from 031f660, and the PR is cleanly mergeable against main.
One required documentation correction before merge:
- In
README.md, change “an argument for finishing the local benchmark” to “an argument for running the local benchmark.” Once this PR lands, the benchmark exists; the remaining action is to run it. This is a factual lifecycle correction, not a change to the preserved Laguna design.
After pushing that one-word fix, let CI rerun. No additional local full-suite run is required for a prose-only edit, but the final GitHub check must pass. Hold for PM approval after the corrected head is green.
The Laguna section closed by calling the unknown ratings an argument for finishing the local benchmark. It was, when that section was written and the benchmark did not exist. Merging this branch is what finishes it, so the sentence would have shipped stale in the same commit that made it so.
1 participant
Closes #20.
Fills in the deferred local-benchmark work:
61-rate-models.shandlib/rate.shmeasure the models this machine serves and print rows forcatalog_ratings(), so a quarter of the recommendation stops resting on aneutral 50.
The rationale, the suite, the grading rules and the open questions are all in
#20 and are not repeated here. What a reviewer most likely wants to disagree
with:
follows from that. It is the reason there is no "write this function and
let's run it" task, and the reason the suite measures agent-shaped
competence rather than code generation.
one of them.
score_confidencenow counts a rating only atmediumor better. Thischanges an existing published rule, deliberately: without it, a single
unrepeated pass would lift the ranking's confidence.
catalog_validate's source rule now branches on the method.local-benchmarkcitesfile:<artifact>.txt;vendor-benchmarkstillrequires https.
Diff
lib/rate.shrate_callis the only function that touches the network.61-rate-models.shtests/cases/rating_test.shlib/catalog.shlib/score.shmediumor better.tests/cases/catalog_test.shREADME.mdVerification
447 tests across 13 suites, all passing, ordinary and network-isolated.
No end-to-end run against a real served model is included, because this branch
was developed on a host with no model loaded. The nine tests that drive
61-rate-models.shend to end exercise the whole path through the curl mock —endpoint resolution, the suite, grading, the artifact and the pasteable row —
but the first real run is still the first real run.
🤖 Generated with Claude Code