Skip to content

Local coding benchmark: measure the ratings instead of leaving them unknown #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rammsguns merged 4 commits into from
Aug 13, 2026
Merged

Local coding benchmark: measure the ratings instead of leaving them unknown #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f8eb114
Measure coding ratings locally, instead of leaving them unknown
rammsguns Aug 12, 2026
031f660
Grade the format tasks strictly, and record what was running
rammsguns Aug 12, 2026
8592d8e
Merge main into local-ratings after #24, #25, #27 and #29 landed
rammsguns Aug 13, 2026
907c441
Say "running" the benchmark, now that it exists
rammsguns Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 201 additions & 0 deletions 61-rate-models.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
#!/usr/bin/env bash
# Rate the models this machine actually serves, on the tasks a coding agent
# actually performs, and write the evidence to a file.
#
# This is what turns catalog_ratings() from a column of `unknown` into
# something with a method, a date and an artifact behind it. It does NOT edit
# the catalog: the table is curated by hand, and a script that rewrites its own
# evidence base is not evidence. It prints the rows to paste, and prints the
# artifact they cite.
#
# Read lib/rate.sh before trusting the number. In particular: nothing the model
# produces is executed, the suite is twelve text-graded tasks, and the
# confidence ceiling is `medium` on purpose.
#
# Usage:
# ./61-rate-models.sh # every served model, one pass each
# ./61-rate-models.sh --repeats 3 # three passes; disagreement -> low
# ./61-rate-models.sh --model qwen3-4b # one served model, by name
# ./61-rate-models.sh --dry-run # print the plan, call nothing
#
# Output -> ~/llm-rating-<date>.txt
set -uo pipefail
RIG_SRC_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$RIG_SRC_DIR/lib/detect.sh"
source "$RIG_SRC_DIR/lib/catalog.sh"
source "$RIG_SRC_DIR/lib/preflight.sh"
source "$RIG_SRC_DIR/lib/rate.sh"

ONLY=""
DRY=0

while (( $# )); do
case "$1" in
--repeats) shift; RATE_REPEATS="${1:-1}" ;;
--repeats=*) RATE_REPEATS="${1#--repeats=}" ;;
--model) shift; ONLY="${1:-}" ;;
--model=*) ONLY="${1#--model=}" ;;
--dry-run) DRY=1 ;;
-h|--help) sed -n '2,21p' "${BASH_SOURCE[0]}"; exit 0 ;;
*) die "unknown argument: $1" ;;
esac
shift
done

[[ "$RATE_REPEATS" =~ ^[1-9][0-9]*$ ]] || die "--repeats must be a positive integer, got '$RATE_REPEATS'"
command -v jq >/dev/null 2>&1 || die "jq is required: sudo apt-get install -y jq"

STAMP="$(date +%Y%m%d-%H%M)"
TODAY="$(date +%F)"
ARTIFACT="$HOME/llm-rating-$STAMP.txt"
ERRFILE="$(mktemp)"
trap 'rm -f "$ERRFILE"' EXIT

BASE="$(preflight_endpoint)"
c_info "Rating against $BASE"

AVAILABLE="$(preflight_check_models "$BASE")" || die \
"no models served at $BASE -- start the stack first (./40-serve.sh), then re-run"

if [[ -n "$ONLY" ]]; then
printf '%s\n' "$AVAILABLE" | grep -qxF "$ONLY" \
|| die "$BASE does not serve '$ONLY'. It serves: $(printf '%s' "$AVAILABLE" | tr '\n' ' ')"
AVAILABLE="$ONLY"
fi

TOTAL_W="$(rate_total_weight)"
NTASKS="$(rate_task_count)"

c_info "suite v$RATE_SUITE_VERSION -- $NTASKS tasks, total weight $TOTAL_W, $RATE_REPEATS repeat(s)"

if (( DRY )); then
printf '%s\n' "$AVAILABLE" | while IFS= read -r served; do
[[ -n "$served" ]] || continue
if id="$(rate_catalog_id "$served")"; then
printf ' %-40s -> %s\n' "$served" "$id"
else
printf ' %-40s -> (not in the catalog; will be measured, not recorded)\n' "$served"
fi
done
exit 0
fi

# --- run --------------------------------------------------------------------

CFG="${LLAMA_SWAP_CFG:-$RIG_DIR/etc/llama-swap.yaml}"

{
printf 'llm-rig local coding rating %s\n' "$STAMP"
printf 'suite: v%s (%s tasks, total weight %s)\n' "$RATE_SUITE_VERSION" "$NTASKS" "$TOTAL_W"
printf 'endpoint: %s\n' "$BASE"
printf 'sampling: temperature=%s seed=%s max_tokens=%s repeats=%s\n' \
"$RATE_TEMPERATURE" "$RATE_SEED" "$RATE_MAX_TOKENS" "$RATE_REPEATS"
printf 'grading: text only. No model output is executed.\n'
# Runtime identity. A rating is only reproducible if you know what was
# running: the served alias does not say which quant, which llama.cpp build
# or which context produced it. Anything that cannot be read is recorded as
# `unavailable` rather than guessed.
printf 'llama.cpp revision: %s\n' "$(rate_llamacpp_rev "$RIG_DIR")"
printf 'llama-swap config: %s\n' "$( [[ -f "$CFG" ]] && printf '%s' "$CFG" || printf '%s' "$RATE_UNAVAILABLE" )"
printf '\n'
} > "$ARTIFACT"

ROWS=""

while IFS= read -r served; do
[[ -n "$served" ]] || continue

cid=""
if ! cid="$(rate_catalog_id "$served")"; then
c_warn "$served is not in the catalog -- measuring it, but no rating row can be written"
cid=""
fi

# Read before the suite runs, except the live context: that needs the model
# loaded, which the first task does.
gguf="$(rate_swap_gguf "$CFG" "$served")"
quant="$(rate_quant_of "$gguf")"
flags="$(rate_swap_flags "$CFG" "$served")"

c_info "$served${cid:+ (catalog: $cid)} quant=$quant"
{
printf 'model: %s\n' "$served"
printf 'catalog-id: %s\n' "${cid:-none}"
printf ' weights: %s\n' "$( [[ "$gguf" == "$RATE_UNAVAILABLE" ]] && printf '%s' "$gguf" || printf '%s' "${gguf##*/}" )"
printf ' quant: %s\n' "$quant"
printf ' serving flags: %s\n' "$flags"
} >> "$ARTIFACT"

got_w=0; answered=0; flips=0

while IFS= read -r task; do
[[ -n "$task" ]] || continue
tid="${task%%;*}"
weight="$(rate_task_get "$tid" weight)"

# Each repeat is graded on its own; the task counts as passed only if every
# repeat passed, and a disagreement is recorded rather than averaged away.
passes=0; errors=0
for (( r = 1; r <= RATE_REPEATS; r++ )); do
# The reason goes through a file rather than RATE_LAST_ERROR: the
# response arrives on stdout, so the call is a command substitution, and
# a variable set inside one does not survive it.
if resp="$(rate_call "$BASE" "$served" "$tid" 2>"$ERRFILE")"; then
if rate_grade "$tid" "$resp"; then
passes=$(( passes + 1 ))
fi
else
errors=$(( errors + 1 ))
printf ' task %-20s error: %s\n' "$tid" \
"$(head -1 "$ERRFILE" 2>/dev/null || printf 'unknown')" >> "$ARTIFACT"
fi
done

if (( errors )); then
verdict="error"
elif (( passes == RATE_REPEATS )); then
verdict="pass"; got_w=$(( got_w + weight )); answered=$(( answered + 1 ))
elif (( passes == 0 )); then
verdict="fail"; answered=$(( answered + 1 ))
else
verdict="unstable"; answered=$(( answered + 1 )); flips=$(( flips + 1 ))
fi

printf ' task %-20s kind=%-6s weight=%s %s (%s/%s)\n' \
"$tid" "$(rate_task_get "$tid" kind)" "$weight" "$verdict" "$passes" "$RATE_REPEATS" \
>> "$ARTIFACT"
printf ' %-20s %s\n' "$tid" "$verdict" >&2
done < <(rate_tasks)

value="$(rate_value "$got_w" "$TOTAL_W")"
conf="$(rate_confidence "$answered" "$NTASKS" "$RATE_REPEATS" "$flips")"

# Now, while the model is still loaded, ask the server what it actually has.
# /props is the only source for this; the config says what was asked for.
ctx="$(rate_live_ctx "$CFG" "$gguf")"
printf ' live n_ctx: %s\n' "$ctx" >> "$ARTIFACT"

printf 'RESULT %s value=%s quant=%s weight=%s/%s answered=%s/%s flips=%s confidence=%s\n\n' \
"${cid:-$served}" "$value" "$quant" "$got_w" "$TOTAL_W" "$answered" "$NTASKS" "$flips" "$conf" \
>> "$ARTIFACT"

c_ok "$served value=$value weight=$got_w/$TOTAL_W confidence=$conf"

# A model that errored on any task is measured but not offered as a rating:
# a partial run is a report, not evidence.
if [[ -n "$cid" ]] && (( answered == NTASKS )); then
ROWS+="$(rate_row "$cid" "$value" "$TODAY" "$ARTIFACT" "$conf")"$'\n'
fi
done < <(printf '%s\n' "$AVAILABLE")

c_info "artifact: $ARTIFACT"

if [[ -n "$ROWS" ]]; then
printf '\n'
c_info "Paste these into catalog_ratings() in lib/catalog.sh, replacing the matching ids:"
printf '%s' "$ROWS"
printf '\n'
c_info "Then re-validate: bash -c 'source lib/models.sh; catalog_validate || printf \"%%s\" \"\$CATALOG_ERRORS\"'"
else
c_warn "No complete result for any catalogued model -- nothing to record"
fi
120 changes: 115 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ claude
| Script | Purpose |
|---|---|
| `./71-verify-runtime.sh` | Query the **running** server's `/props` — confirms live `n_ctx`, flash-attn, KV cache types. Trust this over the config file. Grades its evidence; see below. |
| `./61-rate-models.sh` | Rate the models you serve on a fixed coding suite, and print rows for `catalog_ratings()`. Nothing it grades is executed; see [Rating the models you serve](#rating-the-models-you-serve). |
| `./70-thermal-sweep.sh` | Re-derive the best power limit for your chassis under a heat-soaked load. |
| `./80-try-bigger.sh <hf-repo> [quant]` | Assess, download, auto-tune `--n-cpu-moe` and benchmark a model **larger than VRAM**. Empirically finds the lowest working offload level. `--list` sizes it without downloading. |
| `./19-os-revert.sh` | Undo `10-os-tune.sh`, restoring the values captured before it ran — not assumed defaults. See [Rollback](#what-reversible-means). |
Expand Down Expand Up @@ -104,8 +105,12 @@ not reproducible here, and sorting one vendor's SWE-bench figure against
another's HumanEval figure produces an ordering that means nothing. The
validator enforces the consequence in both directions: a value cannot be
recorded without a method and a source behind it, and a method claiming
evidence cannot be recorded without a value. Filling these in is the deferred
local-benchmark work.
evidence cannot be recorded without a value.

Filling them in is what [`./61-rate-models.sh`](61-rate-models.sh) is for — it
measures the models **you** serve and prints rows you can paste in. The shipped
table stays `unknown` because the shipped table cannot contain your
measurements. See [Rating the models you serve](#rating-the-models-you-serve).

This replaced a single `coding_score` column carrying values from 42 to 88 with
no source, no date and no method — weighted at 25% of the recommendation.
Expand Down Expand Up @@ -178,7 +183,7 @@ One consequence to be aware of: with every coding rating still `unknown`, the
quality term does no discriminating work, so the ranking runs on freshness,
hardware fit, speed and features. On a 31 GB machine that makes Laguna XS 2.1
the top `medium` pick ahead of `qwen3-coder-30b`, on metadata alone. That is
the existing design behaving as designed, and it is an argument for finishing
the existing design behaving as designed, and it is an argument for running
the local benchmark, not for hand-weighting the table.

Running them here: XS 2.1 at `Q4_K_M` is 18.9 GiB and needs both cards
Expand Down Expand Up @@ -286,8 +291,14 @@ publisher, the rating behind a quarter of the weight is not.

Confidence counts three independent kinds of evidence — verified facts, a
sourced rating, and current live data. Three of three is `high`, two is
`medium`, fewer is `low`. **Nothing reaches `high` today**, because no model has
a rating; that ceiling lifts on its own once local benchmarking lands.
`medium`, fewer is `low`. **Nothing reaches `high` on the shipped table**,
because no model has a rating; that ceiling lifts on its own once you record
one with [`./61-rate-models.sh`](61-rate-models.sh).

The rating counts only when the rating itself is `medium` or better, and that
qualifier is load-bearing: a single unrepeated benchmark pass is recorded as
`low`, and a hurried run must not be able to raise the confidence of the
ranking it feeds.

Note what the neutral rating does to the ranking: with the quality term equal
for every model, the total is driven by fit and speed, so the smallest model
Expand Down Expand Up @@ -331,6 +342,105 @@ whose next step downloads tens of gigabytes.
afterwards so a run can be reproduced exactly. Duplicate, out-of-range and
non-numeric answers are refused by name rather than with "invalid input".

### Rating the models you serve

`./61-rate-models.sh` is the answer to "a quarter of the score is a neutral
placeholder". It runs a fixed suite against the models **this machine actually
serves**, at the quant they are actually served at, and writes the evidence to
`~/llm-rating-<date>.txt`.

```bash
./61-rate-models.sh # every served model, one pass each
./61-rate-models.sh --repeats 3 # three passes; any disagreement -> low
./61-rate-models.sh --model qwen3-4b # one model
./61-rate-models.sh --dry-run # show the plan, call nothing
```

Read [`lib/rate.sh`](lib/rate.sh) before trusting a number out of it. Four
things about it are deliberate and constrain what it can claim:

- **Nothing the model produces is executed.** The obvious way to grade
generated code is to run it; that means running text from a model on your
machine, as you, for a score. Every task is graded by reading the response —
an exact answer, a `tool_use` block, a parse. The task set is written around
that constraint rather than pretending it is not there.
- **It measures agent-shaped competence, not SWE-bench.** Read a snippet and
say what it does, pick the right tool with the right arguments, obey an
output format. Those are the failures that make a local model useless as a
Claude Code backend. Tool tasks carry double weight, and all three tools are
offered on every one of them, so a model that always calls the first tool
fails two of the three.
- **Format tasks are graded strictly; comprehension tasks are not.** The
distinction is deliberate and is what makes "obey an output format" a claim
the score actually measures — see the table below.
- **It is comparable across models on your machine and nowhere else.** That is
the comparison the ranking needs, and it is why the method is called
`local-benchmark` rather than `benchmark`.
- **The confidence ceiling is `medium`.** Twelve text-graded tasks at one quant
on one machine does not settle how good a model is at coding. A single pass
is `low`; two clean repeats are `medium`; `high` is not reachable from here.

#### Two grading regimes, on purpose

| Kind | Tasks | Graded on |
| --- | --- | --- |
| `answer` | 6 | The last non-empty line, normalised. Fences, quotes, trailing punctuation and preceding prose are stripped: the question is whether the model knows the answer. |
| `tool` | 3 | A `tool_use` block satisfying a jq filter. Wrapping text is irrelevant — the block either exists with the right arguments or it does not. |
| `oneword` | 1 | **The whole response.** Trimmed of surrounding whitespace, it must be exactly the word. A fence, a full stop or a sentence around it is the failure being measured. |
| `json-only` | 1 | **The whole response.** It must parse as a JSON object on its own — no fence, no prose — and satisfy a jq filter. |
| `diff-only` | 1 | **The whole response.** At least one `@@` hunk header, every non-empty line valid diff syntax, and the required line present. A preamble fails. |

The lenient normaliser would otherwise turn `Let me think.\n\nbash.` into
`bash` and pass a task whose entire subject is formatting. A strict kind may
not call it at all, and a test enforces that structurally, so the bug cannot
come back quietly. A model that cannot suppress its preamble cannot be trusted
to emit a patch a tool will apply — that is the thing `diff-only` measures.

Sampling is pinned — temperature 0, fixed seed — and `--repeats` is what checks
that the pinning held. A task that does not give the same verdict every time
counts as unstable, and one unstable task drops the whole run to `low`: a model
that answers differently at temperature 0 is telling you the measurement is
not stable.

#### The artifact records what was running

A served alias does not identify a runtime. `qwen3-4b` fronts whatever GGUF
`30-models.sh` last downloaded, built by whatever `20-build-llamacpp.sh` last
compiled, at whatever context `40-serve.sh` last configured — so two runs that
disagree could otherwise produce the same catalog row. Each run therefore
records:

| Field | Read from |
| --- | --- |
| llama.cpp revision | `.llamacpp-rev`, written by the build |
| weights | the `-m` path in the generated `llama-swap.yaml`, per model |
| quant | the filename **on disk**, not the catalog's preference — the reason to record it is that the two can differ |
| serving flags | the per-model flags in the same config, including `CUDA_VISIBLE_DEVICES` |
| live `n_ctx` | `/props` on the upstream serving *those* weights |

Anything that cannot be read is written as `unavailable`. The live context is
matched by model path rather than taken from the first server that answers:
with several models loaded, the first answer is some other model's context,
and recording that is worse than recording nothing. The quant also appears on
the `RESULT` line, so two ratings taken at different quants cannot be compared
by accident.

The script **does not edit the catalog**. It prints rows to paste:

```
qwen3-4b;67;2026-08-11;local-benchmark;file:llm-rating-20260811-1930.txt;medium
```

The source is the artifact's **basename**, not a URL. There is no https address
for a file in your `$HOME`, and the validator refuses an invented one —
`local-benchmark` must cite `file:<artifact>.txt`, `vendor-benchmark` must cite
https. A table that is curated by hand is not improved by a script that
rewrites its own evidence base, so the paste is manual and the run that
produced it is a file you can open.

A run where any task errored is written down but **not** offered as a row: a
partial run is a report, not evidence.

### The llama-swap binary is pinned and verified

`40-serve.sh` installs executable code into `/usr/local/bin` as root, so it
Expand Down
28 changes: 25 additions & 3 deletions lib/catalog.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,11 @@ CATALOG_FACT_METHODS=(hf-api card-stated vendor-blog derived)
#
# none no comparable evidence; rating_value must be "unknown"
# vendor-benchmark a number the publisher reports for its own model
# local-benchmark measured on this machine by lib/bench.sh
# local-benchmark measured on this machine by 61-rate-models.sh, against
# the suite in lib/rate.sh, at the quant this rig serves.
# Comparable across models HERE and nowhere else, which is
# the comparison the ranking needs. Its source is the
# artifact basename, not a URL -- see the validator.
CATALOG_RATING_METHODS=(none vendor-benchmark local-benchmark)

CATALOG_RATING_CONFIDENCE=(none low medium high)
Expand Down Expand Up @@ -715,8 +719,26 @@ catalog_validate_ratings_into() {
|| errs+="rating row $n ($id): method '$method' claims evidence but value is unknown"$'\n'
[[ "$conf" != "none" ]] \
|| errs+="rating row $n ($id): method '$method' claims evidence but confidence is 'none'"$'\n'
[[ "$source" == https://* ]] \
|| errs+="rating row $n ($id): rating_source '$source' must be an https URL"$'\n'
# What counts as a source depends on the method, because the two kinds of
# evidence live in different places. A vendor benchmark is published and
# must be linkable. A local benchmark is a file in the runner's own
# $HOME -- there is no URL for it, and inventing an https address so the
# field validates would be a fabrication in the one column whose whole
# job is to say where a number came from.
#
# `file:<basename>`, not an absolute path: the artifact is in the $HOME
# of whoever ran it, and a path from someone else's machine would not
# resolve on yours. The basename is what you look for in your own.
case "$method" in
local-benchmark)
[[ "$source" == file:*.txt && "$source" != *"/"* ]] \
|| errs+="rating row $n ($id): rating_source '$source' must be 'file:<artifact>.txt' -- a basename under \$HOME, produced by 61-rate-models.sh"$'\n'
;;
*)
[[ "$source" == https://* ]] \
|| errs+="rating row $n ($id): rating_source '$source' must be an https URL"$'\n'
;;
esac
if [[ ! "$rdate" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || ! date -d "$rdate" +%Y-%m-%d >/dev/null 2>&1; then
errs+="rating row $n ($id): rating_date '$rdate' is not a real ISO date"$'\n'
fi
Expand Down
Loading