Field tests - #146
Merged
Merged
Field tests#146
Conversation
The README told you what the two modes measure, not which one you want. Adds a three-question flowchart (who is sending requests / how long are the prompts / do the answers quote the prompt back) with five lettered outcomes, and a table giving each one as a diff against the .env that Quick start already told you to copy — so "which setup" has a mechanical answer instead of a reading exercise. Writing it surfaced three things that were wrong: - .env.example suggested CTX=long while leaving SPEC=dflash2 set. DFlash2 past 64k is worth it only for context reproduction; SPEC=mtp CTX=long beats it about 2:1 on everything else (single-user/README.md). Says so now. - .env.example told CTX=huge users to run kvarn/install.sh first. That file is read only by Docker Compose, and the image already builds KVarN in. - single-user/README.md claimed CTX=huge "stays MTP (the script falls back with a message)". The launcher grew a dflash2+huge arm; docs/long-context.md has measured it at 268k tokens of pool since. Points there instead. Drops the "If you are the only user" section, whose content is now two rows of the table plus the link it already carried.
"Help us test more hardware" was mostly an ask for Runpod/Vast credits, which the Ko-fi link and a "compute offer" issue now cover in two lines. What the repo actually needs is runs on cards nobody here owns, so the section becomes the protocol for producing one: set the power limit, run the harness twice and keep the second, check quality, file a report. .github/ISSUE_TEMPLATE/field-report.yml makes the report structured — card, power cap, driver/OS, commit, which setup, the ROW lines, the GSM8K score. Those are the fields that decide whether a number can sit next to the reference row or only in the "measured with their own client" list, and asking for them after the fact has been most of the work on every reproduction so far. config.yml keeps blank issues on and points compute offers and funding at the right places. Two corrections to the protocol while writing it: the harness needs the venv that lives inside the container, so the commands go through `compose exec` (a plain `bash bench/run_benchmarks.sh` finds no `venv/bin/vllm` on the Docker path), and the quality battery needs its GSM8K data downloaded first. Checked on the box: `docker compose exec` reaches a profiled service with and without `--profile` on Compose v2.35.1, and `venv/bin/hf` exists in the published image.
Went through the open issues against the list of measurements the README asks for. Four of them are already someone's hardware, and the section was asking for work that has partly been done: - TP=4 is measured — 4x RTX 5060 Ti on the canonical harness (#105). What is missing is four *Ampere* cards, which is what #135 offers. - 16 GB cards are measured, by that same report. - 12 GB: two boxes serve the model on dual 3060s (#68), with their own clients rather than the harness. - sm80 is not unowned either — two people are mid-bisect on a fault only their cards produce (#98, #72). What a third box buys there is separating the card from the build, which is a different ask than "run the harness". So "Most wanted" now links each row to its thread and says what specifically is still missing, and docs/reproductions/README.md lists the four results that were only in issues: the 5060 Ti four-arm A/B (which re-attributes its own 2.37x to 1.89x KV dtype and 1.12x drafter, and lands the TP4 half of the DFLASH_TOKENS=15 warning), the dual 3060s, the triple 3090 that pins TP=3 as a checkpoint property, and the dual 3090 batch run. sm90 is the only row with nothing behind it at all.
cpuchip
added a commit
to cpuchip/qwen38-27b-rtx3090
that referenced
this pull request
Sep 18, 2026
…: the 0.29 row of the field-tests table now names this branch's pin
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.