docs: how we improve tallyman from session reviews, the error corpus and the eval suite

[paddymul]

Problem

Three ways of finding what to fix in tallyman have produced most of the recent PRs, and none of them is written down:

• A second Claude session reads a driver session's transcript and asks where the model worked around tallyman. Session 4a93fb4e led to feat(mcp): catalog_peek and catalog_query read rows without persisting anything #287 (catalog_query/catalog_peek), feat(mcp): a tool response carries errors recorded since the last one #288 (new_errors) and docs(mcp): a display klass formatter reference, and a hint when a build error has a known fix #289 (display-klass docs and hints).
• The hint code is run over every error recorded on the machine. On 2026-09-29 this found 12 of 135 distinct errors getting a hint, with 2 of those hints wrong.
• The eval suite (test: eval suite — multi-step sessions checked after every step, ported to ADR-011, with results on #216 #258) replays recorded sessions with resets, grid opens and cache wipes between steps. Its first run produced every sort in a worthy entry gets every remaining column as a tie-break; append the input's structural row key instead (needs its own ADR) #259, the diff page builds its own row, stats and keyed tables from JSON next to Buckaroo's compare view; show all data comparison through Buckaroo #260 and recalc reports a follower noop but leaves it stale forever when the revise changes only columns the follower never reads #262 to _clone's cp -c is forked, and the child can SIGSEGV in PROJ's fork handler on macOS; the import then makes a full copy without saying so #267.

An agent asked to "look at session X" currently has to work all of this out again: where transcripts and errors.jsonl live, that tallyman errors are in-band (is_error is false), the rules Paddy gave along the way, and the traps.

What this adds

docs/improving-from-sessions.md, covering:

• where the evidence is: transcript paths and format, and the per-project errors.jsonl, events.jsonl and telemetry.jsonl. It includes a short script that counts tool calls and prints tallyman errors; run on 4a93fb4e and b9d2057f it gives the same Bash counts the reviews reported (48 and 42).
• how to run a driver session and ask for a review, and the steps a useful review takes.
• a table mapping each kind of finding to a change: new tool, reply field, docstring, hint, lint, issue, upstream workaround, or nothing.
• the error-corpus check against the hint code, and rules for writing a hint (anchored regexes, neutral column names, a negative test).
• the eval suite: its files, how to run it, how to add a scenario from a transcript, and what to do with its findings.
• the prompt-pack repo, a checklist for an agent, and a dated table of what is in flight.

It also adds a line to docs/architecture.md's doc index, and a short CLAUDE.md section telling agents to read the doc before reviewing sessions, touching hints or running evals.

Things the doc points out that need a decision

• ~/code/tallyman_nfl_demo/.mcp.json runs the MCP server from /Users/paddy/tallyman, which is at 22f6a3e, 136 commits behind main. Both NFL reviews partly described tools that main had already replaced. Any NFL demo run started now tests that old checkout.
• test: eval suite — multi-step sessions checked after every step, ported to ADR-011, with results on #216 #258 is based on docs/adr-007-009-implemented, which has since reached main through feat(cache): implement ADR-007, ADR-008 and ADR-009 (tallyman-owned materialization, row order, digest stability) #189, so it needs retargeting.
• The base checkout ~/code/tallyman2 still has the untracked first version of the eval files (513c32a), which test: eval suite — multi-step sessions checked after every step, ported to ADR-011, with results on #216 #258 supersedes.

Checks

Docs only. The doc's anchors and relative links resolve against the headings in architecture.md and in the doc itself, and the embedded script was taken out of the markdown and run on two transcripts. CI was not watched.

🤖 Generated with Claude Code