[pipeline-dcdad92d][slice-11/12] Full eval suite (four datasets... by jwbron · Pull Request #206 · Khan/actions

jwbron · 2026-07-03T20:40:23Z

Build the eval suite that measures what this run shipped: four datasets, five metrics, an Opus-4.8 LLM-judge with human audit, overfitting guards, an adversarial hard gate, and the reviewer version stamp that doubles as the single drift-guard surface.

Base PR: #196

What's in this PR

Commits (4):

Persist BRC history for slice-11 (#2548)
implement(tester): slice-11 eval-suite self-tests + CI-gate guard (task-11-6)
implement(coder): slice-11 judge.ts — decouple from slice-8 type; harden aggregate join
implement(coder): slice-11 — full eval suite (datasets, metrics, judge, gates, version stamp)

.../brc-history/pipeline-dcdad92d-implement-slice-11.json       | 2211 ++++++++++++++++++++++++++
 .egg-state/brc-history/pipeline-dcdad92d-implement-slice-11.md  | 2265 +++++++++++++++++++++++++++
 .../eval/corpus/adversarial/adversarial-injection-holdout.json  |   34 +
 .../review/eval/corpus/clean/clean-scope-drops-stale-nit.json   |   37 +
 workflows/review/eval/corpus/clean/clean-typed-refactor.json    |   15 +
 .../review/eval/corpus/golden/golden-approve-with-nit.json      |   49 +
 .../review/eval/corpus/golden/golden-request-changes-authz.json |   34 +
 .../eval/corpus/incidents/incident-federation-compat.json       |   34 +
 .../corpus/synthetic-mutations/mutation-concurrency-async.json  |   34 +
 .../corpus/synthetic-mutations/mutation-data-migrations.json    |   34 +
 .../corpus/synthetic-mutations/mutation-money-payments.json     |   34 +
 workflows/review/eval/gates.ts                                  |  209 +++
 workflows/review/eval/judge.ts                                  |  381 +++++
 workflows/review/eval/metrics.ts                                |  292 ++++
 workflows/review/eval/run-types.ts                              |   17 +
 workflows/review/eval/suite.test.ts                             |  613 ++++++++
 workflows/review/lib/version-stamp.ts                           |  184 +++
 17 files changed, 6477 insertions(+)

This slice

Full eval suite (four datasets, five metrics, judge, version stamp)

Files affected:

workflows/review/eval/corpus/
workflows/review/eval/metrics.ts
workflows/review/eval/judge.ts
workflows/review/eval/gates.ts
workflows/review/lib/version-stamp.ts
workflows/review/eval/suite.test.ts

Tasks (6) + acceptance criteria

task-11-1: Four datasets: incident repros; regenerated synthetic mutations mapped to lenses; golden set (human-comment + revert/follow-up labels); clean set. Built from Khan incident history + consumer PR records WITHOUT writing to consumer repos.
- Acceptance criteria: Four datasets load via the shared loader.
task-11-2: Five metrics: must-catch recall (~100%), golden precision, clean false-block (~0), noise, calibration.
- Acceptance criteria: Metrics computed over a dataset run.
task-11-3: Opus 4.8 LLM-judge with a human-audit sample; thumbs labels (slice 8) calibrate the judge.
- Acceptance criteria: Judge scores a run; audit sample surfaced.
task-11-4: Overfitting guards: golden-set holdout + fresh mutations; adversarial set as a hard gate for automatic mode.
- Acceptance criteria: Holdout separated; adversarial gate enforced.
task-11-5: Reviewer version stamp: hash of prompt+config stamped via the existing review: skip redundant approvals, drop skipped-dimensions note, persist sub-agent outputs #194 HTML marker. This is the SINGLE drift-guard surface (R14 reuses it; no new mechanism).
- Acceptance criteria: Stamp changes when prompt/config changes; readable by a consumer sync check.
task-11-6: Suite self-tests + wire the smoke subset (slice 9) as the CI gate; full suite as scheduled (not per-PR).
- Acceptance criteria: CI runs smoke; full suite invocable on schedule.

Stack

Position: slice 11 of 12 in pipeline pipeline-dcdad92d
Base PR: review: roster restructure, determinism boundary, eval suite #196
Stacked on top of egg/pipeline-dcdad92d/slice-10

changeset-bot · 2026-07-03T20:40:28Z

🦋 Changeset detected

Latest commit: cef4470

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Minor

Not sure what this means? Click here to learn what changesets are.

[Click here if you're a maintainer who wants to add another changeset to this PR](https://github.com/Khan/actions/new/egg/pipeline-dcdad92d/slice-11?filename=.changeset/bright-beers-leave.md&value=---%0A%22review%22%3A%20patch%0A---%0A%0A%5Bpipeline-dcdad92d%5D%5Bslice-11%2F12%5D%20Full%20eval%20suite%20(four%20datasets...%0A)

…ing (slice-1) R1 (task-1-2): standardize Step 6 review submission on one robust submit-pull-request-review call with a real, non-empty body; remove the empty-body branch and any fallback/retry path. E2 (task-1-3): stage /tmp/gh-aw/review/pr-context.json in Step 1 and wire all six sub-agent read blocks to consume it. Extends #194 diff staging; foundation for slice-7 lenses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

task-1-1: workflows/review/lib/finding-schema.ts - FINDING_SCHEMA_VERSION exported constant; each finding carries schema_version and the validator rejects an unrecognized version (breaking-shape guard). - Finding shape (snake_case wire keys, matching #194 per-run JSON artifacts): id, lens, anchor, severity, confidence, evidence_trace, optional suggested_patch, optional pre_merge_obligation, producing_hunt, model_authored_prose. - Anchor is a discriminated union with the required PR-level type (pr | file | line, line supports side + multi-line start_line..line range). - KNOWN_LENSES (11 specialist + always-on/triage) and SEVERITIES exported for reuse by the slice-3 router and slice-2 verdict; lens/severity validated against them. - validateFinding collects ALL violations (diagnosable per-lens drop rate), plus isValidFinding narrowing wrapper and throwing assertFinding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…k-1-4) Covers the exported FINDING_SCHEMA_VERSION constant, well-formed findings across every anchor type (line/range/side, file, PR-level) plus optional suggested_patch/pre_merge_obligation, and malformed findings for every required field. Asserts validateFinding collects ALL violations (per-lens drop-rate diagnosability), and exercises isValidFinding narrowing and assertFinding throwing. 37 tests green under vitest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…nts) - Restore the empty-review-body convention: with inline comments present the verdict is submitted with an empty body; a non-empty body exists only for comment-less reviews and skipped-dimension notes. Keeps the single-call submission rule. - Deliver workflows/review/lib/ to consumer repos: gh-aw's source: import only copies the .md, so the job checks out Khan/actions at the pinned review-v* release (single version surface for prompt + code) and installs its runtime deps. - Drop planning identifiers (E2, R8, slice numbers, operator-directive note) from shipped file comments.

jeresig

Very interesting... I've been reading through the evals and it's unclear to me what we're actually live-evaling. I don't see us checking out any repos, nor running the review.md, as far as I can tell - so what is the judge actually judging? The deterministic tests make sense to me, though!

jeresig · 2026-07-07T16:36:40Z

@@ -1,7 +1,7 @@
 # Causal experiment — webapp PR #40536 (OpenAccess authorization)


I mentioned this in the other PR, but I don't think we need/want this file?

jeresig · 2026-07-07T16:47:26Z

+  workflow_dispatch: {}
+
+jobs:
+  full-suite:


IMO, we should split this into two workflows so that we always run the deterministic portion on every PR - but also I don't even know that we need this, these tests should be running as part of our normal test suite runner - right?

jeresig · 2026-07-07T16:48:54Z

@@ -0,0 +1,150 @@
+# Review full eval suite, scheduled (R5, task-11-6)
+#
+# STAGED FILE, NOT YET ACTIVE. It lives under `.github-staging/` because no


I think you need to fix this?

jeresig · 2026-07-07T16:49:43Z

+name: Review Full Eval Suite
+
+on:
+  schedule:


I'm not sure about this running weekly... I guess it could be ok, but we should find a way to report the results somewhere so that we can see them.

jeresig · 2026-07-07T16:49:59Z

+      - uses: ./actions/shared-node-cache
+      - name: Run judgeCorpus with the live pinned model
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}


I've generated and added an API key to this repo's secrets.

jeresig · 2026-07-07T16:50:36Z

+            echo "ANTHROPIC_API_KEY secret not configured; skipping the live-judge run." >&2
+            exit 0
+          fi
+          cat > ./eval-full-live-judge.mts <<'SCRIPT'


I'm not sure why we're generating this file dynamically, we should just make it a real script instead (that way normal linting/testing/typechecking can happen).

… rendering (slice-2) task-2-1/2-2 verdict.ts: pure computeVerdict(labels + dimension availability + policy conflicts) -> APPROVE | REQUEST_CHANGES | HOLD_FOR_HUMAN. Consumes #194's mechanical label rule (isBlockingLabel); does not re-implement it. R2 gate: a missing core dimension (correctness/skill-severity) or a policy-named conflict holds for a human; a lost pattern-triage is note-and-continue. Documented default blocking threshold = 1 (tunable, not a HITL gate). task-2-3 render-comment.ts: code-owned Conventional-Comment label taxonomy + deterministic label-wrapping/templating from the finding schema; models own all prose. Review-body rendering mirrors review.md Step 6 + skipped-dimension notes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…pshot tests (task-2-4) Truth-table tests for computeVerdict (verdict.test.ts): APPROVE / REQUEST_CHANGES / HOLD_FOR_HUMAN precedence, the R2 core-dimension gate (correctness + skill-severity absence holds; hold dominates a blocking label with all reasons retained), pattern-triage note-and-continue, policy-named conflicts (verbatim detail passthrough), blocking-threshold clamping (>=1) + never-RC-on-zero, and purity (deterministic, non-mutating). Rendering snapshot tests (render-comment.test.ts): isBlockingLabel over the full label taxonomy + unknown-label safe default; labelForFinding severity x lens matrix (conventions => best-practice, specialists => plain); renderComment inline snapshots (label wrap + verbatim prose + optional suggestion block) with verbatim prose/patch passthrough asserted; renderReviewBody inline snapshots for every verdict x inline branch + skipped-dimension notes + non-empty safe-output guard. Fixtures validated through assertFinding so tests can't pass on schema-invalid findings. 50 tests green; full pnpm test suite 253/253; tsc --noEmit clean (node24). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…(task-10-3) Add a task-10-3 verification block to smoke.test.ts that pins the two properties the wave-2 recall/precision rebalance (slice-10, review.md edits 8-13) must not regress, framed in terms of the rebalance's own mechanism rather than the generic slice-9 gate: - No recall regression: every must-catch finding is blocking AND >= medium confidence, so edit-13's posting bar and edit-10's downgrade/refuter pass (which act only on non-blocking low-confidence advisories) provably cannot drop it. Still asserts each is posted by the deterministic path today. - No new false-block: clean cases carry no blocking finding for the rebalance to surface; verdict stays APPROVE with nothing blocking. - The downgrade/collapse path is non-vacuously exercised: a low-confidence advisory exists and is never a must-catch. The rebalance is prompt-only and does not touch the deterministic review path this runner replays, so the smoke set stays green by construction; this block proves why. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…r panel, R6 experiment E8 (coverage first), E11 (confirm before you claim), E12 (cite exact lines or quote) added to the shared R9 producer protocol (Step 3). E9 (blocking requires a concrete failing scenario) added to Step 4 blocking-label criteria. E10 (drop only the refuted; downgrade the uncertain) rewrites the claim-validator keep/drop rule. Blocking-claim refuter panel (task-10-2): new Phase 3b + `claim-refuter` agent (batched/parallel, xhigh per R12), wired to the slice-1 finding confidence field and the slice-2 computed verdict — a refuted blocking claim is downgraded, not deleted, so recomputing computeVerdict over post-refuter labels is the wiring. E13 posting bar in Step 5: ranked posting, inline >= medium confidence, low-confidence in one collapsed <details> section, suggested diffs where clear, no padding. R6 causal experiment (task-10-4): workflows/review/eval/experiments/webapp-40536.md — edits 8+10 via the no-post harness (no consumer write); OpenAccess authorization surfacing recorded against the incident-auth-bypass smoke stand-in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ription)

…ator-confirmed claims An audit of all 319 blocking claims the bot posted on Khan/frontend since the validator shipped (90 PRs, 2026-06-30 to 2026-07-07) found 12 false blocks, of which only 2 were in the refuter panel's addressable class; the panel is unearned machinery (PR #205 discussion). It is removed rather than shipped, recoverable from this history if the eval suite's false-block metric regresses. In its place: - Claim validation is three-state (confirmed / plausible / refuted) with symmetric evidence duties: confirm by citing the failing line, refute by citing the guard. Only a confirmed claim may carry a blocking label into the computed verdict; plausible downgrades to non-blocking. - Author-disputed claims get a usage-depth bar: no re-block unless the re-check traces to actual usage (the frontend #13214 failure shape); otherwise they post as questions. - The no-post runner replays the gate's apply rules deterministically (stage 3b) and the smoke corpus gains six audit-seeded cases: five production false blocks that must approve, one confirmed block that must stay blocking.

…ragraph

…e, gates, version stamp) task-11-1: four datasets loading via the shared loader — golden (human-comment + revert/follow-up ground truth), synthetic-mutations mapped to lenses, clean, and a non-smoke incident repro; holdout/fresh/adversarial tags for the gates. task-11-2: metrics.ts — five metrics (must-catch recall, golden precision, clean false-block, noise, calibration/ECE) computed over a corpus run, pure. task-11-3: judge.ts — Opus-4.8 LLM-judge with a pure request/aggregate core and an injected model seam; deterministic human-audit sample; thumbs (slice-8) calibration pass. task-11-4: gates.ts — golden holdout split + fresh mutations (overfitting report) and the adversarial set as a hard gate for automatic mode. task-11-5: version-stamp.ts — SHA-256 of prompt+config+schema rendered into the existing #194 pr-reviewer HTML marker; the single drift-guard surface (parse + hasDrifted for a consumer sync check). run-types.ts holds the shared EvalRun shape the three eval modules consume. No consumer-repo writes; no benchmark-gated construction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…den aggregate join Addresses reviewer_code_holistic v1 NACK (dangling cross-slice build dependency): judge.ts imported `DownvoteReason` from ../lib/thumbs-sweep, but slice-8 (the thumbs sweep) is not in slice-11's ancestry, so that module does not exist on this branch. The judge consumes thumbs labels as data and never needs the sweep module at build time, so the import was gratuitous coupling. Declared a local `DownvoteReason` union (structurally identical to slice-8's) so judge.ts is self-contained and typechecks against the branch as it stands. Also cleared the two non-blocking advisories while re-proposing: - aggregate() now rejects duplicate finding ids across requests (the join is by bare finding id, only case-unique in the schema — a collision would mis-join), and throws on a score whose finding id matches no request. The docstring now matches the code (previously claimed an unknown-id throw that did not exist). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…sk-11-6) Adds workflows/review/eval/suite.test.ts, the task-11-6 deliverable: - Suite self-tests over the coder's slice-11 modules (loader four-dataset load, five metrics, adversarial/overfitting gates, LLM-judge with a STUB model, and the version-stamp drift surface), exercised against the real corpus plus focused synthetic fixtures built through the public parseCase/runCase path. - Whole-suite green-on-baseline check: every corpus case meets its expected block; must-catch recall = 100%, clean false-block = 0, adversarial hard gate passes (automatic mode allowed). - CI-wiring guard: the smoke subset is a strict subset (the fast per-PR pnpm-test gate), and the ONLY model seam is judgeCorpus (injected model, async, off the deterministic path) -- so the live-judge full suite is scheduled/opt-in, never per-PR. Metrics+gates run over the full corpus with zero model calls. Deterministic: no model, no network, no clock, no randomness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…1-6, resolves gap-1)

… is a committed script on a weekly workflow, version-stamp machinery dropped

…/slice-11 # Conflicts: # workflows/review/eval/corpus/smoke/adversarial-injection-approve.json # workflows/review/eval/smoke.test.ts # workflows/review/lib/lenses.test.ts # workflows/review/review.md

jwbron added egg agent:orchestrator labels Jul 3, 2026

jwbron mentioned this pull request Jul 3, 2026

review: roster restructure, determinism boundary, eval suite #196

Closed

jwbron force-pushed the egg/pipeline-dcdad92d/slice-10 branch from 0c30b5c to 345d59c Compare July 3, 2026 21:36

jwbron force-pushed the egg/pipeline-dcdad92d/slice-11 branch from ee3bc58 to d7427e8 Compare July 3, 2026 21:37

jwbron force-pushed the egg/pipeline-dcdad92d/slice-10 branch from 345d59c to 44ff953 Compare July 3, 2026 21:42

jwbron force-pushed the egg/pipeline-dcdad92d/slice-11 branch from d7427e8 to 41c87b4 Compare July 3, 2026 21:42

egg and others added 7 commits July 3, 2026 14:43

Persist BRC history for slice-1 (#2548)

8826894

slice-1: prettier fixes + changeset

c02da14

slice-1: drop BRC history artifacts from slice PR

b119a01

jwbron force-pushed the egg/pipeline-dcdad92d/slice-10 branch from 44ff953 to baef924 Compare July 3, 2026 21:43

jwbron force-pushed the egg/pipeline-dcdad92d/slice-11 branch from 41c87b4 to 21b1988 Compare July 3, 2026 21:43

jwbron marked this pull request as ready for review July 3, 2026 21:44

jwbron requested a review from jeresig July 3, 2026 21:44

khan-actions-bot requested review from a team and jaredly and removed request for a team July 3, 2026 21:44

jwbron force-pushed the egg/pipeline-dcdad92d/slice-10 branch from baef924 to 8ccb4db Compare July 6, 2026 18:19

jwbron force-pushed the egg/pipeline-dcdad92d/slice-11 branch from 21b1988 to 0a45c34 Compare July 6, 2026 18:19

khan-actions-bot requested a review from a team July 6, 2026 18:20

jeresig requested changes Jul 7, 2026

View reviewed changes

jwbron and others added 3 commits July 7, 2026 11:58

review: drop lib install step; scripts run via npx tsx

123a10f

jwbron and others added 8 commits July 7, 2026 12:24

review: drop low-value smoke corpus size test

35216aa

Persist BRC history for slice-10 (#2548)

f76c328

slice-10: prettier fix + changeset

23aba3b

slice-10: drop BRC history artifacts from slice PR

721f8a5

review: strip remaining plan identifiers (slice-10 and lens prompts)

fa0bfbb

review: drop the webapp-40536 experiment record (moves to the PR desc…

38f3e75

…ription)

jwbron force-pushed the egg/pipeline-dcdad92d/slice-10 branch from 8ccb4db to 38f3e75 Compare July 7, 2026 19:27

jwbron force-pushed the egg/pipeline-dcdad92d/slice-11 branch 2 times, most recently from b4852d4 to f2efbd9 Compare July 7, 2026 19:36

jwbron and others added 10 commits July 7, 2026 13:26

review: strip a reintroduced plan identifier from the verdict-gate pa…

766a8bc

…ragraph

Persist BRC history for slice-11 (#2548)

14c38fb

slice-11: staged scheduled full-eval workflow with live judge (task-1…

b6db587

…1-6, resolves gap-1)

slice-11: prettier fixes + changeset

169a6b8

slice-11: drop BRC history artifacts from slice PR

96b0895

review: strip planning identifiers from the eval suite (slice-11)

4eafaf6

jwbron force-pushed the egg/pipeline-dcdad92d/slice-11 branch from f2efbd9 to 983461b Compare July 7, 2026 20:35

review: eval CI rework; deterministic suite rides node-ci, live judge…

af7ad3a

… is a committed script on a weekly workflow, version-stamp machinery dropped

jwbron force-pushed the egg/pipeline-dcdad92d/slice-11 branch from 983461b to af7ad3a Compare July 7, 2026 20:40

jwbron mentioned this pull request Jul 7, 2026

[pipeline-dcdad92d][slice-1/12] Foundations: finding schema... #197

Merged

jeresig approved these changes Jul 8, 2026

View reviewed changes

Base automatically changed from egg/pipeline-dcdad92d/slice-10 to main July 8, 2026 16:56

Merge remote-tracking branch 'origin/main' into egg/pipeline-dcdad92d…

cef4470

…/slice-11 # Conflicts: # workflows/review/eval/corpus/smoke/adversarial-injection-approve.json # workflows/review/eval/smoke.test.ts # workflows/review/lib/lenses.test.ts # workflows/review/review.md

jwbron merged commit 58f71a5 into main Jul 8, 2026
2 of 3 checks passed

jwbron deleted the egg/pipeline-dcdad92d/slice-11 branch July 8, 2026 16:57

Navigation Menu

Uh oh!

[pipeline-dcdad92d][slice-11/12] Full eval suite (four datasets... - #206

[pipeline-dcdad92d][slice-11/12] Full eval suite (four datasets...#206
jwbron merged 73 commits into
mainfrom
egg/pipeline-dcdad92d/slice-11

jwbron commented Jul 3, 2026

Uh oh!

changeset-bot Bot commented Jul 3, 2026 •

edited

Loading

Uh oh!

jeresig left a comment

Uh oh!

jeresig Jul 7, 2026

Uh oh!

jeresig Jul 7, 2026

Uh oh!

jeresig Jul 7, 2026

Uh oh!

jeresig Jul 7, 2026

Uh oh!

jeresig Jul 7, 2026

Uh oh!

jeresig Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1,7 +1,7 @@
		# Causal experiment — webapp PR #40536 (OpenAccess authorization)

Uh oh!

Conversation

jwbron commented Jul 3, 2026

What's in this PR

This slice

Stack

Uh oh!

changeset-bot Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

jeresig left a comment

Choose a reason for hiding this comment

Uh oh!

jeresig Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeresig Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeresig Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeresig Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeresig Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

jeresig Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

changeset-bot Bot commented Jul 3, 2026 •

edited

Loading