review: make the live eval a measurement tool (repeat aggregation, powered runs, drift watch) #252
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "review": patch | ||
| --- | ||
|
|
||
| Make the live eval instrument a measurement tool, not just a tripwire (the tuning memo's rev-2 sizing items). `eval/aggregate.ts` pools N `live-ab-report.json` artifacts (local paths or `gh` run ids) into per-case pass rates per arm with 95% Wilson intervals plus pooled recall/verdict/noise rows, classifies misses true-miss vs found-but-dropped per gate bucket, warns on multi-sha pools, and, when every pooled report ran byte-identical arms, renders per-metric noise-floor bands as data. `live-ab.ts` gains `--repeats <n>` (the ~$29 targeted-repeats powered run: each arm runs every selected case n times in one dispatch, budget split per arm-run, report rendered through the aggregation core; the adversarial gate is decided by strict majority across repeats instead of the single-run best-of-three retry) and the A/B workflow exposes `cases`/`repeats`/`force_arms` dispatch inputs. A new scheduled workflow (`review-eval-drift.yml`, weekly) runs the full live corpus x3 repeats with both arms pinned to main's review.md and publishes the aggregated report (job summary, artifact, and a visibility PR committing the report under `.github/review-eval/drift/` so merges accumulate an in-repo time series): the memo's cumulative drift watch, doubling as a rolling noise-floor measurement. Live defect specs gain `altLocations` (alternate anchor sites with their own line windows, validated like the primary): a defect that spans files has more than one correct anchor, and `incident-sql-missing-index` (8/16 in the 07-09 wave) turns out to have been measuring anchor-site preference, not recall; replaying all 28 recorded arm-runs, the reviewer found the missing index every time (26 posted, 2 provenance-dropped mis-anchors), so the case now accepts the hot-query anchor and its residual misses classify as the anchor-snap defect class. `eval/match-arbiter.ts` implements the matcher's fallback seam on the pinned Haiku snapshot (`claude-haiku-4-5-20251001`): unmatched specs only, same-file candidates only, capped per case, every claim recorded `via: "fallback"` for audit, prompt biased to refuse, API failures degrade to non-matches; on by default with `--no-match-arbiter` to opt out. Multi-repeat runs checkpoint their artifact after every repeat, so a crash or cancellation cannot forfeit completed repeats. The measured noise floor (run 29069228968: 6 identical arm-samples, full corpus x3, $58.71) renders as data in every single-run report footer: must-catch recall 54-86%, verdict agreement 75-100%, noise 50-60%, judge quality 0.82-0.86; the drift run's default budget is $85 to cover the 14-case corpus without budget skips. `eval/README.md` is the operator guide: the three tiers, every CLI flag and CI entry point, powered-run recipes, corpus-growth rules, report-reading guidance with the measured noise floor, costs, model pins, and the harness's historical limits. Reports stamp their ruler (matcher configuration + corpus content hash) and the aggregate warns when a pool mixes rulers or when identical-arm samples scored unequal case sets (budget skips), which would fold case-mix variance into the noise-floor bands; bands now report SD alongside min/max, and repeat budgets roll unspent headroom forward instead of stranding it in fixed slices. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Review eval drift watch: the tuning memo's "cumulative drift watch". | ||
| # | ||
| # The per-PR A/B only ever prices one PR's marginal review.md delta against | ||
| # its base branch; nothing in that chain notices the reviewer slowly drifting | ||
| # on main (model updates, API behavior, corpus rot). This scheduled run is | ||
| # the memo's answer: the FULL live corpus, 3 repeats per arm, both arms | ||
| # pinned to main's review.md (`--force-arms` past the identity | ||
| # short-circuit), aggregated into per-case pass rates with binomial | ||
| # intervals. Week-over-week comparison happens across artifacts; within one | ||
|
Contributor
There was a problem hiding this comment. suggestion (non-blocking): The drift watch emits weekly snapshots but nothing computes the week-over-week comparison — the header notes it "happens across artifacts" (i.e. by a human), so a slow recall regression on main would sail under the within-run adversarial gate. |
||
| # run the two identical arms double as a rolling noise-floor measurement | ||
| # (the aggregate report renders the per-metric bands). | ||
| # | ||
| # Template: the 2026-07-10 manual cumulative dispatches (runs 29065968954 / | ||
| # 29065976210 / 29065977160, $42.83 for 3 repeats of both arms), now one | ||
| # dispatch via --repeats. Report-only: the aggregate lands in the job | ||
| # summary, the artifact, and a visibility PR (below); the adversarial gate | ||
| # still fails the job on a strict-majority regression, which for a | ||
| # main-vs-main run means production itself regressed and someone should look. | ||
|
|
||
| name: Review Eval Drift | ||
|
|
||
| on: | ||
| schedule: | ||
| # Weekly, Monday 06:23 UTC (off the top of the hour; scheduled-run | ||
| # dispatch is throttled at :00). | ||
| - cron: "23 6 * * 1" | ||
| workflow_dispatch: | ||
| inputs: | ||
| repeats: | ||
| description: "Repeats per arm" | ||
| required: false | ||
| default: "3" | ||
| max_usd: | ||
| description: "Total hard budget across both arms and all repeats" | ||
| required: false | ||
| default: "85" | ||
|
|
||
| permissions: | ||
| # write: the report step commits the aggregate under | ||
| # .github/review-eval/drift/ on a fresh branch and opens a visibility PR. | ||
| contents: write | ||
| pull-requests: write | ||
|
|
||
| concurrency: | ||
| group: review-eval-drift | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| drift: | ||
| name: Full corpus x repeats against main | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 | ||
| with: | ||
| # The baseline arm reads review.md from the base ref via | ||
| # `git show`, so the full history is needed. | ||
| fetch-depth: 0 | ||
| - uses: ./actions/shared-node-cache | ||
| - name: Run the drift watch | ||
| env: | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | ||
| REPEATS: ${{ inputs.repeats || '3' }} | ||
| MAX_USD: ${{ inputs.max_usd || '85' }} | ||
| run: | | ||
| if [ -z "$ANTHROPIC_API_KEY" ]; then | ||
| echo "ANTHROPIC_API_KEY secret not configured; skipping the drift run." >&2 | ||
| exit 0 | ||
| fi | ||
| pnpm dlx tsx workflows/review/eval/live-ab.ts \ | ||
| --base-ref "origin/${{ github.ref_name }}" \ | ||
| --force-arms \ | ||
| --repeats "$REPEATS" \ | ||
| --max-usd "$MAX_USD" \ | ||
| --out out/live-ab-report.json | ||
| - name: Upload the report artifact | ||
| # always(): a partial or gate-failing run still uploads what it wrote. | ||
| if: always() | ||
| uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 | ||
| with: | ||
| name: live-ab-report | ||
| path: out/live-ab-report.* | ||
| if-no-files-found: ignore | ||
| - name: Open the visibility PR | ||
| # A job summary nobody opens is not a drift watch. Commit the report | ||
| # markdown plus the compact aggregate (never the full multi-MB run | ||
| # payload) under .github/review-eval/drift/ and open a PR: it lands | ||
| # in review queues, holds discussion, and merging accumulates an | ||
| # in-repo time series. The path sits under .github/ deliberately: | ||
| # the changeset gate excludes it and it cannot re-trigger the A/B's | ||
| # workflows/review/** filter. Note: GITHUB_TOKEN-created PRs do not | ||
| # trigger CI workflows; the PR is report-only, so merge it directly. | ||
| if: always() | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| if [ ! -f out/live-ab-report.md ]; then | ||
| echo "no report markdown; nothing to publish" >&2 | ||
| exit 0 | ||
| fi | ||
| STAMP="$(date -u +%Y-%m-%d)" | ||
| DIR=".github/review-eval/drift" | ||
| BRANCH="drift-report/${STAMP}-${{ github.run_id }}" | ||
| mkdir -p "$DIR" | ||
|
Contributor
There was a problem hiding this comment. note (non-blocking): This workflow grants |
||
| cp out/live-ab-report.md "$DIR/$STAMP-run${{ github.run_id }}.md" | ||
| # The compact aggregate only; a single-run report has no .aggregate | ||
| # and its full payload (embedded corpus trees) must never be | ||
| # committed, so it gets a pointer at the run artifact instead. | ||
| jq 'if .aggregate then .aggregate | ||
| else {note: "single-run report; full payload in the run artifact", | ||
| runId: "${{ github.run_id }}"} end' \ | ||
| out/live-ab-report.json \ | ||
| > "$DIR/$STAMP-run${{ github.run_id }}.json" || true | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git checkout -b "$BRANCH" | ||
| git add "$DIR" | ||
| git commit -m "review: eval drift report $STAMP (run ${{ github.run_id }})" | ||
| git push origin "$BRANCH" | ||
| gh pr create \ | ||
| --base "${{ github.event.repository.default_branch }}" \ | ||
| --head "$BRANCH" \ | ||
| --title "review: eval drift report $STAMP" \ | ||
| --body-file out/live-ab-report.md | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note (non-blocking): This v8 SHA matches the pin already used by
node-ci.ymlandrelease.yml, so the alignment is correct. Heads up that.github/workflows/review-rereview-sweep.yml:79is now the last workflow still on the v7 SHA — a trivial follow-up would bring the repo to a single pinned version.