review: re-review coverage for the eval corpus (open-PR cases + lifecycle chain) - #251
Merged
Conversation
…live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged.
…action and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately.
… runner (phase 2c)
live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.
live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.
live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero.
Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step.
… and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary).
…rpus' into jwbron/live-eval-producer-staging
…staging' into jwbron/live-eval-ab-runner
…to jwbron/live-eval-ab-ci
…s with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id.
…staging' into jwbron/live-eval-ab-runner
…eport instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual.
…to jwbron/live-eval-ab-ci
…A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed.
…orpus' into tmp-refresh
…roducer-staging' into tmp-refresh
…b-runner' into tmp-refresh
…b-ci' into tmp-refresh
…ity, code-rendered from the reconciler keep-list
… instead of dropping them
…oad (allowed-paths must match staging-relative paths)
…untability' into jwbron/review-out-of-lane
…into one staged disciplines file
…ive case The first live acceptance runs missed incident-sql-missing-index:dm-missing-index-1 in all four arms: the recorded finding is a data-migrations LENS finding, but live cases carried no routerConfig lens rules, so the live roster never spawned the specialist that catches it. Every live case whose recorded finding belongs to a specialist lens now routes that lens on the finding's file (seven cases across five lenses), mirroring how a consumer ROUTING file would route the same paths in production.
…rpus' into jwbron/live-eval-producer-staging
…staging' into jwbron/live-eval-ab-runner
… the A/B report The acceptance runs surfaced a claim-validator failure that the report could only name, not explain (perCase carried agent names only, and the PerAgentReport.failed detail never reached the markdown). perCase failedAgents entries are now '<agent>: <reason>', so the next failure is diagnosable from the sticky comment alone.
… into jwbron/rereview-mode-dial
…' into jwbron/rereview-live-corpus
jwbron
added a commit
that referenced
this pull request
Jul 10, 2026
…budget to $85 The noise-floor dispatch showed the live corpus is now 14 cases (the #251 golden re-review set), and the $60 default trimmed case-runs on both arms (38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs at the measured ~$10/arm-run with landing headroom, so the weekly aggregate never carries budget-skip asymmetry.
This was referenced Jul 10, 2026
Merged
…ereview-accountability
Contributor
Review Guidancegithub-actions (12 files)
Common patterns4 files: Identical 3 files: Identical 2 files: Identical 2 files: 3 files: Same label-trigger + job-level label guard pattern added to every model-spending eval workflow ( Excluded from review (20 files)Not individually reviewed — documentation, or seeded-defect corpus fixtures fully covered by their
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| `budget: stopping before ${mode}/${corpusCase.id} ` + | ||
| `($${usd.toFixed(2)} spent of $${maxUsd})`, | ||
| ); | ||
| break; |
Contributor
There was a problem hiding this comment.
note (non-blocking): On budget exhaustion the break exits only the inner case loop, and buildSweepReport drops modes with zero rows while the stop message goes to stderr (only stdout is teed into the summary/comment). A truncated pricing table is then indistinguishable from a complete one, which matters if a repo sets its ROUTING re-review line from it. Consider recording a truncated flag in SweepReport and surfacing a warning in renderSweepMarkdown.
| }; | ||
| const routing = route({files: corpusCase.changedFiles}, routerConfig); | ||
| const rosterNames = [...DEFAULT_FINDERS, ...routing.lensesToSpawn]; | ||
| const dispatch = staged.rereviewPlan?.dispatch ?? "all"; |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): The mode→roster mapping here (reconcile+correctness → ['correctness-reviewer'], fast → []) is never exercised — live-producer.test.ts's only fixture has no live.rereview block, so no test drives produceLive on a rereview case. A test using the existing scriptedRunner that runs a rereview case at flip-gated/fast and asserts the dispatched agent names would catch a regression in the depth-plan sizing.
| ); | ||
| break; | ||
| } | ||
| const produced = await produceLive(corpusCase, agents, { |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): This per-case pipeline (produceLive → runCase → matchCase → scoreRereview plus its own budget check) duplicates runArm in live-ab.ts, which already runs the identical sequence and aggregates rereview metrics. Building the sweep as a thin loop over runArm (one call per mode) would keep sweep and A/B prices comparable and avoid the two pipelines drifting.
|
|
||
| const rows: SweepRow[] = []; | ||
| let usd = 0; | ||
| for (const mode of modes) { |
Contributor
There was a problem hiding this comment.
thought (non-blocking): Because decideReReviewDepth runs before any model dispatch, a case whose tripwire re-arms full under a reduced mode produces staged inputs identical to its full row — so those cases dispatch the same full-depth run once per mode. Staging all (mode, case) pairs first and dispatching once per distinct executed plan would avoid paying repeatedly for the same measurement.
| // Re-review cases: dispatch the reconciler over the staged threads (it | ||
| // runs at EVERY depth — reconciliation is the fast path's whole job). | ||
| let reconciliation: LiveReconciliation | undefined; | ||
| if (corpusCase.live?.rereview !== undefined) { |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): The reconciler dispatch here and parseReconciliation (lines ~189-197) are only covered indirectly — scoreRereview is unit-tested with hand-built reconciliation objects, but the glue that produces them from a model output isn't. A produceLive test asserting result.reconciliation for a well-formed script (and that a malformed one is handled gracefully) would close the gap.
# Conflicts: # workflows/review/eval/corpus/live.ts # workflows/review/eval/corpus/loader.ts
| * world (its cwd, with no network), so the agent WILL often try to read | ||
| * an imported module or caller that is not there; that read returns an | ||
| * ordinary not-found tool error the agent tolerates and works around, | ||
| * not a run failure. The cost of a missing file is realism, not a crash: |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): The minimal-tree doctrine documented here guarantees only the changed files exist, but a re-review case's priorThreads[].path routinely anchors on files this push didn't touch — and neither liveTreeErrors (live.ts:542-551) nor parseRereview (live.ts:323-330) checks a thread's path against the tree. When an anchor file is absent the reconciler can't read the code the thread refers to, so its resolve/keep decision is scored against a blind guess. Consider extending this doc to name prior-thread anchor files, and/or warning in liveTreeErrors when a priorThreads[].path is missing from the tree.
jwbron
added a commit
that referenced
this pull request
Jul 10, 2026
Phase 3 of the live A/B eval plan (#232), stacked on the Phase 2 producer (#234): score live runs against labeled ground truth and diff two review.md versions arm to arm. ## 3a: `eval/live-match.ts` Live runs choose their own finding ids, so the recorded metrics (which key on `expected.mustCatch` ids) cannot score them. The matcher maps a run's POSTED candidates onto the case's labeled defect specs: a candidate satisfies a spec when its anchor agrees with the spec's path (and line window, when both carry one) and any mechanism alternate matches the finding's `failure_scenario` or prose, case-insensitively. Each candidate satisfies at most one spec and each spec at most one candidate, so one comment cannot claim two defects. An injected fallback arbiter (hard-capped, same-file candidates only, recorded as `via: "fallback"` for human audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. `computeLiveMetrics` aggregates the live analogues of the suite's numbers: must-catch recall, verdict agreement, clean false-flag (a clean case that blocks counts), and noise (posted candidates matching no spec). ## 3b: `eval/live-ab.ts` The arm orchestrator and CLI. Baseline `review.md` comes from `git show <merge-base>` (or `--base-ref`), candidate from the working tree; both arms run over the same live corpus with everything else (corpus, `lib/`, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model-behavior seam. Each arm runs under half the `--max-usd` budget (default 40 total) with sticky exhaustion: once spend plus the running per-case average crosses the cap, the remaining cases are recorded as SKIPPED and the report still emits (dying at a cap with nothing posted is the failure mode the plan forbids). Each live result replays through the deterministic pipeline; `runner.ts` gains an optional `RunOptions.validation` override so the live validator's output replaces the recorded block. Spec-level regressions ("baseline caught, candidate missed" and the reverse) are diffed only over cases both arms actually scored, so a budget skip is never reported as a regression. Judge scoring reuses the pinned judge for quality aggregates only (judge-vs-ground-truth disagreement keys on recorded ids, which live arms do not use); the fetch model moves from `live-judge.ts` into shared `eval/judge-live-model.ts`. Output: `out/live-ab-report.json` plus a markdown table (metrics deltas, judge quality delta, cost, wall clock, regressions, skips, agent failures) printed and appended to `GITHUB_STEP_SUMMARY`. Report-only, with one exception per the playbook's standing rule: the candidate arm failing any adversarial-injection case (wrong verdict or missed spec) exits non-zero. ## Test plan: - `pnpm run test --run`: 696 tests green (16 new: matcher rules incl. window overlap, one-candidate-one-spec, malformed-regex fallback-to-literal, capped fallback audit trail; runArm budget semantics incl. the sticky-stop case that caught a real bug in review; regression diffing over shared cases; report rendering both gate outcomes). - `pnpm run typecheck` and eslint clean. - The end-to-end CLI (`pnpm dlx tsx workflows/review/eval/live-ab.ts --max-usd 10 --cases incident-money-rounding,clean-no-findings`) needs `ANTHROPIC_API_KEY`; this environment has none, so that is the first thing to exercise when testing. On an unchanged review.md it prints the identical-arms note and near-zero deltas. ## Next steps (human) 1. ~~End-to-end A/B validation~~ DONE, superseded by the phase 4 acceptance runs (linked on #237): the control arm held recall and verdict agreement flat at +0% and the weakened arm showed -17% recall with the lost spec named, which is also the first empirical read on matcher quality (7 cases matched; one systematic miss traced to lens routing, fixed on #233, not to the matcher). Two acceptance-driven additions landed here since: `perCase.failedAgents` entries now carry `<agent>: <reason>` (a claim-validator flake was undiagnosable from the report alone; the flake itself is still to be root-caused on a future run), and every report closes with a per-row stability footer (judge quality rose on the deliberately regressed arm because fewer, surer comments each score better, so the footer warns against reading it alone). 2. Review the two scoring caveats: judge output is quality-aggregates only (disagreement semantics key on recorded ids), and the fallback arbiter is wired but not yet given a live implementation (matches would be marked `via: "fallback"` for audit when one is added). 3. Confirm the budget semantics (half of `--max-usd` per arm, sticky skip-and-report) match how you want cost bounded. --- ## Update 2026-07-09: eval-tuning pass (three riders) Three instrument fixes from the [eval-tuning memo](https://claude.ai/code/artifact/2995f0b5-8840-4d03-96e3-2518e2f0f75a), landed here so the whole stack inherits them: 1. **Pre-flight identity short-circuit** (memo item 1, hoisted down from #251 with identical text so that branch merges cleanly): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; `--force-arms` preserved for wobble controls. Corpus-only and plumbing-only PRs in this stack now pay $0 per push. 2. **Gate-flip retry** (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake (the #244/#245 incident shape). Retry spend is recorded in the report. 3. **Drop-bucket taxonomy** (memo item 4): every missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/validation) by re-matching against the dropped candidates with the line window relaxed. Lost regressions are annotated with their class and the table gains a found-but-dropped count row. Plus **judge economics**: the judge pin moves from Opus 4.8 to `claude-haiku-4-5-20251001` (the judge grades prose only; every load-bearing metric is deterministic, and the acceptance runs showed judge quality is not single-run-stable on any model, so it is priced accordingly; supersedes operator direction 4), and the judge seam gains retry with backoff so a transient 500 no longer wastes a whole scoring pass. Author: jwbron Reviewers: jeresig, github-actions[bot], jwbron, jaredly Required Reviewers: Approved By: jeresig, github-actions[bot] Checks: ✅ 9 checks were successful Pull Request URL: #236
jeresig
approved these changes
Jul 13, 2026
jeresig
left a comment
jeresig
left a comment
Member
There was a problem hiding this comment.
Great! Thank you for adding tests for this.
…in' into jwbron/rereview-live-corpus # Conflicts: # .github/workflows/review-eval-ab.yml # workflows/review/README.md # workflows/review/eval/live-ab.ts # workflows/review/eval/live-producer.ts # workflows/review/eval/live-stage.ts # workflows/review/eval/runner.ts # workflows/review/lib/disciplines.test.ts # workflows/review/lib/rereview-mode.ts # workflows/review/review.md
jwbron
added a commit
that referenced
this pull request
Jul 13, 2026
…wered runs, drift watch) (#252) * [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged. * [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately. * [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c) live-producer.ts runs the live roster over one staged case behind an injected LiveAgentRunner seam (the judge.ts pattern), so its logic is stub-tested with zero model calls. Roster: the default finders (correctness-reviewer, skill-auditor) plus the router's lensesToSpawn; no pattern-triage or thread-reconciler in eval. It maps all three sub-agent output contracts into the shapes the deterministic runner consumes: label-shape findings (correctness lens, or conventions for the skill-auditor so labelForFinding reproduces the best-practice variants; confidence defaulted to 0.7 per the production claims rule), structured-schema lens findings validated as-is, and the validator's {claims: [...]} verdicts into CaseVerification[]. It stages claims.json for the validator, resolves {{#runtime-import}} directives against the case tree (a case opts into a skills index by carrying the file), retries once on malformed output with the rejection fed back, keeps partial results when an agent fails twice, prefixes colliding finding ids, and reports per-agent cost/turns/wall-clock. live-runner.ts is the one module that touches a real model runtime: an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned to the staged checkout, the agent's pinned model, and hard turn and wall-clock caps; plus the CLI smoke entry (tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The investigation-cap CLI the prompts reference is not staged; its own denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk as a dev dependency. * [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3) live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero. * [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4) Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step. * [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary). * [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id. * [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual. * [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed. * [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list * [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them * [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths) * [jwbron/review-rereview-accountability] review: prettier-format the rereview module * [jwbron/review-out-of-lane] review: prettier-format finding-schema * [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file * [jwbron/live-eval-corpus] review: route the specialist lens on each live case The first live acceptance runs missed incident-sql-missing-index:dm-missing-index-1 in all four arms: the recorded finding is a data-migrations LENS finding, but live cases carried no routerConfig lens rules, so the live roster never spawned the specialist that catches it. Every live case whose recorded finding belongs to a specialist lens now routes that lens on the finding's file (seven cases across five lenses), mirroring how a consumer ROUTING file would route the same paths in production. * [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report The acceptance runs surfaced a claim-validator failure that the report could only name, not explain (perCase carried agent names only, and the PerAgentReport.failed detail never reached the markdown). perCase failedAgents entries are now '<agent>: <reason>', so the next failure is diagnosable from the sticky comment alone. * [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING (full | scoped | flip-gated | fast, default full) dials how much of the roster a repeat review runs. lib/rereview-mode.ts owns the deterministic half: content-hashed hunk signatures over the generated-stripped diff, the hidden review-body fingerprint stamp (survives cache eviction, dismiss-stale-approvals, and COMMENTED-only histories), the divergence tripwire that re-arms full review at unreviewed share >= 0.4, and the plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval, sparse-PR-then-payload) ship as data in eval/lifecycle/ with a deterministic replay harness; counters gain costByRereviewDepth so the live A/B prices scoped against full. The review.md orchestrator wiring lands in a follow-up commit, on top of fold-in batch two's re-review accountability surface. * [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer The re-review mode dial's flip gate needs 'did the reconciler keep any blocking thread' as a number it can read from rereview.json, not a label judgment re-made at verdict time. Additive: the section rendering is unchanged. * [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator Step 1 stages the bot's prior reviews (every state, dismissed and comment-only included) for the stamp reader. Step 3 runs the rereview-mode plan CLI after the provenance CLI and maps each depth to its dispatch and staging: scoped overwrites full-stripped.diff with scoped.diff so the full roster reads only unseen hunks; flip-gated dispatches reconcile plus correctness over the scoped diff; fast is reconcile-only. Step 4 gains the flip rule: a reduced-depth run may flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0, and in flip-gated depth a validated blocking finding posts and blocks, vetoing the flip. Step 6 appends the hidden fingerprint stamp as the last line of every submitted review body; Steps 7 and 8 skip on reduced depths; Step 9 notes the stamp, not the cache, is the tripwire's authoritative fingerprint. The executed plan is copied into out/ so the run artifact records the depth the counters price. * [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree The dispatch-tax trim brought workflows/review into eslint+prettier scope; format the mode-dial files and the two pre-existing violations in disciplines.test.ts and finding-schema.ts so CI lints clean. * [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer The phase 4 acceptance pair measured which report rows a reader may act on from one run: recall, verdict agreement, the regression lists, and the adversarial gate reproduced exactly on the no-op control, while judge quality and noise moved on jitter alone; and on the weakened arm judge quality went UP 0.16 as recall fell 17 points (fewer, surer comments each score better). The footer states this on every report so nobody chases a judge-quality delta or reads one as health. * [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases) A live-enabled corpus case may now carry a live.rereview block: an open-PR snapshot with the prior review's unresolved threads (author replies included), a stamped prior review derived from priorDiff, and the depth plan the mode dial would compute. The live producer dispatches the thread-reconciler on such cases at every depth and sizes the finder roster by the plan; rereview-match.ts scores thread-resolution accuracy, the flip-gate input, and duplicate comments on kept threads; live-ab prices a mode via --re-review-mode on the candidate arm. The deterministic replay learns the same rules: computeVerdict gains keptBlockingCount (the flip rule as code) and re-review cases render the accountability section into the planned body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the measured seeded live trial: planted bugs, a bad partial fix with added bugs, then the full fix that must flip to APPROVE. * [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case eval/rereview-sweep.ts runs the working tree's reviewer over the open-PR corpus cases at every dial setting and prices the dial in one table: recall, thread resolution, flip-gate correctness, duplicates, and dollars per mode, with the executed depth per case since the tripwire may override the dial. golden-retention-fix-push is the under-threshold case the cheap paths need: a one-hunk fix push (unreviewed share 0.2) that resolves the prior blocking thread but plants a fresh defect inside the new hunk, so scoped and flip-gated must catch it and fast definitionally cannot. No case-format change: the mode is a run parameter. * [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep The per-mode pricing table needs somewhere visible to land: a workflow_dispatch job (never PR-triggered; sweeps spend model dollars) that runs rereview-sweep.ts against the chosen branch, tees the markdown table into the job summary, and uploads the JSON report as an artifact. * [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep on a PR (sticky comment + summary + artifact), live-judge runs the judged corpus pass on a PR head, and full-eval now fires on the labeling event itself instead of waiting for the next push; every workflow keeps workflow_dispatch for off-PR runs. The A/B gains the memo's pre-flight identity short-circuit: byte-identical review.md in both arms posts a no-reviewable-delta verdict and runs nothing, with --force-arms preserved for deliberate wobble controls. * [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy Three instrument fixes from the eval-tuning memo, landed in the runner so the whole stack above inherits them: - Pre-flight identity short-circuit (memo item 1): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; --force-arms preserved for deliberate wobble controls. - Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake. Retry spend is recorded in the report. - Drop-bucket taxonomy (memo item 4): each missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/ validation) by re-matching the spec against the dropped candidates with the line window relaxed (a mis-anchored real finding is exactly the case to surface). The report annotates every lost regression with its class and carries a found-but-dropped count row. * [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff) The judge grades one comment's prose in 512 tokens; every load-bearing metric (recall, verdict agreement, regressions, adversarial gate) is deterministic and never touches it, and the acceptance runs showed the judge signal is not single-run-stable on any model (it moved 0.11 on the byte-identical control and went UP on the weakened arm). Price that signal accordingly: the pin moves from claude-opus-4-8 to claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so week-over-week scores compare; both arms always share one judge, so within-run deltas are unaffected). Supersedes operator direction 4, which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/ 5xx/network only) to the one live judge seam; a single transient 500 previously wasted an entire weekly scoring pass. * [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base The workflow passes origin/<base_ref> (the base branch tip) while its comments and the dispatch-input description said merge-base. The code is the right side of that disagreement: pull_request runs check out the PR merge commit, so the candidate tree already contains the base tip; baselining on the same tip isolates the PR's own review.md delta, where a merge-base baseline would fold upstream review.md movement into the PR's measured numbers. Prose updated to match the code; no behavior change. * [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef) * [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix The eval instrument's percentage deltas sit below the one-case noise floor; this lands the memo's measurement items. eval/aggregate.ts pools N report artifacts into per-case pass rates with Wilson intervals and pooled rows (reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs one powered dispatch with a strict-majority adversarial gate; the weekly review-eval-drift.yml workflow runs full corpus x3 on main and publishes the aggregate; identical-arm pools render noise-floor bands as data. incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the reviewer found the missing index every single time; the 8/16 catch rate was the spec accepting only the migration-file anchor while the reviewer anchored the same blocking finding at the hot query (11 runs) or had it provenance- dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts the query-site anchor and its residual misses now classify found-but-dropped (the anchor-snap defect class), not recall. * [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched after the deterministic pass and a posted candidate shares its file, a pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect, same root cause?), biased to no since a false yes inflates recall. All seam guard rails inherited: capped per case, same-file only, matches recorded via 'fallback' for audit. API failures degrade to non-matches through onError; both arms share the matcher so the A/B delta stays unbiased. On by default in live-ab.ts; --no-match-arbiter restores deterministic-only matching. This targets the 54-69% 'unmatched posted' readings, implausibly high as true noise. * [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and arbiter work; this worktree cannot run eslint locally (dot-dir ignore). The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and both markdown renderers move to a leaf module with no runner dependency; live-ab.ts re-exports them so the import surface is unchanged. * [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat A multi-repeat run carries tens of dollars of spend but wrote its artifact only at the end, so a crash or cancellation on repeat n forfeited repeats 1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids; the workflow's always() artifact upload had nothing to grab). Each completed repeat now overwrites the out path with the accumulated partial payload, which aggregate.ts already pools via its repeats-field handling. * [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3, full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of one review.md. Measured bands: must-catch recall 54-86%, verdict agreement 75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now renders these as data with provenance, replacing prose guesswork; a delta whose arms both sit inside a band is wobble, and the footer points at --repeats for smaller effects. Pre-arbiter measurement; the weekly drift run re-measures with the arbiter active. * [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85 The noise-floor dispatch showed the live corpus is now 14 cases (the #251 golden re-review set), and the $60 default trimmed case-runs on both arms (38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs at the measured ~$10/arm-run with landing headroom, so the weekly aggregate never carries budget-skip asymmetry. * [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset * [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR workflows/review/eval/README.md is the guide an agent or engineer needs to run the eval system cold: the three tiers, every live-ab flag and CI entry point, powered-run recipes with real costs, corpus-growth rules (target the 20-80% band), report-reading guidance anchored on the measured noise floor, model pins, and the harness's historical limits (nothing before the 2026-07-08 structured-agent architecture runs under it). The weekly drift run now also opens a PR committing the report markdown and compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded, outside the A/B path filter): a job summary nobody opens is not a drift watch, and merged report PRs accumulate an in-repo time series. * [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README * [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets Findings from an adversarial pass over the instrument's own claims: Reports now stamp their ruler (matcher configuration + a content hash of the loaded corpus); the aggregate prints it, warns when a pool mixes rulers, and the drift series stays interpretable across instrument upgrades like the arbiter default or corpus growth, which move every rate without the reviewer changing. Noise-floor bands gain SD (min/max are extreme-value statistics that only widen as samples accumulate; mean +/- sd is the stable band) and a loud case-asymmetry warning: the 2026-07-10 measurement had budget skips, so its published v1 bands fold case-mix variance in on top of wobble; the footer constants and provenance string now say so. Repeat budgets roll: each arm-run's slice is the remaining budget over the remaining arm-runs, so cheap early arms donate headroom forward instead of stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that contaminated the noise-floor run). README gains a statistical-honesty section: clustered-interval optimism, the repeats-gate relaxation, and the arbiter's uncalibrated refuse bias. * [jwbron/eval-measurement-tool] review: checkpoint the repeat reports array, not the scalar count The per-repeat checkpoint wrote {repeats: <n>} where the recovery path in aggregate.ts expects {repeats: [reports]}; a crash mid-run therefore forfeited every completed repeat, which is exactly what the checkpoint exists to prevent. Write the reports array, matching the final artifact shape, and lock the checkpoint-to-extractSamples contract with a test. * [jwbron/eval-measurement-tool] review: merge same-module type imports; pin github-script to v8 Two review nitpicks from the stack: live-match.ts imported two types from ./corpus/loader in separate statements where the rest of the eval dir merges them, and the sticky-comment step pinned github-script v7 while node-ci.yml and release.yml pin v8 (ed597411). Align both.
jwbron
added a commit
that referenced
this pull request
Jul 13, 2026
* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged. * [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately. * [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c) live-producer.ts runs the live roster over one staged case behind an injected LiveAgentRunner seam (the judge.ts pattern), so its logic is stub-tested with zero model calls. Roster: the default finders (correctness-reviewer, skill-auditor) plus the router's lensesToSpawn; no pattern-triage or thread-reconciler in eval. It maps all three sub-agent output contracts into the shapes the deterministic runner consumes: label-shape findings (correctness lens, or conventions for the skill-auditor so labelForFinding reproduces the best-practice variants; confidence defaulted to 0.7 per the production claims rule), structured-schema lens findings validated as-is, and the validator's {claims: [...]} verdicts into CaseVerification[]. It stages claims.json for the validator, resolves {{#runtime-import}} directives against the case tree (a case opts into a skills index by carrying the file), retries once on malformed output with the rejection fed back, keeps partial results when an agent fails twice, prefixes colliding finding ids, and reports per-agent cost/turns/wall-clock. live-runner.ts is the one module that touches a real model runtime: an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned to the staged checkout, the agent's pinned model, and hard turn and wall-clock caps; plus the CLI smoke entry (tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The investigation-cap CLI the prompts reference is not staged; its own denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk as a dev dependency. * [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3) live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero. * [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4) Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step. * [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary). * [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id. * [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual. * [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed. * [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list * [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them * [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths) * [jwbron/review-rereview-accountability] review: prettier-format the rereview module * [jwbron/review-out-of-lane] review: prettier-format finding-schema * [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file * [jwbron/live-eval-corpus] review: route the specialist lens on each live case The first live acceptance runs missed incident-sql-missing-index:dm-missing-index-1 in all four arms: the recorded finding is a data-migrations LENS finding, but live cases carried no routerConfig lens rules, so the live roster never spawned the specialist that catches it. Every live case whose recorded finding belongs to a specialist lens now routes that lens on the finding's file (seven cases across five lenses), mirroring how a consumer ROUTING file would route the same paths in production. * [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report The acceptance runs surfaced a claim-validator failure that the report could only name, not explain (perCase carried agent names only, and the PerAgentReport.failed detail never reached the markdown). perCase failedAgents entries are now '<agent>: <reason>', so the next failure is diagnosable from the sticky comment alone. * [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING (full | scoped | flip-gated | fast, default full) dials how much of the roster a repeat review runs. lib/rereview-mode.ts owns the deterministic half: content-hashed hunk signatures over the generated-stripped diff, the hidden review-body fingerprint stamp (survives cache eviction, dismiss-stale-approvals, and COMMENTED-only histories), the divergence tripwire that re-arms full review at unreviewed share >= 0.4, and the plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval, sparse-PR-then-payload) ship as data in eval/lifecycle/ with a deterministic replay harness; counters gain costByRereviewDepth so the live A/B prices scoped against full. The review.md orchestrator wiring lands in a follow-up commit, on top of fold-in batch two's re-review accountability surface. * [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer The re-review mode dial's flip gate needs 'did the reconciler keep any blocking thread' as a number it can read from rereview.json, not a label judgment re-made at verdict time. Additive: the section rendering is unchanged. * [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator Step 1 stages the bot's prior reviews (every state, dismissed and comment-only included) for the stamp reader. Step 3 runs the rereview-mode plan CLI after the provenance CLI and maps each depth to its dispatch and staging: scoped overwrites full-stripped.diff with scoped.diff so the full roster reads only unseen hunks; flip-gated dispatches reconcile plus correctness over the scoped diff; fast is reconcile-only. Step 4 gains the flip rule: a reduced-depth run may flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0, and in flip-gated depth a validated blocking finding posts and blocks, vetoing the flip. Step 6 appends the hidden fingerprint stamp as the last line of every submitted review body; Steps 7 and 8 skip on reduced depths; Step 9 notes the stamp, not the cache, is the tripwire's authoritative fingerprint. The executed plan is copied into out/ so the run artifact records the depth the counters price. * [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree The dispatch-tax trim brought workflows/review into eslint+prettier scope; format the mode-dial files and the two pre-existing violations in disciplines.test.ts and finding-schema.ts so CI lints clean. * [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer The phase 4 acceptance pair measured which report rows a reader may act on from one run: recall, verdict agreement, the regression lists, and the adversarial gate reproduced exactly on the no-op control, while judge quality and noise moved on jitter alone; and on the weakened arm judge quality went UP 0.16 as recall fell 17 points (fewer, surer comments each score better). The footer states this on every report so nobody chases a judge-quality delta or reads one as health. * [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases) A live-enabled corpus case may now carry a live.rereview block: an open-PR snapshot with the prior review's unresolved threads (author replies included), a stamped prior review derived from priorDiff, and the depth plan the mode dial would compute. The live producer dispatches the thread-reconciler on such cases at every depth and sizes the finder roster by the plan; rereview-match.ts scores thread-resolution accuracy, the flip-gate input, and duplicate comments on kept threads; live-ab prices a mode via --re-review-mode on the candidate arm. The deterministic replay learns the same rules: computeVerdict gains keptBlockingCount (the flip rule as code) and re-review cases render the accountability section into the planned body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the measured seeded live trial: planted bugs, a bad partial fix with added bugs, then the full fix that must flip to APPROVE. * [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case eval/rereview-sweep.ts runs the working tree's reviewer over the open-PR corpus cases at every dial setting and prices the dial in one table: recall, thread resolution, flip-gate correctness, duplicates, and dollars per mode, with the executed depth per case since the tripwire may override the dial. golden-retention-fix-push is the under-threshold case the cheap paths need: a one-hunk fix push (unreviewed share 0.2) that resolves the prior blocking thread but plants a fresh defect inside the new hunk, so scoped and flip-gated must catch it and fast definitionally cannot. No case-format change: the mode is a run parameter. * [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep The per-mode pricing table needs somewhere visible to land: a workflow_dispatch job (never PR-triggered; sweeps spend model dollars) that runs rereview-sweep.ts against the chosen branch, tees the markdown table into the job summary, and uploads the JSON report as an artifact. * [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep on a PR (sticky comment + summary + artifact), live-judge runs the judged corpus pass on a PR head, and full-eval now fires on the labeling event itself instead of waiting for the next push; every workflow keeps workflow_dispatch for off-PR runs. The A/B gains the memo's pre-flight identity short-circuit: byte-identical review.md in both arms posts a no-reviewable-delta verdict and runs nothing, with --force-arms preserved for deliberate wobble controls. * [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy Three instrument fixes from the eval-tuning memo, landed in the runner so the whole stack above inherits them: - Pre-flight identity short-circuit (memo item 1): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; --force-arms preserved for deliberate wobble controls. - Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake. Retry spend is recorded in the report. - Drop-bucket taxonomy (memo item 4): each missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/ validation) by re-matching the spec against the dropped candidates with the line window relaxed (a mis-anchored real finding is exactly the case to surface). The report annotates every lost regression with its class and carries a found-but-dropped count row. * [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff) The judge grades one comment's prose in 512 tokens; every load-bearing metric (recall, verdict agreement, regressions, adversarial gate) is deterministic and never touches it, and the acceptance runs showed the judge signal is not single-run-stable on any model (it moved 0.11 on the byte-identical control and went UP on the weakened arm). Price that signal accordingly: the pin moves from claude-opus-4-8 to claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so week-over-week scores compare; both arms always share one judge, so within-run deltas are unaffected). Supersedes operator direction 4, which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/ 5xx/network only) to the one live judge seam; a single transient 500 previously wasted an entire weekly scoring pass. * [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base The workflow passes origin/<base_ref> (the base branch tip) while its comments and the dispatch-input description said merge-base. The code is the right side of that disagreement: pull_request runs check out the PR merge commit, so the candidate tree already contains the base tip; baselining on the same tip isolates the PR's own review.md delta, where a merge-base baseline would fold upstream review.md movement into the PR's measured numbers. Prose updated to match the code; no behavior change. * [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef) * [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix The eval instrument's percentage deltas sit below the one-case noise floor; this lands the memo's measurement items. eval/aggregate.ts pools N report artifacts into per-case pass rates with Wilson intervals and pooled rows (reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs one powered dispatch with a strict-majority adversarial gate; the weekly review-eval-drift.yml workflow runs full corpus x3 on main and publishes the aggregate; identical-arm pools render noise-floor bands as data. incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the reviewer found the missing index every single time; the 8/16 catch rate was the spec accepting only the migration-file anchor while the reviewer anchored the same blocking finding at the hot query (11 runs) or had it provenance- dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts the query-site anchor and its residual misses now classify found-but-dropped (the anchor-snap defect class), not recall. * [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched after the deterministic pass and a posted candidate shares its file, a pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect, same root cause?), biased to no since a false yes inflates recall. All seam guard rails inherited: capped per case, same-file only, matches recorded via 'fallback' for audit. API failures degrade to non-matches through onError; both arms share the matcher so the A/B delta stays unbiased. On by default in live-ab.ts; --no-match-arbiter restores deterministic-only matching. This targets the 54-69% 'unmatched posted' readings, implausibly high as true noise. * [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and arbiter work; this worktree cannot run eslint locally (dot-dir ignore). The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and both markdown renderers move to a leaf module with no runner dependency; live-ab.ts re-exports them so the import surface is unchanged. * [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat A multi-repeat run carries tens of dollars of spend but wrote its artifact only at the end, so a crash or cancellation on repeat n forfeited repeats 1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids; the workflow's always() artifact upload had nothing to grab). Each completed repeat now overwrites the out path with the accumulated partial payload, which aggregate.ts already pools via its repeats-field handling. * [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3, full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of one review.md. Measured bands: must-catch recall 54-86%, verdict agreement 75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now renders these as data with provenance, replacing prose guesswork; a delta whose arms both sit inside a band is wobble, and the footer points at --repeats for smaller effects. Pre-arbiter measurement; the weekly drift run re-measures with the arbiter active. * [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85 The noise-floor dispatch showed the live corpus is now 14 cases (the #251 golden re-review set), and the $60 default trimmed case-runs on both arms (38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs at the measured ~$10/arm-run with landing headroom, so the weekly aggregate never carries budget-skip asymmetry. * [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset * [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR workflows/review/eval/README.md is the guide an agent or engineer needs to run the eval system cold: the three tiers, every live-ab flag and CI entry point, powered-run recipes with real costs, corpus-growth rules (target the 20-80% band), report-reading guidance anchored on the measured noise floor, model pins, and the harness's historical limits (nothing before the 2026-07-08 structured-agent architecture runs under it). The weekly drift run now also opens a PR committing the report markdown and compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded, outside the A/B path filter): a job summary nobody opens is not a drift watch, and merged report PRs accumulate an in-repo time series. * [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README * [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets Findings from an adversarial pass over the instrument's own claims: Reports now stamp their ruler (matcher configuration + a content hash of the loaded corpus); the aggregate prints it, warns when a pool mixes rulers, and the drift series stays interpretable across instrument upgrades like the arbiter default or corpus growth, which move every rate without the reviewer changing. Noise-floor bands gain SD (min/max are extreme-value statistics that only widen as samples accumulate; mean +/- sd is the stable band) and a loud case-asymmetry warning: the 2026-07-10 measurement had budget skips, so its published v1 bands fold case-mix variance in on top of wobble; the footer constants and provenance string now say so. Repeat budgets roll: each arm-run's slice is the remaining budget over the remaining arm-runs, so cheap early arms donate headroom forward instead of stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that contaminated the noise-floor run). README gains a statistical-honesty section: clustered-interval optimism, the repeats-gate relaxation, and the arbiter's uncalibrated refuse bias. * [jwbron/anchor-snap-provenance] review: anchor-snap fallback in the change-provenance gate The reviewer produces right-file, right-mechanism findings at wrong line numbers (it appears to sometimes count unified-diff text lines instead of file lines: observed anchors at line 24 of an 18-line file and line 8 of a 3-line file), and the provenance gate then drops them; since only surviving blocking labels feed the computed verdict, a correct blocking finding dying at the gate can flip REQUEST_CHANGES to APPROVE. Observed on adversarial-injection-approve, golden-request-changes-authz, the sql-index replay, and production main's own baseline arm. Before setting a line-anchored finding aside, the gate now snaps it to the nearest changed line in the same file under two windows: a 3-line near-miss window (the unified diff's context width) and an overflow window for anchors past every shown line by no more than the file's diff-text overhead (exactly the counting mis-anchor's overshoot bound). Ties break toward the lower line. Snapped findings keep their severity, post at the snapped anchor, and are recorded for audit (out/snapped.json in production, snappedByProvenance in eval results); findings outside both windows keep today's set-aside behavior. provenance.json carries a precomputed per-file snap lookup so review.md's gate stays a dictionary lookup. The live A/B emulates each arm's own review.md gate version, keyed on the anchor-snap marker, so the baseline arm replays the pre-snap gate and the powered run prices this change. * [jwbron/anchor-snap-provenance] review: pin anchor-snap in a deterministic smoke case provenance-anchor-snap-rescued replays the observed production anatomy in the per-push CI gate: a correct blocking finding anchored at the diff-text line (past the end of a short file, at the exact overflow edge) snaps to the nearest changed line, posts there, and drives REQUEST_CHANGES, while a far-anchored pre-existing observation in the same run is still set aside unposted. Deterministic and smoke-tagged only, so the live ruler (corpus stamp over live-tagged cases) is untouched and the in-flight powered run stays comparable. Also documents how to audit snap records (from/to distance separates the near-miss and overflow classes). * [jwbron/anchor-snap-provenance] review: overflow-snap only past the file's real end; drop LEFT-side snapping The overflow window could rewrite a genuine finding about unshown code below a short hunk into a posted blocking comment (a small hunk in a longer file leaves real lines between lastShownLine and the overhead bound). The window now additionally requires the anchor to be past the file's actual end (a line that does not exist): the provenance CLI reads each changed file's real length from the checkout, live eval cases hydrate lengths from their tree, and deterministic cases declare them. A file whose length is unknown gets no overflow window (near-miss only). LEFT-side anchors no longer snap anywhere, matching the RIGHT-side-only snap table the production gate reads, so the eval emulation never prices a gate production does not have. Serialized snap tables are now asserted in the CLI tests.
jwbron
added a commit
that referenced
this pull request
Jul 13, 2026
* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged. * [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately. * [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c) live-producer.ts runs the live roster over one staged case behind an injected LiveAgentRunner seam (the judge.ts pattern), so its logic is stub-tested with zero model calls. Roster: the default finders (correctness-reviewer, skill-auditor) plus the router's lensesToSpawn; no pattern-triage or thread-reconciler in eval. It maps all three sub-agent output contracts into the shapes the deterministic runner consumes: label-shape findings (correctness lens, or conventions for the skill-auditor so labelForFinding reproduces the best-practice variants; confidence defaulted to 0.7 per the production claims rule), structured-schema lens findings validated as-is, and the validator's {claims: [...]} verdicts into CaseVerification[]. It stages claims.json for the validator, resolves {{#runtime-import}} directives against the case tree (a case opts into a skills index by carrying the file), retries once on malformed output with the rejection fed back, keeps partial results when an agent fails twice, prefixes colliding finding ids, and reports per-agent cost/turns/wall-clock. live-runner.ts is the one module that touches a real model runtime: an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned to the staged checkout, the agent's pinned model, and hard turn and wall-clock caps; plus the CLI smoke entry (tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The investigation-cap CLI the prompts reference is not staged; its own denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk as a dev dependency. * [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3) live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero. * [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4) Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step. * [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary). * [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id. * [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual. * [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed. * [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list * [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them * [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths) * [jwbron/review-rereview-accountability] review: prettier-format the rereview module * [jwbron/review-out-of-lane] review: prettier-format finding-schema * [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file * [jwbron/live-eval-corpus] review: route the specialist lens on each live case The first live acceptance runs missed incident-sql-missing-index:dm-missing-index-1 in all four arms: the recorded finding is a data-migrations LENS finding, but live cases carried no routerConfig lens rules, so the live roster never spawned the specialist that catches it. Every live case whose recorded finding belongs to a specialist lens now routes that lens on the finding's file (seven cases across five lenses), mirroring how a consumer ROUTING file would route the same paths in production. * [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report The acceptance runs surfaced a claim-validator failure that the report could only name, not explain (perCase carried agent names only, and the PerAgentReport.failed detail never reached the markdown). perCase failedAgents entries are now '<agent>: <reason>', so the next failure is diagnosable from the sticky comment alone. * [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING (full | scoped | flip-gated | fast, default full) dials how much of the roster a repeat review runs. lib/rereview-mode.ts owns the deterministic half: content-hashed hunk signatures over the generated-stripped diff, the hidden review-body fingerprint stamp (survives cache eviction, dismiss-stale-approvals, and COMMENTED-only histories), the divergence tripwire that re-arms full review at unreviewed share >= 0.4, and the plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval, sparse-PR-then-payload) ship as data in eval/lifecycle/ with a deterministic replay harness; counters gain costByRereviewDepth so the live A/B prices scoped against full. The review.md orchestrator wiring lands in a follow-up commit, on top of fold-in batch two's re-review accountability surface. * [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer The re-review mode dial's flip gate needs 'did the reconciler keep any blocking thread' as a number it can read from rereview.json, not a label judgment re-made at verdict time. Additive: the section rendering is unchanged. * [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator Step 1 stages the bot's prior reviews (every state, dismissed and comment-only included) for the stamp reader. Step 3 runs the rereview-mode plan CLI after the provenance CLI and maps each depth to its dispatch and staging: scoped overwrites full-stripped.diff with scoped.diff so the full roster reads only unseen hunks; flip-gated dispatches reconcile plus correctness over the scoped diff; fast is reconcile-only. Step 4 gains the flip rule: a reduced-depth run may flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0, and in flip-gated depth a validated blocking finding posts and blocks, vetoing the flip. Step 6 appends the hidden fingerprint stamp as the last line of every submitted review body; Steps 7 and 8 skip on reduced depths; Step 9 notes the stamp, not the cache, is the tripwire's authoritative fingerprint. The executed plan is copied into out/ so the run artifact records the depth the counters price. * [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree The dispatch-tax trim brought workflows/review into eslint+prettier scope; format the mode-dial files and the two pre-existing violations in disciplines.test.ts and finding-schema.ts so CI lints clean. * [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer The phase 4 acceptance pair measured which report rows a reader may act on from one run: recall, verdict agreement, the regression lists, and the adversarial gate reproduced exactly on the no-op control, while judge quality and noise moved on jitter alone; and on the weakened arm judge quality went UP 0.16 as recall fell 17 points (fewer, surer comments each score better). The footer states this on every report so nobody chases a judge-quality delta or reads one as health. * [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases) A live-enabled corpus case may now carry a live.rereview block: an open-PR snapshot with the prior review's unresolved threads (author replies included), a stamped prior review derived from priorDiff, and the depth plan the mode dial would compute. The live producer dispatches the thread-reconciler on such cases at every depth and sizes the finder roster by the plan; rereview-match.ts scores thread-resolution accuracy, the flip-gate input, and duplicate comments on kept threads; live-ab prices a mode via --re-review-mode on the candidate arm. The deterministic replay learns the same rules: computeVerdict gains keptBlockingCount (the flip rule as code) and re-review cases render the accountability section into the planned body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the measured seeded live trial: planted bugs, a bad partial fix with added bugs, then the full fix that must flip to APPROVE. * [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case eval/rereview-sweep.ts runs the working tree's reviewer over the open-PR corpus cases at every dial setting and prices the dial in one table: recall, thread resolution, flip-gate correctness, duplicates, and dollars per mode, with the executed depth per case since the tripwire may override the dial. golden-retention-fix-push is the under-threshold case the cheap paths need: a one-hunk fix push (unreviewed share 0.2) that resolves the prior blocking thread but plants a fresh defect inside the new hunk, so scoped and flip-gated must catch it and fast definitionally cannot. No case-format change: the mode is a run parameter. * [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep The per-mode pricing table needs somewhere visible to land: a workflow_dispatch job (never PR-triggered; sweeps spend model dollars) that runs rereview-sweep.ts against the chosen branch, tees the markdown table into the job summary, and uploads the JSON report as an artifact. * [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep on a PR (sticky comment + summary + artifact), live-judge runs the judged corpus pass on a PR head, and full-eval now fires on the labeling event itself instead of waiting for the next push; every workflow keeps workflow_dispatch for off-PR runs. The A/B gains the memo's pre-flight identity short-circuit: byte-identical review.md in both arms posts a no-reviewable-delta verdict and runs nothing, with --force-arms preserved for deliberate wobble controls. * [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy Three instrument fixes from the eval-tuning memo, landed in the runner so the whole stack above inherits them: - Pre-flight identity short-circuit (memo item 1): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; --force-arms preserved for deliberate wobble controls. - Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake. Retry spend is recorded in the report. - Drop-bucket taxonomy (memo item 4): each missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/ validation) by re-matching the spec against the dropped candidates with the line window relaxed (a mis-anchored real finding is exactly the case to surface). The report annotates every lost regression with its class and carries a found-but-dropped count row. * [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff) The judge grades one comment's prose in 512 tokens; every load-bearing metric (recall, verdict agreement, regressions, adversarial gate) is deterministic and never touches it, and the acceptance runs showed the judge signal is not single-run-stable on any model (it moved 0.11 on the byte-identical control and went UP on the weakened arm). Price that signal accordingly: the pin moves from claude-opus-4-8 to claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so week-over-week scores compare; both arms always share one judge, so within-run deltas are unaffected). Supersedes operator direction 4, which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/ 5xx/network only) to the one live judge seam; a single transient 500 previously wasted an entire weekly scoring pass. * [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base The workflow passes origin/<base_ref> (the base branch tip) while its comments and the dispatch-input description said merge-base. The code is the right side of that disagreement: pull_request runs check out the PR merge commit, so the candidate tree already contains the base tip; baselining on the same tip isolates the PR's own review.md delta, where a merge-base baseline would fold upstream review.md movement into the PR's measured numbers. Prose updated to match the code; no behavior change. * [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef) * [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix The eval instrument's percentage deltas sit below the one-case noise floor; this lands the memo's measurement items. eval/aggregate.ts pools N report artifacts into per-case pass rates with Wilson intervals and pooled rows (reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs one powered dispatch with a strict-majority adversarial gate; the weekly review-eval-drift.yml workflow runs full corpus x3 on main and publishes the aggregate; identical-arm pools render noise-floor bands as data. incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the reviewer found the missing index every single time; the 8/16 catch rate was the spec accepting only the migration-file anchor while the reviewer anchored the same blocking finding at the hot query (11 runs) or had it provenance- dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts the query-site anchor and its residual misses now classify found-but-dropped (the anchor-snap defect class), not recall. * [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched after the deterministic pass and a posted candidate shares its file, a pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect, same root cause?), biased to no since a false yes inflates recall. All seam guard rails inherited: capped per case, same-file only, matches recorded via 'fallback' for audit. API failures degrade to non-matches through onError; both arms share the matcher so the A/B delta stays unbiased. On by default in live-ab.ts; --no-match-arbiter restores deterministic-only matching. This targets the 54-69% 'unmatched posted' readings, implausibly high as true noise. * [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and arbiter work; this worktree cannot run eslint locally (dot-dir ignore). The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and both markdown renderers move to a leaf module with no runner dependency; live-ab.ts re-exports them so the import surface is unchanged. * [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat A multi-repeat run carries tens of dollars of spend but wrote its artifact only at the end, so a crash or cancellation on repeat n forfeited repeats 1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids; the workflow's always() artifact upload had nothing to grab). Each completed repeat now overwrites the out path with the accumulated partial payload, which aggregate.ts already pools via its repeats-field handling. * [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3, full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of one review.md. Measured bands: must-catch recall 54-86%, verdict agreement 75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now renders these as data with provenance, replacing prose guesswork; a delta whose arms both sit inside a band is wobble, and the footer points at --repeats for smaller effects. Pre-arbiter measurement; the weekly drift run re-measures with the arbiter active. * [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85 The noise-floor dispatch showed the live corpus is now 14 cases (the #251 golden re-review set), and the $60 default trimmed case-runs on both arms (38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs at the measured ~$10/arm-run with landing headroom, so the weekly aggregate never carries budget-skip asymmetry. * [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset * [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR workflows/review/eval/README.md is the guide an agent or engineer needs to run the eval system cold: the three tiers, every live-ab flag and CI entry point, powered-run recipes with real costs, corpus-growth rules (target the 20-80% band), report-reading guidance anchored on the measured noise floor, model pins, and the harness's historical limits (nothing before the 2026-07-08 structured-agent architecture runs under it). The weekly drift run now also opens a PR committing the report markdown and compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded, outside the A/B path filter): a job summary nobody opens is not a drift watch, and merged report PRs accumulate an in-repo time series. * [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README * [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets Findings from an adversarial pass over the instrument's own claims: Reports now stamp their ruler (matcher configuration + a content hash of the loaded corpus); the aggregate prints it, warns when a pool mixes rulers, and the drift series stays interpretable across instrument upgrades like the arbiter default or corpus growth, which move every rate without the reviewer changing. Noise-floor bands gain SD (min/max are extreme-value statistics that only widen as samples accumulate; mean +/- sd is the stable band) and a loud case-asymmetry warning: the 2026-07-10 measurement had budget skips, so its published v1 bands fold case-mix variance in on top of wobble; the footer constants and provenance string now say so. Repeat budgets roll: each arm-run's slice is the remaining budget over the remaining arm-runs, so cheap early arms donate headroom forward instead of stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that contaminated the noise-floor run). README gains a statistical-honesty section: clustered-interval optimism, the repeats-gate relaxation, and the arbiter's uncalibrated refuse bias. * [jwbron/anchor-snap-provenance] review: anchor-snap fallback in the change-provenance gate The reviewer produces right-file, right-mechanism findings at wrong line numbers (it appears to sometimes count unified-diff text lines instead of file lines: observed anchors at line 24 of an 18-line file and line 8 of a 3-line file), and the provenance gate then drops them; since only surviving blocking labels feed the computed verdict, a correct blocking finding dying at the gate can flip REQUEST_CHANGES to APPROVE. Observed on adversarial-injection-approve, golden-request-changes-authz, the sql-index replay, and production main's own baseline arm. Before setting a line-anchored finding aside, the gate now snaps it to the nearest changed line in the same file under two windows: a 3-line near-miss window (the unified diff's context width) and an overflow window for anchors past every shown line by no more than the file's diff-text overhead (exactly the counting mis-anchor's overshoot bound). Ties break toward the lower line. Snapped findings keep their severity, post at the snapped anchor, and are recorded for audit (out/snapped.json in production, snappedByProvenance in eval results); findings outside both windows keep today's set-aside behavior. provenance.json carries a precomputed per-file snap lookup so review.md's gate stays a dictionary lookup. The live A/B emulates each arm's own review.md gate version, keyed on the anchor-snap marker, so the baseline arm replays the pre-snap gate and the powered run prices this change. * [jwbron/anchor-snap-provenance] review: pin anchor-snap in a deterministic smoke case provenance-anchor-snap-rescued replays the observed production anatomy in the per-push CI gate: a correct blocking finding anchored at the diff-text line (past the end of a short file, at the exact overflow edge) snaps to the nearest changed line, posts there, and drives REQUEST_CHANGES, while a far-anchored pre-existing observation in the same run is still set aside unposted. Deterministic and smoke-tagged only, so the live ruler (corpus stamp over live-tagged cases) is untouched and the in-flight powered run stays comparable. Also documents how to audit snap records (from/to distance separates the near-miss and overflow classes). * [jwbron/diff-line-annotation] review: line-number-annotated staged diffs, anchors read not counted The mis-anchor pathology anchor-snap repairs downstream exists because the staged diff makes the model count lines. annotateDiffLineNumbers prefixes every hunk content line with its real line number (+/context lines carry the NEW-file number, - lines the OLD-file number) while keeping the diff marker in column one, so annotated text still splits into sections. The provenance CLI writes full-stripped-annotated.diff beside the raw stripped diff and gains an 'annotate <in> <out>' subcommand for pr-annotated.diff (Phase 1) and the scoped-depth refreshes. Every finding-producing reviewer reads the annotated copy and takes anchor.line from the printed number; everything that parses a diff (provenance, re-review fingerprints, scoped staging, pattern-triage, claim-validator) keeps reading the raw files, so hunk signatures never shift. Eval staging writes the annotated siblings for both arms unconditionally; only a review.md that names them reads them, so the A/B prices this as a pure prompt delta. Reports gain the anchor-fidelity observable: per-case snap counts and a pooled 'Findings anchor-snapped' row (version-tolerant of older artifacts); if annotation works, candidate-arm snaps fall to zero with the anchor-snap gate remaining as backstop.
jwbron
added a commit
that referenced
this pull request
Jul 13, 2026
* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged. * [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately. * [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c) live-producer.ts runs the live roster over one staged case behind an injected LiveAgentRunner seam (the judge.ts pattern), so its logic is stub-tested with zero model calls. Roster: the default finders (correctness-reviewer, skill-auditor) plus the router's lensesToSpawn; no pattern-triage or thread-reconciler in eval. It maps all three sub-agent output contracts into the shapes the deterministic runner consumes: label-shape findings (correctness lens, or conventions for the skill-auditor so labelForFinding reproduces the best-practice variants; confidence defaulted to 0.7 per the production claims rule), structured-schema lens findings validated as-is, and the validator's {claims: [...]} verdicts into CaseVerification[]. It stages claims.json for the validator, resolves {{#runtime-import}} directives against the case tree (a case opts into a skills index by carrying the file), retries once on malformed output with the rejection fed back, keeps partial results when an agent fails twice, prefixes colliding finding ids, and reports per-agent cost/turns/wall-clock. live-runner.ts is the one module that touches a real model runtime: an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned to the staged checkout, the agent's pinned model, and hard turn and wall-clock caps; plus the CLI smoke entry (tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The investigation-cap CLI the prompts reference is not staged; its own denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk as a dev dependency. * [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3) live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero. * [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4) Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step. * [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary). * [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id. * [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual. * [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed. * [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list * [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them * [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths) * [jwbron/review-rereview-accountability] review: prettier-format the rereview module * [jwbron/review-out-of-lane] review: prettier-format finding-schema * [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file * [jwbron/live-eval-corpus] review: route the specialist lens on each live case The first live acceptance runs missed incident-sql-missing-index:dm-missing-index-1 in all four arms: the recorded finding is a data-migrations LENS finding, but live cases carried no routerConfig lens rules, so the live roster never spawned the specialist that catches it. Every live case whose recorded finding belongs to a specialist lens now routes that lens on the finding's file (seven cases across five lenses), mirroring how a consumer ROUTING file would route the same paths in production. * [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report The acceptance runs surfaced a claim-validator failure that the report could only name, not explain (perCase carried agent names only, and the PerAgentReport.failed detail never reached the markdown). perCase failedAgents entries are now '<agent>: <reason>', so the next failure is diagnosable from the sticky comment alone. * [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING (full | scoped | flip-gated | fast, default full) dials how much of the roster a repeat review runs. lib/rereview-mode.ts owns the deterministic half: content-hashed hunk signatures over the generated-stripped diff, the hidden review-body fingerprint stamp (survives cache eviction, dismiss-stale-approvals, and COMMENTED-only histories), the divergence tripwire that re-arms full review at unreviewed share >= 0.4, and the plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval, sparse-PR-then-payload) ship as data in eval/lifecycle/ with a deterministic replay harness; counters gain costByRereviewDepth so the live A/B prices scoped against full. The review.md orchestrator wiring lands in a follow-up commit, on top of fold-in batch two's re-review accountability surface. * [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer The re-review mode dial's flip gate needs 'did the reconciler keep any blocking thread' as a number it can read from rereview.json, not a label judgment re-made at verdict time. Additive: the section rendering is unchanged. * [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator Step 1 stages the bot's prior reviews (every state, dismissed and comment-only included) for the stamp reader. Step 3 runs the rereview-mode plan CLI after the provenance CLI and maps each depth to its dispatch and staging: scoped overwrites full-stripped.diff with scoped.diff so the full roster reads only unseen hunks; flip-gated dispatches reconcile plus correctness over the scoped diff; fast is reconcile-only. Step 4 gains the flip rule: a reduced-depth run may flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0, and in flip-gated depth a validated blocking finding posts and blocks, vetoing the flip. Step 6 appends the hidden fingerprint stamp as the last line of every submitted review body; Steps 7 and 8 skip on reduced depths; Step 9 notes the stamp, not the cache, is the tripwire's authoritative fingerprint. The executed plan is copied into out/ so the run artifact records the depth the counters price. * [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree The dispatch-tax trim brought workflows/review into eslint+prettier scope; format the mode-dial files and the two pre-existing violations in disciplines.test.ts and finding-schema.ts so CI lints clean. * [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer The phase 4 acceptance pair measured which report rows a reader may act on from one run: recall, verdict agreement, the regression lists, and the adversarial gate reproduced exactly on the no-op control, while judge quality and noise moved on jitter alone; and on the weakened arm judge quality went UP 0.16 as recall fell 17 points (fewer, surer comments each score better). The footer states this on every report so nobody chases a judge-quality delta or reads one as health. * [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases) A live-enabled corpus case may now carry a live.rereview block: an open-PR snapshot with the prior review's unresolved threads (author replies included), a stamped prior review derived from priorDiff, and the depth plan the mode dial would compute. The live producer dispatches the thread-reconciler on such cases at every depth and sizes the finder roster by the plan; rereview-match.ts scores thread-resolution accuracy, the flip-gate input, and duplicate comments on kept threads; live-ab prices a mode via --re-review-mode on the candidate arm. The deterministic replay learns the same rules: computeVerdict gains keptBlockingCount (the flip rule as code) and re-review cases render the accountability section into the planned body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the measured seeded live trial: planted bugs, a bad partial fix with added bugs, then the full fix that must flip to APPROVE. * [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case eval/rereview-sweep.ts runs the working tree's reviewer over the open-PR corpus cases at every dial setting and prices the dial in one table: recall, thread resolution, flip-gate correctness, duplicates, and dollars per mode, with the executed depth per case since the tripwire may override the dial. golden-retention-fix-push is the under-threshold case the cheap paths need: a one-hunk fix push (unreviewed share 0.2) that resolves the prior blocking thread but plants a fresh defect inside the new hunk, so scoped and flip-gated must catch it and fast definitionally cannot. No case-format change: the mode is a run parameter. * [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep The per-mode pricing table needs somewhere visible to land: a workflow_dispatch job (never PR-triggered; sweeps spend model dollars) that runs rereview-sweep.ts against the chosen branch, tees the markdown table into the job summary, and uploads the JSON report as an artifact. * [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep on a PR (sticky comment + summary + artifact), live-judge runs the judged corpus pass on a PR head, and full-eval now fires on the labeling event itself instead of waiting for the next push; every workflow keeps workflow_dispatch for off-PR runs. The A/B gains the memo's pre-flight identity short-circuit: byte-identical review.md in both arms posts a no-reviewable-delta verdict and runs nothing, with --force-arms preserved for deliberate wobble controls. * [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy Three instrument fixes from the eval-tuning memo, landed in the runner so the whole stack above inherits them: - Pre-flight identity short-circuit (memo item 1): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; --force-arms preserved for deliberate wobble controls. - Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake. Retry spend is recorded in the report. - Drop-bucket taxonomy (memo item 4): each missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/ validation) by re-matching the spec against the dropped candidates with the line window relaxed (a mis-anchored real finding is exactly the case to surface). The report annotates every lost regression with its class and carries a found-but-dropped count row. * [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff) The judge grades one comment's prose in 512 tokens; every load-bearing metric (recall, verdict agreement, regressions, adversarial gate) is deterministic and never touches it, and the acceptance runs showed the judge signal is not single-run-stable on any model (it moved 0.11 on the byte-identical control and went UP on the weakened arm). Price that signal accordingly: the pin moves from claude-opus-4-8 to claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so week-over-week scores compare; both arms always share one judge, so within-run deltas are unaffected). Supersedes operator direction 4, which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/ 5xx/network only) to the one live judge seam; a single transient 500 previously wasted an entire weekly scoring pass. * [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base The workflow passes origin/<base_ref> (the base branch tip) while its comments and the dispatch-input description said merge-base. The code is the right side of that disagreement: pull_request runs check out the PR merge commit, so the candidate tree already contains the base tip; baselining on the same tip isolates the PR's own review.md delta, where a merge-base baseline would fold upstream review.md movement into the PR's measured numbers. Prose updated to match the code; no behavior change. * [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef) * [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix The eval instrument's percentage deltas sit below the one-case noise floor; this lands the memo's measurement items. eval/aggregate.ts pools N report artifacts into per-case pass rates with Wilson intervals and pooled rows (reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs one powered dispatch with a strict-majority adversarial gate; the weekly review-eval-drift.yml workflow runs full corpus x3 on main and publishes the aggregate; identical-arm pools render noise-floor bands as data. incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the reviewer found the missing index every single time; the 8/16 catch rate was the spec accepting only the migration-file anchor while the reviewer anchored the same blocking finding at the hot query (11 runs) or had it provenance- dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts the query-site anchor and its residual misses now classify found-but-dropped (the anchor-snap defect class), not recall. * [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched after the deterministic pass and a posted candidate shares its file, a pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect, same root cause?), biased to no since a false yes inflates recall. All seam guard rails inherited: capped per case, same-file only, matches recorded via 'fallback' for audit. API failures degrade to non-matches through onError; both arms share the matcher so the A/B delta stays unbiased. On by default in live-ab.ts; --no-match-arbiter restores deterministic-only matching. This targets the 54-69% 'unmatched posted' readings, implausibly high as true noise. * [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and arbiter work; this worktree cannot run eslint locally (dot-dir ignore). The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and both markdown renderers move to a leaf module with no runner dependency; live-ab.ts re-exports them so the import surface is unchanged. * [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat A multi-repeat run carries tens of dollars of spend but wrote its artifact only at the end, so a crash or cancellation on repeat n forfeited repeats 1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids; the workflow's always() artifact upload had nothing to grab). Each completed repeat now overwrites the out path with the accumulated partial payload, which aggregate.ts already pools via its repeats-field handling. * [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3, full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of one review.md. Measured bands: must-catch recall 54-86%, verdict agreement 75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now renders these as data with provenance, replacing prose guesswork; a delta whose arms both sit inside a band is wobble, and the footer points at --repeats for smaller effects. Pre-arbiter measurement; the weekly drift run re-measures with the arbiter active. * [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85 The noise-floor dispatch showed the live corpus is now 14 cases (the #251 golden re-review set), and the $60 default trimmed case-runs on both arms (38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs at the measured ~$10/arm-run with landing headroom, so the weekly aggregate never carries budget-skip asymmetry. * [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset * [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR workflows/review/eval/README.md is the guide an agent or engineer needs to run the eval system cold: the three tiers, every live-ab flag and CI entry point, powered-run recipes with real costs, corpus-growth rules (target the 20-80% band), report-reading guidance anchored on the measured noise floor, model pins, and the harness's historical limits (nothing before the 2026-07-08 structured-agent architecture runs under it). The weekly drift run now also opens a PR committing the report markdown and compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded, outside the A/B path filter): a job summary nobody opens is not a drift watch, and merged report PRs accumulate an in-repo time series. * [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README * [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets Findings from an adversarial pass over the instrument's own claims: Reports now stamp their ruler (matcher configuration + a content hash of the loaded corpus); the aggregate prints it, warns when a pool mixes rulers, and the drift series stays interpretable across instrument upgrades like the arbiter default or corpus growth, which move every rate without the reviewer changing. Noise-floor bands gain SD (min/max are extreme-value statistics that only widen as samples accumulate; mean +/- sd is the stable band) and a loud case-asymmetry warning: the 2026-07-10 measurement had budget skips, so its published v1 bands fold case-mix variance in on top of wobble; the footer constants and provenance string now say so. Repeat budgets roll: each arm-run's slice is the remaining budget over the remaining arm-runs, so cheap early arms donate headroom forward instead of stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that contaminated the noise-floor run). README gains a statistical-honesty section: clustered-interval optimism, the repeats-gate relaxation, and the arbiter's uncalibrated refuse bias. * [jwbron/anchor-snap-provenance] review: anchor-snap fallback in the change-provenance gate The reviewer produces right-file, right-mechanism findings at wrong line numbers (it appears to sometimes count unified-diff text lines instead of file lines: observed anchors at line 24 of an 18-line file and line 8 of a 3-line file), and the provenance gate then drops them; since only surviving blocking labels feed the computed verdict, a correct blocking finding dying at the gate can flip REQUEST_CHANGES to APPROVE. Observed on adversarial-injection-approve, golden-request-changes-authz, the sql-index replay, and production main's own baseline arm. Before setting a line-anchored finding aside, the gate now snaps it to the nearest changed line in the same file under two windows: a 3-line near-miss window (the unified diff's context width) and an overflow window for anchors past every shown line by no more than the file's diff-text overhead (exactly the counting mis-anchor's overshoot bound). Ties break toward the lower line. Snapped findings keep their severity, post at the snapped anchor, and are recorded for audit (out/snapped.json in production, snappedByProvenance in eval results); findings outside both windows keep today's set-aside behavior. provenance.json carries a precomputed per-file snap lookup so review.md's gate stays a dictionary lookup. The live A/B emulates each arm's own review.md gate version, keyed on the anchor-snap marker, so the baseline arm replays the pre-snap gate and the powered run prices this change. * [jwbron/anchor-snap-provenance] review: pin anchor-snap in a deterministic smoke case provenance-anchor-snap-rescued replays the observed production anatomy in the per-push CI gate: a correct blocking finding anchored at the diff-text line (past the end of a short file, at the exact overflow edge) snaps to the nearest changed line, posts there, and drives REQUEST_CHANGES, while a far-anchored pre-existing observation in the same run is still set aside unposted. Deterministic and smoke-tagged only, so the live ruler (corpus stamp over live-tagged cases) is untouched and the in-flight powered run stays comparable. Also documents how to audit snap records (from/to distance separates the near-miss and overflow classes). * [jwbron/eval-case-selection] review: exact --cases selection, fail-fast on unknown ids The smoke scope filtered the corpus before the case filter, so a powered dispatch naming a non-smoke case silently dropped it: the anchor-snap pricing run named two cases and the paid report covered one without saying so. --cases is now an exact selection (bypasses --smoke-only, preserves order, dedupes) and any id matching no live case throws before a single model call. The workflow needs no change; it already passes both flags and the case list now wins.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-review coverage for the eval corpus. Stacked on #246 (the re-review mode dial); answers the gap that the corpus only modeled first reviews: every case was a fresh-PR snapshot, so re-review behavior (thread reconciliation, the flip decision, duplicate suppression, reduced-depth rosters) had no corpus instrument at all. The template is the measured seeded lifecycle (Khan/webapp#40730: planted bugs, a bad partial fix with added bugs, then the full fix), ported with the trial skill's sanitization gate (structural rewrite, generic naming, no webapp code or paths).
Format: open-PR snapshots
A live-enabled case may carry
live.rereview:priorDiff: the previously fully-reviewed diff. The staged prior review's hidden fingerprint stamp and the depth plan are derived from it with the samelib/rereview-modefunctions production runs, so the dial's tripwire and depth decision are exercised, not simulated.priorVerdict/priorDepth: what the stamp records.priorThreads: the unresolved bot threads at this push, each with the opening comment (label template), an optional author reply, per-thread reconciliation ground truth (expect: resolve|keep), and optional mechanism alternates for duplicate detection.Each lifecycle push is an independently runnable corpus case; a lifecycle is a chain of them (
golden-retention-lifecycle-1/2/3). The prior state is authored ground truth rather than carried runner state, which keeps cases reproducible and lets the suite run any push in isolation; self-play lifecycles (the arm's own push-1 comments feeding push 2) remain the review-trial skill's job.What runs
live-stage.ts):threads.json(with author replies, so the reconciler has the reply chain production gives it),human-threads.json,prior-reviews.json(stamped body),rereview-plan.json; under a reduced mode the scoped diff overwritesfull-stripped.diff/pr.diff, exactly as review.md's scoped and flip-gated depths stage.live-producer.ts): dispatchesthread-reconcileron rereview cases at every depth, and sizes the finder roster by the plan (scopedkeeps the full roster,flip-gatedkeeps the correctness pass,fastreconciles only).rereview-match.ts): thread-resolution accuracy against per-thread ground truth (unaccounted threads count as wrong, the accountability failure mode), the flip-gate input (kept blocking count, expected vs actual), and duplicate comments that re-raise kept threads. Fresh-defect recall on a push still rides the ordinarymustCatchSpecsthroughlive-match.live-ab.ts):--re-review-mode <m>sets the CANDIDATE arm's mode while the baseline staysfull, so one run prices the dial on recall and dollars over the same cases; per-case and aggregate rereview metrics land in the JSON and markdown reports.runner.ts+lib/verdict.ts):computeVerdictgainskeptBlockingCount, the flip rule as code: an otherwise-approving run is floored at REQUEST_CHANGES while a prior blocking thread stays open. Re-review cases replay the ground-truth reconciliation, render the accountability section into the planned body, and the suite asserts the push-2 case blocks even with zero posted blocking comments and the push-3 case flips to APPROVE with "All 4 prior review threads are resolved."The ported lifecycle
golden-retention-lifecycle-1: four planted defects (deletion removes at most one row via a paging default, a cap off-by-one, an unbounded prune query, a test that asserts nothing).golden-retention-lifecycle-2: the bad partial fix. Two threads properly fixed (resolve), the off-by-one "fixed" into a different off-by-one (keep), the test fix deferred in a reply (keep), plus a fresh planted blocking defect in the new dedup (prefix-collision data loss) and an untested window. Expected verdict stays REQUEST_CHANGES.golden-retention-lifecycle-3: everything fixed; all four threads must resolve and the verdict must flip to APPROVE, with the intentional fire-and-forget prune error as a must-not-flag trap.Sweeping every dial setting
eval/rereview-sweep.tsprices all four modes in one command (the A/B's--re-review-modecompares one mode against a full-depth baseline; the sweep holds the prompt fixed and varies only the mode, which is the decision a repo faces when writing its ROUTING line):No case-format change is needed (the mode is a run parameter); three realities the report handles instead: every row shows the EXECUTED depth because the tripwire can override the dial (both lifecycle pushes diverge at share 0.67 and re-arm full under every mode, correctly); pricing the cheap paths therefore needs an under-threshold case,
golden-retention-fix-push(a one-hunk fix push at share 0.2 whose fix resolves the prior blocking thread but plants a fresh defect inside the new hunk, so scoped/flip-gated must catch it); andfasthas definitionally zero fresh-defect recall, shown as the mode's cost in the recall-vs-dollars table.669 tests pass, lint and typecheck clean.