review: move deep-reasoning reviewer roles to Fable 5

[jwbron]

De-scoped 2026-07-21 per review: the swap covers correctness-reviewer only. Now based on main (the corpus stack #270/#273 and the drift/budget PRs have landed); the de-scoped roster has its own powered A/B below.

What

Switches the review workflow's correctness-reviewer from claude-opus-4-8 to claude-fable-5 (effort high, unchanged). Bug-finding recall is the workflow's load-bearing metric, and both the original five-role bundle A/B and the de-scoped re-run concentrate their gain in correctness-adjacent rows.

Deliberately NOT moved, per review (jeresig) and per the measurements:

• claim-validator stays Opus at xhigh: the bundle's Fable validator did not improve the precision gate, and the de-scoped re-run confirms the noise increase never came from the validator (it reproduces without it; see below).
• holistic / completeness / test-adequacy stay Opus: the bundle A/B could not attribute any gain to them, and both known consumers enable them in ROUTING, so they price into every full review.
• The orchestrator, thread-reconciler / skill-auditor / conventions, and all specialist lenses stay Opus (the lenses deliberately: Fable's cyber safety classifiers can false-positive refuse benign security analysis, and a refused lens is a silent coverage hole). pattern-triage stays Sonnet 4.6; first-principles already ran Fable 5.

Each stay-on-Opus reason is pinned in the agent definition in review.md. The weekly drift budget is sized from the de-scoped roster's measured rate (~$1.08/case-arm-run): 180 case-arm-runs ~ $194/week, cap 220.

The de-scoped measurement (powered A/B)

Nine pooled dispatches (runs 29855626692-29855643020), same shard-and-pool structure as the bundle run: 30 live cases x 3 repeats per arm; baseline = all-Opus roster, candidate = this branch. Aggregated with eval/aggregate.ts.

Metric	Opus 4.8	Fable 5 correctness-only	(five-role bundle, superseded)
Must-catch recall	82% (101/123)	89% (110/123)	83% -> 90%
True misses	22	13	21 -> 12
Verdict agreement	96%	96%	94% -> 97%
Noise (unmatched posted)	43%	49%	43% -> 49%
Judge mean quality	0.88	0.86	0.87 -> 0.86
Adversarial hard gate	—	PASSED (all shards)	PASSED
Cost	$72.12	$97.19 (+35%)	+73%

The de-scope kept the entire recall gain at half the cost delta. The improved rows are the same previously in-band/floor spec rows as the bundle run (incident-sql-missing-index:dm-default-backfill 1/3 -> 3/3, trial-retention-deletion:erasure-default-limit-one 0/3 -> 2/3, :erasure-ignores-delete-helper 1/3 -> 2/3, :erasure-prune-error-swallowed 0/3 -> 1/3, golden-retention-fix-push:quota-exceeded-never-true 2/3 -> 3/3, golden-retention-lifecycle-2:retention-prune-unhandled-rejection 2/3 -> 3/3), so the bundle's gain is now attributed: it was the correctness reviewer.

What the re-run disambiguated. The noise increase (43% -> 49%) reproduces without the Fable validator or whole-change reviewers: it is the Fable correctness reviewer posting more companion findings (219 posted vs 178). Likewise the one repeated verdict regression: golden-retention-lifecycle-3 (expected-clean re-review) 3/3 -> 1/3, where two candidate samples flag the deliberate single-batch PRUNE_BATCH prune as blocking. Note both arms in this run predate #271 (merged since), whose correctness-lens language explicitly names the bounded-batch prune as the fix shape, not a defect; the first drift run on merged main will show whether that attenuates this. If not, the corpus-side option is a bounded-prune must-not-flag trap on lifecycle-3.

Statistical honesty. The pooled +7-point recall delta is z ~ 1.7 (suggestive, not conclusive at 95%; pooled intervals are optimistic per the eval README). As with the bundle run, the per-case pattern is the stronger evidence: six hard rows improved, none regressed on recall, true misses down 41%. The aggregate stamps the expected mixed-corpus-hash warning (three shard groups select different case subsets); within every shard both arms share the same ruler.

The superseded five-role bundle measurement (runs 29783522498-29783549087) is preserved in the edit history and summarized in the table's last column.

Known risks to watch in production

• Occasional refusal stop on cyber-adjacent diffs hitting correctness-reviewer (surfaces as a missing agent result; none observed across ~380 live case-arm-runs to date). The weekly drift corpus is the soft-refusal detector: a silently refusing correctness reviewer craters must-catch recall on the security-adjacent cases.
• Author-facing chattiness: the Fable correctness reviewer posts more companion findings (noise 49% vs 43% on the eval's strict matcher). The reviewer-side dedup gap (two comments per defect) is the known follow-up that would absorb most of this.
• Over-blocking on expected-clean re-review pushes (lifecycle-3 above); watch the drift series now that review: teach the correctness and caching lenses to flag unbounded reads #271's lens carve-out is on main.

[changeset-bot]

🦋 Changeset detected

Latest commit: 77e0278

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Changes requested — see inline comments.

[github-actions]

Changes requested — see inline comments.

[github-actions]

Approved — no blocking issues found.

[github-actions]

Review live A/B

Baseline: origin/main (review.md 1f18e0251d8f); candidate: working tree (review.md 56aa662fbaad).

Ruler: matcher deterministic+arbiter; corpus cc11988c0918 (9 cases).

Metric	Baseline	Candidate	Delta
Must-catch recall	100%	100%	+0%
Verdict agreement	89%	89%	+0%
Noise (unmatched posted)	50%	61%	+11%
Clean false flags	0	0
Judge mean quality	0.91	0.90	-0.01
Cost	$7.32	$10.17
Wall clock	1066s	1309s
Cases run / skipped	9 / 0	9 / 0
Misses found-but-dropped	0	0
Findings anchor-snapped	0	0

Adversarial hard gate: PASSED on the candidate arm.

Single-run-stable rows: recall, verdict agreement, regressions, adversarial gate. Judge quality and noise are not: they jitter run-to-run at this corpus size, and a regressed reviewer can score HIGHER on judge quality (fewer, surer comments each read better). Recall against the labeled specs is the load-bearing metric.

Measured noise floor (identical arms, run 29069228968, 2026-07-10, 6 arm-samples, full corpus x3, pre-arbiter; budget skips left the samples on unequal case sets, so these v1 bands also carry case-mix variance): must-catch recall 54%-86% (sd 10%), verdict agreement 75%-100% (sd 9%), noise (unmatched posted) 50%-60% (sd 3%), judge mean quality 82%-86% (sd 2%). A single-run delta whose arms both sit inside a band is indistinguishable from run-to-run wobble; use --repeats to resolve smaller effects.

[jeresig]

Wait - all of these are on fable?? That seems like extreme overkill here - and not just that but also with high levels of effort.

[jwbron]

I don't know if I'd call this "extreme overkill". These are all reasoning tasks and fable performs significantly better at reasoning on high.

I'm not set on this selection though, maybe it makes sense to drop some of these back to opus but it will require more evals. The results from this experiment are promising though, fable is clearly a notably better reviewer than opus.

[jwbron]

Agreed, and de-scoped in 48c367f: only correctness-reviewer moves to Fable now (plus first-principles, which already ran it). The bundle A/B's own data supports the narrowing: the recall gain concentrated in correctness-adjacent rows, the Fable claim-validator measurably did not improve the precision gate (noise 43% -> 49%, one wrong blocking flag on a clean case), and the whole-change reviewers' contribution was unattributed while their cost is real; both this repo and webapp enable them in ROUTING, so the "free until enabled" framing was wrong in practice. Each stay-on-Opus reason is now pinned in the agent definition, the drift budget is provisionally re-derived (240 -> 200), and a fresh A/B of the de-scoped roster is running (runs 29855626692-29855643020, same shard-and-pool structure); the PR description will be updated with its numbers before this merges. One caveat for the record: attributing the gain to the correctness reviewer is inference from where the improved rows sit, not a per-role measurement, which is exactly what the re-run checks.

[jwbron]

fable's analysis above. Running a full A/B test now.

[jwbron]

De-scoped A/B results are in (runs 29855626692-29855643020, 30 cases x 3 repeats per arm): the correctness-only swap keeps the entire recall gain (82% -> 89%, true misses 22 -> 13, matching the five-role bundle within noise) at half the cost delta (+35% vs +73%). The re-run also settled the attribution question in your favor: the noise increase and the one verdict regression (over-blocking on the expected-clean lifecycle-3 re-review case) reproduce without the Fable validator or whole-change reviewers, so those roles were adding cost, not signal. Full table and analysis in the updated PR description; drift budget re-sized from the measured rate (~$1.08/case-arm-run -> cap 220).

[github-actions]

Review Guidance

github-actions (2 files)

File	Reason
review.md	Shared reviewer prompt executed in every consuming repo; swaps the default-roster correctness-reviewer's pinned model to claude-fable-5.
review-eval-drift.yml	Weekly scheduled workflow; raises the drift-run spend cap (120 to 200), which gates how much of the corpus runs before budget-skipping.

Common patterns

2 files: correctness-reviewer model swapped from claude-opus-4-8 to claude-fable-5 (workflows/review/review.md, workflows/review/README.md).

-model: claude-opus-4-8
+model: claude-fable-5

2 files: eval budget cap re-sized for the Fable corpus (30 live cases x3) — default and recipe max_usd raised to 200 (.github/workflows/review-eval-drift.yml, workflows/review/eval/README.md).

-        default: "120"
+        default: "200"