diff --git a/.changeset/review-fable-deep-reasoning.md b/.changeset/review-fable-deep-reasoning.md new file mode 100644 index 00000000..9967e65d --- /dev/null +++ b/.changeset/review-fable-deep-reasoning.md @@ -0,0 +1,5 @@ +--- +"review": minor +--- + +Move the `correctness-reviewer` from Opus 4.8 to Fable 5 (`claude-fable-5`). Bug-finding recall is the workflow's load-bearing metric, and the 2026-07-20 pooled A/B's recall gain (83% to 90%, true misses down 43%) concentrated in correctness-adjacent rows, so the 2x per-token price goes to the one role where the measurement supports it. Deliberately NOT moved, per review: `claim-validator` stays Opus at xhigh (the Fable arm measurably did not improve the precision gate: noise 43% to 49%, one wrong blocking flag on a clean case); the opt-in whole-change reviewers (`holistic`, `completeness`, `test-adequacy`) stay Opus (the A/B measured the full-Fable bundle, so their contribution is unattributed, and both known consumers enable them in ROUTING so they are not free in practice); the procedural roles (orchestrator, `thread-reconciler`, `skill-auditor`, `conventions`) stay Opus (instruction-following over staged context, and the orchestrator is the token-heaviest agent under a 20-minute timeout); and the specialist lenses stay Opus because Fable's cyber safety classifiers can refuse benign security-focused analysis, and a refused security lens would be a silent coverage hole. `first-principles` already ran Fable 5. The firewall pricing pin (awf v0.27.27) and the `models:` cost block already cover `claude-fable-5`, so no sandbox or pricing change is needed. diff --git a/.github/workflows/review-eval-drift.yml b/.github/workflows/review-eval-drift.yml index de498f36..5c60bea2 100644 --- a/.github/workflows/review-eval-drift.yml +++ b/.github/workflows/review-eval-drift.yml @@ -35,16 +35,23 @@ on: # case-arm-runs (~$0.82 each) with the tail of the corpus # budget-skipped, which also contaminated the noise-floor bands # (asymmetric case mix). The 23-case corpus (both new mutation cases - # are live-tagged) needs 23 x 3 repeats x 2 arms = 138 case-arm-runs, - # ~$113; 120 clears it while staying a hard stop against runaways, - # though the headroom is thin: the ~$0.82/run mean comes from a run - # whose tail was budget-skipped, so it may understate. Repeats stay - # at 3: they are - # the power axis, and per-case repeat tiering would make every - # noise-floor sample a different case mix. + # are live-tagged) needed 23 x 3 repeats x 2 arms = 138 case-arm-runs, + # ~$113; 120 cleared it while staying a hard stop against runaways. + # Repeats stay at 3: they are the power axis, and per-case repeat + # tiering would make every noise-floor sample a different case mix. + # + # Re-sized 2026-07-21 for the Fable-5 correctness-reviewer plus the + # discrimination cases added alongside it: 30 live cases (2 clean + + # 5 golden + 6 incidents + 7 smoke + 10 synthetic-mutations) x 3 + # repeats x 2 arms = 180 case-arm-runs. The de-scoped roster measured + # ~$1.08 per case-arm-run in the pooled A/B (runs + # 29855626692-29855643020: $97.19 / 90), so a drift week with both + # arms on this roster is ~$194; 220 clears it with real headroom, + # which matters now that a budget-skipped tail is a red run (the + # caseAsymmetry gate). If the swap is reverted, dial back toward 160. description: "Total hard budget across both arms and all repeats" required: false - default: "120" + default: "220" permissions: # write: the report step commits the aggregate under @@ -71,7 +78,7 @@ jobs: env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} REPEATS: ${{ inputs.repeats || '3' }} - MAX_USD: ${{ inputs.max_usd || '120' }} + MAX_USD: ${{ inputs.max_usd || '220' }} run: | if [ -z "$ANTHROPIC_API_KEY" ]; then echo "ANTHROPIC_API_KEY secret not configured; skipping the drift run." >&2 diff --git a/workflows/review/README.md b/workflows/review/README.md index 1ce3cbc0..678e98ca 100644 --- a/workflows/review/README.md +++ b/workflows/review/README.md @@ -297,21 +297,32 @@ sub-agent models — this table is the human-facing summary: | orchestrator | `claude-opus-4-8` | high | Owns every GitHub/safe-output decision | | `pattern-triage` | `claude-sonnet-4-6` | medium | Cheap first-pass triage | | `thread-reconciler` | `claude-opus-4-8` | medium | Reconciliation | -| `correctness-reviewer` | `claude-opus-4-8` | high | Whole-change reviewer | +| `correctness-reviewer` | `claude-fable-5` | high | Whole-change reviewer; bug-finding recall is the load-bearing metric | | `skill-auditor` | `claude-opus-4-8` | high | Whole-change reviewer | | `holistic` | `claude-opus-4-8` | high | Opt-in whole-change reviewer (`enable` in `ROUTING`) | | `completeness` | `claude-opus-4-8` | high | Opt-in whole-change reviewer (`enable` in `ROUTING`) | | `test-adequacy` | `claude-opus-4-8` | high | Opt-in whole-change reviewer (`enable` in `ROUTING`) | | `conventions` | `claude-opus-4-8` | medium | Opt-in advisory targeted check (`enable` in `ROUTING`) | | `first-principles` | `claude-fable-5` | high | Opt-in advisory-only; reviews the change's justification | -| `claim-validator` | `claude-opus-4-8` | xhigh | Adversarial claim validation | +| `claim-validator` | `claude-opus-4-8` | xhigh | Adversarial claim validation; stays Opus (the Fable arm did not improve precision) | | specialist lenses | `claude-opus-4-8` | high | Opt-in via `lens=` in `ROUTING`; the security & auth lens is xhigh | Only the orchestrator and the default roster (`pattern-triage`, `correctness-reviewer`, `skill-auditor`, `thread-reconciler`, `claim-validator`) run by default; every other row is opt-in via `ROUTING` and earns its line through -the eval suite. Per-role Fable-5 / Sonnet experiment arms are eval-suite -measurements to run after the suite exists. +the eval suite. Exactly two roles run Fable 5: `first-principles` (from day +one, for perspective diversity) and `correctness-reviewer` (the 2026-07-20 +A/B's recall gain concentrated in correctness-adjacent rows, and bug-finding +recall is the load-bearing metric). Everything else stays on Opus 4.8 +deliberately: the A/B measured the full-Fable bundle, so the other roles' +contribution is unattributed while their cost is not (both known consumers +enable the whole-change reviewers in `ROUTING`, so they are not free in +practice); the Fable `claim-validator` measurably did not improve the +precision gate (noise 43% -> 49%); and the specialist lenses stay Opus +because Fable's cyber safety classifiers can refuse benign security-focused +analysis, and a refused security lens would be a silent coverage hole. Any +further per-role promotion (or Sonnet step-down) earns its line through its +own eval-suite arm. ### Feedback signal: thumbs sweep and live counters diff --git a/workflows/review/eval/README.md b/workflows/review/eval/README.md index b00b9fe1..76a7f403 100644 --- a/workflows/review/eval/README.md +++ b/workflows/review/eval/README.md @@ -10,7 +10,7 @@ comment is the reference for its internals. | --- | --- | --- | --- | | Deterministic suite (vitest) | The pipeline code: router, gates, verdict, rendering, matching, aggregation. Replays recorded findings; no model calls. | $0 | Every push (`pnpm test --run`) | | Live A/B, smoke subset | One PR's marginal `review.md` delta: real model sub-agents run from BOTH the base branch's and the PR's review.md over live-tagged smoke cases. A tripwire, not a measurement. | ~$10/PR | Every PR touching `workflows/review/**` | -| Powered / scheduled runs | Recall effects, priced with repeats and binomial intervals; run-to-run wobble; cumulative drift vs main. | ~$29-85 | `workflow_dispatch`, weekly cron | +| Powered / scheduled runs | Recall effects, priced with repeats and binomial intervals; run-to-run wobble; cumulative drift vs main. | ~$29-220 | `workflow_dispatch`, weekly cron | Single-run percentage deltas below the measured noise floor mean nothing (see "Reading a report" below). Any recall claim needs a powered run. @@ -66,13 +66,16 @@ gh workflow run review-eval-ab.yml --ref \ -f cases=adversarial-injection-approve,golden-request-changes-authz \ -f repeats=10 -f max_usd=45 -# Cumulative measurement vs production (~$60): full corpus x3 against main +# Cumulative measurement vs production (~$170 at post-Fable-swap measured +# rates: ~$0.80/case-arm-run on the all-Opus baseline arm, ~$1.08 on the +# Fable-correctness candidate arm, 30 live cases x3): full corpus x3 gh workflow run review-eval-ab.yml --ref \ - -f base_ref=origin/main -f full=true -f repeats=3 -f max_usd=85 + -f base_ref=origin/main -f full=true -f repeats=3 -f max_usd=200 -# Noise floor / wobble control: identical arms, full corpus x3 +# Noise floor / wobble control (~$194 measured with both arms on the +# Fable-correctness roster): identical arms, full corpus x3 gh workflow run review-eval-ab.yml --ref \ - -f base_ref=origin/ -f force_arms=true -f full=true -f repeats=3 -f max_usd=85 + -f base_ref=origin/ -f force_arms=true -f full=true -f repeats=3 -f max_usd=220 # Pool reports across dispatches (run ids or local paths) pnpm dlx tsx workflows/review/eval/aggregate.ts ... [--out ] diff --git a/workflows/review/review.md b/workflows/review/review.md index 6bd6b562..ec0a2c82 100644 --- a/workflows/review/review.md +++ b/workflows/review/review.md @@ -175,7 +175,7 @@ engine: model: claude-opus-4-8 timeout-minutes: 20 -# claude-fable-5 (the first-principles reviewer's pinned model) is not in the +# claude-fable-5 (pinned by first-principles and correctness-reviewer) is not in the # AI-credits pricing table of the firewall api-proxy that gh-aw <= v0.81.x pins # (gh-aw-firewall v0.27.11), and the proxy rejects any un-priced model with a 400, # so the first-principles dispatch fails on every run where it is enabled. Two @@ -1717,9 +1717,11 @@ return `{"findings": [], "hunts": [...]}` with the hunt states still recorded. --- name: correctness-reviewer description: Classifies each changed file's risk and reviews the diff for correctness defects; returns JSON. -model: claude-opus-4-8 +model: claude-fable-5 # effort: high — launch default (whole-change reviewer). gh-aw has no per-agent # effort field yet; the per-role model/effort table lives in the README. +# Fable 5: bug-finding recall is this workflow's load-bearing metric, and +# stronger real-defect detection is Fable's headline gain over Opus 4.8. --- You are a correctness-focused code reviewer. You have **no GitHub access** — read the diff and file list from disk and return your result as JSON only. @@ -2114,7 +2116,11 @@ Return ONLY this JSON object (no prose, no code fence): name: claim-validator description: Re-checks each candidate review comment against the actual code and the repo's best-practice skills, and drops or corrects the ones that are wrong; returns JSON. model: claude-opus-4-8 -# effort: xhigh — launch default (claim-validator). +# effort: xhigh — launch default (claim-validator). Deliberately NOT moved to +# Fable 5 with the correctness reviewer: in the 2026-07-20 pooled A/B the +# Fable validator did not offset the higher flag rate (noise 43% -> 49%, one +# wrong blocking flag on a clean case), so the precision gate stays on Opus +# until an arm shows otherwise; prompt tightening is the queued follow-up. --- You are a skeptical validator. Other reviewers proposed the comments in `/tmp/gh-aw/review/claims.json`; your job is to catch the ones that are **wrong** — @@ -2496,16 +2502,17 @@ If the changed behavior is adequately tested, return {"findings": []}. name: first-principles description: A diverse-perspective, advisory-only sanity check on whether the change should exist as written; returns findings as JSON. model: claude-fable-5 -# effort: high — launch default. Runs on Fable 5 (claude-fable-5) day one for a -# genuinely different perspective. Advisory-only, never blocks. +# effort: high — launch default. Ran on Fable 5 (claude-fable-5) from day one; +# the correctness reviewer joined it after the 2026-07-20 A/B. Advisory-only, +# never blocks. --- You are the **first-principles** reviewer. Your single mandate is to review the **justification for the change, not the change itself**: where `holistic` asks whether the diff hangs together, you step outside the change's own framing and ask whether it **should exist as written**. Your primary input is the stated rationale — the PR title/description and the problem it claims to solve — read against the diff, -not the diff line by line. You run on a different model (Fable 5) on purpose, -to bring a perspective the other reviewers do not. You have **no GitHub access** — read +not the diff line by line. You are prompted for a deliberately different +perspective than the other reviewers, so bring one. You have **no GitHub access** — read from disk and return JSON only. **You are advisory-only and you never block.** Every finding you return MUST carry a