review: label identical-arm eval runs as wobble controls; raise the drift budget - #267
Conversation
…s as wobble controls; size the drift budget to the corpus
🦋 Changeset detectedLatest commit: 6dcec10 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Review live A/BNo reviewable delta: review.md is byte-identical in both arms (baseline |
This comment has been minimized.
This comment has been minimized.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…note prefix test, red-run on case asymmetry Review feedback on the wobble-control relabel PR, plus consolidation across the open budget-touching PRs: - Hand the drift max_usd bump (85 -> 120 -> 240) to the corpus-growth stack, which re-derives it from the grown corpus at Fable-arm rates; three PRs were editing the same lines. The eval README recipe figures move with it. This PR keeps only the relabel work. - Add the identical-arm dropped-spec regression test: every existing identical-arm fixture had no drops, so the arm A/arm B drop-note prefix branch never rendered and a regression to base:/cand: would ship unnoticed. - Fail the drift job when the aggregate flags caseAsymmetry on the noise floor (after the artifact upload and visibility PR): a corpus outgrowing the budget becomes a red run instead of a footnote.
Review Guidancegithub-actions (3 files)
Common patterns2 files: Identical-arm detection added to the report renderers — on matching arm SHAs each retitles to "wobble control" and relabels Baseline/Candidate to Arm A/Arm B. 2 files: New tests assert the wobble-control heading, Arm A/Arm B columns, and "run-to-run wobble" text for identical-arm inputs. Excluded from review (1 file)Not individually reviewed — generated, formatting-only, or fully explained by a common pattern above:
|
| --head "$BRANCH" \ | ||
| --title "review: eval drift report $STAMP" \ | ||
| --body-file out/live-ab-report.md | ||
| - name: Fail on a contaminated noise floor |
There was a problem hiding this comment.
question (non-blocking): This red-run gate lands while its trigger is already live and the paired budget bump was pulled out of this PR. The last committed drift report carries caseAsymmetry: true, and this diff keeps MAX_USD at 85 (the 85→120 bump was handed to the corpus-growth stack), so the next scheduled run will likely trip this exit 1 on the same corpus. The step's own remedy ("Raise max_usd and re-dispatch") also isn't reachable on the cron path without that bump. Is the intent for the weekly job to sit red until the corpus-growth stack lands, or should this gate land together with (or after) the budget raise?
2 participants
What
Two follow-ups from the first scheduled drift report (#265):
Identical-arm eval reports no longer read as an A/B. When both arms ran the same review.md (only possible under
--force-arms: the weekly drift watch and manual wobble controls), the single-run, multi-repeat, and aggregate renderers now retitle themselves "Review wobble control (identical arms)", relabel Baseline/Candidate to Arm A/Arm B, and state up front that between-arm deltas are run-to-run wobble, not a prompt effect. The aggregate additionally leads with the noise-floor bands: on an identical-arm pool the bands are the product and the per-case table is the raw material. The aggregate keys the relabel offnoiseFloorbeing present, which is computed iff every pooled sample ran identical arms.A contaminated noise floor is a red run, not a footnote. The 2026-07-20 run spent $84.33 of its $85 cap and budget-skipped cases, which the report flags as contaminating the noise-floor bands with case-mix variance; but the flag lives inside the report body, so the degradation persists silently until someone reads it. The drift job now fails when the aggregate reports
caseAsymmetry(after the artifact upload and visibility PR, bothif: always()), so a corpus outgrowing the budget surfaces as a red run. The budget resize itself (85 -> 120 -> 240) moved to the corpus-growth stack (review eval: fix floor-case ground truth, mine noise pool, mint mid-band cases #270/review: move deep-reasoning reviewer roles to Fable 5 #266), which owns the sizing; three open PRs were editing the same lines.Why the relabel matters
#265 renders a "Baseline vs Candidate" per-case table with small deltas between two runs of the same prompt; that framing invites reading wobble as a result. The drift run's whole point is that within-run deltas are noise, so the report should say so in its own headline.
Testing
pnpm test --run workflows/review/eval/(new cases cover the multi-repeat, single-run, and aggregate relabels, the noise-floor-first ordering, the identical-arm drop-note prefix, and that A/B reports keep the Baseline/Candidate framing)pnpm typecheck, prettier clean on changed files