Skip to content

review eval: fix floor-case ground truth, mine noise pool, mint mid-band cases #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jwbron merged 7 commits into from
Jul 21, 2026
Merged

review eval: fix floor-case ground truth, mine noise pool, mint mid-band cases #270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
590adb1
[review-eval/corpus-discrimination] review eval: fix floor-case groun…
jwbron Jul 20, 2026
ffc11cd
[review-eval/corpus-discrimination] review eval: annotate real defect…
jwbron Jul 20, 2026
3618cb1
[review-eval/corpus-discrimination] review eval: mint two mid-band re…
jwbron Jul 20, 2026
be418e8
[review-eval/corpus-discrimination] review eval: raise the drift budg…
jwbron Jul 20, 2026
f31ee46
[fix-270] [review-eval/corpus-discrimination] review eval: add the ch…
jwbron Jul 20, 2026
a04f231
[review-eval/corpus-discrimination] eval: fold in triage-branch fixtu…
jwbron Jul 21, 2026
e306982
[review-eval/corpus-discrimination] eval: blockingOnly trap pin; reco…
jwbron Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/review-eval-corpus-discrimination.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"review": patch
---

Eval-only: improve the discrimination power of the review eval corpus. Fixes the floor-case ground truth found by the 2026-07-20 drift triage (lifecycle-3's void prune was a real defect the reviewer was right to block, dedup-eventual-consistency's expected verdict contradicted its own premise, retention-unbounded-prune was retired as unevidenceable in-tree), repairs order-sensitive spec regexes that under-counted deterministic matches, annotates four real defects mined from the drift run's unmatched-posted noise pool, mints two mid-band retention/lifecycle cases, and raises the weekly drift budget default from 85 to 120 USD so the full corpus clears without budget skips. No change to the shipped review workflow.
15 changes: 13 additions & 2 deletions .github/workflows/review-eval-drift.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,20 @@ on:
required: false
default: "3"
max_usd:
# Sized from the 2026-07-20 run (29724668102): $84.33 for 103
# case-arm-runs (~$0.82 each) with the tail of the corpus
# budget-skipped, which also contaminated the noise-floor bands
# (asymmetric case mix). The 23-case corpus (both new mutation cases
# are live-tagged) needs 23 x 3 repeats x 2 arms = 138 case-arm-runs,
# ~$113; 120 clears it while staying a hard stop against runaways,
# though the headroom is thin: the ~$0.82/run mean comes from a run
# whose tail was budget-skipped, so it may understate. Repeats stay
# at 3: they are
# the power axis, and per-case repeat tiering would make every
# noise-floor sample a different case mix.
description: "Total hard budget across both arms and all repeats"
required: false
default: "85"
default: "120"

permissions:
# write: the report step commits the aggregate under
Expand All @@ -60,7 +71,7 @@ jobs:
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
REPEATS: ${{ inputs.repeats || '3' }}
MAX_USD: ${{ inputs.max_usd || '85' }}
MAX_USD: ${{ inputs.max_usd || '120' }}
run: |
if [ -z "$ANTHROPIC_API_KEY" ]; then
echo "ANTHROPIC_API_KEY secret not configured; skipping the drift run." >&2
Expand Down
13 changes: 12 additions & 1 deletion workflows/review/eval/corpus/golden/golden-retention-fix-push/case.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"live"
],
"category": "golden",
"description": "The under-tripwire re-review pricing case: an open PR touched five regions of quota.ts and the prior full review blocked one (an off-by-one in the remaining-quota math). The fix push rewrites only that region (unreviewed share 0.2, below the tripwire threshold), so scoped/flip-gated/fast actually execute their reduced paths. The fix resolves the thread but plants a fresh defect in the same hunk: the remaining count is cached under a shared key with no user id, leaking quota across users. scoped and flip-gated stage exactly that hunk and must catch it; fast dispatches no finder and cannot, which is the recall cost the mode sweep prices.",
"description": "The under-tripwire re-review pricing case: an open PR touched five regions of quota.ts and the prior full review blocked one (an off-by-one in the remaining-quota math). The fix push rewrites only that region (unreviewed share 0.2, below the tripwire threshold), so scoped/flip-gated/fast actually execute their reduced paths. The fix resolves the thread but plants a fresh defect in the same hunk: the remaining count is cached under a shared key with no user id, leaking quota across users. scoped and flip-gated stage exactly that hunk and must catch it; fast dispatches no finder and cannot, which is the recall cost the mode sweep prices. The rewritten hunk also leaves quotaExceeded unsatisfiable (remaining is clamped to zero, then compared < 0, so the quota is never enforced); a second real defect annotated 2026-07-20 from the drift noise pool.",
"changedFiles": [
{
"path": "src/notes/quota.ts",
Expand Down Expand Up @@ -68,6 +68,17 @@
"quota-remaining",
"another user|cross.user|leak"
]
},
{
"key": "quota-exceeded-never-true",
"path": "src/notes/quota.ts",
"lineStart": 30,
"lineEnd": 36,
"mechanism": [
"never (returns? )?true|always (returns? )?false|can never (be true|be negative|exceed)|unsatisfiable",
"Math\\.max\\(0.{0,80}< 0|clamp(ed|s)?.{0,80}< 0|< 0 is (always )?false|0 < 0",
"unlimited (notes|saves)|quota (is )?never enforced|save is (always )?allowed|never blocked"
]
}
],
"rereview": {
Expand Down
32 changes: 18 additions & 14 deletions workflows/review/eval/corpus/golden/golden-retention-lifecycle-1/case.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"live"
],
"category": "golden",
"description": "Push 1 of the retention lifecycle (sanitized port of a seeded live trial): a per-user note cap with pruning, and note deletion wired into account deletion. Four planted defects: deletion removes at most one note (facade pageSize defaults to 1), the prune offset keeps one note fewer than the cap, the prune query loads the entire result set, and the prune test asserts nothing the cap needs.",
"description": "Push 1 of the retention lifecycle (sanitized port of a seeded live trial): a per-user note cap with pruning, and note deletion wired into account deletion. Four planted defects: deletion removes at most one note (facade pageSize defaults to 1), the prune offset keeps one note fewer than the cap, the fire-and-forget prune discards rejections with no handler (an unhandled rejection can crash the process), and the prune test asserts nothing the cap needs. The former retention-unbounded-prune spec (pageSize \"all\" into one deleteMulti) was retired 2026-07-20: the facade documents no batch limit, so the defect was not evidenceable in-tree and read 0% across every drift sample; mutation-retention-batch-delete-limit covers the mechanism with the limit documented.",
"changedFiles": [
{
"path": "src/notes/purge-user-data.ts",
Expand Down Expand Up @@ -71,22 +71,24 @@
"source": "correctness",
"finding": {
"schema_version": 2,
"id": "retention-unbounded-prune",
"id": "retention-prune-unhandled-rejection",
"lens": "correctness",
"anchor": {
"type": "line",
"path": "src/notes/retention.ts",
"line": 17,
"line": 9,
"side": "RIGHT"
},
"severity": "blocking",
"confidence": 0.9,
"confidence": 0.85,
"evidence_trace": [
"src/notes/retention.ts: pageSize \"all\" on the prune query"
"src/notes/retention.ts:9 fires pruneNotes with `void` and no rejection handler",
"db.query/deleteMulti inside pruneNotes can reject on a transient datastore error",
"an unhandled promise rejection terminates the Node process under default settings"
],
"failure_scenario": "A user far over the cap triggers prune; pageSize \"all\" loads every stale row into memory in one call, so a large backlog makes prune (and the save that fired it) balloon instead of deleting in bounded batches.",
"producing_hunt": "correctness:unbounded-read",
"model_authored_prose": "The prune query loads the entire stale result set at once (`pageSize: \"all\"`). Delete in bounded batches so a large backlog cannot balloon a save."
"failure_scenario": "A transient datastore error makes pruneNotes reject after a save; the `void`-discarded promise has no .catch, so the rejection is unhandled and crashes the process under Node's default --unhandled-rejections=throw; the opposite of the comment's 'a failing prune must not fail the save'.",
"producing_hunt": "correctness:unhandled-rejection",
"model_authored_prose": "`void pruneNotes(...)` discards the promise but not its rejection: a failing prune becomes an unhandledRejection and can crash the process. Attach `.catch(() => {})` (or log the error) so the documented fire-and-forget intent actually holds."
}
},
{
Expand Down Expand Up @@ -122,7 +124,7 @@
"verification": "confirmed"
},
{
"id": "retention-unbounded-prune",
"id": "retention-prune-unhandled-rejection",
"verification": "confirmed"
},
{
Expand All @@ -136,7 +138,7 @@
"mustCatch": [
"retention-purge-limit",
"retention-cap-off-by-one",
"retention-unbounded-prune",
"retention-prune-unhandled-rejection",
"retention-test-asserts-nothing"
]
},
Expand Down Expand Up @@ -168,12 +170,14 @@
]
},
{
"key": "retention-unbounded-prune",
"key": "retention-prune-unhandled-rejection",
"path": "src/notes/retention.ts",
"lineStart": 6,
"lineEnd": 10,
"mechanism": [
"pageSize.*all",
"entire result|unbounded|all (stale )?rows",
"batch"
"unhandled (promise )?rejection|unhandledRejection",
"no \\.?catch|without (a )?\\.?catch|no rejection handler|neither awaited nor caught",
"crash(es|ed)?.{0,30}process|terminat(es|e|ing).{0,30}process|process (crash|exit)"
]
},
{
Expand Down
24 changes: 21 additions & 3 deletions workflows/review/eval/corpus/golden/golden-retention-lifecycle-2/case.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"live"
],
"category": "golden",
"description": "Push 2 of the retention lifecycle: a bad partial fix with added bugs, as an open-PR re-review snapshot. The purge and the unbounded prune are properly fixed; the off-by-one is 'fixed' to a different off-by-one (offset MAX+1) so its thread must be KEPT; the test fix is deferred in a reply so its thread must be KEPT; and the new dedup feature plants a fresh blocking defect (a lowercased 8-char prefix key silently drops distinct notes) plus an untested dedup window.",
"description": "Push 2 of the retention lifecycle: a bad partial fix with added bugs, as an open-PR re-review snapshot. The purge and the unbounded prune are properly fixed; the off-by-one is 'fixed' to a different off-by-one (offset MAX+1) so its thread must be KEPT; the test fix is deferred in a reply so its thread must be KEPT; and the new dedup feature plants a fresh blocking defect (a lowercased 8-char prefix key silently drops distinct notes) plus an untested dedup window. The fire-and-forget prune still has no rejection handler (carried from push 1, annotated 2026-07-20 from the drift noise pool). The overcorrected off-by-one and the still-vacuous test are deliberately NOT specs: their prior threads are KEPT, so fresh findings there are re-review duplicates, not recall.",
"changedFiles": [
{
"path": "src/notes/purge-user-data.ts",
Expand Down Expand Up @@ -107,9 +107,27 @@
"key": "retention-dedup-window-untested",
"path": "src/notes/retention.ts",
"mechanism": [
"window.*(untested|no test)|no test.*window",
"window.{0,60}(untested|no test|not tested|never tested)|no test.{0,60}window",
"DEDUP_WINDOW",
"boundary"
"(boundary|edge).{0,60}(window|dedup)|(window|dedup).{0,60}(boundary|edge)",
"dedup.{0,80}(untested|no test|never (tested|exercised|covered|verifie[sd])|not (covered|tested|exercised|verified))|never verifies.{0,60}dedup",
"(test|coverage).{0,80}dedup"
],
"altLocations": [
{
"path": "src/notes/retention.test.ts"
}
]
},
{
"key": "retention-prune-unhandled-rejection",
"path": "src/notes/retention.ts",
"lineStart": 29,
"lineEnd": 33,
"mechanism": [
"unhandled (promise )?rejection|unhandledRejection",
"no \\.?catch|without (a )?\\.?catch|no rejection handler|neither awaited nor caught",
"crash(es|ed)?.{0,30}process|terminat(es|e|ing).{0,30}process|process (crash|exit)"
]
}
],
Expand Down
Loading
Loading