review: human-resolved bot threads become an adjudicated suppression corpus

[jwbron]

What

Resolving a bot thread now means "settled" instead of "open season for a rephrase". The staging fetches resolution state (isResolved + resolvedBy over GraphQL) and writes a new adjudicated-threads.json: bot-opened threads a HUMAN resolved. The dispatcher runs a second suppression pass over that corpus after open-thread suppression, using the same defect-identity match (same path, no line window, the #245 similarity floors), and drops any non-blocking candidate that re-derives an adjudicated defect before validation ever sees it.

Why

Suppression was keyed on threads.json, which stages only UNRESOLVED threads, so resolving a bot thread quietly removed the defect from the suppression shield; the next run could re-post the same concern with fresh wording as a brand-new thread, which every later accountability recap then enumerated as "still unaddressed".

Measured on webapp#41290: the author replied to and resolved six variants of one concern at moderation_helpers.go:135 over two days, and a seventh rephrasing posted anyway (review 4875909452), then headlined every subsequent recap. From the author's side that is indistinguishable from the bot ignoring every resolution; it is the exact "resolving the comments does nothing" report from the review-bot feedback thread.

Semantics

Two deliberate asymmetries against open-thread suppression, both fail-safe:

• A thread the BOT resolved never joins the corpus. The reconciler resolving a thread means a code change fixed the defect; a fixed defect that reappears is a fresh finding that must post. Membership requires a non-empty human resolvedBy (an unattributable resolver, e.g. a deleted account, reads as no adjudication).
• A blocking candidate is never suppressed by the corpus. An adjudicated thread is closed and floors nothing, so suppressing a blocker on it would let a re-confirmed blocking defect vanish without a trace. This is also the regression escape hatch the old "resolved threads never suppress" rule protected: a fixed-then-regressed defect worth stopping the PR for re-presents at blocking severity and posts.

A candidate matching BOTH corpora is attributed to the OPEN thread (pass order is a property of the composed entry point, not the call site), so the verdict floor reads the open thread's blocking state. Suppression records carry adjudicated: true for the audit trail; they can never floor the verdict (the floor requires the candidate's label to be blocking, which no adjudicated suppression's is). Every guard fails closed toward posting a duplicate, never toward dropping a finding, and an older staging without the file degrades to an empty corpus.

Structure

Split per the max-lines convention: the corpus filter, the suppression pass, and the composed two-pass entry point live in dedup-adjudicated.ts (dedup.ts exports its matcher internals). A follow-up commit extracted the dependency-free text-similarity primitives (STOPWORDS, contentTokens, bigrams, intersectionSize) into dedup-text.ts, byte-identical, to keep dedup.ts under the 1000-line cap. Dispatch-level tests in dispatch-adjudicated.test.ts, unit tests in dedup-adjudicated.test.ts, staging tests extend stage-threads.test.ts with the producer-to-consumer byte bind.

Tests

New: corpus admission and every fail-closed guard; non-blocking suppression with the adjudicated marker; the blocking exemption; both-corpora attribution to the open thread; staging partition (human-resolved bot thread lands in the adjudicated file and nowhere else, bot-resolved and human-opened resolved threads excluded, no resolvedBy leak into threads.json); end-to-end dispatch suppression pre-validation and the blocking re-flag reaching the validator. Full workflows/review suite green; tsc and eslint clean.

Follow-ups (not in this PR)

• Downvote adjudication: feed the thumbs sweep's incorrect/duplicate answers into the same corpus.
• Accountability-section damping for unchanged non-blocking threads.
• /reconsider per-finding re-evaluation (needs a design note first).

[changeset-bot]

🦋 Changeset detected

Latest commit: 1c2a123

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Review Guidance

Common patterns

2 files: collectUnresolvedThreads renamed to collectReviewThreads at its definition and its call site (old name re-introduced as a thin filter wrapper in threads.ts)

- const allThreads = await collectUnresolvedThreads(ghGraphql, owner, repoName, prNumber);
+ const fetchedThreads = await collectReviewThreads(ghGraphql, owner, repoName, prNumber);

2 files: Parallel documentation additions describing adjudicated-threads.json and the adjudicated-thread suppression pass in both the feature overview and the agent-facing staging reference

-duplicate; a suppressed blocking candidate still floors the verdict), and
+duplicate; a suppressed blocking candidate still floors the verdict),
+adjudicated-thread suppression (a non-blocking candidate re-deriving a defect
+a human settled by resolving the bot's thread posts no new thread; a blocking
+candidate is never suppressed this way, so a regression re-flag stays
+visible), and

[github-actions]

Approved — no blocking issues found.
Note: correctness-reviewer not assessed this run (correctness-reviewer output unavailable).
Note: skill-auditor not assessed this run (skill-auditor output unavailable).
Note: test-adequacy not assessed this run (test-adequacy output unavailable).
Note: holistic not assessed this run (holistic output unavailable).
Note: first-principles not assessed this run (first-principles output unavailable).
Note: conventions not assessed this run (conventions output unavailable).
Note: documentation not assessed this run (documentation output unavailable).

[github-actions]

Review live A/B

Baseline: origin/main (review.md b839d2b5a7aa); candidate: working tree (review.md 524e646e7e53).

Ruler: matcher deterministic+arbiter; corpus cc11988c0918 (9 cases).

Metric	Baseline	Candidate	Delta
Must-catch recall	100%	100%	+0%
Verdict agreement	89%	78%	-11%
Noise (unmatched posted)	59%	50%	-9%
Clean false flags	0	0
Judge mean quality	0.86	0.90	+0.04
Cost	$8.92	$9.66
Wall clock	1238s	1470s
Cases run / skipped	9 / 0	9 / 0
Misses found-but-dropped	0	0
Findings anchor-snapped	0	0
Cross-source claims merged (of candidates)	3 / 25 (0 by clusterer at $0.27 / 91s, 6 proposed member(s) rejected)	4 / 22 (0 by clusterer at $0.27 / 97s, 8 proposed member(s) rejected)

Adversarial hard gate: PASSED on the candidate arm.

Single-run-stable rows: recall, verdict agreement, regressions, adversarial gate. Judge quality and noise are not: they jitter run-to-run at this corpus size, and a regressed reviewer can score HIGHER on judge quality (fewer, surer comments each read better). Recall against the labeled specs is the load-bearing metric.

Measured noise floor (identical arms, run 29069228968, 2026-07-10, 6 arm-samples, full corpus x3, pre-arbiter; budget skips left the samples on unequal case sets, so these v1 bands also carry case-mix variance): must-catch recall 54%-86% (sd 10%), verdict agreement 75%-100% (sd 9%), noise (unmatched posted) 50%-60% (sd 3%), judge mean quality 82%-86% (sd 2%). A single-run delta whose arms both sit inside a band is indistinguishable from run-to-run wobble; use --repeats to resolve smaller effects.

[github-actions]

Note: re-review ran at scoped depth (re-review mode scoped).

[github-actions]

thought (non-blocking): New file dedup-text.ts (STOPWORDS/contentTokens/bigrams/intersectionSize extracted from dedup.ts) is not mentioned in the PR's Structure section. The extraction is a behavior-neutral code move (verified: the primitives are byte-identical to those deleted from dedup.ts, and dedup-adjudicated.ts does not even import it), so this is silent scope rather than a functional gap — the otherwise-exhaustive Structure section just omits this one added file. Non-blocking; the feature itself fully matches its stated intent.

[github-actions]

thought (non-blocking): Extraction restores compliance but almost no headroom under the 1000-line cap. I verified the premise holds: without this split dedup.ts would be ~1007 lines against the @khanacademy/eslint-config 1000-line cap, so the extraction is justified. But only the three dependency-free text primitives (~33 lines) were carved out, and only dedup.ts imports dedup-text.ts — the seam was chosen for minimal motion, not durable headroom, leaving the file one small feature away from the cap again.

A sketch, not a committable replacement:

While the file is already being split, consider extracting a larger coherent tier (e.g., the floor constants plus the scoring/matcher helpers that dedup-adjudicated.ts consumes) into the shared module, so the next change to dedup.ts isn't immediately renegotiating the line budget.

[jwbron]

Post-deploy data point for the family-matching follow-up (window 2026-08-11T16:00Z through 2026-08-12T19:35Z in webapp; the reviewer deployed at the window start): webapp#41290 now carries five bot threads on one mechanism (the tmv2 CEDAR side effect on flagged turns) across two files. Pre-deploy: r3756443933 and r3756443949 (tmv2.go). Post-deploy: r3767202100 and r3767482526 (SPEC.md, different anchors), plus r3761098824, which the author downvoted with reason duplicate. The corpus matcher here keys on path plus the similarity floors, so cross-file variants of an adjudicated defect will still post. Not asking to widen this PR; recording the measured shape the mechanism-level follow-up needs to catch.