Repository navigation
review: drop a subject that restates the discussion; stage body-size stats #365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
0116e7d
36d3da7
fab4730
3a2547d
e746282
542d051
3da9c08
68591d0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| "review": minor | ||
| --- | ||
|
|
||
| Stop posting a subject line that restates the discussion, and stage body-size stats with every submission plan. | ||
|
|
||
| The 2026-08-20 by-version audit of webapp reviews found a prose repetition cluster in the v1.11.0-v1.13.0 windows (5 of 29 blind-judged bodies restated one fact two to four times, vs 1 of 60 before), and the root cause is mechanical: `joinProse` concatenates the label contract's `subject` and `discussion` verbatim, producers routinely emit a subject that restates a discussion claim, and the v1.8.0 task-mode removal (#284/#288/#289) deleted the orchestrator rewrite pass that used to absorb the overlap. `joinProse` now drops a subject whose folded tokens are all contained in the discussion's first sentence (stopwords ignored on the subject side, light inflection folding on both sides, markdown and trailing punctuation stripped); a subject restating a later sentence or carrying any token the first sentence lacks is kept whole. `buildClaims`' first-sentence split then recovers the discussion's own opening claim as the subject, so no downstream field goes empty, and the `failure_scenario` salvage for a dropped subject reads the discussion instead, keeping dedup's `comparedText` on its discussion branch. Expected output-shape effect: median inline comment bodies shrink by up to one sentence each (the drop removes exactly the duplicated subject line and adds nothing); review bodies are unchanged. | ||
|
|
||
| The same audit found the +60% median-body step (557 to 889 chars across webapp's v1.7.0 to v1.11.0 bump) shipped with no changeset naming it, and a by-version audit four versions later was the detection mechanism. `submission-plan.json` now carries `bodyStats` (comment count, median/p90/max/total rendered chars over the final comment bodies, footer rides included, plus the review body's length), echoed in the CLI's run-log summary, so the next render-path regression is visible in the first runs' artifacts; a copy of the plan is staged under `out/`, the one directory the run uploads. Artifact-only; nothing gates on it. The README's versioning section now requires render-path changesets to state their expected output-shape effect. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -87,18 +87,140 @@ export type Candidate = { | |
| authorDispute?: string; | ||
| }; | ||
|
|
||
| /** | ||
| * Fold a prose token toward its stem so an inflection difference does not | ||
| * defeat the restatement check below ("drops" vs "dropped", "cache" vs | ||
| * "caches"). Deliberately crude: strip one of ing/ed/es/s, collapse a | ||
| * doubled final consonant ("dropped" -> "dropp" -> "drop"), then strip a | ||
| * trailing "e" so "caches" -> "cach" meets "cache" -> "cach". Both sides | ||
| * of every comparison fold identically, and a miss is safe — the subject | ||
| * is kept and the body merely stays as long as it is today. | ||
| */ | ||
| const foldToken = (token: string): string => { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. suggestion (non-blocking): ~110 lines of stemming/stopword machinery for a mode whose common case is a verbatim prefix. Three of the four positive tests are near-verbatim restatements; only one needs inflection folding. The repo already has a normalize-and-compare-prefix idiom for precisely "does this field restate that one" in A sketch, not a committable replacement: review detailsfound by first-principles |
||
| if (token.length <= 3) { | ||
| return token; | ||
| } | ||
| let folded = token; | ||
| for (const suffix of ["ing", "ed", "es", "s"]) { | ||
| if (folded.endsWith(suffix) && folded.length - suffix.length >= 3) { | ||
| folded = folded.slice(0, folded.length - suffix.length); | ||
| break; | ||
| } | ||
| } | ||
| if (/([b-df-hj-np-tv-z])\1$/.test(folded)) { | ||
| folded = folded.slice(0, -1); | ||
| } | ||
| return folded.length >= 4 && folded.endsWith("e") | ||
| ? folded.slice(0, -1) | ||
| : folded; | ||
| }; | ||
|
|
||
| /** | ||
| * Function words that carry no claim content; ignored on the SUBJECT side | ||
| * of the restatement check so "turns are dropped" still matches "drops | ||
| * turns" (the sentence has no "are"). Never filtered from the sentence | ||
| * side — there they can only help containment, not hurt it. | ||
| */ | ||
| const STOPWORDS = new Set([ | ||
|
khan-actions-bot marked this conversation as resolved.
Resolved
|
||
| "a", | ||
| "an", | ||
| "the", | ||
| "is", | ||
| "are", | ||
| "was", | ||
| "were", | ||
| "be", | ||
| "been", | ||
| "being", | ||
| "it", | ||
| "its", | ||
| "this", | ||
| "that", | ||
| "these", | ||
| "those", | ||
| "and", | ||
| "or", | ||
| "of", | ||
| "to", | ||
| "in", | ||
| "on", | ||
| "by", | ||
| "for", | ||
| "with", | ||
| "as", | ||
| "at", | ||
| "so", | ||
| ]); | ||
|
|
||
| /** | ||
| * The comparable word tokens of a prose fragment: markdown emphasis | ||
| * stripped, lowercased, internal punctuation kept (`counts.go` is one | ||
| * token) but trailing punctuation shed ("turns." and "turns" are the same | ||
| * word). | ||
| */ | ||
| const proseTokens = (text: string): string[] => | ||
| ( | ||
| text | ||
| .toLowerCase() | ||
| .replace(/[`*_]/g, "") | ||
| .match(/[a-z0-9][a-z0-9./:-]*/g) ?? [] | ||
| ).map((token) => token.replace(/[./:-]+$/, "")); | ||
|
|
||
| /** | ||
| * Whether the subject merely restates the discussion's FIRST sentence: | ||
| * every folded subject token already appears there, so prepending the | ||
| * subject adds repetition and no vocabulary. This is the W4-W5 prose | ||
| * failure mode the 2026-08-20 version audit measured (5 of 29 sampled | ||
| * bodies restated one fact two to four times, vs 1 of 60 in W0-W3): | ||
| * producers routinely emit a `subject` that restates the `discussion` | ||
| * lede, and the v1.8.0 task-mode removal deleted the orchestrator rewrite | ||
| * pass that used to absorb the overlap (PRA-46). | ||
| * | ||
| * First-sentence-only is deliberate. When the drop fires, `buildClaims`' | ||
| * first-sentence split recovers the discussion's opening sentence as | ||
| * `claim.subject`, and that string is a visible header downstream (the | ||
| * HOLD_FOR_HUMAN and over-cap collapsed lists, `renderPrLevelFold`), so it | ||
| * must be the claim; matching a later sentence would leave setup prose | ||
| * there. A subject restating a later sentence, one summarizing across | ||
| * sentences, or one carrying any token the first sentence lacks is kept | ||
| * whole. The comparison is an unordered token bag, so a subject reusing | ||
| * the sentence's exact vocabulary to state a different relation would be | ||
| * dropped too; accepted, since the audited failure mode is restatement and | ||
| * the sentence carrying that vocabulary still posts. | ||
| */ | ||
| const subjectRestatesDiscussion = ( | ||
| subject: string, | ||
| discussion: string, | ||
| ): boolean => { | ||
| const subjectTokens = proseTokens(subject) | ||
| .filter((token) => !STOPWORDS.has(token)) | ||
| .map(foldToken); | ||
| if (subjectTokens.length === 0) { | ||
| return false; | ||
| } | ||
| const firstSentence = discussion.split(/(?<=[.!?])\s/, 1)[0] ?? ""; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. question (non-blocking): The sentence split review detailsfound by skill-auditor (out-of-lane) | also flagged by conventions
khan-actions-bot marked this conversation as resolved.
Resolved
khan-actions-bot marked this conversation as resolved.
Resolved
|
||
| const sentenceTokens = new Set(proseTokens(firstSentence).map(foldToken)); | ||
| return subjectTokens.every((token) => sentenceTokens.has(token)); | ||
| }; | ||
|
|
||
| /** | ||
| * Join the label contract's `subject` and `discussion` into one prose block. | ||
| * A subject with no terminal punctuation gets a sentence break, not a bare | ||
| * space (run 29897276810 posted "...memory Both TestExpiration..."); the | ||
| * break also keeps `buildClaims`' first-sentence split recovering the | ||
| * subject. | ||
| * | ||
| * A subject that restates the discussion's opening sentence | ||
|
khan-actions-bot marked this conversation as resolved.
Resolved
|
||
| * ({@link subjectRestatesDiscussion}) is dropped instead of joined: the | ||
| * posted body then opens with the discussion's own first claim, and | ||
| * `buildClaims`' first-sentence split recovers that as the subject, so no | ||
| * downstream field goes empty; the body loses the duplicate sentence. | ||
| */ | ||
| export const joinProse = (subject: string, discussion: string): string => { | ||
| if (discussion === "") { | ||
| return subject.trim(); | ||
| } | ||
| if (subject === "") { | ||
| if (subject === "" || subjectRestatesDiscussion(subject, discussion)) { | ||
|
khan-actions-bot marked this conversation as resolved.
Resolved
khan-actions-bot marked this conversation as resolved.
Resolved
khan-actions-bot marked this conversation as resolved.
Resolved
|
||
| return discussion.trim(); | ||
| } | ||
| const trimmed = subject.trimEnd(); | ||
|
|
@@ -167,9 +289,20 @@ const fromLabelShape = ( | |
| // valid labels with only {id, anchor, discussion}; rejecting it for | ||
| // the missing failure_scenario voided the whole correctness | ||
| // dimension twice, which is strictly worse than validating against | ||
| // the discussion prose. | ||
| // the discussion prose. A subject joinProse drops (restatement) | ||
| // salvages from the discussion too: dedup's comparedText reads the | ||
| // discussion only when failure_scenario prefix-matches | ||
| // claim.subject (dedup.ts), and after the drop claim.subject is | ||
| // the discussion's first sentence, which prefix-matches the | ||
| // discussion itself but not an inflected or reordered dropped | ||
| // subject; salvaging that subject would compare the claim on one | ||
| // sentence plus its own restatement, the exact shape run | ||
| // 30301235749 failed to merge. | ||
| failure_scenario: | ||
| raw["failure_scenario"] ?? (subject !== "" ? subject : discussion), | ||
| raw["failure_scenario"] ?? | ||
| (subject !== "" && !subjectRestatesDiscussion(subject, discussion) | ||
| ? subject | ||
| : discussion), | ||
| producing_hunt: `dispatch:${agentName}`, | ||
| model_authored_prose: joinProse(subject, discussion), | ||
| // Suggestion salvage, like the anchor/subject salvage above: run | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.