Skip to content
9 changes: 9 additions & 0 deletions .changeset/review-subject-restatement-and-body-stats.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
"review": minor
---

Stop posting a subject line that restates the discussion, and stage body-size stats with every submission plan.

The 2026-08-20 by-version audit of webapp reviews found a prose repetition cluster in the v1.11.0-v1.13.0 windows (5 of 29 blind-judged bodies restated one fact two to four times, vs 1 of 60 before), and the root cause is mechanical: `joinProse` concatenates the label contract's `subject` and `discussion` verbatim, producers routinely emit a subject that restates a discussion claim, and the v1.8.0 task-mode removal (#284/#288/#289) deleted the orchestrator rewrite pass that used to absorb the overlap. `joinProse` now drops a subject whose folded tokens are all contained in a single discussion sentence (stopwords ignored on the subject side, light inflection folding on both sides, markdown and trailing punctuation stripped); a subject carrying any token no single sentence holds is kept whole, so the check can only remove a duplicate sentence, never content. `buildClaims`' first-sentence split then recovers the discussion's own opening claim as the subject, so no downstream field goes empty.
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved

The same audit found the +60% median-body step (557 to 889 chars across webapp's v1.7.0 to v1.11.0 bump) shipped with no changeset naming it, and a by-version audit four versions later was the detection mechanism. `submission-plan.json` now carries `bodyStats` (comment count, median/p90/max/total rendered chars over the final comment bodies, footer rides included, plus the review body's length), echoed in the CLI's run-log summary, so the next render-path regression is visible in the first runs' artifacts. Artifact-only; nothing gates on it. The README's versioning section now requires render-path changesets to state their expected output-shape effect.
14 changes: 14 additions & 0 deletions workflows/review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -824,6 +824,20 @@ bump lands in the same Version Packages commit that gets tagged.
literals do not match the `review` package version (releases v1.3.0 through
v1.4.0 shipped still pointing at v1.2.2, before the sync existed).

### Output-shape changes belong in the changeset

A change to the render path (`lib/submission.ts`, `lib/render-comment.ts`,
`lib/dispatch-contracts.ts` prose composition) that alters what authors see
must say so in its changeset, including the expected direction of body-size
change. The v1.8.0 task-mode removal shipped a +60% median-body step (557 to
889 chars across webapp's v1.7.0 to v1.11.0 bump, sketch share 15.6% to 48.6%)
with changesets framed entirely as conformance wins; the regression went
unattributed for four versions until a by-version audit reconstructed it
(PRA-46). Each staged `submission-plan.json` now carries `bodyStats`
(per-comment median/p90/max/total chars and the review body's length), so
compare a release's first live artifacts against the prior version's when a
render change ships.

### Version attribution

Semver is the behavior contract: a release that changes the reviewer's behavior bumps
Expand Down
59 changes: 59 additions & 0 deletions workflows/review/lib/dispatch-contracts.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,65 @@ describe("label-contract enforcement (run 29897276810)", () => {
expect(joinProse("Only a subject", "")).toBe("Only a subject");
expect(joinProse("", "Only a discussion.")).toBe("Only a discussion.");
});

it("drops a subject that restates a discussion sentence (PRA-46 W4-W5 repetition mode)", () => {
// Verbatim restatement: the discussion opens with the subject.
expect(
joinProse(
"The merger drops flagged turns.",
"The merger drops flagged turns whenever moderation flags a turn mid-stream.",
),
).toBe(
"The merger drops flagged turns whenever moderation flags a turn mid-stream.",
);
// Inflected restatement: "dropped" vs "drops" still folds together.
expect(
joinProse(
"Flagged turns are dropped by the merger.",
"The merger drops flagged turns because the filter runs before the merge, so a flagged turn never reaches the sink.",
),
).toBe(
"The merger drops flagged turns because the filter runs before the merge, so a flagged turn never reaches the sink.",
);
// Markdown wrapping does not defeat the comparison.
expect(
joinProse(
"`counts.go` recomputes the total.",
"counts.go recomputes the total on every call.",
),
).toBe("counts.go recomputes the total on every call.");
// Restating a LATER sentence is the same duplication.
expect(
joinProse(
"A delete leaves the stale entry behind.",
"The cache is written in save(). A delete leaves the stale entry behind.",
),
).toBe(
"The cache is written in save(). A delete leaves the stale entry behind.",
);
});

it("keeps a subject that carries information the opening sentence lacks", () => {
// "never invalidated" is not in the first sentence: kept whole.
expect(
joinProse(
"The cache is never invalidated.",
"The cache is written in save(). A delete leaves the stale entry behind.",
),
).toBe(
"The cache is never invalidated. The cache is written in save(). A delete leaves the stale entry behind.",
);
// A subject summarizing ACROSS sentences (no single sentence holds
// all its tokens) is a genuine lede and survives.
expect(
joinProse(
"save() caches, delete leaves it stale.",
"The cache is written in save(). A delete leaves the stale entry behind.",
),
).toBe(
"save() caches, delete leaves it stale. The cache is written in save(). A delete leaves the stale entry behind.",
);
});
});

describe("label-shape lens assignment", () => {
Expand Down
117 changes: 116 additions & 1 deletion workflows/review/lib/dispatch-contracts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,18 +87,133 @@ export type Candidate = {
authorDispute?: string;
};

/**
* Fold a prose token toward its stem so an inflection difference does not
* defeat the restatement check below ("drops" vs "dropped", "cache" vs
* "caches"). Deliberately crude: strip one of ing/ed/es/s, collapse a
* doubled final consonant ("dropped" -> "dropp" -> "drop"), then strip a
* trailing "e" so "caches" -> "cach" meets "cache" -> "cach". Both sides
* of every comparison fold identically, and a miss is safe — the subject
* is kept and the body merely stays as long as it is today.
*/
const foldToken = (token: string): string => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (non-blocking): ~110 lines of stemming/stopword machinery for a mode whose common case is a verbatim prefix. Three of the four positive tests are near-verbatim restatements; only one needs inflection folding. The repo already has a normalize-and-compare-prefix idiom for precisely "does this field restate that one" in dedup.ts's comparisonKey/comparedText.

A sketch, not a committable replacement:

Consider starting with normalized substring containment (`comparisonKey(subject)` contained in `comparisonKey(sentence)`) and adding the stemmer only if live artifacts show the inflected case is common enough to matter — the token-fold path can then be justified by measured misses rather than anticipated ones.
review details found by first-principles

if (token.length <= 3) {
return token;
}
let folded = token;
for (const suffix of ["ing", "ed", "es", "s"]) {
if (folded.endsWith(suffix) && folded.length - suffix.length >= 3) {
folded = folded.slice(0, folded.length - suffix.length);
break;
}
}
if (/([b-df-hj-np-tv-z])\1$/.test(folded)) {
folded = folded.slice(0, -1);
}
return folded.length >= 4 && folded.endsWith("e")
? folded.slice(0, -1)
: folded;
};

/**
* Function words that carry no claim content; ignored on the SUBJECT side
* of the restatement check so "turns are dropped" still matches "drops
* turns" (the sentence has no "are"). Never filtered from the sentence
* side — there they can only help containment, not hurt it.
*/
const STOPWORDS = new Set([
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
"a",
"an",
"the",
"is",
"are",
"was",
"were",
"be",
"been",
"being",
"it",
"its",
"this",
"that",
"these",
"those",
"and",
"or",
"of",
"to",
"in",
"on",
"by",
"for",
"with",
"as",
"at",
"so",
]);

/**
* The comparable word tokens of a prose fragment: markdown emphasis
* stripped, lowercased, internal punctuation kept (`counts.go` is one
* token) but trailing punctuation shed ("turns." and "turns" are the same
* word).
*/
const proseTokens = (text: string): string[] =>
(
text
.toLowerCase()
.replace(/[`*_]/g, "")
.match(/[a-z0-9][a-z0-9./:-]*/g) ?? []
).map((token) => token.replace(/[./:-]+$/, ""));

/**
* Whether the subject merely restates one sentence of the discussion:
* every folded subject token already appears in a single discussion
* sentence, so prepending the subject adds repetition and no vocabulary.
* This is the W4-W5 prose failure mode the 2026-08-20 version audit
* measured (5 of 29 sampled bodies restated one fact two to four times, vs
* 1 of 60 in W0-W3): producers routinely emit a `subject` that restates a
* `discussion` claim, and the v1.8.0 task-mode removal deleted the
* orchestrator rewrite pass that used to absorb the overlap (PRA-46).
* Per-sentence containment is the conservative direction — a subject
* summarizing ACROSS sentences (tokens no single sentence holds) is a
* genuine lede and is kept whole, as is one carrying any token the
* discussion lacks.
*/
export const subjectRestatesDiscussion = (
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
subject: string,
discussion: string,
): boolean => {
const subjectTokens = proseTokens(subject)
.filter((token) => !STOPWORDS.has(token))
.map(foldToken);
if (subjectTokens.length === 0) {
return false;
}
return discussion.split(/(?<=[.!?])\s/).some((sentence) => {
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
const sentenceTokens = new Set(proseTokens(sentence).map(foldToken));
return subjectTokens.every((token) => sentenceTokens.has(token));
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
});
};

/**
* Join the label contract's `subject` and `discussion` into one prose block.
* A subject with no terminal punctuation gets a sentence break, not a bare
* space (run 29897276810 posted "...memory Both TestExpiration..."); the
* break also keeps `buildClaims`' first-sentence split recovering the
* subject.
*
* A subject that restates the discussion's opening sentence
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
* ({@link subjectRestatesDiscussion}) is dropped instead of joined: the
* posted body then opens with the discussion's own first claim, and
* `buildClaims`' first-sentence split recovers that as the subject, so
* nothing downstream loses a field — only the duplicate sentence.
*/
export const joinProse = (subject: string, discussion: string): string => {
if (discussion === "") {
return subject.trim();
}
if (subject === "") {
if (subject === "" || subjectRestatesDiscussion(subject, discussion)) {
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
return discussion.trim();
}
const trimmed = subject.trimEnd();
Expand Down
58 changes: 58 additions & 0 deletions workflows/review/lib/submission.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import {evaluateDispatchConformance} from "./dispatch-gate";
import {labelForFinding, renderComment} from "./render-comment";
import {renderRereviewStamp, STAMP_SCHEMA_VERSION} from "./rereview-mode";
import {
computeBodyStats,
MAX_VERBATIM_FOLD_CHARS,
renderClaimComment,
runSubmissionCli,
Expand Down Expand Up @@ -83,6 +84,22 @@ const priorApprove = (): Record<string, string> => ({
}),
});

describe("computeBodyStats", () => {
it("reports nearest-rank percentiles over comment body lengths", () => {
const comments = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100].map(
(length) => ({path: "a.ts", line: 1, body: "x".repeat(length)}),
);
expect(computeBodyStats(comments, "body")).toEqual({
comments: 10,
medianChars: 50, // nearest-rank: ceil(0.5 * 10) = 5th of 10
p90Chars: 90, // ceil(0.9 * 10) = 9th of 10
maxChars: 100,
totalChars: 550,
bodyChars: 4,
});
});
});

describe("runSubmissionCli", () => {
it("plans REQUEST_CHANGES with the fixed body line when a blocking claim posts", () => {
const fs = makeFakeFs(
Expand Down Expand Up @@ -143,6 +160,47 @@ describe("runSubmissionCli", () => {
expect(plan.body).toContain("Approved — no blocking issues found.");
});

it("stages body-size stats measured over the final rendered bodies (PRA-46)", () => {
const fs = makeFakeFs(
staged({
depth: "full",
claims: [claim()],
noteLines: [],
reconciliation: {resolve: [], keep: []},
}),
);
const plan = runSubmissionCli(fs);
// One comment: every percentile is that comment's FINAL length
// (attribution footer included), not the bare claim render.
const commentChars = plan.comments[0]?.body.length as number;
expect(plan.bodyStats).toEqual({
comments: 1,
medianChars: commentChars,
p90Chars: commentChars,
maxChars: commentChars,
totalChars: commentChars,
bodyChars: plan.body.length,
});
expect(plan.comments[0]?.body).toContain("review details");
// The stats ride the staged artifact for the live drift watch.
expect(
JSON.parse(fs.files[`${REVIEW}/submission-plan.json`]).bodyStats,
).toEqual(plan.bodyStats);
});

it("stages zeroed comment stats when nothing posts inline", () => {
const fs = makeFakeFs(staged({depth: "full", claims: []}));
const plan = runSubmissionCli(fs);
expect(plan.bodyStats).toEqual({
comments: 0,
medianChars: 0,
p90Chars: 0,
maxChars: 0,
totalChars: 0,
bodyChars: plan.body.length,
});
});

it("applies the reduced-depth flip floor from kept blocking threads", () => {
const stamp = renderRereviewStamp({
schemaVersion: STAMP_SCHEMA_VERSION,
Expand Down
Loading
Loading