Repository navigation
feat(review): ask each fix for its test, and rule on non-convergence #9596
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 6 commits
e631d95
78ae615
0accb99
51cd13f
cefb01e
df6d20e
26e69ba
fd52412
61949c5
22786dd
c6cdd43
910c700
a0d09e0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2471,6 +2471,31 @@ describe('buildRoleBrief — every agent, not just the territory ones', () => { | |
| expect(p).not.toMatch(/If you find no issues, say/i); | ||
| }); | ||
|
|
||
| it('welds the fix-witness format into the launched finder briefs', () => { | ||
| // The fix-witness mandate is pinned in SKILL.md by SKILL.test.ts, but | ||
| // this half is the one that actually reaches the agents: the | ||
| // FINDING_FORMAT embedded in every finder brief. Deleting the Fix | ||
| // witness line — or the exemption clause below it — shipped green once, | ||
| // because no test read a BUILT brief; launched finders would stop being | ||
| // asked for the criterion and Step 7's posting rule would go inert on | ||
| // every agent-built round. Pin both halves through the brief. | ||
| const brief = buildRoleBrief(PLAN, '1a'); | ||
| expect(brief).toContain( | ||
| '**Fix witness:** <the test that must go RED if that fix is removed', | ||
| ); | ||
| expect(brief).toContain('**This field never gates reporting**'); | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in df6d20e, taking your suggestion as written — the exemption tail is now pinned beside the prefix in the built brief. Mutation-probed: deleting the clause from |
||
| // The exemption TAIL, pinned beside the prefix. The prefix assertion | ||
| // above stops before it, so deleting or rewording `or "N/A" ...` shipped | ||
| // green — and the two copies of the finding format (SKILL.md, pinned by | ||
| // SKILL.test.ts, and this embedded one) could drift on exactly that | ||
| // clause. Finders would then read a brief that mandates Fix witness with | ||
| // no way out, and rounds would start demanding tests for fixes that add | ||
| // no guard at all — a rename, a comment, a docs line. | ||
| expect(brief).toContain( | ||
| 'or "N/A" when the fix adds no guard, branch or behaviour a test can pin', | ||
| ); | ||
| }); | ||
|
|
||
| it('injects generic repository context into reviewers and a narrow verification boundary into Agent 7', () => { | ||
| const contextPlan = { | ||
| ...PR_PLAN, | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -217,6 +217,7 @@ const FINDING_FORMAT = `Format each finding using this structure: | |||||
| - **Issue:** <one-line statement of the defect> | ||||||
| - **Failure scenario:** <the concrete trigger and the concrete wrong outcome: what input, state, timing, or config makes this code misbehave, and what incorrect output / crash / leak / exposure results> | ||||||
| - **Suggested fix:** <concrete code suggestion when possible, or "N/A"> | ||||||
| - **Fix witness:** <the test that must go RED if that fix is removed — the test file and the behaviour it pins — or "N/A" when the fix adds no guard, branch or behaviour a test can pin> | ||||||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Suggestion] The review-agent prompt's new
Suggested change
中文说明review-agent 提示词中新增的 — deepseek-v4-flash via Qwen Code /review (v0.21.11)
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Deferred to the next round — this round's batch was bounded (~8 findings) and spent on the three Criticals first. Same work as rc:3824051695: pin the agent-prompt 中文说明延迟到下一轮——本轮批次有上限(约 8 条 finding),优先处理了三条 Critical。与 rc:3824051695 同一工作:在
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Deferred to the next round — this round's batch was bounded (~8 findings) and spent on the three Criticals first. Same work as rc:3824047036: pin the 中文说明延迟到下一轮——本轮批次有上限(约 8 条 finding),优先处理了三条 Critical。与 rc:3824047036 同一工作:在 agent-prompt.test.ts 中钉住 Fix witness 格式行与「永不阻断上报」条款。下一轮处理。 |
||||||
| - **Severity:** Critical | Suggestion | Nice to have | ||||||
| - **Confidence:** high | low | ||||||
|
|
||||||
|
|
@@ -229,7 +230,9 @@ const FINDING_FORMAT = `Format each finding using this structure: | |||||
| - A line too long to quote whole — a multi-KB single-line Markdown paragraph — may be quoted as a distinctive verbatim **fragment** of at least 12 characters (measured after whitespace collapse); it resolves to the line containing it. | ||||||
| - Fill in **File** and the line number anyway. The path selects the file and the line breaks a tie when the snippet genuinely repeats. Neither is trusted as the answer. | ||||||
|
|
||||||
| **The failure scenario is the finding's evidence, and it gates reporting.** For a quality finding, state the concrete cost instead of a crash — what is duplicated, wasted, or made harder to change — or quote the rule it violates. A **Suggestion** or **Nice to have** whose failure scenario you cannot fill in concretely **is not a finding: do not report it.** A suspected **Critical** whose trigger you cannot pin down IS still reported, at \`Confidence: low\`, with the scenario naming the mechanism and what remains uncertain — a later verification stage rules on it. "This looks risky", with no nameable trigger and no nameable cost, is how a hallucinated finding reaches a pull request.`; | ||||||
| **The failure scenario is the finding's evidence, and it gates reporting.** For a quality finding, state the concrete cost instead of a crash — what is duplicated, wasted, or made harder to change — or quote the rule it violates. A **Suggestion** or **Nice to have** whose failure scenario you cannot fill in concretely **is not a finding: do not report it.** A suspected **Critical** whose trigger you cannot pin down IS still reported, at \`Confidence: low\`, with the scenario naming the mechanism and what remains uncertain — a later verification stage rules on it. "This looks risky", with no nameable trigger and no nameable cost, is how a hallucinated finding reaches a pull request. | ||||||
|
|
||||||
| **A fix that adds a guard owes a test that fails without it — say so in the finding.** The fix round is this loop's largest single source of its own next round: measured across six multi-round pull requests, roughly a third of every post-first-round finding was introduced by the fix immediately before it, and the dominant shape was a guard or branch added with no test of its own. The suite re-runs only the tests that exist, so an unwitnessed guard passes every gate and its hole comes back as next round's finding. So when your **Suggested fix** adds or changes a guard, a branch, or a behaviour, fill **Fix witness** with the test that must go red without it — the file and what it asserts — and, where you can, the mutation that proves it: remove the guard, run that test, watch it fail. Write \`N/A\` when there is genuinely nothing to pin — a rename, a comment, a docs line, a type-only change, a fix whose whole content is deleting code. **This field never gates reporting**: a finding whose fix you cannot pin is still filed, with \`N/A\`. It is an acceptance criterion for the author, not a bar for you.`; | ||||||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Suggestion] The Fix-witness mandate and the N/A exemption contradict each other inside the same new paragraph: the mandate says a fix that "adds or changes a guard, a branch, or a behaviour" must fill Fix witness, while the exemption list immediately grants
Suggested change
中文说明Fix-witness 强制条款与 N/A 豁免在同一个新段落内互相矛盾:强制条款说「新增或改变 guard、分支或行为」的修复必须填 Fix witness,而豁免清单紧接着把 — deepseek-v4-flash via Qwen Code /review (v0.21.11)
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Deferred to the next round — this round's batch was bounded (~8 findings) and spent on the three Criticals first. The deletion-exemption contradiction (a guard removal is both a deletion and a behaviour change) needs the carve-out wording; next round, together with the matching agent-prompt paragraph. 中文说明延迟到下一轮——本轮批次有上限(约 8 条 finding),优先处理了三条 Critical。删除豁免的矛盾(移除 guard 既是删除又是行为改变)需要豁免排除措辞;下一轮与 agent 提示词对应段落一并处理。 |
||||||
|
|
||||||
| /** | ||||||
| * What not to report. | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deferred to the next round, not declined: a two-line assertion pinning the N/A exemption tail. Critical-only mode is active with the window over its test budget (423/400 net test lines), so this round's test budget went to the regression tests witnessing the Critical fixes; the prompt line the pin protects is untouched this round and the finding does not go stale.
中文说明
延后到下一轮,并非拒绝:这是钉住 N/A 豁免尾句的两行断言。当前处于仅处理 Critical 的模式,且窗口测试预算已超(净测试行 423/400),本轮的测试预算用于见证 Critical 修复的回归测试;该断言保护的提示词文本本轮未被触碰,发现不会过时。