Repository navigation
perf(review): extend the convergence pair to 3B (chunked) reviews #8903
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
434e9c7
e3c274d
1da9264
04f1179
876d315
3e231e5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,6 +45,7 @@ import { | |
| DEADLINE_ENV, | ||
| RESERVE_ENV, | ||
| COMPOSE_FLOOR_ENV, | ||
| TOOL_CONCURRENCY_ENV, | ||
| readBudgetStop, | ||
| readRoundStamps, | ||
| } from './lib/deadline.js'; | ||
|
|
@@ -2658,6 +2659,7 @@ describe('the reverse-audit budget gate — the loop must end by reporting', () | |
| afterEach(() => { | ||
| delete process.env[DEADLINE_ENV]; | ||
| delete process.env[RESERVE_ENV]; | ||
| delete process.env[TOOL_CONCURRENCY_ENV]; | ||
| process.exitCode = undefined; | ||
| for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true }); | ||
| }); | ||
|
|
@@ -3103,6 +3105,59 @@ describe('the reverse-audit budget gate — the loop must end by reporting', () | |
| // A refusal is not an admission. | ||
| expect(readRoundStamps(plan)).toHaveLength(1); | ||
| }); | ||
|
|
||
| it('prices the 3B pair as one admission — round 2 bears the pair wall', () => { | ||
| // Round 2's build lands seconds after round 1's stamp, so nothing has | ||
| // measured a round yet; the price is both members' wall in waves of the | ||
| // tool-concurrency pool. PLAN has three chunks; at a 2-slot pool each | ||
| // round runs two waves and the pair three, so round 2 pays 3/2 of the | ||
| // round estimate — and the gate refuses it when the reserve plus that | ||
| // does not fit, even though round 1 (one estimate) just admitted. | ||
| process.env[TOOL_CONCURRENCY_ENV] = '2'; | ||
| process.env[RESERVE_ENV] = '600'; | ||
| process.env[DEADLINE_ENV] = String(Math.floor(Date.now() / 1000) + 3000); | ||
| const plan = call('reverse-audit', { 'all-chunks': true, round: 1 }); | ||
| expect(process.exitCode).toBeUndefined(); | ||
| expect(readRoundStamps(plan).some((st) => st.round === 1)).toBe(true); | ||
|
|
||
| (writeStdoutLine as unknown as Mock).mockClear(); | ||
| call('reverse-audit', { 'all-chunks': true, round: 2 }, plan); | ||
| // Reserve 600 + pair price 2700 = 3300 > the 3000 remaining. | ||
| expect(process.exitCode).toBe(4); | ||
| expect((writeStdoutLine as unknown as Mock).mock.calls).toHaveLength(0); | ||
| expect(readBudgetStop(plan)?.entry).toBe( | ||
| 'reverse audit — stopped before round 2 by the review time budget', | ||
| ); | ||
| expect(readRoundStamps(plan)).toHaveLength(1); | ||
| }); | ||
|
|
||
| it('admits the 3B pair when the reserve plus the pair wall fits', () => { | ||
| process.env[TOOL_CONCURRENCY_ENV] = '2'; | ||
| process.env[RESERVE_ENV] = '600'; | ||
| process.env[DEADLINE_ENV] = String(Math.floor(Date.now() / 1000) + 3400); | ||
| const plan = call('reverse-audit', { 'all-chunks': true, round: 1 }); | ||
| expect(process.exitCode).toBeUndefined(); | ||
| (writeStdoutLine as unknown as Mock).mockClear(); | ||
| call('reverse-audit', { 'all-chunks': true, round: 2 }, plan); | ||
| expect(process.exitCode).toBeUndefined(); | ||
| expect(readRoundStamps(plan).map((st) => st.round)).toEqual([1, 2]); | ||
| expect(readBudgetStop(plan)).toBeNull(); | ||
| }); | ||
|
|
||
| it('prices the pair at one round when the pool holds both fan-outs at once', () => { | ||
| // The default 10-slot pool holds all six auditors of PLAN's 3-chunk | ||
| // pair in one wave, so round 2 pays one round estimate — a flat 2x | ||
| // price would refuse this admission (reserve 600 + 3600 > 3000) and | ||
| // gut the pair's admission win near the deadline. | ||
| process.env[RESERVE_ENV] = '600'; | ||
| process.env[DEADLINE_ENV] = String(Math.floor(Date.now() / 1000) + 3000); | ||
| const plan = call('reverse-audit', { 'all-chunks': true, round: 1 }); | ||
| expect(process.exitCode).toBeUndefined(); | ||
| (writeStdoutLine as unknown as Mock).mockClear(); | ||
| call('reverse-audit', { 'all-chunks': true, round: 2 }, plan); | ||
| expect(process.exitCode).toBeUndefined(); | ||
| expect(readRoundStamps(plan).map((st) => st.round)).toEqual([1, 2]); | ||
| }); | ||
| }); | ||
|
|
||
| describe('per-chunk retirement — cold territories stop costing a round', () => { | ||
|
|
@@ -3310,6 +3365,30 @@ describe('per-chunk retirement — cold territories stop costing a round', () => | |
| expect(keysOf(2)).toHaveLength(3); | ||
| }); | ||
|
|
||
| it('the 3B pair: round 2 builds every chunk with round 1 still in flight (no round-1 transcripts)', () => { | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Suggestion] This test proves that both prompt builders can run without round-1 transcripts, but it does not pin the effect-bearing SKILL.md instruction that both 3B rounds must launch in the same response. — Concrete cost: a later edit can serialize the skill while this test stays green because it calls 中文说明这个测试证明了两个 prompt builder 可以在没有 round-1 transcript 时运行,但没有固定真正影响行为的 SKILL.md 约束:两个 3B rounds 必须在同一个 response 中启动。具体代价:后续可以把 skill 改回串行,而该测试仍然通过,因为它自己顺序调用 — Qwen Code via Qwen Code /review (v0.21.9) |
||
| // The convergence pair on 3B — the latency lever: rounds 1 and 2 are | ||
| // launched together, so round 2's builder runs BEFORE round 1's auditors | ||
| // have returned any transcript. Round 2 must still fan out to every chunk | ||
| // (the retirement schedule only reads history at k >= 3, so nothing here | ||
| // depends on round 1's records existing) and stamp its own admission, so | ||
| // the two rounds' auditors run concurrently instead of one round-wall | ||
| // apart. Pins the mechanism the SKILL 3B-pair orchestration relies on. | ||
| const r1 = runRound(1); // built, but no transcripts written for it | ||
| expect(r1).toContain('3 auditors required this round — one per chunk.'); | ||
| const r2 = runRound(2); // round 1's transcripts don't exist yet at this point | ||
| expect(r2).toContain('3 auditors required this round — one per chunk.'); | ||
| expect(r2).not.toContain('retirement:'); | ||
| expect(keysOf(1)).toHaveLength(3); | ||
| expect(keysOf(2)).toHaveLength(3); | ||
| // Both admissions are stamped, so the deadline gate prices each and the | ||
| // clock advances a round per stamp. | ||
| const rounds = readRoundStamps(plan) | ||
| .map((s) => s.round) | ||
| .sort(); | ||
| expect(rounds).toContain(1); | ||
| expect(rounds).toContain(2); | ||
| }); | ||
|
|
||
| it('round 3 skips a chunk dry in rounds 1 and 2, and the note names it', () => { | ||
| answerRound(1, { 13: DRY, 14: YIELD, 15: YIELD }); | ||
| answerRound(2, { 13: DRY, 14: YIELD, 15: YIELD }); | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,7 +45,7 @@ import { writeStdoutLine, writeStderrLine } from '../../utils/stdioHelpers.js'; | |
| import { launchToolBudget, reverseAuditRoundCap } from './lib/budget.js'; | ||
| import { | ||
| clearBudgetStop, | ||
| expectedRoundSeconds, | ||
| expectedAdmissionSeconds, | ||
| readRoundStamps, | ||
| reverseAuditBudgetExhausted, | ||
| reverseAuditBudgetMessage, | ||
|
|
@@ -1862,12 +1862,17 @@ function requireAuditableChunks(report: PlanReport): DiffChunk[] { | |
| * round was refused: the caller builds nothing. The admission STAMP is not | ||
| * written here — it lands after the build succeeds, in each build path: the | ||
| * stamp is what the next round's gate measures cost from, and a build that | ||
| * throws must not leave one behind. | ||
| * throws must not leave one behind. `fanOutWidth` is the auditors this | ||
| * round fans out (1 for a whole-diff round): when the previous round is | ||
| * still in flight — the convergence pair's second member — the price | ||
| * covers both members' wall in waves of the tool-concurrency pool, not | ||
| * just this round's (deadline.ts `expectedAdmissionSeconds`). | ||
| */ | ||
| function admitReverseAuditRound( | ||
| planPath: string, | ||
| round: number | undefined, | ||
| cap: number, | ||
| fanOutWidth: number, | ||
| ): boolean { | ||
| // The plan's round cap first: deterministic, and cheaper than the | ||
| // deadline arithmetic. The full cap normally; a reduced cap for a huge | ||
|
|
@@ -1900,7 +1905,7 @@ function admitReverseAuditRound( | |
| } | ||
| const spent = reverseAuditBudgetExhausted( | ||
| process.env, | ||
| expectedRoundSeconds(planPath, round), | ||
| expectedAdmissionSeconds(planPath, round, fanOutWidth, process.env), | ||
| ); | ||
| if (spent !== null) { | ||
| writeBudgetStop(planPath, spent, round); | ||
|
|
@@ -2009,6 +2014,7 @@ function runAllChunks( | |
| planPath, | ||
| round, | ||
| reverseAuditRoundCap(report.budget), | ||
| chunks.length, | ||
| ) | ||
| ) { | ||
| return; | ||
|
|
@@ -2403,7 +2409,9 @@ function runAgentPrompt(args: AgentPromptArgs): void { | |
| // admits on the reserve alone hands the terminal round a start right at | ||
| // the boundary, which is the killed-mid-verification failure one round | ||
| // wide. The round's cost is the previous round's, measured admission to | ||
| // admission. The admission is stamped AFTER the build succeeds (below), | ||
| // admission — except when this round launches with the previous one still | ||
| // in flight (the convergence pair), where it covers both. The admission is | ||
| // stamped AFTER the build succeeds (below), | ||
| // never here: the stamp is what the next round's gate measures cost from, | ||
| // and a build that throws must not leave one behind — priced from a | ||
| // failed build, the next round would be floored to the 600s minimum, | ||
|
|
@@ -2421,6 +2429,7 @@ function runAgentPrompt(args: AgentPromptArgs): void { | |
| args.plan, | ||
| args.round, | ||
| reverseAuditRoundCap(report.budget), | ||
| 1, | ||
| ) | ||
| ) { | ||
| return; | ||
|
|
@@ -2471,14 +2480,17 @@ function runAgentPrompt(args: AgentPromptArgs): void { | |
| hasChunk && | ||
| !readRoundStamps(args.plan).some((s) => s.round === (args.round ?? null)) | ||
| ) { | ||
| const planChunkIds = ( | ||
| Array.isArray(report.chunks) ? (report.chunks as DiffChunk[]) : [] | ||
| ) | ||
| .map((c) => c?.id) | ||
| .filter((id): id is number => typeof id === 'number'); | ||
| if (args.round !== undefined) { | ||
| let schedule: RoundSchedule | null = null; | ||
| try { | ||
| schedule = scheduleReverseAuditRound( | ||
| args.plan, | ||
| (Array.isArray(report.chunks) ? (report.chunks as DiffChunk[]) : []) | ||
| .map((c) => c?.id) | ||
| .filter((id): id is number => typeof id === 'number'), | ||
| planChunkIds, | ||
| args.round, | ||
| process.env, | ||
| typeof report.diffPathAbsolute === 'string' | ||
|
|
@@ -2500,6 +2512,7 @@ function runAgentPrompt(args: AgentPromptArgs): void { | |
| args.plan, | ||
| args.round, | ||
| reverseAuditRoundCap(report.budget), | ||
| planChunkIds.length, | ||
| ) | ||
|
Comment on lines
2512
to
2516
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Deferred this round (not dropped): the wiring is correct as written — the finding itself classifies this as a missing guard, not a live defect — and this round landed the cheaper fixes under a budget warning. Test recipe preserved for the follow-up: one handler-level test admitting round 1 via 中文说明本轮推迟(并非丢弃):接线按现状是正确的——发现本身将其归类为缺失的防护,而非现存缺陷——且本轮在预算警告下落地了成本较低的修复。为后续保留的测试配方:一个 handler 级测试,先通过 |
||
| ) | ||
| return; | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Declined — no code change exists: the suggested fix is "N/A for the code" by its own text. The hunk in this file is formatting-only (a destructuring re-wrap), and the review's own explicit run against the PR bundle passed 11/11. The residual risk named — the
Integration Tests (CLI, No Sandbox)job being skipped on this PR — is a merge-time check owned by the workflow/maintainer, not something a branch change can fix. Leaving this thread open so the check stays visible at merge time.中文说明
拒绝——不存在可做的代码改动:建议修复按原文即为 "N/A for the code"。该文件中被改动的 hunk 仅是格式化(解构重排),且评审自己针对 PR bundle 的显式运行 11/11 通过。所指的残余风险——
Integration Tests (CLI, No Sandbox)作业在本 PR 上被跳过——是 workflow/维护者在合入时的检查项,不是分支改动能解决的。线程保持打开,使该检查在合入时保持可见。