feat(review): measured failure attribution, round ledger, richer mutants, doc parity - #8218
Conversation
… A/B, per-hunk probes - qwen review test-plan: rule on the PR Test Plan's checkable claims (paths, npm scripts, test counts) against the reviewed tree; contradictions and differing counts are disclosed via compose-review, never capping. - qwen review base-tree: build the merge base in a sibling worktree so the verifier can A/B a comparative claim instead of reading it; swept by cleanup. - test-efficacy: third probe kind — reverse-apply one hunk at a time and re-run the affected tests, attributing a still-green suite to the specific change nothing gates; shares the mutants' budget window, runs last.
… summaries Both measured on a live /review of #8176 with the built CLI: - test-plan's observedTestCounts strips SGR sequences before matching; a color-enabled pipe interleaves them BETWEEN tokens, and the count claim fell to 'unchecked' with the summary right there in the report. - build-test's trimOutput rescues runner summary lines from the omitted middle (like module-resolution errors): a failing suite's tail is all failure details and npm epilogue, which pushed the one-line summary out of the kept text entirely.
|
Thanks for the PR! Re-running the gate now that #8215 has landed and this stack has been retargeted to
Moving on to code review. 🔍 中文说明感谢贡献！#8215 已合并、本栈已 retarget 到
进入代码审查。🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code reviewRead independently first: for "make test-failure attribution a measurement," I'd add a subcommand that takes the build-test report plus a built base tree, reruns only the failed commands on base, parses failing files from each side, and diffs the sets (not counts — a flaky suite fails different names across two runs of the same tree). That is exactly what No Critical blockers. Notes, all non-blocking:
Files changed (13 of 13)
Test evidenceThis is an unattended CI run, so nothing here was built or executed by the reviewer — but unlike the previous pass, the PR's own CI now runs on the reviewed commit, because #8215 landed and the stack was retargeted to
The green Linux suite includes the new Sandboxed verification would settle the remaining gap: 中文说明代码审查先独立设想：要把"测试失败归因"变成测量，我会加一个子命令——接收 build-test 报告加一棵已构建的 base 树，只在 base 上重跑失败的命令，从两侧解析失败文件，再对集合做差（而非计数——flaky 套件在同一棵树的两次运行中会失败于不同的测试名）。这正是 无 Critical 阻断项。以下均为非阻断观察：
测试证据这是无人值守的 CI 运行，因此评审者此处未构建、未执行任何东西——但与上一轮不同，本 PR 自己的 CI 现在会在被审 commit 上运行，因为 #8215 已合并、本栈已 retarget 到 绿色的 Linux 套件包含新增的 沙箱验证可以定案剩余缺口： — Qwen Code · qwen3.8-max-preview Reviewed at |
|
Confidence: 4/5 — clean review with no blockers and green CI on the reviewed commit; the only reservations are non-blocking (cross-platform CI is matrix-skipped, the round ledger is prose-driven, and the integrated Stepping back: this is genuinely good work, and the one reservation I had last time is gone. My independent proposal for measured attribution was exactly the file-set-diff subcommand the PR ships, and I didn't find a simpler path it missed. The design is consistently conservative — every selector is biased toward silence (no merge base, unparsed output, timed-out or un-built base, unrecognised command, exhausted budget → fall back and say so, with its own reason), so the cost of the new machinery is missed candidates, never wrong findings. The two notes from the prior pass are both addressed in the current head: the asymmetric What changed since the defer: #8215 landed, the stack was retargeted to Non-blocking, for the maintainer's awareness: (1) macOS/Windows/integration are matrix-skipped on this trigger, so the suite only ran on Linux — the Windows path parse is unit-tested, but a maintainer may want the full matrix before merge; (2) the findings ledger is prose-driven (cache schema + SKILL steps) with no subcommand validating ledger shape yet, as the author discloses; (3) a Approving, pinned to the reviewed commit. ✅ 中文说明置信度：4/5 —— 各阶段评审干净、无阻断项，且被审 commit 上 CI 全绿；唯一的保留是非阻断性的（跨平台 CI 被矩阵跳过、轮次台账由 prose 驱动、集成的 退一步看：这确实是高质量的工作，而我上一次的保留已经消失。我对"实测归因"的独立设想，恰好就是 PR 交付的按文件集做差的子命令，也没找到它漏掉的更简路径。设计一以贯之地保守——每个选择器都偏向沉默（无 merge base、输出不可解析、base 超时或未构建、命令不可识别、预算耗尽 → 回退并说明，且各有其因），所以新机制的代价是漏掉候选，绝不是错误的 finding。上一轮的两点观察在当前 head 中均已处理：非对称 自上次 defer 以来的变化：#8215 已合并，本栈已 retarget 到 非阻断，供维护者知悉：(1) 本次触发下 macOS/Windows/integration 被矩阵跳过，套件只在 Linux 上跑过——Windows 路径解析有单测，但维护者或许希望合并前跑完整矩阵；(2) findings 台账由 prose 驱动（缓存 schema + SKILL 步骤），尚无子命令校验台账形状（作者已披露）；(3) 跑一次 批准，钉在被审 commit 上。✅ — Qwen Code · qwen3.8-max-preview Reviewed at |
All measured in the review (#8215 review comment): - test-plan: linear-time bold-heading scan (the old pattern backtracked catastrophically on an unclosed ** line an untrusted PR body controls); a flag preceding the npm script yields no claim instead of a false 'no package defines this script'. - test-efficacy: a hunk probe's restore recreates the parent directory a reverse-applied 'new file' hunk removed (the ENOENT from finally lost the verdict and marked every remaining hunk inconclusive); hunks get their own skippedForBaseline instead of mislabeling a red baseline as a budget skip; splitDiffIntoHunks re-captures the file header at every diff --git boundary; a hunk-survived finding notes when it restates an inert file-level revert at hunk granularity. - base-tree: idempotent fast path keyed on a build marker + HEAD check — concurrent verifier shards reuse one built tree instead of sweeping it out from under each other mid-A/B (a fabricated base-side difference with a deterministic source tag was the worst case); cost wording is now 'an install and a build' everywhere it was 'one extra build'.
aa59e92 to
38664ab
Compare
|
Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration. 中文请勿对活跃的 PR 执行 rebase 或 force-push，因为这会使已有的评审评论失效。另外，供日后参考：作为集成流程的一部分，机器人始终会自动将所有改动压缩（squash）为单个提交。 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
中文说明
已审查——无阻断问题。 建议见行内评论。
— qwen3.8-max-preview via Qwen Code /review
…line (#8215) A per-hunk probe reported `survived` whenever the green baseline probes still passed with the hunk reverted. When the hunk's own collocated test dropped out of the baseline (a probe-tree import error collects nothing), the remaining green probes prove only that THEY do not cover the hunk, so the verdict is now `inconclusive` — the same dropped-test asymmetry the mutants already hold. Also scope the hunk-survived cross-reference note to the hunk's own collocated test, and let test-plan match a workspace-scoped run of the plan's bare command instead of falling through to the manifest on an exact-string miss.
…his branch The two blocking findings, reproduced on this PR's own Test Plan: - test-plan files no false contradicted notes: npm rulings move from a four-verb denylist to an allowlist (the run form + npm's script aliases — the ~fifty other builtins each used to become 'no package defines this script'); a slash token is claimed as a repo path only with evidence (an extension or ./ prefix), never when it is a flag's value (--repo owner/repo) or under the review's own temp root; HEADING_LINE_RE drops the same quadratic shape its bold sibling was rewritten to remove. - base-tree gets a real mutual-exclusion lock around sweep+add+build (mkdirSync test-and-set; the loser returns busy instead of deleting the tree the winner is mid-install in), and a failed build writes a settled marker so later shards stop re-paying the install to relearn 'unavailable'. Also: Agent 7's brief now names hunk-survived and the hunks.* counters (it is the report's only consumer, and the finding class was invisible); hunk findings anchor at the first ADDED line instead of up to three context lines above the change.
… round ledger, richer mutants, doc parity Shaped by a live round-2 maintainer re-verification (#7818): - qwen review test-delta: rerun the PR side's failed test commands on the built merge base and diff the failing FILE SETS — netNew is the PR's own failure by measurement (whatever files the diff touches), shared is pre-existing by measurement. Counts are never compared: a flaky suite fails different test names between runs of the same tree. An unfinished base rerun attributes nothing. - Round ledger: the incremental review cache persists confirmed findings under round-scoped ids (R1-2); a high-effort re-review rules on each (fixed / still stands / cannot tell) and opens its report with the table, the way a human round-2 comment opens with 'M1 is fixed'. - Three replacement mutation operators in test-efficacy: drop '?? fallback', force a comparison-bearing guard condition to 'true', drop a '+ CONST' term. Each survivor maps to one crisp untested-protection sentence. A line whose raw text and literal-blanked code view disagree yields no candidate — an edit index computed across the two views once spliced 'iftrue 0)' into a guard, and a mangled mutant reads as inconclusive while quietly spending a cap slot. - Quality brief: documentation-parity lens — a new user-facing surface whose siblings are documented is a Suggestion naming the sibling precedent; no documented sibling, no finding.
38664ab to
e896669
Compare
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
中文说明
已审查——无阻断问题。 建议见行内评论。
— qwen3.8-max-preview via Qwen Code /review
…ry diff guard, error convention (#8215)
…est-plan false positives, hunk-probe ranges (#8215) - base-tree: only stamp a base tree available when runBuildTest actually compiled something (ok AND npm toolchain AND a non-empty build). An `unsupported` handoff or an empty npm scope returns ok:true having built nothing; marking that tree available let an A/B read the absence of a build as a behavioural difference. - cleanup: sweep the stale base-tree build lock a killed builder leaves behind. - test-plan: read the root manifest's scripts directly so a root-only script survives when the root defines no build/test; bail on the inline --root=./dir rebasing form; stop treating a positional after an inline --flag=value as the flag's value; prefer a failed scoped run when ruling a bare command; anchor the npm script alias to a full token so `yarn test:unit` is not truncated. - test-efficacy: exclude `\ No newline at end of file` from the startLine offset count; compute the mutant-overlap range from the header's new-side span so it no longer overshoots into a closely following hunk.
Review —
|
| candidates | commits producing ≥1 | |
|---|---|---|
| deletion (pre-PR pool) | 9 | 6 / 40 |
| replacement (new) | 215 | 17 / 40 |
Per-commit replacement counts include 45, 39, 33, 28, 18 — 11 of the 17 overflow MAX_MUTANTS = 8, several by 20-38. Before this PR the cap was essentially unreachable (max 3 in one commit). guard-true is the driver: any single-line if with a comparison qualifies, which is most guards in most diffs.
The cap keeps selection at 8, so the report stays bounded. The cost lands on the time budget instead. Mutants and hunk probes share one window (startedAt + 540s − 300s ≈ 240 s minus the baseline run), each experiment costs a full runProbeSuite, and mutants drain it first (test-efficacy.ts:1659-1704). At a ~30 s probe suite that is ~4 runs total — so on any diff with ≥4 replacement candidates (12 of 40 commits sampled), hunkCandidates gets zero and reports skippedForBudget. Previously ~34 of 40 commits produced no mutants at all and hunk probes had the whole window.
That means hunk-survived — the finding class the stacked base #8215 adds, MAX_HUNK_PROBES = 6 — stops firing on most PRs, and nothing in the diff says so. The comment that authorises the ordering is now load-bearing on a premise the PR itself contradicts:
test-efficacy.ts:1671-1676— "a safety-verb mutant is the higher-precision experiment, so it gets the budget first"
Replacement mutants are explicitly not that: the PR queues them behind deletions in the cap precisely because deletions "have the track record". Yet in the time budget they outrank hunk probes anyway. Two ways out, either fine: give replacements their own sub-cap (2-3 slots) so deletions and hunk probes keep theirs, or move replacements after hunk probes in the time loop to match the stated precision ordering. Whichever — the ordering comment needs to be rewritten to say what it now means.
🟠 Important — test-delta has no whole-command budget
--timeout is per command, default 300 s, and runTestDelta loops over every failed command with no aggregate deadline. build-test's own --timeout help says it is "Kept strictly below the 600s (600000ms)" tool ceiling, and test-efficacy reserves an explicit TOTAL_BUDGET_MS = 540_000. Three failed test commands — routine on a monorepo where the widening loop scopes several workspaces — is 900 s worst case, killed by the tool ceiling with no report written at all, discarding the base-tree install+build that was just paid for.
test-efficacy's pattern applies directly: one deadlineAt, priced per entry, remaining entries disclosed rather than attempted.
🟠 Important — run() drops three deliberate properties of build-test's run()
test-delta.ts:386-404 reimplements build-test.ts:213-239 and diverges on all three points build-test's comments call out as intentional:
- No
trimOutput.CommandResult.outputis documented "Trimmed output" (build-test.ts:77). test-delta stores the raw base output — up to the 64 MBmaxBuffer— intoentries[].base.output, which is thenJSON.stringify(…, null, 2)'d into--outand printed whole to stdout. A failing monoreponpm testis easily hundreds of KB; Agent 7 reads that file.trimOutput(…)is already imported-adjacent and keeps exactly the tail the parser needs. - Timeout detection is
r.error?.message?.includes('ETIMEDOUT')instead of build-test'serror.code === 'ETIMEDOUT' || (r.signal === 'SIGTERM' && r.status === null). I confirmed the message form does work on the default path (msg = "spawnSync /bin/sh ETIMEDOUT"), so this is not live today — but build-test callserror.code"the authoritative signal" and added the SIGTERM fallback on purpose, and one case it catches that the substring does not is amaxBufferoverflow kill (ENOBUFS → SIGTERM,status: null). A missed timeout here is not cosmetic: it is exactly thebaseUnusable = falsepath from the Critical above. - No
stdio: ['ignore', 'pipe', 'pipe']—build-test.ts:222, "A build that asks a question is a build that hangs until the deadline." test-delta reruns those same commands without the guard.
Simplest resolution: export build-test's run and call it, rather than keeping a second copy that will keep drifting.
🔵 Suggestions
The overlap test is vacuous. test-efficacy.test.ts — "emits one candidate per line — a safety-verb line is not also mutated by replacement" — uses cache.delete(key);, which yields no replacement candidate under any implementation (replacementMutantsOf('cache.delete(key);', …) === null, confirmed). I deleted the continue at test-efficacy.ts:681 and the test still passed. Its own comment names the right input: map.delete(k) ?? fallback. Using cache.delete(key) ?? fallback; does exercise it — I checked, it selects the deletion candidate with operator: undefined, which is the claim.
The literal-blanking skip removes the most common ?? shape. raw.trim() !== codeLine → null is sound, and the iftrue 0) story justifies it. But codeLines blanks string contents without preserving length (scanFileLines, buf += ch + ch), so every line carrying a string literal is skipped — including const name = cfg.name ?? 'unknown';, which produced no candidate in my run. The headline ?? config.getModel() example is the literal-free minority. A recovery that keeps the index safety fully: detect on codeLine, then re-run the same regex against raw.trim() and splice at that match's index — no index ever crosses views, and the string-fallback case comes back.
Equivalent mutants will file unactionable Suggestions. guard-true on a guard that is redundant with the body survives by construction: if (list.length > 0) { for (const x of list) use(x); } → if (true) { … } is semantically identical, as is if (m.get(k) !== undefined) return m.get(k); (both verified to be selected). The survivor message then says "no test pins when the guard must NOT fire", which the author cannot act on. Worth one hedging clause in the guard-true message, or a filter for conditions whose body is itself iteration/optional-chained.
Ledger id continuity is under-specified. Step 1 says to read the cache's findings, but not its round — and round is what the R<round>-<n> ids are built from. A round-2 run that does not read it emits R1-* ids again and collides with the carried-forward R1-* entries it is supposed to be ruling on, which is the whole continuity mechanism. Step 8 says "write every confirmed finding of this round under a fresh R<round>-<n> id" while Step 6 says a still-standing finding is re-reported "under its original id" — reconcilable, but only if you read both. Making Step 1 read round and Step 8 say carried-forward entries keep their original id closes it. Separately: the ledger has no bound. fixed entries leave, still stands entries never do, and Step 6 requires ruling on every entry — a PR at round 6 with a disputed Suggestion pays for it every round.
failingFilesOf minor gaps (all fail safe to unparsed → path rule, so noting only): [\w@./-] excludes \, so Windows-style paths never match — relevant given the 🪟 *.test.* / *.spec.* filenames are recognised, so a __tests__/foo.ts convention yields nothing. term-drop on x = + CONST_ABC; produces x =; (a broken mutant → inconclusive → a spent cap slot) — contrived, but the same class the raw/code-view guard exists to prevent.
What's good
- The
timedOut → attributes nothingbranch, and the test comment that records it shipped wrong first because the test asserted only the note text. That is the right instinct and the right disclosure; the Critical above is the same bug one condition over. - One candidate per line, deletions first, every skip counted — the cap discipline is consistent with the rest of the file.
replacementMutantsOf's balanced-paren scan handles} else if, nested calls, and&&chains correctly (if (i < arr.length && arr[i] !== x) break;→if (true) break;); the??simple-fallback lookahead correctly refusesx ?? y + zand correctly takes the last link ofa ?? b ?? c. I probed ~20 shapes and found no unsound edit outside thex = + CONSTcuriosity above.- The doc-parity lens is well-bounded — "no documented sibling, no finding" is the difference between a convention check and a reviewer's taste, and it says so.
Verdict
Request changes, on the Critical alone — the netNew promotion off an unusable base run is a false-blocker generator wired straight into a verification-exempt path, and the guard is three tokens. The hunk-probe starvation is the one I'd want settled before merge too, since it silently degrades the feature shipping directly beneath this one. Everything else is comfortably follow-up material.
- test-delta (Critical, #8218 review): a base rerun that fails with ZERO parseable failing files (install/toolchain failure) no longer reads as 'base green' — it attributes nothing, like a timeout, instead of promoting every PR-side failure to net-new. - extract-step (Critical, #8225 review): every line of a multi-line env value is comment-prefixed — an unprefixed block-scalar second line sat in the emitted script as an executable line. - extract-step: expressionsOf no longer misses expressions containing their own brace (format('{0}')). - cleanup (#8215 review): sweeps an orphaned base-tree build lock, which otherwise wedges the A/B for that PR permanently.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (1)
packages/cli/src/commands/review/test-delta.ts:98
relativeToRoot()strips the worktree/root prefix using a case-sensitivestartsWith. On Windows (case-insensitive paths), a drive-letter or path segment case mismatch between--pr-worktree/--baselineand the runner’s reported path can prevent stripping, leaving absolute paths in the failing-file identity and causing pre-existing failures to be misclassified asnetNew. Consider doing the prefix comparison case-insensitively on win32 while preserving the original-cased relative path in the return value.
const norm = (v: string) => v.replace(/\\/g, '/').replace(/\/+$/, '');
const f = norm(file);
const r = root ? norm(root) : '';
const rel = r && f.startsWith(`${r}/`) ? f.slice(r.length + 1) : f;
return rel.replace(/^\.\//, '');
`trimOutput` rescues module errors and runner summaries out of the omitted middle, not the per-file FAIL lines this command reads. A base suite whose failure section overruns the tail budget therefore lost failing files into the gap - and a SHORT base set is the dangerous direction, because netNew is the PR side minus the base side. Every file the trim hid came back as a Critical attributed to this PR by "measurement". Parse the raw text, report the bounded one. The PR side cannot be fixed here: it is read out of build-test's stored output, trimmed before this command existed. That loss runs the other way - it understates `shared`, never invents a netNew - so the entry carries `prTruncated` and the note says the list may be partial. A silence-biased gap is still a gap the author should hear about. Also names both selection caps in the mutant-skip diagnostic. The count accumulates replacement sub-cap drops, and with 2 deletions and 6 replacements the total is exactly MAX_MUTANTS: the main cap never fires, yet 3 are dropped, and the message sent the reader looking for a pool of 11 candidates that never existed.
This command reads a report off disk and then executes the strings in it with `shell: true`, in the base worktree. Nothing else in the pipeline re-executes a value it read back from a file, so nothing else has to care where that value came from - this does, and the provenance is worse than it looks: the command is `npm test --workspace="<dir>"`, the workspace token is a directory, and a directory is a name a pull request chooses. `packages/x";curl …|sh;"` is a legal path in git and on Linux and it round-trips through the report into a shell. Restricting to the emitter's own grammar costs nothing real, since that grammar is the two shapes build-test produces. A command outside it is skipped and disclosed, the same treatment everything else this command cannot do already gets, so a future shape degrades to "judge it by the diff" rather than to arbitrary execution.
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
中文说明
— qwen3.8-max-preview via Qwen Code /review
|
@qwen-code /triage |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 280 passed · 0 failed · 280 total 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：280 通过 · 0 失败 · 280 总计 Verification reportPR #8218 Deep Verification —
|
| # | Capability | Where | Mechanically testable? |
|---|---|---|---|
| 1 | Measured failure attribution (test-delta) |
test-delta.ts (+453, new) |
Yes — central claim |
| 2 | Three replacement mutation operators | test-efficacy.ts (+228) |
Yes — secondary |
| 3 | Documentation-parity lens | agent-briefs.ts (Agent 4 brief prose) |
No — prompt text |
| 4 | Round ledger | SKILL.md (cache schema + Step 6/8 prose) |
No — model-followed prose |
Central claim: test-delta reruns the PR side's failed test commands on a built base tree and diffs the failing file sets — netNew (PR's own) vs shared (pre-existing) — with conservative silence wherever it cannot rule (unparsed, base timed out, base failed-without-naming-a-file, command outside the emitter grammar). Secondary: the three replacement operators are high-precision (conservative selectors), and the shell:true re-execution of a report-read command is injection-safe.
Central claim — A/B / wire-oracle evidence
test-delta is a new command, so the base arm has no equivalent code path to diff against; the load-bearing question is whether its classification logic is correct and pinned. Two instruments:
(a) Mock-free harness through the real spawnSync path (harness-delta.mjs, capture 01-test-delta-ab-real-spawn.png). The unit tests inject an exec seam, so the production run() — spawnSync(shell:true) → npm test → raw failingFilesOf parse → trimOutput → spawnTimedOut — is never exercised by them. This harness drives the compiled runTestDelta with no seam, building real minimal npm packages as the base tree so npm test actually runs and emits vitest-shaped output:
| Cell | Scenario (real npm test in base tree) |
Oracle | Result |
|---|---|---|---|
| A | same failure both sides, base exit 1 | shared=[flaky], netNew=[], base exitCode observed = 1 |
✅ |
| B | PR-only + shared failure | netNew=[new], shared=[flaky] |
✅ |
| C | base FAILED TO RUN (exit 1, Cannot find module, no FAIL lines) |
netNew=[] (not promoted), note "did not measure the base" |
✅ |
| D | base green (exit 0, no FAIL) | every PR failure → netNew |
✅ |
| E | FAIL line buried in the trimmed middle (>8 kB prose) | raw parse still sees it → shared; output trimmed |
✅ |
| F | base rerun times out (real SIGTERM/ETIMEDOUT, 8 s sleep / 2 s deadline) | base.timedOut=true, netNew=[] |
✅ |
| G | grammar guard — 10 metacharacter injection shapes + 2 legit + 1 ../ |
see Security below | ✅ |
| H | failingFilesOf parser edges (project token, root strip, ANSI, ❯, truncated token) |
7/7 | ✅ |
28/28 assertions pass. Cell C is the load-bearing safety property — a base that exits non-zero without naming a failing file (unbuilt tree, missing install, workspace the PR added) must NOT promote PR-side failures to netNew. Proven end-to-end through a real npm test exiting 1. Capture: 01-test-delta-ab-real-spawn.png.
(b) Mutation matrix (mutation-matrix.mjs, capture 02-mutation-matrix.png) — revert one central hunk in the source, run the PR's own test, confirm red, restore:
| Mutant | Hunk reverted | Killed by | Suite failures |
|---|---|---|---|
| M1 | drop baseUnusable's "failed without naming a file" clause |
"does NOT read a base that failed to RUN" | 1 |
| M2 | permissive grammar /^npm test/ |
"never hands a command outside the emitter grammar" | 1 |
| M3 | positive control — netNew ignores the base set |
"attributes a PR-only failure as netNew…" | 4 |
| M4 | drop replacementMutantsOf's raw.trim() !== codeLine guard |
"yields NOTHING when the raw line and the code view disagree" | 1 |
4/4 killed, M3 (positive control) confirms the harness detects red, and the named test appears among the failures each time. Source restored cleanly (git status --porcelain empty). The central tests are not vacuous and do not pass for the wrong reason. Capture: 02-mutation-matrix.png.
Security — the shell:true re-execution surface
test-delta reads a command out of a report file and re-executes it with shell: true, guarded only by RERUNNABLE_COMMAND_RE = /^npm test(?: --workspace="[\w@./-]+")?$/. The report's workspace token is a directory name a PR can choose, so this is the sharpest consequence. Cell G tested it with ground truth — each hostile shape carries a touch /tmp/td-pwnedN payload, so an executed injection leaves an artifact:
- GROUND TRUTH: no
/tmp/td-pwned*file was created — no injection ran. - All 10 metacharacter shapes refused (
;, backtick,$(), single-quote,&&,|,>, embedded space, newline, bare;). - Every command that DID run is structurally
npm test [--workspace="<metacharacter-free>"]— the invariant that makesshell:truesafe, since[\w@./-]contains no shell metacharacter. - Both legit emitter shapes (
npm test,npm test --workspace="packages/core") ran; the grammar matches whatbuild-testactually emits (build-test.ts:67).
Command injection is disproved, not merely unobserved. The grammar matches the real emitter exactly, so there is no false-negative under-attribution for legitimate commands either.
Secondary claim — replacement mutation operators
harness-selectors.mjs (capture 03-replacement-selectors.png) drives the compiled selectMutants end-to-end (the real literal-blanking scanner produces codeLine), 21/21:
- Positive:
coalesce(return a ?? defaultValue;→return a;),guard-true(if (a !== b) {→if (true) {),term-drop(const x = base + MAX_EXTRA;→const x = base;), and the brace-lesselse if (a !== 1)shape all fire and edit the line correctly. - Conservative gates (the PR Test Plan's exact claims + measured siblings): a string-bearing line
if (s === ")")→ no candidate (raw ≠ code view);if (ready)→ none (no comparison);if (isRecord<string>(v))→ none (the generic-call<stringtrap — the trailing-\srequirement excludes it);if (arr.map(x => x).length)→ none (the arrow=>is not read as a>comparison); a comment-bearing line → none. A real comparisonif (a > 5 && …)still fires. - Deletion priority:
state.clear();stays a deletion mutant (nooperator, nomutated). - Sub-cap:
REPLACEMENT_SUB_CAP=3— 6 guard-true candidates yield exactly 3 selected + 3 counted inskippedForCap, withinMAX_MUTANTS=8.
The new end-to-end integration test (runs a REPLACEMENT mutant end-to-end and reports the survivor) passes, exercising the real runOneMutant write→probe→classify path for the lines[line-1] = mutated branch.
Targeted gate (affected suites, unmutated)
| Suite | Tests | Result |
|---|---|---|
test-delta.test.ts |
24 | ✅ pass |
test-plan.test.ts |
73 | ✅ pass |
test-efficacy.test.ts |
104 | ✅ pass |
test-efficacy.integration.test.ts |
25 | ✅ pass |
| Total | 226 | 0 failures |
Scaling probe (untrusted input): test-plan parses the author-controlled PR body. A ReDoS ladder over extractTestPlanSection + extractClaims at 2 k / 5 k / 20 k / 65 536 chars (the GitHub body cap) across five hostile shapes (repeated Test Files labels, dense N passed tokens, the historical unclosed-** shape, many backticked spans, fenced # comments): worst single call 140 ms at the 65 k cap, the unclosed-** shape 0.1 ms (the historical backtracking is fixed). No catastrophic backtracking (harness-redos.mjs).
Findings
None blocking.
Observation (informational, by-design, not exploitable): grammar admits .. in the workspace token
RERUNNABLE_COMMAND_RE accepts npm test --workspace="packages/../etc/passwd" because . and / are intentionally in the character class. This is path-traversal-shaped over-acceptance, not command injection: the accepted string is still structurally npm test --workspace="<no-metacharacter>", so shell:true cannot be made to run a second command — confirmed by ground truth (no marker file created when the payload rode alongside such a token). Blast radius is nil: the command runs npm test against an odd directory inside a disposable base worktree in the sandbox, where npm simply fails to resolve the workspace (→ baseUnusable → no attribution). The PR's stated security claim is specifically about shell injection (packages/x";curl|sh;"), which holds; .. being in the class is documented design ([\w@./-]), not an oversight. No reviewer action required; recorded so the boundary is named precisely.
Not covered
- Round ledger (capability 4) and doc-parity lens (capability 3) are prose, not executable code — the ledger is a cache schema + Step 6/8 instructions in
SKILL.md, and doc-parity is a paragraph in Agent 4's brief (agent-briefs.ts). I verified both texts are present and wired (thetest-delta/base-treeinstructions in Agent 7's brief, thefindings[]ledger schema, the parity paragraph), but there is no code path to assert against; adherence is model-followed, exactly as the PR states ("no subcommand validates ledger shape yet", "a live incremental round-2 run has not been driven end to end"). - Real
base-tree↔test-deltaintegration was not driven end-to-end: the harness hand-built the base npm packages rather than invokingqwen review base-treeto build a real merge base. The classification logic and the realspawnSyncpath are proven; the upstream base-tree build is feat(review): Test Plan claim check, base-tree A/B harness, per-hunk probes #8215's surface (already in the base tip) and was not re-verified here. - Repo-wide gate not run: the PR Test Plan cites 43 files / 1185 tests for the whole
reviewdirectory; I ran the four affected files (226 tests). Unaffected review subcommands were out of scope. - Per-commit attribution is out of reach: the checkout is depth 2 (shallow), so only the merge commit, base tip, and PR head are local; the 20 commits in the metadata
commits[]array are not individually exercisable. Verified the aggregateHEAD^1..HEADdiff. - Windows/macOS behaviour of
test-delta(thefailingFilesOfWindows-path branch is unit-tested but not run on Windows); the lane is Linux.
Methodology
CI merge-ref checkout (HEAD=merge 6041f088, HEAD^1=base bd855991, HEAD^2=PR head e7b3f670), node:22-bookworm, Node v22.23.2, no zstd. npm ci + npm run build were pre-run; harnesses import the compiled packages/cli/dist/src/commands/review/*.js. Four harnesses (all in this dir, rerunnable): harness-delta.mjs drives runTestDelta with no exec seam — real minimal npm packages emit vitest-shaped FAIL output so the production spawnSync/raw-parse/trimOutput/spawnTimedOut path is under test; harness-selectors.mjs drives selectMutants/replacementMutantsOf; harness-redos.mjs times the untrusted-body parsers; mutation-matrix.mjs reverts central hunks in source, runs the PR's own vitest, and restores from the clean tree. Internal workspace link verified pointing into the head tree (readlink -f node_modules/@qwen-code/qwen-code-core → packages/core), so the head-side harnesses load head code. Raw per-harness logs are in logs/; evidence captures in evidence/.
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
Review SummaryPR #8218 — feat(review): four round-2 borrowings — measured failure attribution, round ledger, richer mutants, doc parity Stacked on #8215, 13 files changed, +1506 lines. CI all green. What it does
Tests
AssessmentHigh-quality PR. Silence-biased design throughout — every selector errs toward not producing findings. Security-conscious (shell injection guard). Type system used effectively. DESIGN.md rationale sections are excellent. No issues found. — Qwen Code /review (v0.5.0) |
ytahdn
left a comment
ytahdn
left a comment
There was a problem hiding this comment.
LGTM — well-designed, thoroughly tested, silence-biased throughout. No issues found.
#8218 landed as a squash, so its content now arrives on this branch from two histories at once and every shared file conflicts. Four of them, all resolved to this branch: the three `extract-step` registrations and its DESIGN sections are this PR's own contribution and main has no copy, and the ledger's `fixed` ruling keeps the sibling-entrance sentence that main's squashed copy predates. Checked the pair that a wrong resolution here would silently break: the registered subcommands and the list the error message names are still identical, extract-step included.
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 1/100 轮）。改动内容与我反驳保留之处如下： Review feedback addressed — PR #8218One change this round; the rest of the feedback was verified already resolved in Change made
Bot inline
|
…ings artifact The rebase onto #8218 left two id schemes for one finding: the incremental cache's cross-round ledger names findings R<round>-<n>, while the findings artifact accepted any unique id. Same defect, two names, and the outcome ledger and next round's report could no longer be joined. The artifact now uses the R-ids whenever the run writes the cache ledger. Conflict resolutions from the rebase itself: review.ts keeps both new subcommands (test-delta from main, findings from this branch); the documentation-parity check #8218 added to the old Agent 3 brief lands in 3c, the consistency slice that owns sibling-parity checks.
…om Claude /review (QwenLM#8315) * feat(review): borrow recall, a fix loop, and size-derived budget from Claude /review Three changes, from a comparison of this skill against Claude Code's `/code-review`. The orchestration half of that comparison went the other way — nothing there has the worktree isolation, the transcript-backed coverage gate, or the single computed verdict — so what is borrowed is what it does better: how much it surfaces, what a finding *is*, and what a small diff costs. Recall ------ The Exclusion Criteria are a filter on what KIND of thing is a finding. Read as a confidence bar — which is how an agent under "silence is better than noise" reads them — they license dropping anything half-believed, and that drop is invisible: no later stage sees a candidate that was never filed. Every stage after the finders removes wrong findings; none can add a missing one. Each finder brief now carries the counterweight explicitly, and the Step 4 verifier deliberately does not get it. Code quality was one agent holding six unrelated checks — the shape this skill already refuses for invariant agents, on measured evidence (PR QwenLM#6457: one agent with an eight-item checklist found 1 of 5 defects; the same model split three ways found all 5). Split into 3a reuse/duplication, 3b altitude/abstraction fit, 3c consistency/clarity. 12 -> 14 agents in 3A. Low was one undirected pass capped at 8, and its only alternative was a nine-subagent fan-out. It is now an angle rotation in one context — line-by-line, removed behaviour, language pitfalls, wrapper routing, reuse/dead code, sibling consistency, then a gap sweep — dedup-only, no re-judging, cap 10. Still zero subagents. --fix and findings as data -------------------------- `--fix` is `--comment` reflected and gated on the opposite target: `--comment` writes to a pull request, `--fix` writes to a working tree, so a PR review (whose tree is the ephemeral worktree Step 9 deletes) ignores it with a warning. An effective `--fix` floors the effort at medium — editing the user's files on an unverified finding is the same mistake as posting one. New `qwen review findings` canonicalizes the findings into a JSON artifact the terminal report, the saved report and the review JSON all read, instead of three transcriptions of one list. With `--outcomes` it merges the fixer's ledger and REFUSES one that does not account for every finding: a fixer that applies six of nine and reports six has not lied about any of them, it has silently shortened the list. Size-derived budget ------------------- New `plan.budget`, computed from srcDiffLines the way the topology gate is and recorded in the plan rather than passed as a flag, so every reader sees one number. Scopes the low tier's angle count and sweep, the Agent 8 ceiling (0 below 80 source lines — "one domain dominates" is a judgement, and one made about forty lines finds a dominant domain every time), and the verify shard width. It never scales a dimension away: that is the roster's answer and the roster reads effort. Not included: per-model prompt routing. Claude's table exists because it was measured per model family; shipping an invented mapping into this skill is the kind of change its own review would flag. Tests: 39 files, 1215 assertions. * docs(review): align counts and level descriptions left stale by the 3a/3b/3c split Round-1 self-review findings on this branch: - SKILL.md medium tier still named 'quality (Agent 3)'; the Step-1 low bullet and Step 3C heading hardcoded six angles though plan.budget scales them 3-6; and no fallback was stated for a plan written by an older CLI that carries no budget field (falls back to the flat pre-budget behaviour — more coverage, never less). - DESIGN.md still labelled the 12-agent roster '(current)', kept the ten-lens / crosses-twelve topology arithmetic beside the updated fourteen-agent copy, keyed the re-gating cost row to the 12-agent roster, and described low as one pass ≤8 and medium as unverified inline angles — both contradicting the SKILL.md this PR ships. The LLM-call-budget and Fork-Subagent sections were still summed for 12 agents. - findings.ts: validateFindings accepted outcome but dropped outcomeNote, so the canonical artifact did not round-trip — a skipped finding fed back through --input kept its outcome and lost its reason. Tests: 40 review files green, including two new round-trip cases. * docs(review): one id per finding across the cache ledger and the findings artifact The rebase onto QwenLM#8218 left two id schemes for one finding: the incremental cache's cross-round ledger names findings R<round>-<n>, while the findings artifact accepted any unique id. Same defect, two names, and the outcome ledger and next round's report could no longer be joined. The artifact now uses the R-ids whenever the run writes the cache ledger. Conflict resolutions from the rebase itself: review.ts keeps both new subcommands (test-delta from main, findings from this branch); the documentation-parity check QwenLM#8218 added to the old Agent 3 brief lands in 3c, the consistency slice that owns sibling-parity checks. * test(cli): add findings to the pinned review subcommand surface review.test.ts pins the exact subcommand list and sits one directory above the review/ glob the branch's local runs used, so the new findings subcommand never met it until CI. Ubuntu was the only matrix leg that ran. --------- Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com> Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
…rol, seven lenses (QwenLM#8261) * feat(review): borrowed-verification trio — test-plan check, base-tree A/B, per-hunk probes - qwen review test-plan: rule on the PR Test Plan's checkable claims (paths, npm scripts, test counts) against the reviewed tree; contradictions and differing counts are disclosed via compose-review, never capping. - qwen review base-tree: build the merge base in a sibling worktree so the verifier can A/B a comparative claim instead of reading it; swept by cleanup. - test-efficacy: third probe kind — reverse-apply one hunk at a time and re-run the affected tests, attributing a still-green suite to the specific change nothing gates; shares the mutants' budget window, runs last. * fix(review): survive real runner output — ANSI-laced and trimmed-away summaries Both measured on a live /review of QwenLM/qwen-code#8176 with the built CLI: - test-plan's observedTestCounts strips SGR sequences before matching; a color-enabled pipe interleaves them BETWEEN tokens, and the count claim fell to 'unchecked' with the summary right there in the report. - build-test's trimOutput rescues runner summary lines from the omitted middle (like module-resolution errors): a failing suite's tail is all failure details and npm epilogue, which pushed the one-line summary out of the kept text entirely. * fix(review): address the eight findings from live review of this branch All measured in the review (QwenLM/qwen-code#8215 review comment): - test-plan: linear-time bold-heading scan (the old pattern backtracked catastrophically on an unclosed ** line an untrusted PR body controls); a flag preceding the npm script yields no claim instead of a false 'no package defines this script'. - test-efficacy: a hunk probe's restore recreates the parent directory a reverse-applied 'new file' hunk removed (the ENOENT from finally lost the verdict and marked every remaining hunk inconclusive); hunks get their own skippedForBaseline instead of mislabeling a red baseline as a budget skip; splitDiffIntoHunks re-captures the file header at every diff --git boundary; a hunk-survived finding notes when it restates an inert file-level revert at hunk granularity. - base-tree: idempotent fast path keyed on a build marker + HEAD check — concurrent verifier shards reuse one built tree instead of sweeping it out from under each other mid-A/B (a fabricated base-side difference with a deterministic source tag was the worst case); cost wording is now 'an install and a build' everywhere it was 'one extra build'. * fix(cli): never score a hunk survived when its own test left the baseline (#8215) A per-hunk probe reported `survived` whenever the green baseline probes still passed with the hunk reverted. When the hunk's own collocated test dropped out of the baseline (a probe-tree import error collects nothing), the remaining green probes prove only that THEY do not cover the hunk, so the verdict is now `inconclusive` — the same dropped-test asymmetry the mutants already hold. Also scope the hunk-survived cross-reference note to the hunk's own collocated test, and let test-plan match a workspace-scoped run of the plan's bare command instead of falling through to the manifest on an exact-string miss. * fix(review): silence-bias hardening from four live review rounds of this branch The two blocking findings, reproduced on this PR's own Test Plan: - test-plan files no false contradicted notes: npm rulings move from a four-verb denylist to an allowlist (the run form + npm's script aliases — the ~fifty other builtins each used to become 'no package defines this script'); a slash token is claimed as a repo path only with evidence (an extension or ./ prefix), never when it is a flag's value (--repo owner/repo) or under the review's own temp root; HEADING_LINE_RE drops the same quadratic shape its bold sibling was rewritten to remove. - base-tree gets a real mutual-exclusion lock around sweep+add+build (mkdirSync test-and-set; the loser returns busy instead of deleting the tree the winner is mid-install in), and a failed build writes a settled marker so later shards stop re-paying the install to relearn 'unavailable'. Also: Agent 7's brief now names hunk-survived and the hunks.* counters (it is the report's only consumer, and the finding class was invisible); hunk findings anchor at the first ADDED line instead of up to three context lines above the change. * feat(review): four round-2 borrowings — measured failure attribution, round ledger, richer mutants, doc parity Shaped by a live round-2 maintainer re-verification (QwenLM/qwen-code#7818): - qwen review test-delta: rerun the PR side's failed test commands on the built merge base and diff the failing FILE SETS — netNew is the PR's own failure by measurement (whatever files the diff touches), shared is pre-existing by measurement. Counts are never compared: a flaky suite fails different test names between runs of the same tree. An unfinished base rerun attributes nothing. - Round ledger: the incremental review cache persists confirmed findings under round-scoped ids (R1-2); a high-effort re-review rules on each (fixed / still stands / cannot tell) and opens its report with the table, the way a human round-2 comment opens with 'M1 is fixed'. - Three replacement mutation operators in test-efficacy: drop '?? fallback', force a comparison-bearing guard condition to 'true', drop a '+ CONST' term. Each survivor maps to one crisp untested-protection sentence. A line whose raw text and literal-blanked code view disagree yields no candidate — an edit index computed across the two views once spliced 'iftrue 0)' into a guard, and a mangled mutant reads as inconclusive while quietly spending a cap slot. - Quality brief: documentation-parity lens — a new user-facing surface whose siblings are documented is a Suggestion naming the sibling precedent; no documented sibling, no finding. * feat(review): render adjudication, workflow step extraction, three verification lenses Shaped by a live sanitizer-PR verification (QwenLM/qwen-code#8147): - Render-adjudication capability (opt-in): with QWEN_REVIEW_SCRATCH_REPO set, the verifier may post a minimal payload to that user-designated repo and rule on GitHub's own rendered HTML — the measured case being an @ -> @ defusal every local reading called sound while GitHub's real renderer registered the mention and fired the notification. Absent the setting, rendering claims honestly cap at low confidence / cannot tell. Step 7's write ban names the carve-out explicitly. - qwen review extract-step: lift one workflow step's run: script verbatim into an executable, with env (as comments, never half-substituted exports), every ${{ }} site listed unevaluated, and a heuristic invoked- command list as the stubbing starting point. With base-tree, both arms of a by-hand workflow A/B become two invocations. yaml declared as a cli dependency (previously resolved only via hoisting). - Three brief lenses: a borrowed protection idiom missing what made it work at home (the code ancestor did the protecting; only the entity was copied); a second parser for an authoritatively-parsed format is a divergence hunt; tests that pin the mechanism instead of the effect, and oracles that mirror the implementation's own model. * feat(review): sibling-entrance discipline for the fixed verdict From round 6 of the live sanitizer verification (QwenLM/qwen-code#8147): the fix closed the fence-shaped entrance into a raw-HTML block, and the code-span entrance beside it — same divergence, adjacent syntax — stayed open. A re-check that tests only the reported input rules 'fixed' over a hole one backtick away. Both fixed-verdict sites in SKILL Step 6 (the open-Criticals re-check and the round ledger) now require enumerating a divergence-class defect's sibling entrances before ruling fixed; a still-open sibling is a NEW finding, never a reason to withhold the original's fixed — the two rulings stay separate so the second hole cannot ship unreviewed. * feat(review): three measured-verification lenses from live rounds 8037/8005 - Threshold-boundary scan: when a fix is a ratio/length guard, hold the issue's own variables fixed and binary-search the boundary where the behaviour flips; put the number next to what the issue reports. Live case: a prose-ratio guard covered the edit/write_file half of its issue and silently declined the run_shell_command half (~473-char boundary). - Delimiter self-injection named as the first parser-differential probe: a no-escaping extractor fed its own close tag truncates silently. - Shared-gate state enumeration: a deliberate-design defence extends only to the states it argues — an input-hold argued for 'active' silently froze three idle states sharing the same gate. The sibling-entrance rule, applied to a state machine instead of a syntax. * feat(review): carry the round ledger in the posted review body The ledger shipped as a local cache file and its first multi-round live use exposed the flaw: four model-comparison rounds re-reviewed the same two PRs and every round opened from scratch — medium never reads the cache, and from CI or another clone there is no cache at all. The one artifact every environment can see, the posted review, carried nothing machine-readable. - compose-review embeds an HTML-comment marker (invisible on GitHub) carrying this round's findings and round number; the round comes from the side file pr-context recovered, +1 — never from the model. - pr-context recovers the reviewing account's latest marker, renders it as a 'Previous /review round (machine ledger)' work-list section, and writes it beside the context file for compose-review's round counter. Own-account only, latest round wins, fail-quiet on any malformation — the marker is data the next round rules on, never authority it obeys. - Step 6's previous-round ruling now runs at medium too (recovery is free; medium still writes no cache and posts nothing); the local cache is demoted to fallback for rounds that never posted. * fix(review): address review feedback — false-positive hardening, binary diff guard, error convention (#8215) * fix(review): address review feedback — base-tree availability gate, test-plan false positives, hunk-probe ranges (#8215) - base-tree: only stamp a base tree available when runBuildTest actually compiled something (ok AND npm toolchain AND a non-empty build). An `unsupported` handoff or an empty npm scope returns ok:true having built nothing; marking that tree available let an A/B read the absence of a build as a behavioural difference. - cleanup: sweep the stale base-tree build lock a killed builder leaves behind. - test-plan: read the root manifest's scripts directly so a root-only script survives when the root defines no build/test; bail on the inline --root=./dir rebasing form; stop treating a positional after an inline --flag=value as the flag's value; prefer a failed scoped run when ruling a bare command; anchor the npm script alias to a full token so `yarn test:unit` is not truncated. - test-efficacy: exclude `\ No newline at end of file` from the startLine offset count; compute the mutant-overlap range from the header's new-side span so it no longer overshoots into a closely following hunk. * feat(review): mined disciplines — effective-diff guard, positive control, seven lenses From a full sweep of 108 maintainer verification comments (2026-07-31): - fetch-pr detects an EMPTY merge-base diff (work already landed — the skill stops and recommends close-as-superseded) and a diff collapsed to a residual by overlapping merged PRs (disclosed; body claims about the rest are description-of-history). - test-efficacy runs a POSITIVE CONTROL before spending the mutant window: an injected always-failing test must turn the runner red, or every would-be survivor is re-classed inconclusive — a dead runner cannot kill, and its survivors were the false gap-reports measured live. - Brief lenses: tested-fix + contamination statement; race verdicts as rates with load amplification (lower idle rate without structural change is luck); triangulate-and-label for unreachable authorities; arm-identity and comparator negative-control for A/B; deleted code's incidental effects; self-defeating fallbacks (trigger set within cannot-succeed set); reachability adjudication (fix vs scaffolding); flipped-assertion audit; shape-vs-provenance authorization probing. * fix(review): four fixes from the stack's own round-2 reviews - test-delta (Critical, #8218 review): a base rerun that fails with ZERO parseable failing files (install/toolchain failure) no longer reads as 'base green' — it attributes nothing, like a timeout, instead of promoting every PR-side failure to net-new. - extract-step (Critical, #8225 review): every line of a multi-line env value is comment-prefixed — an unprefixed block-scalar second line sat in the emitted script as an executable line. - extract-step: expressionsOf no longer misses expressions containing their own brace (format('{0}')). - cleanup (#8215 review): sweeps an orphaned base-tree build lock, which otherwise wedges the A/B for that PR permanently. * fix(review): close the remaining round-2 findings across the stack - selectMutants: replacement operators get a 3-slot sub-cap inside the shared cap — measured 24x pool inflation (215 vs 9 candidates over 40 real commits) was draining the time window hunk probes draw from last, silently un-shipping the hunk-survived finding class; sub-cap drops are counted in skippedForCap. - test-delta: whole-command budget (540s, mirroring test-efficacy) so three failed commands cannot blow the 600s tool ceiling and lose the report; run() regains build-test's three deliberate properties (stdin ignored, error.code/SIGTERM timeout detection, trimmed output). - test-plan: pasted unified diffs inside Evidence blocks shed no path claims (+++ b/<path> once ruled contradicted on a correct body); gitignored paths rule unchecked — absent at the reviewed commit by construction, the .qwen/ reasoning generalized. - agent-prompt: the Agent 7 efficacy block now enumerates all FOUR finding kinds including hunk-survived, the hunks.* skip counters, and harnessValidated — the brief and the block in the same prompt no longer present two different complete sets (the sibling-enumeration lesson, applied to our own prompt). * fix(review): address review feedback — diff-header false positives, stale prompt enumeration, added-file hunk probes (#8215) * fix(review): close the round-3 findings on the stack #8261 review: - the dead-harness re-class now runs BEFORE findings are built (a failed positive control could still file survivor findings — the Critical); it also covers the file-level revert probe's inert verdicts, the control pays for its run out of the shared budget (no more silently eaten skippedForBudget slot), a failed control stops the window instead of manufacturing survivors to re-class, and its note names all three causes that share the green-control shape. - emptyDiff no longer fires on a FAILED diff capture (close-as-superseded off an infrastructure error would close a live PR); countDiffChangedLines counts --x/++x body lines. #8215 review: - the cd target itself is no longer claimed as a path (a bare dir carries no evidence; only the file tokens it resolves are); - 'Test Files N passed' is a FILE count — variable-length lookbehind keeps it out of the test-count claims; - base-tree's handler gains the pipeline's catch-and-exit-1 convention; - probeCreateFailureDetail folded into the shared worktreeCreateFailureDetail. * fix(review): address review feedback — cd-base exclusion, Test Files count guard, base-tree error handling, probe delegation (#8215) * fix(review): port the collocated-dropout test to the post-#8050 runner seam Merging main brought #8050's Windows-portability refactor, which resolves the probe runner through vitest/package.json's bin — a node_modules/.bin fake is dead weight it never reads. The 8215-only collocated-dropout test still installed the old .bin fake, so the REAL vitest ran its fixtures, price.test.ts genuinely passed, and the hunk scored survived. The test now overrides the fake package's vitest.mjs like every post-refactor test. * fix(review): close the round-4 findings from both stack reviews From the #8215 review: trimOutput's summary rescue is capped at 40 lines (uncapped, 40k lines of 'Test <n>: …' prose made the whole trim a no-op — 1.6MB in, 1.6MB out); a spaceless # line is prose per the ATX rule GitHub applies (a pasted #!/bin/bash outside a fence once ended the Test Plan mid-body), with the bare-#-run crash on the closing scan guarded; a base-tree build lock older than 30 minutes is a corpse and is swept rather than reporting busy for the rest of the review. From the #8261 review: a pasted diff's BODY lines shed no path claims (-packages/old/gone.ts matched PATH_RE and ruled a false contradicted — the syntax-line filter alone was vacuous against the real failure mode); a gitignored file that nonetheless EXISTS rules reproduces (the ignore guard only ever downgrades a would-be contradiction); test-delta's note now explains each unattributed command (timeout vs toolchain failure) instead of leaving the reader to infer why it is in neither list. * fix(review): bound the summary rescue, apply the ATX heading rule, sweep stale build locks The three 8215-layer findings from the latest review, fixed at this layer (they were first patched further up the stack, where the reviewer of THIS PR cannot see them): - trimOutput's summary rescue is capped at 40 lines — uncapped, 40k lines of 'Test <n>: …' prose voided the trim entirely (measured 1.6MB in, 1.6MB out) and the bounded-output contract is the whole point. - A '#' with no following whitespace is prose, not a heading (the ATX rule GitHub applies): '#8176', '#tag', an unfenced '#!/bin/bash' no longer end the Test Plan section mid-body; the bare-#-run crash on the closing scan is guarded. - A base-tree build lock older than 30 minutes is a corpse left by a killed builder — swept and rebuilt instead of reporting busy for the rest of the review. * fix(review): EEXIST-only lock busy, bun test alias, chained cd bail, fence backreference Four live findings from the latest inline review round (the rest of the round was already fixed upstream by the takeover bot - verified by probing head behavior rather than re-reading the threads): - base-tree's lock catch distinguishes EEXIST (a concurrent builder, busy) from EPERM/EROFS/ENOSPC (this run's own failure, reported as such, not as a busy that will never clear). - "bun test" is bun's built-in runner, not a package-script alias: it runs whether or not any manifest defines test, so ruling it against the scripts table filed a false contradicted. - A chained cd matches the leading-cd shape but the single-hop resolver joined file tokens against the FIRST directory; it now bails like the exotic-cd case. - codeSpans' fence regex closes on its own marker via backreference; a tilde fence line inside a backtick block ended the span early and lines after it were lost to extraction. * fix(review): make the ledger marker actually reach the posted body Three Criticals from the #8255 review, all real: - The marker was appended in the CLI handler, AFTER composeReview() returned, so it only ever reached the composed JSON on disk. submit posts what the PURE function returns, so no posted review carried a marker, latestOwnLedger always returned null, and the whole feature was inert end to end. The append now lives inside composeReview, fed by a draftedComments seam that both CLI boundaries fill from the same array they count (and both strip from a model-written state JSON). Three end-to-end tests now assert on the body composeReview returns. - pr-context's side-file write preceded the mkdir that creates its directory and was unguarded: an ENOENT failed the whole command over a best-effort carry-forward. mkdir first, write guarded. - CANONICAL_LGTM_RE is anchored, so a trailing marker made every no-op round "worth showing" and prior rounds started rendering in full. The marker is now stripped before the filter and the count walk, not only before rendering. Plus the three thread findings: ledger cells are escaped before landing in the context table (a title with a pipe could forge rows), parseLedger takes the LAST marker and type-checks/normalises what it recovers, and the stray verdict doc comment is reattached to verdictLine. * fix(review): close the ten open findings on this PR - guard-true tested for a comparison anywhere after `if (`, including the then-body, so `if (ready) emit(a !== b);` admitted a mutant on the comparison-less condition the gate exists to exclude. It now tests the condition span only. - The `survived` detail said "when it changes" for legacy DELETION mutants too; it now matches the operator. - test-delta's `unparsed` required both sides to parse zero files, so a PR-side failure whose FAIL lines the trim scattered was silently dropped whenever the base rerun happened to parse. netNew/shared come from the PR side, so the PR side alone decides. - failingFilesOf now matches Windows path shapes (backslashes, C:) - a missed parse is an unattributed failure, not a loud error. - The replacement branch of runOneMutant (write-file -> run-probe -> classify) had no end-to-end test; one now drives a coalesce operator through the real handler and asserts the mutated line, the verdict, the operator-specific wording, and that the shared tree is untouched. - Two tests were vacuous with respect to what they promised: the baseline-dir test never asserted the cwd (its helper swallowed the argument - fixed at the helper), and the one-candidate-per-line test used an input that never triggered the replacement path, so the `continue` under test was not load-bearing. - Reattached the orphaned selectMutants JSDoc; reworded the SKILL line. * fix(review): the test helper's cwd parameter is required, not optional CI's `tsc --build` failed on test-delta.test.ts: the exec seam always passes a cwd, but the helper's signature marked it optional, so pushing it into a string[] was `string | undefined`. Missed locally because vitest runs through esbuild, which strips types without checking them - the suite was green while the build was red. The gate to run before pushing a type-level change is `npx tsc --build`, not the test suite. * fix(review): a base rerun that could not RUN attributes nothing Two Criticals from the latest review, both reachable on the brief's own happy path: - baseUnusable covered only timeouts. Every other way the base side can fail to run - an unbuilt base tree, a missing install, a workspace the PR ADDED (npm test --workspace cannot resolve on base), an ENOBUFS truncation - exits non-zero with zero FAIL lines, which this code read as a green base. Every PR-side failure then became netNew: the strongest evidence the command emits, manufactured from a base that never ran a test. It now attributes nothing and says why. - Timeout detection was the weaker substring form the sibling explicitly rejects; an external SIGTERM (container stop, cancelled job) set neither an ETIMEDOUT message nor an exit code and fed straight into the above. build-test now exports spawnTimedOut and test-delta asks the same question rather than re-deriving it. Also: the base output is trimmed (it precedes the verdict fields in the report the agent reads, so an untrimmed megabyte truncates exactly what the command produces); the guard-true gate no longer reads an arrow function's => as a comparison (every predicate guard was a candidate - the if (ready) noise the gate exists to exclude); the term-drop message no longer calls a string concatenation a reserve term; the unparsed note describes its own PR-side-only condition; and the ledger's Step 6/Step 8 now agree that a still-standing finding keeps its id. * fix(review): extract-step resolves all three env/defaults levels, and comments every env line Two silent-wrongness defects in a command whose whole value is fidelity. `env:`, `shell:` and `working-directory:` are three-level settings on GitHub — workflow, job, step, nearest wins — and only the step level appears in the step's own text. Reading step-level alone reproduced by machine the exact transcription error this command exists to remove: measured, a step under a job-level `NODE_ENV: production` and a workflow-level `GLOBAL_FLAG` extracted with `env: { LOCAL: '1' }` and `workingDirectory: undefined`, so the emitted script ran with both unset and nothing said so. Not a contrived shape: this repo carries workflow-level `env:` in 7 workflows, job-level `env:` in 10, and job-level `defaults.run` in qwen-triage.yml — the workflow the command's own test plan names. The three levels now merge with the runner's precedence, and `envSources` records which level each key came from, so an inherited value is visible rather than indistinguishable from the step's own. The env block was commented per ENTRY, not per LINE. A YAML block scalar (qwen-autofix.yml's `SETTINGS_JSON: |-`) reaches the header as a multi-line string, so its continuation lines landed in command position — and under the `set -e` the header itself emits, the extracted step died in its own preamble before its `run:` body ran. Every line is commented now. Tests pin the effect, not the mechanism: `executableLines()` asserts nothing but the `run:` body ever reaches command position, plus a `bash -n` parse check. Verified to flip — all five new assertions fail against the pre-fix implementation (`{ LOCAL: '1' }`, `undefined` working directory, three executable lines instead of one, and a real `bash -n` syntax error). * fix(review): compare failing files by a normalised, project-keyed identity Critical: the two sides run in DIFFERENT roots (the PR worktree and the base tree), and netNew/shared compared the parsed paths verbatim - so an absolute-path runner turned every pre-existing failure into a fabricated Critical, with the authority of a measurement behind it. Paths are now normalised against each run's own root (and backslashes to /, so a Windows path compares with its POSIX-printed twin), which is why test-delta gained --pr-worktree. The identity also keeps the vitest project token: dropping it collapsed same-named test files across workspaces, so a PR-caused failure in one package could read as pre-existing because another package has a file by the same name - the worse failure direction. Also from the same review, all of them reachable on the brief's own path: - The base rerun now inherits build-test's stdio: ['ignore','pipe','pipe'] ("a build that asks a question is a build that hangs until the deadline") and its trimOutput, which matters because entries[].base precedes the verdict fields in the report the agent reads. - The brief gates on base-tree's `available`, not just its `path`: a tree that was created but did not build populates path too, and measuring against it turns an infrastructure failure into Criticals. - A programmatic caller omitting `timeout` no longer sends NaN into spawnSync. - MutantCandidate is a discriminated union, so an operator without its replacement line - which would delete a line while reporting "with its ?? fallback dropped" - is unrepresentable. - The comparison class no longer requires a trailing space (if (a<b) is the same guard, just unformatted) and matches a brace-less else if. - DeltaEntry.unparsed's doc now describes the PR-side-only condition it actually implements. * fix(review): restore the whole-command budget, keep generics out of guard-true Round-1 findings from a fresh review of this PR: - test-delta had no aggregate deadline: --timeout is PER command and defaults to 300s, so three failed commands is 900s against Agent 7's 600s ceiling - killed with NO report at all, discarding the base-tree install and build just paid for. TOTAL_BUDGET_MS mirrors the one test-efficacy reserves; commands it cannot fit are disclosed. - guard-true matched generic calls: `if (isRecord<string>(v))` produced a mutant, and a type-guard predicate is exactly the `if (ready)` shape whose survivors the gate calls noise. The trailing \s is required, not an accidental asymmetry with [!=]== - telling `a<b` from `fn<T>(x)` needs a parser, and the gate is silence-biased by design. - --pr-worktree had no contract test, and its failure mode is the worst here: arriving undefined, root stripping silently stops and every pre-existing failure becomes a fabricated netNew. The new test feeds parseSync's output straight into runTestDelta and asserts an attribution only reachable when both roots were stripped (verified red against the snake_case field shape that shipped once already). - Merged the two consecutive doc comments on prWorktree. * fix(review): a budget-shortened deadline is not the same fact as a slow rerun Round-2 finding on the budget just restored: `Math.min(perCommandMs, remaining)` can hand a rerun far less than --timeout, and if it dies there the note said only "timed out - infrastructure, not evidence". True, but it sends the reader hunting a hang that is really an exhausted budget - and unlike a real timeout, a rerun with budget to spare would still measure it. The note now names those commands separately and says so. Verified red against removing the tracking line. * fix(review): brace-tolerant stub list, pipefail fidelity, and extract-step in the briefs Round-3 findings on this PR, fixed. `expressionsOf` matched `[^}]*`, so any expression containing a brace — `format('refs/pull/{0}/head', …)`, `fromJSON('{"a":1}')` — was not mis-listed but DROPPED. A stub list reads as "these are all the values to supply", so a silent omission is a value that never gets stubbed. It now scans forward to the closing `}}`, and reports nothing for an unterminated site rather than swallowing the rest of the text. Declaring `shell: bash` is not the runner's default `bash`. The default is `bash -e {0}`; a declared `bash` (at any level) is `bash --noprofile --norc -eo pipefail {0}`, and a pipeline whose middle stage fails aborts under one and not the other. The header now carries `set -eo pipefail` or `set -e` accordingly — 163 of this repo's 434 `run:` steps are under a declared bash and were getting the weaker one. A `shell:` value is also a command template (`perl {0}`), so only its first word goes in the shebang and the whole template is recorded beside it. `extract-step` was registered on the CLI and mentioned in DESIGN.md, and nowhere in SKILL.md or the agent briefs — the runtime prompts. The capability was unreachable by the agents it was written for. The verifier's brief now carries it next to the A/B paragraph it composes with, and Step 4 summarises it. Also: env ordered nearest-first (measured on qwen-autofix.yml:route:0, merge order put 20 inherited entries ahead of the step's own 26 in a 49-line header); a valueless `FOO:` renders as the empty string, not `"null"`, and a non-scalar as JSON rather than `[object Object]`; a missing file no longer reports as a parse failure; DESIGN.md's lens count matches its list. The test oracle is rebuilt around the property instead of a filter: the file is the header plus the body verbatim, and every line before the body is a comment or a directive the test names. The old helper dropped `set -e` unconditionally, so it could not tell the header's from one the body legitimately contains — and would have gone green on a header that leaked exactly that line. 434 real `run:` steps swept: 0 non-verbatim bodies, 0 live header lines, 0 missed expression sites, 0 out-of-order env, 0 `bash -n` failures. * fix(review): restore the replacement sub-cap, stranded on a downstream branch Round-3 finding, and the third instance of one class: an 8218-layer fix committed on the 8261 branch, four PRs above the code it belongs to. Measured over 40 real commits, the replacement operators produce ~24x the deletion pool (215 vs 9 candidates; guard-true drives it). Every mutant run drains the same window hunk probes draw from LAST, so uncapped, most diffs with any replacement candidates leave hunk probing zero runs - the hunk-survived finding class silently stops firing and nothing says so. Three slots, and what the sub-cap drops is counted in skippedForCap rather than lost. Also swept the other direction: diffed every review file against its 8261 copy to confirm nothing else 8218-layer is stranded up there. The remaining divergence is 8261's own (the positive control, its lenses). * fix(review): one severity predicate, carried ids, and a lossless marker Round-1 review of the ledger marker, six findings. Two contradicted contracts this package already states elsewhere: - `buildLedger` restated the severity predicate as a bare `startsWith` while `severityOf` — what `countInlineFindings` and the unmarked-comment gate both use — trims first. A Critical whose body opened with a newline was counted, was posted, blocked the merge, and was silently absent from the ledger, shifting the id of every finding after it. It classifies through `severityOf` now, and strips the marker with the shared prefixes instead of a second copy of the literals. - Ids were assigned by array position every round, so the report saying `R1-2 still stands` shipped beside a marker recording that claim as `R3-4` — and `LedgerFinding.id` documented itself "stable across re-reports". Step 6 now mandates writing the original id into the re-report (`**[Critical]** R1-2: …`), `buildLedger` reads it back, and a duplicate or absent id falls back to the next free id of this round. Four smaller ones: - `stripLedgerMarker` removed only the FIRST marker while `parseLedger` reads the LAST, so a body holding two leaked the trusted marker into model-facing prose and un-filtered the canonical LGTM it was appended to. - The `--` guard rewrote data to an em dash: a finding about `--comment` reached the next round as `—comment`. It escapes at the JSON layer now, which is comment-safe and lossless. - `file` was capped on read (200) but not on write; the cap is one constant binding both halves. - `submit`'s path — the one GitHub's payload is built from — had no test that the marker reaches the posted body. Plus: tie-break `latestOwnLedger` on review id when `submitted_at` ties, neutralise a backtick in the location code span, skip the `currentUser()` round-trip when the PR has no reviews, and say why a stale side file is deliberately left in place. * docs(review): complete the "delta cannot rule" enumeration in both places Round-4 finding. The brief and SKILL.md each listed three cases where test-delta attributes nothing - unparsed, timed-out base, no merge base - but the code has five: the later rounds of this PR added "a base rerun that failed without naming any failing file" (it did not measure the base) and "a command the whole-command budget could not fit". Two enumerations of the same set with different membership, in the two places an agent reads. That is the sibling-enumeration lesson this skill teaches, applied to its own prose for the second time: the fix is not just adding the missing members but saying that the report names each case with its own reason rather than folding them into one. * fix(review): the invokes list was mostly prose, not commands Round-4 finding on this PR. `invokes` is documented as a heuristic starting point, and imprecision is fine — but measured over this repo's 434 real `run:` steps it was reporting 435 distinct "commands", 267 of them appearing exactly once, with a worst case of 63 entries made up of words like `CI`, `Evidence`, `PR` and `and`. A list that size, mostly prose, is not a starting point. Three causes, each measured: - A `${{ … }}` expression is not shell, and it routinely contains `||`. Splitting on that as a pipeline separator reported both operands as commands (`matrix.arch`, `github.event.inputs.version`). Expressions are now masked to an opaque token before the split; one sitting in command position contributes nothing, which is honest — what it expands to is unknown here by design. - A heredoc body is input to a command, not a list of them. Its lines were scanned as commands, terminator included. 12 steps in this repo carry one. - The `name=value` skip stepped over the prefix and took the NEXT word as the command — but for a quoted value with spaces that word is inside the value: `EVIDENCE_SECTION=$'### Evidence images'` reported `Evidence`. Quoted spans are now blanked out, with the quote carried across lines so a multi-line string's continuation lines are data too. Command substitutions are read first, so `body="$(sanitize < "$REPORT")"` still reports `sanitize`. A `#` preceded by whitespace ends the live part of a line, so an apostrophe in a trailing comment cannot open a span and eat the rest of the script. Measured after: 435 distinct commands to 187, singletons 267 to 104, worst case 63 to 27 — and the worst case is now real commands (`awk cat chmod curl git jq mktemp pkill tar tee timeout`) plus the script's own shell functions. Also added the builtins a stub could not intercept anyway (`eval`, `exec`, `source`, `unset`, `command`, …) to the keyword set. 434 steps re-swept: 0 non-verbatim bodies, 0 live header lines, 0 missed expression sites, 0 out-of-order env, 0 `bash -n` failures. * docs(review): the rationale named only one of the ways base goes unmeasured Third and last copy of the enumeration the previous commit fixed. The bullet's headline already generalised - "base attributes nothing it did not finish" - but its body named only the timeout, so a reader learning the contract from the rationale would conclude timeouts are the only unusable case. Name the set, and say why the report keeps the reasons apart: "we could not measure" and "we measured nothing" are different facts to the author, and only one of them is about their PR. * fix(review): a quoted `<<EOF` is not a heredoc, and a continued line is one command Round-5 findings, both in the previous commit's own scanner. A heredoc opener was matched over the whole line, so one inside a string started heredoc mode: `echo "write <<EOF for a heredoc"` made every later line wait for a terminator that never arrives. The failure is not a missing entry but a missing REST — measured, a three-line script returned `[]` instead of `[curl, jq]`, empty and entirely plausible. Opener detection moved inside the quote walk, where it only fires outside quotes; the quoted forms (`<<'EOF'`) are consumed by the match, so their quotes never open a span either. A backslash-continued command was scanned as several lines, which puts the next ARGUMENT in command position — this is where `apt-get install -y \` / ` libx11-dev` reported the package as an invoked command. Continuations are now joined into one logical line before scanning. Measured after: 187 distinct commands to 185, and the singleton tail is now dominated by real ones — PowerShell cmdlets, macOS tooling, and the scripts' own shell functions. 434 real steps re-swept: 0 non-verbatim bodies, 0 live header lines, 0 missed expression sites, 0 out-of-order env, 0 `bash -n` failures. * fix(review): case labels, a second heredoc, and an expression in command position Round-6 findings, from running the scanner against adversarial shapes rather than reading it again. Two are UNDER-reports, which is the worse direction: a command missing from the list is a stub the verifier never writes, so the extraction reaches the real network. - A `case` pattern label stopped the scan on its own line: `blocked) gh api x` reported nothing, losing `gh`. The label is now stepped over like a `name=` prefix. - Only the first heredoc opener on a line was tracked, so `cat <<A <<B` left the second body and its terminator read as commands (`B`, `y`). Openers are queued and consumed in order. - Masking an expression to a QUOTED token let the quote-stripper delete it entirely, so `${{ steps.x.outputs.cmd }} arg` reported `arg` as the command. The token now survives stripping and cannot match a command word, so an expression in command position contributes nothing. Ten further adversarial shapes were already correct and are pinned as regression guards: nested `$( )` in quotes, subshells, function definitions, indented heredoc terminators, backticks, bare redirects, adjacent and empty `${{ }}` sites, and a JSON literal inside an expression. Verified to flip — exactly the three above fail against the previous commit, the other ten pass. 434 real steps re-swept: 0 non-verbatim bodies, 0 live header lines, 0 missed expression sites, 0 out-of-order env, 0 `bash -n` failures. * fix(review): annotate the continuation accumulator so tsc can type it `invokedCommandsOf`'s backslash-continuation loop failed to compile with TS7022: the narrowed type of `pending` at the join line is the union of the loop-entry value and the back edge, and the back edge is computed from the join itself. The declaration's own annotation does not break that cycle - control-flow narrowing runs after it - so the checker gives up and calls the result `any`. Caught only on a forced rebuild. `tsc --build` had been reporting this tree clean off a stale .tsbuildinfo, which is the same shape of gap that put a type error into CI last week: the test runner strips types, so the suite stayed green either way. Gate with --force. * fix(review): a file-count label stops counting at the end of its line `Test Files 45 passed` filing its 45 as a differing TEST count was fixed once, with a lookbehind on the bare-count pattern. That only ever rejected the all-green shape. The moment any file fails, the runner prints `Test Files 1 failed | 44 passed (45)`, the label is no longer adjacent to the number, and 44 comes through as a test count - so the note reads "claimed 44, observed 1323" on exactly the runs whose summary someone would paste. Adjacency was the wrong invariant; the line is. Masking from the label to end-of-line is distance-independent and picks up jest's `Test Suites: 1 failed, 44 passed, 45 total` at no cost. The label keeps its `Test` word on purpose: a first cut matched a bare `files` and blanked the line in "expect all four files and 471 tests to pass", silencing a real claim. An existing test caught that, which is the argument for the rule being as narrow as it is - anything that suppresses claims is worth exactly its narrowness. Also drops the now-dead lookbehind, which a reviewer had separately (and wrongly) called a JS syntax error; variable-length lookbehind is legal in V8, and the module parsed fine. It goes because the line rule subsumes it, not because it was broken. * fix(review): address round-1 review of the mined-disciplines batch The positive control could state a verdict it never measured, and its own failure path shipped untested. Ten findings from the review of #8261: - `runControlMutant` returns `boolean | null`. An unreadable probe file injected no test and ran nothing, but returned `false` — which the caller reads as "the injected always-failing test stayed green", re-classes every survivor with that sentence, and discards the whole mutant/hunk window over an I/O error. `null` is the file's own third-outcome rule; the window stays spendable and the note says the harness was not validated. - `mutants.skippedForControl` / `hunks.skippedForControl`. A control that came back red stopped the run with candidates unprobed while every `skipped*` counter read 0 — the silent cap the brief teaching agents to read those counters rules out. Their own reason, not the budget's. - Cover the control's failure half: a dead-runner integration case (survivors re-classed, revert probe's `inert` re-classed, findings empty, nothing spent) and a could-not-set-up case pinning that `null` does NOT discard the window. Both branches, and the re-class block, were previously unreachable by any test. - `rulePath`: one existence check, not two. The second was unreachable behind the first and its note — "exists at the reviewed commit (the diff does not change it)" — silently retired, collapsing tracked state and this run's build output onto one sentence. The ignore status now picks the wording. - `expressionsOf`: forbid the body from crossing another `${{`. A malformed site above a real one swallowed it into the blob, so `${{ github.event.comment.body }}` stopped being enumerated as an injection site — the one direction this helper must not fail in. - `countDiffChangedLines`: count by hunk position, not prefix shape. A deleted line whose content starts `--` arrives as `--- …` (markdown rules, YAML markers, SQL comments), and every dropped line pushes the ratio toward a false `collapsedFromUpstream`. - `emptyDiff` is gated on `baseFetchFailed` too. A stale local base ref that already contains the head commits diffs to empty, and the skill acts on that by recommending close-as-superseded. - `test-delta`: price the budget slot against the command's own measured duration instead of a flat 5s floor, which admitted commands that could only time out and then disclosed them as "infrastructure, not evidence". `skippedForBudget` becomes a structured field; `now` becomes a seam so the test stops reassigning the global `Date.now`. - Drop `probeCreateFailureDetail` (no production caller after the swap to `worktreeCreateFailureDetail`) and the duplicated base-lock sweep in `cleanup.ts`, ten lines above the identical pre-existing one. - `--` before the path in `check-ignore`, and the comment says plainly that no `-`-leading claim survives extraction today. * fix(review): parse the base rerun before its output is trimmed `trimOutput` rescues module errors and runner summaries out of the omitted middle, not the per-file FAIL lines this command reads. A base suite whose failure section overruns the tail budget therefore lost failing files into the gap - and a SHORT base set is the dangerous direction, because netNew is the PR side minus the base side. Every file the trim hid came back as a Critical attributed to this PR by "measurement". Parse the raw text, report the bounded one. The PR side cannot be fixed here: it is read out of build-test's stored output, trimmed before this command existed. That loss runs the other way - it understates `shared`, never invents a netNew - so the entry carries `prTruncated` and the note says the list may be partial. A silence-biased gap is still a gap the author should hear about. Also names both selection caps in the mutant-skip diagnostic. The count accumulates replacement sub-cap drops, and with 2 deletions and 6 replacements the total is exactly MAX_MUTANTS: the main cap never fires, yet 3 are dropped, and the message sent the reader looking for a pool of 11 candidates that never existed. * test(review): pin the `sh` set-line to exactly what the runner uses A reviewer asked for `expect(script).not.toContain('set -e')` on a `shell: sh` step. The opposite is correct - GitHub runs that step as `sh -e {0}` - but the thread was right that nothing pinned it either way. Assert both halves: `set -e` is present, `pipefail` is not. Dropping the line makes an extracted `sh` step run past a failure the runner stops on; adding pipefail claims a bash feature `sh` does not have. * fix(review): rerun only the command shapes build-test emits This command reads a report off disk and then executes the strings in it with `shell: true`, in the base worktree. Nothing else in the pipeline re-executes a value it read back from a file, so nothing else has to care where that value came from - this does, and the provenance is worse than it looks: the command is `npm test --workspace="<dir>"`, the workspace token is a directory, and a directory is a name a pull request chooses. `packages/x";curl …|sh;"` is a legal path in git and on Linux and it round-trips through the report into a shell. Restricting to the emitter's own grammar costs nothing real, since that grammar is the two shapes build-test produces. A command outside it is skipped and disclosed, the same treatment everything else this command cannot do already gets, so a future shape degrades to "judge it by the diff" rather than to arbitrary execution. * test(review): pin the empty-diff and collapse detectors Both lived inline in a large object literal with no test, and one of them is the most expensive disclosure this command emits: the SKILL acts on `emptyDiff` by recommending the PR be closed as superseded. Dropping the 200-line floor, flipping the 4x comparison, or losing either empty-diff guard turned nothing red. Extracted as two pure predicates and pinned at the boundaries, including the two guards that are not about the diff at all - a capture that threw leaves diffText empty, and a merge base resolved from a stale local ref can already contain the head commits. Either would have closed a live PR on an infrastructure error. Checked by mutation rather than by reading: all four edits above now fail exactly one assertion each. * test(review): pin the 30s budget floor, and stop keeping its arithmetic twice Two findings from the same review round, both about budget math nobody was checking. The 30s floor separates "skipped, judge it by the diff" from "started, timed out, disclosed as infrastructure". Every existing budget test left it non-binding - two sit hundreds of seconds from the boundary and the third is capped by the per-command deadline - so regressing it to the old 5s turned nothing red. Verified by doing exactly that: 26 green before the new test, one red after. A one-second command admitted into a twenty-second window comes back labelled `budgetClamped`, which is the mislabelling the priced floor exists to prevent. The other is the same duplication lesson this skill keeps arriving at, in its own tests: one test carried the per-run budget arithmetic in a block comment AND at the `now:` argument, and when the per-run figure changed only one copy moved. The block also predated the positive control, so it omitted a whole suite run. Deleted the copy rather than correcting it - two agreeing copies today are two disagreeing copies later. * fix(review): the working directory is a setting the extraction was losing Two findings, both of them this file failing its own stated argument. The stub list dropped it. `expressions` covered the script and the env and nothing else, so a `working-directory: ${{ github.workspace }}/x` produced an empty list and a summary line reading "0 ${{ }} site(s) to stub" - and `expressionsOf`'s own comment says why that is the failure this list cannot afford: the caller reads it as "these are all the values to supply". Widened to every setting the command carries, the `shell:` template included. The emitted script never mentioned it. The env block is commented into the header precisely so a reader of the script alone can see it; the working directory changes what the script does just as much and was in the metadata only. The argument for reading all three levels, written in this file, is that a step run "in the wrong directory, and nothing says so" is the transcription error the command exists to remove - which is exactly what the header did. It is a comment rather than a `cd` for the same reason env is comments, not exports: the value may hold `${{ … }}` and this command substitutes nothing. Both pinned, and both checked by deleting the fix: each mutation fails exactly one of the two new tests. * fix(review): $(( )) is arithmetic, and a heredoc's form decides where it ends Three fixes and one deliberate non-fix, all measured against this repo's own 434 `run:` steps rather than argued. `$(( ))` was read as a command substitution, so `N=$((N + 1))` reported `N` as a command to stub. It was the single largest source of junk in the list: 196 distinct "commands" across the corpus, 165 without it. A plain `<<WORD` heredoc ends only on a line that is exactly WORD; the loose match ended it on an indented `EOF` inside the body and then read the body as commands, which is how `rm` got reported for a script that never runs it. `<<-` stays looser than bash (any leading whitespace, not just tabs) because looser can only end a body early, and this file's priority is that an under-report is the worse direction. `[^()]*` matched only the innermost `$( )`, so `X=$(gh api $(u))` lost `gh` - a missed stub, and the extraction reaches the network. Depth counted now, and the assignment-prefix skip no longer steps over an unclosed `X=$(gh` into reading `api` as the command. The non-fix is recorded where the next reader will hit it: the quote walk is flat while shell quoting nests, and over ~300 lines the drift reports fragments of jq source as commands. Inserting a separator where a blanked span was removes nine of those, but it splits `a"X"b`, which is one word to the shell, and the minimal reproducer for the difference is 296 lines - nothing short enough to pin it. A scanner nobody can pin costs more than the junk it removes. * fix(review): refuse an ambiguous step name, and report errors like the siblings Two findings, both about a message the caller never gets. A job may legally hold two steps with the same name. The selector took the first and said nothing, which is the failure this file's own header names - "picks the same-named step from the wrong job" - and it is worst in the use the command exists for: A/B extraction runs it once per tree, so a PR that adds or reorders a duplicate leaves the two sides comparing different steps while reporting on one. Refused out loud now, naming the indices; the index is always available and never ambiguous. The handler also let every throw propagate, so five carefully separated messages - cannot read vs cannot parse vs no job vs no step vs no `run:` - all arrived as "An unexpected critical error occurred" under a stack trace. `base-tree` and `test-plan` in the same directory already catch, write the message, and set exit 1. Matched. The separation between "the path is wrong" and "the YAML is wrong" only pays if the caller sees it. * docs(review): the brief's list of limits was one short "Two limits worth knowing before you spend the step" became three when the selector started refusing an ambiguous step name, and the count went stale in the same commit that made it wrong. This is the enumeration drift the skill teaches, in the text that teaches it. The added entry says what to do rather than only what happens: pass the index, which is what an A/B wants regardless — the two trees have to select the same step, and a name that moved between them is exactly how they stop doing that. * chore(review): drop a scratch probe file that reached the branch `packages/cli/inert.mjs` was a throwaway harness for sweeping the repo's own workflows; its cleanup ran with a relative path from the wrong directory and it got committed by the next `git add -A`. It failed CI and not the local hook because the two lint different sets: lint-staged filters to `*.{js,jsx,ts,tsx}`, which does not include `.mjs`, while CI's flat config picks the file up regardless of `--ext`. Verified with CI's own command this time, not the hook's. * fix(review): the collapse disclosure needs the guard its sibling already has Three findings from the round on this branch, all accepted. `isEmptyDiff` refuses to rule when the merge base came from a possibly stale local ref, because such a base can already contain the head commits and diff to empty. The PARTIAL form of that cause lands on `isCollapsedFromUpstream` instead - a stale ref holding most of the head commits shrinks the recomputed diff past the 4x ratio - and the flag then tells Agent 0 to read the body as description-of-history when the body may be perfectly current and the real cause is a fetch failure. Two predicates over the same evidence, one guarded and one not. Guarded now, and pinned: removing the guard fails exactly one test. The pre-set of `mutantsSkippedForBudget`/`hunksSkippedForBudget` in the no-budget-for-control branch was dead in every path and worse than dead in one: both loops run with `harnessValidated` still null, re-check the same budget, and set their own counters - and the hunk loop's figure excludes the collocated-probe inconclusives it pushes first, which the pre-set did not. Dropped, with the reason where the assignment was. `runControlMutant`'s bound is now stated in its doc: it validates ONE file, so a collector that silently drops a different probe file passes the control while that file's survivors stand. The per-file baseline gate bounds what is left. A `true` there is read as covering the run, so what it does not cover belongs next to it. * fix(review): bound the whole ledger marker, not just each of its fields The per-field caps leave the total unbounded: fifty findings at full width serialize to just under 17,000 characters. The module's own paragraph calls the marker "a footnote, never a payload" and reasons about GitHub's 65,536-character body limit, so the total is the number that claim is about and nothing was computing it. Set the budget from measurement rather than from the limit. Across every review this pipeline has posted on its own stack (n=66) the body runs a median of 721 characters, p90 2,178, max 3,925 - the limit has ~61 KiB of headroom, and an over-long marker was never going to fail a post. The real cost is proportion: 17,000 characters of invisible payload on a 3,925-character review is four times more marker than review. 8 KiB holds fifty findings at realistic widths without dropping any, and the truncation path is reached only by a ledger no round has produced. What it drops travels with it. A `dropped` count is the difference between a list the next round reads as complete and one that says it is not - the same distinction this module already draws everywhere else. * fix(review): a partial ledger must not render as a complete one The size cap added in the previous commit can drop entries, and the rendered section still opened with "**Every entry below is owed a this-round ruling**" and said nothing about what was missing. The `dropped` count existed and reached the only place a reader sees the list as nothing at all — which is the failure it was added to prevent, committed one step after adding the field. It now says the list is partial, how many are missing, and that absence is not evidence a finding was fixed. Second, the table escaping escaped `|` but not `\`. `\|` in a title became `\\|`, which markdown reads as an escaped backslash followed by a LIVE separator: the forged row the escaping exists to prevent, produced by the escaping. Backslash first now, and the test counts separators the way markdown does rather than by pattern, because a lookbehind counter is fooled by exactly the input under test. * fix(review): an empty ledger title is a cap the next round cannot lift A comment that is nothing but its severity marker produced a ledger entry with an empty title. That does not merely degrade the entry, it jams the review: the next round is told every ledger entry is owed a this-round ruling, has no claim text to rule on, answers `cannot tell`, and `cannot-tell-existing-critical` is one of the cap reasons. Nothing between rounds changes, so the cap comes back every round. Dropping the entry would hide a Critical that really was posted, so it keeps its place and gets the one handle there is - the location it was filed at, or "the review body" for a body-level one. The field's stated job is "enough for the next round to re-locate the claim", and that is what a text-less finding still has. Found by probing `buildLedger` with ten malformed comment shapes; the other nine - carried ids, id collisions, two comments carrying the same id, `R2D2` not being an id, a leading newline before the marker - all behaved correctly and needed no change. * fix(review): count both caps as dropped, not just the byte one Found by running the three previous fixes together on one round: 51 findings in, 24 kept, and the marker said 26 were missing. `LEDGER_MAX_FINDINGS` truncates before the byte cap ever runs, and measuring `dropped` against the already-sliced list left the count cap's share silent - so the field added two commits ago to stop a truncated list reading as complete was itself under-reporting the truncation. Measured against what came IN now, and pinned across all four combinations: count cap alone, byte cap alone, both, and neither. Kept plus dropped equals given, in every one. * fix(review): the ignore check was the one git spawn without a deadline `isGitIgnored` reached for a bare `execFileSync` while every other git invocation in these commands goes through the package helper, which carries `GIT_TIMEOUT_MS` and `GIT_TERMINAL_PROMPT=0`. That constant's own comment is the argument: "a hang must still end". This one ran against a worktree the review does not control and could not. No behavioural delta for any input that does not hang - all 80 test-plan tests pass unchanged - so it ships without a new test rather than with a mocked-spawn assertion that would pin the call shape instead of the behaviour. * docs(review): a control killed mid-run did not "never run at all" SKILL's `harnessValidated` paragraph enumerated four ways the value lands on `null` - no green baseline, no candidates, no budget, an unreadable probe file - and all four are "the control never ran". There is a fifth: `runControlMutant` throws when its deadline kills the run or the runner cannot be spawned, the outer catch swallows it, and the value stays `null` rather than becoming a fabricated `false`. That is the path a slow machine takes. The ruling is unchanged - neither validated nor refuted, a survivor stands but unconfirmed - so this is about what the terminal says, not what it decides. An agent quoting "the control never ran at all" for a control that ran and was killed states something untrue about the run, and the enumeration is what it quotes from. --------- Co-authored-by: qwen-code-dev-bot <qwen-code-dev@service.alibaba.com> Co-authored-by: Qwen Code <qwen-code@users.noreply.github.com> Co-authored-by: Qwen Code Bot <qwen-code-bot@users.noreply.github.com> Co-authored-by: Qwen Code Autofix <qwen-code-autofix@users.noreply.github.com>
|
Released in v0.21.3. |
What this PR does
Stacked on #8215. Four more
/reviewcapabilities, borrowed from the shape of a live round-2 maintainer re-verification (#7818's round-2 comment):qwen review test-delta). Agent 7 has always judged failures by path — a failure in a touched file is a Critical, in an untouched file pre-existing — which misclassifies in both directions. The new subcommand reruns the PR side's failed test commands on the merge base feat(review): Test Plan claim check, base-tree A/B harness, per-hunk probes #8215'sbase-treebuilds, and diffs the failing file sets:netNewfails on the PR side only (the PR's own, whatever files the diff touches),sharedfails on base too (pre-existing by measurement). Counts are deliberately never compared — a flaky suite fails different test names between two runs of the same tree — and a base rerun that times out attributes nothing rather than promoting PR-side failures to evidence.R1-2), and a high-effort incremental re-review owes each one a ruling — fixed / still stands / cannot tell — under the same trace-the-mechanism bar the open-Criticals re-check already enforces, rendered as a table at the top of the report.test-efficacy, each lifted from a survivor the deletion operator cannot express: drop a?? fallback(the miss path is frequently the entire safety property), force a comparison-bearing guard condition totrue(a survivor means nothing pins when the guard must not fire — the round-2 fix in feat(cli): add /model --compaction for configurable chat compression model #7818 shipped exactly so), and drop a+ UPPER_CONSTterm (an unpinned reserve in an estimate). A line whose raw text and literal-blanked code view disagree yields no candidate at all — an edit index computed across the two views once splicediftrue 0)into a guard, and a mangled mutant reads asinconclusivewhile quietly spending a cap slot. Deletion mutants keep cap priority; every skip is counted.Why it's needed
Round-2 re-verification is where a hands-on maintainer produces the evidence a from-scratch review cannot: what became of last round's findings, which failures are genuinely the PR's, and whether the fixes made in response to review are themselves tested. Each of the four is the deterministic half of one of those habits, moved into the pipeline so every re-review gets it.
Reviewer Test Plan
How to verify
cd packages/cli && npx vitest run src/commands/review src/commands/review.test.ts; expect all 43 files and 1185 tests to pass, including the newtest-delta.test.tsand thereplacementMutantsOf/selectMutants — replacement operatorssuites intest-efficacy.test.ts.packages/core/src/skills/bundled/review/DESIGN.mdand the wiring inSKILL.md(Agent 7 downstream, Step 6 previous-round table, Step 8 cache format).test-deltamechanically: point--reportat a build-test JSON whosetest[]carries a failure,--baselineat any built checkout of the base; confirm a failure present on both sides lands inshared, one absent from base innetNew, and that a green report answers "nothing to attribute" without running anything.replacementMutantsOf(' if (s === ")") …', …)(string-bearing line) yields no candidate;if (ready)(no comparison) yields none;state.clear();still selects the deletion mutant, not a replacement.Evidence (Before & After)
The attribution case is the previous live run of this pipeline (#8215's evidence):
npm test --workspace=packages/corefailed with three environment-sensitive files (extensionManager,session-writer-lease) that the model had to judge pre-existing by path.test-delta's test suite pins that exact scenario as a measurement — both files fail on base,netNewempty. The mutation operators' three shapes are the three surviving mutations of #7818's round-2 report (?? config.getModel()untested, the guard-skip condition untested,+ COMPACT_MAX_OUTPUT_TOKENSunpinned); each now maps to a selector with a matching unit test. Non-UI change otherwise: N/A.Tested on
Environment (optional)
Unit suites only for this branch (the pipeline integration paths it extends were exercised live on this machine under #8215).
Risk & Scope
test-deltafalls back to the existing path judgment wherever it cannot rule (no merge base, unparsed output, timed-out base rerun), and the mutation operators skip any line whose code view disagrees with its raw text. The cost of that bias is missed candidates, never wrong findings.round,findings— older caches without them are handled), additive mutant fields (operator,mutated— deletion mutants keep the legacy shape).Linked Issues
Stacked on #8215 (uses its
base-tree; retarget tomainonce #8215 lands). Workflow reference: the round-2 maintainer verification on #7818.中文说明
本 PR 做了什么
叠在 #8215 之上。为
/review再增加四个能力，借鉴自一次真实的二轮 maintainer 复验（#7818 的 round-2 评论）：qwen review test-delta）。Agent 7 一直按路径判断失败归属——改动文件中的失败算 Critical、未触及文件中的算预存在——两个方向都会误判。新子命令在 feat(review): Test Plan claim check, base-tree A/B harness, per-hunk probes #8215 的base-tree构建出的 merge base 上重跑 PR 侧失败的测试命令，并对比失败文件集：netNew只在 PR 侧失败（无论 diff 是否触及该文件，都是 PR 自己的问题），shared在 base 上同样失败（实测预存在）。刻意不比较计数——flaky 套件在同一棵树的两次运行中失败的测试名都不同——base 侧重跑超时则不归因任何东西，而不是把 PR 侧失败提升为证据。R1-2）持久化已确认的 findings，高档增量复审对每一条负有裁定义务——fixed / still stands / cannot tell——采用与既有 open-Criticals 复查相同的"追踪机制"标准，并在报告顶部以表格呈现。?? fallback（miss 路径往往就是整个安全属性）、把含比较的 guard 条件强制为true（幸存意味着没有测试钉住 guard 不该触发的情形——feat(cli): add /model --compaction for configurable chat compression model #7818 的 round-2 修复正是如此裸奔上线）、删+ UPPER_CONST项（估算中未钉住的预留量）。原始文本与字面量剥离后的代码视图不一致的行不产生任何候选——曾有一次跨视图计算的编辑下标把 guard 拼接成iftrue 0)，而损坏的变异体以inconclusive呈现、还悄悄占掉一个 cap 名额。删除变异体保持 cap 优先级；所有跳过都有计数。为什么需要
二轮复验正是人工 maintainer 能产出从零评审给不出的证据的地方：上一轮的 findings 都怎么样了、哪些失败真正属于本 PR、以及为回应评审而做的修复本身是否有测试。这四项分别是其中一种习惯的可确定性化的一半，移入流水线后每次复审都能得到。
Reviewer 测试计划
验证方式
cd packages/cli && npx vitest run src/commands/review src/commands/review.test.ts；预期 43 个文件、1185 项测试全部通过，含新增的test-delta.test.ts以及test-efficacy.test.ts中的replacementMutantsOf/selectMutants — replacement operators套件。packages/core/src/skills/bundled/review/DESIGN.md新增的四节依据，以及SKILL.md的接线（Agent 7 downstream、Step 6 前轮表格、Step 8 缓存格式）。test-delta：--report指向test[]携带失败的 build-test JSON，--baseline指向任一已构建的 base checkout；确认两侧都失败的文件落入shared、base 上不失败的落入netNew、全绿报告不运行任何东西即回答"nothing to attribute"。replacementMutantsOf(' if (s === ")") …', …)（含字符串的行）不产生候选；if (ready)（无比较）不产生；state.clear();仍选择删除变异体而非替换。前后对比证据
归因场景来自本流水线上一次实测（#8215 的证据）：
npm test --workspace=packages/core带着三个环境敏感文件（extensionManager、session-writer-lease）失败，模型只能按路径判断预存在。test-delta的测试套件把这一确切场景钉为测量——两个文件在 base 上同样失败、netNew为空。变异算子的三个形态即 #7818 round-2 报告的三个幸存变异（?? config.getModel()无测试、guard 跳过条件无测试、+ COMPACT_MAX_OUTPUT_TOKENS未钉住）；每个现在都有对应选择器和配套单测。其余非 UI 变更：N/A。测试平台
环境（可选）
本分支仅单元测试（其扩展的流水线集成路径已在 #8215 下于本机实测）。
风险与范围
test-delta在无法裁定处（无 merge base、输出不可解析、base 重跑超时）回退到既有路径判断，变异算子跳过代码视图与原文不一致的行。这种偏向的代价是漏掉候选，绝不是错误的 finding。round、findings——无这些字段的旧缓存已兼容处理）、增量式变异体字段（operator、mutated——删除变异体保持旧形状）。关联 Issue
叠在 #8215 之上（使用其
base-tree；#8215 合并后 retarget 到main）。工作流参考：#7818 上的二轮 maintainer 复验。