feat(verify): sweep sibling shapes, calibrate replays, measure suggested fixes - #8242
Conversation
…ted fixes The verify-pr skill produced a `merge-ready` report on a head that leaks live HTML. Measured on #8147 at b0ae340 with two control arms and one treatment arm (real qwen3.8-max-preview, isolated QWEN_HOME each, arms differing only by this file): both control arms concluded `merge-ready`, the treatment arm concluded `findings` and shipped a measured two-line fix. The decisive arm is the control replicate. It spent the same budget as the treatment arm and built a stronger instrument — a micromark+GFM rendering oracle, 23 hostile fixtures, 254 assertions. It built the HTML-block-times-fence fixture and separately probed code spans, and never crossed the two axes. Re-running that arm's own oracle on the fixture it never built renders a live <img> and a live <a href>, so it would have failed its own guarantee-1 assertion. The miss was which fixtures the agent thought to construct, which is what these rules supply. Added to the A/B method: sweep the sibling shapes a parser-class fix does not pin; treat untrusted text reaching a parser as a scaling question and probe it with a timed ladder; audit a stated cost against the repo's own accounting of the same mechanism; enumerate the unnamed siblings of an accepted-tradeoff list. Added to shared-state probing: an instruction in a prompt is not an invariant, and interleavings that fabricate a result outrank ones that crash. Added to vacuity: run the mutation in reverse (a suite green with and without a candidate fix pins nothing along that axis), and read a test's name as a claim about its fixture. Workflow replays now calibrate against a real production artifact before their cells are believed, and suggested fixes ship with collateral measured. The follow-up status enum gains `worsened`, because a fix can move an accepted tradeoff. Rules whose effect was not measured are marked as such in the PR description rather than presented as validated.
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
|
Thanks for the PR! Template: headings differ from the template ( Problem: observed and measured, not theoretical. The PR documents a concrete miss — the Direction: aligned. This hardens an internal QA skill — the verify-pr skill is the last line of defence before a maintainer reads a report, and rules that close documented misses are squarely in scope. No auth, sandbox, model, telemetry, or public-contract surface touched. Approach: the scope feels right. Seven rules, each anchored to a specific measured failure, with the "what is not validated" section honestly naming which rules lack A/B evidence (the four from #8215). The diff is focused — only the skill file and its test, no drive-by changes. One observation: the rules are dense and some paragraphs run long (the scaling-question rule is ~15 lines of prose). That's inherent to the "measured example" format and consistent with the file's existing style, so not a concern — just noting it for the maintainer's awareness. Risk: no elevated risk signals. Changed files are Moving on to code review. 🔍 中文说明感谢贡献！ 模板：标题与模板不同（ 问题：已观测且有实测数据，非理论性加固。PR 记录了一次具体的漏检—— 方向：对齐。此 PR 加固内部 QA skill——verify-pr skill 是 maintainer 阅读报告前的最后防线，关闭已记录漏检的规则完全在范围内。未触及 auth、sandbox、model、telemetry 或公共契约面。 方案：范围合理。七条规则，每条锚定一次具体实测失败，"未验证"部分诚实地标明了哪些规则缺少 A/B 证据（来自 #8215 的四条）。diff 聚焦——仅 skill 文件及其测试，无顺手改动。一点观察：规则较密集，部分段落较长（scaling-question 规则约 15 行 prose）。这是"实测示例"格式的固有特征，与文件既有风格一致，不构成顾虑——仅供 maintainer 知悉。 风险：无升级风险信号。变更文件为 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code reviewIndependent proposal: given the documented miss (verify-pr returned Comparison with the diff: the PR matches this closely and executes it well. The rules land in the right structural positions — sibling sweep and scaling ladder extend the A/B method section right after the existing "fix that holds only for the reported input shape" rule; the cost audit expands the cache-pool paragraph; the accepted-tradeoff boundary follows "report which ones do NOT hold"; prompt-is-not-invariant sits with the concurrency-adjacent content; reverse mutation and test-name-as-claim go in the vacuity section; workflow calibration expands the existing workflow/CI rule; measured suggested fix expands the report contract. No rule is orphaned or misplaced. No critical blockers. No AGENTS.md violations. Two observations, neither blocking:
The test file adds 17 assertions to the existing TestingFinal CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 The ubuntu test suite is still running. macOS/Windows/Integration are skipped — expected for a change that touches only No behavioural claim to settle with 中文说明代码审查独立方案： 鉴于已记录的漏检（verify-pr 在一个通过 code-span × HTML-block 交叉泄漏活 HTML 的 head 上返回了 与 diff 的比较： PR 与此方案高度吻合且执行良好。规则落在正确的结构位置——邻门扫描和规模阶梯扩展 A/B 方法节，紧接既有的"修复仅对被报告形态成立"规则之后；成本审计扩展缓存池段落；已接受权衡边界跟随"报告哪些不成立"；指令非不变量与并发邻接内容放在一起；反向变异和测试名即声明放在空洞性节；工作流校准扩展既有工作流/CI 规则；修复实测扩展报告契约。无规则孤立或错放。 无关键阻塞项。无 AGENTS.md 违规。两点观察，均不阻塞：
测试文件在既有的 测试ubuntu 测试套件仍在运行。macOS/Windows/集成测试已跳过——对于仅触及 无需以 — Qwen Code · qwen3.8-max-preview Reviewed at |
|
Confidence: 5/5 — clean across every stage; the problem is measured, the rules are well-anchored, and the honesty about what's not validated is exactly what this skill exists to enforce. This PR does something rare: it applies the verify-pr skill's own standards to the skill itself. The three-arm A/B against #8147 is the kind of evidence the skill demands of others — ground truth re-established from the actual production functions, not cited from a prior round; the replicate arm spending the same budget as treatment and building the stronger instrument, yet still missing because it never crossed two axes. The treatment arm's find traces directly to the added text, and the PR says so. The seven rules land in the right structural positions and each carries its measured example. The four from #8215 that lack A/B validation on this file are named as such — "unvalidated, drawn from a hands-on round's findings" — which is the report-contract rule applied to the PR's own claims. The The test assertions pin each rule's presence with the established Approval deferred until CI lands green on 中文说明置信度：5/5 —— 各阶段均干净；问题有实测数据，规则锚定良好，对未验证内容的诚实正是此 skill 存在的意义。 此 PR 做了一件少见的事：以 verify-pr skill 自身的标准来审视 skill 本身。针对 #8147 的三臂 A/B 是该 skill 要求他人提供的那种证据——ground truth 从实际生产函数重新建立，而非引用前轮结论；复现臂花费与处理臂相同的预算并造了更强的仪器，却仍因从未交叉两个轴而漏检。处理臂的发现可直接溯源到新增文本，PR 也如此说明。 七条规则落在正确的结构位置，每条都携带实测示例。来自 #8215 的四条缺少对本文件 A/B 验证的规则被明确标注——"未验证，源自一轮实操验证的发现"——这是报告契约规则应用于 PR 自身声明。 测试断言以既有的 审批推迟至 CI 在 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
中文说明
未发现问题。LGTM！✅
— qwen3.8-max-preview via Qwen Code /review
Six more rules, each taken from a hands-on verification round that found something the skill could not have produced. From #8037, a fallback parser for XML tool calls: a bug report enumerates its own cases, and those names are coverage claims the fix inherits — holding the issue's reported 1,898-character preamble fixed and varying only the tool showed the guard declining run_shell_command, which the issue names explicitly, so the fix covered half of what it was filed against. The same round found one defect with a loud variant (schema rejects a dropped argument) and a silent one (a truncated file), which fixes the ordering: rank variants by observability, not blast radius — the same ordering the concurrency rule already uses. And its harness replayed the reported wire bytes without reproducing the model-side degradation that produces them, so Not covered now asks which of the two a reproduction actually has. From #8005, a Goal runtime driven through a real TUI: walking the PR's own Reviewer Test Plan step by step turned an unrunnable step into the round's sharpest finding, because no code path could insert real user input into an active turn and the feature's completion criterion needed exactly that. Proving it took a census rather than a reading — 30 captured verifier payloads carried one evidence kind and never the required one. And its mutation matrix is the reason survivors are now gated on a positive control: two mutations survived 429/429 and 326/326, believable only because a third, expected-caught mutation turned one test red. None of these six were measured with an A/B on this file; they are drawn from rounds that found real defects. The PR description says so.
|
@qwen-code /takeover |
qwen-code-dev-bot
added
the
autofix/takeover
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管：autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突，直到移除标签或达到轮次上限。移除 |
|
🔀 Base updated: red check(s) [Test (ubuntu-latest, Node 22.x)] pass on current main — merged current main via update-branch; CI will re-run. 中文说明🔀 已更新 base：红色检查 [Test (ubuntu-latest, Node 22.x)] 在当前 main 上通过 —— 已通过 update-branch 合入当前 main，CI 将重新运行。 |
Review: the tests hold up, the description's accounting does notVerified at head Verdict: merge-ready. The test side is clean and — checked individually — non-vacuous. One Medium finding about what the A/B actually measured, three Low findings about the description and the file's organization. None of them block. 1. [Medium] The shipped file is not the artifact the arms measuredThe branch is two commits, and the second one landed after the arms ran (the body says so: "Added after the arms above"):
So the treatment arm that returned That matters because of this PR's own thesis. The round-6 result is explicitly not about capability, budget, or oracle quality — it is about which fixtures the agent thought to construct, i.e. the prompt's content is the causal mechanism. Changing the mechanism by 8% after measuring it is the file's own rule at L350 turned inward: "A limit measured in isolation does not transfer to the real call site." The I am not claiming dilution occurs — see Not covered. I am claiming the report should say the measured arm ≠ head. Either add one line to the body, or land 2. [Low] The unvalidated accounting is inconsistent, and four rules are double-listedThe body says "Second batch — six more rules" and then "Unvalidated: all ten rules in the second batch above". Six vs ten. The gap is the four rules attributed to #8215 (scaling ladder, prompt-is-not-an-invariant, name-is-a-claim, cost audit). Those are already enumerated in the first-batch Change section — and they are all in So the body lists them twice and then declares them unvalidated despite their shipping inside the arm that was measured. The count that survives audit is: six rules added after the arms, not ten. 3. [Low] Both numbers in the Tests section are wrong at this head
In a PR whose stated point is to stop asserting that prose works, the description's own counts are the cheapest thing to keep true. 4. [Low]
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
中文说明
— qwen3.8-max-preview via Qwen Code /review
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： Autofix review round: no code change this roundI triaged every feedback point and verified the PR locally. None of the findings is a correctness defect in the changed code, so this round makes no commit. Details below. CI —
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
— qwen3.8-max-preview via Qwen Code /review
ReviewTwo files, no product code: What I verifiedWorktree at
Findings1. The new replay-calibration precondition cannot be met on a first round, and the rule states no fallback. (medium)
And the snapshot is The rule closes with "A replay that cannot reproduce a known real output is measuring your harness, not the PR." On a first-round workflow PR that sentence tells the agent its only available instrument is worthless, with no stated exit. Worth noting this is exactly the configuration the arms ran in (snapshot "without comments"), so whatever the treatment arm actually did here went unmeasured. Suggest one sentence after "…name the diffs you allowed":
2. The ladder rule has no time cap and can eat the budget it is spending. (medium)
The example already contains the fix — "over 100 s" is a cap, not a measurement. Make it explicit:
3. Two new instructions ship unpinned, against the file's own convention. (low) The test's own preamble (
Both are new normative text, and the PR's own rule 5 is that a fix ships with its fixture. 4. The description's arithmetic does not match the diff — in a PR whose thesis is that counts are sacred. (low)
5. The text the arms measured is not the text being merged. (low, strategic — no action required) The A/B validated a The premise the arms established is precisely that rule text steers which fixtures an agent constructs. That is also the mechanism by which ten more rules could dilute the first batch's salience — and it is the one thing the arms could have measured cheaply and did not. Not a blocker, and the disclosure is exactly right; but a single confirmation arm on the merged text against the same #8147 fixture would close the loop, and its absence is the strongest argument for splitting the unvalidated ten into a follow-up. 6. Section placement. (nit)
VerdictGood change, and the four rules I'd bet on (sibling sweep, reverse mutation, positive control, measured suggested fix) are the ones with evidence behind them. Finding 1 is the one I would fix before merge — it plants a contradiction between two sections of the same prompt, and a contradicted agent picks one side silently. Finding 2 is a one-line edit. The rest are follow-up material. 中文说明已验证worktree 位于
发现1. 新增的 replay 校准前置条件在首轮根本无法满足，且规则未给出退路。（中）
于是唯一可达的"真实产物"只剩 而规则的收尾是"无法复现已知真实输出的重放，测量的是你的装置而不是 PR"。对首轮的 workflow PR 而言，这句话等于告诉 agent：你唯一能用的仪器没有价值——却没给出口。顺带一提，本 PR 各臂正是跑在这个配置下（快照"不含 comments"），所以处理臂在这一点上实际怎么做的，恰恰没有被测量。 建议在"name the diffs you allowed"后补一句：无法取得真实产物时（首轮、无 token、无 2. 阶梯规则没有时间上限，可能吃掉它正在花的预算。（中）
修法其实已经藏在例子里——"over 100 s"本身就是一个上限而非测量值。把它写明：每一级都在 3. 两条新指令未被断言钉住，违反了该文件自己的约定。（低） 测试自身的前言（
两者都是新增的规范性文本，而本 PR 自己的第 5 条正是"修复应与它的 fixture 一同交付"。 4. 描述里的数字与 diff 对不上——而这个 PR 的主张恰恰是"计数神圣"。（低）
5. 各臂测量的文本，并不是要合入的文本。（低，策略性，不需要改动） A/B 验证的是携带第一批规则的 各臂确立的前提正是：规则文本会左右 agent 去构造哪些 fixture。而这同样也是"再加十条可能稀释第一批显著性"的机制——这恰恰是各臂本可以低成本测量却没有测的一件事。这不构成阻塞，披露也完全到位；但用合入后的文本对同一个 #8147 fixture 再跑一个确认臂就能闭环，而它的缺席正是"把未验证的十条拆成后续 PR"的最强理由。 6. 章节归属。（nit）
结论好改动，我最看好的四条（邻门扫描、反向变异、正对照、修复实测）恰好都是有证据支撑的。发现 1 是我会在合入前处理的一条：它在同一份 prompt 的两个章节之间埋下矛盾，而被矛盾夹住的 agent 会静默地选一边。发现 2 是一行的事。其余可以放到后续。 |
…inning, section placement (#8242)
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 1/100 轮）。改动内容与我反驳保留之处如下： Review feedback addressedFinding 1 — Calibration precondition has no first-round fallback (medium) → FixedAdded an explicit uncalibrated exit path after the calibration requirement in the workflow/CI bullet. When no real emitted artifact is retrievable — a first round, no token, no Finding 2 — Ladder rule has no time cap (medium) → FixedAdded three lines to the scaling-ladder bullet: "Run each rung under Finding 3 — Two new instructions ship unpinned (low) → FixedAdded three test assertions:
Finding 4 — Description arithmetic does not match the diff (low) → DeclinedThe PR description is managed outside this checkout; this workflow has no GitHub credentials to edit it. The counts should be corrected when the description is next updated. Finding 5 — Arms measured different text than what merges (low, strategic) → DeclinedThe reviewer explicitly states "no action required" and "not a blocker." The disclosure in the description is acknowledged as correct. A confirmation arm on the merged text is a follow-up item. Finding 6 — Section placement (nit) → FixedMoved four non-A/B bullets (bug-report-as-coverage-spec, Reviewer Test Plan walk, census, observability ranking) from Verification
中文说明已处理的评审反馈发现 1 — 校准前置条件在首轮无退路（中）→ 已修复在 workflow/CI 条目的校准要求之后，新增了一条明确的未校准退出路径。当无法取得真实产物时（首轮、无 token、无 发现 2 — 阶梯规则没有时间上限（中）→ 已修复在规模阶梯条目中新增三行："每一级都在 发现 3 — 两条新指令未被断言钉住（低）→ 已修复新增三条测试断言：
发现 4 — 描述中的数字与 diff 不符（低）→ 拒绝PR 描述由本 checkout 之外管理；本工作流没有 GitHub 凭据来编辑它。计数应在下次更新描述时修正。 发现 5 — 各臂测量的文本与要合入的文本不同（低，策略性）→ 拒绝评审者明确写了"不需要改动"且"不构成阻塞"。描述中的披露被认为是到位的。对合入后文本的确认臂属于后续事项。 发现 6 — 章节归属（nit）→ 已修复将四条非 A/B 条目（缺陷报告即覆盖声明、走 Reviewer Test Plan、普查、按可观测性排序）从 验证
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审；第 100 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
中文说明
已审查——无阻断问题。 建议见行内评论。
— qwen3.8-max-preview via Qwen Code /review
| // is 0.0ms and proves nothing — only the ladder shows the curve, and | ||
| // only tracing the body to a fork contributor makes it a finding. | ||
| expect(flat).toContain('is a scaling question'); | ||
| expect(flat).toContain('probe it with a **ladder**'); |
There was a problem hiding this comment.
[Suggestion] These two assertions pin literal ** markdown emphasis markers — this one and expect(flat).toContain('Distinguish reproducing the **shape**'); at line 2456 — the first such cases in a file whose ~48 other skill-pinning assertions all pin bare words. flat = verifySkill.replace(/\s+/g, ' ') collapses whitespace only and does not strip **, so the assertions match the raw bold markers rather than the rule's words. — Concrete cost: a future doc-normalization pass (or editor) that changes **ladder** to *ladder* or drops the bold leaves the rule fully intact but fails the carries the maintainer-round verification techniques test, sending a maintainer to discover the test was pinning emphasis syntax, not the rule its header comment says it pins.
| expect(flat).toContain('probe it with a **ladder**'); | |
| expect(flat).toContain('probe it with a'); |
The same fix applies at line 2456: pin 'Distinguish reproducing the' without the markers.
中文说明
这两条断言钉住了字面的 ** markdown 强调标记——本条以及第 2456 行的 expect(flat).toContain('Distinguish reproducing the **shape**');——是本文件中的首例（其余约 48 条 skill 钉住断言都钉裸词）。flat = verifySkill.replace(/\s+/g, ' ') 只折叠空白、不剥离 **，因此断言匹配的是原始加粗标记，而非规则的措辞。
具体代价： 未来一次文档规范化（或编辑器）把 **ladder** 改成 *ladder* 或去掉加粗后，规则本身完好无损，但 carries the maintainer-round verification techniques 测试会失败，迫使维护者去发现：测试钉住的是强调语法，而非其头部注释所声称的那条规则。建议同样去掉标记钉住裸词：本行改为 expect(flat).toContain('probe it with a');，第 2456 行改为 expect(flat).toContain('Distinguish reproducing the');，与文件既有约定一致。
— qwen3.8-max-preview via Qwen Code /review
|
🤖 Addressed the latest review feedback (round 2/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 2/100 轮）。改动内容与我反驳保留之处如下： Review feedback addressed[rc:3694026582] Skill-pinning assertions should not include
|
Review (round 3) — the previous round's fixes landed; the last commit weakened two pinsRe-verified at head Verdict: merge-ready. Four of the six findings from the earlier rounds are genuinely closed at this head. One new Low finding comes from the very last commit, and the description's arithmetic is still the weakest artifact in the PR. Closed since the last round
Verified good
Findings1. [Low] The head commit stripped the payload word out of two pins, and both now pass on text that has lost the rule.
The intent is right — Bare-word replacements, each verified unique in the head file today:
The third row is not a defect — the emphasis wraps the whole sentence there, so the longer string is available for free. 2. [Low] Two sentences in the ladder bullet now each declare themselves "the finding".
The second sentence is pre-existing; the first arrived with the round-2 timeout fix. Read as an instruction they disagree: one says report the rung that timed out, the other says a single capped rung proves nothing without the curve. An agent that hits the cap on rung 1 cannot tell from this whether it has a finding or a budget excuse. Also 3. [Low] Half the new section does not belong to its own heading. 4. [Low] The description's accounting is still self-contradicting — the one artifact this PR is about.
5. [Info] The file is now the largest skill in the repo, and only this PR's first batch has ever been measured. Trend on the same file: No action here, and the disclosure in the body is the right call. But the arms measured recall on a new axis and nothing measured retention on the old ones — and the PR's own thesis is that rule text is the causal mechanism for which fixtures an agent builds, which is exactly the mechanism by which rule #27 could displace rule #4. The cheap closing arm is a regression arm, not another discovery arm: re-run the merged text against a fixture a previous round's control did find, and confirm it is still found. One arm, same harness, and it converts "we assume no dilution" into a number. Related, and cheaper: every rule now ships with a verbatim string lock, so the file can only ever grow — rewording for concision breaks tests. Pinning a stable anchor instead (an HTML comment rule id next to each bullet, Not covered
Methodology
中文说明在 head 结论：可合入。 上一轮的 6 条发现中有 4 条在本 head 已真正闭环（校准退路、timeout 上限、两条未钉住的指令、新章节），"23 条断言"现在也确实成立。新增 1 条来自最后一次提交的低级别问题，另有描述侧的计数仍未修。 已验证
发现
未覆盖
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
— qwen3.8-max-preview via Qwen Code /review
…out, relocate misplaced bullets (#8242)
|
🤖 Addressed the latest review feedback (round 3/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 3/100 轮）。改动内容与我反驳保留之处如下： Round 3 review — summaryAll findings are from @wenshao's issue-level review (round 3). No inline comments this round. Finding 1 [Low] — Weakened assertion payload words → ImplementedThe head commit
Each replacement was verified unique (exactly 1 occurrence) in the head SKILL.md before applying. Finding 2 [Low] — Contradictory "the finding" sentences + unbound
|
| 修改前 | 修改后 |
|---|---|
'probe it with a' |
'ladder' |
'Distinguish reproducing the' |
'not the model-side degradation that produces it' |
'is a scaling question' |
'Untrusted text reaching a parser is a scaling question' |
每条替换均已验证在 head SKILL.md 中唯一（恰好 1 次出现）。
发现 2 [低] — 矛盾的 "the finding" 句 + 未绑定的 <N> → 已实施
- 将
<N>绑定为具体默认值：timeout 30/>30 s。 - 调和两句话：撞上上限的那一级现在是 "the evidence"（数据点），超线性曲线仍然是 "the finding"。在第 1 级就撞上限的 agent 现在有明确的解读：该级是证据，曲线才构成发现。
发现 3 [低] — Scoping 标题下的错位条目 → 已实施
将两条要点从 ### Scoping from the report and the plan 移至其内容所属位置：
- "Rank a defect's variants by observability" → 移至其交叉引用的并发规则（"An instruction in a prompt is not an invariant"）之后，使 "the concurrency rule above" 确实在上方。
- "Prove a negative by census" → 移至 vacuity 章节中 "count at the destination, not at the component boundary" 之后，与其他计数/插桩材料相邻。
Scoping 章节现在只保留其两条 scoping 要点（bug 报告枚举、Reviewer Test Plan）。
发现 4 [低] — PR 描述的计数问题 → 拒绝（超出代码变更范围）
"六条与十条"的矛盾、#8215 的 commit 归属、以及 "107/107 → 116/116" 的计数均在 PR 正文中，由工作流管理——不在分支代码中。代码变更无法修复这些问题；需要工作流重新生成 PR 正文。
发现 5 [信息] — 文件大小观察 → 无需操作（评审者明确立场）
评审者声明 "No action here"，并将大小趋势作为背景披露。建议（回归臂、基于锚点的钉住、it.each）已记录供未来轮次参考，但不在本轮评审范围内。
验证
npm run build— 通过npm run typecheck— 通过npm run lint— 通过npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/qwen-triage-workflow.test.js— 116 通过 (116)
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。
Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审；第 100 轮后本 bot 停止并将 PR 交给人工。
🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
中文说明
未发现问题。LGTM！✅
— qwen3.8-max-preview via Qwen Code /review
Verification round — real harnesses at
|
| # | mutation applied to SKILL.md |
suite |
|---|---|---|
| M0 | none (control) | GREEN 116/116 |
| M1 | positive control — reword a rule the PR pins (gets its siblings swept → gets its neighbours visited) |
RED 1 failed / 115 passed |
| M2 | delete the round-1 fix — the uncalibrated exit path |
GREEN 116/116 |
| M3 | delete the round-2/3 fix — the timeout 30 budget cap |
GREEN 116/116 |
| M4 | undo the round-3 fix — rename ### Scoping from the report and the plan |
GREEN 116/116 |
M1 is what makes M2–M4 worth believing: the harness can turn this suite red, so the three survivors are real. grep agrees — uncalibrated, timeout 30 and is the evidence appear in SKILL.md and zero times in the test file.
This is finding 3 from round 1 ("two new instructions ship unpinned"), reintroduced once per fix commit: the normative text added to close a review finding is the text that arrives without a fixture. Three strings, each verified to occur exactly once at head:
expect(flat).toContain('presenting it as calibrated is what is not allowed');
expect(flat).toContain('the rung that hits the cap is the evidence');
expect(flat).toContain('Scoping from the report and the plan');Low, not blocking — but it is the file's own convention and this PR's own rule 5, and the cost is three lines.
4. The body's ground truth reproduces exactly — and the loop has closed downstream
The rules only earn their place if the failure they are anchored to is real, so I re-established it here instead of citing the description. html_escape / emit_block / emit_report were extracted verbatim from .github/workflows/qwen-triage.yml at b0ae340b with a YAML parser (never retyped), driven under the step's own contract (bash --noprofile --norc, set -euo pipefail), and the output judged by GitHub's own renderer — POST /markdown, mode=gfm — not a local approximation.
| fixture | b0ae340b (the head the arms measured) |
cea325f79c (#8147 today) |
|---|---|---|
| HTML block × code span | 1 live <img>, 1 live <a href> |
0 / 0, <img escaped |
HTML block × code span × </details> |
1 live <img>, fold 2 opens / 3 closers |
0 / 0, 2 opens / 2 closers |
| HTML block × fence (control) | 0 / 0, <img escaped |
0 / 0 |
| LF fence closed with CRLF | @everyone renders as a live user-mention, no ZWSP |
0, falls back to escaped embedding |
| LF fence closed with LF (control) | 0, ZWSP applied | 0, ZWSP applied |
| benign report (control) | clean | clean |
Both numbers in the description land on the nose: "1 live <img> + 1 live <a href>" and "the fold escapes its wrapper (2 opens, 3 closers)", with the fence form returning 0 (the probe is not simply leaky) and the emit_block base arm escaping everything (the control is alive, not empty). The CRLF finding the treatment arm produced — LF-opened fence, CRLF close, [ \t]*$ cannot match the \r, inFence never closes while GitHub normalizes and renders the region as prose — reproduces as a live mention under the bot identity.
The right-hand column is the part worth keeping: at #8147's current head every one of those fixtures returns zero, including the two I built without reference to its tests. The defect these rules were distilled from is real, and it is closed.
5. Nits
- The ladder example can no longer be produced by the procedure it illustrates.
SKILL.md:216-223now says to run each rung undertimeout 30and record the cap as>30 s— and then illustrates with "0.96 s, 3.2 s, 14.4 s, then over 100 s". Under the rule as written that last cell reads>30 s. Either the example predates the cap and should say so, or the number should be restated. - The description's arithmetic still disagrees with itself, and it is the one artifact this PR is about: "Second batch — six more rules" vs "all ten rules in the second batch above" (ten is right), and
107/107twice where this head measures 116/116. "Twenty-three assertions added" is now exactly true. The body is maintainer-editable, so this is a two-minute fix outside the branch. - Scope, for the record: the
verifyjob re-pins.qwenfrom the base commit before the agent runs, so this text governs the PRs verified after it merges — including, deliberately, not this one.
Not covered
- I did not re-run the three-arm A/B. ~25 min / ~5 M tokens per arm against real
qwen3.8-max-previewis outside this round's budget. Everything above establishes that the anchor defect is real and that the assertions bite; none of it is evidence that the added prose changes an agent's behaviour. That claim still rests on the arms in the description alone, and the ten second-batch rules remain unvalidated by the body's own statement. - The base arm's
448 byteswas not re-derived — byte counts are fixture-dependent and my fixtures are not the arms'. I reproduced the property (0 live tags,<imgescaped), not the figure. - No dilution measurement. The file is 44,669 → 56,149 bytes; there is no mechanical cap on skill size anywhere in
packages/core/src/skills, so growth is a token-budget question, not a correctness one. Whether rule 请问这个 qwen3-coder-max 这个模型，从哪里找到？ 从 阿里云百炼平台 没找到这个模型 #27 displaces rule Are you interested in AI Terminal? #4 is unmeasured here, as it was in the arms. - The illustrative figures inside the new prose were not re-derived (the 1,898-character preamble bisect, the 30 verifier payloads, the 5→8 escaping artifact). A wrong figure misleads an agent but breaks nothing mechanical.
Methodology
git worktree at f86468f2bc, root node_modules symlinked. Teeth audit: a Node script parsing every expect(flat).toContain(...) literal (including the prettier-wrapped multi-line form) out of the head test file, bagged against the base file to isolate net-new strings, each matched for presence and multiplicity against both SKILL.md revisions under the test's own flattening. Mutation harness: programmatic edit of SKILL.md, full vitest run per mutation, file restored and git diff confirmed clean afterwards. Ground truth: yaml@2.9.0 parse of the workflow at each ref → verbatim function slices → bash --noprofile --norc → gh api -X POST /markdown -f mode=gfm, with the live-tag counts taken from the rendered HTML. Raw outputs and fixtures were kept in a local scratch directory outside the repo; the four screenshots above are those runs, not mock-ups.
中文说明
验证轮次 —— 在 f86468f2bc 上跑真实装置
在 head f86468f2bc（merge-base 912f7399）的干净 git worktree 中构建并运行，Linux、Node v22.22.2、root node_modules 软链入。以下全部是跑出来的，没有一条沿用自此前轮次。
结论：可合入。 各项门禁全绿；23 条新断言在正对照下都有牙；整个 PR 所依赖的那个实测事实，在本轮从零搭建的仪器上分毫不差地复现。一条低级别发现 —— 第 1–3 轮评审的三次修复各自都没有配断言，而这恰是本 PR 存在的意义所在的那条约定 —— 另有两条 nit。均不阻塞。
1. 门禁
vitest 269/269，覆盖读取该 skill 的四个套件（qwen-triage-workflow 116，另加 qwen-autofix-workflow、verify-capture、qwen-pr-review-workflow）；两个文件 prettier --check 干净；eslint --max-warnings 0 退出码 0。（截图见上文第 1 张）
2. 23 条断言都有牙
从 head 测试文件中解析出全部 expect(flat).toContain(...) 字面量，按测试自身的 \s+ → ' ' 归一后分别匹配 head 与 base 的 SKILL.md：base 27 → head 50，净新增 23 条，每条在 head 恰好命中一次、在 base 为零，且无删除。第 3 轮的"关键词回填"修复确实落地并成立 —— 'ladder'、'not the model-side degradation that produces it'、'Untrusted text reaching a parser is a scaling question' 均唯一命中，规则一旦被改写就会变红，而不会靠残句蒙混过关。（第 2 张）
3. [低] 三次评审回应的修复各自都未被钉住
我把本 PR 新增的规则反过来用在了本 PR 自己身上：反向变异，并配上这份 PR 现在自己要求的正对照。
| # | 对 SKILL.md 施加的变异 |
套件 |
|---|---|---|
| M0 | 无（对照） | GREEN 116/116 |
| M1 | 正对照 —— 改写一条 PR 钉住的规则（gets its siblings swept → gets its neighbours visited） |
RED 1 失败 / 115 通过 |
| M2 | 删除第 1 轮修复 —— uncalibrated 退出路径 |
GREEN 116/116 |
| M3 | 删除第 2/3 轮修复 —— timeout 30 预算上限 |
GREEN 116/116 |
| M4 | 撤销第 3 轮修复 —— 重命名 ### Scoping from the report and the plan |
GREEN 116/116 |
M1 正是让 M2–M4 值得采信的那一行：装置确实能让这个套件变红，因此那三个存活者是真的。grep 也印证：uncalibrated、timeout 30、is the evidence 出现在 SKILL.md 中，而在测试文件里一次都没有。
这就是第 1 轮的发现 3（"两条新指令未被钉住"），被每一个修复提交各重演了一次：为了闭环某条评审发现而新增的规范性文本，恰恰是不带 fixture 就进来的那一段。 三条字符串，均已验证在 head 唯一出现：
expect(flat).toContain('presenting it as calibrated is what is not allowed');
expect(flat).toContain('the rung that hits the cap is the evidence');
expect(flat).toContain('Scoping from the report and the plan');低级别、不阻塞 —— 但这是该文件自己的约定、也是本 PR 自己的第 5 条规则，代价只有三行。（第 3 张）
4. 描述里的 ground truth 分毫不差地复现 —— 且下游的环已经闭合
这些规则要站得住，前提是它们所锚定的失败是真的，所以我没有引用描述，而是在本地重新建立了一遍。从 b0ae340b 的 .github/workflows/qwen-triage.yml 中用 YAML 解析器逐字抽出 html_escape / emit_block / emit_report（绝不重打），在该 step 自身的契约下驱动（bash --noprofile --norc、set -euo pipefail），并用 GitHub 自己的渲染器判定输出 —— POST /markdown、mode=gfm —— 而不是本地近似实现。
| fixture | b0ae340b（各臂所测的 head） |
cea325f79c（#8147 当前 head） |
|---|---|---|
| HTML 块 × 代码跨度 | 1 个活 <img>、1 个活 <a href> |
0 / 0，<img 已转义 |
HTML 块 × 代码跨度 × </details> |
1 个活 <img>，折叠 2 开 / 3 闭 |
0 / 0，2 开 / 2 闭 |
| HTML 块 × 围栏（对照） | 0 / 0，<img 已转义 |
0 / 0 |
| LF 开启、CRLF 关闭的围栏 | @everyone 渲染为活的 user-mention，无 ZWSP |
0，回退到转义嵌入 |
| LF 开启、LF 关闭（对照） | 0，ZWSP 已施加 | 0，ZWSP 已施加 |
| 良性报告（对照） | 干净 | 干净 |
描述里的两个数字都精确命中："1 个活 <img> + 1 个活 <a href>" 与 "折叠逃出包裹层（2 开、3 闭）"；围栏形态返回 0（说明探针并非"什么都漏"），emit_block base 臂全部转义（说明对照是活的，不是空的）。处理臂产出的 CRLF 发现 —— LF 开启的围栏以 CRLF 关闭，[ \t]*$ 无法匹配行尾的 \r，inFence 永不关闭，而 GitHub 归一后按 prose 渲染 —— 复现为一次以机器人身份触发的活 mention。
右侧那一列才是最值得留下的：在 #8147 的当前 head 上，上述每个 fixture 都返回零，包括两个我在完全不参考它的测试的情况下自行构造的形态。这些规则所提炼自的缺陷是真的，而且已经闭合。（第 4 张）
5. Nits
- 阶梯的例子已经无法由它自己所示范的流程产生。
SKILL.md:216-223现在要求每一级在timeout 30下运行、把上限记为>30 s，紧接着却示范 "0.96 s、3.2 s、14.4 s，然后超过 100 s"。按新写的规则，最后一格只能读作>30 s。要么点明该例子早于这条上限，要么把数字改写。 - 描述的计数仍自相矛盾，而这恰是本 PR 的主题："第二批 —— 另外六条" 与 "第二批的全部十条" 并存（十是对的）；
107/107出现两次，而本 head 实测为 116/116。"新增 23 条断言" 现在是准确的。正文可由维护者直接编辑，这是分支之外两分钟的事。 - 顺带记录一下作用域：
verifyjob 在 agent 运行前会从 base commit 重新 pin.qwen，因此这份文本管的是它合入之后被验证的 PR —— 按设计，也包括不管它自己。
未覆盖
- 未重跑三臂 A/B。 每臂约 25 分钟 / 5M tokens（真实
qwen3.8-max-preview），超出本轮预算。以上全部工作确立的是"锚定的缺陷是真的"以及"断言确实咬得住"；没有任何一项能证明新增的散文改变了 agent 的行为。 那个论断仍然只靠描述中的各臂支撑，而第二批的十条按 body 自己的说法仍未验证。 - base 臂的
448 bytes未复核 —— 字节数依赖 fixture，而我的 fixture 不是各臂的那一套。我复现的是性质（0 个活标签、<img已转义），不是那个数字。 - 未测量稀释效应。 文件从 44,669 增至 56,149 字节；
packages/core/src/skills中不存在任何对 skill 体积的机械上限，因此增长是 token 预算问题，不是正确性问题。第 27 条会不会挤掉第 4 条，在这里和在各臂里一样未被测量。 - 新散文中引用的实测数字未复核（1,898 字符前缀的分界、30 份 verifier 载荷、5→8 的转义膨胀）。数字错会误导 agent，但不会破坏任何机械环节。
方法
git worktree 于 f86468f2bc，root node_modules 软链。断言审计：一个 Node 脚本解析 head 测试文件中全部 expect(flat).toContain(...) 字面量（含 prettier 折行后的多行形态），与 base 文件做多重集差以分离净新增字符串，再按测试自身的归一方式对两个 SKILL.md 版本逐条检查命中与重数。变异装置：以程序方式改写 SKILL.md，每个变异跑一次完整 vitest，事后还原并用 git diff 确认干净。ground truth：用 yaml@2.9.0 解析两个 ref 的 workflow → 逐字切出函数 → bash --noprofile --norc → gh api -X POST /markdown -f mode=gfm，活标签计数取自渲染后的 HTML。原始输出与 fixture 保存在仓库之外的本地临时目录；上面四张截图即这些运行本身，不是示意图。
🤖 Verified with Claude Code · model claude-opus-5[1m]
ReviewReviewed at head What I re-measured
The vacuity check was scripted rather than eyeballed: extract every Two design details of that pin are worth naming because they are easy to get wrong and this PR got them right:
Findings1. The ladder rule contradicts its own measured example. 2. One bullet in a section of paragraphs. 3. PR description inconsistencies. Cosmetic, but this PR's whole thesis is not asserting unmeasured things, so they read louder than they otherwise would:
Observations (not merge conditions)
VerdictThe mechanism is sound and the gates are green: 116/116, prettier stable, eslint clean, and all 23 new assertions demonstrably pin new text. Nothing here blocks merge. Finding 1 is a real internal contradiction in shipped guidance and is a two-word fix; finding 3 is description-only. Both are worth doing before merge because they are cheap and because this file is read as a specification for how to be careful. 中文摘要在 复测结果:测试 116/116 通过( 问题:
观察项(非合并条件): 结论:机制成立,门禁全绿,23 条新断言确实钉住新文本,无阻塞项。问题 1 是已交付指南内部的真实矛盾、改两个词即可;问题 3 仅涉及描述。两者都建议合并前处理——成本极低,而这份文件是被当作"如何做到严谨"的规范来读的。 |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (3)
scripts/tests/qwen-triage-workflow.test.js:2476
- This expectation matches only the first half of the sentence (it ends at "becomes a"), so the test would still pass if the key term ("finding") or the rest of the rule were accidentally removed or changed. Using the full phrase makes the assertion meaningfully pin the rule text.
// #8005 F3: two mutations survived 429/429 and 326/326 — believable
// only because a third, expected-caught mutation turned one test red.
// Without it, "no coverage" and "harness never ran" look identical.
expect(flat).toContain(
'A surviving mutation needs a positive control before it becomes a',
);
.qwen/skills/verify-pr/SKILL.md:66
- The follow-up status enum lists
fixed / stands / worsened / superseded / declined-with-rationale, but the next sentence refers to "deferred" rows. Sincedeferredis not one of the listed status values, the instructions are internally inconsistent and it’s unclear how a follow-up table should label deferred items.
a status table. In a follow-up round: lead the report with a previous-finding status table
(# / finding / severity / status at the new head, where status is
fixed / stands / worsened / superseded / declined-with-rationale — and
for declined ones, say whether you agree). Declined and deferred rows are
not exempt from re-measurement: a fix can move an accepted tradeoff, and
scripts/tests/qwen-triage-workflow.test.js:2390
- This assertion hard-codes the follow-up status enum string. If
deferredis included as a status (as implied by the adjacent SKILL.md text), this test will fail until it’s updated to match the documented enum.
This issue also appears on line 2471 of the same file.
expect(flat).toContain(
'fixed / stands / worsened / superseded / declined-with-rationale',
);
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
Reviewed at f86468f (started at 4e53037; the head moved twice mid-review, so this covers both follow-up commits). The ci-bot Suggestion is fixed and the round-3 commit is a genuine improvement on three axes I had checked and would otherwise have raised myself:
timeout <N>→timeout 30binds a placeholder the agent could not have resolved on its own, andis the finding→is the evidencefor a cap-hit is the more precise claim (the curve across rungs is the finding; one capped rung is one data point).- The two relocated bullets fix a broken cross-reference.
Rank a defect's variants by observabilitysays "the same ordering as the concurrency rule above" and now sits at line 368, immediately after the race bullet at :367 — before the move that "above" pointed into a different section.Prove a negative by censuslanded next to "count at the destination, not at the component boundary", which is the rule it is the counting-side mirror of.
On the pins I checked all 50 mechanically rather than by sampling — extracted every toContain literal, applied the same \s+ → ' ' flattening the test does, and confirmed 0 missing at this HEAD (that matters here: 'Calibrate the replay before believing it' spans a line break and only matches post-flattening). Three pins are under 18 chars — 'in both orders', 'A/A control', 'ladder' — and each occurs exactly once in the flattened skill, so none is loose. 'ladder' alone would be weak, but it is paired with the 53-character 'Untrusted text reaching a parser is a scaling question', which carries the rule's identity; the round-3 change traded a mid-length pin for a strong-plus-token pair, which is net stronger. No ** markers remain in any toContain literal.
Internal consistency: the new obligations are conditional on PR shape rather than unconditional work — the sibling sweep triggers on "a fix that closes one instance of a bug class", the ladder on "When the PR adds or changes a regex, tokenizer, or scanner that runs over input an outsider writes", the suggested-fix measurement only when a round produced a candidate patch. They live in the methodology sections, not in the priority-ordered budget list at 121-132, so "when time runs out, ship the report with what ran" plus the mandatory Not covered listing still govern, and the ladder carries its own cap in-line. No collision with the ≈110-minute budget and no contradiction with counts-drive-the-verdict: a both-green suggested fix routes to the unpinned-axis finding, not to a verdict change.
Embedded claims verified against the tree rather than trusted: scripts/verify-capture.mjs exists (and the text names the right fallback — declare it under Not covered, do not rebuild the pipeline by hand), and the node scripts/lint.js warning is accurate — the no-arg path really reaches runPrettier() → prettier --write ., which is why the skill directs the read-only lane at the individual --actionlint/--yamllint/--eslint flags.
On scope discipline, +183 lines of prose invites the Simplicity First question, so I looked for speculative additions: every added rule carries a measured example bound to a specific round (the code-span-in-raw-HTML sibling and list-nested fold from #8147 R6, the ladder timings, the SIGPIPE-misclassified log oracle, the prompt-instruction-is-not-an-invariant race, the coverage-spec enumeration). Each generalises an observed miss rather than guessing at a future one, which is the right side of the line for a skill whose failure mode was a rigorous run inside too small a frame.
Ran the suite locally: 116/116 at 4e53037, and 113/116 at this HEAD. The three failures are screens sponsored runs…, hosts only valid… PNGs, and survives symlink escapes… — all spawn-heavy tests that exercise qwen-triage.yml shell steps, and this PR touches only SKILL.md and adds test lines (+119/-0, no existing assertion modified), so the workflow they drive is byte-identical to main. Two pass standalone; the sponsored-run screen is borderline against its own 60s it() cap on this machine, which the flag override cannot lift. CI is fully green at this HEAD.
|
Released in v0.21.3. |
Problem
The
verify-prskill returnedmerge-ready, 63/63 assertions passed on a head that emits live HTML into a bot-authored comment. Not a weak run: it extracted the realemit_reportfrom the workflow YAML, drove it through realbash, killed 6/6 mutants, and ran the suite at 108/108. It was thorough inside a frame that never contained the bug.What was measured
Three arms against #8147 at
b0ae340b(the round-6 head whose code-span hole was already established by a hands-on round), realqwen3.8-max-preview, isolatedQWEN_HOMEper arm, identical merge worktree, metadata snapshot in the CI's own shape and without comments — the arms differ only by.qwen/skills/verify-pr/SKILL.md. Ground truth, scoring, and the decision rule were pre-registered before any arm ran.merge-ready63/63merge-ready254/254findings64 pass / 4 failThe decisive arm is the replicate, not the first control. It spent the same budget as the treatment arm and built the stronger instrument — a micromark + GFM
allowDangerousHtmlrendering oracle, 23 hostile fixtures, 254 assertions. It built09-htmlblock-fence(HTML block × fence) and it separately probed markers inside code spans. It never crossed the two axes. Re-running that arm's own oracle on the fixture it never built:09-htmlblock-fence— built<img>, 0 live<a href>→ passes<img>, 1 live<a href>→ fails its own guarantee 1So the miss was not capability, budget, or oracle quality. It was which fixtures the agent thought to construct — which is exactly what a rule can supply. The treatment arm's own words trace to the added text: "a parser, so sibling shapes need sweeping … the test only covers the fence variant."
The treatment arm also produced a finding the hands-on round had not made: an LF-opened fence closed with CRLF leaves
\ron the line, the close regex[ \t]*$cannot match it,inFencenever closes — while GitHub normalizes CRLF→LF per CommonMark, closes the fence, and renders the region as prose, so@everyonefires under the bot identity. Reproduced independently.Ground truth was re-established here rather than cited:
emit_report,emit_blockandhtml_escapeextracted verbatim fromb0ae340band driven underbash --noprofile --norcwith the step's ownset -euo pipefail. Head leaks 1 live<img>+ 1 live<a href>and the fold escapes its wrapper (2 opens, 3 closers); the fence form returns 0 (the probe is not simply leaky) and the base arm returns 0 with<imgescaped in 448 bytes (the control is alive, not empty).Change
Seven rules, each anchored to the failure it exists for.
A/B method — sweep the sibling shapes a parser-class fix does not pin (a fix's own new test pins the reported shape by construction); treat untrusted text reaching a parser as a scaling question and probe it with a timed ladder (2 k → 20 k), tracing the input back to a writer and checking the claimed escape hatch really excludes the path; audit a stated cost against the repo's own accounting of the same mechanism; enumerate the unnamed siblings of an accepted-tradeoff list.
Shared-state probing — an instruction in a prompt is not an invariant: when a safety property lives in a brief and the same change hands the resource to N concurrent agents, find the interleaving, and rank interleavings by what they produce, because a race that fabricates a result outranks one that crashes.
Vacuity — run the mutation in reverse: a suite green both with and without a candidate fix proves the suite pins nothing along that axis, and the report should name the fixture that would go red. And read a test's name as a claim about its fixture — not vacuity (the assertion can fail, the scenario runs), but a fixture that is not the shape the name promises.
Workflow/CI PRs — extract the step verbatim with a YAML parser, run it under the step's own shell contract, and calibrate before believing it: require the base arm to reproduce a real artifact the production step already emitted, naming the diffs allowed. A replay that cannot reproduce a known real output is measuring the harness, not the PR.
Report contract — a suggested fix is measured, not eyeballed (hostile fixtures clean, benign fixtures byte-identical, suite counts unchanged); the follow-up status enum gains
worsened, and declined/deferred rows are re-measured rather than carried.Second batch — six more rules, from three more hands-on rounds
Added after the arms above, from rounds that found real defects rather than from an A/B on this file:
/reviewcapability PR): treat untrusted text reaching a parser as a scaling question and probe it with a timed ladder; an instruction in a prompt is not an invariant, and interleavings that fabricate a result outrank ones that crash; a test's name is a claim about its fixture; audit a stated cost against the repo's own accounting of the same mechanism.run_shell_command, which the issue names explicitly. Rank a defect's variants by observability, not blast radius: a dropped argument is schema-rejected and loud, a truncated value is silent, and the silent one is the finding. And Not covered now asks whether a reproduction has the shape or the cause — that round replayed the reported wire bytes without reproducing the model-side degradation producing them.The measured artifact is not the shipped one
Stated because a reviewer had to derive it, and derived it low. The arms did not read either commit on this branch — the staged copies are
sha256:82dff804…, 7,633 words / 744 lines, matching no committed state:5538eaeca7a029179186f86468f2bc(autofix rounds since)So +1,335 words / +17.5% has been added to the instrument after it was measured, and ten rules post-date the arms — the four from #8215 landed in the first commit but after the arms had already run (
grepfor each of their four phrases against the staged arm file: 0 hits; against5538eaeca7: 1 each). The64/4cell is evidence about a file that will never run, and by this PR's own thesis — the prompt's content is the causal mechanism — that gap matters. It is not a claim that the extra words dilute anything; no such measurement exists.What is not validated
Stated plainly, because the point of the exercise was to stop asserting that prose works:
worsenedenum needs a follow-up round withprevious-report.mdpresent; this was a first round, so the scenario never occurred.Tests
Twenty-three assertions added (at
f86468f2bc; autofix rounds are still adding to this branch, so this count is pinned to that head rather than left floating) tocarries the maintainer-round verification techniques, each commented with the failure it pins, following the file's existing convention.scripts/tests/qwen-triage-workflow.test.js116/116 atf86468f2bc; prettier stable on a second--check; eslint--max-warnings 0clean.One defect this caught that no gate would: prettier reads a line-initial
>100 sas a blockquote and cascades>onto the nine following lines — in a file that is a prompt, a paragraph silently becomes a quotation. Only the diff shows it.中文说明
问题
verify-prskill 在一个会向机器人评论中注入活 HTML 的 head 上给出了merge-ready，63/63 断言全过。这并不是一次敷衍的运行：它从 workflow YAML 中抽出真实的emit_report、经真实bash驱动、6/6 变异全杀、套件 108/108。它在一个从未包含该 bug 的框架内做到了完备。实测了什么
三臂对照 #8147 @
b0ae340b（该 head 的 code-span 漏洞已由一轮实操验证确立），真实qwen3.8-max-preview，每臂独立QWEN_HOME，同一 merge worktree，元数据快照采用 CI 自身的字段形态且不含 comments——各臂之间仅.qwen/skills/verify-pr/SKILL.md不同。ground truth、判分标准与决策规则在任何一臂开跑前预先登记。merge-ready63/63merge-ready254/254findings64 过 / 4 挂决定性的是复现臂，不是第一个控制臂。 它花了与处理臂相同的预算，并且造了更强的仪器——micromark + GFM
allowDangerousHtml渲染预言机、23 个敌意 fixture、254 条断言。它建了09-htmlblock-fence（HTML 块 × 围栏），也单独探测了代码跨度内的标记伪造。它从未把这两个轴交叉。用该臂自己的预言机去跑它从未构造的那个 fixture：09-htmlblock-fence（已构造）<img>、0 个活<a href>→ 通过<img>、1 个活<a href>→ 击穿它自己的保证 1所以这次漏检与能力、预算、预言机质量都无关，而在于agent 想到去构造哪些 fixture——而这恰好是一条规则能够提供的东西。处理臂自己的措辞可直接溯源到新增文本："a parser, so sibling shapes need sweeping … the test only covers the fence variant."
处理臂还产出了那轮实操验证没有做出的发现：以 LF 开启、以 CRLF 关闭的围栏会在行尾留下
\r，闭合正则[ \t]*$无法匹配，inFence永不关闭——而 GitHub 按 CommonMark 将 CRLF 归一为 LF、关闭围栏、把该区域按 prose 渲染，于是@everyone以机器人身份触发。已独立复现。ground truth 在此重新建立，而非引用既有结论：从
b0ae340b逐字抽取emit_report、emit_block与html_escape，在bash --noprofile --norc下、施加该 step 自身的set -euo pipefail驱动。head 侧漏出 1 个活<img>+ 1 个活<a href>，且折叠逃出包裹层（2 个开启、3 个闭合）；围栏形态返回 0（说明探针并非"什么都漏"），base 侧返回 0 且<img被转义、输出 448 字节（说明对照是活的，不是空的）。改动
七条规则，每条都锚定它所对应的那次失败。
A/B 方法 —— 对 parser 类修复扫描它未钉住的邻接形态（修复自带的新测试按构造只会钉住被报告的那个形态）；把进入解析器的不可信文本当作规模问题，用计时阶梯（2 k → 20 k）探测，并把输入溯源到写入者、核实声称的豁免路径确实排除了该条路径；对 PR 声明的成本，对照仓库自身对同一机制的记账进行审计；枚举"已接受权衡"清单中未被点名的同胞。
共享状态探测 —— 写在 prompt 里的指令不是不变量：当安全属性寄居于一份 brief，而同一改动又把该资源交给 N 个并发 agent 时，去找出那个交错；并按产出对交错排序，因为伪造出结果的竞争比崩溃的竞争更危险。
空洞性检查 —— 反向执行变异：套件在打与不打候选补丁两种状态下都绿，恰恰证明该维度未被钉住，报告应点名那个本该变红的 fixture。以及把测试的名字读作对其 fixture 的一条声明——这不是空洞（断言能失败、场景也确实运行），而是 fixture 并非名字所承诺的形态。
Workflow / CI 类 PR —— 用 YAML 解析器逐字抽取该 step，在其自身的 shell 契约下运行，并在采信之前先校准：要求 base 臂复现生产 step 已经产出过的真实产物，并点名允许的差异。无法复现已知真实输出的重放，测量的是装置本身，而不是 PR。
报告契约 —— 建议修复必须实测而非目测（敌意 fixture 归零、良性 fixture 逐字节一致、套件计数不变）；follow-up 状态枚举新增
worsened，且 declined / deferred 行需重新测量而非直接沿用。第二批 —— 另外六条，来自三轮实操验证
在上述各臂之后加入，源自真实发现缺陷的验证轮次，而非对本文件的 A/B：
/review能力 PR）：把进入解析器的不可信文本当作规模问题，用计时阶梯探测；写在 prompt 里的指令不是不变量，且伪造出结果的交错比崩溃的交错更危险；测试的名字是对其 fixture 的一条声明；对 PR 声明的成本，对照仓库自身对同一机制的记账进行审计。run_shell_command，而 issue 恰恰点名了它。按可观测性而非影响面给缺陷变体排序：丢失参数会被 schema 大声拒绝，值被截断则是静默的，而静默的那个才是 finding。此外 Not covered 现在要求说明：你的复现拿到的是报文形态还是成因——那一轮复现了所报的链路字节，却没有复现产生它的模型侧降级。哪些未经验证
明确写出，因为这次工作的意义正是不再断言 prose 有效：
worsened枚举需要存在previous-report.md的 follow-up 轮，而本次是首轮，该场景从未发生。测试
在
carries the maintainer-round verification techniques中新增 23 条断言（测于f86468f2bc；autofix 仍在向本分支追加，故该计数锚定到该 head 而非悬空），每条都按该文件既有约定注明它所钉住的那次失败。scripts/tests/qwen-triage-workflow.test.js在f86468f2bc上 116/116；prettier 二次--check稳定；eslint--max-warnings 0干净。顺带抓到一个任何门禁都不会报的缺陷：prettier 会把行首的
>100 s读成引用块，并把>级联到其后九行——在一个本身就是 prompt 的文件里，一段文字就此静默地变成了引文。只有 diff 能看见它。