Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 24 additions & 10 deletions packages/cli/src/commands/review/run-skill-parity.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,7 @@ const repoRoot = resolve(
'..',
);

const SKILL_PATH = join(
repoRoot,
'packages/core/src/skills/bundled/review/SKILL.md',
);
const SKILL_DIR = join(repoRoot, 'packages/core/src/skills/bundled/review');

/** The `{target}` token, rendered per class exactly as the skill defines it. */
const TARGETS = {
Expand All @@ -46,14 +43,31 @@ const TARGETS = {
};

describe('run pins match the bundled skill templates', () => {
const skill = existsSync(SKILL_PATH)
? readFileSync(SKILL_PATH, 'utf8').replace(/\r\n/g, '\n')
// The skill is a corpus since #9787, and each oracle reads only the
// files the owning step is GUARANTEED to see: the composed-name template
// lives in the core SKILL.md (injected on every run), the report stems in
// references/persistence.md (the one reference loaded before Step 8 on
// every run that has a Step 8). Accepting a template from ANY corpus file
// would stay green on a move into a verdict-gated file that many runs
// never load, while those runs improvise artifact names.
const coreSkill = existsSync(join(SKILL_DIR, 'SKILL.md'))
? readFileSync(join(SKILL_DIR, 'SKILL.md'), 'utf8').replace(/\r\n/g, '\n')
: null;
const step8Corpus = (() => {
if (coreSkill === null) return null;
const persistence = join(SKILL_DIR, 'references', 'persistence.md');
return existsSync(persistence)
? `${coreSkill}\n${readFileSync(persistence, 'utf8').replace(/\r\n/g, '\n')}`
: null;
})();

// A sparse or partial checkout has no skill to read; the pins are still
// covered by run.test.ts's own cases. Failing here would report a checkout
// shape as a contract drift.
const itWithSkill = skill === null ? it.skip : it;
const itWithSkill = coreSkill === null ? it.skip : it;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R2-1: The sparse-checkout skip guard keys only on the existence of SKILL.md, but the Step 8 report-stems oracle now also reads references/persistence.md. A sparse/partial checkout that has SKILL.md but not that file runs the stems test instead of skipping it, and the test then fails as if the skill's templates had drifted: step8Corpus falls back to the core body, the stems regex finds zero stems (all three live only in persistence.md), and the arrayContaining assertion reports a checkout shape as a contract drift — the exact misattribution the guard's own comment above exists to prevent. Pre-split the same checkout passed because the stems lived in SKILL.md itself, so the split widens the set of checkout shapes that fail spuriously. A probe at the reviewed commit reproduced it: removing only persistence.md fails reportPatternFor accepts Step 8's report stems with expected [] to deeply equal ArrayContaining ["local", "pr-<number>", "<filename>"], and splitting the guard flips that to 1 passed | 1 skipped. Split the guard so the stems oracle skips unless persistence.md exists too, and use it for the stems test:

Suggested change
const itWithSkill = coreSkill === null ? it.skip : it;
const itWithSkill = coreSkill === null ? it.skip : it;
const itWithStep8 =
coreSkill !== null &&
existsSync(join(SKILL_DIR, 'references', 'persistence.md'))
? it
: it.skip;
中文说明

sparse-checkout 跳过守卫只以 SKILL.md 是否存在为条件,但 Step 8 的报告文件名干(report stems)oracle 现在还会读取 references/persistence.md。当稀疏/部分检出包含 SKILL.md 但缺少该文件时,stems 测试会照常运行而不是被跳过,然后以「技能模板发生漂移」的姿态失败:step8Corpus 回退到核心正文,stems 正则一个也匹配不到(三个名干全部只存在于 persistence.md),arrayContaining 断言会把检出形态误报为合同漂移 —— 这正是上方守卫注释本要防止的误归因。拆分前同样的检出是绿的(名干当时就在 SKILL.md 里),因此本次拆分扩大了会被误报失败的检出形态。探针在受审提交上复现:仅删除 persistence.mdreportPatternFor accepts Step 8's report stemsexpected [] to deeply equal ArrayContaining ["local", "pr-<number>", "<filename>"] 失败;拆分守卫后同一形态变为 1 passed | 1 skipped。建议把守卫拆开,让 stems oracle 在 persistence.md 不存在时也跳过,并在 stems 测试中改用新守卫。

— qwen3.8-max via Qwen Code /review (v0.22.0)

// The stems oracle additionally reads references/persistence.md, so a
// checkout that has SKILL.md but not that file must skip it too.
const itWithStep8 = step8Corpus === null ? it.skip : it;

itWithSkill('composedNameFor renders Step 6’s --out template', () => {
// The template as the skill writes it, e.g.
Expand All @@ -66,7 +80,7 @@ describe('run pins match the bundled skill templates', () => {
// class, so no future suffix character has to be foreseen.
const m =
/--out\s+\.qwen\/tmp\/(qwen-review-\{target\}-composed\.json)(?=\s|$)/.exec(
skill as string,
coreSkill as string,
);
// A null here means SKILL.md no longer writes that `--out` line: update
// composedNameFor and this oracle together to the new template.
Expand All @@ -78,11 +92,11 @@ describe('run pins match the bundled skill templates', () => {
}
});

itWithSkill('reportPatternFor accepts Step 8’s report stems', () => {
itWithStep8('reportPatternFor accepts Step 8’s report stems', () => {
// The stems as the skill lists them, e.g.
// `.qwen/reviews/<YYYY-MM-DD>-<HHMMSS>-pr-<number>.md`
const stems = [
...(skill as string).matchAll(
...(step8Corpus as string).matchAll(
/`\.qwen\/reviews\/<YYYY-MM-DD>-<HHMMSS>-([^`]+)\.md`/g,
),
].map((s) => s[1]);
Expand Down
11 changes: 11 additions & 0 deletions packages/core/src/skills/bundled-skills.integration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,4 +63,15 @@ describe('bundled SKILL.md files', () => {
fs.existsSync(path.join(datavizDir, 'references', 'anti-patterns.md')),
).toBe(true);
});

it('ships the review verdict-gated reference files with the bundled skill', () => {
const reviewDir = path.join(bundledDir, 'review');

expect(fs.existsSync(path.join(reviewDir, 'SKILL.md'))).toBe(true);
for (const name of ['posting.md', 'persistence.md', 'aone.md']) {
expect(fs.existsSync(path.join(reviewDir, 'references', name))).toBe(
true,
);
}
});
});
416 changes: 31 additions & 385 deletions packages/core/src/skills/bundled/review/SKILL.md

Large diffs are not rendered by default.

163 changes: 162 additions & 1 deletion packages/core/src/skills/bundled/review/SKILL.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,24 @@ const skillDir = path.dirname(fileURLToPath(import.meta.url));
const POINTER_RE = /\(measured; DESIGN\.md — ([^()\n]+(?:\([^()\n]*\))?)\)/g;
const POINTER_OPEN = '(measured; DESIGN.md — ';

function skillBody(): string {
// The verdict-gated reference files (#9787): Step 7, Step 8 and the Aone
// paths live beside the core body and are read on demand. The split moved
// whole sections verbatim, so every revert guard below governs the full
// corpus, whichever file the guarded text now lives in.
const REFERENCE_FILES = ['posting.md', 'persistence.md', 'aone.md'];

function coreBody(): string {
return fs.readFileSync(path.join(skillDir, 'SKILL.md'), 'utf8');
}

function referenceBody(name: string): string {
return fs.readFileSync(path.join(skillDir, 'references', name), 'utf8');
}

function skillBody(): string {
return [coreBody(), ...REFERENCE_FILES.map(referenceBody)].join('\n');
}

function incidentPointers(body: string): string[] {
return [...body.matchAll(POINTER_RE)].map(([, title]) => title.trim());
}
Expand Down Expand Up @@ -609,6 +623,42 @@ describe('bundled review skill', () => {
);
});

it('keeps the presubmit example on the host rule', () => {
// Revert guard: presubmit was the one Step 7 subcommand example
// missing the host flag; on an auth-config-only GHE clone a dropped
// `--host` routes its platform queries at github.com — the same
// failure class the meta pins above guard.
const body = skillBody();
expect(body).toContain(
'[--new-findings .qwen/tmp/qwen-review-{target}-new-findings.json] \\\n [--host <host>]',
);
});

it('pins the publish-assets weave as the last, all-or-nothing step', () => {
// Revert guard: `--findings-out` is written only after the push and
// the manifest succeed; without the clause the artifact's failure
// contract is unstated, and a mid-publish failure reads as a partial
// weave or a reason not to re-run.
const body = skillBody();
expect(body).toContain(
'the `--findings-out` rewrite runs only after every file has landed and the manifest is written',
);
expect(body).toContain(
'a run that fails partway through the push is completed by an idempotent re-run',
);
});

it('names the deferral channel in the bodyCriticals sources', () => {
// Revert guard: compose-review relocates a `Critical` entry written
// into `deferredSuggestions` into the body Criticals (a Critical is
// never deferred); the bodyCriticals bullet must name that mechanical
// relocation beside the two model-written sources.
const body = skillBody();
expect(body).toContain(
'a `Critical` entry placed in `deferredSuggestions` is relocated here, never deferred',
);
});

it('keeps the lightweight capture on fetch-diff with the plan-diff host note', () => {
// Revert guard: restoring a prose `gh pr diff > file` here (or dropping
// the plan-diff --host note) must fail a test, not slip through — the
Expand Down Expand Up @@ -845,6 +895,17 @@ describe('bundled review skill', () => {
expect(body).not.toContain(
'`pr-context` and `comment-status` have no Aone backing',
);
// The last three skip residues this change removes — the setup-batch
// parenthetical, the comment-status guard clause, and the Step 6
// no-report clause. The positive assertions above stay green if a
// merge resolution or partial revert re-adds any of them, while Aone
// runs skip comment-status again; the replacement contract is the
// a1-backed report's existence in Step 6's re-check.
expect(body).not.toContain('drops out of the batch');
expect(body).not.toContain('leaving a two-call batch');
expect(body).not.toContain('the command has no backing');
expect(body).not.toContain('skips the command with the Step 1 batch');
expect(body).toContain('on an Aone target it runs a1-backed');
});

it('keeps the corrected Aone --comment contract, not merge residue', () => {
Expand Down Expand Up @@ -951,4 +1012,104 @@ describe('bundled review skill', () => {
);
expect(new Set(advertised)).toEqual(new Set(declared));
});

it('ships the verdict-gated reference files beside the core body', () => {
// The split (#9787) moves whole steps, not rules: the core keeps the
// gates and the invariants that bind runs which never load a file, and
// each reference owns one conditional territory.
for (const name of REFERENCE_FILES) {
expect(referenceBody(name).length).toBeGreaterThan(1000);
}
expect(referenceBody('posting.md')).toContain('# Step 7: Submit PR review');
expect(referenceBody('persistence.md')).toContain(
'# Step 8: Save review report and cache',
);
expect(referenceBody('aone.md')).toContain('# Aone Code paths');
});

it('gates every reference file on the verdict in the core body', () => {
// A run must learn from the injected core alone WHICH file to read and
// when; a gate that moved into the file it gates would be unreadable.
const core = coreBody();
expect(core).toContain('**Reference files, gated by this verdict.**');

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-8: This test claims in its name and comment to pin "WHICH file to read and when", but its assertions stop at the file-enumeration prefixes — none of the three load-condition clauses in the core gate list (SKILL.md:78-80) is asserted by any test in the repo (grepped: zero hits for "posting is live", "cross-repo lightweight", "GitHub runs never", etc.). A probe confirmed the gap: rewriting posting.md's bullet to Load it before Step 7 on every run. with the enumeration prefix intact passes 35/35 — the mutant survives; adding toContain pins for the condition clauses fails the mutant and passes on the pristine corpus. The trigger is not hypothetical for this file — merge residue demonstrably happens here (the sibling 'not merge residue' test exists because of one), and the gate conditions are the mechanism this PR introduces.

Suggested change
expect(core).toContain('**Reference files, gated by this verdict.**');
expect(core).toContain('**Reference files, gated by this verdict.**');
expect(core).toContain('Load it when, and only when, posting is live');
expect(core).toContain('Load it before Step 8 on every run except cross-repo lightweight mode');
expect(core).toContain('Load it before `match-remote` when the target is Aone');
中文说明

该测试的名称与注释声称要钉住「读哪个文件、何时读」,但其断言止步于文件枚举报头 —— 核心门控表(SKILL.md:78-80)中三个加载条件从句,整个仓库没有任何测试断言(grep 证实:"posting is live"、"cross-repo lightweight"、"GitHub runs never" 等均零命中)。探针确认了缺口:把 posting.md 的条件从句改写为 Load it before Step 7 on every run.(保留枚举报头)仍通过 35/35 —— 变异体存活;为条件从句补上 toContain 钉住后,变异体失败、原始语料通过。对本文件而言这并非假想触发 —— 合并残留在这里有实锤先例(姊妹测试『not merge residue』正是因此存在),而门控条件正是本 PR 引入的机制。

— qwen3.8-max via Qwen Code /review (v0.22.0)

// Pin each enumeration prefix together with its load-condition clause
// as ONE contiguous substring: checked separately, a rewrite that swaps
// two clauses between bullets ships green while a report-only run loads
// the wrong file. The gating is the mechanism this split introduces.
expect(core).toContain(
'`references/posting.md` — Step 7 (authorisation, anchors, presubmit, `submit`, the 422/head-drift recovery, `publish-assets`). Load it when, and only when, posting is live',
);
expect(core).toContain(
'`references/persistence.md` — Step 8 (report, artifact registration, incremental cache). Load it before Step 8 on every run except cross-repo lightweight mode',
);
expect(core).toContain(
'`references/aone.md` — the Aone paths (see the Aone note below). Load it before `match-remote` when the target is Aone',
);
});

it('keeps the write prohibition and the posting gates in the core body', () => {
// The one-sentence write ban and the PR-only/high-only posting rule must
// bind a run that never loads posting.md — the bypass they guard against
// does not wait for the gate file.
const core = coreBody();
expect(core).toContain(
'`qwen review submit` is the only write path in this skill',
);
expect(core).toContain('Posting is a PR-only, high-only action');
// The step headings stay in core so every "Step 7" / "Step 8" cross-
// reference in the corpus resolves to the pointer that forwards.
expect(core).toContain('## Step 7: Submit PR review');
expect(core).toContain('## Step 8: Save review report and cache');
// The compose-state field list relocated to Step 6 references the
// never-in-body rule whose full text moved to posting.md; the entry must
// restate the rule's substance so a report-only run (which never loads
// posting.md) still sees why a Suggestion must not ride the review body.
expect(core).toContain('does not filter review bodies');
});

it('moved the sections whole — no step body duplicated across files', () => {
const core = coreBody();
const corpus = skillBody();
// Distinctive openings of the moved sections: present in exactly one
// file of the corpus, and absent from the core. The corpus-wide count
// alone would pass a revert that keeps a section in the core, and the
// absence-from-core alone passes a copy duplicated BETWEEN the
// reference files — an Aone --comment run loads both posting.md and
// aone.md, so one run would then obey two potentially divergent
// copies of the same step.
expect(corpus.match(/\*\*Use the "Create Review" API/g)).toHaveLength(1);
expect(corpus.match(/### Report persistence/g)).toHaveLength(1);
expect(
corpus.match(/run `\/review` \*\*from inside a clone of that repo\*\*/g),
).toHaveLength(1);
expect(core).not.toContain(
'**Use the "Create Review" API to submit verdict + inline comments',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R1-5: This test's name and comment promise the moved sections are "present in exactly one file", but the three section-opening pins assert absence from the CORE only — duplication between the three reference files passes. Only the separate modelId pin enforces corpus-wide uniqueness (toHaveLength(1)). A probe confirmed the gap: appending the "Create Review" opening to aone.md (so it exists in both aone.md and posting.md) passes 35/35 as shipped; corpus-wide toHaveLength(1) pins fail the mutant and pass on the pristine corpus. The harm path is real: an Aone --comment run loads both posting.md and aone.md (core gate block, SKILL.md:76-81), so a duplicated step body governs one run in two potentially divergent copies — and this PR's own head merge committed exactly this residue shape once (the sibling 'keeps the corrected Aone --comment contract, not merge residue' test exists because of it).

Promote the three pins to the modelId shape over skillBody():

const corpus = skillBody();
expect(corpus.match(/\*\*Use the "Create Review" API/g)).toHaveLength(1);
expect(corpus.match(/### Report persistence/g)).toHaveLength(1);
expect(corpus.match(/run `\/review` \*\*from inside a clone of that repo\*\*/g)).toHaveLength(1);

(optionally keeping the not.toContain(core) halves).

中文说明

该测试的名称与注释承诺被搬迁的小节「恰好存在于一个文件中」,但三个小节开头钉住只断言了核心正文中的缺席 —— 三个参考文件之间的重复可以通过。只有独立的 modelId 钉住通过 toHaveLength(1) 强制全语料唯一。探针确认了缺口:把 "Create Review" 开头追加到 aone.md(使其同时存在于 aone.md 与 posting.md)按现状仍通过 35/35;改为全语料 toHaveLength(1) 钉住后变异体失败、原始语料通过。危害路径真实存在:Aone --comment 运行会同时加载 posting.md 与 aone.md(核心门控块,SKILL.md:76-81),重复的步骤正文会让同一次运行受两份可能分叉的副本约束 —— 而本 PR 自己的头部合并就曾提交过完全相同的残留形态(姊妹测试『keeps the corrected Aone --comment contract, not merge residue』正是因此存在)。

— qwen3.8-max via Qwen Code /review (v0.22.0)

);
expect(core).not.toContain('### Report persistence');
expect(core).not.toContain(
'run `/review` **from inside a clone of that repo**',
);
// The compose-state field list relocated from Step 7 to Step 6's Verdict
// section: one copy in the corpus, in the core.
expect(corpus.match(/- `modelId` — for the footer\./g)).toHaveLength(1);
expect(core).toContain('- `modelId` — for the footer.');
});

it('keeps template tokens out of the raw-loaded reference files', () => {
// BundledSkillLoader interpolates only the core body it injects; the
// reference files are read raw via read_file, so a token there reaches
// the run unreplaced: a literal `(v{{cliVersion}})` draft footer is
// one stripReviewFooter cannot match (the version span excludes
// braces), so every posted comment carries the broken token above the
// canonical footer, and a `{{model}}` copied into the cache JSON
// fails the next round's same-model anchor gate.
for (const name of REFERENCE_FILES) {
expect(referenceBody(name)).not.toMatch(/\{\{[^}]+\}\}/);
}
// The reference files' footer templates name YOUR_MODEL_ID, whose
// value the loader prepends to the injected core body — but only when
// the core body carries a model token; without one the declaration
// vanishes and the templates dangle.
expect(/{{model}}|YOUR_MODEL_ID/.test(coreBody())).toBe(true);
});
});
Loading
Loading