Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions docs/design/2026-08-27-review-critical-axes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Review: Critical direction/baseline axes and the convergence floor (#10291)

Date: 2026-08-27
Issue: #10291 — `Critical` is one bit carrying three axes; add
direction/baseline fields so the convergence floor can actually floor.

## Problem

The convergence posture (`floor: critical`, from round 6 or on the
flat-trend signal) is the review loop's only damper, and on #9659 it stopped
damping at round 17: new Criticals per round ran 7, 5, 3, 5, 4, 1, 6 over
rounds 17–23 with near-zero false positives. The floor could filter nothing
because the `Critical` class collapsed three orthogonal merge decisions into
one bit — which way the defect fails, what it is measured against, how often
it triggers. Of the ~25 findings the issue classified (31 new in all across
the seven rounds), about 6 certified falsely (a wrong result presented as
correct); the other ~19 were fail-closed corners on defenses
added in earlier rounds, most of them zero-regression against the merge base
(before the PR, a sparse checkout had no incremental review either).

## Design

### Two fields, one deferral shape

Findings gain two optional enum fields, stated by the Step 4 verifier off the
witness that confirmed the finding:

- `direction`: `certifies-falsely` (the code produces a wrong result it
presents as correct — a wrong output, a silent corruption, a bypassed
check, a decision over state nobody read) | `fails-closed` (refuses,
wedges, crashes or degrades to its own absence without a wrong result).
- `baseline`: `regression` (the merge base handles the trigger correctly; the
change breaks it — the A/B's base arm is correct) | `new-surface` (the
failing path does not exist at the merge base — the change adds the
feature, defense or branch the defect lives in).

At a floor resolved to `critical` (explicit, `auto` from round 6, or `auto`
with `flatRounds` at its bar), **a Critical that is `fails-closed` AND
`new-surface` is recorded, not requested** — a typed deferral like a
Suggestion's. Every other Critical posts: `certifies-falsely` at either
baseline, `regression` in either direction, and any Critical with a missing
or self-contradicting axis. The rounds-2–5 code-age rule never touches a
Critical. The issue's optional third axis (trigger frequency) is not
carried.

Replaying #9659 under the rule: the ~6 certification-direction findings
still block, the ~19 fail-closed corners route to follow-up, and the loop
plausibly settles around round 19 instead of 23.

### Carriers

| Surface | Form | Reader |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| Findings artifact (`qwen review findings`) | `direction` / `baseline` enums, refused when misspelled | report, `report_findings`, the compose state |
| `report_findings` tool (core) | same optional enums in the schema | TUI / Web Shell / ACP display (pass-through) |
| Drafted Critical comment | claim-line bracket tags `[fails-closed] [new-surface]` beside the id, copied from the artifact | `floorEnforcedReroute` (backstop), `buildLedger` (marker) — claim line only |
| Typed deferral entry (`deferredSuggestions`) | `severity: "Critical"` + both axes | `splitDeferralChannel` — deferred only when licensed |
| Ledger marker | `d: 'c' \| 'f'`, `b: 'r' \| 'n'` per finding; unknown values normalise to absent | next round's side-file work-list table (`Critical (fails-closed, new-surface)`), autofix tooling (#9907) |

### The licence

The Critical split reads the ENFORCEMENT resolution of the floor
(`criticalFloorInEffect`), computed once per compose and threaded to the
ledger build, the body composer and the mechanism-health check. A Critical
entry arriving when the floor is not in effect is relocated into the body
Criticals and posts — the fail-toward-posting direction every arm of the
floor takes. A deferred Critical never caps, never withholds the anchor,
leaves the work list, and — on its first deferral — joins the closure mint
by the id its title carries, so the successor-chain sentinel does not read
it as a fix. A re-deferral bears an id the previous posted work list never
Comment thread
wenshao marked this conversation as resolved.
Resolved
held, fails the mint's membership leg, and that round's mint fails closed
(absence never reads as a fix, at the cost of that round's true closures);
carrying the round's deferred ids in the marker would let a re-deferral
join by them and is left for a follow-up.

### Trust boundaries

- Tags are read off the claim line's head slot only — the machine tokens
before the title, like `[probe]`: a title that merely quotes a tag is
prose, the body's tail is writable surface, and a forged pair in either
would otherwise defer a drafted Critical.
- An axis carrying both of its tags reads as unclassified — the backstop
never guesses a blocker out of the posting set.
- The marker fields decide nothing in code; a spelling this version does not
know is normalised away (like the stand-in flag `k`), never used to reject
the entry.
- The verifier is told to omit an axis its witness cannot settle. The
artifact and the typed channel refuse a misspelled axis rather than
silently reading it as absent (which would post a deferrable blocker
without anyone seeing the drop).

### Disclosure

- The deferral list line names a deferred Critical and its tags:
`src/sparse.ts:12 — [review] Critical [fails-closed] [new-surface] …`.
- The deferral header adds why: "N Critical(s) among them are deferred by
their axes — fails-closed on new surface, …". Bilingual.
- The floor-enforcement note and `submit`'s stderr line count moved
Suggestions and moved Criticals separately.
- The verdict line says "N finding(s) deferred" (was "non-Critical
finding(s)").
- The relocation note reads "a Critical is deferred only as fails-closed on
new surface at a critical floor; this one posts".

## Implementation outline

1. `core/tools/report-findings.ts`: `FINDING_DIRECTIONS`, `FINDING_BASELINES`,
schema fields, pass-through; `ReportedFinding` type; core exports.
2. `cli/commands/review/findings.ts`: `DIRECTIONS`/`BASELINES`, `Finding`
fields, strict validation, round-trip, terminal render.
3. `cli/commands/review/lib/ledger.ts`: `LedgerFinding.d`/`.b`, normalisation,
`axesOf` renderer.
4. `cli/commands/review/compose-review.ts`: axis-tag readers, typed-entry
axes, licensed split, Critical reroute arm (seam counts adjusted per
severity), ledger stamping via `readClaim`, disclosures, verdict wording,
`deferrableFindingsInline`.
5. `submit.ts` (stderr wording), `pr-context.ts` (work-list table),
`lib/agent-briefs.ts` (verify brief states the axes).
6. SKILL.md / posting.md (the model-side contract), DESIGN.md.

## Explicitly not done

- **No trigger-frequency axis.** Named optional in the issue; a "corner" is a
judgement no witness settles.
- **No auto-filed issues.** `submit` is the only write path; a deferred
Critical's record is the posted deferral line and the artifact's
`D<round>-<n>` entry, and filing the follow-up issue stays with the author.
- **No Web Shell rendering.** The renderer ignores the new fields; nothing
it displays is decided by them.
2 changes: 1 addition & 1 deletion docs/users/configuration/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ Settings are organized into categories. Most settings should be placed within th
| `review.attribution` | boolean | Append the attribution footer naming the model and CLI version (e.g. `_— qwen3-coder via Qwen Code /review (v0.21.2)_`) to review bodies and inline comments posted by `/review`. Disable to post reviews without visible AI attribution: the footer is omitted and posted comments and body lists lose their `**[Critical]**`/`**[Suggestion]**` markers. The posts stay identifiable in the raw source: each carries an invisible severity marker (`<!-- qwen-review critical -->`) and the review body carries a ledger marker (`<!-- qwen-review-ledger ... -->`) — anything reading comment bodies (GitHub API automation, the workflows this setting couples to) still recognizes a `/review` artifact, and presubmit duplicate detection recognizes the reviewing account's earlier posts by the severity marker, though unattributed posts from other accounts escape it. Another consequence: qwen-autofix's Critical-only mode (engaged after round 5, or earlier when a counting window's diff-growth budget trips) no longer recognizes the posted findings as Critical and defers them. Disabling also withholds the model from the machine-ledger marker embedded in the review body, so in a fresh environment (CI, another clone — anywhere without a review cache) the incremental anchor recovered from the last posted review fails the same-model check and the re-review falls back to full-range. | `true` |
| `review.effort` | enum | Default effort for `/review` when neither `--effort` nor a project-remembered explicitly typed level applies: `"low"`, `"medium"`, `"high"`, or `"auto"` (the built-in rule: high for PRs, medium for local changes). An explicit or remembered level wins; an effective `--comment` still forces high and `--fix` still floors at medium. | `"auto"` |
| `review.comment` | boolean | Treat every PR `/review` as if `--comment` was passed: findings are posted to the pull request without the flag. The post still binds to the PR named in the invocation. Enable only if you always want reviews published. | `false` |
| `review.severityFloor` | enum | The lowest severity a PR `/review` posts when `--severity-floor` is not given: `"auto"` (the round-adaptive default — Suggestions post through round 5, only Criticals from round 6, with otherwise-postable high-confidence Suggestions recorded and deferred, and rounds 2–5 deferring new Suggestions on code unchanged since the previous round; low-confidence and Nice-to-have findings stay terminal-only), `"critical"` (that posture from round 1), or `"suggestion"` (Suggestions post at every round; turns the convergence posture off). Non-PR targets have no rounds and ignore this. | `"auto"` |
| `review.severityFloor` | enum | The lowest severity a PR `/review` posts when `--severity-floor` is not given: `"auto"` (the round-adaptive default — Suggestions post through round 5, only Criticals from round 6, with otherwise-postable high-confidence Suggestions — and a Critical classified fails-closed on new surface — recorded and deferred, and rounds 2–5 deferring new Suggestions on code unchanged since the previous round; low-confidence and Nice-to-have findings stay terminal-only), `"critical"` (that posture from round 1), or `"suggestion"` (Suggestions post at every round; turns the convergence posture off). Non-PR targets have no rounds and ignore this. | `"auto"` |
| `review.reverseAuditRounds` | number | Lower the reverse-audit loop's round cap for every high-effort review. The cap otherwise follows the diff topology (10 small / 5 chunked; a huge diff is 3 with a review deadline and 5 without). This can only **lower** whichever tier applies: a value below 3, above the tier, or not a whole number above zero is ignored. Cutting the cap does not make reviews converge sooner — the loop ends on two consecutive dry rounds — it makes them stop before converging more often, and every such stop caps the verdict at Comment. | `0` (unset) |

These settings are read from operator scopes only (User, System, and SystemDefaults); values in a workspace `.qwen/settings.json` are ignored, so a repository cannot set review policy for its reviewers.
Expand Down
20 changes: 20 additions & 0 deletions packages/cli/src/commands/review/agent-prompt.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -626,6 +626,26 @@ describe('agent-prompt (command boundary)', () => {
expect(briefText).toContain('A confirmed Critical returns its witness.');
expect(briefText).toContain('witness: not run —');
expect(briefText).toContain('sweep the real population');
// The two decision axes (#10291) ride the same witness: the brief
// defines both values of each, ties the routing consequence to the
// ONE combination the floor defers, and tells the verifier to omit
// rather than guess — a guess on either axis completes the pair and
// takes a blocker off the pull request.
expect(briefText).toContain(
'A confirmed Critical also returns its two decision axes',
);
for (const value of [
'direction: certifies-falsely',
'direction: fails-closed',
'baseline: regression',
'baseline: new-surface',
]) {
expect(briefText).toContain(value);
}
expect(briefText).toContain(
'only a Critical that is both fails-closed and new-surface is recorded as a deferral',
);
expect(briefText).toContain('OMIT that line rather than guess');
} finally {
rmSync(dir, { recursive: true, force: true });
}
Expand Down
Loading
Loading