Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/review-trial-protocol-hygiene.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"review": patch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question (non-blocking): Does a doc outside workflows/review/ warrant a review: patch bump? The skill lives at repo-level .claude/skills/review-trial/, outside the review package, and the changeset's own closing line says "No change to the shipped review workflow" — yet this bump cuts review-v1.9.1 and rewrites the pinned review-v<semver> literal in review.md. If the changeset gate only needs a file to exist, pnpm changeset --empty lands the lessons without minting a contentless release; keep the patch bump only if trial docs are deliberately versioned with the package.

---

Skill-doc only: record five operational lessons from the 2026-07 preview trials in the `review-trial` skill. (1) Never force-push a trial branch between rounds: a round that cannot anchor on a prior fingerprint reports `stampSource: null` in `out/rereview-plan.json` and executes `depth: full` with `staging: whole-diff` even where ROUTING says `scoped`, which is indistinguishable from a genuine carrier gap and silently voids that round's scoped-cost sample. `stampSource` in the plan artifact is the authoritative anchoring signal and must be read every round; the carrier internals that make a rebase the wrong suspect (content-hashed hunk signatures, a `commitSha` that is written but never read back) are dated implementation properties, not contracts. (2) Use a fresh lock filename per trial: GitHub's workflow registry keys the display name to the file path and keeps the stale one, so two trials that reused `review-preview.lock.yml` both listed as "PR Reviewer" while the files declared "PR Reviewer Preview v1.8.0" and "...v1.9.0". Concurrency was unaffected, but a run list naming every arm after the production reviewer invites the "did production just run?" scare the naming rule exists to prevent. (3) A lifecycle plan that means to exercise open-thread suppression must leave at least one finding unfixed, since a push where the fixer repairs everything lets the reconciler resolve every thread and `threads.json` comes back empty. (4) Count runs by the EVENTS a step fires, and prefer resuming a closed trial PR over building a new lifecycle: a closed PR fires no `pull_request` event (pushes are free), `reopened` is in the reviewer's trigger list (exactly one run), and pushing to an open PR fires `synchronize` (one more each time), so push-while-closed then reopen buys a round for one run. A closed PR whose last round's comments are still unresolved already IS a partially-fixed round; webapp#41204 was reopened that way for the round that made open-thread suppression fire in production for the first time (run 30650642317), which had been budgeted at two runs. Unresolving a thread is a GraphQL mutation an agent's tooling may refuse, so it belongs to the operator. (5) Keep fault injections out of the diff under review: mis-staging to trip a fail-open reporter works (run 30654454047 reported `unusableThreads: 9`), but injecting via the prompt or lock file on the trial branch makes that edit the newly-changed code, so all five of that run's candidates landed on the prompt file and none on the fixture, the duplicate re-posting the reporter warns about went unobserved, and the reviewers spent the round correctly flagging the injected prose as agent-directing instructions. The cleanup step gains the corollary the closed-PR shortcut requires: a branch that outlives its PR is reverted to a safe state and its resumable details (unresolved threads, run IDs, what a reopen fires) go in the closing comment, since a surviving branch carrying a planted panic is a trap and a reopen reviews immediately. No change to the shipped review workflow.
93 changes: 90 additions & 3 deletions .claude/skills/review-trial/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,11 @@ missing, stop and ask; do not improvise a defect table from the branch diff.
tables isolate the dial's recall and dollar deltas.
4. **Lifecycle plan** (optional): the push-2 content (fixes mixed with fresh
seeds, plus their defect-table rows) and the push-3 content (everything
fixed).
fixed). If the plan means to exercise **open-thread suppression**, it must
leave at least one finding unfixed: a push where the fixer repairs
everything lets the reconciler resolve every thread, `threads.json` comes
back empty, and suppression has nothing left to suppress, so the round
proves nothing about it either way.
5. **Budget approval**: project the cost FIRST and confirm. Project in the
units the cap enforces: the firewall api-proxy's credit meter
(`ai_credits_this_response` sums in the run's token-usage log), NOT the
Expand Down Expand Up @@ -78,7 +82,17 @@ Per-arm trigger setup:

- `repo-default`: nothing extra; the repo's reviewer triggers normally.
- `workflow @ ref`: commit the compiled workflow for that ref onto the
scaffolding branch. **Give it a workflow name distinct from the repo's
scaffolding branch, under a **fresh lock filename per trial** (e.g.
`review-preview-v1-9-0.lock.yml`, not a reused `review-preview.lock.yml`).
GitHub's workflow registry keys the display name to the file PATH and keeps
the stale one: two consecutive trials that reused one filename both showed
up in the Actions list as "PR Reviewer" while the files on the branches
declared "PR Reviewer Preview v1.8.0" and "...v1.9.0". Nothing about
concurrency broke (the groups were distinct, and the production workflow
recorded zero runs on either branch), but a run list that names every arm
after the production reviewer is unreadable and invites exactly the "did
production just run?" scare the naming rule exists to prevent.
**Give it a workflow name distinct from the repo's
own reviewer and from every other arm**: same-named gh-aw workflows share
a per-PR concurrency group and silently cancel each other (this ate a run
in the original trial). Its `if:` must exclude the scaffolding branch
Expand Down Expand Up @@ -130,6 +144,62 @@ let each arm re-review, and collect again; repeat for push-3. Score each push
separately: re-review behavior (thread resolution, duplicate suppression,
scoping to new hunks) is half the point of the lifecycle.

**Never force-push a trial branch between rounds.** Every push after the first
review is forward-only; rebase, squash, or re-parent the stack BEFORE the first
arm reviews it. The reason is measurement integrity, not correctness: a round
that cannot anchor on a prior fingerprint reports `stampSource: null` in
`out/rereview-plan.json` and executes `depth: full` with `staging: whole-diff`
even where ROUTING says `scoped`, which is visually indistinguishable from a
genuine carrier gap and silently voids the scoped-cost sample that round was
supposed to produce. `stampSource` in that artifact is the authoritative answer
to "did this round anchor?"; read it every round rather than inferring the
carrier state from the shape of the push, and discard scoped-cost samples from
any round that did not anchor.

The carrier internals below explain why a rebase is usually the wrong suspect,
but they are incidental properties of today's implementation rather than
contracts, so do not reason from them in place of the artifact. As of 2026-07:
the hunk signature is content-hashed over each hunk's added and removed lines,
so SHA rewrites and shifted line numbers do NOT move it
(`workflows/review/lib/rereview-mode.ts`), and the cache record's `commitSha`
is written but never read back. On that basis a rebase alone should not refuse
the carrier, and what actually loses the anchor is losing the RECORD (cache
eviction, or a credit-capped run that died before Step 9 wrote it); a
force-pushed round is exactly the round where that goes unnoticed and gets
blamed on the rebase. If a future change starts validating `commitSha` on
restore, this paragraph is what goes stale, not the `stampSource` rule above.

**A closed trial PR is usually a cheaper source of the round you need than a
new lifecycle.** Before building a two-run push-2, check whether a previous
trial already left the shape lying around: a CLOSED PR whose last round's
comments are still unresolved is a partially-fixed round waiting to be
resumed, because its findings were never addressed. webapp#41204 was reopened
this way to get the round that made open-thread suppression fire in production
for the first time (run 30650642317: six threads kept, six suppressions,
REQUEST_CHANGES floored by the suppression floor alone), which the plan had
budgeted two runs for and got in one.

Control the run count through the events, not through hope. On a closed PR no
`pull_request` event fires at all, so pushes are free; `reopened` IS in the
reviewer's trigger list, so reopening fires exactly one run; and pushing to an
already-open PR fires `synchronize`, which is one more run each time. The
sequence that costs a single run is therefore: push the fixture state while the
PR is still closed, adjust thread resolution, then reopen. Unresolving a thread
is a GraphQL mutation (`unresolveReviewThread`) that an agent's tooling may
refuse; hand it to the operator rather than working around it.

**Fault injection: keep it out of the diff under review.** Exercising a
fail-open reporter by deliberately mis-staging works (run 30654454047 tripped
`threadSuppressionUnavailable` with `unusableThreads: 9` by telling the
orchestrator to omit `resolved`), but if the injection edits the prompt or lock
file ON the trial branch, that edit becomes the newly-changed code: every one
of that run's five candidates landed on `review-preview.md` and none on the Go
fixture, so the reporter fired while the duplicate re-posting it warns about
went unobserved, and the reviewers spent the round flagging the injected prose
as blocking (correctly: it reads as instructions aimed at the agent). Land the
injection in the same push as the fixture change you want re-reviewed, or force
full depth, and expect the injected text itself to draw findings.

When an arm runs a reduced re-review mode (`re-review` in ROUTING), also
record per push: the executed depth and tripwire fields from the run's
`out/rereview-plan.json` artifact, and the billed cost, so the report can
Expand Down Expand Up @@ -180,10 +250,27 @@ Close every trial PR with a comment linking the report, delete the
onto a long-lived branch. Leave the collected transcripts with the operator
(attach the report to the PR or issue that motivated the trial).

One exception, and it is the reason the closed-PR shortcut above works: keep a
branch whose round you expect to resume, and then leave it in a state someone
can reopen safely. Revert every deliberate fault on it (reintroduced defects,
mis-staging injections) in the same pass that closes the PR, and put the
resumable state in the closing comment: which threads are still unresolved,
which run IDs the round produced, and what a reopen would trigger. A surviving
trial branch that still carries a planted panic or a prompt injection is a trap
for whoever reopens it, and the reopen fires a review immediately.

## Guardrails (recap)

- Costs projected and approved before the first PR exists.
- Seeds and ground truth are operator-authored, always.
- One arm per PR; distinct workflow names; exactly one reviewer per PR.
- One arm per PR; distinct workflow names; a fresh lock filename per trial;
exactly one reviewer per PR.
- Forward-only pushes once a round has been reviewed; no force-pushes mid-trial.
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
- Read `stampSource` from every round's plan artifact; scoped-cost samples from
non-anchored rounds are void.
- Budget runs by counting the events a step fires (`reopened` and `synchronize`
each cost one review; a closed PR fires none), not by counting pushes.
- Any branch that outlives its PR is reverted to a safe state, with the
resumable details in the closing comment.
- Score before cleanup; export before cleanup; never lose transcripts.
- Faithful reporting, including the arm you expected to win losing.
Loading