Repository navigation
review-trial: record three operational lessons from the preview trials #306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "review": patch | ||
| --- | ||
|
|
||
| Skill-doc only: record five operational lessons from the 2026-07 preview trials in the `review-trial` skill. (1) Never force-push a trial branch between rounds: a round that cannot anchor on a prior fingerprint reports `stampSource: null` in `out/rereview-plan.json` and executes `depth: full` with `staging: whole-diff` even where ROUTING says `scoped`, which is indistinguishable from a genuine carrier gap and silently voids that round's scoped-cost sample. `stampSource` in the plan artifact is the authoritative anchoring signal and must be read every round; the carrier internals that make a rebase the wrong suspect (content-hashed hunk signatures, a `commitSha` that is written but never read back) are dated implementation properties, not contracts. (2) Use a fresh lock filename per trial: GitHub's workflow registry keys the display name to the file path and keeps the stale one, so two trials that reused `review-preview.lock.yml` both listed as "PR Reviewer" while the files declared "PR Reviewer Preview v1.8.0" and "...v1.9.0". Concurrency was unaffected, but a run list naming every arm after the production reviewer invites the "did production just run?" scare the naming rule exists to prevent. (3) A lifecycle plan that means to exercise open-thread suppression must leave at least one finding unfixed, since a push where the fixer repairs everything lets the reconciler resolve every thread and `threads.json` comes back empty. (4) Count runs by the EVENTS a step fires, and prefer resuming a closed trial PR over building a new lifecycle: a closed PR fires no `pull_request` event (pushes are free), `reopened` is in the reviewer's trigger list (exactly one run), and pushing to an open PR fires `synchronize` (one more each time), so push-while-closed then reopen buys a round for one run. A closed PR whose last round's comments are still unresolved already IS a partially-fixed round; webapp#41204 was reopened that way for the round that made open-thread suppression fire in production for the first time (run 30650642317), which had been budgeted at two runs. Unresolving a thread is a GraphQL mutation an agent's tooling may refuse, so it belongs to the operator. (5) Keep fault injections out of the diff under review: mis-staging to trip a fail-open reporter works (run 30654454047 reported `unusableThreads: 9`), but injecting via the prompt or lock file on the trial branch makes that edit the newly-changed code, so all five of that run's candidates landed on the prompt file and none on the fixture, the duplicate re-posting the reporter warns about went unobserved, and the reviewers spent the round correctly flagging the injected prose as agent-directing instructions. The cleanup step gains the corollary the closed-PR shortcut requires: a branch that outlives its PR is reverted to a safe state and its resumable details (unresolved threads, run IDs, what a reopen fires) go in the closing comment, since a surviving branch carrying a planted panic is a trap and a reopen reviews immediately. No change to the shipped review workflow. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question (non-blocking): Does a doc outside
workflows/review/warrant areview: patchbump? The skill lives at repo-level.claude/skills/review-trial/, outside thereviewpackage, and the changeset's own closing line says "No change to the shipped review workflow" — yet this bump cutsreview-v1.9.1and rewrites the pinnedreview-v<semver>literal inreview.md. If the changeset gate only needs a file to exist,pnpm changeset --emptylands the lessons without minting a contentless release; keep thepatchbump only if trial docs are deliberately versioned with the package.