review-consumer-bump: a skill for rolling reviewer releases out to consumers

[jwbron]

Adds .claude/skills/review-consumer-bump/SKILL.md: the playbook for rolling a review-v<version> (and optionally autofix-v<version>) release out to every consuming repo, one PR per consumer. Written from the 2026-08-20 rollout (kore-marketplace#11, agent-settings#76, webapp#41661, #356), where every pitfall it records was hit live.

The judgment calls it carries:

• consumers are discovered by org-wide code search on the installed source: line each time, never from a list in the skill (lists drift; the search found exactly the 4 current consumers when tested, and the command in the skill is the tested percent-encoded form since the raw query 400s intermittently)
• gh aw update is explicitly banned with the two observed failures: it treats review-v<version> tags as branches and repins to main's head SHA, and its 3-way merge emptied a consumer's review.md to 0 bytes
• the replacement is 4 commands around git merge-file, with the LOCAL OVERRIDE inventory check before and after
• compile side effects are classified keep-or-revert (setup-action pin moves: keep; stripped merge=ours gitattribute: revert; gh-aw v0.85.x deleting agentics-maintenance.yml: keep)
• verification is two gates: providers in both awf-config payloads of the lock, and the consumer-config checker run from a version-matched tag checkout with a path (not a name) as --repo, both of which failed silently or confusingly when done wrong this week

Mostly skill-doc, "review": patch changeset per the #306 precedent. Review feedback also had the gh aw update ban propagated to the shipped docs: workflows/review/README.md now points updates at the merge flow instead of the tool, and one frontmatter comment in workflows/review/review.md drops the "gh aw update preserves it" claim. No behavior change to the shipped workflow.

[changeset-bot]

🦋 Changeset detected

Latest commit: 3aa6f35

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

_{review details}

_{review-v1.17.1 | schema 2 | depth full | re-review scoped blocking-only | enable holistic,completeness,test-adequacy,first-principles,conventions,documentation}

[jwbron]

Took three of the review's findings in 0fb321e, plus the nitpick:

• The .gitattributes collision was real: the post-compile git checkout -- .gitattributes also discards the stale-line removal the next bullet asks for. The skill now says the removal happens after the final compile. Went with an ordering rule rather than git checkout -p; interactive patching is a worse instruction for an agent-executed skill.
• The consumer-discovery search now fails loudly past 100 hits (total_count guard in the jq) instead of silently dropping the overflow.
• The gh aw update inconsistency was the strongest of the low-confidence batch: review-onboarding prescribed it in 4 places for the exact operation this skill documents failing twice. All 4 now point at the merge flow here.
• Headings renumbered to Step 1..6 to match the siblings.

Two follow-ups, deliberately not in this PR:

• workflows/review/README.md still recommends gh aw update in 4 places; that's shipped consumer-facing content, so it gets its own PR.
• The --repo-takes-a-path footgun in check-consumer-config.ts is fixable with an existence check on the argument; code change, not a docs change.

On the changeset question: repo precedent settles it. The Lints job fails without a changeset (baa5fc7 records exactly this), every prior skill-doc PR took a patch bump, and the changeset already says the shipped workflow is unchanged, which is the signal to skip a rollout for it.

[github-actions]

Changes requested — see inline comments.
All 2 prior review threads are resolved.
Note: divergence tripwire re-armed a full review (unreviewed share 1.00).

_{review details}

_{review-v1.17.1 | schema 2 | depth full | re-review scoped blocking-only | enable holistic,completeness,test-adequacy,first-principles,conventions,documentation}

[jwbron]

Took all five in d4496b4.

The blocking one first: the providers gate now counts the two payloads separately, exactly per the sketch. Verified against this repo's lock: grep -c '"providers"' returns 2 (lines 150 and 983, the MODEL_COSTS env line plus the unescaped payload) and grep -c '\\"providers\\"' returns 1 (line 1597, the escaped payload). The old check's answer of 2 really was a coincidence; it never saw the second payload at all.

The rest:

• Step 2 now opens with git -C <actions> fetch --tags and a test -s on both extracted files before the merge, since a 0-byte theirs reads as an upstream deletion and the conflict-marker gate can't see that
• dropped the "four commands" count rather than recounting it; with the new guard it kept drifting anyway
• added the "What stays human" gate per the sketch: consumer set and version confirmation before branching, force-push as the operator's call
• propagated the ban to workflows/review/README.md (install section now points at the skill's merge flow, and the three "gh aw update preserves it" claims are reworded); turns out there was one more occurrence the grep list missed, the observability comment in workflows/review/review.md, so that's reworded too. Comment-only, and the installed copy stays pinned so it picks the wording up at the next bump; changeset updated to say so.

[github-actions]

Guidance for reviewers

Triage notes for reviewers: risky files by owning team, repeated changes, and files excluded from review.

github-actions (2 files)

File	Reason
SKILL.md	.claude/skills/review-consumer-bump/SKILL.md — new skill that drives an agent to open mutating PRs in every consuming repo, so a wrong command or missing guard here lands as a bad merge in someone else's installed review.md.
SKILL.md	.claude/skills/review-onboarding/SKILL.md — the hunk rewrites the runnable update recipe an agent follows against a consumer repo.

Common patterns

3 files: Replace every reference to gh aw update (as the upgrade mechanism) with references to the manual 3-way merge flow / review-consumer-bump skill.

- a local edit; `gh aw update` preserves it.
+ a local edit; the 3-way merge update flow preserves it.

Excluded from review (1 file)

Not individually reviewed — generated, formatting-only, or
fully explained by a common pattern above:

• workflows/review/review.md — pattern-only

_{review details}

_{review-v1.17.1 | schema 2 | depth full | re-review scoped blocking-only | enable holistic,completeness,test-adequacy,first-principles,conventions,documentation}

[github-actions]

All 5 prior review threads are resolved.
Note: divergence tripwire re-armed a full review (unreviewed share 0.67).

_{review details}

_{review-v1.17.1 | schema 2 | depth full | re-review scoped blocking-only | enable holistic,completeness,test-adequacy,first-principles,conventions,documentation}

[github-actions]

note (non-blocking): The gh aw update doc sweep misses the checker's user-facing source-missing warning. Grepped the tracked tree for gh aw update: after this PR the only remaining live prescriptions are workflows/review/lib/check-consumer-config.ts:540 and .github/workflows/review.md:168. The latter is correct to leave alone (it is this repo's installed copy, frozen to its pinned tag by review-pins.test.ts), but line 540 is emitted output the new skill's own Step 5 tells operators to read and quote into PR bodies.

Lower-confidence observations (2)

• .claude/skills/review-consumer-bump/SKILL.md:89 suggestion (non-blocking): The merge procedure is deterministic and guard-heavy — it reads like a script that was written as prose. _{(first-principles)}
• workflows/review/README.md:182 thought (non-blocking): The upgrade procedure now lives in a maintainer agent skill while the consumer-facing doc only points at it. _{(first-principles)}

_{review details}

_{found by completeness | also flagged by holistic}

[github-actions]

question (non-blocking): The gh aw update ban is unconditioned and the root cause may not be upstream's. Failure (1) is that gh-aw does not read review-v<version> as a release tag because it is not bare semver — that is a consequence of this repo's own tag scheme, and failure (2) (merge emptying a file to 0 bytes) is a data-loss bug worth reporting since other Khan repos use the same tool. Was an upstream issue filed, and does the ban carry a version bound plus a condition under which it is re-evaluated?

_{review details}

_{found by first-principles}

[github-actions]

Review live A/B

Baseline: origin/main (review.md 604183e7637f); candidate: working tree (review.md 8ee1dd35aa6c).

Ruler: matcher deterministic+arbiter; corpus 0441b7c6a581 (10 cases).

Metric	Baseline	Candidate	Delta
Must-catch recall	100%	100%	+0%
Verdict agreement	90%	90%	+0%
Noise (unmatched posted)	63%	65%	+3%
Clean false flags	1	1
Judge mean quality	0.83	0.87	+0.03
Cost	$10.00	$9.52
Wall clock	1570s	1584s
Cases run / skipped	10 / 0	10 / 0
Misses found-but-dropped	0	0
Findings anchor-snapped	0	0
Cross-source claims merged (of candidates)	4 / 28 (0 by clusterer at $0.30 / 139s, 8 proposed member(s) rejected)	7 / 33 (1 by clusterer at $0.38 / 195s, 10 proposed member(s) rejected)

Adversarial hard gate: PASSED on the candidate arm.

Single-run-stable rows: recall, verdict agreement, regressions, adversarial gate. Judge quality and noise are not: they jitter run-to-run at this corpus size, and a regressed reviewer can score HIGHER on judge quality (fewer, surer comments each read better). Recall against the labeled specs is the load-bearing metric.

Measured noise floor (identical arms, run 29069228968, 2026-07-10, 6 arm-samples, full corpus x3, pre-arbiter; budget skips left the samples on unequal case sets, so these v1 bands also carry case-mix variance): must-catch recall 54%-86% (sd 10%), verdict agreement 75%-100% (sd 9%), noise (unmatched posted) 50%-60% (sd 3%), judge mean quality 82%-86% (sd 2%). A single-run delta whose arms both sit inside a band is indistinguishable from run-to-run wobble; use --repeats to resolve smaller effects.

[github-actions]

Approved — no blocking issues found.
3 of 5 prior review threads resolved; 2 still unaddressed as of 3aa6f35:

2 non-blocking threads still open

• question (non-blocking) .claude/skills/review-consumer-bump/SKILL.md:71: The gh aw update ban is unconditioned and the root cause may not be upstream's. Failure (1) is that gh-aw does not rea...
• note (non-blocking) workflows/review/README.md:183: The gh aw update doc sweep misses the checker's user-facing source-missing warning. Grepped the tracked tree for `gh...

Non-blocking observations (2)

• .claude/skills/review-consumer-bump/SKILL.md:41 question (non-blocking): Step 1's discovery query matches only ...review.md@review-v pins, so it cannot find a consumer whose source: line is a raw SHA or an unpinned default-branch install — including the exact damage the skill's own "Do not use gh aw update" section says the tool leaves behind ("repins to the head commit of main as a raw SHA instead of the target tag"). _{(skill-auditor (out-of-lane))}
• .claude/skills/review-consumer-bump/SKILL.md:170 suggestion (non-blocking): --repo (consumer-path) renames the placeholder the repo's other checker invocations call (consumer). _{(conventions)}

Note: 2 finding(s) not re-posted (already tracked in open review threads). Note: re-review ran at scoped depth (re-review mode scoped, blocking-only).

_{review details}

_{review-v1.18.0 | schema 2 | depth scoped | re-review scoped blocking-only | enable holistic,completeness,test-adequacy,first-principles,conventions,documentation}