review: dedupe lens discipline snippets into one staged disciplines file - #247
Conversation
…live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged.
…action and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately.
… runner (phase 2c)
live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.
live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.
live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero.
Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step.
… and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary).
…rpus' into jwbron/live-eval-producer-staging
…staging' into jwbron/live-eval-ab-runner
…to jwbron/live-eval-ab-ci
…s with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id.
…staging' into jwbron/live-eval-ab-runner
…eport instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual.
…to jwbron/live-eval-ab-ci
…A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed.
…orpus' into tmp-refresh
…roducer-staging' into tmp-refresh
…b-runner' into tmp-refresh
…b-ci' into tmp-refresh
…ity, code-rendered from the reconciler keep-list
… instead of dropping them
…oad (allowed-paths must match staging-relative paths)
🦋 Changeset detectedLatest commit: 0345c27 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…untability' into jwbron/review-out-of-lane
…into one staged disciplines file
8e015ff to
813767c
Compare
…o jwbron/review-trial-skill
…-skill' into jwbron/review-out-artifact-upload
…ut-artifact-upload' into jwbron/review-rereview-accountability
…untability' into jwbron/review-out-of-lane
… into jwbron/review-dispatch-tax
…ereview-accountability
This comment has been minimized.
This comment has been minimized.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| themselves match, never this instruction or the sed command's own text.) | ||
| Then verify the staged file carries the schema section: | ||
| `grep -q '## Structured finding schema and hunts' /tmp/gh-aw/review/disciplines.md`. | ||
| If that verification fails (e.g. `$GH_AW_PROMPT` is unset in a future engine), fall |
There was a problem hiding this comment.
suggestion (non-blocking): The heredoc fallback (lines 322-326) reproduces the ~92-line disciplines section byte-for-byte, but the verify on line 321 greps for only one heading (## Structured finding schema and hunts). A paraphrased or truncated LLM transcription of the other six sections would still pass that single-heading check, so lenses could run on degraded discipline text with no signal. Consider a line-count or content-hash check so a degraded fallback fails loudly.
|
|
||
| The section between the markers below is the single copy of the discipline text | ||
| every **specialist lens** follows. It used to be stamped verbatim into all eleven | ||
| lens definitions and paid on every dispatch; now the lenses read it once from |
There was a problem hiding this comment.
question (non-blocking): Does staging net a per-dispatch token reduction, or relocate the cost? Each of the 11 lens pointers now Reads the ~5.2KB disciplines file as a tool result (still input tokens) plus a pointer block and an extra round-trip, where ten lenses previously carried a shorter condensed variant inline. Worth one measured run to confirm the dispatch-tax saving before the rest of the stack builds on it — or reframing the changeset around drift-elimination, which is the unambiguous win here.
| lines). Stage them for the lens sub-agents with one mechanical extraction — the | ||
| engine writes this rendered prompt to the path in `$GH_AW_PROMPT`: | ||
| ``` | ||
| sed -n '/^<!-- BEGIN REVIEW DISCIPLINES -->$/,/^<!-- END REVIEW DISCIPLINES -->$/p' \ |
There was a problem hiding this comment.
note (non-blocking): The test exercises sedRangeExtract against the source review.md, not the rendered $GH_AW_PROMPT that the runtime sed actually reads. If the compiler ever indents or transforms the marker lines, the anchored sed stages an empty file and the fallback fires on every run — behavior stays neutral, but the cost win is silently lost and no test covers that path. Mitigated by the verify+fallback guard and prior precedent for extracting $GH_AW_PROMPT, so non-blocking.
# Conflicts: # workflows/review/eval/corpus/live.ts
jeresig
left a comment
jeresig
left a comment
There was a problem hiding this comment.
I'm skeptical that this is going to have much impact - I think it's just moving stuff around? Isn't the caching cumulative based upon the previous input tokens? But I guess we can see in the testing.
|
You're right to be skeptical, and working the mechanics through makes it worse than a wash on the sub-agent side: each lens definition shrinks by ~1.3KB at dispatch, but each lens then reads the ~5.2KB staged disciplines file as a tool result (plus a round trip), and the section itself now lives in the main body the orchestrator carries. Caching discounts the repeats either way, so the dispatch premium this set out to close may not move; the body already defers that question to the next lifecycle re-run. The reason to keep the change is drift, not tokens: the eleven stamped copies had already diverged (security-auth carried fuller bounded-investigation and untrusted-input wording than the other ten), and unifying on one source removes a correctness hazard in what the lenses are told. I've updated the description to state the read-side arithmetic plainly. Longer term this staging step migrates to a pre-workflow step with the rest of the staging layer (same direction as your #246 comments), where the single-source file is the shape that work wants anyway. |
…n against real sed, not only the modeled replica
Review Guidancegithub-actions (1 file)
Common patterns1 file ( Excluded from review (2 files)Not individually reviewed — generated, formatting-only, or
|
…n' into jwbron/review-dispatch-tax
…pies the main merge re-imported (the shared disciplines section carries the text)
2 participants
Summary
Fourth of five PRs in the fold-in batch two stack (stacked on #245; the stack is based on
mainsince 2026-07-10). The top of the batch-two stack isjwbron/review-skill-rule-quote(#248).The v1.4.0 re-run's cost premium vs the v1.3.1 baseline sits at dispatch time (uncached input +51-65%, cache writes +18-34% per run), partly because the discipline snippets (bounded investigation, untrusted input, lens-owned skills incl. quote-the-rule, the schema-rules trailer, the tri-state hunt contract) are stamped verbatim into every one of the eleven specialist-lens definitions and paid on every dispatch.
The mechanics, grounded in the gh-aw v0.81.6 source:
## agent:sections are extracted from the markdown at compile time into per-agent definitions (pkg/parser/sub_agent_extractor.go), and each dispatch pays its definition's full text; the orchestrator's rendered prompt (the main body only) is written to disk at the engine-provided$GH_AW_PROMPTpath. So the shared text now lives once, in a marker-delimited "REVIEW DISCIPLINES" section at the end of the main body, and:sed -n '/^<!-- BEGIN ... -->$/,/^<!-- END ... -->$/p' "$GH_AW_PROMPT" > /tmp/gh-aw/review/disciplines.md. The patterns are whole-line-anchored on purpose, so the sed command's own text and prose mentions of the markers can never open the range. A grep verifies the staged file carries the schema section; the fallback is a byte-for-byte heredoc copy, never a paraphrase. Prompt and disciplines version atomically (both live in review.md), with no coupling to the lib-checkout ref.Behavior-neutral by design: the instruction content is unchanged. Where the eleven stamped copies had already drifted (security-auth carried a fuller bounded-investigation and untrusted-input wording than the compact copies in the other ten), the disciplines section unifies on the fullest wording, so no lens loses instruction it previously had. The label-shape reviewers (correctness, skill-auditor, claim-validator, and the whole-change reviewers) are deliberately out of scope: their variants differ materially (per-agent investigation-cap CLI id semantics,
discussionvsevidence_trace), so deduping them is not the mechanical win it is for the lenses.Failure scenario this guards against, and how it is pinned: if the extraction silently produced an empty file, every lens would run without its disciplines (no quote-the-rule, no schema rules).
lib/disciplines.test.tsreplicates the exact sed range semantics against review.md and asserts the extraction is non-empty and carries all seven section headings, that every lens points at the staged file and carries no residual copy, and that the label-shape reviewers still carry theirs; Step 1 additionally verifies the staged file at run time before any lens is dispatched.Net effect on review.md: 246 insertions, 474 deletions (about 16.6KB removed); each lens's dispatched definition shrinks by roughly 1.3KB. Whether that fully closes the measured dispatch premium is for the next lifecycle re-run to say; the duplication and the drift are gone either way.
Read-side arithmetic, stated plainly (raised in review by jeresig and the bot reviewer): each lens now reads the ~5.2KB staged disciplines file where its stamped copy was ~1.3KB, plus one tool round trip, and the disciplines section itself moves into the main body the orchestrator carries. So per-run tokens are at best a wash and may tick up; the change is justified by drift-elimination (a single source for what the lenses are told), not by a priced token saving. Under the deterministic-orchestrator staging migration (see #246 review discussion) this extraction becomes a pre-workflow step and the single-source file is the shape that work wants anyway.
cc @jeresig
Testing
npx vitest run workflows/review: 508 tests green (24 new inlib/disciplines.test.ts).npx tsc --noEmitclean.