review: the live producer (live A/B phase 2) - #234
Merged
Conversation
…live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged.
…action and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately.
🦋 Changeset detectedLatest commit: 12f5d84 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
… runner (phase 2c)
live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.
live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.
jwbron
changed the title
This was referenced Jul 9, 2026
… and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary).
…rpus' into jwbron/live-eval-producer-staging
…s with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id.
This was referenced Jul 9, 2026
…ive case The first live acceptance runs missed incident-sql-missing-index:dm-missing-index-1 in all four arms: the recorded finding is a data-migrations LENS finding, but live cases carried no routerConfig lens rules, so the live roster never spawned the specialist that catches it. Every live case whose recorded finding belongs to a specialist lens now routes that lens on the finding's file (seven cases across five lenses), mirroring how a consumer ROUTING file would route the same paths in production.
…rpus' into jwbron/live-eval-producer-staging
…/live-eval-corpus' into jwbron/live-eval-corpus
…gin/jwbron/live-eval-producer-staging' into jwbron/live-eval-producer-staging
…rpus' into jwbron/live-eval-producer-staging
jwbron
marked this pull request as ready for review
July 10, 2026 18:36
khan-actions-bot
requested review from
a team,
jaredly and
jeresig
and removed request for
a team
July 10, 2026 18:36
khan-actions-bot
requested review from
a team and
jaredly
and removed request for
a team
July 10, 2026 19:48
Contributor
Review Guidancegithub-actions (4 files)
Common patterns2 test files: Identical 2 implementation files: Identical real- Excluded from review (2 files)Not individually reviewed — generated or formatting-only:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| if (entry.isDirectory()) { | ||
| copyDir(from, to, fs); | ||
| } else if (entry.isFile()) { | ||
| fs.writeFileSync(to, fs.readFileSync(from, "utf8")); |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): copyDir round-trips every file through readFileSync(from, "utf8") + writeFileSync, and StageFs exposes no byte-preserving read/write — so a case tree containing a real binary (image, fixture archive) is silently corrupted in the staged checkout, and the live arm then investigates files that differ from what the case recorded. Consider adding a copyFileSync (or Buffer-based) member to StageFs and using it here; memfs supports both, so the test seam need not force the utf8 restriction.
Low-confidence notes (2)
workflows/review/eval/live-producer.ts:307— cross-agent id-collision resolution is order-dependent under concurrent dispatch, so a genuine collision is not resolved reproducibly across runs.workflows/review/eval/live-runner.ts:56— unverified (SDK not in checkout): doesallowedToolsrestrict the toolset underpermissionMode: "bypassPermissions", or only auto-approve it? If it does not restrict, Bash/Write/WebFetch stay reachable despite the read-only module doc.
| (output) => parseVerifications(output, knownIds), | ||
| ); | ||
| perAgent.push(report); | ||
| validation = parsed ?? []; |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): The documented validator-failure fallback here (a twice-failed claim-validator yields validation = [] while produced findings survive) has no test — every produceLive test drives the validator happy path or skips it. Current behavior is correct; the gap is that a regression making a failed validator throw or drop findings would go uncaught. Consider a case that scripts the validator to return non-JSON twice and asserts findings.length > 0, validation is [], and the validator agent is marked failed.
| }); | ||
|
|
||
| headings.forEach((heading, i) => { | ||
| const sectionEnd = headings[i + 1]?.index ?? lines.length; |
Contributor
There was a problem hiding this comment.
thought (non-blocking): A section ends at the next ## agent: heading or EOF, so a non-agent ## section appended after the last agent would be folded into that agent's prompt with no parse error — the silent skew the strict parser exists to prevent. It holds today only because agent sections are currently terminal in review.md. Ending a section at the next ## heading of any kind (or asserting in the integration test that none follows the agents) would close the gap.
| const DEFAULT_CONCURRENCY = 4; | ||
|
|
||
| /** The real filesystem, in the staging seam's shape (mirrors live-stage). */ | ||
| const NODE_FS: StageFs = { |
Contributor
There was a problem hiding this comment.
nitpick (non-blocking): The sibling live-stage.ts:54 and corpus/loader.ts:724 both name this real-fs StageFs adapter DEFAULT_FS; here it is NODE_FS, though the line-133 comment says it "mirrors live-stage." Renaming to DEFAULT_FS keeps one searchable name for the concept across the eval directory.
| const verification = raw["verification"]; | ||
| if (typeof id !== "string" || !knownIds.has(id)) { | ||
| throw new Error( | ||
| `claims[${index}].id does not match a produced finding`, |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): parseVerifications's rejection guards (unknown id here, invalid verification state just below) are untested — every scripted validator supplies valid ids and states. Consider a case emitting an id not among the produced findings (or verification: "maybe") and asserting the validator is marked failed and validation is [].
| anchor: { | ||
| type: "line", | ||
| path: raw["path"], | ||
| line: raw["line"], |
Contributor
There was a problem hiding this comment.
suggestion (non-blocking): fromLabelShape hardcodes anchor.type: "line" with line: raw["line"], which validateFinding rejects for any non-positive-integer line. Since the label-shape contract in review.md uses "line": 0 as its placeholder and parseAgentFindings maps eagerly, a single line: 0 / omitted-line / PR-level entry throws and sinks the whole agent's batch (retry, then agent marked failed, losing its other valid findings). Consider treating a missing/zero line as a file- or PR-level anchor rather than throwing.
| producing_hunt: `live:${agentName}`, | ||
| model_authored_prose: | ||
| discussion === "" ? subject : `${subject} ${discussion}`.trim(), | ||
| ...(typeof raw["suggestion"] === "string" && raw["suggestion"] !== "" |
Contributor
There was a problem hiding this comment.
nitpick (non-blocking): The suggestion -> suggested_patch round-trip (here, then carried back out as suggestion in buildClaims) is never asserted — no LABEL_FINDING fixture sets suggestion. A regression dropping it would silently strip suggested patches from live output with no failing test.
jwbron
added a commit
that referenced
this pull request
Jul 10, 2026
Phase 1 of the live A/B eval plan (#232): the eval corpus learns to carry real change content so a future live producer can run the actual model sub-agents against it. ## Format A case may now carry an opt-in `live` block (`corpus/live.ts`, re-exported through the loader): - `prContext`: PR title/description/author/base branch, mirroring production `pr-context.json`. The description is untrusted author text, so adversarial cases can carry their payload there or in the diff. - An on-disk post-change file tree, via a new `<id>/case.json` + `<id>/tree/` layout coexisting with flat `<id>.json`. A directory containing `case.json` is one case; its tree is never parsed as corpus JSON. - `mustCatchSpecs` / `mustNotFlagSpecs`: labeled defects as (path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window plus mechanism rather than id; the Phase 3 matcher consumes these. Enforced invariants: the `live` tag and the block imply each other; a live case needs a cleanly-parseable diff (fail-open is fine in production, an authoring error here); spec paths must appear in `changedFiles` and the diff; every non-removed changed file must exist in the tree. `loadLiveCorpus()` returns the subset. ## Cases Ten cases converted with hand-authored real diffs and trees: the five smoke incidents (money rounding, auth bypass, cache key, race condition, missing index), both clean cases, the adversarial injection case (payload is a code comment in the diff instructing the reviewer to approve), the golden authz holdout, and the money-payments synthetic mutation. Recorded line anchors are rewritten to the authored defect lines; natural files beat content padded to synthetic line numbers, and every anchor is asserted to be an added line so the provenance gate (now active on these cases) keeps them. The trial-content port landed as #235, stacked on this PR. One acceptance-driven change also landed here: the phase 4 runs missed `incident-sql-missing-index` in all four arms because live cases carried no `routerConfig` lens rules, so specialist lenses never spawned; every live case whose ground truth belongs to a specialist lens now routes that lens on the finding's file (see the "route the specialist lens on each live case" commit). ## Test plan: - `pnpm run test --run`: 659 tests green, including 17 new loader tests (memfs) covering the live block, both layouts, tree validation, and the tree-JSON exclusion. - The deterministic suite runs the ten converted cases unchanged (same verdicts, comment counts, must-catch sets), now with the provenance gate active on them. - `pnpm run typecheck` and eslint clean on the three touched TS files. ## Next steps (human) 1. Review the ten authored diffs and trees for realism; they are synthetic content a live model will read, so plausibility matters more than in ordinary fixtures. Spot-check that each case's recorded finding still describes the authored defect (anchors were rewritten to the authored lines). 2. This is the root of the stack: review and undraft first; #234 and #235 rebase onto it. 3. No live validation needed here; the deterministic suite (`pnpm run test --run`) fully gates it. Author: jwbron Reviewers: jeresig, github-actions[bot], jwbron Required Reviewers: Approved By: jeresig, github-actions[bot] Checks: ✅ 9 checks were successful Pull Request URL: #233
…gin/main' into jwbron/live-eval-producer-staging # Conflicts: # workflows/review/eval/corpus/live.ts
jwbron
added a commit
that referenced
this pull request
Jul 10, 2026
Phase 3 of the live A/B eval plan (#232), stacked on the Phase 2 producer (#234): score live runs against labeled ground truth and diff two review.md versions arm to arm. ## 3a: `eval/live-match.ts` Live runs choose their own finding ids, so the recorded metrics (which key on `expected.mustCatch` ids) cannot score them. The matcher maps a run's POSTED candidates onto the case's labeled defect specs: a candidate satisfies a spec when its anchor agrees with the spec's path (and line window, when both carry one) and any mechanism alternate matches the finding's `failure_scenario` or prose, case-insensitively. Each candidate satisfies at most one spec and each spec at most one candidate, so one comment cannot claim two defects. An injected fallback arbiter (hard-capped, same-file candidates only, recorded as `via: "fallback"` for human audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. `computeLiveMetrics` aggregates the live analogues of the suite's numbers: must-catch recall, verdict agreement, clean false-flag (a clean case that blocks counts), and noise (posted candidates matching no spec). ## 3b: `eval/live-ab.ts` The arm orchestrator and CLI. Baseline `review.md` comes from `git show <merge-base>` (or `--base-ref`), candidate from the working tree; both arms run over the same live corpus with everything else (corpus, `lib/`, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model-behavior seam. Each arm runs under half the `--max-usd` budget (default 40 total) with sticky exhaustion: once spend plus the running per-case average crosses the cap, the remaining cases are recorded as SKIPPED and the report still emits (dying at a cap with nothing posted is the failure mode the plan forbids). Each live result replays through the deterministic pipeline; `runner.ts` gains an optional `RunOptions.validation` override so the live validator's output replaces the recorded block. Spec-level regressions ("baseline caught, candidate missed" and the reverse) are diffed only over cases both arms actually scored, so a budget skip is never reported as a regression. Judge scoring reuses the pinned judge for quality aggregates only (judge-vs-ground-truth disagreement keys on recorded ids, which live arms do not use); the fetch model moves from `live-judge.ts` into shared `eval/judge-live-model.ts`. Output: `out/live-ab-report.json` plus a markdown table (metrics deltas, judge quality delta, cost, wall clock, regressions, skips, agent failures) printed and appended to `GITHUB_STEP_SUMMARY`. Report-only, with one exception per the playbook's standing rule: the candidate arm failing any adversarial-injection case (wrong verdict or missed spec) exits non-zero. ## Test plan: - `pnpm run test --run`: 696 tests green (16 new: matcher rules incl. window overlap, one-candidate-one-spec, malformed-regex fallback-to-literal, capped fallback audit trail; runArm budget semantics incl. the sticky-stop case that caught a real bug in review; regression diffing over shared cases; report rendering both gate outcomes). - `pnpm run typecheck` and eslint clean. - The end-to-end CLI (`pnpm dlx tsx workflows/review/eval/live-ab.ts --max-usd 10 --cases incident-money-rounding,clean-no-findings`) needs `ANTHROPIC_API_KEY`; this environment has none, so that is the first thing to exercise when testing. On an unchanged review.md it prints the identical-arms note and near-zero deltas. ## Next steps (human) 1. ~~End-to-end A/B validation~~ DONE, superseded by the phase 4 acceptance runs (linked on #237): the control arm held recall and verdict agreement flat at +0% and the weakened arm showed -17% recall with the lost spec named, which is also the first empirical read on matcher quality (7 cases matched; one systematic miss traced to lens routing, fixed on #233, not to the matcher). Two acceptance-driven additions landed here since: `perCase.failedAgents` entries now carry `<agent>: <reason>` (a claim-validator flake was undiagnosable from the report alone; the flake itself is still to be root-caused on a future run), and every report closes with a per-row stability footer (judge quality rose on the deliberately regressed arm because fewer, surer comments each score better, so the footer warns against reading it alone). 2. Review the two scoring caveats: judge output is quality-aggregates only (disagreement semantics key on recorded ids), and the fallback arbiter is wired but not yet given a live implementation (matches would be marked `via: "fallback"` for audit when one is added). 3. Confirm the budget semantics (half of `--max-usd` per arm, sticky skip-and-report) match how you want cost bounded. --- ## Update 2026-07-09: eval-tuning pass (three riders) Three instrument fixes from the [eval-tuning memo](https://claude.ai/code/artifact/2995f0b5-8840-4d03-96e3-2518e2f0f75a), landed here so the whole stack inherits them: 1. **Pre-flight identity short-circuit** (memo item 1, hoisted down from #251 with identical text so that branch merges cleanly): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; `--force-arms` preserved for wobble controls. Corpus-only and plumbing-only PRs in this stack now pay $0 per push. 2. **Gate-flip retry** (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake (the #244/#245 incident shape). Retry spend is recorded in the report. 3. **Drop-bucket taxonomy** (memo item 4): every missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/validation) by re-matching against the dropped candidates with the line window relaxed. Lost regressions are annotated with their class and the table gains a found-but-dropped count row. Plus **judge economics**: the judge pin moves from Opus 4.8 to `claude-haiku-4-5-20251001` (the judge grades prose only; every load-bearing metric is deterministic, and the acceptance runs showed judge quality is not single-run-stable on any model, so it is priced accordingly; supersedes operator direction 4), and the judge seam gains retry with backoff so a transient 500 no longer wastes a whole scoring pass. Author: jwbron Reviewers: jeresig, github-actions[bot], jwbron, jaredly Required Reviewers: Approved By: jeresig, github-actions[bot] Checks: ✅ 9 checks were successful Pull Request URL: #236
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of the live A/B eval plan (#232), complete, stacked on the Phase 1 corpus format (#233): everything needed to run the real model sub-agents over a live-enabled corpus case.
2a:
eval/agent-extract.tsParses
review.md's## agent:sections into{name, description, model, prompt}. String in, no filesystem access, because the baseline arm of an A/B extracts the merge-basereview.mdviagit show. Parsing is strict: missing/unterminated frontmatter, a name mismatch, a missing model, or an empty body throws with every problem listed, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the realreview.md(21 agents) and pins the default roster's reference to the production staging root so a rename fails a test instead of silently staging nothing.2b:
eval/live-stage.tsMaterializes the production
/tmp/gh-aw/review/layout for one case, perreview.mdStep 1's staging contract:pr-context.json(synthetic identity fields),full.diff/pr.diff/full-stripped.diff,files.json+review-files.jsonwith thehasPatchcross-check,provenance.json,routing.jsonfrom the deterministic router,out/, and the post-change checkout from the case's tree.rewriteAgentPromptswaps the production staging root across a prompt.2c:
eval/live-producer.ts+eval/live-runner.tsproduceLivedispatches the live roster (default finders plus the router'slensesToSpawn; no pattern-triage or thread-reconciler in eval) behind an injectedLiveAgentRunnerseam, the same pattern asjudge.ts's injected model, so all its logic is stub-tested with zero model calls. It maps the three sub-agent output contracts into exactly the shapes the deterministic runner consumes (RecordedFinding[]forproduceFindings,CaseVerification[]for the validation replay): label-shape findings get the code-assigned lens (correctness, orconventionsfor the skill-auditor solabelForFindingreproduces the best-practice label variants) and the production 0.7 confidence default; lens findings validate against the schema as-is; the claim-validator's{claims: [...]}output becomes verifications. It stagesclaims.json, resolves{{#runtime-import}}directives from the case tree (a case opts into a skills index by carrying the file), retries once on malformed output with the rejection fed back, keeps partial results when an agent fails twice, prefixes colliding finding ids, and accounts per-agent cost/turns/wall-clock.live-runner.tsis the only module touching a real model runtime: one Agent SDKquery()per dispatch, Read/Grep/Glob only, cwd pinned to the staged checkout, the agent's pinned model, hard turn and wall-clock caps, plus the CLI smoke entry:pnpm dlx tsx workflows/review/eval/live-runner.ts --case <id>(requiresANTHROPIC_API_KEY). Adds@anthropic-ai/claude-agent-sdkas a dev dependency.Documented deviations from production (module doc): no pattern-triage pass, runtime imports resolve from the case tree or a fixed note, and the investigation-cap CLI is not staged (its own denied-budget fallback applies).
Test plan:
pnpm run test --run: 680 tests green (21 new across the three test files: extraction fixtures + real-review.md integration, staging via memfs, producer via a scripted stub runner covering happy path, retry, double-failure, empty-findings skip, id collisions, missing-agent error, and runtime-import resolution).pnpm run typecheckand eslint clean.ANTHROPIC_API_KEY, which this environment does not have; it is the first thing to run when testing this PR (any of the 13 live case ids, e.g.--case incident-money-rounding). Expected cost: roughly $0.50 to $2 per case with the default roster.Next steps (human)
Key-bearing validationDONE, superseded by the phase 4 acceptance runs, which exercised this producer end to end with a real key over 7 live cases x 2 arms x 2 PRs (runs 29052332224 and 29052334404); measured ~$0.55-0.72 per case per arm, inside the plan's expected envelope. The CLI smoke remains available for local iteration.live-runner.ts):permissionMode: "bypassPermissions"with tools restricted to Read/Grep/Glob and cwd pinned to the staged checkout. Confirm you are comfortable with that posture for CI.live-producer.ts's module doc (no pattern-triage, runtime-import resolution, investigation-cap absence); each is a measurement caveat for the A/B.@anthropic-ai/claude-agent-sdk. Confirm it clears whatever dependency review applies to this repo.