review: hand off out-of-lane observations instead of dropping them - #245
Merged
Conversation
…live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged.
…action and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately.
… runner (phase 2c)
live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.
live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.
live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero.
Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step.
… and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary).
…rpus' into jwbron/live-eval-producer-staging
…staging' into jwbron/live-eval-ab-runner
…to jwbron/live-eval-ab-ci
…s with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id.
…staging' into jwbron/live-eval-ab-runner
…eport instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual.
…to jwbron/live-eval-ab-ci
…A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed.
🦋 Changeset detectedLatest commit: cbb2999 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…orpus' into tmp-refresh
…roducer-staging' into tmp-refresh
…b-runner' into tmp-refresh
…b-ci' into tmp-refresh
…ity, code-rendered from the reconciler keep-list
… instead of dropping them
…oad (allowed-paths must match staging-relative paths)
jwbron
force-pushed
the
jwbron/review-rereview-accountability
branch
from
July 9, 2026 21:43
f85b74f to
491a983
Compare
jwbron
force-pushed
the
jwbron/review-out-of-lane
branch
from
July 9, 2026 21:43
df3d369 to
7b5318c
Compare
Contributor
Author
This was referenced Jul 9, 2026
This comment has been minimized.
This comment has been minimized.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| * block on its own. | ||
| */ | ||
| export type OutOfLaneObservation = { | ||
| /** Repo-relative path the observation anchors on. */ |
Contributor
There was a problem hiding this comment.
thought (non-blocking): The production orchestrator now routes out-of-lane observations, but the live-eval producer (eval/live-producer.ts parseAgentFindings) reads only findings[]. So in a review-trial a handed-off observation is dropped and the arm scores as if it never happened — the same drop this PR closes on the production path. Worth teaching the eval harness to read out_of_lane_observations[] in a follow-up so the trial can measure this feature.
| * returned, so a producer's malformed handoff is diagnosable from the run | ||
| * artifact rather than silently dropped. | ||
| */ | ||
| export const validateOutOfLaneObservation = ( |
Contributor
There was a problem hiding this comment.
question (non-blocking): validateOutOfLaneObservation is referenced only by the tests — the orchestrator prose in review.md never invokes it (unlike the provenance gate, which runs a real CLI). Is the intent that shape enforcement stays eval/test-only, as validateFinding already is? If so, the PR description's "shape is code-validated" slightly overstates the live behavior; a one-line note that the live gate is the orchestrator prompt would settle it.
| lens, a concern outside its domain). Do not discard these. Convert each observation | ||
| into a candidate comment in the same label-bearing shape as every other candidate: | ||
| `path`/`line` from the observation, `subject` from its `observation` text verbatim, | ||
| `failure_scenario` verbatim, and the label **`question (non-blocking)`** — the label |
Contributor
There was a problem hiding this comment.
thought (non-blocking): The handoff channel has a lower evidentiary bar than the lane it feeds. A finding must carry evidence_trace and an honest confidence; an out-of-lane observation carries neither, so once converted to a candidate it has no confidence and defaults to 0.7 (review.md:693) — above the 0.5 medium posting bar — leaving the claim-validator as the only gate before it posts. Consider code-assigning out-of-lane candidates a below-default confidence so only validator-confirmed ones clear the bar.
| is code-assigned, never model-chosen: an out-of-lane observation is a handoff, not a | ||
| vetted finding, so it can never block on its own (and the `claim-validator` never | ||
| upgrades severity). Set the candidate's `source` to `"<agent> (out-of-lane)"`. From | ||
| here each one flows through the identical change-provenance gate → scope filter → |
Contributor
There was a problem hiding this comment.
thought (non-blocking): Nothing merges same-line candidates from different sources within a run (the only dedup layers are the thread-reconciler and the scope filter), so the motivating case — the skill-auditor hands off a correctness bug the correctness-reviewer also finds — can post the same line twice. Also suggested_lane is collected on the observation but never consumed in the routing paragraph: either use it for an ownership/dedup check or drop the field.
# Conflicts: # workflows/review/eval/corpus/live.ts
Contributor
Review Guidancegithub-actions (2 files)
Common patterns
Excluded from review (3 files)Not individually reviewed — generated, formatting-only, or fully explained by a common pattern above:
|
jwbron
added a commit
that referenced
this pull request
Jul 10, 2026
Phase 3 of the live A/B eval plan (#232), stacked on the Phase 2 producer (#234): score live runs against labeled ground truth and diff two review.md versions arm to arm. ## 3a: `eval/live-match.ts` Live runs choose their own finding ids, so the recorded metrics (which key on `expected.mustCatch` ids) cannot score them. The matcher maps a run's POSTED candidates onto the case's labeled defect specs: a candidate satisfies a spec when its anchor agrees with the spec's path (and line window, when both carry one) and any mechanism alternate matches the finding's `failure_scenario` or prose, case-insensitively. Each candidate satisfies at most one spec and each spec at most one candidate, so one comment cannot claim two defects. An injected fallback arbiter (hard-capped, same-file candidates only, recorded as `via: "fallback"` for human audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. `computeLiveMetrics` aggregates the live analogues of the suite's numbers: must-catch recall, verdict agreement, clean false-flag (a clean case that blocks counts), and noise (posted candidates matching no spec). ## 3b: `eval/live-ab.ts` The arm orchestrator and CLI. Baseline `review.md` comes from `git show <merge-base>` (or `--base-ref`), candidate from the working tree; both arms run over the same live corpus with everything else (corpus, `lib/`, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model-behavior seam. Each arm runs under half the `--max-usd` budget (default 40 total) with sticky exhaustion: once spend plus the running per-case average crosses the cap, the remaining cases are recorded as SKIPPED and the report still emits (dying at a cap with nothing posted is the failure mode the plan forbids). Each live result replays through the deterministic pipeline; `runner.ts` gains an optional `RunOptions.validation` override so the live validator's output replaces the recorded block. Spec-level regressions ("baseline caught, candidate missed" and the reverse) are diffed only over cases both arms actually scored, so a budget skip is never reported as a regression. Judge scoring reuses the pinned judge for quality aggregates only (judge-vs-ground-truth disagreement keys on recorded ids, which live arms do not use); the fetch model moves from `live-judge.ts` into shared `eval/judge-live-model.ts`. Output: `out/live-ab-report.json` plus a markdown table (metrics deltas, judge quality delta, cost, wall clock, regressions, skips, agent failures) printed and appended to `GITHUB_STEP_SUMMARY`. Report-only, with one exception per the playbook's standing rule: the candidate arm failing any adversarial-injection case (wrong verdict or missed spec) exits non-zero. ## Test plan: - `pnpm run test --run`: 696 tests green (16 new: matcher rules incl. window overlap, one-candidate-one-spec, malformed-regex fallback-to-literal, capped fallback audit trail; runArm budget semantics incl. the sticky-stop case that caught a real bug in review; regression diffing over shared cases; report rendering both gate outcomes). - `pnpm run typecheck` and eslint clean. - The end-to-end CLI (`pnpm dlx tsx workflows/review/eval/live-ab.ts --max-usd 10 --cases incident-money-rounding,clean-no-findings`) needs `ANTHROPIC_API_KEY`; this environment has none, so that is the first thing to exercise when testing. On an unchanged review.md it prints the identical-arms note and near-zero deltas. ## Next steps (human) 1. ~~End-to-end A/B validation~~ DONE, superseded by the phase 4 acceptance runs (linked on #237): the control arm held recall and verdict agreement flat at +0% and the weakened arm showed -17% recall with the lost spec named, which is also the first empirical read on matcher quality (7 cases matched; one systematic miss traced to lens routing, fixed on #233, not to the matcher). Two acceptance-driven additions landed here since: `perCase.failedAgents` entries now carry `<agent>: <reason>` (a claim-validator flake was undiagnosable from the report alone; the flake itself is still to be root-caused on a future run), and every report closes with a per-row stability footer (judge quality rose on the deliberately regressed arm because fewer, surer comments each score better, so the footer warns against reading it alone). 2. Review the two scoring caveats: judge output is quality-aggregates only (disagreement semantics key on recorded ids), and the fallback arbiter is wired but not yet given a live implementation (matches would be marked `via: "fallback"` for audit when one is added). 3. Confirm the budget semantics (half of `--max-usd` per arm, sticky skip-and-report) match how you want cost bounded. --- ## Update 2026-07-09: eval-tuning pass (three riders) Three instrument fixes from the [eval-tuning memo](https://claude.ai/code/artifact/2995f0b5-8840-4d03-96e3-2518e2f0f75a), landed here so the whole stack inherits them: 1. **Pre-flight identity short-circuit** (memo item 1, hoisted down from #251 with identical text so that branch merges cleanly): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; `--force-arms` preserved for wobble controls. Corpus-only and plumbing-only PRs in this stack now pay $0 per push. 2. **Gate-flip retry** (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake (the #244/#245 incident shape). Retry spend is recorded in the report. 3. **Drop-bucket taxonomy** (memo item 4): every missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/validation) by re-matching against the dropped candidates with the line window relaxed. Lost regressions are annotated with their class and the table gains a found-but-dropped count row. Plus **judge economics**: the judge pin moves from Opus 4.8 to `claude-haiku-4-5-20251001` (the judge grades prose only; every load-bearing metric is deterministic, and the acceptance runs showed judge quality is not single-run-stable on any model, so it is priced accordingly; supersedes operator direction 4), and the judge seam gains retry with backoff so a transient 500 no longer wastes a whole scoring pass. Author: jwbron Reviewers: jeresig, github-actions[bot], jwbron, jaredly Required Reviewers: Approved By: jeresig, github-actions[bot] Checks: ✅ 9 checks were successful Pull Request URL: #236
…' into jwbron/review-out-of-lane # Conflicts: # workflows/review/README.md
jwbron
changed the base branch from
jwbron/review-rereview-accountability
to
main
July 13, 2026 18:45
This was referenced Jul 21, 2026
jwbron
added a commit
that referenced
this pull request
Jul 30, 2026
…ice 2) (#282) * [jwies/review-scripted-dispatch] review: script-driven dispatch behind the ROUTING dispatch dial (orchestrator slice 2) * [jwies/review-scripted-dispatch] review: sync the workspace lockfile with the agent-sdk dependency * [jwies/review-scripted-dispatch] review: parse the validator's claims-array contract; dispute cap as a mechanical floor; feedback hardening * [jwies/review-scripted-dispatch-local] review: lenient contract extraction, findings-key tolerance, and the malformed-output retry in the dispatcher (trial run 29893634730) * [jwies/review-scripted-dispatch-local] review: honest dedupe-test naming; dispatch-directive arity/duplicate and runCli dispatchMode coverage (re-review feedback) * [jwies/review-scripted-dispatch] review: enforce the label contract in fromLabelShape (salvage near-misses, reject unknown labels) and join subject/discussion with a sentence break Trial run 29897276810's correctness-reviewer emitted a ReportFindings-style shape ({file, line, anchor, summary, severity, verdict, category, discussion}) instead of the label contract. The parse accepted it, defaulted label to the empty string with no anchor, and produced four label-less anchor-less claims that rendered as bare '**:**' paragraphs in the review body and demoted a BLOCKING correctness finding to advisory. Now fromLabelShape salvages the near-miss fields first (path from anchor.path/file, subject from summary), then rejects any finding whose label is not in the Conventional Comments vocabulary, so parseWithRetry re-dispatches the reviewer once with its corrective note instead of accepting a shape the contract never allowed. The same run's prose join glued subject to discussion with a bare space ("...memory Both TestExpiration..."); joinProse now inserts a sentence break when the subject lacks terminal punctuation, which also keeps buildClaims' first-sentence split recovering the subject. * [jwies/review-scripted-dispatch] review: merge cross-source duplicate claims before validation (#245) Trial run 29897276810 posted the AddDate months-vs-days defect four times (correctness, completeness, first-principles, and a skill-auditor out-of-lane handoff), and each copy was separately validated; claim validation is the largest sub-agent cost line, so duplicates now merge before the validator dispatch, not after. The merge is conservative and deterministic: different sources only, same path within a two-line window, and a token-similarity floor (Jaccard, overlap, and shared-bigram thresholds calibrated on that run's real outputs, where same-defect pairs scored at least 0.20 Jaccard and different-defect pairs on the same lines at most 0.05). The survivor is the highest-severity copy; it gains an 'also flagged by' note, adopts a merged duplicate's suggestion or author dispute when it lacks its own, and every merge is recorded in dispatch-result.json so run reports can count them. Replayed against the run's real out/ files this merges 13 claims to 9: the AddDate group collapses into the blocking correctness finding, and the missing-composite-index defect (flagged by correctness and holistic in different framings) collapses into one. * [jwies/review-scripted-dispatch] review: raise the per-sub-agent dispatch timeout to 15 minutes and record timeouts readably Trial run 29901690493 shed both default finders: the correctness-reviewer and skill-auditor were aborted at exactly the 5-minute DEFAULT_TIMEOUT_MS while every lighter reviewer finished in 60-115 seconds, and the staged error record read only 'Claude Code process aborted by user' (the SDK's generic abort message). The heavy investigators routinely need 5-10 minutes; the prior pin's correctness pass ran about 8 minutes to completion. The cap is a hang backstop, not a budget (credit spend is metered by the sandbox api-proxy), so it now sits at 15 minutes, and the CLI runner translates a timer abort into 'sub-agent timed out after Xms' so the out-file record and the run report name the cause. * [jwies/review-scripted-dispatch] review: salvage a label-valid finding whose failure_scenario is missing Trial run 29906543140's correctness pass twice emitted findings with valid labels, anchor objects, and rich discussions but no subject and no failure_scenario; the schema requires a non-empty failure_scenario, so both replies were rejected and the correctness dimension was voided (disclosed, but a voided default finder is strictly worse than a weaker fallback). The salvage now falls back subject -> discussion for the failure_scenario, so the claim-validator attacks the discussion prose instead of nothing. * [jwies/review-scripted-dispatch] review: salvage a title-keyed subject in the label-shape near-miss path Trial run 29908199997's correctness pass drifted to {id, file, anchor, label, category, title, discussion}: valid label, but the subject lived under 'title', so the salvage produced an empty subject and (pre-c35675f) an empty failure_scenario, voiding the dimension a third time. 'title' now joins 'summary' in the subject fallback chain. * [jwies/review-scripted-dispatch] review: salvage suggested_patch in the label-shape near-miss path fromLabelShape read only the contract's `suggestion` key, so trial run 29943085279's correctness pass, which drifted into its ReportFindings-style shape (defect-13 lineage), salvaged the AddDate finding but dropped the one-line fix it carried under `suggested_patch`; the posted comment (Khan/webapp#41029) had no suggestion fence. Accept a non-empty `suggested_patch` when `suggestion` is absent, exactly like the anchor and summary/title salvages beside it; `suggestion` still wins when both are present. * [jwies/review-scripted-dispatch] review: drop the same-path line window in the cross-source duplicate merge dedupeClaims required anchors within two lines of each other, so trial run 29943085279 posted the missing-deletion-test defect four times in one review (expiration_test.go:15 and :58, expiration.go:62 and :38). For same-path pairs the calibrated similarity floor (describesSameDefect) now carries the precision alone, the stance suppressOpenThreadDuplicates already takes and documents. Replayed against that run's claims.json this merges exactly the :15/:58 pair and nothing else; the AddDate issue and the untested-behavior thought on the very same line stay unmerged (the false-merge guard), and the test-adequacy/first-principles pair scores below the floor and stays two comments. Cross-file merging stays out of scope: it needs its own strictly higher calibration, and a missed merge only costs a duplicate comment. * [jwies/review-scripted-dispatch] review: star-guard the duplicate merge so a chain-only member never drops Union-find chains A~B~C through a bridging claim that bundles two defects (measured: a test-adequacy finding naming both the missing test and the unbounded read clears the floor against each correctness finding separately while the two never clear it against each other), and the collapse silently dropped a distinct blocking finding; with the same-path line window gone the bridge can span a whole file. A group member now merges only when it clears describesSameDefect against the survivor directly; chain-only members stay their own claims (a missed merge only costs a duplicate comment). Both recorded trial merges are unaffected: run 29897276810's four-way group is pairwise-complete, run 29943085279's is a direct pair. * [jwies/review-scripted-dispatch] review: stage dispatch-result.json under out/ so the artifact carries it runDispatch wrote dispatch-result.json only to /tmp/gh-aw/review, which the Step 9 artifact upload does not include, so run 29943085279's post-hoc could not tell whether the correctness output arrived through the submit_result tool or the text fallback: perAgent (structuredFinal, retried, per-agent usd), merges, and threadSuppressions never left the runner. The same content now also stages as out/dispatch-result.json. The dispatch gate reads out/ files by fixed or routing-planned name only (rereview-plan.json already stages there as a non-agent file), so the extra file changes no gate rule. * [jwies/review-scripted-dispatch] review: malformed reviewFiles entries fail toward triage-unavailable The dispatcher filtered triage's reviewFiles to strings and treated the filtered-empty result as an empty review, dispatching no finders and rendering no disclosure, while the gate's empty-reviewFiles waiver reads the RAW staged array (dispatch-gate.ts triageEmptiedReview): a triage output like reviewFiles: [1, 2] therefore produced a guaranteed false block (rule 1 correctness-missing has no disclosure escape, plus shed-undisclosed for every planned extra) on a conforming dispatcher. A non-empty array with zero string entries now takes the triage-unavailable path: review everything, disclose the skipped triage. A genuinely empty array still empties the review and the gate waiver still applies. * [jwies/review-scripted-dispatch] review: show the correctness reviewer one complete example finding The correctness pass emitted the ReportFindings-style shape in run 29943085279 even with the submit_result instruction appended (defect-13 lineage), and the salvage path is demonstrably lossy (the same run's suggested_patch drop). The output contract now carries one filled-in example finding naming the required keys and the drift keys not to substitute. Format emphasis only; no review rule or severity guidance changes. Verified with gh aw compile against a scratch consumer layout (.github/aw/review/config.md staged): 0 errors, 0 warnings. * [jwies/review-scripted-dispatch] review: record the trial hardening in the scripted-dispatch changeset * [jwies/review-scripted-dispatch] review: recalibrate cross-source claim dedup on the 30301235749 trial Run 30301235749 (Khan/webapp#41118) had five sources flag one TTL-unit defect at expiration.go:38, merged none of them, and posted three blocking comments plus a note for a single defect; it also made one false merge, folding the skill-auditor's AddDate handoff at :38 into the unrelated missing-test todo at :62. Replaying the module over that run's real pre-dedup claims (out/claims.json is written AFTER the merge, so the input is that file with the recorded merge undone and the skill-auditor handoff restored from out/skill-auditor.json) shows the floors were not the cause. A label-shape reviewer that omits failure_scenario is handed its own subject back as one, so the default correctness pass, a one-line title with all of its evidence in discussion, reached the comparison as eleven content tokens repeated twice and scored 0.13-0.23 Jaccard against the four discursive copies on a 0.20 floor. No threshold can fix that: run 29943085279's distinct same-line issue/thought pair (0.192/0.350/3) outscores two of the true duplicates on both ratios, so any floor loose enough to catch the cluster admits it. A claim whose failure scenario only restates its subject is now compared on its discussion instead. Only those claims: feeding the discussion to every claim was measured and rejected, since it lifts that same-line negative to ten shared bigrams, above every real duplicate. The floors then split in two tiers, an identical (path, line) from two sources being evidence of one defect while a differing line is weak evidence of two: exact anchor at 0.14/0.34/4, any other line on the same path at 0.20/0.35/6. The tighter other-line tier is what rejects the false merge (five shared bigrams; both real different-line duplicates share six or more). Replayed over the two runs whose pre-validation claim sets survive as artifacts, all 58 same-path cross-source pairs land on ground truth: 30301235749 collapses its five :38 copies into the correctness issue and keeps the :62 todo, 29897276810's three-way :38 group merges exactly as before, and no other pair moves. Nothing over-merges. Run 29943085279 staged no claims.json, so its pairs stay fixture-held, checked against the comments it actually posted. Both floors are minima over real claims and the margins are one unit (4 bigrams against 3, 6 against 5); dedup.test.ts carries the run's real claim texts so the next recalibration re-derives them rather than nudging them. * [jwies/review-scripted-dispatch] review: validate the validator's corrected fields per key `applyVerifications` applied every `corrected` field through one string-or-number check, so `corrected.label` could drift out of the label vocabulary and `corrected.line` could land a string in a numeric anchor. An out-of-vocabulary label on a confirmed BLOCKING claim fails `isBlockingLabel` and silently drops it from the REQUEST_CHANGES verdict: the drift class `fromLabelShape` already gates on the finder side, left ungated on the validator side. Validate per key and keep the finder's original when a correction is rejected. * [jwies/review-scripted-dispatch] review: split the self-contained glob matcher and dispatch-mode tests out main's lens-payload seam (#281) left router.ts at 999 lines against the 1000-line cap, so this branch's twelve lines of dispatchMode plumbing breached it, and the unioned dispatch-mode tests did the same to router.test.ts. Move the section already marked self-contained into glob-match.ts (re-exported from router.ts so no importer changes) and the dispatch-directive CLI tests into router-dispatch-mode.test.ts, the same split dispatch-contracts.ts took from dispatch.ts.
jwbron
added a commit
that referenced
this pull request
Jul 30, 2026
…ad suppression, deterministic cache record (#288) * [jwies/review-decision-chokepoints-local] review: rule 5 anchors on the cache-memory stamp; rule 6 reads the reconciler out-file leniently * [jwies/review-trial-followups] review: post-trial follow-ups; structured sub-agent finals, open-thread suppression, deterministic cache record The three highest-value in-repo suggestions from the lifecycle trial's consolidated report (suggestions h, g, and b on #284): - Structured sub-agent finals: every scripted dispatch exposes an in-process submit_result MCP tool whose input is validated against the agent's exact output contract at the tool boundary; a drifted shape is rejected back to the model in-session with the precise contract error (defect 13: three drift shapes in five runs, two voided correctness dimensions). Free-text finals remain the fallback. - Open-thread suppression at dispatch: a candidate describing a defect an open bot thread already tracks posts no duplicate (same path plus the #245 similarity floor; reconciler-resolved threads exempt); a suppressed blocking candidate still floors the verdict at REQUEST_CHANGES. - Deterministic Step 9 cache write: in scripted mode the orchestrator invokes lib/cache-record.ts once after emitting; the fingerprint-carrier fields are copied verbatim from staged files and corroborated against the safe-output queue. risksPatternsKey becomes code-owned end to end. Verification: 1254 tests green, typecheck and eslint clean, frontmatter compiles clean under gh-aw v0.81.6 (gate step ordering verified against the compiled lock; the cache writer cannot be a post-step because the compiled agent job uploads cache-memory before post-steps run). * [jwies/review-trial-followups] review: address #288 review feedback - computeRisksPatternsKey: the risk-tier filter matched the literal "moderate" while the triage contract emits Medium/High; every medium-risk file was silently dropped from the signature, so Step 7 could not detect a guidance change confined to medium-risk files. Accept medium (moderate stays tolerated) and pin the contract vocabulary in the fixtures. - Open-thread suppression floor: record the matched thread opener's blocking-ness (threadBlocking) and floor the verdict only when both the suppressed candidate and the thread opener are blocking. A false-positive blocking candidate matching an open non-blocking thread can no longer force REQUEST_CHANGES with no validation and no visible blocking comment. - risks-patterns-key.txt now stages at full depth only. Step 7 skips every reduced depth including scoped, and a scoped run's triage sees only the scoped subset; staging that narrower signature could collapse the standing full-run guidance. - The cache-record CLI emits ::warning on corroboration/staging refusals (a systematic refusal permanently stales the fingerprint); the benign no-ops (task mode, gate-blocked) stay quiet. - readQueue parses the safe-output JSONL per line, so one truncated tail line no longer discards the whole queue. - dispatch-runner gets direct tests via a mocked Agent SDK: the submit_result accept/reject handler, the structured final beating the free-text final, and the salvage of an accepted payload after the session dies. * [jwies/review-trial-followups] review: enforce bot-authored openers on open-thread suppression in code threads.json staging is prompt-executed (review.md asks the orchestrator for the unresolved github-actions[bot] threads; stage-pr deliberately does not stage threads yet), and the dispatcher read the opener body without ever reading its author, so a mis-staged human thread on the same path could silently suppress a bot candidate before validation; its free-text opener would also read as non-blocking and skip the verdict floor. The OpenThread build moves to openThreadsFromStaged (dedup.ts, beside its consumers) and admits only threads whose opening comment the bot authored, the same code-level stance stage-pr already takes for prior-reviews.json. Fails closed: a thread without a bot opener never suppresses, worst case a duplicate comment. Eval staging (live-stage.ts) already writes bot-authored openers. * [jwies/review-trial-followups] review: record the suppression hardening in the followups changeset * [jwies/review-trial-followups] review: address the reviewer feedback on the trial follow-ups Three items from the bot review on #288. 1. `threadOpenerIsBlocking` matched a hand-rolled `\(blocking\)`, which cannot match `issue (blocking, best-practice)`, a real BLOCKING_LABELS entry. A suppressed re-confirmation of such a thread recorded `threadBlocking: false`, the both-blocking floor in submission.ts never fired, and the run could flip to APPROVE over a still-open, re-confirmed blocking objection: exactly the invariant the suppression adds. The opener's label is now parsed off the template and classified by `isBlockingLabel`, the taxonomy's own rule (spacing inside the parens normalized first). 2. Open-thread suppression depended on staged threads being unresolved, but only review.md prose enforced that, the same gap the bot-authorship hardening closed. `get_review_comments` returns `is_resolved` per thread, so review.md now stages it as `resolved` and `openThreadsFromStaged` admits only threads that carry an explicit false. Fails closed like the author check: worst case is a duplicate comment. 3. The cache-record writer trusted the plan alone when no safe-output queue was readable, citing the dispatch gate. The gate runs on the post-ingest agent_output.json, after cache-memory is committed, so it cannot protect the record: a run whose emission silently failed recorded "review posted" with the current fingerprints, and the next run stamped against an unreviewed diff. An unreadable queue is now a refuse-with-warn, except the one shape that legitimately queues nothing (the Step 6 redundant-approval skip), keeping the failure pointed at a fuller next review. Suite green (1278 tests), typecheck and lint clean. * [jwies/review-trial-followups] review: fold the NOTIFIED signature into the staged guidance key review.md's Step 7 defines the canonical risks/patterns signature to include `notified.json`'s signature and treats the NOTIFIED match set as an independent repost trigger, but `computeRisksPatternsKey` built the key from risks, patterns, exclusions and owners only. Because the key is staged at Step 4, a NOTIFIED-only change produced an identical key on both sides of the Step 7 compare: the guidance comment never reposted and a newly subscribed `.github/NOTIFIED` @team was silently never mentioned, with the notified-less key persisting as the baseline. * [jwies/review-trial-followups] review: compute the NOTIFIED signature in the submission CLI The NOTIFIED fold was dead at runtime in scripted mode. `runSubmissionCli` read `notified.json`, but the notified CLI does not run until Step 7 and nothing stages that file earlier, so `signature` was always undefined and `computeRisksPatternsKey` dropped the `notified:` component on every real run. A `.github/NOTIFIED`-only change then staged a key identical to the prior run's, Step 7 read the guidance as unchanged, and a newly-subscribed team was never mentioned: exactly the failure the fold was added to fix. The existing test passed only because it injected the signature directly. Compute it here instead of depending on step ordering. `notified.ts`'s `runCli` is pure over staged inputs the pre-agent step writes as hard prerequisites (`files.json`, `full.diff`), so it is safe to call this early, and it stages `notified.json` as a side effect, which makes Step 7's own invocation idempotent rather than first-of-its-kind. The tests go in a new `submission-notified.test.ts` rather than another block in `submission.test.ts`, which sits within ten lines of the 1000-line max-lines budget and went over when they were added there. None of them injects a signature, which is the point: one asserts the component lands without `notified.json` staged first, one asserts a NOTIFIED-only change moves the key (the assertion that fails against the read-from-file version), and one asserts a repo without the file still stages the pre-feature key.
Merged
jwbron
added a commit
that referenced
this pull request
Jul 31, 2026
…rds cannot validate a comparedText change Run 30650642317 raised a second question about dedup precision. Three claims from three sources described one defect (the zero-value doc/panic contradiction) and the run merged none of them, while suppression matched all three to the same open thread on a STRICTER floor: clearing the harder bar and failing the easier one points at the text basis, not the thresholds. `comparedText` feeds merge `subject + failure_scenario` (or `subject + discussion` only when the failure restates the subject), so the evidence a reviewer writes in `discussion` is usually absent from the merge comparison and present in suppression's. Reading `discussion` unconditionally merges two of those three pairs (correctness/documentation 2 -> 12 shared bigrams, first-principles / documentation 2 -> 4 at the same line) and leaves the whole suite green, which looks like a free win and is not one: the three false-merge guards leave `discussion` at the `claim()` default "d", one character the tokenizer drops, so the candidate is a no-op for them, and dedup.ts's own comment records the opposite outcome on the real claims (run 29943085279's distinct same-line issue/thought pair reaching ten shared bigrams). So this documents the gap instead of shipping the change. Suppression masked the cost on that run; against a resolved thread the same three claims post three comments for one defect, the #245 problem. Re-deriving the tradeoff needs the real claim sets for runs 29943085279 and 30301235749 with their discussions, which these fixtures do not carry.
This was referenced Jul 31, 2026
Merged
jwbron
added a commit
that referenced
this pull request
Jul 31, 2026
… the first staged one (#310) * [jwies/suppression-best-match] review: attribute a suppression to its best-matching open thread, not the first staged one Open-thread suppression took the FIRST staged thread a candidate cleared the similarity floor against. `threadBlocking` is read off that thread and feeds submission.ts's verdict floor, so staging order could decide a verdict. Measured on webapp#41204 run 30650642317, the first live run where suppression fired at all: the fresh test-adequacy todo ("No test covers Record's documented zero-valued-Window guarantee") cleared the floor against both the blocking nil-map panic thread and its own exact counterpart, a non-blocking nitpick whose opener reads "The documented zero-value-Window guarantee has no test". The blocking thread was staged first because it was older, so the suppression recorded threadBlocking: true and added a second entry to the floor. That run's REQUEST_CHANGES was already correct on another suppression's strength, so nothing was mis-decided, but the direction is the dangerous one, and the same accident can hand a candidate a non-blocking thread when a blocking one is its real counterpart, silently lowering the floor instead. Ranked on jaccard first because it is the length-normalized metric: raw sharedBigrams grows with the opener's length and overlap divides by the smaller token set, so both favor the longer, more discursive thread. On the observed pair jaccard (0.253 vs 0.218) and bigrams (13 vs 12) both pick the true counterpart while overlap alone (0.468 vs 0.511) picks the wrong one, so bigrams break jaccard ties and overlap never ranks. Comparisons are strictly-better, so staging order stays the final tiebreak and dispatch-result.json stays reproducible. The regression case stages the blocking thread first, as the live run did, and asserts the same attribution with the pair staged either way round. * [jwies/suppression-best-match] review: record why the false-merge guards cannot validate a comparedText change Run 30650642317 raised a second question about dedup precision. Three claims from three sources described one defect (the zero-value doc/panic contradiction) and the run merged none of them, while suppression matched all three to the same open thread on a STRICTER floor: clearing the harder bar and failing the easier one points at the text basis, not the thresholds. `comparedText` feeds merge `subject + failure_scenario` (or `subject + discussion` only when the failure restates the subject), so the evidence a reviewer writes in `discussion` is usually absent from the merge comparison and present in suppression's. Reading `discussion` unconditionally merges two of those three pairs (correctness/documentation 2 -> 12 shared bigrams, first-principles / documentation 2 -> 4 at the same line) and leaves the whole suite green, which looks like a free win and is not one: the three false-merge guards leave `discussion` at the `claim()` default "d", one character the tokenizer drops, so the candidate is a no-op for them, and dedup.ts's own comment records the opposite outcome on the real claims (run 29943085279's distinct same-line issue/thought pair reaching ten shared bigrams). So this documents the gap instead of shipping the change. Suppression masked the cost on that run; against a resolved thread the same three claims post three comments for one defect, the #245 problem. Re-deriving the tradeoff needs the real claim sets for runs 29943085279 and 30301235749 with their discussions, which these fixtures do not carry.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Third of five PRs in the fold-in batch two stack (stacked on #244; the stack is based on #238,
jwbron/review-trial-skill). The top of the batch-two stack will bejwbron/review-skill-rule-quote.In run 2 of the v1.4.0 re-run lifecycle the skill-auditor found a real issue (dedup reads via an eventually-consistent Query immediately after PutMulti) and dropped it, verbatim: "that's a correctness concern, not a quotable skill-rule violation, so I'll leave it". That was the correct call under quote-the-rule, but the observation died with it.
Failure scenario fixed: a reviewer investigating a rule notices a genuine correctness defect, correctly declines to report it as a skill violation, and has no other channel; the defect reaches the PR author in no form and ships.
The handoff:
out_of_lane_observations[]alongsidefindings[]:path, optionalline, the concern stated concretely, a required concretefailure_scenario(the claim the validator attacks, same discipline as findings), and an optionalsuggested_lane.question (non-blocking); the label is never model-chosen, because a handoff is not a vetted finding and must not block on its own (and the claim-validator never upgrades severity). Candidates then flow through the identical provenance gate, scope filter, claims.json, and validation path as every other candidate; validation into the correctness lane, rather than re-dispatching the correctness reviewer, keeps the handoff inside the run's existing budget.validateOutOfLaneObservation/isValidOutOfLaneObservationinlib/finding-schema.ts, with the same collect-every-error contract asvalidateFindingso a malformed handoff is diagnosable from the run artifact. It is a sibling type, not aFindingfield, soFINDING_SCHEMA_VERSIONstays 2 and no serialized finding is invalidated.The per-lens handoff paragraph and JSON field are stamped into each of the 11 lens definitions (mechanically, identical text); the next PR in the stack (dispatch-tax trim) folds exactly this kind of repeated block into the shared staged disciplines context, so the duplication added here is transitional within the same stack.
cc @jeresig
Testing
npx vitest run workflows/review: 484 tests green (7 new observation-schema tests: minimal and full shapes, required fields, error collection, line bounds).npx tsc --noEmitclean.