From ec38c46f18e17348643b64fa48d3fd4e485037f6 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Thu, 20 Aug 2026 20:23:08 +1200 Subject: [PATCH 1/3] docs(launchpad): correct #118's plan against #117's now-merged state The plan's ALREADY TRUE and BUDGET sections, written 2026-08-13, assumed none of #120 or #117 had landed -- 'THE PRODUCER OF THIS STAGE'S INPUT DOES NOT EXIST', 'no findings.py, no run_dimensions.py'. As of today every one of #117's 12 steps is built, and 11 of 12 are merged to launchpad (STEP 8's 15 recorded reviewer outputs are committed on this exact branch, PR #252 open). Corrected in place, struck through rather than deleted, per this plan's own citation-rot discipline (the same convention it already uses for its own prior corrections): the precondition chain #120 -> #117 -> #118 is discharged, not pending, and the actual risk STEP 8/9 now carry is the OPPOSITE of what was documented -- real recorded dimension output already exists and should be the primary input for #118's own fixtures, not document-only synthesis. Also fixed one stale line citation (review.py:62 -> review.py:86, verified against the actual file rather than trusted). STEP 8/9's own text still describes synthesizing fixtures from the contract document alone; BUDGET's correction flags this explicitly so whoever implements those steps updates that text rather than building against it as written. No steps renumbered, no done-when criteria changed -- this is a documentation correction only, ahead of implementation. Signed-off-by: Serina Mcfall --- .../2026-08-13-issue-118-adjudication.md | 66 +++++++++++++++---- 1 file changed, 52 insertions(+), 14 deletions(-) diff --git a/launchpad/plans/2026-08-13-issue-118-adjudication.md b/launchpad/plans/2026-08-13-issue-118-adjudication.md index 15f270a2bc2..5fd632825d9 100644 --- a/launchpad/plans/2026-08-13-issue-118-adjudication.md +++ b/launchpad/plans/2026-08-13-issue-118-adjudication.md @@ -21,12 +21,10 @@ each observable on its own — splitting is the reader's call, not this plan's: Planned as written below. ALREADY TRUE (verified against git, the working trees and the GitHub API, not notes) - Nothing of #118 is built. Branch feat/review-agent-adjudication is at e14f5fafb + ~~Nothing of #118 is built. Branch feat/review-agent-adjudication is at e14f5fafb and `git rev-list --left-right --count origin/launchpad...HEAD` reports 0 0. - `git ls-files | grep -iE 'adjudic|review-agent'` matches nothing. - `launchpad/plans/` does not exist on this branch — it is created by this plan. - Toolchain present: python3 3.12.3, gh 2.93.0. - THIS ISSUE SITS AT THE END OF A THREE-LINK CHAIN, AND NONE OF IT IS MERGED. + `git ls-files | grep -iE 'adjudic|review-agent'` matches nothing.~~ + ~~THIS ISSUE SITS AT THE END OF A THREE-LINK CHAIN, AND NONE OF IT IS MERGED. #120's containment tree — contain.py, fetch.py, review.py, run_controls.py and CONTAINMENT.md — is three pushed commits on `feat/review-agent-untrusted-input` (618789584, e072fba55, c64ff7958) and `git rev-list --left-right --count @@ -41,7 +39,28 @@ ALREADY TRUE (verified against git, the working trees and the GitHub API, not n no dimension definition anywhere on it. So the contract this stage consumes exists ONLY as a normative plan document, and no producer of it has ever run. See BUDGET — this is the defining risk of #118 and - it is different in kind from the one #117 carried. + it is different in kind from the one #117 carried.~~ + **Corrected 2026-08-20, one week after the above was written — struck through, + not deleted, per this plan's own citation-rot discipline below.** #120's + containment tree and every one of #117's 12 steps are now BUILT AND MERGED to + `origin/launchpad`, except #117's own STEP 8 (15 recorded reviewer outputs), + which is committed and pushed on `feat/review-agent-recordings-v2` (PR #252, + open) — this plan is written from a worktree branched off that exact tip, so + STEP 8's recordings are directly available to this issue's own STEP 8/9 as real + producer output, not a hypothetical. Verified today, not assumed: `git + rev-list --left-right --count origin/launchpad...HEAD` from this worktree + reports `0 1` — this branch is `origin/launchpad` plus exactly that one + commit. `git ls-tree -r origin/launchpad -- launchpad/review-agent/` lists + `findings.py`, `run_dimensions.py`, `contain.py`, `fetch.py`, `review.py`, + `FINDINGS.md`, `CONTAINMENT.md`, three files under `dimensions/`, five under + `fixtures/dimensions/`, and (on this branch only, pending #252) fifteen under + `recordings/`. `find . -iname "*adjudic*"` still returns nothing anywhere in + the tree — of everything above, only #118 itself remains unbuilt, which is + the one claim from the struck-through block that still holds. **The defining + risk BUDGET names below (a contract with no producer) is now a non-issue**: + 15 real recorded dimension reports already exist and are the right input for + STEP 8/9's fixtures, not a document-only synthesis. See BUDGET's own + correction for what that changes. #117's contract is settled at its fourth revision and this plan honours it without renegotiating. Ten finding fields — dimension, severity, anchor, file, line, defect, failure, finding_id, entry_point, evidence. Eleven envelope @@ -104,8 +123,10 @@ ALREADY TRUE (verified against git, the working trees and the GitHub API, not n with `review.SEVERITY_ORDER.get(finding["severity"], 9)` and renders an out-of-ladder severity under its "malformed finding" heading, sorting it last and triggering the incomplete banner — its review's finding 2 raised the bare - subscript and its Outcomes table records the fix. `review.py:62` uses `.get( - f.severity, 9)` for the same reason. + subscript and its Outcomes table records the fix. ~~`review.py:62`~~ + **(corrected 2026-08-20: `review.py:86`, per this plan's own citation-rot + rule — checked against the actual file, not the working line number pinned + when this was written)** uses `.get(f.severity, 9)` for the same reason. This plan cited the bare subscript as live, and that citation was true when written and false forty minutes later: `feat/review-agent-publish` committed 47482549e at 2026-08-13 08:48:13 revising the plan under this plan's feet. The @@ -1004,7 +1025,7 @@ GATES No verify gate is installed in this checkout — .claude/settings.json an harness. STEP 11 now pairs each control with a named, targeted mutation of the production code it will eventually test, on that harness's model. -BUDGET STEP 9 eats the budget. The thing most likely to derail the issue is +BUDGET ~~STEP 9 eats the budget. The thing most likely to derail the issue is STEP 8, and it is a different risk from the one #117 carried. THE PRODUCER OF THIS STAGE'S INPUT DOES NOT EXIST. #117 depended on #120's tree, which was at least written and pushed — signatures could be re-verified against @@ -1020,14 +1041,31 @@ BUDGET STEP 9 eats the budget. The thing most likely to derail the issue is branch with review.py, contain.py, run_controls.py AND #117's findings.py, obtained by those issues merging to launchpad and this branch rebasing — never by copying, which creates a second source of truth and a guaranteed conflict when - they land. Before STEP 2, re-verify review.SEVERITY_ORDER's location and #117's + they land.~~ + **Corrected 2026-08-20 (struck through, not deleted — see ALREADY TRUE's own + correction above, same date).** #120 and #117 are BOTH MERGED to `origin/launchpad` + (every step except #117's own STEP 8, which is committed on the branch this plan + is written from, PR #252 open). This branch needed no rebase to get + `launchpad/review-agent/` — it was already there, cloned from `origin/launchpad` + plus one commit. The precondition chain is DISCHARGED, not pending. + **The actual risk STEP 8/9 now carry is the opposite of the one described above:** + not "no producer exists to synthesize fixtures from," but "real recorded dimension + output already exists (#117's own STEP 8, 15 files under `recordings/`) and STEP + 8/9 below must be built to CONSUME it as the primary input, not to re-synthesize + fixtures from ADJUDICATION.md alone as originally planned." Re-read STEP 8/9 below + with that in mind before implementing; if their text still describes + document-only synthesis, that text is what needs updating, not this correction. + Before STEP 2, re-verify review.SEVERITY_ORDER's location and #117's field names against what has actually been committed by then rather than trusting - the citations in ALREADY TRUE. That instruction paid for itself twice on #117, - where a stale measurement and a moved symbol both invalidated a step's - specification between drafts. + the citations in ALREADY TRUE — worth repeating even though the chain is now + merged, since a field could still move before this plan's own STEP 2 lands. STEP 9 is five recordings plus four before-and-after pairs, each needing a real run against a real model, and it is the step where "recorded from a real run" - quietly becomes "hand-written to look like one". STEP 3 decides its cost: if the + quietly becomes "hand-written to look like one" — #117's own STEP 8 found exactly + this failure mode in its own first draft (a review caught byte-identical prose + across three supposedly-independent dimension recordings) and fixed it by + disclosing the sampling method honestly rather than fabricating variation; the + same discipline applies here. STEP 3 decides its cost: if the judge is not an injected callable — if a model call is hardcoded at its call site — then STEP 9 stops being recording and becomes a rewrite of STEPs 3, 6 and 7. From cbe13bb5906aac2bcd01a8727c2aeb9fc26d54f0 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Thu, 20 Aug 2026 20:37:31 +1200 Subject: [PATCH 2/3] feat(launchpad): ADJUDICATION.md -- the verdict contract (#118 STEP 1) Adds launchpad/review-agent/ADJUDICATION.md, a sibling to FINDINGS.md and CONTAINMENT.md in the same normative voice: the contract #119 will read once #118's adjudication stage runs. Settles: the verdict enum (CONFIRMED/REFUTED/UNPROVEN, input/output finding_id SETS equal, UNPROVEN the default and REFUTED never one); six added finding fields (verdict, verdict_evidence, reported_severity, severity, severity_reason, duplicate_of); three escalate-never-approve prohibitions (no approval-carrying field, a REFUTED finding still published, a downgrade allowed but visible); the nine-key adjudication block with completion_marker last, carrying #117's own nonce rather than minting one; dedupe (survivor-selection rule, both- directions linking, duplicates still individually verdicted); total refutation as a flag rather than a silent clean pass; and containment findings passed through byte-identically, never adjudicated. Also updates CONTAINMENT.md's 'Contract for later stages' table to point at this document from #118's existing row, without touching that row's literal 'must call contain.findings_for' prescription -- the plan's own OPEN section flags that prescription as unsatisfiable from this stage's actual JSON-on- stdin input, and explicitly defers correcting the table to whoever owns #120. ADJUDICATION.md names the mismatch plainly and states the deferral rather than silently resolving a decision the plan marks as not a builder's to make. Ahead of this commit, corrected the plan itself (launchpad/plans/ 2026-08-13-issue-118-adjudication.md, previous commit): its ALREADY TRUE and BUDGET sections, written before #117 existed as code, are now stale -- #117's 12 steps are all built, 11 of 12 merged. Struck through per the plan's own citation-rot convention, not deleted. Reviewed independently (serina:review-code) before commit: two findings, one Medium (a 'see section Dedupe' cross-reference to a section that did not exist -- fixed by adding one, covering the survivor rule and both-directions linking from the plan's STEP 7) and one Low (findings_out==findings_in described as proving no silent drop, when only the stronger finding_id set-equality claim actually guards a drop-and-invent swap -- reworded to state which guarantee is load-bearing). Both fixed and re-verified. Signed-off-by: Serina Mcfall --- launchpad/review-agent/ADJUDICATION.md | 206 +++++++++++++++++++++++++ launchpad/review-agent/CONTAINMENT.md | 2 +- 2 files changed, 207 insertions(+), 1 deletion(-) create mode 100644 launchpad/review-agent/ADJUDICATION.md diff --git a/launchpad/review-agent/ADJUDICATION.md b/launchpad/review-agent/ADJUDICATION.md new file mode 100644 index 00000000000..b2a2e375bb1 --- /dev/null +++ b/launchpad/review-agent/ADJUDICATION.md @@ -0,0 +1,206 @@ +# ADJUDICATION.md — the adjudication stage's verdict contract + +Normative, and a sibling to `CONTAINMENT.md` and to `FINDINGS.md` in the same voice. +This is the document [#119](https://github.com/launchpad-26/buzz/issues/119) reads to +know what arrives once [#118](https://github.com/launchpad-26/buzz/issues/118) +(adjudication) has run. + +## The verdict + +Exactly one verdict per finding, from `CONFIRMED | REFUTED | UNPROVEN`. "Exactly one" is +the whole criterion: a finding present on input and absent from output is a defect in +this stage, not a tidy-up — the contract states that the input and output `finding_id` +sets are **equal**, not that output is a subset of input. + +**The default is `UNPROVEN`, never `REFUTED`.** Absence of confirmation is not +refutation. An adjudicator that cannot reach the location a finding names, cannot parse +the finding, times out, or returns unusable output yields `UNPROVEN` with a reason. This +is the fail-closed direction: a wrongly-`UNPROVEN` finding still reaches a human; a +wrongly-`REFUTED` one reaches them wearing a dismissal. + +## The added finding fields + +On top of `FINDINGS.md`'s ten. Names are final here: + +| field | meaning | +|---|---| +| `verdict` | `CONFIRMED \| REFUTED \| UNPROVEN` | +| `verdict_evidence` | what the adjudicator established, in its own words — required and non-empty on **all three** verdicts, including `REFUTED` and `UNPROVEN`. An `UNPROVEN` with no reason is indistinguishable from a stage that skipped the finding. | +| `reported_severity` | the reporting dimension's value, preserved verbatim | +| `severity` | the **re-rated** value — `FINDINGS.md`'s field, overwritten here | +| `severity_reason` | required whenever `severity != reported_severity` | +| `duplicate_of` | `finding_id` of the survivor, or null — see § Dedupe, below | + +`severity` carries the re-rating and `reported_severity` preserves the original because +#119 ranks by `finding["severity"]`, so this field name decides what the published review +leads with. The dimension's original value stays readable beside it, which is what +#118's issue requires — "readable", not merely "recoverable from history" — and the two +are only the same field if #119 changes what it renders. + +**Severity is re-rated on every finding, including `REFUTED` ones.** The rating answers +"how bad if true", which is a separate question from "is it true". Rating only the +confirmed findings would leave `REFUTED` findings carrying an unexamined severity into +#119's sort. + +### `severity` and `reported_severity` must both be in `review.SEVERITY_ORDER` + +Stated as a guarantee this stage makes to #119, and as **defence in depth** rather than +#119's only defence: #119 sorts with `.get(severity, 9)` and routes an unrecognised +severity to its own malformed-finding heading, so a bad value is survivable there — but +this stage is the one that **creates** bad values by re-rating, and a producer that +relies on its consumer's default has moved the failure rather than removed it. + +The guarantee is on the **effective** severity — the re-rating where one was made, +`reported_severity` where none was. Both must be checked: a finding arriving with an +out-of-ladder `reported_severity` that the judge happens to agree with is never +re-rated at all, so a guard watching only re-ratings never fires and the bad value is +copied into `severity` untouched. + +**This cannot happen if the input document is validated first, and this stage does so.** +`run_adjudication.py`'s `main` runs #117's own `findings.validate` against the input +document *before* any adjudication logic touches it, and exits non-zero — adjudicating +nothing — when validation fails. An input carrying `reported_severity: "Info"` fails +`FINDINGS.md`'s own severity rule and never reaches this stage's re-rating logic at all. +This is stronger than "this stage is agnostic about its producer": #119 is agnostic +about *its* producer because #119 cannot re-validate a document it did not assemble from +parts, while this stage *can*, because its input document is exactly #117's own output +shape and the validator that checks it already exists. Refusing outright is also the +only answer that needs no invention: there is no legal value to preserve +`reported_severity` *as*, once it already arrived broken, and inventing one would be this +stage silently deciding what the dimension actually meant. + +So an out-of-ladder **effective** severity this stage can still produce — one its own +re-rating created from a legal `reported_severity` — is an adjudication **failure** for +that finding: `UNPROVEN`, `severity` set to the nearest legal value with the refusal +stated in `severity_reason`, and `reported_severity` left unchanged (input validation +already guarantees it was legal to begin with). This stage may not silently decide an +unrateable finding is a small one, and it is never asked to invent a value for a field +that arrived already broken. + +## Escalate, never approve + +Stated as three concrete prohibitions rather than a slogan, because a slogan is not +checkable: + +1. **No field in this contract can carry an approval, a merge recommendation, or a + pass.** There is no `approved`, no `mergeable`, no `verdict: OK`. A judge cannot emit + what the record cannot hold. This binds field names and enumerated values, **not free + text**, and the limit is stated because it is otherwise invisible: `verdict_evidence`, + `severity_reason`, and `notes` are free strings, and nothing mechanical stops a judge + writing "this looks fine to me, recommend merge" into one. A control that grepped + them for approving phrasing would itself be a keyword filter — the kind of narrow + guard the review dimensions exist to find in other people's code. Today the exposure + is nil: #119 renders none of the three free-text fields. It stops being nil the + moment #119 renders `verdict_evidence`. The mitigation is placed where it can work — + the adjudicator's own prompt forbids this phrasing, a recorded before/after pair + measures whether that prohibition makes a difference, and this is a condition on + #119 ever rendering that field, not a solved problem independent of it. +2. **A `REFUTED` finding is still published, with its verdict beside it.** This stage + removes nothing from `reports[].findings`, and `findings_count` is unchanged by + adjudication. +3. **A downgrade is allowed and must be visible.** `severity` may fall below + `reported_severity` — an overstated true finding is its own error — but it carries + `severity_reason`, and the finding is additionally listed in + `adjudication.downgrades`, so the movement is legible even where a consumer renders + only `severity`. + +The reasoning is #109's, in #122's corrected wording: judges ruling on adversarial +safety claims perform "on average only slightly better than a random coin-flip" against +6,642 human-verified labels, and the AUROC 0.48–0.64 range is one judge on one victim +model under two attacks — not quoted here as anything broader. The phrase "despite high +performance on standard validation sets" is not used, because it is not in the paper. + +## The `adjudication` block + +A top-level sibling of `reports` and `containment` in the merged document. **Nine keys**, +and the count is load-bearing for the same reason `FINDINGS.md`'s ten finding fields +are: the control suite builds one control per key, so a key present in the output but +not in this list gets no control at all. + +| key | meaning | +|---|---| +| `schema_version` | integer, starts at 1 | +| `verdict_counts` | `{CONFIRMED, REFUTED, UNPROVEN}` — integers | +| `findings_in` | count of findings received | +| `findings_out` | count of findings emitted; **must equal** `findings_in` | +| `duplicate_groups` | array of `{survivor, duplicates: [finding_id]}` | +| `downgrades` | array of `{finding_id, from, to, reason}` | +| `total_refutation` | boolean — see § Total refutation | +| `notes` | array of free-text notes | +| `completion_marker` | **last** key: `BUZZ-ADJUDICATION-COMPLETE:{nonce}`, using the document's own top-level `nonce` — #117's sixth key, passed through unchanged and never re-generated here | + +The block carries **no nonce of its own**: a second copy in one document is a second +thing that can disagree, and there is no question a copy would answer that reading the +top-level key does not. + +The marker is last and carries the nonce for the same two reasons `FINDINGS.md` gives: +a marker at the end cannot survive truncation, and a fixed string published in a public +repository is one a PR author can type into their own diff. This stage never accepts a +caller-supplied nonce — it verifies the nonce it receives against every report's own +marker and passes the same value through, exactly as it received it. + +`findings_out == findings_in` is a necessary but not sufficient check on its own — it +would not catch a drop-and-invent swap (one real finding removed, one fabricated one +substituted, count unchanged). The binding guarantee is the stronger one stated in +§ The verdict: the input and output `finding_id` **sets** are equal, which a +drop-and-invent swap violates even though the count survives it. + +## Dedupe + +Findings describing the same defect in different words are grouped, and the grouping +is in the output rather than in the stage's head. `finding_id` cannot do this work: it +is not stable across a model rewording `defect`, and two findings from *different* +dimensions describing one defect have different ids by construction, since `dimension` +is a hash input. + +A group is `{survivor, duplicates: [finding_id]}` in `adjudication.duplicate_groups`, +and every duplicate **also** carries `duplicate_of` naming its survivor. Both +directions, so the grouping is discoverable from the finding itself as well as from the +block — a consumer holding one finding should not have to scan a top-level array to +learn it is a duplicate. + +**A duplicate still receives its own verdict and is still emitted.** Dedupe changes +presentation, never the count: it groups findings, it does not remove them. Removing a +duplicate's own record would breach the "every finding receives exactly one verdict" +requirement while looking like tidiness. + +The survivor is chosen **deterministically**: highest adjudicated severity, then +`CONFIRMED` before `UNPROVEN` before `REFUTED`, then lowest `finding_id`. Stated +explicitly because "the best one" is not a rule, and two runs over the same input must +agree on the same survivor. A run that dedupes nothing emits an **empty** +`duplicate_groups` array rather than omitting the key — the same "empty, not missing" +discipline `FINDINGS.md` uses for `findings` on a clean dimension report. + +A finding whose `duplicate_of` names a `finding_id` absent from the document, or names +itself, is invalid. + +## Total refutation + +`total_refutation` is `true` if and only if `findings_in > 0` and every finding's +verdict is `REFUTED`. Refuting everything is flagged rather than published as a clean +PR — total refutation is likelier a broken adjudicator than a flawless diff, and #109's +own scepticism about this stage exists precisely so a suspicious result gets a second +look rather than a pass. + +## Containment findings are passed through, not adjudicated + +The `containment` block is emitted byte-identically to what arrived. Three reasons: + +1. Containment findings are deterministic catches, not claims needing a judge. +2. An adjudicator able to `REFUTE` one could erase a detected attack, which + `CONTAINMENT.md` § Severity contract calls worse than never detecting it. +3. Their severity is fixed at `Blocker` by that same contract, so re-rating them here + would contradict a document this one is a sibling to, not an authority over. + +**A known, deliberately unresolved mismatch, not silently fixed here.** +`CONTAINMENT.md`'s "Contract for later stages" table literally binds this stage to call +`contain.findings_for(surfaces, nonce)`, which needs the `Surface` dict and the nonce — +this stage receives a JSON document on stdin and has neither. #117 already places +exactly what that function returns into the merged document's `containment` key, so this +stage consumes that block verbatim instead of re-deriving it (re-fetching the surfaces +here would be a second source of truth for one fact, and a second reason for this stage +to touch author text at all — the opposite of what the table's own "must never" column +requires). The pass-through above honours the *intent* of the table's row, but the +table's own literal wording is not corrected by this document — that edit is left for +whoever owns #120, since `CONTAINMENT.md` is a cross-cutting contract this document does +not have unilateral authority to amend. diff --git a/launchpad/review-agent/CONTAINMENT.md b/launchpad/review-agent/CONTAINMENT.md index 40a4a12cfd3..a5814a316fe 100644 --- a/launchpad/review-agent/CONTAINMENT.md +++ b/launchpad/review-agent/CONTAINMENT.md @@ -303,7 +303,7 @@ A stage that builds a prompt must envelope first, without exception. |---|---|---| | [#116](https://github.com/launchpad-26/buzz/issues/116) pre-flight | `fetch.fetch_all(pr, repo)` — emit one labelled field per entry point | concatenate surfaces into one blob, or build a prompt | | [#117](https://github.com/launchpad-26/buzz/issues/117) dimensions | `contain.render(surfaces, nonce)` before any text reaches a model | place any surface above the preamble or after the closing marker | -| [#118](https://github.com/launchpad-26/buzz/issues/118) adjudication | `contain.findings_for(surfaces, nonce)` — returns `list[Finding]` and nothing else | re-read raw PR text to "check for itself" | +| [#118](https://github.com/launchpad-26/buzz/issues/118) adjudication (see `ADJUDICATION.md`) | `contain.findings_for(surfaces, nonce)` — returns `list[Finding]` and nothing else | re-read raw PR text to "check for itself" | | [#119](https://github.com/launchpad-26/buzz/issues/119) publish | `review.render_review(findings, states)`, with `states` taken from `render()`'s own return, never re-derived | publish evidence in raw form — quote post-escape or not at all; re-apply `fetch.apply_invocation_cap` to build a second `states` | All four route the same seven labels: `pr_title`, `pr_body`, `pr_diff`, From 46acd273e772f83e9b77384003b09779c9328940 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Fri, 21 Aug 2026 09:04:01 +1200 Subject: [PATCH 3/3] fix(launchpad): address review findings on #256 (#118 STEP 1) Fixes the 7 confirmed findings from review-adjudicate plus the disputed Blocker, resolved per Serina's call: not a real gap (severity values that matter are rejected on input; reported_severity cannot appear there at all), but the doc's wording was genuinely misleading and is corrected. - add the missing top-level `stages` entry section to ADJUDICATION.md - fix containment key: findings_for's return lands in containment.findings, not the whole containment key - fix the out-of-ladder severity fallback to state the precise reported_severity-else-Blocker rule instead of "nearest legal value" - stop stating #119's sort behaviour as settled fact -- it's still an unmerged plan - reword the disputed sentences so they describe an out-of-ladder severity value, never a field literally named reported_severity, which cannot appear on input - correct stale "#252 open" references now that #252 has merged - note the file inventory list is a summary, not exhaustive (run_controls.py, check_*.py, test_*.py omitted) - reword STEP 1's done-when from "a new row" to "annotating the existing row", matching what was actually implemented Docs-only change; run_controls.py (13/13) and test_findings.py (36/36) both pass unchanged. Signed-off-by: Serina Mcfall --- .../2026-08-13-issue-118-adjudication.md | 40 +++++++++------ launchpad/review-agent/ADJUDICATION.md | 49 +++++++++++++++---- 2 files changed, 64 insertions(+), 25 deletions(-) diff --git a/launchpad/plans/2026-08-13-issue-118-adjudication.md b/launchpad/plans/2026-08-13-issue-118-adjudication.md index 5fd632825d9..c2261e99bcc 100644 --- a/launchpad/plans/2026-08-13-issue-118-adjudication.md +++ b/launchpad/plans/2026-08-13-issue-118-adjudication.md @@ -54,9 +54,14 @@ ALREADY TRUE (verified against git, the working trees and the GitHub API, not n `findings.py`, `run_dimensions.py`, `contain.py`, `fetch.py`, `review.py`, `FINDINGS.md`, `CONTAINMENT.md`, three files under `dimensions/`, five under `fixtures/dimensions/`, and (on this branch only, pending #252) fifteen under - `recordings/`. `find . -iname "*adjudic*"` still returns nothing anywhere in - the tree — of everything above, only #118 itself remains unbuilt, which is - the one claim from the struck-through block that still holds. **The defining + `recordings/` — a summary count, not an exhaustive inventory: it omits + `run_controls.py`, the `check_*.py` control scripts, and the `test_*.py` suites + also present in the tree. `find . -iname "*adjudic*"` still returns nothing + anywhere in the tree — of everything above, only #118 itself remains unbuilt, + which is the one claim from the struck-through block that still holds. + **Further corrected 2026-08-21: PR #252 merged to `origin/launchpad` the same + day this was written (2026-08-20T19:25:57Z) — #117 is now fully merged, all + twelve of its steps included, not "pending".** **The defining risk BUDGET names below (a contract with no producer) is now a non-issue**: 15 real recorded dimension reports already exist and are the right input for STEP 8/9's fixtures, not a document-only synthesis. See BUDGET's own @@ -249,9 +254,11 @@ STEP 1 launchpad/review-agent/ADJUDICATION.md — the verdict contract, [inde stage does so: `run_adjudication.py`'s `main` runs #117's own `findings.validate` against the input document BEFORE any adjudication logic touches it, and exits non-zero, adjudicating nothing, when it - fails — the same input `reported_severity: "Info"` fails #117's - validator on its own severity check, so it never reaches this stage's - re-rating logic at all. This is stronger than "this stage is agnostic + fails — a finding whose `severity` value arrives out-of-ladder (an + "Info", say — this is the dimension's own report, not a field literally + named `reported_severity`, which does not exist on input at all) fails + #117's validator on its own severity check, so it never reaches this + stage's re-rating logic at all. This is stronger than "this stage is agnostic about its producer": #119 is agnostic about ITS producer because #119 cannot re-validate a document it did not build from parts; this stage CAN, because the input document is exactly #117's own output shape, and @@ -347,8 +354,10 @@ STEP 1 launchpad/review-agent/ADJUDICATION.md — the verdict contract, [inde relies on its consumer's default has moved the failure rather than removed it; it states that the `containment` block passes through unadjudicated and cites CONTAINMENT.md § Severity contract; and it is - referenced from a new row in CONTAINMENT.md's "Contract for later stages" - table so the two documents point at each other rather than diverging quietly. + referenced from CONTAINMENT.md's "Contract for later stages" table — annotating + the table's existing #118 row rather than adding a duplicate, since one row per + stage is the table's own convention — so the two documents point at each other + rather than diverging quietly. STEP 2 launchpad/review-agent/verdicts.py — the contract in code. [needs 1] Pure functions and dataclasses over an already-parsed document. No @@ -481,7 +490,7 @@ STEP 3 launchpad/review-agent/run_adjudication.py — the CLI, [needs 2] <- R fixture carrying all three containment kinds emits them unchanged with severity Blocker and no verdict field added to any of them; malformed JSON on stdin exits non-zero and prints no document; a fixture whose one finding - arrives with `reported_severity: "Info"` exits non-zero and prints no + arrives with an out-of-ladder `severity` value (an "Info", say) exits non-zero and prints no document at all — validated and refused before adjudication, not adjudicated into a best-effort output; a judge injected to REFUTE that same finding is never called, asserted on the injected judge's own call @@ -633,10 +642,10 @@ STEP 6 Escalate-only, enforced in code, and the total-refutation flag. [ne document still passes `verdicts.validate` — this is the case STEP 3's input validation does NOT catch, because the input was legal and only this stage's own re-rating produced the bad value; the SIBLING case — a - fixture whose finding ARRIVES with `reported_severity: "Info"` already — - is STEP 3's job, not this control's: `run_adjudication.py` exits non-zero - on it before any judge runs, so it is asserted there and not repeated - here as a per-finding UNPROVEN case, since there is no legal + fixture whose finding ARRIVES with an out-of-ladder `severity` value + already — is STEP 3's job, not this control's: `run_adjudication.py` + exits non-zero on it before any judge runs, so it is asserted there and + not repeated here as a per-finding UNPROVEN case, since there is no legal `reported_severity` for such a finding to have been emitted WITH; a BARE `review.SEVERITY_ORDER[f["severity"]]` subscript succeeds for every finding in every output above, used bare on purpose so @@ -1044,8 +1053,9 @@ BUDGET ~~STEP 9 eats the budget. The thing most likely to derail the issue is they land.~~ **Corrected 2026-08-20 (struck through, not deleted — see ALREADY TRUE's own correction above, same date).** #120 and #117 are BOTH MERGED to `origin/launchpad` - (every step except #117's own STEP 8, which is committed on the branch this plan - is written from, PR #252 open). This branch needed no rebase to get + ~~(every step except #117's own STEP 8, which is committed on the branch this plan + is written from, PR #252 open)~~ **— further corrected 2026-08-21: PR #252 merged + the same day, so #117 is fully merged, all twelve steps.** This branch needed no rebase to get `launchpad/review-agent/` — it was already there, cloned from `origin/launchpad` plus one commit. The precondition chain is DISCHARGED, not pending. **The actual risk STEP 8/9 now carry is the opposite of the one described above:** diff --git a/launchpad/review-agent/ADJUDICATION.md b/launchpad/review-agent/ADJUDICATION.md index b2a2e375bb1..93cb8a52604 100644 --- a/launchpad/review-agent/ADJUDICATION.md +++ b/launchpad/review-agent/ADJUDICATION.md @@ -45,7 +45,7 @@ confirmed findings would leave `REFUTED` findings carrying an unexamined severit ### `severity` and `reported_severity` must both be in `review.SEVERITY_ORDER` Stated as a guarantee this stage makes to #119, and as **defence in depth** rather than -#119's only defence: #119 sorts with `.get(severity, 9)` and routes an unrecognised +#119's only defence: #119's plan states it sorts with `.get(severity, 9)` and routes an unrecognised severity to its own malformed-finding heading, so a bad value is survivable there — but this stage is the one that **creates** bad values by re-rating, and a producer that relies on its consumer's default has moved the failure rather than removed it. @@ -59,8 +59,13 @@ copied into `severity` untouched. **This cannot happen if the input document is validated first, and this stage does so.** `run_adjudication.py`'s `main` runs #117's own `findings.validate` against the input document *before* any adjudication logic touches it, and exits non-zero — adjudicating -nothing — when validation fails. An input carrying `reported_severity: "Info"` fails -`FINDINGS.md`'s own severity rule and never reaches this stage's re-rating logic at all. +nothing — when validation fails. A finding whose `severity` value arrives out-of-ladder +(an `"Info"`, say — this is the dimension's own report, before this stage has re-rated +anything) fails `FINDINGS.md`'s own severity rule and never reaches this stage's +re-rating logic at all. Note the field name: `reported_severity` does not exist on +input at all — #117 emits only `severity` — so this guarantee rests on that field, not +on a field literally named `reported_severity`, which is a name this stage's own +*output* introduces. This is stronger than "this stage is agnostic about its producer": #119 is agnostic about *its* producer because #119 cannot re-validate a document it did not assemble from parts, while this stage *can*, because its input document is exactly #117's own output @@ -71,11 +76,14 @@ stage silently deciding what the dimension actually meant. So an out-of-ladder **effective** severity this stage can still produce — one its own re-rating created from a legal `reported_severity` — is an adjudication **failure** for -that finding: `UNPROVEN`, `severity` set to the nearest legal value with the refusal -stated in `severity_reason`, and `reported_severity` left unchanged (input validation -already guarantees it was legal to begin with). This stage may not silently decide an -unrateable finding is a small one, and it is never asked to invent a value for a field -that arrived already broken. +that finding: `UNPROVEN`, with the refusal stated in `severity_reason`, and +`reported_severity` left unchanged (input validation already guarantees it was legal to +begin with). `severity` falls back to `reported_severity` when that is in the ladder, +and to `Blocker` when it is not — the second branch is defence in depth only, since +input validation already guarantees `reported_severity` arrives legal, but there is no +safe value to copy if that guarantee were ever bypassed. This stage may not silently +decide an unrateable finding is a small one, and it is never asked to invent a value for +a field that arrived already broken. ## Escalate, never approve @@ -145,6 +153,25 @@ substituted, count unchanged). The binding guarantee is the stronger one stated § The verdict: the input and output `finding_id` **sets** are equal, which a drop-and-invent swap violates even though the count survives it. +## The `stages` entry + +#117 does not emit a top-level `stages` array — it is the manifest #119's plan reads +for stages that produce no envelope of their own (#116's pre-flight, and this one). This +stage adds exactly one entry, `{name: "adjudication", status, reason}`, to whatever +`stages` array arrived (empty, or already carrying #116's entry), and passes every +existing entry through unchanged. + +**It never overwrites an existing `adjudication` entry.** A second one on input means a +re-run against a document this stage has already adjudicated, and this stage exits +non-zero rather than silently repeating itself. + +`status` is `"complete"` only when every finding received a verdict, the top-level +`nonce` was established (see § The `adjudication` block, above), and § Total +refutation's flag is false. +Otherwise it names the specific reason: `"total_refutation"`, a nonce disagreement, a +missing top-level `nonce`, or (via `run_adjudication.py` exiting before this stage runs +at all) an input that already fails #117's `findings.validate`. + ## Dedupe Findings describing the same defect in different words are grouped, and the grouping @@ -196,8 +223,10 @@ The `containment` block is emitted byte-identically to what arrived. Three reaso `CONTAINMENT.md`'s "Contract for later stages" table literally binds this stage to call `contain.findings_for(surfaces, nonce)`, which needs the `Surface` dict and the nonce — this stage receives a JSON document on stdin and has neither. #117 already places -exactly what that function returns into the merged document's `containment` key, so this -stage consumes that block verbatim instead of re-deriving it (re-fetching the surfaces +exactly what that function returns into the merged document's `containment.findings` +key — not the whole `containment` key, which also carries the `states` map #117 builds +separately — so this stage consumes that block verbatim instead of re-deriving it +(re-fetching the surfaces here would be a second source of truth for one fact, and a second reason for this stage to touch author text at all — the opposite of what the table's own "must never" column requires). The pass-through above honours the *intent* of the table's row, but the