Skip to content

Add the required verdict-review-gate: sealed cross-family evidence for every changed verdict row - #135

Merged
seathatflowsinourveins merged 18 commits into
mainfrom
claude/verdict-review-gate
Sep 23, 2026
Merged

seathatflowsinourveins merged 18 commits into
mainfrom
claude/verdict-review-gate

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

This adds a required CI check, verdict-review-gate, as the control for a single-maintainer repository. A PR that changes any layer verdict row cannot merge without the sealed cross-family review evidence the verdict pipeline defines (#122, #124).

What it enforces for every changed (catalog, layer_id, run_id) row outside the grandfathered 20260922 wave:

  • Lane returns: the sealed returns exist, are registered, and match the row's sealed_sha256. They share one packet_sha256, and they come from distinct families.
  • Agreement: agreement is recomputed from the two winner_keys sets, so a relabelled row fails.
  • Winners: winners must equal the chosen lane's keys resolved through the sealed packet listed in run-manifest.json.
  • Packets: withheld keys are rejected recursively.
  • Hash bindings: run_manifest_sha256, adjudication_sha256 and single_lane_decision_sha256 must all match.
  • Disagreements: a disagree row needs both-family, both-order adjudication with no refutation.
  • Single-lane rows: they need an authorization under docs/decisions/, carrying single-lane-authorization: <catalog>/<layer_id>, that is already at the base.
  • platform_status: it must be the value platform_status() derives from receipts bound to the sealed pin and refs that are already at the base.
  • Waves: there is no rollback, deletion or re-keying of rows. Earlier waves stay frozen.
  • Trust inputs: they are derived from what the gate and validators actually read.
  • Base comparison: the base is the merge commit's first parent, checked against the payload, and the job fails closed.
  • Validators: landscape.py and build_verdicts --check run too.
  • Pass-through: a PR that touches no verdict row passes with "no verdict rows changed".

Recorded residuals (see the decision record):

  • Sealed lane returns certify themselves; the gate proves consistency, not authenticity.
  • A PR can edit its own validate.yml job. pull_request_target and workflow_run are rejected by the repository's zizmor policy, and required workflows exist only for organisations.
  • Two lows carry proposed fixes for the tooling owner.

Review. Five adversarial Opus review rounds plus raw reruns closed:

  • a decoy ledger;
  • rollback, deletion and relabelling;
  • fail-open error paths and duplicate JSON keys;
  • platform_status forgery;
  • same-PR authorization;
  • a pin change behind an unchanged status.

It has 142 gate tests, all negative controls mutation-checked, and real-checkout mutations.

Checks. At the merge with main, including #124:

The target .github/main-ruleset.json adds verdict-review-gate as a required check. I'll apply it after merge, after notifying the coordinating session.

🤖 Generated with Claude Code

…rows

With one maintainer and zero required approvals, a PR changing a verdict row
could merge unreviewed. scripts/verdict_review_gate.py compares ledger rows at
the merge base with the head, keyed by (catalog, layer_id, run_id), and requires
for every changed non-grandfathered row: a registered wave document and run
manifest listing the row; registered, hash-matching sealed returns from two
distinct families; the agreement recomputed from the returns (review finding
1); winners resolved through the sealed packet, failing closed without it
(finding 6); a both-family, both-order, unrefuted adjudication for disagree
rows; an explicit docs/decisions single-lane-authorization line (finding 2);
receipt-derived platform_status-only changes; and frozen earlier wave entries
(finding 4). It then runs landscape.py and build_verdicts.py --check.

validate.yml gains the verdict-review-gate job (every pull_request, no path
filter; contents: read; harden-runner first; no persisted credentials; base
sha via env). The target main ruleset adds the context; the coordinator
applies it after merge.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…evidence

A changed non-grandfathered row now also needs each winner's evidence_class and
why_selected to be the chosen lane's and its pin to be the sealed packet's; a
pending row may carry no winners; every changed platform_status value must be
the one scripts/platform_status.py derives. Changes under the sealed verdict
artifacts or the landscape/sota-convergence catalogs run the repository
validators even when no row changed, so a deleted sealed lane file fails.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ed rules, published fields

- Read rows from build_verdicts.LEDGER_FILES and fail when the head's landscape manifest names
  other ledgers (a decoy could be validated in place of the published ledger).
- Run the base commit's gate against the PR checkout in CI, and fail a verdict change that also
  changes the gate's trust base in the same comparison.
- Tie a changed winner's repository and recipe_ref to the sealed packet candidate, and the
  published alternatives, verdict_overturn_when and overturn_protocol to what record_verdicts
  derives from the sealed returns; open_gaps changes are re-checked.
- Fail closed (exit 2) when a base or head ledger, manifest or wave registry exists but cannot be
  read or parsed.
- Add a negative-control test for each closed bypass and record the review in the decision.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… compare, #124 bindings

F1: a base row cannot disappear, roll back to an older or grandfathered wave, or move to any
wave but the newest registered one (then it needs the full new-wave evidence); one row per layer.
F2: the PR-editable job definition is recorded as an accepted residual with its rejected
alternatives, mitigations and overturn; a test asserts the push-to-main run executes the gate.
F3: rows, wave documents and the registry compare on parsed values, so a pure reformat passes
and is not a verdict change for the trust-base rule; documented that a changed row value in a
non-grandfathered wave needs a new recorded wave.
F4: TRUST_PATHS adds scripts/validate.py, lane-provenance.json and the host-receipt schema; a
test derives the transitive imports and rule inputs from the modules.
F5: on pull_request the base is the merge commit's first parent, cross-checked against the
payload base.sha; push-to-main uses github.event.before and fails closed on all-zero.
F6: packets resolve through the run manifest (packet_sha256, retained_packets,
packets_sha256sums) and carry no withheld key at any depth; run_manifest_sha256,
adjudication_sha256 and single_lane_decision_sha256 are required and bound, with #124's names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…n message

The measured entry lists the new controls, the four real-checkout F1 mutations (each exit 1),
the real-ledger reformat (exit 0 with both validators) and the #124 overlay run. Two
adjudication tests accept the message #124's run_manifest_row_issue reports first; the
no-new-wave move message no longer prints None.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…closed on git errors and duplicate keys

Third review of the verdict-review-gate: a recorded newest-wave row could be
relabelled pending_lanes or no_selection with its winners cleared and still
pass. The gate now derives the status record_verdicts.py writes for the sealed
evidence (agreement, adjudication, single-lane decision, remaining
alternatives) and rejects no_selection on a new-wave row. A failing git
diff/ls-files raises a read error (exit 2); every parse rejects duplicate JSON
keys; #124's constant values are pinned so the alignment test asserts before
#124 merges. The free-form newest-wave fields are documented.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… single-lane authorizations

Fourth review of the verdict-review-gate:
- G1: every winner field except platform_status must equal record_verdicts.build_winners output
  (evidence_refs and pin included, no extra key), and platform_status derives from receipts bound
  to the sealed pin and refs; a pin the packet lacks binds only when the base's row candidates
  carry it. The canonical repository index must derive the same alternatives at the base.
- G2: a single-lane decision record must exist byte-identical at the base.
- G3: the trust-path test derives head-side reads with an audit hook; adoption/manifest.json
  platform_profiles is bound as a rule input; base tree reads use ls-tree and fail closed.
- G4: the workflow asserts HEAD is the merge commit whose second parent is the PR head.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…kout mutations

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d workflow

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ed SOTA manifests

Fifth review of the verdict-review-gate. A changed platform_status value may
rank no higher than platform_status() derives from only the cited evidence
files and host receipts already at the base with the same bytes and
registered there with that sha256, so a sealed citation made to resolve, or
a receipt added, in the same PR cannot raise a status. The SOTA manifest
every base wave registers, the newest included, keeps its pointer and value.
merge_base fails closed on git errors other than no common history. The
decision record adds the self-attested sealed-returns residual and rewords
the overturn line and workflow comment as a consistency check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ckout mutations

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… fall back to the base row's candidate pin

Sixth review: a candidates-only pin change with a same-PR receipt could keep a
declared status the sealed pin does not support. The no-packet-pin fallback now
comes from the base row, and an unchanged value is skipped only when pin,
evidence_refs and evidence_class are unchanged. Three negative controls, each
mutation-checked; two low findings recorded as residuals with proposed fixes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-gate

# Conflicts:
#	manifests/evidence.json
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-23T18:26:32.886035Z 6bfeb24 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…rict bootstrap and type-strict values

- H1: the pull_request trigger adds the edited type (a base-branch change) and the
  verdict-review-gate job fails closed on a pull_request whose base branch (PR_BASE_REF from
  github.base_ref, through env) is not main, so a retargeted PR cannot keep a stale green run.
- M1: when the head registers a wave newer than the base's newest, that newest wave's registry
  entry (sha256 included) and document are frozen byte for byte.
- L4: the head's own gate runs only when base...HEAD adds scripts/verdict_review_gate.py (A);
  any other base without the gate fails closed.
- L5: frozen values compare as canonical json.dumps(sort_keys=True), so 1, 1.0 and true differ.
- Hardening: changed paths are listed with git diff -z / git ls-files -z and split on NUL.
- L3: the decision record states that the pinning tests are head-defined too, and corrects the
  "visible workflow edit" claim with the retarget path and its fix.

Each new negative control is mutation-checked (reverting its fix fails its test).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…se on origin/main

A newly registered wave must be the only new run id, the head's newest and
newer than every base wave, so build_verdicts.py --check always regenerates
it; two new waves or a backdated wave fail. The PR gate step also requires
the merge commit's first parent to be a commit on origin/main, so a merge
commit built on a stacked branch fails closed. Fix the record's H1
cross-reference and document both checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 round 2 lows)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…2 review lows

A throwaway PR (#143) showed that the edited run after a retarget checks out the merge commit
built on the old base; the round-2 origin/main ancestry check failed it closed, and close and
reopen rebuilt it on main, where the gate passed. The error message now names that recovery.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins
seathatflowsinourveins merged commit 3541678 into main Sep 23, 2026
22 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/verdict-review-gate branch September 23, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant