Repository navigation
Verdict-pipeline integrity before the 20260923 re-record: per-wave CI, unbiased packets, two-family rule, survivorship, receipt-backed status - #122
Conversation
…-record Fixes seven defects of the peer update-path audit (wf_77c0ea46-091, main 38847e5): 1. Per-wave CI: wave documents are frozen and hash-registered in catalogs/sota-convergence/layer-verdict-waves.json; build_verdicts --check verifies each wave and regenerates only the newest one and the handbook. 2. --withhold-labels strips popularity/recency fields from every candidate and component copy and lists them in withheld; default packets unchanged. 3. codex_absent stays pending_lanes unless --allow-single-lane names a dated decision record; landscape.py rejects a single-lane recorded row without it. 4. Lane returns declare model.family (anthropic/openai patterns, families differ); adjudications record judge {model, family} and the stripped-packet sha256; a winner needs both families in both orders, else a sealed split. 5. Every wave writes run-manifest.json (packets, lane outcomes, SHA256SUMS); landscape.py requires each new-wave row there with both lanes. 6. Linux accepted needs a cited registered receipt/evidence artifact, through a one-line adapter with PR #117's platform_status call shape (#117 not merged). 7. layer-verdict-lane.js vendored under SHA256SUMS; new-wave returns carry provenance (claude_lane.py writes the Claude lane's, codex_lane.py its own). Grandfathered run id 20260922 (GRANDFATHERED_RUN_IDS in scripts/landscape.py); all committed 20260922 artifacts unchanged. Two-family adjudication recorded as keep-but-compare in docs/decisions/2026-09-23-verdict-integrity.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion gaps Fix round for the independent Opus review of 26c8f3a: - Grandfather (major): record_verdicts.py --run-id is required and --write refuses a grandfathered id once --root holds that wave (sealed directory or registered wave document); a grandfathered run writes no run manifest. build_verdicts.py --write refuses to regenerate or re-register a registered grandfathered wave even as the newest. GrandfatheredWavePinTests pins the 20260922 document, its registry entry and the 122-file sealed directory. - lane_packets.withhold_popularity strips a private copy of each shared upstream record, so packet output no longer depends on ledger order. - Linux "accepted" never counts evidence/artifacts/layer-verdicts-*/ files. - Single-lane decision records must name the layer id as a whole token. - Provenance and judgments are bound, not only format-checked: the append-only tools/sota-convergence/lane-provenance.json lists accepted lane code (full workflow_path, not basename), record time also requires the current vendored bytes and this checkout's codex_lane.py/lane-prompt.md, landscape.py re-checks sealed returns against the registry, and each new-wave judgment's stripped_packet_sha256 must equal the layer's sealed packet hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/main at 9432720 (PR #117, scripts/platform_status.py) and resolve the six findings of the rereview:integrity pass: 1. record_verdicts.platform_status_for calls the shared platform_status(platform_id, winner, context) for every platform, with load_context(root) once per run. landscape.py re-checks a new-wave row on every platform with declared_status_error (ENFORCED_PLATFORMS stays {"macos-arm64"} as on main, for grandfathered rows). The shared registered_evidence_refs now excludes evidence/artifacts/layer-verdicts-*, so the recorder, landscape.py and component_matrix.py agree. The Linux-only adapter code is deleted and the stale "#117 is not on main" text fixed. 2. The new-wave checks run for any row whose sealed_base or lane run id names a non-grandfathered wave, even without run ids; recorded rows must seal the lanes their agreement implies. 3. winner_lane is validated after the two-family rule: a unanimous single-family adjudication is a split with null or the chosen lane. 4. record_verdicts --checked-at defaults to the --run-id date. 5. codex_lane.py writes model.name from --model (passed as codex exec -m) or the event stream, never from the response text; registry entry appended. 6. --withhold-labels withholds upstream.latest always (stricter option), with upstream.prerelease and pin_behind_upstream. manifests/evidence.json: main's version, rehashed and sorted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ave, withhold the newcomer flag, cover the unreviewed macOS receipt Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f19193e83
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if withhold: | ||
| # Every packet, manifest-mode trading packets included, loses popularity and | ||
| # recency signals; the default (no --withhold-labels) build is unchanged. | ||
| packet = withhold_popularity(packet) |
There was a problem hiding this comment.
When --withhold-labels is omitted—as in the current command in recipes/sota-convergence-practice.md—this branch leaves both decision labels and the newly identified popularity/recency fields in the packets. record_verdicts.py does not verify a withholding marker or packet contents, so a non-grandfathered wave can be recorded and pass CI even though its judges saw the biased fields this change is intended to exclude. Require the withholding policy for new waves or make the recorder reject packets that do not declare and satisfy it.
Useful? React with 👍 / 👎.
| if git(agentlab_root, "diff", "--quiet", "HEAD", "--", workflow).returncode != 0: | ||
| raise ProvenanceError(f"{workflow} differs from HEAD in {agentlab_root}; commit it before recording") | ||
| return {"workflow_path": workflow, "workflow_sha256": digest, "agentlab_commit": head.stdout.strip()} |
There was a problem hiding this comment.
If the workflow path is untracked but contains the vendored bytes, git diff --quiet HEAD -- <path> exits successfully, so this records the checkout's HEAD as agentlab_commit even though that commit does not contain the workflow at all. A later checkout of the recorded revision therefore cannot reproduce the claimed source; check that the path is tracked at HEAD, for example with git ls-files --error-unmatch or by reading HEAD:<path>, before accepting it.
AGENTS.md reference: AGENTS.md:L10-L15
Useful? React with 👍 / 👎.
| model = dict(data.get("model") or {}) | ||
| if resolved_model: | ||
| model["name"] = resolved_model | ||
| model["family"] = LANE_FAMILY |
There was a problem hiding this comment.
When --resolved-model is omitted, model.name is copied from layer["final"], which is the model-authored structured response rather than runner-owned workflow metadata. A malformed return can claim another matching Anthropic name such as sonnet and pass record_verdicts.py, leaving a false model identity in sealed evidence despite the workflow being statically bound to opus; overwrite the name from the workflow's top-level runner result or require --resolved-model.
AGENTS.md reference: AGENTS.md:L12-L15
Useful? React with 👍 / 👎.
| if (!refuted.length && !majors.length) return { catalog: p.catalog, layer_id: p.layer_id, packet_sha256: p.sha256, proposal, votes, revised: null, final: proposal } | ||
| const revised = await agent(PACKET + '\n' + fill(p) + fixed(p) + `\nThis is the revision round. Your earlier proposal was: ${JSON.stringify(proposal)}. Independent refuters found: ${JSON.stringify(votes)}. Resolve every blocking and major finding by re-reading the cited evidence; keep what the evidence supports, drop or correct what it does not, and add what was missing. Return the complete revised object.`, { label: `propose:revise:${p.catalog}/${p.layer_id}`, phase: 'Revise', agentType: 'semantic-evidence-reviewer', model: 'opus', effort: 'high', schema: LANE_RETURN }) | ||
| return { catalog: p.catalog, layer_id: p.layer_id, packet_sha256: p.sha256, proposal, votes, revised, final: revised || proposal } |
There was a problem hiding this comment.
If either refuter returns null, it is filtered out and the no-findings branch can publish the proposal as final; if a refuter does report a blocking or major issue but the revision call returns null, revised || proposal publishes that already-refuted proposal instead. claude_lane.py then discards the vote envelope and writes only final, so record_verdicts.py cannot detect either incomplete chain and may use the proposal in a recorded winner. Require both refuter results and, after any material finding, a non-null revision before setting final.
AGENTS.md reference: AGENTS.md:L65-L68
Useful? React with 👍 / 👎.
| if not isinstance(layer.get("final"), dict): | ||
| without_final.append(name) | ||
| continue |
There was a problem hiding this comment.
When the work directory already contains a Claude return and a later workflow result names that layer with final: null, this branch skips the write but leaves the old file in place; layers listed only in lost are likewise never cleaned up. If the packet hash is unchanged, record_verdicts.py accepts that stale file and records the current run's outcome as sealed instead of missing, hiding the failed attempt. Delete stale outputs for without_final and lost layers or write each attempt into an isolated directory.
AGENTS.md reference: AGENTS.md:L65-L68
Useful? React with 👍 / 👎.
| if not grandfathered: | ||
| sealed_writes.append((root / sealed_base / RUN_MANIFEST_NAME, | ||
| run_manifest_text(work_dir, run_date, sealed_base, outcomes, rejections))) |
There was a problem hiding this comment.
For a new wave this only seals run-manifest.json; the packet JSON files and their SHA256SUMS remain solely in the external work directory. Once that directory is removed, the repository contains only claimed hashes, so reviewers cannot inspect the exact withheld packet seen by the lanes or reproduce the run when non-default seed, manifest, candidate mode, or date arguments were used. Copy the packet artifacts into the sealed wave, as the grandfathered wave does, or retain all generator inputs needed to reproduce and verify their bytes.
Useful? React with 👍 / 👎.
| const a = args && typeof args === 'object' ? args : {} | ||
| const issues = [] | ||
| const nonblank = (v) => typeof v === 'string' && v.trim().length > 0 | ||
| const REPO = nonblank(a.repo) ? a.repo : '.' |
There was a problem hiding this comment.
The workflow accepts arbitrary caller-supplied prompt text, but the Claude provenance written later contains only the workflow path/hash and agent-lab commit. A caller can therefore run the registered workflow with modified decision instructions and still produce a return that passes all provenance checks as though it used the checked-in lane-prompt.md; record and validate the prompt hash, as the Codex lane already does, or remove this override.
AGENTS.md reference: AGENTS.md:L10-L15
Useful? React with 👍 / 👎.
| adjudication_path = f"{sealed_base}/adjudication/{lanes_field['claude'].get('run_id')}.json" | ||
| adjudication_file = safe_file(root, adjudication_path) | ||
| require(adjudication_file.is_file(), | ||
| str(key) + f" disagree verdict needs its sealed adjudication {adjudication_path}") | ||
| issue, result = judge_adjudication(json.loads(adjudication_file.read_text(encoding="utf-8")), | ||
| grandfathered=grandfathered, packet_sha256=packet_sha256) |
There was a problem hiding this comment.
Adjudication validation is nested under status == "recorded", but the recorder deliberately seals a two-family split while leaving the row pending_lanes. For such a row, CI validates the two lane returns and run manifest but never requires or parses the referenced adjudication, so a malformed or replaced split record can pass while the open gap still presents it as the observed basis for remaining pending. Persist a structured adjudication reference and validate it for pending disagree rows that sealed a split.
AGENTS.md reference: AGENTS.md:L65-L68
Useful? React with 👍 / 👎.
| data = lane_return(layer, provenance, args.resolved_model) | ||
| (out_dir / name).write_text(json.dumps(data, indent=1, sort_keys=True) + "\n", encoding="utf-8") |
There was a problem hiding this comment.
The output path is built directly from catalog and layer_id in the supplied workflow-result JSON without validating either as a filename component. A malformed result can use an absolute catalog or ../ segments and make (out_dir / name).write_text(...) overwrite an arbitrary path outside --work-dir; validate these fields against the supported catalog/layer identities and reject path separators before writing.
Useful? React with 👍 / 👎.
…ane authorization, immutable waves, sealed refutations, retained packets (#124) * Fix the #122 review findings before the 32-row re-record: bind rows to their sealed wave Independent review of catalog #122 (head 2f19193), findings 1, 2, 3, 5 and 6, plus 7 and 9. Each has a regression test that fails on 97a56e2. 1. landscape.py recomputes a new-wave row's agreement from both sealed returns' winner_keys, resolved through the retained packet, and requires a recorded row's winners to equal the chosen lane's set (same_winner/codex_absent: claude; disagree: the adjudication's winner_lane); an unrecorded new-wave row carries no winners. 2. --allow-single-lane needs a dated docs/decisions/ record carrying the line "single-lane-authorization: <catalog>/<layer_id>"; the record's sha256 is stored on the row (lanes.single_lane_decision_sha256) and re-checked in CI. 3. A new wave's --write refuses once its run-manifest.json exists unless --append-rows names only absent rows, and never overwrites a sealed file with other bytes. Rows store lanes.run_manifest_sha256 and lanes.adjudication_sha256; landscape.py verifies both and rejects any file in a new wave's sealed folder that no row or the run manifest references. 5. Re-vendor layer-verdict-lane.js from agent-lab 6aab001 (sha256 2eb9c10d...), registered in lane-provenance.json; the return schema gains the refutation summary, claude_lane.py copies it, codex_lane.py drops it from the strict copy, and record_verdicts.py and landscape.py reject a Claude return whose final was refuted or unknown or lacks a lens vote. 6. --write retains the wave's packets and SHA256SUMS under <sealed_base>/packets/ (manifest retained_packets) and refuses withheld keys; CI re-checks hashes and withheld keys with the policy lane_packets.py now imports from landscape.py. 7. claude_lane.py and codex_lane.py write <lane>/failures.json; the run manifest records a "failed" outcome with its reason. 9. The kept "adopted" marker is documented; the candidate-only note is withheld. Findings 4 and 8 are left for a later PR. GRANDFATHERED_RUN_IDS and every 20260922 artifact are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix the fix-round review: recursive withheld-key check, manifest-bound adjudications, stray wave folders Independent review of fcee72b: - MEDIUM: withheld_packet_keys checked only the positions withhold_popularity strips, so nested release/popularity keys, a top-level newcomers list, a non-null review_status and decisions[].selection passed CI and --write. scripts/landscape.py now walks the whole packet (is_withheld_packet_key, shared with lane_packets.withhold_popularity, which strips at any depth), rejects the disposition labels on candidate/component copies, and requires the always-listed policy labels in withheld[] (proof of --withhold-labels). Exemptions: requirement-gated archived/license and the packet's checked_at. Real 2026-09-23 packets leaked upstream.latest_flag (tag release/2025-11-28); it is now withheld. Default 2026-09-22 packets are byte-identical. - LOW: the run manifest entry lists a sealed adjudication's sha256; landscape.py requires it to equal the row's lanes.adjudication_sha256 and re-hashes the file. - LOW: a layer-verdicts-<id> folder with a non-alphanumeric id is rejected, not skipped. - INFO (lane policy edge): kept deliberately and recorded in docs/decisions/2026-09-23-verdict-integrity.md with its overturn condition. Regression tests (fail on fcee72b): test_lane_packets RecursiveWithheldKeyTests; test_landscape test_retained_packets_are_checked_at_every_depth, test_a_sealed_adjudication_is_bound_by_the_run_manifest, test_a_layer_verdicts_folder_no_row_can_name_is_rejected; test_record_verdicts ReviewOf122Tests test_a_packet_carrying_a_disposition_label_or_a_nested_withheld_key_is_refused, test_the_run_manifest_lists_each_sealed_adjudication. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point the vendored lane at its merged agent-lab commit e6953e5 (#33) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Update the SHA256SUMS entry for vendored-lanes.json after pointing it at e6953e5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Resolve the five Codex review threads on #124 - claude_lane: delete a layer's previous return when its layer now fails or its packet is lost, before failures.json is written. - codex_lane: delete the old output for a layer whose attempts all fail. - record_verdicts: --append-rows fails before writing when a named row has no packet or lane return in the work dir. - landscape: a retained SHA256SUMS must list every retained packet exactly once with its hash; malformed or duplicate lines are rejected. - landscape: a sealed winner key must name an adopted candidate. Each fix has a regression test that fails on the previous code. The new codex_lane.py hash is appended to lane-provenance.json. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse to seal a wave whose SHA256SUMS the landscape check rejects record_verdicts now applies landscape.py's retained-SHA256SUMS rule before any write: exactly the retained packets, each once, with its actual sha256. A --write can no longer seal a wave CI would reject, since a sealed wave is never re-recorded. The test PacketWriter keeps one line per packet, as lane_packets.py does, and the three SHA256SUMS tests now expect the refusal and still check landscape.py against a wave tampered after sealing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: seathatflowsinourveins <234074349+seathatflowsinourveins@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 participant
Summary
This closes the seven findings of agent-lab-e9's verified audit (
wf_77c0ea46-091) of the layer-verdict pipeline, before the 20260923 cross-family re-record of all 32 rows. No ledger row changes here. Each item has a regression test that fails on the base and passes here; 34 tests failed and 6 errored on38847e5.layer-verdicts-<run-id>.jsonis frozen and hash-registered incatalogs/sota-convergence/layer-verdict-waves.json.build_verdicts.py --checkverifies every wave against its hash and against the rows that name it, and regenerates only the newest wave plus the handbook block.--writerefuses to rewrite an older wave.--withhold-labels, packets drop stars, forks, watchers, every*_atfield,upstream.latest,prerelease,pin_behind_upstreamand thenewcomerflag.archivedandlicensestay only when the requirement names them. Each stripped field is listed inwithheld. Default packets are byte-identical to the base.codex_absentrow stayspending_lanes.--allow-single-lane PATHrequires a dated decision record naming the layer, stored on the row, andlandscape.pyrejects the row without it.model.family; the two families must differ and match per-lane patterns. An adjudicated winner needs judges from both lane families, each family covering both presentation orders, all picking the same lane with no refutation. Anything else is a sealed split. This is checked in bothrecord_verdictsandlandscape.py.evidence/artifacts/layer-verdicts-<run-id>/run-manifest.json, recording every packet and each lane's outcome: a sealed return, a rejection with its reasons, or a missing return.landscape.pyrequires every new-wave row to appear with both lanes.record_verdicts.platform_status_forcalls the sharedscripts/platform_status.py(#117) for every platform, including the macOS flip rule.landscape.pyappliesdeclared_status_errorto new-wave rows. Layer-verdict artifacts do not count as receipts.ENFORCED_PLATFORMSstays as it is on main; the re-record PR widens it.layer-verdict-lane.jsis vendored underexamples/claude-native/workflowswith a SHA256SUMS entry. New-wave returns carry runner-written provenance and family:claude_lane.pyfor the Claude lane,codex_lane.py --modelfor the Codex lane. The Codex model name never comes from the model's own response.Hardening added in the review rounds:
--checked-atdefaults to the date in the run id.Grandfathering
GRANDFATHERED_RUN_IDS = {"20260922"}covers the sealed 2026-09-22 wave. It keeps its Opus-only, both-order adjudications, and needs no family or provenance fields and no run manifest. Every 20260922 artifact is unchanged.Measured under the shared rule: 13 of the 33 Linux
acceptedwinners, across 6 rows, would deriveconditional. They are re-derived in the re-record.Keep-but-compare
The rule "both lane families agree in both orders" is recorded in
docs/decisions/2026-09-23-verdict-integrity.md. It would be overturned by a qualified third-family judge (for example the local Qwen3-8B-AWQ worker) that clears a preregistered judge-agreement bar on the sealed adjudication packets.Review
fix_required, 1 major and 4 minor.The shared
scripts/platform_status.pygains one change, which excludes layer-verdict artifacts from registered receipts. Its owner, finalize-llm-harness-catalogs, has been told.Checks (merged with main 6e47651)
python3 -m unittest: 2,888 tests OK, 330 skipped.validate.py,evidence_manifest.py --check,landscape.py,build_verdicts.py --check,component_matrix.py --checkandtrading_gates.py --check: all pass.🤖 Generated with Claude Code