Skip to content

Jev adopt-now policy: crosswalk digit routing (A3) and screened-out labels (A7), strict reviewer schema (A4), Jev rules and the TypeSafe key's P1 consumer - #675

Merged
seathatflowsinourveins merged 8 commits into
mainfrom
c5/jev-policy
Oct 3, 2026
Merged

seathatflowsinourveins merged 8 commits into
mainfrom
c5/jev-policy

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Owner

Suggested title: Jev adopt-now policy: crosswalk screened_out routing (A7, A3), reviewer return schema (A4), README rules (A1-A3) and the TypeSafe key's provider consumer

Label: lane:foundation

Scope

  • What this PR changes: it applies the adopt-now policy items of section 3.1 of the Jev and TypeSafe design report. The crosswalk stops letting the TypeSafe screen set a final decision (A7). A pair that no reviewer saw is recorded as screened_out, and a gap that has candidate pairs but no reviewed pair gets the gap status screened_out instead of open. A pair whose gap text contains a digit always goes to review (A3). The semantic-evidence-reviewer role gets a return schema that the caller supplies (A4, all five copies). A1-A3 and A7 are written down as README rules. The root promptfoo config of blueprints/native-skill-practice/, the native-skill-practice Jev provider, becomes a recorded consumer of the TypeSafe key through the id-based runner. Nothing here calls Jev, changes a host, or changes a catalog verdict.
  • Base commit: d2fc3803e (origin/main at the first push). The brief's base was ecea28654. I rebased because main's 54a96ff3d (New WSL local models: settle the generation and embedding slots by their preregistered measurement #647) re-sorts the evidence-registry pair that Restore sorted manifests/evidence.json files[] after two adjacent custody merges #666 targeted, and Restore sorted manifests/evidence.json files[] after two adjacent custody merges #666 closed unmerged at 2026-10-03T18:17:17Z. origin/main has since moved to 1f5a791b0. git merge-tree --write-tree origin/main HEAD exits 0, so the branch still merges cleanly, including manifests/evidence.json.
  • Lane: lane:foundation
  • Owned paths touched:
    • tools/sota-convergence/gap_crosswalk.py, tests/test_gap_crosswalk_routing.py (new)
    • blueprints/native-skill-practice/README.md, blueprints/native-skill-practice/semantic-evidence-reviewer.schema.json (new), tests/test_semantic_reviewer_schema.py (new)
    • the five reviewer copies: .claude/agents/, adoption/agents/claude/ and examples/claude-native/agents/semantic-evidence-reviewer.md (still byte-identical); adoption/agents/codex/workers/semantic-evidence-reviewer.toml; examples/codex-native/agents/semantic-evidence-reviewer.toml
    • adoption/agents/codex/workers/SHA256SUMS: one row. This path is outside the brief's allowed paths (see "Decisions to veto").
    • docs/secret-storage.md: line 20 only. adoption/credential-inventory.json: the typesafe entry only.
    • manifests/evidence.json: last commit. Compared with main, it updates eight rows and adds four: the schema, its test, the routing test and adoption/agents/claude/semantic-evidence-reviewer.md. The .claude/agents/ copy has no row. On main none of the eleven .claude/agents/ files is registered, and only .claude/settings.json and one skill are registered under .claude/, so this follows that convention [inference].
  • Overlap with open PRs:

SOTA sources

  • Design source, kept outside the repository and cited by hash: "Jev and TypeSafe: typed judgments for the foundation and north-star R&D", 2026-10-03, revision r1, sha256 6fdd8bc20258e6cd7695c0a07bdd412dd8f48b653ee310a4bc04b2a580e35c70. Sections used: 3.1 (A1-A7), 5.0, 5.1 (Harness, and Cost and custody rows), 6.1, 6.2 and 7.
  • Workflow schema validation: Claude Code workflows, https://code.claude.com/docs/en/workflows. If you pass a schema on an agent() call, the subagent returns JSON matching the shape, and "If the subagent's output still fails validation after five attempts, the call fails" (read 2026-10-03). The installed client is 2.1.288, and claude --help describes --json-schema as "JSON Schema for structured output validation".
  • Codex structured output: https://developers.openai.com/codex/noninteractive says to "use --output-schema to request a final response that conforms to a JSON Schema". The installed codex-cli is 0.159.3, whose codex exec --help lists "--output-schema Path to a JSON Schema file describing the model's final response shape".
  • Strict schema form: the repository precedent tools/sota-convergence/adjudication-judge.schema.json (draft 2020-12, additionalProperties: false, every property required, no minLength, so that Codex strict output accepts it).
  • TypeSafe version policy, failure modes and terms: docs.typesafe.ai models.md says to pin the version ID and gives no retirement window. The jev-1.13 jaggedness page lists nine failure modes. The Master Customer Agreement (updated 2026-09-23) covers Telemetry in §4.1 and §4.3. The report re-read all three and hashed the pages (15351ada…, 5d471be7…); this PR cites them through the report.
  • typesafe-ai/skills, for the gated A5 record, verified 2026-10-03T17:49:53Z:
    • git ls-remote https://github.com/typesafe-ai/skills shows HEAD = refs/heads/main = refs/tags/v0.5.7 = 65a39f393687675ce170e6094757de20370365b9, committed 2026-09-12T05:42:05Z as "Release v0.5.7".
    • A blobless clone gives 65a39f39…:skills/typesafe-ai = tree 678c375b883794bf4292715a43acbba5c84681aa. SKILL.md at that commit has sha256 71ea90d7906c6554c4f4c460ef7361b2d26f59116ccdae986dc6d997b9389f52 and 10,040 bytes, equal to adoption/skills/manifest.json.
  • gitleaks 8.30.1: pinned in catalogs/sota-convergence/manifest-20260929.json (secrets-credentials, ci-supply-chain and quality-evaluation layers).
  • Key delivery: tools/credentials/credential_run.py, documented in docs/secret-storage.md, "Using a key". The typesafe entry's status user_only_paid is in the runner's INJECTABLE_STATUSES. The inventory form follows the existing claude-oauth-token entry, which records the runner form under loaders.

Evidence-class table

Claim Evidence class Command / receipt
The retained 92bb279 crosswalk still reproduces byte for byte under its recorded 2026-09-23 rule local_integration python3 tools/sota-convergence/gap_crosswalk.py build --check (exit 0, "crosswalk up to date"); tests.test_gap_crosswalk 8/8
A new crosswalk records a pair that no reviewer saw as screened_out and names its rule in method.routing. A gap with candidate pairs but no reviewed pair gets the gap status screened_out, a gap whose reviewed pair does not address it stays open, and the page's Totals table lists screened_out. A gap text with a digit sends every pair to review, and the build refuses such a pair if it has no review synthetic tests.test_gap_crosswalk_routing, CrosswalkRoutingTests (4 tests)
Negative control: the 2026-09-23 rule fails the A7, A7 gap-status and A3 checks on the same fixture and reads the unreviewed gap as open. Earlier tool versions fail the new tests synthetic the in-suite control passes. Tool at 8d300d71e (blob 15895771): exit 1, FAILED (failures=1), 'open' != 'screened_out'. Tool at ecea28654 (blob 81fb44f2): exit 1, FAILED (failures=3, errors=1): 'not_addressed' != 'screened_out'; 'open' != 'screened_out'; (False, None) != (True, 'digit_in_gap'); no attribute pair_entry
Retained 92bb279 inputs (offline): the old rule reproduces all 835 recorded pair entries. Of the 725 unreviewed pairs, the new rule refuses 369, which have a digit in the gap text (98 gaps in 19 layers), and marks the other 356 screened_out. 135 gaps have only unreviewed pairs, 73 of them without a digit local_integration (retained records) tests.test_gap_crosswalk_routing, RetainedLedgerRoutingTests, plus read-only counts over catalogs/landscape/gap-crosswalk-92bb279.json
The reviewer schema is strict, its dispositions match response.cjs, and every copy names the schema and the three A4 fields. The schema also holds source_refs, correction, limits, skill_path and model, beyond the three fields section 3.1 names. Codex strict output requires additionalProperties: false, so a three-field schema would forbid the citations, corrections and limits that the role body already requires synthetic (structural) tests.test_semantic_reviewer_schema 3/3; a mutated schema is the control
The role carriers keep their pins and rules local_integration test_codex_roles 38, test_codex_agents 19, test_install_claude_profile 79 (4 YAML tests skipped, also skipped on main), test-contract-mutations.mjs 74/74, test-envelope.mjs 254/254
The inventory and runbook agree, and every consumer path exists local_integration test_credential_status 44, test_credential_run 92; scripts/validate.py (runs inventory_errors)
A live agent({schema}) or codex exec --output-schema call with this schema not run residual: needs a model call

Local commands run

These ran on HEAD f4ad846a6 with TMPDIR=/var/tmp/c5-jev-policy nice -n 19:

$ python3 scripts/validate.py
exit 0 {"components": 69, "hashed_files": 9545, "profiles": 4, "receipts": 193, "status": "passed"}
$ python3 tools/sota-convergence/gap_crosswalk.py build --check
exit 0 crosswalk up to date
$ python3 tools/sota-convergence/build_verdicts.py --check
exit 0 {"status": "checked", "waves": ["20260922"], "current": "20260922", ...}
$ python3 tools/sota-convergence/gap_wave_ledger.py --wave gap-wave2-20260923 --wave gap-wave3-20260923 --owner gap-resolution --check
exit 0 {"status": "checked", "counts": {"advanced": 59, "settled": 10}}
$ python3 scripts/component_matrix.py --check
exit 0 {"rows": 32, "status": "checked"}
$ python3 scripts/evidence_manifest.py --check
exit 0 {"files": 9545, "status": "passed"}
$ python3 -m unittest tests.<module>   (run per module)
exit 0 each: test_gap_crosswalk 8, test_gap_crosswalk_routing 5, test_semantic_reviewer_schema 3, test_credential_status 44,
test_credential_run 92, test_secret_path_guard 82 (skipped=2), test_canary_proof 143, test_credential_boot_receipt 21,
test_component_matrix 63, test_codex_roles 38, test_codex_agents 19, test_install_claude_profile 79 (skipped=4)
$ (blueprints/native-skill-practice) node --test test-contract.cjs
exit 0 (tests 6, pass 6, fail 0)
$ (examples/claude-native/workflows) node test-contract-mutations.mjs ; node test-envelope.mjs
exit 0 SUMMARY passed=74 failed=0 total=74 ; exit 0 SUMMARY passed=254 failed=0 total=254
$ tests.test_gap_crosswalk_routing with its TOOL path set to the tool blob at 8d300d71e (the working tree untouched)
exit 1 FAILED (failures=1)   # 'open' != 'screened_out' in test_a7_a_gap_with_no_reviewed_pair_is_not_open
$ tests.test_gap_crosswalk_routing with its TOOL path set to the tool blob at ecea28654
exit 1 FAILED (failures=3, errors=1)
$ git merge-tree --write-tree origin/main HEAD   (origin/main 1f5a791b0)
exit 0
$ python3 -m unittest   (CI's full discovery, .github/workflows/validate.yml:224)
exit 1 Ran 9959 tests in 2677.748s FAILED (failures=3, skipped=850)
$ python3 -m unittest tests.test_adaptive_paper_metrics
exit 0 Ran 39 tests OK
$ python3 -m unittest tests.test_gpt6_family_tiering_20260926
exit 1 FAILED (failures=1)   # "run_arm: the temporary directory /tmp must lie outside the state directory and every repository"
$ python3 -m unittest tests.test_windows_terminal_defaults
exit 1 FAILED (failures=1)   # Lists differ: ['auth_storage_failure'] != []

None of the three failing modules, or any file they read, is changed by this branch. test_adaptive_paper_metrics ("exporter never registered its target") passes when run alone, so its failure in the full run was load timing. The other two fail because of this host. This host has a /tmp/.git, which the runner refuses as a temporary directory inside a repository. The installed Claude Code 2.1.288 binary knows the notification type auth_storage_failure, which the table in tests/test_windows_terminal_defaults.py lacks; CI skips that test because it has no installed client.

Decision record

There is none. Section 3.1 does not call for one, and section 6.1 limits this PR to README text, the five reviewer copies and the crosswalk change. The dated rationale and the overturn condition are in the README section "Jev rules (2026-10-03)" and in the routing comment of gap_crosswalk.py. That overturn condition is two preregistered, held-out, human-labelled replications (P1, then P1b) before the screen may set a final status.

Gated items (not in this PR)

  1. A1 canary freeze. Freeze the 10 A1 canary cases with their stored answers. They are not frozen: the draft P1 freeze packet (local branch c5/jev-p1 at b9f93db42, not yet on origin; blueprints/native-skill-practice/p1/draft/freeze-manifest.draft.json) defines the role at line 17, records "canary": [] at line 117 and lists canary under missing_roles at line 127. Sections 3.1 A1 and 5.0 make the canary a precondition of every run: run it first, and stop if it flips more than 2 of 10. Storing the answers takes Jev calls [inference], so this waits on the same user go as the P1 run.
  2. A5 catalog pin record. This is the first precondition of option 1, and it is blocked. Setting the candidate:typesafe-ai-skills winner's pin in catalogs/landscape/foundation.json makes python3 tools/sota-convergence/build_verdicts.py --check exit 1 with "foundation/instructions-skills names wave 20260922 but differs from its frozen entry in catalogs/sota-convergence/layer-verdicts-20260922.json (re-record it under a new --run-id instead)", and the check also flags the handbook block. The receipt recorder reads only pin: host_receipts.winner_pins reads it, and "unpinned" never binds. So a separate field would not unblock the recorder. The fix is a re-record of the instructions-skills row under a new run id, which is the layer-verdict lane's job. It touches layer-verdicts-<run-id>.json, layer-verdict-waves.json, docs/grand-catalog-handbook.md and the row's lanes.sealed_base. The verified ref and tree above make the pin edit itself one step. The skill-preload re-probe on Claude Code 2.1.288, the new distribution and Codex stays a residual.
  3. The crosswalk's readers. scripts/component_matrix.py (lines 209 and 222) counts only gaps whose status is open, so it would drop screened_out gaps from its open counts. tools/sota-convergence/gap_wave_ledger.py (line 149) schedules every executable_now gap that is not settled, so it would schedule a screened_out gap without first reviewing its pairs. Both must treat screened_out as unknown before any crosswalk built under the new rule replaces the retained 92bb279 file. Both are outside this PR's paths, and both read only that file today, so nothing changes before then.
  4. The retained 92bb279 crosswalk under A7 alone. Relabelling its 725 unreviewed pairs screened_out takes no model call. It needs a rule id without the digit route (not added here, because nothing would use it yet) and an owner for re-records outside this PR's paths: catalogs/landscape/gap-crosswalk-92bb279.json, docs/gap-crosswalk-92bb279.md and the gap-wave ledgers that embed crosswalk_sha256 (gap_wave_ledger.py:161). 135 gaps would change from open to screened_out, so item 3 comes first.
  5. The digit route (A3) applied to that crosswalk after the fact (optional). This needs Opus reviews of the 369 unreviewed pairs whose gap text contains a digit (98 gaps in 19 layers). The 73 gaps without a digit whose pairs are all unreviewed would stay screened_out.
  6. P1's harness as a key consumer. When c5/jev-p1 lands on main, add blueprints/native-skill-practice/p1/promptfooconfig.yaml to the typesafe entry's environment_only_consumers. That file is the P1 harness and reads the key at lines 34, 55 and 76. Reword docs/secret-storage.md:20 and the entry's notes at the same time. It cannot go in earlier, because scripts/credential_status.py:180-189 rejects a consumer path that does not exist. The brief pointed at the root config's line 17, which belongs to the 2026-09-21 eight-case diagnostic, so this PR now names that file the native-skill-practice Jev provider rather than the P1 run.
  7. P1 live run. The promptfoo provider smoke checks are model calls and wait for the user's go. That go rests on the coordinator's tracker entry (about 17:41Z). The user did not state it in this session, so the user should confirm it at review. Whether the TypeSafe account is funded is unknown.
  8. A6, the on listing on this host. This belongs to the client-configuration lane.

Decisions to veto

  • adoption/agents/codex/workers/SHA256SUMS is outside the brief's allowed paths. Its one changed row (semantic-evidence-reviewer.toml, ee924bbe… to 0a406ec9…) follows from editing the pinned Codex worker copy: tools/adoption/codex_roles.py pins that file through this SHA256SUMS, and test_codex_roles passes 38/38 with the new row. Accept the path, or veto A4's Codex worker copy with it.
  • The branch base is d2fc3803e, not the brief's ecea28654. See "Scope".
  • The gap-level marker is a new gap status value, screened_out, and not a count field. A reader that knows only open therefore cannot mistake such a gap for a reviewed one. A gap that mixes reviewed pairs that do not address it with screened-out pairs stays open, and its screened-out pairs show only per pair (56 such gaps in the retained ledger).
  • No A7-only rule id is added (gated item 4).
  • adoption/agents/claude/semantic-evidence-reviewer.md is registered; the .claude/agents/ copy is not (see "Scope").
  • The reviewer schema holds eight fields beyond the three that section 3.1 names (see the evidence table).

Host evidence

Not applicable: this PR adds or changes nothing under evidence/hosts/.

Coordinator confirmations (2026-10-03)

  • The user's P1 go: the user's answer in the coordinator session at about 17:38Z, verbatim: "1 and all further can be pretending with stages". It was given to the question of how far the Jev measurements go, and its option one was the first measurement only. It covers the TypeSafe key's new P1 consumer and P1's staged run. "Pretending" is the user's word for "pending": P2 and P3 follow in stages. The live P1 run still needs the frozen canary and the label pack first.
  • A5: the pin record waits for the layer-verdict lane to re-record instructions-skills under a new run id. build_verdicts.py --check rightly refuses an edit to the frozen 2026-09-22 entry.

Checklist

  • New/changed GitHub Actions are pinned to a full commit SHA with a version comment (no floating tags). (none changed)
  • New/changed workflows declare top-level permissions: contents: read (or a narrower, explicitly justified addition). (none changed)
  • No secrets are printed, logged or committed; no new required secret was added without a documented owner. (No value was read. The existing optional typesafe key gains one documented consumer.)
  • No new paid hosting, subscription or billing surface was introduced.
  • Peer-owned untracked files and worktrees were preserved (not deleted, moved or overwritten).

🤖 Generated with Claude Code

Scout and others added 7 commits October 3, 2026 14:31
…d (A3)

A pair that no reviewer saw now gets final_decision screened_out, which is no
final status and counts toward no gap status, instead of a screen-set
not_addressed. A pair whose gap text contains a digit always enters review.
Every new crosswalk takes this rule and records it in method.routing. The
retained 92bb279 ledger keeps its 2026-09-23 rule, so build --check still
reproduces it byte for byte; re-recording it under the new rule first needs
reviews of its 369 unreviewed pairs whose gap text contains a digit.

tests/test_gap_crosswalk_routing.py builds a synthetic crosswalk under each
rule. The 2026-09-23 rule is the negative control: it fails the A7 and A3
checks, and so does the tool at ecea286. A second test runs both rules over
the retained 92bb279 judgments and reviews offline.

Source: design report "Jev and TypeSafe: typed judgments for the foundation and
north-star R&D" (2026-10-03, revision r1, sha256 6fdd8bc2), section 3.1, A3
and A7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blueprints/native-skill-practice/semantic-evidence-reviewer.schema.json holds,
per case, case_id, final_disposition (supported, contradicted or insufficient)
and retained_provider_disposition, plus the source references, correction and
limits the role already returns, and skill_path and model. It follows the
strict form of tools/sota-convergence/adjudication-judge.schema.json (every
property required, no other allowed, no minLength), which Codex strict output
needs.

Agent frontmatter has no schema field, so the five copies of the role name the
schema and the caller supplies it: agent({agentType, schema}) in a workflow,
which validates the return and fails after five invalid attempts
(code.claude.com/docs/en/workflows), the coordinator for a single Agent-tool
call, and codex exec --output-schema for the Codex copies
(developers.openai.com/codex/noninteractive). The three Claude copies stay
byte-identical; adoption/agents/codex/workers/SHA256SUMS carries the Codex
worker's new digest.

tests/test_semantic_reviewer_schema.py checks the strict form (a mutated copy is
the control), the disposition labels against response.cjs, and that every copy
names the schema and the three fields.

Source: design report "Jev and TypeSafe: typed judgments for the foundation and
north-star R&D" (2026-10-03, revision r1, sha256 6fdd8bc2), section 3.1, A4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…key for P1

blueprints/native-skill-practice/README.md gains a "Jev rules (2026-10-03)"
section at its end, so existing line citations of the file still hold: the
version pin and drift canary (A1), the custody rule (A2), the controls for the
nine documented failure modes, with every claim that contains a digit sent to
Opus (A3), the reviewer schema at the call site (A4) and the crosswalk's
screened_out routing (A7). None of them adopts Jev for a feature.

The key's consumers now include the P1 claim-verification run of the promptfoo
provider (promptfooconfig.yaml reads the variable from its environment), each
consumer started through the id-based runner tools/credentials/credential_run.py
(docs/secret-storage.md, "Using a key"). The inventory entry follows the
claude-oauth-token precedent: the runner form under loaders, the config under
environment_only_consumers. The README's run command uses the runner instead of
--env-file. A trading-lane use still needs its own, separately authorized key.
The coordinator relayed the user's 2026-10-03 go for P1.

Source: design report "Jev and TypeSafe: typed judgments for the foundation and
north-star R&D" (2026-10-03, revision r1, sha256 6fdd8bc2), sections 3.1, 5.1
(harness and custody rows) and 6.2 (key consumers and delivery).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file updates the digests of the eight changed files
that main already registers and adds the reviewer schema and the two new test
modules at their sorted places. .claude/agents/ and adoption/agents/claude/
copies stay unregistered, as on main. scripts/evidence_manifest.py --check
passes, and so does scripts/validate.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…not open (A7)

Under the 2026-10-03 routing rule, a gap whose candidate pairs were all
screened out read "open", the same value as a gap whose reviewed pairs do
not address it. It now gets its own gap status, screened_out, and the
rendered Totals table lists it. The retained 92bb279 crosswalk keeps the
2026-09-23 rule and still reproduces byte for byte. The routing comment,
method.routing.rule and the page now describe only what the tool records:
the two readers (scripts/component_matrix.py, gap_wave_ledger.py) read only
the retained ledger and do not handle screened_out yet.

The fixture gains a gap whose pairs are all screened out and a gap whose
reviewed pair does not address it. The new test fails on the previous tool
(8d300d7) and on the base tool (ecea286).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d its readers (A7), Jev provider naming

A1: the 10 canary cases and their stored answers are not frozen yet (the
draft P1 freeze packet lists the canary as a missing role), so the README no
longer says the packet names them. A7: the README describes the gap status
screened_out, names the two readers that do not handle it yet, and splits
the retained crosswalk's re-record: A7 alone takes no model call, while the
digit route (A3) applied after the fact adds 369 Opus reviews. The runbook
line and the inventory note name the root promptfoo config as the
native-skill-practice Jev provider, not as the P1 run; the P1 harness is a
different file that is not on main yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Updates the digests of gap_crosswalk.py, its routing test, the
native-skill-practice README, docs/secret-storage.md and
adoption/credential-inventory.json, and registers
adoption/agents/claude/semantic-evidence-reviewer.md, which the A4 commit
changed without a row. The .claude/agents/ copy stays unregistered, as all
eleven .claude/agents/ files are on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 3, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-03T21:51:37.755594Z f4ad846 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

💡 Codex Review

- **Return.** Return one JSON object in the shape of the repository's `blueprints/native-skill-practice/semantic-evidence-reviewer.schema.json`: a `cases` list in which each case gives `case_id`, your `final_disposition` (supported, contradicted or insufficient), the `retained_provider_disposition` exactly as supplied (null when none was), `source_refs`, `correction` and `limits`, plus `skill_path` and `model` ("unavailable" when the client does not expose them). Keep the retained provider judgment apart from your own source review. The caller passes that schema with `codex exec --output-schema`, which checks the shape of your final response.

P1 Badge Ship the schema with installed reviewer roles

When this role is installed user-wide or copied into another project, the documented installers copy only the agent definition (tools/adoption/install_claude_profile.py:12 and examples/codex-native/README.md:13-16), while the referenced schema remains exclusively in this checkout. Consequently, a caller following the new requirement to pass this path to codex exec --output-schema or agent({agentType, schema}) cannot resolve it and the structured review fails before the reviewer runs. Install/copy the schema alongside the role or reference a stable packaged location.

AGENTS.md reference: AGENTS.md:L7-L7


"schema_version": 1, "id": CROSSWALK_ID, "checked_at": "2026-09-23",
"scope": ("Maps the 2026-09-22 gap-resolution receipts onto every open_gaps entry of the 32 landscape rows at "
"source_revision, and gives each gap a blocker category and an owner for the next evidence wave. A status "
"here records only whether an existing receipt addresses the gap; it changes no verdict. Rows re-recorded "
"after source_revision need a new crosswalk."),

P2 Badge Parameterize provenance for new crosswalks

When CROSSWALK_ID is anything other than the retained gap-crosswalk-92bb279, build() selects the newly introduced routing policy and records method.routing, but it still stamps the resulting evidence as checked on 2026-09-23 and describes it as mapping the 2026-09-22 receipts. Thus a future crosswalk built through this new code path is published with the retained run's provenance rather than its own date and inputs; make this metadata crosswalk-specific instead of hard-coding the historical values.

AGENTS.md reference: AGENTS.md:L14-L14



P1 Badge Reject reviewer responses that omit all cases

For any review packet, the schema currently accepts {"cases":[],"skill_path":"","model":""} because cases has no minItems and no caller added by this change checks that the returned IDs equal the requested IDs. Since the documented workflow treats validation against this schema as the return gate, a reviewer can omit every assigned case and still be accepted after the first attempt, silently leaving the semantic evidence unreviewed. Require a nonempty array and have the caller verify the exact expected case-ID set.


REQUEST_TIMEOUT_MS=15000 python3 ../../tools/credentials/credential_run.py typesafe -- \
timeout 180s promptfoo eval -c promptfooconfig.yaml \
--no-cache --max-concurrency 1 --repeat 1 --no-share --no-write --no-table \
--output /private/path/to/results.json

P1 Badge Gate live Jev calls on the required canary

When a user follows this newly credential-wired command, Promptfoo immediately makes the eight uncached Jev requests, even though the A1 policy below says every run must first rescore a ten-case drift canary and that no run may start until those cases are frozen. Neither this command nor gap_crosswalk.py judge performs or verifies that canary, so the documented path can produce paid, policy-invalid results under an unmeasured model drift. Add an executable preflight gate before exposing either live-call path.


"final_disposition": {
"type": "string",
"enum": [
"supported",
"contradicted",
"insufficient"
]

P1 Badge Preserve the crosswalk's three review decisions

When the new digit route sends crosswalk pairs to the semantic-evidence-reviewer named by gap_crosswalk.py, this schema cannot return the decisions that pair_entry() consumes: the crosswalk requires settles, partially, or not_addressed, but the enforced enum permits only supported, contradicted, or insufficient. There is no documented lossless mapping—especially for partially—so the newly required reviews cannot be fed into review-results.json without an unvalidated translation or bypassing the schema. Use a task-specific crosswalk schema or preserve the crosswalk decision vocabulary in this contract.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

SOURCE ACCEPT WITH LIMITS for #675 head f4ad846a68c7fa17e14c28b5febec1381c5eab55, as a policy and reviewer-source change. The policy explicitly adopts Jev for no feature; feature/default qualification and the preregistered human-labelled replications remain separate.

Root independently recomputed all 27 present old/head/base/main body bindings (23 unique blobs, including all 14 head paths), eight supporting repository bodies and three pinned upstream bodies. All 12 registered payload byte/hash bindings match. The 22:08:03Z public guard still names this head; the earlier metadata-query HTTP503 is retained separately.

Carry these limits:

  • The user-wide role requires the caller to pass the repository schema via --output-schema. Inspected role installers copy the role carriers. External-workspace callers therefore need an explicit schema locator or retained-checkout prerequisite; standalone schema availability is unqualified.
  • The crosswalk consumes review-results.json. Its protocol differs from the new reviewer schema, but the inspected sources do not establish an executable linkage between them. This is unqualified interoperability, not an observed integration failure. Its generic routing branch retains historical date/scope metadata; future current-evidence publication needs explicit provenance treatment.
  • The canary is an unfrozen manual prerequisite. No canary run or automatic enforcement is accepted here. The external design-report digest alone does not verify that report's empirical claims.

This is source acceptance. Current-main refresh, prospective merge preservation, installed-role behavior, native calls, labels and P1 execution remain separate checks. No tests, service/model calls, credential access, installation or Claude-owned edits occurred in this review.

Only manifests/evidence.json conflicted: main's registry taken, the 12 PR-owned rows
re-registered. docs/secret-storage.md also changed on main and merged without
conflict; its row carries the merged content and the PR's own hunk is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

SOURCE ACCEPT WITH LIMITS for #675 head 52b3fbb908c23380a7540cca3fd17762e3e3a0be, carrying the accepted f4ad policy verdict. This is source carry, with the same policy/runtime limits.

Root independently rechecked all 14 current owned body/Git bindings and the three full current/base/main registry bindings, all 12 owned payload registrations, 10,262 foreign actual-base Git membership tuples in their returned order, 9,557 full ordered foreign registry rows and all non-files registry fields. The actual base is ed60d4384c8cc495007b888342cc523a9b62758f. Twelve owned payload identities are reused unchanged; the owned docs/secret-storage.md plus/minus policy lines remain literal-exact through its merge with main. The full diff headers/context differ, so this is a policy-hunk comparison rather than whole diff-byte equality. The unregistered role file retains its prior status.

The policy still adopts Jev for no feature. Human-labelled replication, role-return/legacy review-result interoperability, external-workspace schema location and native runtime qualification remain as in the prior verdict. Existing pinned primary source acceptance is reused because those payloads and references are unchanged. No new installation, feature admission or model call follows from this merge.

Observed main 984bf4b273f1737a4cb65acb29c9f03e1d044d8a adds one later foreign decision document/registration absent from this raw branch. No prospective merge was executed, and this raw gap is not an eventual-deletion claim. The earlier HTTP503 guard failure remains retained; subsequent exact-head source metadata returned0. No root repository edit, auth/credential/config access, host apply, trial, test or merge occurred.

@seathatflowsinourveins
seathatflowsinourveins merged commit b29036f into main Oct 3, 2026
25 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the c5/jev-policy branch October 3, 2026 23:03
@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

Actual source landing closed: main b29036f05905f75df934603eb8f1d2248a01c543 has sole parent 984bf4b273f1737a4cb65acb29c9f03e1d044d8a. Primary MERGED metadata binds this commit to exact #675 head 52b3fbb908c23380a7540cca3fd17762e3e3a0be, merged 23:03:23Z. Root recomputed all three full commit identities, all 25 present before/after body/Git bindings and all 78 returned capture streams; all 39 native source commands returned 0.

All 13 landed owned content tuples and 12 complete owned registry rows match the accepted head. Against the actual parent, root independently rederived preservation of 10,263 full foreign Git tuples/order and 9,558 complete foreign registry rows/order, all nonfile fields and the unchanged 193 receipts/29 convergence records. Registry rows become 9,566→9,570. The parent's Polaris retirement document/row is retained despite its raw-branch gap. The semantic-evidence-reviewer source remains unregistered.

This closes actual public source integration. The source verdict's limits remain: adopt Jev for no feature until qualification, publisher-reported fixture/canary execution, native role/interoperability and target-host acceptance are separate. No new test, model trial, install, measured default or native lifecycle acceptance was performed for this carry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant