Repository navigation
Jev adopt-now policy: crosswalk digit routing (A3) and screened-out labels (A7), strict reviewer schema (A4), Jev rules and the TypeSafe key's P1 consumer - #675
Conversation
…d (A3) A pair that no reviewer saw now gets final_decision screened_out, which is no final status and counts toward no gap status, instead of a screen-set not_addressed. A pair whose gap text contains a digit always enters review. Every new crosswalk takes this rule and records it in method.routing. The retained 92bb279 ledger keeps its 2026-09-23 rule, so build --check still reproduces it byte for byte; re-recording it under the new rule first needs reviews of its 369 unreviewed pairs whose gap text contains a digit. tests/test_gap_crosswalk_routing.py builds a synthetic crosswalk under each rule. The 2026-09-23 rule is the negative control: it fails the A7 and A3 checks, and so does the tool at ecea286. A second test runs both rules over the retained 92bb279 judgments and reviews offline. Source: design report "Jev and TypeSafe: typed judgments for the foundation and north-star R&D" (2026-10-03, revision r1, sha256 6fdd8bc2), section 3.1, A3 and A7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blueprints/native-skill-practice/semantic-evidence-reviewer.schema.json holds,
per case, case_id, final_disposition (supported, contradicted or insufficient)
and retained_provider_disposition, plus the source references, correction and
limits the role already returns, and skill_path and model. It follows the
strict form of tools/sota-convergence/adjudication-judge.schema.json (every
property required, no other allowed, no minLength), which Codex strict output
needs.
Agent frontmatter has no schema field, so the five copies of the role name the
schema and the caller supplies it: agent({agentType, schema}) in a workflow,
which validates the return and fails after five invalid attempts
(code.claude.com/docs/en/workflows), the coordinator for a single Agent-tool
call, and codex exec --output-schema for the Codex copies
(developers.openai.com/codex/noninteractive). The three Claude copies stay
byte-identical; adoption/agents/codex/workers/SHA256SUMS carries the Codex
worker's new digest.
tests/test_semantic_reviewer_schema.py checks the strict form (a mutated copy is
the control), the disposition labels against response.cjs, and that every copy
names the schema and the three fields.
Source: design report "Jev and TypeSafe: typed judgments for the foundation and
north-star R&D" (2026-10-03, revision r1, sha256 6fdd8bc2), section 3.1, A4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…key for P1 blueprints/native-skill-practice/README.md gains a "Jev rules (2026-10-03)" section at its end, so existing line citations of the file still hold: the version pin and drift canary (A1), the custody rule (A2), the controls for the nine documented failure modes, with every claim that contains a digit sent to Opus (A3), the reviewer schema at the call site (A4) and the crosswalk's screened_out routing (A7). None of them adopts Jev for a feature. The key's consumers now include the P1 claim-verification run of the promptfoo provider (promptfooconfig.yaml reads the variable from its environment), each consumer started through the id-based runner tools/credentials/credential_run.py (docs/secret-storage.md, "Using a key"). The inventory entry follows the claude-oauth-token precedent: the runner form under loaders, the config under environment_only_consumers. The README's run command uses the runner instead of --env-file. A trading-lane use still needs its own, separately authorized key. The coordinator relayed the user's 2026-10-03 go for P1. Source: design report "Jev and TypeSafe: typed judgments for the foundation and north-star R&D" (2026-10-03, revision r1, sha256 6fdd8bc2), sections 3.1, 5.1 (harness and custody rows) and 6.2 (key consumers and delivery). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file updates the digests of the eight changed files that main already registers and adds the reviewer schema and the two new test modules at their sorted places. .claude/agents/ and adoption/agents/claude/ copies stay unregistered, as on main. scripts/evidence_manifest.py --check passes, and so does scripts/validate.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…not open (A7) Under the 2026-10-03 routing rule, a gap whose candidate pairs were all screened out read "open", the same value as a gap whose reviewed pairs do not address it. It now gets its own gap status, screened_out, and the rendered Totals table lists it. The retained 92bb279 crosswalk keeps the 2026-09-23 rule and still reproduces byte for byte. The routing comment, method.routing.rule and the page now describe only what the tool records: the two readers (scripts/component_matrix.py, gap_wave_ledger.py) read only the retained ledger and do not handle screened_out yet. The fixture gains a gap whose pairs are all screened out and a gap whose reviewed pair does not address it. The new test fails on the previous tool (8d300d7) and on the base tool (ecea286). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d its readers (A7), Jev provider naming A1: the 10 canary cases and their stored answers are not frozen yet (the draft P1 freeze packet lists the canary as a missing role), so the README no longer says the packet names them. A7: the README describes the gap status screened_out, names the two readers that do not handle it yet, and splits the retained crosswalk's re-record: A7 alone takes no model call, while the digit route (A3) applied after the fact adds 369 Opus reviews. The runbook line and the inventory note name the root promptfoo config as the native-skill-practice Jev provider, not as the P1 run; the P1 harness is a different file that is not on main yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Updates the digests of gap_crosswalk.py, its routing test, the native-skill-practice README, docs/secret-storage.md and adoption/credential-inventory.json, and registers adoption/agents/claude/semantic-evidence-reviewer.md, which the A4 commit changed without a row. The .claude/agents/ copy stays unregistered, as all eleven .claude/agents/ files are on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
💡 Codex ReviewWhen this role is installed user-wide or copied into another project, the documented installers copy only the agent definition ( AGENTS.md reference: AGENTS.md:L7-L7 native-agent-stack/tools/sota-convergence/gap_crosswalk.py Lines 407 to 411 in f4ad846 When AGENTS.md reference: AGENTS.md:L14-L14 For any review packet, the schema currently accepts native-agent-stack/blueprints/native-skill-practice/README.md Lines 96 to 99 in f4ad846 When a user follows this newly credential-wired command, Promptfoo immediately makes the eight uncached Jev requests, even though the A1 policy below says every run must first rescore a ten-case drift canary and that no run may start until those cases are frozen. Neither this command nor When the new digit route sends crosswalk pairs to the ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
SOURCE ACCEPT WITH LIMITS for #675 head Root independently recomputed all 27 present old/head/base/main body bindings (23 unique blobs, including all 14 head paths), eight supporting repository bodies and three pinned upstream bodies. All 12 registered payload byte/hash bindings match. The 22:08:03Z public guard still names this head; the earlier metadata-query HTTP503 is retained separately. Carry these limits:
This is source acceptance. Current-main refresh, prospective merge preservation, installed-role behavior, native calls, labels and P1 execution remain separate checks. No tests, service/model calls, credential access, installation or Claude-owned edits occurred in this review. |
Only manifests/evidence.json conflicted: main's registry taken, the 12 PR-owned rows re-registered. docs/secret-storage.md also changed on main and merged without conflict; its row carries the merged content and the PR's own hunk is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
SOURCE ACCEPT WITH LIMITS for #675 head Root independently rechecked all 14 current owned body/Git bindings and the three full current/base/main registry bindings, all 12 owned payload registrations, 10,262 foreign actual-base Git membership tuples in their returned order, 9,557 full ordered foreign registry rows and all non-files registry fields. The actual base is The policy still adopts Jev for no feature. Human-labelled replication, role-return/legacy review-result interoperability, external-workspace schema location and native runtime qualification remain as in the prior verdict. Existing pinned primary source acceptance is reused because those payloads and references are unchanged. No new installation, feature admission or model call follows from this merge. Observed main |
|
Actual source landing closed: main All 13 landed owned content tuples and 12 complete owned registry rows match the accepted head. Against the actual parent, root independently rederived preservation of 10,263 full foreign Git tuples/order and 9,558 complete foreign registry rows/order, all nonfile fields and the unchanged 193 receipts/29 convergence records. Registry rows become 9,566→9,570. The parent's Polaris retirement document/row is retained despite its raw-branch gap. The semantic-evidence-reviewer source remains unregistered. This closes actual public source integration. The source verdict's limits remain: adopt Jev for no feature until qualification, publisher-reported fixture/canary execution, native role/interoperability and target-host acceptance are separate. No new test, model trial, install, measured default or native lifecycle acceptance was performed for this carry. |
Suggested title: Jev adopt-now policy: crosswalk screened_out routing (A7, A3), reviewer return schema (A4), README rules (A1-A3) and the TypeSafe key's provider consumer
Label:
lane:foundationScope
screened_out, and a gap that has candidate pairs but no reviewed pair gets the gap statusscreened_outinstead ofopen. A pair whose gap text contains a digit always goes to review (A3). Thesemantic-evidence-reviewerrole gets a return schema that the caller supplies (A4, all five copies). A1-A3 and A7 are written down as README rules. The root promptfoo config ofblueprints/native-skill-practice/, the native-skill-practice Jev provider, becomes a recorded consumer of the TypeSafe key through the id-based runner. Nothing here calls Jev, changes a host, or changes a catalog verdict.d2fc3803e(origin/main at the first push). The brief's base wasecea28654. I rebased because main's54a96ff3d(New WSL local models: settle the generation and embedding slots by their preregistered measurement #647) re-sorts the evidence-registry pair that Restore sorted manifests/evidence.json files[] after two adjacent custody merges #666 targeted, and Restore sorted manifests/evidence.json files[] after two adjacent custody merges #666 closed unmerged at 2026-10-03T18:17:17Z. origin/main has since moved to1f5a791b0.git merge-tree --write-tree origin/main HEADexits 0, so the branch still merges cleanly, includingmanifests/evidence.json.lane:foundationtools/sota-convergence/gap_crosswalk.py,tests/test_gap_crosswalk_routing.py(new)blueprints/native-skill-practice/README.md,blueprints/native-skill-practice/semantic-evidence-reviewer.schema.json(new),tests/test_semantic_reviewer_schema.py(new).claude/agents/,adoption/agents/claude/andexamples/claude-native/agents/semantic-evidence-reviewer.md(still byte-identical);adoption/agents/codex/workers/semantic-evidence-reviewer.toml;examples/codex-native/agents/semantic-evidence-reviewer.tomladoption/agents/codex/workers/SHA256SUMS: one row. This path is outside the brief's allowed paths (see "Decisions to veto").docs/secret-storage.md: line 20 only.adoption/credential-inventory.json: thetypesafeentry only.manifests/evidence.json: last commit. Compared with main, it updates eight rows and adds four: the schema, its test, the routing test andadoption/agents/claude/semantic-evidence-reviewer.md. The.claude/agents/copy has no row. On main none of the eleven.claude/agents/files is registered, and only.claude/settings.jsonand one skill are registered under.claude/, so this follows that convention [inference].docs/secret-storage.mdin one hunk,@@ -1554,3 +1554,10 @@. This PR changes line 20 only, so the hunks are far apart.catalogs/landscape/foundation.jsonat lines 48-62, 4560 and 4582-4610. This PR does not edit that file, because A5's pin record is gated (see "Gated items" below).manifests/evidence.json. Whichever merges later re-registers its files under the hot-file protocol.SOTA sources
6fdd8bc20258e6cd7695c0a07bdd412dd8f48b653ee310a4bc04b2a580e35c70. Sections used: 3.1 (A1-A7), 5.0, 5.1 (Harness, and Cost and custody rows), 6.1, 6.2 and 7.schemaon anagent()call, the subagent returns JSON matching the shape, and "If the subagent's output still fails validation after five attempts, the call fails" (read 2026-10-03). The installed client is 2.1.288, andclaude --helpdescribes--json-schemaas "JSON Schema for structured output validation".--output-schemato request a final response that conforms to a JSON Schema". The installed codex-cli is 0.159.3, whosecodex exec --helplists "--output-schema Path to a JSON Schema file describing the model's final response shape".tools/sota-convergence/adjudication-judge.schema.json(draft 2020-12,additionalProperties: false, every property required, nominLength, so that Codex strict output accepts it).models.mdsays to pin the version ID and gives no retirement window. The jev-1.13 jaggedness page lists nine failure modes. The Master Customer Agreement (updated 2026-09-23) covers Telemetry in §4.1 and §4.3. The report re-read all three and hashed the pages (15351ada…,5d471be7…); this PR cites them through the report.git ls-remote https://github.com/typesafe-ai/skillsshows HEAD =refs/heads/main=refs/tags/v0.5.7=65a39f393687675ce170e6094757de20370365b9, committed 2026-09-12T05:42:05Z as "Release v0.5.7".65a39f39…:skills/typesafe-ai= tree678c375b883794bf4292715a43acbba5c84681aa. SKILL.md at that commit has sha25671ea90d7906c6554c4f4c460ef7361b2d26f59116ccdae986dc6d997b9389f52and 10,040 bytes, equal toadoption/skills/manifest.json.catalogs/sota-convergence/manifest-20260929.json(secrets-credentials, ci-supply-chain and quality-evaluation layers).tools/credentials/credential_run.py, documented indocs/secret-storage.md, "Using a key". Thetypesafeentry's statususer_only_paidis in the runner'sINJECTABLE_STATUSES. The inventory form follows the existingclaude-oauth-tokenentry, which records the runner form underloaders.Evidence-class table
python3 tools/sota-convergence/gap_crosswalk.py build --check(exit 0, "crosswalk up to date");tests.test_gap_crosswalk8/8screened_outand names its rule inmethod.routing. A gap with candidate pairs but no reviewed pair gets the gap statusscreened_out, a gap whose reviewed pair does not address it staysopen, and the page's Totals table listsscreened_out. A gap text with a digit sends every pair to review, and the build refuses such a pair if it has no reviewtests.test_gap_crosswalk_routing, CrosswalkRoutingTests (4 tests)open. Earlier tool versions fail the new tests8d300d71e(blob15895771): exit 1,FAILED (failures=1),'open' != 'screened_out'. Tool atecea28654(blob81fb44f2): exit 1,FAILED (failures=3, errors=1):'not_addressed' != 'screened_out';'open' != 'screened_out';(False, None) != (True, 'digit_in_gap'); no attributepair_entryscreened_out. 135 gaps have only unreviewed pairs, 73 of them without a digittests.test_gap_crosswalk_routing, RetainedLedgerRoutingTests, plus read-only counts overcatalogs/landscape/gap-crosswalk-92bb279.jsonresponse.cjs, and every copy names the schema and the three A4 fields. The schema also holdssource_refs,correction,limits,skill_pathandmodel, beyond the three fields section 3.1 names. Codex strict output requiresadditionalProperties: false, so a three-field schema would forbid the citations, corrections and limits that the role body already requirestests.test_semantic_reviewer_schema3/3; a mutated schema is the controltest_codex_roles38,test_codex_agents19,test_install_claude_profile79 (4 YAML tests skipped, also skipped on main),test-contract-mutations.mjs74/74,test-envelope.mjs254/254test_credential_status44,test_credential_run92;scripts/validate.py(runsinventory_errors)agent({schema})orcodex exec --output-schemacall with this schemaLocal commands run
These ran on HEAD
f4ad846a6withTMPDIR=/var/tmp/c5-jev-policy nice -n 19:None of the three failing modules, or any file they read, is changed by this branch.
test_adaptive_paper_metrics("exporter never registered its target") passes when run alone, so its failure in the full run was load timing. The other two fail because of this host. This host has a/tmp/.git, which the runner refuses as a temporary directory inside a repository. The installed Claude Code 2.1.288 binary knows the notification typeauth_storage_failure, which the table intests/test_windows_terminal_defaults.pylacks; CI skips that test because it has no installed client.Decision record
There is none. Section 3.1 does not call for one, and section 6.1 limits this PR to README text, the five reviewer copies and the crosswalk change. The dated rationale and the overturn condition are in the README section "Jev rules (2026-10-03)" and in the routing comment of
gap_crosswalk.py. That overturn condition is two preregistered, held-out, human-labelled replications (P1, then P1b) before the screen may set a final status.Gated items (not in this PR)
c5/jev-p1atb9f93db42, not yet on origin;blueprints/native-skill-practice/p1/draft/freeze-manifest.draft.json) defines the role at line 17, records"canary": []at line 117 and listscanaryundermissing_rolesat line 127. Sections 3.1 A1 and 5.0 make the canary a precondition of every run: run it first, and stop if it flips more than 2 of 10. Storing the answers takes Jev calls [inference], so this waits on the same user go as the P1 run.candidate:typesafe-ai-skillswinner'spinincatalogs/landscape/foundation.jsonmakespython3 tools/sota-convergence/build_verdicts.py --checkexit 1 with "foundation/instructions-skills names wave 20260922 but differs from its frozen entry in catalogs/sota-convergence/layer-verdicts-20260922.json (re-record it under a new --run-id instead)", and the check also flags the handbook block. The receipt recorder reads onlypin:host_receipts.winner_pinsreads it, and "unpinned" never binds. So a separate field would not unblock the recorder. The fix is a re-record of the instructions-skills row under a new run id, which is the layer-verdict lane's job. It toucheslayer-verdicts-<run-id>.json,layer-verdict-waves.json,docs/grand-catalog-handbook.mdand the row'slanes.sealed_base. The verified ref and tree above make the pin edit itself one step. The skill-preload re-probe on Claude Code 2.1.288, the new distribution and Codex stays a residual.scripts/component_matrix.py(lines 209 and 222) counts only gaps whose status isopen, so it would dropscreened_outgaps from its open counts.tools/sota-convergence/gap_wave_ledger.py(line 149) schedules everyexecutable_nowgap that is not settled, so it would schedule ascreened_outgap without first reviewing its pairs. Both must treatscreened_outas unknown before any crosswalk built under the new rule replaces the retained 92bb279 file. Both are outside this PR's paths, and both read only that file today, so nothing changes before then.screened_outtakes no model call. It needs a rule id without the digit route (not added here, because nothing would use it yet) and an owner for re-records outside this PR's paths:catalogs/landscape/gap-crosswalk-92bb279.json,docs/gap-crosswalk-92bb279.mdand the gap-wave ledgers that embedcrosswalk_sha256(gap_wave_ledger.py:161). 135 gaps would change fromopentoscreened_out, so item 3 comes first.screened_out.c5/jev-p1lands on main, addblueprints/native-skill-practice/p1/promptfooconfig.yamlto thetypesafeentry'senvironment_only_consumers. That file is the P1 harness and reads the key at lines 34, 55 and 76. Reworddocs/secret-storage.md:20and the entry's notes at the same time. It cannot go in earlier, becausescripts/credential_status.py:180-189rejects a consumer path that does not exist. The brief pointed at the root config's line 17, which belongs to the 2026-09-21 eight-case diagnostic, so this PR now names that file the native-skill-practice Jev provider rather than the P1 run.onlisting on this host. This belongs to the client-configuration lane.Decisions to veto
adoption/agents/codex/workers/SHA256SUMSis outside the brief's allowed paths. Its one changed row (semantic-evidence-reviewer.toml,ee924bbe…to0a406ec9…) follows from editing the pinned Codex worker copy:tools/adoption/codex_roles.pypins that file through this SHA256SUMS, andtest_codex_rolespasses 38/38 with the new row. Accept the path, or veto A4's Codex worker copy with it.d2fc3803e, not the brief'secea28654. See "Scope".screened_out, and not a count field. A reader that knows onlyopentherefore cannot mistake such a gap for a reviewed one. A gap that mixes reviewed pairs that do not address it with screened-out pairs staysopen, and its screened-out pairs show only per pair (56 such gaps in the retained ledger).adoption/agents/claude/semantic-evidence-reviewer.mdis registered; the.claude/agents/copy is not (see "Scope").Host evidence
Not applicable: this PR adds or changes nothing under
evidence/hosts/.Coordinator confirmations (2026-10-03)
instructions-skillsunder a new run id.build_verdicts.py --checkrightly refuses an edit to the frozen 2026-09-22 entry.Checklist
permissions: contents: read(or a narrower, explicitly justified addition). (none changed)typesafekey gains one documented consumer.)🤖 Generated with Claude Code