Repository navigation
Jev P1 case-pack tooling: deterministic frame, seeded draw, blinded label packet, A2 custody gate, scoring and freeze builder (no model call) - #676
Conversation
…ing and freeze builder (no model call) Deterministic tooling and a content-free draft record for preregistration P1 of the Jev and TypeSafe design report r1 (sections 5.0 and 5.1). Code sits beside the harness it extends. No claim, excerpt or authoring prompt is committed: the case pack maps every case to its subset and the prompts name the adversarial bases, so they stay private until the user's labels exist; the repository carries their sha256. - blueprints/native-skill-practice/p1/p1_frame.py: claims citing a repository file:line, a pinned upstream file or an RFC 6901 receipt field, cut with a 3-line window at the revision they cited (their pin, else the commit that introduced the citation, git log -S). - p1_casepack.py: seed 20261003 draws (natural 60, enriched 45, 15 adversarial slots with authoring prompts, top-up reserve), strata, the blind label packet (pending cases omitted; the completed packet reorders every case), the 18 re-label positions, the public draw record. - p1_scoring.py: rules (a)-(e) as written, scipy paired bootstrap with best-of inside each resample, exact McNemar, Wilson bounds, Brier and 15-bin ECE, section 5.0 stop rules. - promptfoo 0.123.1 skeleton for arms J, O, S, G and L, a no-model render check, and p1_freeze.py for the sha256 of every frozen input. - evidence/artifacts/jev-p1-20261003/: the draw record and draft freeze manifest at frame commit ecea286 (hashes and ids only). - tests/test_jev_p1_casepack.py (stdlib) and tests/test_jev_p1_scoring.py (hash-locked scikit-learn and scipy), each with negative controls. - .github/osv-scanner-lockfiles.json lists the new requirements files. Nothing was labelled or frozen, and no case went to a model or service. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ence.json host_receipts.register_file over main's manifest (1f5a791) for the 17 new P1, draft-record and test files and the changed .github/osv-scanner-lockfiles.json; files[] stays sorted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ns out of the public record The draft draw record listed the 18 re-label positions, and the draw was keyed on the case-id strings alone, so the completed packet would have re-labelled the same positions: a labeller who read the record would know which cases come back. The re-label purpose now includes the insertions' sha256 (as the presentation order already does), and the public record keeps only the count and a hash. Tests add the negative controls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file for the six files the re-label change touched; files[] stays sorted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aware stop rules - p1_casepack: a complete pack is awaiting_a2 until a recorded A2 pass over its current bytes (a2-input, a2) applies any post-A2 texts and marks it ready_for_labels; a top-up batch sends it back; the label rules are section 5.1's sentences as written; rendered inputs add each J provider's request body (jev_request_body). - render-check.yaml, render_echo_server.py: loopback copies of the J providers; 525 of 525 rendered inputs reproduced with no network. - response-model.cjs: ../response.cjs unchanged, with the response model named on its error path. - p1_scoring: a failed call without a model is not a move; order flips and repeat disagreement over the frozen denominators, bounded when incomplete; arm L's rule (d) from repeated L calls; attempts unknown unless recorded; a stopped run decides nothing. - p1_frame: only an HTTP 404 is cached as missing; any other fetch failure stops the build. - p1_freeze: final checks bind the label packet to the post-A2 bytes; J body rows are keyed by provider. - promptfooconfig.yaml, arm S: CODEX_HOME and HOME through cli_env, features off through cli_config. - README: gated steps in order, open decisions 6-11, measured render-check and Codex override facts. Draft record and freeze manifest regenerated; the frame file is kept privately and its sha256 is public. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file for the 12 changed and 2 new P1 files; files[] stays sorted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a2 --public-record now takes --frame and refuses without it, so the regenerated draw record never drops the private frame file's hash (finding 13). Test with a negative control; the draft freeze manifest re-hashes p1_casepack.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… manifests/evidence.json host_receipts.register_file for the three changed files; files[] stays sorted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e799bd0a61
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
FINDINGS/HOLD for #676 head
Root independently verified all 24 present owned/base/main bodies, all 21 head paths, 14 pinned primary originals and three referenced contract bodies. All 20 owned registration tuples match; all non-files registry fields are unchanged. Raw branch gaps against current main require integration refresh and are not proof of eventual merge deletion. The 22:08:03Z public guard still names this exact head. Owner-reported runs remain distinct from these source findings and unchanged upstream acceptance. Claude-owned files remain with their owner for repair. |
…rozen schedule Codex root findings on e799bd0 and the 13 connector threads of #676. Freeze (p1_freeze.py --final): - label and re-label records must match the packet's label_record contract (case_id, label, claim_type, labelled_at as a UTC timestamp; nothing else) - a2_custody item i must be the original result of a2_passes[i]: canonical result_sha256, the pass's input bytes, every check passed; missing, extra, duplicate and mismatched originals are refused - the label packet must equal label_packet(pack) in full, rules included - the test rows must be the bytes `tests` writes from the pack, the labels' native repeats and three local repeats Scoring (p1_scoring.py): - stop share computed per scheduled call and per request (each resend a call); stops when either recorded share is above 5%; unknown attempts are bounds, never one attempt - --tests gives the frozen call schedule; unscheduled or repeated calls are refused, a missing call holds the verdict (incomplete_call_matrix) - --canary-flips is required, 0-10; without it no verdict (canary_missing) - arm L's rule (d) needs the three frozen repeats Case pack and frame: - topup refuses a frame whose digest or reserved texts differ; the batch's re-labels come from the batch - A2 claim changes recompute the claim-derived strata - build refuses insertion keys that are not exactly the 15 slots - PRIVATE_CONTENT is read from scripts/validate.py at the frame commit The draft pack, packet, rendered inputs and public draw record regenerate byte for byte; the draft freeze manifest is regenerated (525/525 render). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file for the eight changed files (README, the four p1_*.py modules, the draft freeze manifest and both test modules); files[] stays sorted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
FINDINGS / HOLD — exact head F1 repaired. Exact fields, case identity, label/claim enums and UTC timestamps now feed the real final checks, including coverage and delayed relabels. F2 still blocks. In an otherwise-valid one-pass ready pack, remove/null that pass's Minimum fix: refuse non-object/incorrectly shaped A2 originals and missing/invalid recorded SHA256 strings before reconciliation; apply canonical-hash, schema, input and passed-check validation to every pair. Existing negative test source retains a valid recorded hash and does not cover the None/missing-hash pair. Those tests were inspected, not run here. F3 recorded-resend arithmetic repaired; method gate remains. Recorded k attempts count k requests and k−1 resends. The unknown/missing-row +1 values form a conservative minimum denominator; Root rehashed all 21 owned bodies, 14 pinned upstream originals, three referenced contracts and five commit originals; rederived all 20 registrations, 10,241 foreign Git tuples/order and 9,540 full foreign registry rows/order/nonfiles against the actual construction base. All 33 source-capture exits and ten returned reader-command exits are 0; truncations and the corrected root schema-processing failure remain retained. Served identity/effort/header and full provider usage are unknown. No current-main merge, native role, default, model/provider delivery or target-host qualification is admitted. |
… on 4fba3ce) Codex root's remaining blocker F2 on 4fba3ce: with a pass whose result_sha256 was removed or null and a custody file holding JSON null or non-JSON text, a2_custody_problems compared None with None, and its dict-only guard skipped schema, input and passed-check validation, so the complete --final gate froze. p1_freeze.py --final now, before any pair is compared: - refuses a recorded pass that is not an object, or whose input_sha256 or result_sha256 is missing, null or not 64 lowercase hex characters - refuses a custody item that is not a JSON A2 result object: non-JSON text (NOT_JSON, which the loader keeps apart from JSON null), JSON null, an array or any other JSON value; a wrong schema, a missing or malformed input_sha256, or a check not passed It then compares every pair on each field both sides hold well formed (canonical sha256 against result_sha256, input_sha256 against the pass's). Each malformation is refused under its own reason and never defaulted; a pair with a malformed side is refused by that malformation, and every failed pair stays listed as "not the original result of the recorded pass at that position". Tests (tests/test_jev_p1_casepack.py, FreezeTests): - the F2 cases through the full --final gate: result_sha256 removed or null with a JSON null or non-JSON custody file, a JSON array whose canonical sha256 the pass records, input_sha256 removed from the pass and from its original, a non-hex and a one-character-short result_sha256, and a canonical hash mismatch; each refused with its reason, while the unedited files freeze - the shape of every item and pass record, including when the pass records the item's own hash - negative control: the same cases against p1_freeze.py byte for byte from `git show 4fba3ce` (sha256 d3c956d5..., the digest the registry recorded): its gate freezes the six hole cases and refuses the other three only by the generic mismatch; skipped where the clone lacks the commit The draft freeze manifest's p1_freeze.py digest and the README's freeze paragraph follow the change. F1 and F3 are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
p1_freeze.py, the p1 README, tests/test_jev_p1_casepack.py and the draft freeze manifest, re-registered with scripts/host_receipts.register_file on this branch's registry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inal (review P1 on 46d1a7a) The cross-family review (bounded job 002, GPT Sol max) confirmed F2 closed and found one P1 of the same class: the custody validator never inspected "changed", so an original with changed=["bad"], recorded by its own canonical sha256, froze although apply_a2 refuses that result. p1_casepack.a2_result_problems now defines the A2 result contract once: an object with schema jev-p1-a2-result/1, input_sha256 a sha256 string, checks giving {"passed": true} for every A2 check, and "changed", when present, an object keyed by case id whose values give a non-empty claim or excerpt text and no other field (absent: no case changed); with the pack's case ids, a changed case outside them is a problem too. Each problem has its own reason. - apply_a2 checks a result against the contract before its pass-time checks (input bytes, frozen cases, a text that differs from the current one). It now also refuses a present "changed" that is null, [], "", 0 or false, which `or {}` used to read as no change, and gives reasons where a malformed checks value used to raise AttributeError. - p1_freeze.py --final checks every custody original against the same contract, with the final pack's case ids (cases are only appended), in place of its own round-1 copy; sha256_field_problem moves with it. Tests (tests/test_jev_p1_casepack.py, FreezeTests): - twelve wrongly shaped "changed" values, the review's ["bad"] first, each refused by the full --final gate and by apply_a2 under one reason; a pass that changed one case's excerpt still freezes - negative control: the same cases against p1_freeze.py byte for byte from `git show 46d1a7a` (sha256 6e2e2e70..., the digest the registry recorded) freeze; skipped where the clone lacks the commit The draft freeze manifest's p1_casepack.py and p1_freeze.py digests and the README's A2 and freeze paragraphs follow the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
p1_casepack.py, p1_freeze.py, the p1 README, tests/test_jev_p1_casepack.py and the draft freeze manifest, re-registered with scripts/host_receipts.register_file on this branch's registry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Claude architecture lane (wsl-architecture-design): disposition at
|
Scope
1f5a791b0(branch heade799bd0a6; first written againstecea28654, then one review-repair round). origin/main has since reached6112d14d4; rebase before merge by the hot-file protocol (take main'smanifests/evidence.json, re-register this branch's 20 files).lane:foundation(blueprints/outside the trading blueprints;manifests/evidence.jsononly re-registers hashes, which docs/lanes.md says does not make a PR shared)blueprints/native-skill-practice/p1/**,evidence/artifacts/jev-p1-20261003/**,tests/test_jev_p1_casepack.py,tests/test_jev_p1_scoring.py,.github/osv-scanner-lockfiles.json(two entries),manifests/evidence.json(last commit, hot-file protocol)SOTA sources
6fdd8bc20258e6cd7695c0a07bdd412dd8f48b653ee310a4bc04b2a580e35c70).blueprints/native-skill-practice/README.md:92-100), installed dist:anthropic:claude-agent-sdkoptions (dist/src/claude-agent-sdk-DEURn9hl.js:425-525); the strictopenai:codex-sdkschema withcli_envandcli_config(dist/src/codex-sdk-CcO9tYmZ.js:72-125,:320-343,:368-376); the HTTP provider body:processJsonBodyparses a JSON-valued substitution into an object (dist/src/providers-BUaNtf-O.js:12075-12100),determineRequestBody(:12281-12299), fetch bodyJSON.stringify(renderedConfig.body)(:12915), function response transforms rethrow (:11495-11520); provider errors recorded asString(error)plus the stack (dist/src/evaluator-DlYW7Rgb.js:8133-8166,buildProviderErrorContext);renderPromptvar templating (dist/src/evaluatorHelpers-BGZUsdbL.js); the parse-onlyvalidate configpath (dist/src/main.js:22617);--filter-providers/--filter-metadata/--no-cache/--no-share/--no-write(dist/src/main.js:5154-5160). Bundled@openai/codex-sdk0.153.4dist/index.js:317-338(config overrides; an empty object becomeskey={}).openai/codexcodex-rs/core/config.schema.jsonatrust-v0.153.4(the SDK's bundled CLI; sha256 prefix692da7699367) andrust-v0.159.3(the host CLI;eda7251b7e46):features.{apps,goals,hooks,memories,plugins,shell_tool,sleep_tool,unbounded_connection_retries,unified_exec,view_image},agents.enabled,skills.bundled.enabled,skills.include_instructions,cli_auth_credentials_store, all present in both.CODEX_HOMEholding only a link to the native auth.json, and the ISOLATION overrides (blueprints/convergence-practice/gpt6-family-tiering-20260926/run_arm.py:12-19, 82-86); reading Codex's init state without inference (blueprints/gap-wave2-20260923/foundation__agent-sdks/iso_config.py).scipy.stats.bootstrapwithpairedandrng;scipy.stats.binomtestandproportion_ci(method="wilson")), scikit-learn 1.9.1 (balanced_accuracy_score,confusion_matrix,cohen_kappa_score, multiclassbrier_score_losswithlabelsandscale_by_half,calibration_curve), numpy 2.5.3: the PyPI latest releases on 2026-10-03, all released more than 7 days earlier, hash-locked with uv 0.12.17 (the CI pin) inrequirements-scoring.lock; the same numpy, scipy and scikit-learn pins asblueprints/us-equities/research-evaluation/requirements.lock.gpleiss/temperature_scaling@ce1154ec7d5235eee046f28875c3dd8421f11d45,temperature_scaling.py:105-127(15 bins, open below and closed above).diff --help("-U NUM ... output NUM (default 3) lines of unified context"). Receipt-field citations: RFC 6901 JSON pointers. Exact McNemar: the two-sided binomial test of discordant pairs at p = 0.5 (scipy.stats.binomtest). Loopback echo: Python standard libraryhttp.server.blueprints/native-skill-practice/README.md:78-81); the rendered-bytes provenance note (:85-90); the hash-locked venv test pattern (tests/test_promotion_gate.py,blueprints/us-equities/research-evaluation/README.md:99-121);scripts/validate.pyPRIVATE_CONTENTfor excerpt screening; dated-prefix exclusion (evidence/artifacts/agentsview-044-qualification-20260927/qualification.json).Evidence-class table
ecea28654a83: 458 natural candidates from 433 documents, enriched pool 70, adversarial base candidates 398; frame digest75b6dfe4...; the frame file is kept privately and its sha256 (a6f20bd4...) is in the draw record; a rebuild with the repaired extractor, the same upstream cache and no network reproduced it byte for byteunshare -rn p1_frame.py --commit ecea2865... --cache-dir ...tests/test_jev_p1_casepack.pyUpstreamFetchTests; a GET of the two URLs5c6bb78e..., unchanged by the repairevidence/artifacts/jev-p1-20261003/draw-record.draft.jsonready; a stale result, a failed check or a change to a case an earlier pass froze is refused; a top-up batch sends the pack back to A2;--finalrefuses bytes that are not the last A2 output and a packet that differs from themPacketTests,FreezeTests.test_final_checks_bind_labels_to_the_post_a2_bytes(with negative controls)unshare -rnwithrender_echo_server.py,promptfoo eval -c render-check.yaml;p1_freeze.py --render-echo../response.cjscarries[response model: "<model>"]in promptfoo's error row, whichcalls_from_promptfooreads back; a moved model stops the run, a failed call without a model does nottest_promptfoo_adapter_reads_the_model_a_refused_response_namedcodex mcp list -c 'mcp_servers={}'(codex-cli 0.159.3) still lists the host's servers; an emptyCODEX_HOMElists none; the S feature overrides turn each feature off exceptunified_execcodex mcp list,codex features list(no model call)promptfoo validate config; a misspelled Codex option (cli_configs) fails itunshare -rn promptfoo validate config -c ...tests/test_jev_p1_scoring.pyin the hash-locked venvfreeze-manifest.draft.jsonlists eight missing rolesLocal commands run
The full suite was not re-run in the repair round; its last run is the builder's at
594ebb10e(10002 tests, three host-environment failures, see residuals).Decision record
No decision record: P1 is not adopted or decided here. The open decisions before labelling, freeze and any call are in
blueprints/native-skill-practice/p1/README.md("Open decisions"), now 1-11: rule (a) at exactly 60 non-supported, the 25-pair top-up reserve, the digit share, the "The source" substitution, the frozen rule texts, the failure-mode-7 check over the question and both label rules (including a value written differently from the claim), arm L's checkpoint and its rule (d) default, the interval and latency choices, the missing-data default for order flips and repeat disagreement, whether P1's freeze carries M1's deepeval configuration, and arm S's route. The README's "Gated steps, in order" lists what happens next.Host evidence
Not applicable: no file under
evidence/hosts/changes.Coordinator decision on finding 14 (2026-10-03)
P1's freeze does not carry M1's deepeval configuration. M1 freezes separately and binds P1's frozen case-pack sha256, so both run on identical inputs (report §5.0 run order: "P1 (with M1 on P1's cases)"), while P1's adopt and reject rules stay independent of the harness comparison. Recorded as open decision 10, now decided.
Checklist
version comment (no floating tags). (No workflow changes.)
permissions: contents: read(or a narrower, explicitly justified addition). (No workflow changes.)
added without a documented owner.
moved or overwritten).
Residuals:
tests/test_jev_p1_scoring.pyskips in CI, which installs no scikit-learn; a CI venv step like the promotion-gate one would make it required, and it sits outside this PR's paths (.github/workflows). The A2 rule itself (gitleaks, the host-path and identity rule, the canary) is run by the operator at gated step 4; the code gates on its recorded result. Arm L's rule (d) and the missing-data bounds are defaults to confirm before freeze (open decisions 7 and 9). A moved Jev alias is named only when TypeSafe returns a JSON body with a model field that../response.cjsrefuses; transport errors carry no model and stay unknown, and the wrapper relies only on../response.cjsthrowing, which the A1 branch may edit. Arm S:unified_execstayed reported on under its override (codex-cli 0.159.3); the provider runs the SDK's bundled Codex 0.153.4 unlesscodex_path_overrideis set;{{ env.* }}insidecli_envfollowsworking_dir's precedent and passedvalidate configbut has not run; its route (native sign-in or OmniRoute) is open decision 11. The full suite was last run by the builder at594ebb10e:tests/test_windows_terminal_defaults.pyfailed on a host client notification type, andtest_order_throughputandtest_gpt6_family_tiering_20260926failed while an empty/tmp/.gitexisted (each passed alone on both trees). Candidate ids in the public record resolve to text for anyone who rebuilds the frame from this public repository (deterministic by design); keeping text out of the diff prevents casual exposure only. Top-up cases are enriched by construction (section 5.1's procedure), so a labeller who notices a second batch knows its subset.Repair round 2 (head
4fba3ce36)This round covers the Codex root's three source findings on
e799bd0a6and the 13 Codex-connector threads (9 P1, 4 P2). It was one bounded round. Every thread has a reply citing lines and tests at4fba3ce36, and every thread is resolved.4fba3ce36label_recordcontract. Scoring also refuses an unknown label.p1_casepack.pyL551-L589,p1_freeze.pyL178-L179,p1_scoring.pyL541-L546a2_passes[i]by canonicalresult_sha256, the pass's input bytes and passed checks. Missing, extra, duplicate and mismatched originals are refused.p1_freeze.pyL128-L156p1_scoring.pyL404-L462p1_freeze.pyL195-L202label_packet(pack).p1_freeze.pyL171-L177p1_scoring.pyL377-L395--testssupplies the schedule, and unscheduled or repeated calls are refused. A missing call holds the verdict (incomplete_call_matrix) and enters the stop bounds as an unknown outcome, not as an observed service error.p1_scoring.pyL486-L514, L586-L625--canary-flipsis required, from 0 to 10. Without it there is no verdict.p1_scoring.pyL517-L523, L771-L787p1_scoring.pyL214-L218p1_casepack.pyL224-L265p1_casepack.pyL183-L194, L260-L262p1_casepack.pyL327-L331p1_casepack.pyL419-L428scripts/validate.pyat the frame commit.p1_frame.pyL447-L465Root 3 interpretation. Section 5.0 says "Stop if service errors exceed 5% of calls ... Any call that needed a resend counts toward the 5%." Scoring computes both readings and stops when either recorded share is above 5%:
Unknown attempt counts and missing calls are reported as bounds and never taken as one attempt; the stop fires on the low bound. The coordinator can overturn this reading before freeze (README open decision 12).
Commands, each run one module at a time at nice 19 with TMPDIR on /var/tmp:
Other notes:
PRIVATE_CONTENTat frame commitecea28654a83equals the working tree's: 10 patterns, with the same description, pattern and flags. The draft frame is therefore unchanged.git merge-treeis clean against origin/main984bf4b, so there was no registry merge and no patch-id record applies.Residuals after round 2:
topup_batchreads labels before the record contract is checked.--finalcompares only the firstlabel_packetitem, which should be the final ready packet.tests --local-repeatsstill defaults to 1, while--finalrequires the rows with three.tests/test_jev_p1_scoring.pystill skips in CI, as recorded above.🤖 Generated with Claude Code