Skip to content

Jev P1 case-pack tooling: deterministic frame, seeded draw, blinded label packet, A2 custody gate, scoring and freeze builder (no model call) - #676

Merged
seathatflowsinourveins merged 14 commits into
mainfrom
c5/jev-p1
Oct 4, 2026
Merged

seathatflowsinourveins merged 14 commits into
mainfrom
c5/jev-p1

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes, in one or two sentences: P1's deterministic case-pack tooling and a content-free draft record for the Jev and TypeSafe design report r1 (sections 5.0 and 5.1): frame extractor, seeded draws (seed 20261003), strata, the blind label packet behind an A2 custody gate, adversarial slots with authoring prompts, scoring with scikit-learn and scipy, a promptfoo 0.123.1 skeleton for arms J, O, S, G and L with a no-network render check that covers arm J's real request body, a freeze-manifest builder, and the gated steps in order. No case text is committed (it stays private until the user's labels exist, named here by sha256); nothing was labelled or frozen, and no case went to a model or a remote service.
  • Base commit: 1f5a791b0 (branch head e799bd0a6; first written against ecea28654, then one review-repair round). origin/main has since reached 6112d14d4; rebase before merge by the hot-file protocol (take main's manifests/evidence.json, re-register this branch's 20 files).
  • Lane: lane:foundation (blueprints/ outside the trading blueprints; manifests/evidence.json only re-registers hashes, which docs/lanes.md says does not make a PR shared)
  • Owned paths touched: blueprints/native-skill-practice/p1/**, evidence/artifacts/jev-p1-20261003/**, tests/test_jev_p1_casepack.py, tests/test_jev_p1_scoring.py, .github/osv-scanner-lockfiles.json (two entries), manifests/evidence.json (last commit, hot-file protocol)

SOTA sources

  • Specification: Jev and TypeSafe design report r1, sections 3.1 (A2), 5.0, 5.1 and 6.1 (host-private coordination record, sha256 6fdd8bc20258e6cd7695c0a07bdd412dd8f48b653ee310a4bc04b2a580e35c70).
  • promptfoo 0.123.1 (the selected harness, blueprints/native-skill-practice/README.md:92-100), installed dist: anthropic:claude-agent-sdk options (dist/src/claude-agent-sdk-DEURn9hl.js:425-525); the strict openai:codex-sdk schema with cli_env and cli_config (dist/src/codex-sdk-CcO9tYmZ.js:72-125, :320-343, :368-376); the HTTP provider body: processJsonBody parses a JSON-valued substitution into an object (dist/src/providers-BUaNtf-O.js:12075-12100), determineRequestBody (:12281-12299), fetch body JSON.stringify(renderedConfig.body) (:12915), function response transforms rethrow (:11495-11520); provider errors recorded as String(error) plus the stack (dist/src/evaluator-DlYW7Rgb.js:8133-8166, buildProviderErrorContext); renderPrompt var templating (dist/src/evaluatorHelpers-BGZUsdbL.js); the parse-only validate config path (dist/src/main.js:22617); --filter-providers/--filter-metadata/--no-cache/--no-share/--no-write (dist/src/main.js:5154-5160). Bundled @openai/codex-sdk 0.153.4 dist/index.js:317-338 (config overrides; an empty object becomes key={}).
  • Codex configuration schema, openai/codex codex-rs/core/config.schema.json at rust-v0.153.4 (the SDK's bundled CLI; sha256 prefix 692da7699367) and rust-v0.159.3 (the host CLI; eda7251b7e46): features.{apps,goals,hooks,memories,plugins,shell_tool,sleep_tool,unbounded_connection_retries,unified_exec,view_image}, agents.enabled, skills.bundled.enabled, skills.include_instructions, cli_auth_credentials_store, all present in both.
  • Repository recipes followed: per-call CODEX_HOME holding only a link to the native auth.json, and the ISOLATION overrides (blueprints/convergence-practice/gpt6-family-tiering-20260926/run_arm.py:12-19, 82-86); reading Codex's init state without inference (blueprints/gap-wave2-20260923/foundation__agent-sdks/iso_config.py).
  • scipy 1.18.1 (scipy.stats.bootstrap with paired and rng; scipy.stats.binomtest and proportion_ci(method="wilson")), scikit-learn 1.9.1 (balanced_accuracy_score, confusion_matrix, cohen_kappa_score, multiclass brier_score_loss with labels and scale_by_half, calibration_curve), numpy 2.5.3: the PyPI latest releases on 2026-10-03, all released more than 7 days earlier, hash-locked with uv 0.12.17 (the CI pin) in requirements-scoring.lock; the same numpy, scipy and scikit-learn pins as blueprints/us-equities/research-evaluation/requirements.lock.
  • ECE: Guo, Pleiss, Sun and Weinberger, "On Calibration of Modern Neural Networks", ICML 2017, arXiv:1706.04599, eq. 3; reference code gpleiss/temperature_scaling@ce1154ec7d5235eee046f28875c3dd8421f11d45, temperature_scaling.py:105-127 (15 bins, open below and closed above).
  • Excerpt window: GNU diffutils 3.10 diff --help ("-U NUM ... output NUM (default 3) lines of unified context"). Receipt-field citations: RFC 6901 JSON pointers. Exact McNemar: the two-sided binomial test of discordant pairs at p = 0.5 (scipy.stats.binomtest). Loopback echo: Python standard library http.server.
  • Repository precedents: cases built from retained excerpts and labelled before inference (blueprints/native-skill-practice/README.md:78-81); the rendered-bytes provenance note (:85-90); the hash-locked venv test pattern (tests/test_promotion_gate.py, blueprints/us-equities/research-evaluation/README.md:99-121); scripts/validate.py PRIVATE_CONTENT for excerpt screening; dated-prefix exclusion (evidence/artifacts/agentsview-044-qualification-20260927/qualification.json).

Evidence-class table

Claim Evidence class Command / receipt
Frame at ecea28654a83: 458 natural candidates from 433 documents, enriched pool 70, adversarial base candidates 398; frame digest 75b6dfe4...; the frame file is kept privately and its sha256 (a6f20bd4...) is in the draw record; a rebuild with the repaired extractor, the same upstream cache and no network reproduced it byte for byte local_integration unshare -rn p1_frame.py --commit ecea2865... --cache-dir ...
Only an HTTP 404 is cached as missing; a 429, a 503, a URL error or a timeout stops the build; both cached 404s returned 404 again at 2026-10-03T21:26:31Z synthetic + local_integration tests/test_jev_p1_casepack.py UpstreamFetchTests; a GET of the two URLs
Draw: natural 60, enriched 45, 15 adversarial slots (5 per form; Opus 8, Sol 7), top-up reserve 25, 18 re-labels (count and hash only); drawn-ids sha256 5c6bb78e..., unchanged by the repair local_integration evidence/artifacts/jev-p1-20261003/draw-record.draft.json
A complete pack waits for a recorded A2 pass over its current bytes before its packet is ready; a stale result, a failed check or a change to a case an earlier pass froze is refused; a top-up batch sends the pack back to A2; --final refuses bytes that are not the last A2 output and a packet that differs from them synthetic PacketTests, FreezeTests.test_final_checks_bind_labels_to_the_post_a2_bytes (with negative controls)
promptfoo 0.123.1 reproduces all 525 rendered inputs of the 105 complete draft cases byte for byte: 210 prompts from the echo provider and 315 J request bodies (compact JSON, state as an object) from loopback copies of the J providers local_integration unshare -rn with render_echo_server.py, promptfoo eval -c render-check.yaml; p1_freeze.py --render-echo
A J response refused by ../response.cjs carries [response model: "<model>"] in promptfoo's error row, which calls_from_promptfoo reads back; a moved model stops the run, a failed call without a model does not local_integration + synthetic loopback probe with a synthetic row (no case text); test_promptfoo_adapter_reads_the_model_a_refused_response_named
codex mcp list -c 'mcp_servers={}' (codex-cli 0.159.3) still lists the host's servers; an empty CODEX_HOME lists none; the S feature overrides turn each feature off except unified_exec local_integration codex mcp list, codex features list (no model call)
Both configs pass promptfoo validate config; a misspelled Codex option (cli_configs) fails it local_integration unshare -rn promptfoo validate config -c ...
Rules (a)-(e), reject and inconclusive, frozen denominators with bounds, arm L's rule (d) from repeated calls, the stop rules, a stopped run's verdict, bootstrap, McNemar, Wilson, Brier and ECE behave as specified synthetic tests/test_jev_p1_scoring.py in the hash-locked venv
No arm, model or remote service call; nothing labelled or frozen source_review the diff; freeze-manifest.draft.json lists eight missing roles

Local commands run

$ python3 scripts/validate.py
{"components": 69, "hashed_files": 9560, "profiles": 4, "receipts": 193, "status": "passed"}
exit 0
$ python3 -m unittest tests.test_jev_p1_casepack
Ran 40 tests ... OK  (exit 0)
$ JEV_P1_REQUIRE_SCORING_DEPS=1 <venv>/bin/python -m unittest tests.test_jev_p1_scoring
Ran 26 tests ... OK  (exit 0)
$ python3 -m unittest tests.test_jev_p1_scoring        # system python, no scikit-learn
OK (skipped=26)  (exit 0; untested there, see residuals)
$ JEV_P1_REQUIRE_SCORING_DEPS=1 python3 -m unittest tests.test_jev_p1_scoring
FAILED (errors=1)  (exit 1; the guard refuses a silent skip)
$ python3 -m unittest tests.test_agentsview_qualification tests.test_catalog_freshness_propose tests.test_osv_lockfile_coverage tests.test_validate
Ran 208 tests ... OK  (exit 0)
$ python3 -m unittest tests.test_evidence_manifest tests.test_gitleaks_config tests.test_host_receipts tests.test_secret_path_guard tests.test_claude_repository_evidence
Ran 318 tests ... OK (skipped=2)  (exit 0)
$ unshare -rn sh -c 'ip link set lo up; python3 render_echo_server.py & ...; promptfoo eval -c render-check.yaml --filter-metadata arm_group=render --no-cache --no-share --no-write --no-table --output <private>/render.json'
exit 0; 0 failed, 0 errors; p1_freeze.py --render-echo: expected 525, matched 525, passed
$ unshare -rn promptfoo validate config -c promptfooconfig.yaml ; ... -c render-check.yaml ; ... -c <copy with cli_configs>
exit 0 ; exit 0 ; exit 1 (Unrecognized key: "cli_configs")
$ unshare -rn python3 p1_frame.py --commit ecea28654a835fff2cc3651bab77ca0e46b9bec5 --cache-dir <cache> --out <private>
exit 0; sha256 a6f20bd46ab57fde... equal to the retained frame file

The full suite was not re-run in the repair round; its last run is the builder's at 594ebb10e (10002 tests, three host-environment failures, see residuals).

Decision record

No decision record: P1 is not adopted or decided here. The open decisions before labelling, freeze and any call are in blueprints/native-skill-practice/p1/README.md ("Open decisions"), now 1-11: rule (a) at exactly 60 non-supported, the 25-pair top-up reserve, the digit share, the "The source" substitution, the frozen rule texts, the failure-mode-7 check over the question and both label rules (including a value written differently from the claim), arm L's checkpoint and its rule (d) default, the interval and latency choices, the missing-data default for order flips and repeat disagreement, whether P1's freeze carries M1's deepeval configuration, and arm S's route. The README's "Gated steps, in order" lists what happens next.

Host evidence

Not applicable: no file under evidence/hosts/ changes.

Coordinator decision on finding 14 (2026-10-03)

P1's freeze does not carry M1's deepeval configuration. M1 freezes separately and binds P1's frozen case-pack sha256, so both run on identical inputs (report §5.0 run order: "P1 (with M1 on P1's cases)"), while P1's adopt and reject rules stay independent of the harness comparison. Recorded as open decision 10, now decided.

Checklist

  • New/changed GitHub Actions are pinned to a full commit SHA with a
    version comment (no floating tags). (No workflow changes.)
  • New/changed workflows declare top-level permissions: contents: read
    (or a narrower, explicitly justified addition). (No workflow changes.)
  • No secrets are printed, logged or committed; no new required secret was
    added without a documented owner.
  • No new paid hosting, subscription or billing surface was introduced.
  • Peer-owned untracked files and worktrees were preserved (not deleted,
    moved or overwritten).

Residuals: tests/test_jev_p1_scoring.py skips in CI, which installs no scikit-learn; a CI venv step like the promotion-gate one would make it required, and it sits outside this PR's paths (.github/workflows). The A2 rule itself (gitleaks, the host-path and identity rule, the canary) is run by the operator at gated step 4; the code gates on its recorded result. Arm L's rule (d) and the missing-data bounds are defaults to confirm before freeze (open decisions 7 and 9). A moved Jev alias is named only when TypeSafe returns a JSON body with a model field that ../response.cjs refuses; transport errors carry no model and stay unknown, and the wrapper relies only on ../response.cjs throwing, which the A1 branch may edit. Arm S: unified_exec stayed reported on under its override (codex-cli 0.159.3); the provider runs the SDK's bundled Codex 0.153.4 unless codex_path_override is set; {{ env.* }} inside cli_env follows working_dir's precedent and passed validate config but has not run; its route (native sign-in or OmniRoute) is open decision 11. The full suite was last run by the builder at 594ebb10e: tests/test_windows_terminal_defaults.py failed on a host client notification type, and test_order_throughput and test_gpt6_family_tiering_20260926 failed while an empty /tmp/.git existed (each passed alone on both trees). Candidate ids in the public record resolve to text for anyone who rebuilds the frame from this public repository (deterministic by design); keeping text out of the diff prevents casual exposure only. Top-up cases are enriched by construction (section 5.1's procedure), so a labeller who notices a second batch knows its subset.

Repair round 2 (head 4fba3ce36)

This round covers the Codex root's three source findings on e799bd0a6 and the 13 Codex-connector threads (9 P1, 4 P2). It was one bounded round. Every thread has a reply citing lines and tests at 4fba3ce36, and every thread is resolved.

Item Disposition Where at 4fba3ce36
Root 1 and thread k5: label records Fixed. Every label and re-label record is checked against the packet's label_record contract. Scoring also refuses an unknown label. p1_casepack.py L551-L589, p1_freeze.py L178-L179, p1_scoring.py L541-L546
Root 2 and thread k-: A2 custody Fixed. Custody item i is bound to a2_passes[i] by canonical result_sha256, the pass's input bytes and passed checks. Missing, extra, duplicate and mismatched originals are refused. p1_freeze.py L128-L156
Root 3: stop share and resends Fixed as a stated interpretation; see below. p1_scoring.py L404-L462
k1: test rows Fixed. The rows file is compared byte for byte with the rows generated from the pack, the labels' native repeats and three local repeats. p1_freeze.py L195-L202
lL: label packet Fixed. The whole packet is compared with label_packet(pack). p1_freeze.py L171-L177
k7: arm L repeats Fixed. Rule (d) needs the three frozen L repeats. p1_scoring.py L377-L395
lC: call matrix Fixed. --tests supplies the schedule, and unscheduled or repeated calls are refused. A missing call holds the verdict (incomplete_call_matrix) and enters the stop bounds as an unknown outcome, not as an observed service error. p1_scoring.py L486-L514, L586-L625
lE: canary Fixed. --canary-flips is required, from 0 to 10. Without it there is no verdict. p1_scoring.py L517-L523, L771-L787
lH: J-3 latency Rejected. Section 5.1 defines the latency as "the wall time of three concurrent calls" (README open decision 8). Only the comment changed. p1_scoring.py L214-L218
lA: top-up frame Fixed. The frame digest and the reserved text hashes are checked before a top-up. p1_casepack.py L224-L265
lJ: top-up re-labels Fixed. The batch's re-labels are drawn from the batch. p1_casepack.py L183-L194, L260-L262
lP: strata after A2 Fixed. Claim-derived strata are recomputed. p1_casepack.py L327-L331
lR: insertion keys Fixed. Insertion keys must be exactly the 15 slot ids. p1_casepack.py L419-L428
lT: private-content rules Fixed. The rules are read from scripts/validate.py at the frame commit. p1_frame.py L447-L465

Root 3 interpretation. Section 5.0 says "Stop if service errors exceed 5% of calls ... Any call that needed a resend counts toward the 5%." Scoring computes both readings and stops when either recorded share is above 5%:

  • Per scheduled call, which is section 5.1's unit: a call that failed or needed a resend counts once.
  • Per request, where every resend is itself a call: k recorded attempts are k requests, and k-1 of them were resent.

Unknown attempt counts and missing calls are reported as bounds and never taken as one attempt; the stop fires on the low bound. The coordinator can overturn this reading before freeze (README open decision 12).

Commands, each run one module at a time at nice 19 with TMPDIR on /var/tmp:

$ python3 -m unittest tests.test_jev_p1_casepack
Ran 52 tests ... OK  (exit 0)
$ JEV_P1_REQUIRE_SCORING_DEPS=1 <venv>/bin/python -m unittest -v tests.test_jev_p1_scoring
Ran 32 tests ... OK, 0 skipped  (exit 0)
$ python3 scripts/evidence_manifest.py --check
{"files": 9560, "status": "passed"}  (exit 0)
$ python3 scripts/validate.py
{"components": 69, "hashed_files": 9560, "profiles": 4, "receipts": 193, "status": "passed"}  (exit 0)
$ python3 .../p1_casepack.py build --frame <private>/frame.json --out-dir <scratch> --public-record <scratch>/record.json
$ python3 .../p1_casepack.py tests --pack <scratch>/case-pack.draft.json --rows <scratch>/rows.jsonl --rendered <scratch>/rendered.json
the case pack, label packet and rendered inputs equal the private draft files;
record.json equals draw-record.draft.json byte for byte  (both exit 0)
$ python3 .../p1_freeze.py --spec <private draft spec> --render-echo <retained render output> --out .../freeze-manifest.draft.json
render check 525 of 525 matched; external hashes unchanged; only the four code hashes moved  (exit 0)

Other notes:

  • PRIVATE_CONTENT at frame commit ecea28654a83 equals the working tree's: 10 patterns, with the same description, pattern and flags. The draft frame is therefore unchanged.
  • No merge was needed. git merge-tree is clean against origin/main 984bf4b, so there was no registry merge and no patch-id record applies.
  • The full suite was not run, because session 0c's CPU measurement window was open.

Residuals after round 2:

  • Resend evidence: no arm records attempt counts today, so the resend part of the stop rule is checked only where an attempt count is recorded (README open decision 12). Whether such a run may issue a verdict is a decision before freeze.
  • The freeze does not re-check the number of top-up cases against the labels' deficit rule, and topup_batch reads labels before the record contract is checked.
  • Arm D has no frozen schedule, so D calls are refused as unscheduled until a D row group exists.
  • --final compares only the first label_packet item, which should be the final ready packet.
  • tests --local-repeats still defaults to 1, while --final requires the rows with three.
  • tests/test_jev_p1_scoring.py still skips in CI, as recorded above.

🤖 Generated with Claude Code

Scout and others added 8 commits October 3, 2026 15:43
…ing and freeze builder (no model call)

Deterministic tooling and a content-free draft record for preregistration P1 of the Jev and
TypeSafe design report r1 (sections 5.0 and 5.1). Code sits beside the harness it extends.
No claim, excerpt or authoring prompt is committed: the case pack maps every case to its subset
and the prompts name the adversarial bases, so they stay private until the user's labels exist;
the repository carries their sha256.

- blueprints/native-skill-practice/p1/p1_frame.py: claims citing a repository file:line, a
  pinned upstream file or an RFC 6901 receipt field, cut with a 3-line window at the revision
  they cited (their pin, else the commit that introduced the citation, git log -S).
- p1_casepack.py: seed 20261003 draws (natural 60, enriched 45, 15 adversarial slots with
  authoring prompts, top-up reserve), strata, the blind label packet (pending cases omitted;
  the completed packet reorders every case), the 18 re-label positions, the public draw record.
- p1_scoring.py: rules (a)-(e) as written, scipy paired bootstrap with best-of inside each
  resample, exact McNemar, Wilson bounds, Brier and 15-bin ECE, section 5.0 stop rules.
- promptfoo 0.123.1 skeleton for arms J, O, S, G and L, a no-model render check, and
  p1_freeze.py for the sha256 of every frozen input.
- evidence/artifacts/jev-p1-20261003/: the draw record and draft freeze manifest at frame
  commit ecea286 (hashes and ids only).
- tests/test_jev_p1_casepack.py (stdlib) and tests/test_jev_p1_scoring.py (hash-locked
  scikit-learn and scipy), each with negative controls.
- .github/osv-scanner-lockfiles.json lists the new requirements files.

Nothing was labelled or frozen, and no case went to a model or service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ence.json

host_receipts.register_file over main's manifest (1f5a791) for the 17 new P1, draft-record and
test files and the changed .github/osv-scanner-lockfiles.json; files[] stays sorted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ns out of the public record

The draft draw record listed the 18 re-label positions, and the draw was keyed on the case-id
strings alone, so the completed packet would have re-labelled the same positions: a labeller who
read the record would know which cases come back. The re-label purpose now includes the
insertions' sha256 (as the presentation order already does), and the public record keeps only the
count and a hash. Tests add the negative controls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file for the six files the re-label change touched; files[] stays sorted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aware stop rules

- p1_casepack: a complete pack is awaiting_a2 until a recorded A2 pass over its
  current bytes (a2-input, a2) applies any post-A2 texts and marks it
  ready_for_labels; a top-up batch sends it back; the label rules are section
  5.1's sentences as written; rendered inputs add each J provider's request
  body (jev_request_body).
- render-check.yaml, render_echo_server.py: loopback copies of the J providers;
  525 of 525 rendered inputs reproduced with no network.
- response-model.cjs: ../response.cjs unchanged, with the response model named
  on its error path.
- p1_scoring: a failed call without a model is not a move; order flips and
  repeat disagreement over the frozen denominators, bounded when incomplete;
  arm L's rule (d) from repeated L calls; attempts unknown unless recorded; a
  stopped run decides nothing.
- p1_frame: only an HTTP 404 is cached as missing; any other fetch failure
  stops the build.
- p1_freeze: final checks bind the label packet to the post-A2 bytes; J body
  rows are keyed by provider.
- promptfooconfig.yaml, arm S: CODEX_HOME and HOME through cli_env, features
  off through cli_config.
- README: gated steps in order, open decisions 6-11, measured render-check and
  Codex override facts. Draft record and freeze manifest regenerated; the frame
  file is kept privately and its sha256 is public.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file for the 12 changed and 2 new P1 files; files[] stays sorted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a2 --public-record now takes --frame and refuses without it, so the regenerated
draw record never drops the private frame file's hash (finding 13). Test with a
negative control; the draft freeze manifest re-hashes p1_casepack.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… manifests/evidence.json

host_receipts.register_file for the three changed files; files[] stays sorted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 3, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-03T21:48:27.595185Z e799bd0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e799bd0a61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread blueprints/native-skill-practice/p1/p1_freeze.py Outdated
Comment thread blueprints/native-skill-practice/p1/p1_freeze.py
Comment thread blueprints/native-skill-practice/p1/p1_scoring.py Outdated
Comment thread blueprints/native-skill-practice/p1/p1_freeze.py
Comment thread blueprints/native-skill-practice/p1/p1_casepack.py Outdated
Comment thread blueprints/native-skill-practice/p1/p1_casepack.py Outdated
Comment thread blueprints/native-skill-practice/p1/p1_freeze.py Outdated
Comment thread blueprints/native-skill-practice/p1/p1_casepack.py
Comment thread blueprints/native-skill-practice/p1/p1_casepack.py
Comment thread blueprints/native-skill-practice/p1/p1_frame.py Outdated
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

FINDINGS/HOLD for #676 head e799bd0a617263b2d12a47731676d08ff240e339. These are source-contract findings; no freeze, fixture, test or model call was executed by root.

  1. Final freeze can accept unlabelled or invalid label records. The guard checks case-key coverage and selected timestamps, but does not require a label field or validate its value. The declared required role includes label and claim type, while scoring later dereferences every label. With otherwise valid pack/render inputs, records containing timestamps but no labels can satisfy these freeze predicates and then fail scoring. Validate both labels and relabels against the declared record contract before returning/writing a frozen manifest.
  2. A2 custody files are hashed without binding them to the recorded passes. Each pass records a canonical result_sha256, but the final builder never reconciles the supplied a2_custody originals with those canonical hashes. An unrelated nonempty custody file can be hashed as that role. Require exact coverage of the recorded A2 results in their declared canonical hashing domain, with missing/extra/duplicate or mismatched originals refused.
  3. The implemented stop calculation does not count resends as calls. Section 5.0's source contract says resends count, yet its denominator is len(calls) and any result row with multiple attempts contributes only one failed-or-resent row. Reconcile this implementation with the frozen call/attempt rule and original attempt evidence; keep unknown outcomes unknown rather than inferring every retry outcome from the aggregate attempt count.

Root independently verified all 24 present owned/base/main bodies, all 21 head paths, 14 pinned primary originals and three referenced contract bodies. All 20 owned registration tuples match; all non-files registry fields are unchanged. Raw branch gaps against current main require integration refresh and are not proof of eventual merge deletion. The 22:08:03Z public guard still names this exact head. Owner-reported runs remain distinct from these source findings and unchanged upstream acceptance. Claude-owned files remain with their owner for repair.

Scout and others added 2 commits October 3, 2026 18:49
…rozen schedule

Codex root findings on e799bd0 and the 13 connector threads of #676.

Freeze (p1_freeze.py --final):
- label and re-label records must match the packet's label_record contract
  (case_id, label, claim_type, labelled_at as a UTC timestamp; nothing else)
- a2_custody item i must be the original result of a2_passes[i]: canonical
  result_sha256, the pass's input bytes, every check passed; missing, extra,
  duplicate and mismatched originals are refused
- the label packet must equal label_packet(pack) in full, rules included
- the test rows must be the bytes `tests` writes from the pack, the labels'
  native repeats and three local repeats

Scoring (p1_scoring.py):
- stop share computed per scheduled call and per request (each resend a
  call); stops when either recorded share is above 5%; unknown attempts are
  bounds, never one attempt
- --tests gives the frozen call schedule; unscheduled or repeated calls are
  refused, a missing call holds the verdict (incomplete_call_matrix)
- --canary-flips is required, 0-10; without it no verdict (canary_missing)
- arm L's rule (d) needs the three frozen repeats

Case pack and frame:
- topup refuses a frame whose digest or reserved texts differ; the batch's
  re-labels come from the batch
- A2 claim changes recompute the claim-derived strata
- build refuses insertion keys that are not exactly the 15 slots
- PRIVATE_CONTENT is read from scripts/validate.py at the frame commit

The draft pack, packet, rendered inputs and public draw record regenerate
byte for byte; the draft freeze manifest is regenerated (525/525 render).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host_receipts.register_file for the eight changed files (README, the four
p1_*.py modules, the draft freeze manifest and both test modules); files[]
stays sorted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

FINDINGS / HOLD — exact head 4fba3ce362a67a145103d239246f6582ba5e679a. Reviewed old e799bd0a617263b2d12a47731676d08ff240e339; actual construction base 1f5a791b02a230aced670c88bab3d3d0ebcf401a. The remaining F2 failure survived the bounded Sol repair; one requested Astra/max consequential source read through the existing pool returned this verdict, native exit 0. No fixture or malformed actual freeze was executed or observed.

F1 repaired. Exact fields, case identity, label/claim enums and UTC timestamps now feed the real final checks, including coverage and delayed relabels.

F2 still blocks. In an otherwise-valid one-pass ready pack, remove/null that pass's result_sha256; provide one custody file containing JSON null or non-JSON text. The predicate gets given=[None], recorded=[None]: counts and equality pass, duplicate detection excludes None, and the dictionary-only guard skips schema/input/passed-check validation. The loader and final construction introduce no separate guard. Case hashing excludes that pass metadata; valid labels, relabels, label packet, rendered inputs, test rows, role files and last A2 hashes can stay unchanged. Thus the complete final gate can accept this conceptual construction.

Minimum fix: refuse non-object/incorrectly shaped A2 originals and missing/invalid recorded SHA256 strings before reconciliation; apply canonical-hash, schema, input and passed-check validation to every pair. Existing negative test source retains a valid recorded hash and does not cover the None/missing-hash pair. Those tests were inspected, not run here.

F3 recorded-resend arithmetic repaired; method gate remains. Recorded k attempts count k requests and k−1 resends. The unknown/missing-row +1 values form a conservative minimum denominator; requests_known is not authenticated traffic, and missing rows prove no request was sent. Preserve unknown usage/bounds. Decision 12 remains required before freeze; scoring does not independently HOLD for unknown attempts. This review chooses no new experimental rule.

Root rehashed all 21 owned bodies, 14 pinned upstream originals, three referenced contracts and five commit originals; rederived all 20 registrations, 10,241 foreign Git tuples/order and 9,540 full foreign registry rows/order/nonfiles against the actual construction base. All 33 source-capture exits and ten returned reader-command exits are 0; truncations and the corrected root schema-processing failure remain retained. Served identity/effort/header and full provider usage are unknown. No current-main merge, native role, default, model/provider delivery or target-host qualification is admitted.

Scout and others added 4 commits October 3, 2026 21:01
… on 4fba3ce)

Codex root's remaining blocker F2 on 4fba3ce: with a pass whose
result_sha256 was removed or null and a custody file holding JSON null or
non-JSON text, a2_custody_problems compared None with None, and its
dict-only guard skipped schema, input and passed-check validation, so the
complete --final gate froze.

p1_freeze.py --final now, before any pair is compared:
- refuses a recorded pass that is not an object, or whose input_sha256 or
  result_sha256 is missing, null or not 64 lowercase hex characters
- refuses a custody item that is not a JSON A2 result object: non-JSON
  text (NOT_JSON, which the loader keeps apart from JSON null), JSON null,
  an array or any other JSON value; a wrong schema, a missing or malformed
  input_sha256, or a check not passed
It then compares every pair on each field both sides hold well formed
(canonical sha256 against result_sha256, input_sha256 against the pass's).
Each malformation is refused under its own reason and never defaulted; a
pair with a malformed side is refused by that malformation, and every
failed pair stays listed as "not the original result of the recorded pass
at that position".

Tests (tests/test_jev_p1_casepack.py, FreezeTests):
- the F2 cases through the full --final gate: result_sha256 removed or
  null with a JSON null or non-JSON custody file, a JSON array whose
  canonical sha256 the pass records, input_sha256 removed from the pass
  and from its original, a non-hex and a one-character-short
  result_sha256, and a canonical hash mismatch; each refused with its
  reason, while the unedited files freeze
- the shape of every item and pass record, including when the pass
  records the item's own hash
- negative control: the same cases against p1_freeze.py byte for byte
  from `git show 4fba3ce` (sha256 d3c956d5..., the digest the registry
  recorded): its gate freezes the six hole cases and refuses the other
  three only by the generic mismatch; skipped where the clone lacks the
  commit

The draft freeze manifest's p1_freeze.py digest and the README's freeze
paragraph follow the change. F1 and F3 are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
p1_freeze.py, the p1 README, tests/test_jev_p1_casepack.py and the draft
freeze manifest, re-registered with scripts/host_receipts.register_file on
this branch's registry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inal (review P1 on 46d1a7a)

The cross-family review (bounded job 002, GPT Sol max) confirmed F2
closed and found one P1 of the same class: the custody validator never
inspected "changed", so an original with changed=["bad"], recorded by its
own canonical sha256, froze although apply_a2 refuses that result.

p1_casepack.a2_result_problems now defines the A2 result contract once:
an object with schema jev-p1-a2-result/1, input_sha256 a sha256 string,
checks giving {"passed": true} for every A2 check, and "changed", when
present, an object keyed by case id whose values give a non-empty claim
or excerpt text and no other field (absent: no case changed); with the
pack's case ids, a changed case outside them is a problem too. Each
problem has its own reason.

- apply_a2 checks a result against the contract before its pass-time
  checks (input bytes, frozen cases, a text that differs from the current
  one). It now also refuses a present "changed" that is null, [], "", 0
  or false, which `or {}` used to read as no change, and gives reasons
  where a malformed checks value used to raise AttributeError.
- p1_freeze.py --final checks every custody original against the same
  contract, with the final pack's case ids (cases are only appended), in
  place of its own round-1 copy; sha256_field_problem moves with it.

Tests (tests/test_jev_p1_casepack.py, FreezeTests):
- twelve wrongly shaped "changed" values, the review's ["bad"] first,
  each refused by the full --final gate and by apply_a2 under one reason;
  a pass that changed one case's excerpt still freezes
- negative control: the same cases against p1_freeze.py byte for byte
  from `git show 46d1a7a` (sha256 6e2e2e70..., the digest the registry
  recorded) freeze; skipped where the clone lacks the commit

The draft freeze manifest's p1_casepack.py and p1_freeze.py digests and
the README's A2 and freeze paragraphs follow the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
p1_casepack.py, p1_freeze.py, the p1 README, tests/test_jev_p1_casepack.py
and the draft freeze manifest, re-registered with
scripts/host_receipts.register_file on this branch's registry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Claude architecture lane (wsl-architecture-design): disposition at 38b6780a15e8c6f32137e6f698b7ce9e012bd672. It replaces the Codex root's HOLD on 4fba3ce3 and the cross-family job 002 finding on 46d1a7a0.

  • F2, root: closed at 46d1a7a0. A missing, null or invalid result_sha256, or a custody file holding JSON null, non-JSON text or an array, is now refused with its own reason. Bounded job 002 (GPT Sol max, read-only) confirmed all four of root's constructions are refused.
  • The P1 from job 002: a wrongly shaped A2 changed value (["bad"]) was accepted. Closed at 38b6780a.
    • a2_result_problems (p1_casepack.py L309) is now the single A2 result contract, used by both apply_a2 and the freeze custody gate (p1_freeze.py L136, L173, L204).
    • 12 malformed cases are refused by both gates under the same reason, and a valid change still freezes.
    • Negative control: the 46d1a7a0 predicate freezes all 12 cases.
  • Verification:
    • tests.test_jev_p1_casepack: 57 tests OK.
    • The scoring tests: 32 OK, in the README's hash-locked virtual environment.
    • scripts/validate.py passed.
    • merge-tree against main e0c329ae is clean.
  • Residuals, recorded and not fixed (bounded loop; this was the last repair round):
    • extra top-level and checks fields stay allowed, as apply_a2 has always allowed them;
    • changed is not cross-checked against the final case texts;
    • duplicate JSON keys are not refused;
    • the history-based controls go blind after a squash merge;
    • three malformed shapes are refused by a crash rather than a reason;
    • F1 and root's F3 method gate (Decision 12) are unchanged.
  • It lands under the agreed rule when the required checks are green on this head.

@seathatflowsinourveins
seathatflowsinourveins merged commit fca0da2 into main Oct 4, 2026
25 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the c5/jev-p1 branch October 4, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant