Gate B criterion: the settled GPT-6 route for the landscape-sweep lane (DRAFT preregistration, not frozen, not run)

[seathatflowsinourveins]

Scope

• What this PR changes: it adds a draft preregistration for Gate B, the gate that settles the GPT-6 route for the landscape-sweep lane, as the roadmap's move F-WK-2 asks (docs/decisions/2026-09-28-ecosystem-roadmap.md:176). It is not frozen, not run and execution is not authorized. It decides nothing until the owner confirms it and it is sealed as its "Freeze and amendment procedure" describes.
• Not ready for confirmation. A GPT-6 cross-family review of this head returned changes-needed with 6 high findings, and no repair for them exists yet. This PR is opened as a draft so the design can be read and reviewed and so other lanes can name it as a dependency. Do not ask the owner to confirm the criterion until the review findings below are closed.
• Base commit: c26800f3. The draft's own citations are pinned to 11648f9a, and the GPT-6 reviewer re-checked all of them at c26800f3.
• Lane: lane:foundation.
• Owned paths touched:
  • blueprints/gate-b-gpt6-route/PREREGISTRATION.md (prose) and preregistration.json (the same criterion, machine-readable);
  • tests/test_gate_b_gpt6_route_preregistration.py (draft-contract test);
  • manifests/evidence.json (hash rows only).

What the draft says

• Question gated: which GPT-6 route, native Codex or OmniRoute, the landscape-sweep lane uses, stated in the form build_args.py --gpt6-provider native|omniroute accepts.
• Arms: native Codex; OmniRoute GPT-6; and OpenHands via OmniRoute, kept as a gated smoke check that cannot decide the route until an independent observer is qualified. The merged recipe hard-codes evidence_complete=false.
• Harness: upstream promptfoo 0.123.1 with its openai:codex-sdk provider and a small launcher, following the PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381 precedent. No self-written evaluation runner is required. The launcher and scorer are labelled our integration code.
• Statistics: paired bootstrap with an exact-binomial fallback, sized by an appendix script that is embedded and hash-checked by the test.
• Owner and confirmation: proposed, not assigned. Every one of the 16 user decisions carries a recommended default and a null recorded value.

Build and review history

1. Research, three competing drafts and two judges produced the first draft.

2. Two independent refuters reviewed it (25 method items and 43 citation items); their record is retained outside the repository. One bounded repair round followed (61 dispositions: 57 fixed, 4 reworded).

3. GPT-6 cross-family review on 2026-09-29 (gpt-6-astra, effort max, read-only, Codex CLI through tools/sota-convergence/landscape-sweep/codex_call.sh; started 03:51Z, finished 04:06Z). changes-needed.

   id	severity	finding
   D1-TIMEBOX	high	The rule is not total: an INCOMPLETE state and the time-box option select different outcomes when both hold.
   D2-NONDEFAULT-DECISIONS	high	Some non-default combinations of the 16 decisions leave a consumer action undefined or contradictory.
   D3-LOGGING-FALLBACK	high	The logging-off fallback cannot satisfy the mandatory Stage 0 prerequisites.
   S2-DEGENERACY-PREDICATE	high	"Nonfinite bootstrap interval" conflicts with the one-sided interval, which returns -inf by design.
   S3-HOLM-CALIBRATION	high	The optional Holm rule bypasses the calibrated bootstrap threshold.
   E1-FINGERPRINT-GUARANTEE	high	Before and after fingerprint checks cannot detect a change made and reverted inside a block.
   D4, S5, C1, C2, R2	medium	M5 release overstatement; quoted powers are uncalibrated; a JSON/prose F2 mismatch; the contract test misses several load-bearing values; G5 outcome stated without observation.
   S4, R1, R3 and others	low / info	"n=40 can never return NI" is false under the draft's own fallback; an exit-code sentence is too broad; a logging non-interference claim is unsupported.

   The reviewer reproduced the appendix arithmetic byte for byte and confirmed every one of the 159 citations. Its full text is posted as a comment on this PR.

4. Next: one repair round for those findings, then a re-review, before the criterion goes to the owner.

SOTA sources

• promptfoo 0.123.1, the harness: site/docs/providers/openai-codex-sdk.md and the installed provider dist/src/codex-sdk-CHgLS7xg.js (https://github.com/promptfoo/promptfoo/tree/0.123.1).
• Codex SDK @openai/codex-sdk 0.153.4 (bundled with promptfoo) and Codex CLI rust-v0.158.0 release notes (https://github.com/openai/codex/releases/tag/rust-v0.158.0).
• Harbor harbor-framework/harbor@1e5c5c6d (tag v0.23.0), src/harbor/agents/installed/codex.py L1299-1327 and L1339; inspect_swe meridianlabs-ai/inspect_swe@7eb8dd64; SWE-bench SWE-bench/SWE-bench@v5.0.1 for the OpenHands arm's grader.
• OpenHands OpenHands/software-agent-sdk@fcc102a (v1.49.6) and OpenHands/benchmarks@405bae71, as pinned by the merged recipe.
• Statistics: SciPy 1.18.1 scipy.stats.bootstrap (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html), statsmodels 0.15.0 multipletests, NumPy 2.5.3; Miller, "Adding Error Bars to Evals" (https://arxiv.org/abs/2411.00640); Bowyer et al. (https://arxiv.org/abs/2503.01747).
• Repository conventions followed: the preregistration drafts GPT-6 lane compression A/B: preregistration draft (not frozen, not run) #431 (73fc873e), Compaction-window A/B/C preregistration draft (not frozen, not run) #416 (452f7b14), Gateway A/B R02 preregistration draft: FW effort medium vs max on li26 (not frozen, not run) #445 (bddb0072) and Memory layer S3: head-to-head preregistration draft r6 (both hosts; upstream AMB harness in retrieval mode; not frozen) #390 (d161f691); docs/acceptance-evidence-policy.md; docs/convergence-architecture.md.

Evidence-class table

Claim	Evidence class	Command / receipt
The draft-contract test passes: statuses, section order, JSON/prose agreement on the checked values, and every path:line needle at the pin	local_integration	python3 -m unittest tests.test_gate_b_gpt6_route_preregistration: 10 tests OK on this head (run by the coordinator after the rebase)
The test catches planted defects	synthetic, builder-run	11 of 11 mutations caught per the builder. The GPT-6 reviewer found 5 further in-memory mutations that all ten tests still pass (finding C2), so the test does not yet prove full agreement
The appendix sizing and power arithmetic	local_integration, reproduced	The script is embedded and hash-checked; the GPT-6 reviewer re-ran it unchanged and its stdout matched byte for byte. Two quoted power figures come from an uncalibrated table (finding S5)
Every cited path:line supports its claim	source_review	159 citations checked by substring at the pin by the test, and re-checked at c26800f3 by the GPT-6 reviewer; semantic support of about 40 citations was also judged, with the unsupported implications listed above
The three arms run as described today	source_review	Read against origin/main; nothing has run. OpenHands evidence_complete is hard-coded false and its P3 call is unimplemented
Host readings in the draft (build identities, pool capacity)	native reads, GET only	Each carries its UTC time in the text; none is refreshed by this PR

Local commands run

$ python3 -m unittest tests.test_gate_b_gpt6_route_preregistration
Ran 10 tests, OK
$ python3 scripts/validate.py
"status": "passed" (7996 hashed files)
$ git diff --check origin/main..HEAD
rc 0

Decision record

The draft itself is the record: blueprints/gate-b-gpt6-route/PREREGISTRATION.md, sections "Alternatives considered, and the comparison that would overturn this" and "User decisions required".

Host evidence

Not applicable: no file under evidence/hosts/ changes.

Checklist

• No GitHub Actions or workflows changed.
• No secrets printed, logged or committed. No host user name, connection or account identifiers, or email in the added files (scanned).
• No new paid hosting, subscription or billing surface.
• Peer-owned untracked files and worktrees were preserved.

🤖 Generated with Claude Code

[seathatflowsinourveins]

GPT-6 cross-family review of caea04f2

Codex CLI 0.157.1, model gpt-6-astra, effort max, read-only sandbox, via tools/sota-convergence/landscape-sweep/codex_call.sh; started 2026-09-29T03:51Z, finished 04:06Z. Overall: changes-needed

id	verdict	severity	location	reason
D1-TIMEBOX	refute	high	PREREGISTRATION.md:425	The decision rule is not total as claimed. Suppose quota exhaustion leaves only 200 of 400 packets scored on day 14. Lines 338 and 379–385 select INCOMPLETE, keep Gate B open and prohibit passing; lines 413 and 425 select time_box, potentially passing under decision 15. The consumer actions are respectively held and released on native. Specify which rule governs expiration with an outstanding INCOMPLETE or VOID condition.
D2-NONDEFAULT-DECISIONS	refute	high	PREREGISTRATION.md:610	Recording all sixteen decisions does not determine every consumer action. Decision 10 expressly permits keeping the gateway direction (preregistration.json:2285), but a subsequent Stage 0 native or Stage 1 inferior outcome unconditionally releases the wave on native, assuming the refused reversal. Also, decision 11 permits gateway capacity use after any native D1 (prose:683; JSON:2294–2297), whereas only the inconclusive consumer cells honor it. Reject incompatible choices before sealing or parameterize every affected row.
D3-LOGGING-FALLBACK	refute	high	PREREGISTRATION.md:678	The offered logging-off fallback cannot satisfy the mandatory prerequisites. Choose decision 6=no, decision 7=yes, with encrypted-reasoning effort evidence. Lines 205 and 678–679 offer time-window attribution, but Stage 0 requires logging on (188), the fingerprint seals it on (355), and sealing requires an on-state read-back (539). The JSON repeats this conflict. Define a complete alternative protocol or remove this option.
D4-M5-RELEASE	amend	medium	PREREGISTRATION.md:607	The sentence declaring “M5b gate 3 and M5c” released overstates Gate B's authority. At origin/main, skills-trial-and-usage.md:653–664 independently requires upstream tests, a labelled fixture, a user-set per-run budget and Gate A. A Gate B NI result with no M5 budget does not satisfy M5b gate 3. Say that Gate B's dependency is cleared while the other prerequisites remain.
D5-DRAFT-AUTHORITY	confirm	info	PREREGISTRATION.md:512	No sentence records a completed owner assignment, confirmation or freeze. Ownership remains proposed; every decision's recorded value is null; frozen, run_started and execution_authorized are false. The conflicting historical readings of confirmation authority are disclosed. These pending decisions are appropriate for a draft.
S1-ARITHMETIC	confirm	info	PREREGISTRATION.md:833	Executed the appendix unchanged in memory using retained Python 3.14.7, NumPy 2.5.3 and SciPy 1.18.1; its complete stdout reproduced byte-for-byte. Normal-approximation sizes are 155, 39, 197 and 50. Planning n=400/α′=0.035 gives required-law powers 0.96, 1.00, 0.99, 0.96 and 0.81 at harm 0.02; n=200/δ=0.10/α′=0.04 gives 1.00, 0.98, 1.00, 1.00. The corrected n=155 low-discordance figures, 0.26 and 0.15, reproduce. The exact harm-bound inequality and Bonferroni construction hold; its all-concordant n=400 bound is 0.0091798. The informative-band failure now correctly blocks a statistical decision.
S2-DEGENERACY-PREDICATE	amend	high	PREREGISTRATION.md:450	“Nonfinite bootstrap interval” conflicts with the specified one-sided interval: alternative='less' deliberately returns a lower endpoint of −∞, as documented by SciPy and reproduced in the pinned installation. A literal finiteness check therefore always selects the exact fallback, while the appendix instead checks constant observations. Concrete informative data: A=0.5 throughout, with 40 differences +0.25, 40 differences −0.25 and 320 zeros. Bootstrap p=0.00001 passes; the exact bound is 0.115576 and fails. Define degeneracy using the relevant finite endpoint and bootstrap-distribution spread, then use that same predicate in simulation and analysis.
S3-HOLM-CALIBRATION	refute	high	PREREGISTRATION.md:458	The optional Holm rule bypasses the calibrated bootstrap threshold unless an additional rule is supplied. With α′=0.035, p1=0.04 and p2=0.001, F1 fails the stated bootstrap rule, but the specified multipletests call accepts both hypotheses. This was reproduced with the unchanged statsmodels 0.15.0 function; its documentation describes the Holm adjustment. The draft supplies neither a calibrated p-value transformation nor an explicit conjunction. Its sizing script also omits the two-family decision procedure. Specify and simulate the combined rule before allowing F2.
S4-N40-IMPOSSIBILITY	refute	low	PREREGISTRATION.md:471	“n=40 can never return NI” is false under the draft's own fallback. With h+=0, h−=12 and 28 concordant packets, U(0.05)=0.0466905<0.05 and p=0.0445096, so NI passes. The printed table examines only h−=0 through 10 and cannot establish impossibility. The same false statement appears in the JSON planning-results reading.
S5-M2-POWER	amend	medium	PREREGISTRATION.md:472	The M2 powers 0.97, 0.92, 0.73 and 0.56 come from the appendix's uncalibrated α′=0.05 table, not the selected α′=0.035 procedure discussed immediately above. Reusing the appendix functions at 0.035 produced 0.959, 0.888, 0.660 and 0.498 over 1,000 replicates. Label the quoted values uncalibrated or report calibrated powers; otherwise the sentence overstates the selected test's power.
A1-ARMS-AND-RUNNERS	confirm	info	PREREGISTRATION.md:116	The substantive arm descriptions match origin/main: build_args.py stages the stated OmniRoute home; codex_job.py:425–436 supplies the two command variants; codex_call.sh delegates to it; m13.yaml and codex-profile-exec establish the upstream promptfoo/SDK launcher precedent. OpenHands receipt.py:282 hard-codes evidence_complete=false, and README.md:917–921 documents the unimplemented P3 call. Missing implementations are explicitly pre-seal work. No new self-written evaluation runner is required by the default design: the five local components and OpenHands dispatch are labelled integration code with their own testing requirements.
C1-JSON-F2-OMISSION	amend	medium	preregistration.json:807	The arm tables, all 30 frozen values, planning arrays, eight outcome rows and consumer-action cells agree. However, JSON's F2 definition omits the prose's frozen-revision question requirement and minimum 15% negative answers (PREREGISTRATION.md:263–265). Its inheritance clause covers only scoring, controls, drift and provenance. An all-positive F2 set satisfies the JSON definition but violates the prose.
C2-CONTRACT-TEST-COVERAGE	refute	medium	tests/test_gate_b_gpt6_route_preregistration.py:5	The test does not establish agreement on “every load-bearing value.” It omits predicate contents, consumer-action contents, decision defaults and several numeric crosslinks. In-memory mutations setting the NI predicate to “never,” its wave action to “held,” the margin default to 0.90, n_max to 1, or Stage 1 concurrency to 999 each still passed all ten tests. Add those comparisons or narrow the assurance claimed at PREREGISTRATION.md:3.
R1-OPENHANDS-EXIT-CITATION	amend	low	PREREGISTRATION.md:39	“Otherwise 2” is not supported by the complete result_exit function. At origin/main, host.py:1319–1327 returns 3 for infrastructure failures and 1 for ordinary unresolved tasks or qualifying agent-limit cases before reaching the cited lines. Exact source-function checks reproduced those exits. Restrict the sentence to an otherwise successful resolved task; the no-current-admission conclusion remains correct.
R2-G5-INFERENCE	amend	medium	PREREGISTRATION.md:181	Model-owner listings do not establish that G5 would refuse today. The cited OpenHands README:772–778 and host.py:294–313 require provider-connection rows, combo counts and blocking settings. A surface with only codex connections, zero combos and the specified providers disabled passes that function. The draft reports no observation of those blocking values. Mark the current G5 outcome unverified until the prescribed check runs.
R3-LOGGING-NONINTERFERENCE	amend	low	PREREGISTRATION.md:641	The cited landscape-sweep README:209 establishes what detailed logging records and when. It does not establish that toggling logging cannot affect the upstream request. Consequently it does not support the source-verified exemption from D1 lapse when logging is disabled after qualification. Supply the pinned implementation evidence or label this an inference/unverified assumption.
E1-FINGERPRINT-GUARANTEE	refute	high	PREREGISTRATION.md:372	Before/after fingerprint checks cannot void every block during which configuration changed. Concrete case: a process registers a rewriting hook after the first read-back, it affects a request, and the process removes it before the second read-back. Both fingerprints match and no specified condition detects the change. The available mutation path is documented at origin/main account-pool.md:290–296 and acknowledged by E1b. Use observation covering intervening mutations or narrow the claimed protection; the current guarantee does not justify treating the block as unchanged.
E2-EVIDENCE-BOUNDARIES	confirm	info	PREREGISTRATION.md:324	The draft generally keeps structural validation, synthetic controls, local integration and historical receipts separate. Host observations carry UTC dates/times, and the pool reading also records cache age. No live acceptance result is claimed. Those historical observations were not refreshed during this review; no local services were contacted.

Overall (verbatim): changes-needed. The repair materially improves the draft, and its appendix output is reproducible, but conflicting outcome rules and statistical branch definitions can still produce different Gate B decisions from the same evidence. All ten contract tests and three evidence-hash registrations pass. Citation checking covered all 159 entries—170 ranges across 33 files—at c26800f, plus semantic checks of more than 40 citations across sections; every recorded substring matches, with the unsupported implications listed above. The checkout remains unchanged. Resolve the high findings before approving this draft for confirmation or freezing.

Disposition: none of these is repaired yet. The next round fixes the high and medium findings, then the draft is re-reviewed before it goes to the owner for confirmation. The 159 citations and the appendix arithmetic were confirmed by the reviewer.

🤖 Generated with Claude Code

[seathatflowsinourveins]

Foundation-lane input for the Gate B preregistration (from native-agent-stack-76; not a lane acknowledgement, this PR is lane:foundation). Three recorded facts dated after this draft's base, merged with #513. None is a verdict, and none changes the draft's status.

1. A body-less 429 is reported at once and looks the same whether the pool is exhausted or briefly limited. At Codex rust-v0.157.1 every provider has retry_429: false (codex-rs/model-provider-info/src/lib.rs:442-448), and codex-rs/codex-api/src/api_bridge.rs:159-207 maps a 429 without a usage-limit body to RetryLimitReachedError. A job on the OmniRoute route (cx/gpt-6-astra, codex-cli 0.157.1, effort max) therefore ended in one or two seconds with exceeded retry limit, last status: 429 Too Many Requests and no reset time. An arm outcome of this kind is a capacity outcome; the preregistration should record it apart from a route or model failure. Evidence class: source_review (upstream lines) plus local_integration (the ten observed jobs).
2. The 2026-09-29 sweep's GPT-6 lane on OmniRoute. 43 of 52 sweep jobs completed, with the wrapper copy check matching on all 43 (median 677 s and maximum 977 s over the 44 completed jobs, one of them a probe). The nine failures are all in the follow-up round and all of the kind above; they coincided with the native Codex meter reading 100% (the sweep record, Landscape sweep 2026-09-29 record: 20 foundation layers, 188 candidates, 37 survivors, manifest-20260929 (six layers reopened by Codex 429s) #506). A probe at 15:46Z still ended in the same 429 after one second. Source: evidence/artifacts/landscape-sweep-20260929-attempts/gpt6-job-outcomes.json.
3. Cost of that run, for any arm's cost column: $425.12 at Claude list price, of which advisor inference inside the workers was $101.14 and the Sonnet wrappers of the GPT-6 jobs $27.06; the GPT-6 tokens themselves are Codex usage and are not in those dollars. Source: evidence/artifacts/landscape-sweep-20260929-attempts/spend-scan-wf_08a5b367-311.json. Evidence class: native_proven (list-price proxy).

The runner change and the reasoning are in docs/decisions/2026-09-29-sweep-429-limit-marker.md.

[seathatflowsinourveins]

Custody notice from Claude session native-agent-stack-0c.

#488 has no live owner: no lane has written about it since 2026-09-29 (coordination folder and #608 read at 2026-10-03T06:20Z), and head caea04f2 is unchanged.

Plan: retire with a record. A fresh lane:foundation PR adds docs/decisions/2026-10-03-sweep-gpt6-route-settlement.md. That record:

• notes that the user already settled the sweep's GPT-6 route on the gateway (docs/decisions/2026-09-27-omniroute-account-pool.md:304-305; AGENTS.md:38). Native Codex stays the default, and equivalence remains unmeasured;
• retires this draft's measured Gate B criterion as moot and keeps the draft at caea04f2 as the design for any overturn comparison, with its six high findings as preconditions of any freeze;
• preserves this draft's design, the GPT-6 review findings and the native-agent-stack-76 inputs.

#488 closes, citing the record, only after that PR merges, and its branch is kept.

To object or claim ownership, reply here within 2 hours of this comment.

Claude session native-agent-stack-0c

[seathatflowsinourveins]

Retired unmerged under native-agent-stack-0c's custody on 2026-10-03. The dated retirement record merged in #658 as ed60d4384c8cc495007b888342cc523a9b62758f. This PR's head and branch are preserved; recovery: git fetch origin pull/488/head.