seathatflowsinourveins · Oct 2, 2026 · Oct 2, 2026 · Oct 2, 2026 · Oct 3, 2026 · Oct 3, 2026
@@ -0,0 +1,134 @@
+# Decision: a clean-room, two-family audit of the definitive manifest on 43 slots, with a deep-dive dossier of every candidate (2026-10-03)
+
+Lane: foundation. North-star action served: the new WSL's install record rests on picks that two model families reach
+independently, from the candidates' upstream code and README, without the instruction files that name candidates.
+
+Status: a record. It changes no row of the definitive manifest. Contests go to the manifest's owner, the pin conflict
+goes to the user, and the memory result goes to the head-to-head (#526).
+
+## Decision
+
+`evidence/artifacts/new-wsl-definitive-round-20261002/` records a clean-room round over the 43 foundation slots that
+#591 left open. It compares each slot with the definitive manifest at #602's merge commit (`675bdd51`, pinned by
+sha256).
+
+**Method.**
+- Every one of the 177 candidates got a dossier written from its upstream README and code: 148 at a tagged release, 23
+  at a snapshot without a tag, and 6 from page sources. Sonnet 5.5 wrote it at max; Opus 5.5 verified it at max against
+  5 cited claims, with at most one repair.
+  - 176 dossiers pass verification. One still fails after its repair, and the judges saw that verdict.
+- Both families then decided every unit in two seeded candidate orders, and a critic followed each family's deciders:
+  - Claude Opus 5.5 at max, through `claude -p --safe-mode --restricted --strict-mcp-config`;
+  - GPT-6 Astra at max, through `codex exec` with an isolated `CODEX_HOME` on the OmniRoute gateway.
+- Probes show that no instruction file reached either family (`clean-room.json`).
+- Skills are the exception on the GPT side. Codex listed the user's installed skills and its own system skills to the
+  GPT judges, and some read skill files by path. That exposes four slots on the GPT side (Limits); the Claude side ran
+  in safe mode without skills.
+- No GPT judge read the other family's records: an audit of all 2,504 GPT shell commands found none (`run-notes.json`,
+  `gpt_read_audit`).
+- Slots where the families differed, or where one family was undetermined, went to four adjudicators: one per family, in
+  both A/B orders.
+
+**Preregistration.** The plan was frozen before the first dossier and amended five times.
+- Amendments 1 to 3 came before any decision: the runner's prompt fix, the retargeting to an audit of #602, and the
+  re-verification of 13 dossiers that lost their verification to usage-limit stops.
+- Amendments 4 and 5 came after the results. Both are defect fixes, described under Limits.
+- Every amendment is hashed, and `freeze.py --check` verifies the chain.
+
+**Round outcome** (`selection.json`): 43 slots.
+- 35 definitive: 33 where the families agreed, and 2 adjudicated four of four (the deep-research harness and memory).
+- 7 left to measurement.
+- 1 user-pin conflict, where the families agreed on a repository other than the pin.
+
+The contamination audit of the raw returns has 0 hits.
+
+## Result against the definitive manifest
+
+From `audit-of-manifest.json`, at #602. The same verdicts hold against main after #620 (`cfd08b85…`): no slot's
+verdict, state or pick differs. #620 adds five rows the round does not cover.
+
+| Verdict | Slots |
+| --- | --- |
+| Agree | 22: zizmor, attest, Syft, Dependabot, actionlint, Serena, Ubuntu 26.04.1, QMD, git, gh, Worktrunk, difftastic, Compose, Trail of Bits skills, sandbox-runtime, OTel Collector, Prometheus, Inspect AI, Harbor, mise, Restic, betterleaks |
+| Contest | 7. #602 installs nothing; both families picked github/codeql-action (findings publication), claude-code-action (GitHub-side agent review), sem (agent structural diff), Phoenix (route qualification), promptfoo (CI regression gate), chezmoi (dotfiles) and trufflehog (credential verification) |
+| Nominates | 4 split rows. Loki (events store), Grafana (dashboards), SocratiCode (code search) and agent-browser (browser interaction) go to those rows' named measurements and settle nothing by themselves |
+| Cross-check | Memory. All four adjudicators chose ai-memory. The head-to-head (#526) stands; this is a nomination for it, not a decision |
+| Pin agree | Deep-research harness: gpt-researcher, adjudicated four of four, which is one of the two pinned harnesses |
+| Pin conflict | Agent runtime worker. Both families picked openai/codex over the pinned OpenHands software-agent-sdk. The pin stays until the user decides |
+| Not settled | 7. Client-native LSP (plugin or none), GPT gateway (OmniRoute or none; the pin stands unconfirmed), document ingestion (Docling or MinerU), local model server (#598's gate stands), workflow engine (Prefect or Dagu), usage meter (agent-console or ccusage) and page-text extraction (trafilatura or none) |
+
+**How to read the contests.** Every contested slot asked "which single tool does this job, or none?", and "none" was
+allowed in each.
+- Both families judged that a tool beats "none" for the job as the slot states it, and the critics cited upstream code
+  for the capability.
+- #602 resolved the same rows on job ownership: another installed tool already owns the job, or the layer's requirement
+  does not ask for it. Inspect AI owns prompt evaluation, mise and the bootstrap own configuration, and difftastic owns
+  structural diffs.
+- The round's judges saw the sibling slots of their unit but did not weigh overlap with the rest of the installed stack.
+- So the contests are clean-room picks for whoever keeps these jobs as slots. They are not evidence that the jobs need
+  owners. Whether they do is the manifest owner's call.
+
+## Limits
+
+- **GPT-side skill exposure.** An isolated `CODEX_HOME` does not hide `~/.agents/skills`, and codex lists the skills it
+  finds. The GPT judges therefore saw the names of the user's installed skills. On four slots the GPT vote is not
+  independent of the user's installed tools; the Claude vote is:
+  - browser interaction: an `agent-browser` skill, shipped by that contender;
+  - the task-scoped skill pack: skills from trailofbits/skills, mattpocock/skills and others;
+  - findings publication: a `codeql` skill;
+  - the runtime-worker slots: codex's own system skills.
+
+  The clean-room probe tested instruction files, not skills. A re-run should point codex's skill discovery at an empty
+  folder.
+- **Unequal live web evidence.** Both families had the same dossiers and source clones.
+  - The Claude judges ran 117 web searches through Claude Code's WebSearch tool, which uses a helper model (Haiku 4.5).
+    The results are not retained, so whether every search returned results is unverified. The live web also carries
+    this repository's public records, which name the manifest's picks, and a result read but not cited would not show
+    in the contamination audit.
+  - The GPT judges' 157 searches through the gateway came back empty, except 2 in adjudication. The gateway has no
+    credentialed search provider (the Mac gateway record, Limits).
+  - Agreement rests on the shared evidence. Where the families differed, this asymmetry is one possible factor
+    (`run-notes.json`).
+- **Amendment 4 came after the results.** The comparison script treated `installs_nothing_extra` as a "none" pick. That
+  contradicts amendment 2's rule text, under which a row's pick is the repository it names.
+  - The fix turned build provenance (attest) and dependency updates (Dependabot) from contest into agree.
+  - The pre-fix output is kept as `audit-of-manifest.before-amendment-4.json`.
+- **Amendment 5 also came after the results.** It answers the cross-family review of #663 and changes no outcome
+  status or verdict:
+  - An undetermined family's two decider picks now count as finalists. Document ingestion therefore lists Docling as
+    well as MinerU, the two its settling measurement compares.
+  - The run record now includes every dossier call.
+  - `compare.py` parses URLs instead of matching them as text; its output is byte-identical.
+  - `freeze.py` rejects an amendment that re-adds a frozen file.
+- **A runner defect cost three GPT calls.** The runner created a GPT answer folder only after the call returned.
+  - The first GPT call of each stage lost its answer and was retried successfully.
+  - Cost: 1,763 seconds, 2.37 million input tokens and 43 thousand output tokens.
+  - No outcome changed. The runner is kept as it ran (`run-notes.json`, `runner_defect`).
+- **Adjudicator anonymity.** The returns were shown as A and B in both orders. Their text can still carry family signals,
+  such as contender names that coincide with the families; the returns were not audited one by one.
+- **Evidence class.** These are model judgments on source review. Nothing was installed or measured.
+- **Usage** (`run-notes.json`) is a lower bound. Attempts lost at usage-limit stops are not counted.
+
+## Alternatives
+
+- **Publish a separate architecture from the round.** Rejected in amendment 2. #602 is the install record and has one
+  owner.
+- **Re-run the GPT half after configuring a search provider.** Not done: it needs a credential the user holds. It would
+  remove the web-evidence asymmetry and is the first step if a contest is pursued.
+- **Treat the contests as overturns.** Rejected: they answer a different question than #602's job-ownership rule.
+
+## Overturn
+
+- If the manifest's owner keeps any of the seven jobs as its own slot, the round's pick for that slot is the clean-room
+  candidate.
+- If the user drops the agent-runtime-worker pin, Codex is the two-family pick.
+- If the head-to-head (#526) runs, ai-memory is the round's nomination.
+- A re-run with working GPT web search, or with the job-overlap question asked, replaces this audit for the slots it
+  covers.
+
+## SOTA sources
+
+- The candidates' upstream repositories at the release tags recorded in each dossier.
+- `claude -p` (Claude Code), with its `--safe-mode`, `--restricted` and `--strict-mcp-config` flags.
+- `codex exec` of [openai/codex](https://github.com/openai/codex), through [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute).
+- #591's criteria and decision rule, byte-identical; the definitive manifest at #602 and at #620.
@@ -0,0 +1,44 @@
+# Clean-room definitive round of the new-WSL architecture (2026-10-02)
+
+A clean-room audit of the definitive manifest (#602) on the 43 slots that #591 left open, plus a deep-dive dossier of
+every candidate's upstream code and README. Both model families decide each slot in a clean room under #591's decision
+rule, and `compare.py` sets each result against the manifest row the slot maps to: agree, contest, nominates (a row
+waiting for its measurement), cross-check (a measurement row), pin agree or conflict, or not settled. The manifest is
+the install record; this round changes none of its rows and publishes no architecture of its own. A contest is a
+hand-off to the manifest's owner and a pin conflict goes to the user (amendment 2, which retargeted the round after
+#602 merged and before any decision ran). It replaces nothing in #591's, #589's or #602's folders.
+
+Why a clean room: `codex exec` loads `$CODEX_HOME/AGENTS.md`, and workflow subagents receive the global `CLAUDE.md`;
+both files name candidates of this field, so the earlier two-family agreement on those layers was not independent
+(`evidence/artifacts/new-wsl-clean-install-selection-20261001/cross-family/instruction-probe.json`). Here no judge, writer or verifier
+sees an instruction file, hook or MCP server (`clean-room.json`, with its probes). Skills are the exception on the
+GPT side: codex listed the user's skills and its own system skills to the GPT judges, which exposes four slots
+(`run-notes.json`, `gpt_read_audit`); the Claude side ran in safe mode without skills.
+
+| File | What it is |
+| --- | --- |
+| `units.json`, `contenders.json`, `build_units.py` | 19 units, 43 open slots asked as neutral function questions, and every candidate of each unit's field (the frozen #589 packets, plus the runtime rows' fields written for this round); prior picks and labels are left out |
+| `criteria.txt` | #591's frozen criteria, byte-identical |
+| `dossier-*.json/txt`, `verify-*` | The deep-dive dossier (README read in full, code, packaging, tests and CI, claim checks against code, cited as path:line at the revision) and its verification |
+| `decide-*`, `critic-*`, `adjudicate-*` | #591's decision rules, applied per unit with several slots, to dossiers and cloned sources |
+| `decision-rule.txt` | When a slot is definitive, how splits are adjudicated, and what happens when adjudication does not settle |
+| `clean-room.json` | The exact flags and isolated configuration of both clients, and the probes that show no instruction file reaches them |
+| `run_round.py`, `freeze.py` | The runner (clone, dossiers, packets, decide, critic, adjudicate) and the preregistration writer; `freeze.py --check` verifies the frozen files through the amendment chain |
+| `audit-observations.json` | A byte-identical copy of the upstream audit's observations frozen in the preregistration; the packets state its sampling limits (the first five release assets queried for attestations, one page of check runs) |
+| `compare.py` | The comparison with the definitive manifest at #602's merge commit (pinned by sha256); writes `audit-of-manifest.json` |
+| `preregistration.json` | The sha256 of every frozen file, recorded before the first dossier |
+| `preregistration-amendment-1.json`, `preregistration-amendment-2.json` | The runner's prompt fix (before the decide stage), and the retargeting to an audit of #602 with the comparison rule, the row mapping, the audit labels and the run notes (before any packet or decision) |
+| `selection.json`, `run-record.json`, `dossiers/` | The assembled outcome per slot under the decision rule, every attempt with its usage and the sha256 of each private original, and the 177 verified dossiers (sanitized) |
+| `audit-of-manifest.json`, `audit-of-manifest.before-amendment-4.json`, `preregistration-amendment-4.json` | The comparison with #602, and the defect fix of amendment 4 (made with the results known) with the output before it |
+| `preregistration-amendment-5.json` | The fixes after the cross-family review of #663 and its CI (made with the results known): finalists of an undetermined family, dossier calls in the run record, URL parsing in compare.py, freeze.py additions |
+| `run-notes.json` | The coordinator's run facts: timeline, the ordering that kept a family's decisions away from the other family's deciders, the unequal live web evidence, usage, the usage-limit stops |
+| `reverify.py`, `preregistration-amendment-3.json` | The frozen verification loop re-run on the 13 dossiers whose verifier attempts a usage-limit stop ended, with no dossier repaired twice (before any packet or decision) |
+
+Private originals (clones, raw returns, event streams) stay outside the repository; the assembled record lists their
+sha256. One change to the committed copies: in three dossiers (`anthropics__claude-plugins-official.json`,
+`facebookincubator__glean.json`, `scip-code__scip.json`), 40-character commit hashes are shortened to 12 characters
+(3, 3 and 7 values). The repository's secret scanner reads a 40-character value in a file that mentions Sourcegraph
+as an access token. The values still resolve, and no other field changed. The assembled copies had sha256
+`7cb10c9c…`, `8d448c0a…` and `b7007f33…` before the change. The judges read the private originals, which keep the
+full hashes. In the betterleaks dossier the quoted inline-allow marker is written `betterleaks[:]allow`:
+the repository's tests forbid the literal marker in any tracked file, because betterleaks would honor it. This is source review by model judges with adversarial critics: no candidate is installed or measured here.
@@ -0,0 +1,7 @@
+You adjudicate one slot on which two independent reviewers, A and B, named different defaults after their own adversarial critics. You do not know who A and B are, and their order carries no meaning.
+
+__CRITERIA__
+
+Read the slot's question and field (unit.json), the candidates' dossiers (dossiers/), the cloned sources (clones/) and the two returns (returns.json). Re-check at its source every fact on which either return turns. Then choose A or B: the default that the criteria, applied as written, support better on the evidence. Choose unsettled only when the evidence cannot separate them; then name the single measurement, runnable on the target host, that would. Not evidence: stars, downloads, popularity, current use, list order, or which reviewer argued more. Evidence is limited to your working directory and public sources, within at most 6 web searches.
+
+Slot: __SLOT__
@@ -0,0 +1,14 @@
+{
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["unit_id", "slot_id", "choice", "default", "reason", "checks", "settling_measurement"],
+  "properties": {
+    "unit_id": {"type": "string"},
+    "slot_id": {"type": "string"},
+    "choice": {"type": "string", "enum": ["A", "B", "unsettled"]},
+    "default": {"type": ["string", "null"]},
+    "reason": {"type": "string"},
+    "checks": {"type": "array", "items": {"type": "object", "additionalProperties": false, "required": ["fact", "source", "holds"], "properties": {"fact": {"type": "string"}, "source": {"type": "string"}, "holds": {"type": "boolean"}}}},
+    "settling_measurement": {"type": ["string", "null"]}
+  }
+}