Record the failed hosted trial of unittest-parallel on the two required test jobs

docs/harness-defaults.md

@@ -86,6 +86,11 @@ When a claim or action proves wrong, correct it where it was relayed and record
 
 | Date | Anti-pattern | What happened | Rule or check that prevents it | Where enforced |
 | --- | --- | --- | --- | --- |
+| 2026-10-03 | Pointing an observed record's frozen inputs at a copy of the plan instead of at the files the plan froze | The first observed convergence record of the suite-parallelism trial listed the preregistration copy as its source, input and evaluation artifact, and the oracle's trimmed output as an evaluation input, so `scripts/validate_convergence.py` re-checked none of the preregistered bytes; the delta review of #652 found it | Keep byte copies, with the same sha256, of every file the preregistration froze and list them under the preregistration's own role lists; never list an oracle output as a frozen input; name a lock copy so that no dependency scanner pattern matches it | this log; the trial outcome artifacts under `evidence/artifacts/suite-parallelism-trial-20261003/frozen/` and their README |
+| 2026-10-03 | Putting a free-form receipt in place of a preregistered observed convergence record | The suite-parallelism trial's outcome pull request (#652) first kept the failed trial's per-run data only in a receipt and stored the preregistered record as `.json.txt`, so `scripts/validate_convergence.py --all-recorded` checked nothing about the trial: no artifact hashes, run and failure coverage or usage fields. The automated review of #652 found it | Write the observed record that the preregistration names, in a path that does not re-trigger the trial, validate it with `scripts/validate_convergence.py` and list it in `convergence_records`; a receipt adds detail but does not replace it | AGENTS.md:18; this log; [trial outcome record](decisions/2026-10-03-suite-parallelism-trial-outcome.md) and its observed record |
+| 2026-10-03 | Closing a substantive trial without its completeness critique | The same outcome record listed the runner classes the trial did not evaluate "for a pending survey" and recorded no critique of the missed modality, sources or candidate classes. The review of #652 found it | End every substantive research or adoption unit with the completeness critic, mark each relayed candidate claim as verified against its upstream source or as a lead, and record the seeds for the layer's next landscape sweep in the unit's record | AGENTS.md:18; this log; the trial outcome record's "Completeness critique" |
+| 2026-10-03 | Preregistering a hosted whole-suite trial after a preflight of one module | The unittest-parallel 1.8.6 trial was preregistered after its only parallel preflight, a run of the 29-test wrapper module `tests.test_native_maintenance`; the plan's whole-suite local preflight never ran, as the preregistration itself recorded. The hosted run then ended every parallel run with `TypeError: cannot pickle '_contextvars.Context' object`, raised by the two modules with `IsolatedAsyncioTestCase` classes; the third failing module, of dynamically built classes, raises a `PicklingError` that the pool's first-exception rule masks (`Lib/multiprocessing/pool.py:822` at CPython v3.12.3). Each of the three modules reproduces its error locally under the candidate runner, so a local whole-suite run on the 48-core host would very likely have shown the crash before the hosted run (inferred from the per-module reproductions; the whole suite was not run locally under the candidate runner) | Run the whole suite once under the candidate runner locally before preregistering a hosted trial, and treat a traceback at the end of that run as a failed preflight: unittest-parallel raises a pickling error only after every other suite has run | This log; [trial outcome record](decisions/2026-10-03-suite-parallelism-trial-outcome.md) |
+| 2026-10-03 | Leaving the record path of a self-triggering trial workflow undecided until after the run | The trial workflow triggers on `pull_request` with `paths` set to its own file (`.github/workflows/macos-suite-parallel-trial.yml:41-44` at `1e4bb5ab`). A pull request's paths filter is evaluated on its three-dot diff, which always contains a newly added workflow, so every push to the trial branch and every pull request that adds the file re-runs the whole trial (38 jobs, 20 of them on macOS). The observed record could not be pushed to the trial branch or land with the workflow without re-running the trial, so it went to a separate pull request | Decide the record path before the first push: a separate pull request that does not contain the workflow, or a one-shot guard in the workflow | This log; [trial outcome record](decisions/2026-10-03-suite-parallelism-trial-outcome.md); paths filter: [github/docs at `06854646`](https://github.com/github/docs/blob/068546469ae7f079368d12f969991a045121c4a0/data/reusables/actions/workflows/triggering-a-workflow-paths5.md) |
 | 2026-10-04 | Replacing a historical launcher candidate while claiming refusal preservation | PR #685 round two replaced the prior GNU env branch and command tail with a uutils reading, and coreutils dispatch skipped the raw keyring start. The supplied R1–R5b oracle found lost main and round-one refusals despite a passing filtered replay. The expanded comparison fails at `ec382e21` and passes after retaining all candidates. | Queue the union of raw, GNU and uutils command tails, scan raw dispatch words before unwrapping, and compare both pinned baselines over the supplied oracle and existing fixture rows. | `tests/test_secret_path_guard.py`, `test_review_685_monotonicity_against_pinned_baselines`, with hash-pinned originals in `tests/fixtures/guard_685/`; returned evidence in [the repair record](decisions/2026-09-24-secret-storage.md#addendum-2026-10-04-pr-685-bounded-repair). |
 | 2026-10-04 | Sharing launcher option parsing because implementations share a literal string splitter | PR #685 round one treated GNU and uutils env split options alike. Uutils 0.10.0 preprocesses only literal split prefixes, keeps scanning after assignments, and lets clap discard inferred split values; the incorrect allow controls could dump the environment. The corrected review tests produce 95 failing subtests at `a7888d31`. | Check each implementation's preprocessing and final option parser independently; refuse if either reading refuses, and replay against both main and the repair's starting head. | `tests/test_secret_path_guard.py`, `test_review_685_uutils_split_preprocessing`; sources and returned evidence in [the secret-storage repair record](decisions/2026-09-24-secret-storage.md#addendum-2026-10-04-pr-685-bounded-repair). |
 | 2026-10-04 | Letting a no-command refusal hide a broken launcher test | The PR #685 split-string row without a trailing operand already returned `environment_dump` at `385e3f65`. Adding `EXAMPLE_OTHER` makes that same revision allow the command and the repaired guard refuse it. | Include a harmless trailing operand when testing effective launcher argv so an absent-command fallback cannot supply the expected refusal. | `tests/test_secret_path_guard.py`, `test_review_685_env_split_string_effective_argv`; the retained round-two baseline comparison. |

evidence/artifacts/suite-parallelism-trial-20261003/README.md

@@ -0,0 +1,74 @@
+# Suite-parallelism trial, run 37109532421 (2026-10-03)
+
+Sanitized excerpts and byte copies of the preregistered trial of unittest-parallel 1.8.6 on the whole test suite
+(draft pull request #646, head `1e4bb5ab7c79faa83e4cd8d04cf8dc6d33c7b0de`, closed unmerged). Only the ubuntu-24.04
+jobs ran; the run was cancelled before any macos-15 job received a runner. The GitHub run records expire under the
+repository's 90-day retention and the 18 artifacts on 2026-11-02, so these files and their hashes are the durable copy.
+
+The convergence record of the trial is [`experiment.json`](experiment.json), the observed record (status
+`observed`, decision `reject`, 36 observations) that `scripts/validate_convergence.py` validates and
+`manifests/evidence.json` lists in `convergence_records`. The receipt
+[`suite-parallelism-trial-20261003.json`](../../receipts/suite-parallelism-trial-20261003.json) holds the detailed
+per-run data, the claim text, the sha256 and size of every original file (`data.original_files`,
+`data.hosted_artifacts`) and of every file here (`data.retained_files`). The decision is recorded in
+[`2026-10-03-suite-parallelism-trial-outcome.md`](../../../docs/decisions/2026-10-03-suite-parallelism-trial-outcome.md).
+
+| File | Source | Treatment |
+| --- | --- | --- |
+| `experiment.json` | Generated by a script from the raw run directories (`exit-code.txt`, `step-seconds.txt`, `command.txt`, `meta.json`, `runtime.json`, `git-status.txt`, `log.txt`), the jobs API records (read with GET, steps included) and `result.json`, with the preregistered fields carried over unchanged | The observed convergence record: one observation per timed ubuntu-24.04 run (12), per ubuntu-24.04 control job (4) and per macos-15 job that never started (20). Its observations cite the files below by sha256, and its `frozen_inputs` cite the 15 byte copies under `frozen/` (below) in the preregistration's role lists and order, with the preregistration copy at the end of `sources` |
+| `preregistration-experiment.json.txt` | `blueprints/convergence-practice/macos-suite-parallelism-20261003/experiment.json` at the trial head, status `planned` | Byte copy (sha256 `305fc2fa2ddd2c0a0eba4d85d748807db87302bde59a6b6489de866bee416c08`). The `.txt` suffix keeps `scripts/validate_convergence.py --all-recorded` from reading it as an undeclared convergence record, which it rejects, and from checking its frozen-input hashes against files that exist only on the trial branch |
+| `result-trimmed.json` | `result.json` of the `compare` artifact (815,073 bytes) | Trimmed: `decision_rule`, `inputs`, `id_mapping`, `controls` and `verdicts` unchanged; per run every field except the parser notes (kept as a count), the missing ids (kept once per level in `missing_id_sets`) and the mismatch records of missing ids (the others are kept in full). The S side of those dropped records, each missing id's serial outcome, is kept as counts per level and class (`s_outcome_totals`, `s_outcome_by_class`). An observation artifact, not a frozen input |
+| `summary.md`, `checkout-status.json` | The `compare` artifact | Byte copies |
+| `crash-tail-L4-r1.txt`, `crash-tail-L4F-r1.txt`, `crash-tail-L4C-r1.txt` | `log.txt` of the first repeat of each parallel arm | Last 30 lines, sanitized |
+| `log-key-lines.txt` | `log.txt` of all 20 run directories | First line, `Ran` and status lines, last line, sanitized |
+| `control-logs.txt` | The eight control and crash-control runs, synthetic fixtures executed on the hosted runner | Full logs, sanitized |
+| `serial-durations-S.txt` | `log.txt` of the three serial runs | The `--durations 25` tables with their `Ran` and status lines |
+| `local-reproductions.txt` | This record's local reproductions: the toy modules on CPython 3.12.3 and 3.13.16, the three real modules on 3.12.3 only and the order-throughput module on both | Commands, exit codes and output, sanitized; the two toy test modules' source |
+
+Sanitization: the runner's checkout path became `<workspace>` and any other path under the runner's home
+directory became `<runner-home>`. In the local reproductions, interpreter and virtual-environment prefixes
+became `<cpython-3.12.3>`, `<cpython-3.13.16>`, `<venv-3.12.3>` and `<venv-3.13.16>`. Nothing else was changed,
+and the copies under `frozen/` are not sanitized at all.
+
+## `frozen/`: byte copies of the 15 frozen files
+
+The preregistration froze 15 files by sha256 (`frozen_inputs` of `preregistration-experiment.json.txt`). The
+originals exist only on the trial branch, at the head `1e4bb5ab7c79faa83e4cd8d04cf8dc6d33c7b0de` of draft pull
+request #646, which was closed unmerged on 2026-10-03. Each file below was written with
+`git show 1e4bb5ab7c79faa83e4cd8d04cf8dc6d33c7b0de:<original path>` and is byte-identical to its original, so its
+sha256 equals the preregistered one. That equality is the point: the observed record's `frozen_inputs` cite these
+copies with the preregistered sha256 values, in the preregistration's role lists and order, so
+`scripts/validate_convergence.py` re-checks on `main` the exact bytes that the preregistration froze (15 of 15 equal
+on 2026-10-03). The receipt's `data.retained_files` lists each copy's sha256 and size. All 15 passed
+`python3 scripts/validate.py --scan-file` and are kept whole (264,054 bytes); none is sanitized, because that would
+change its hash.
+
+Each name is the original path with the trial directory `blueprints/convergence-practice/macos-suite-parallelism-20261003/`
+(`<trial>` below) shortened to `trial`, the leading dot of `.github` dropped and every `/` replaced by `-`, plus
+`.txt`, except the two lock copies. The `.txt` suffix keeps test discovery, workflow scanning and
+`scripts/validate_convergence.py --all-recorded` from reading a copy as a live file. A `.txt` name containing
+`requirements` is what Python dependency scanners and `tests/test_osv_lockfile_coverage.py` match, so the two lock
+copies are named `trial-ci-suite-<os>-pins.frozen`: neither `requirements` nor `.txt` appears in the name (bytes and
+sha256 unchanged). That this keeps them out of GitHub's dependency graph is expected from the name patterns and is not
+verified; check the dependency graph and Dependabot alerts on the pushed head.
+
+| Copy | Original at `1e4bb5ab` | Role in `frozen_inputs` | What it is | Bytes |
+| --- | --- | --- | --- | --- |
+| `tests-test_native_maintenance.py.txt` | `tests/test_native_maintenance.py` | sources | The B1 fix: the wrapper that loads the convergence-practice blueprint tests and registers each loaded module under a child name of itself, so that a spawned worker finds their classes by name | 2,389 |
+| `trial-ci-suite-macos-pins.frozen` | `.github/requirements-ci-suite-macos.txt` | sources | The macOS hash lock (unittest-parallel 1.8.6 and coverage 7.16.2) | 1,136 |
+| `trial-ci-suite-linux-pins.frozen` | `.github/requirements-ci-suite-linux.txt` | sources | The Linux hash lock (the same two packages) | 1,316 |
+| `trial-controls-test_zz_trial_controls.py.txt` | `<trial>/controls/test_zz_trial_controls.py` | inputs | The controls file: a pass, a skip, a failure, an error, a failing subtest, an expected failure, an unexpected success, a multi-line description and a failing `setUpClass` | 1,975 |
+| `trial-controls-crash_control-test_crash.py.txt` | `<trial>/controls/crash_control/test_crash.py` | inputs | The crash control: one test that ends its process with `os._exit(3)` | 476 |
+| `trial-controls-control_expectations.json.txt` | `<trial>/controls/control_expectations.json` | inputs | The expected results of the controls file run alone and of the crash control | 3,131 |
+| `github-workflows-macos-suite-parallel-trial.yml.txt` | `.github/workflows/macos-suite-parallel-trial.yml` | evaluation | The trial workflow: the inventory, control, arm and compare jobs | 56,566 |
+| `trial-compare.py.txt` | `<trial>/compare.py` | evaluation | The oracle: run eligibility, controls, id mapping and the decision rule | 79,570 |
+| `trial-ids.py.txt` | `<trial>/ids.py` | evaluation | The inventory script (discovery only) | 3,438 |
+| `trial-README.md.txt` | `<trial>/README.md` | evaluation | The trial's rules: arms, run contract and controls | 24,695 |
+| `trial-test_compare.py.txt` | `<trial>/test_compare.py` | evaluation | The oracle's own tests, which the compare job runs before the oracle | 57,574 |
+| `trial-test_b1_failure_mode.py.txt` | `<trial>/test_b1_failure_mode.py` | evaluation | The test of the B1 wrapper's failure mode (a blueprint that fails to load becomes one failing test) | 7,586 |
+| `trial-make_fixtures.py.txt` | `<trial>/make_fixtures.py` | evaluation | The fixture generator, which `test_compare.py` imports and runs | 12,699 |
+| `trial-fixtures-index.json.txt` | `<trial>/fixtures/index.json` | evaluation | The index of the oracle's fixture logs, with each log's sha256 | 11,011 |
+| `trial-fixtures-inventory-controls.txt.txt` | `<trial>/fixtures/inventory-controls.txt` | evaluation | The id inventory of the controls file for those fixtures | 492 |
+
+The observed record's `sources` ends with a sixteenth entry, `preregistration-experiment.json.txt` (the plan itself,
+not one of the 15 frozen files). `result-trimmed.json` is not a frozen input.

evidence/artifacts/suite-parallelism-trial-20261003/checkout-status.json

@@ -0,0 +1,128 @@
+{
+  "dirty": [],
+  "ok": true,
+  "rule": "every run directory's git-status.txt exists and is empty",
+  "runs": [
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4-controls",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4-crash",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4-r1",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4-r2",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4-r3",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4C-controls",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4C-crash",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4C-r1",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4C-r2",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4C-r3",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4F-controls",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4F-crash",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4F-r1",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4F-r2",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-L4F-r3",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-S-controls",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-S-crash",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-S-r1",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-S-r2",
+      "reason": null,
+      "status": ""
+    },
+    {
+      "clean": true,
+      "name": "ubuntu-24.04-S-r3",
+      "reason": null,
+      "status": ""
+    }
+  ],
+  "schema": "suite-parallelism-checkout-status/1"
+}