Repository navigation
Trial: unittest-parallel for the two required test jobs (draft, preregistered, never merged) - #646
Closed
seathatflowsinourveins wants to merge 17 commits into
Closed
seathatflowsinourveins wants to merge 17 commits into
seathatflowsinourveins wants to merge 17 commits into
Conversation
tests/test_native_maintenance.py loaded three blueprint test files with spec_from_file_location under bare names, inside load_tests, and never registered them in sys.modules. unittest-parallel 1.8.6 pickles each test suite into a spawned worker (src/unittest_parallel/main.py:150-160 at craigahobbs/unittest-parallel@bda5d77), and pickle finds a class by its module name, so the six classes could not be pickled: _pickle.PicklingError: Can't pickle <class 'memory_patch_evidence_tests.EvidenceTests'>: import of module 'memory_patch_evidence_tests' failed Load the three files at import time of the wrapper as the child modules f"{__name__}.<short>", registered in sys.modules before exec_module (the Python docs importlib recipe "Importing a source file directly"). A spawned worker that unpickles one of the classes imports the wrapper, whose import-time code registers the child, and the import system then returns it from sys.modules (CPython 3.13.16 Lib/importlib/_bootstrap.py:1306-1316; Modules/_pickle.c find_class calls PyImport_Import). load_tests still returns loader.loadTestsFromModule for each module, in the same order; the blueprint files are unchanged. The 29 tests and their outcomes are unchanged. The six classes' test ids gain the wrapper's module name as a prefix, for example memory_patch_evidence_tests.EvidenceTests.test_retained_incomplete_attempt becomes tests.test_native_maintenance.memory_patch_evidence_tests. EvidenceTests.test_retained_incomplete_attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and real-run fixtures Stdlib-only oracle for the preregistered unittest-parallel 1.8.6 trial (blueprints/convergence-practice/macos-suite-parallelism-20261003/): - ids.py lists the sorted, de-duplicated test ids from discovery only (honours load_tests, runs no test body); 9,800 at 56473e4, equal to the Ran 9800 of the macOS full-suite log of run 37073310931 at that commit. - compare.py parses serial and parallel -v logs with rules cited from CPython v3.13.16/v3.12.3 Lib/unittest and unittest-parallel bda5d77 main.py, checks per-run eligibility against the trial inventory and the S baseline (flaky S ids excluded), verifies the base-to-trial id rewrite, evaluates the controls and applies the preregistered speed rule. - controls/: a known-outcome control file, a crash control and their expectations; never placed under tests/. - fixtures/: sanitized logs of real local runs of the controls under python3 -m unittest -v and six unittest_parallel flag sets, plus declared mutations the oracle must flag; test_compare.py covers them. Local integration evidence only; no GitHub-hosted run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unner Trial for a draft pull request that is never merged (plan W1): runs the whole suite serially and with craigahobbs/unittest-parallel 1.8.6 (release commit bda5d77dc1a2fa2df90f5f5a7de297ea375e345c) on macos-15 (arms S, P3, P3F, P3C, P4) and ubuntu-24.04 (S, L4, L4F, L4C), three repeats each, plus separate control jobs (control suite and os._exit crash control under a 300 s limit), an id inventory of the trial head and of base 56473e4, and a compare job that runs the oracle's compare.py. Every job checks out and asserts the pull request head. The Linux arms reproduce the steps the validate job runs before its suite; both arms jobs install the two npm pins the runtime tripwires require; the controls are split per runner because the hardening tests need a literal runs-on label per job. The two locks pin unittest-parallel 1.8.6 and coverage 7.16.2 (its only dependency) with sha256 values checked against PyPI JSON: cp313 macOS arm64 and cp312 manylinux x86_64 (the ubuntu-24.04 image's python3 3.12.3). Both are added to the OSV-Scanner inventory, and the workflow to the expected set of tests/test_workflow_security_coverage.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
controls/control_expectations.json asks for each control file to be copied alone into an empty directory and run there with the arm's exact command, stored beside the timed runs as <os>-<arm>-controls and <os>-<arm>-crash, and compare.py reads both kinds from one --results directory. The control jobs now do that (the 300 s limit is unchanged), write meta.json without step_seconds (the controls are not timed), upload into the results tree, and the compare job downloads the control artifacts into the same directory. Because the serial arm's command is the whole-suite form the shell-parser and skills-yaml ratchets read, both control jobs carry validate.yml's two npm pin steps unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Integration cross-check of the three W1 units (the oracle's README,
compare.py and controls/control_expectations.json against the workflow):
- compare: pass --expected-sha with the pull request head. Without it
compare.py takes the most common checkout_sha as the frozen corpus,
which the oracle's README calls weaker than asserting it.
- compare: require a clean checkout after every run. The workflow wrote
git-status.txt for each run but nothing read it, so the plan's oracle
element "empty git status --porcelain" was never checked. A new step
(if: always()) treats a missing file or any status text as dirty,
writes checkout-status.json beside result.json and fails the job when
a run is dirty. compare.py is unchanged; the record names the two
parts of the oracle.
- compare: run the oracle's own tests (test_compare.py) on the image's
python3 before compare.py, fail-closed. The README asks the workflow
to run them; the oracle was only tested on CPython 3.13 and 3.14.
- inventory: install the Linux trial lock before ids.py, as the README
asks ("Generate the inventory on the runner after installing the
lock").
Local integration checks (not a hosted run): actionlint exit 0; zizmor
1.30.1 offline on this file, no findings; tests.test_workflow_hardening
and tests.test_workflow_security_coverage OK (2 skips, the same as at
the base); the PyYAML job-parser cross-check OK; the new step on clean,
dirty, missing and empty result trees exits 0, 1, 1, 1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blueprints/convergence-practice/macos-suite-parallelism-20261003/experiment.json freezes plan W1's trial before any hosted run: status planned, decision trial, lane native-agent-engineering, no observations. - Arms: S (python3 -m unittest -v --durations 25) against unittest-parallel 1.8.6 P3, P3F, P3C, P4 on macos-15 and L4, L4F, L4C on ubuntu-24.04, three repeats each, at the draft pull request's head with the trial lock installed in every arm. - Oracle: compare.py's README rules (part 1) plus the workflow's clean-checkout step (part 2), id_mapping.ok required; controls and an os._exit(3) crash control under a 300 s bound in separate jobs. - Decision rule fixed in advance: eligible only with zero mismatches; adopt the fastest eligible arm only if its median <= 0.60 x S median and its max <= 0.75 x S's fastest run; prefer P3F (L4F) within 10%; otherwise reject and keep the record as a failed attempt. - Frozen by sha256: the B1 fix, both locks, the controls and their expectations, the workflow, compare.py, ids.py, the oracle README and test_compare.py. - Sources re-read on 2026-10-03 with GETs: unittest-parallel at bda5d77 (main.py, README), the PyPI wheels, CPython at the v3.13.16 commit cbc944f4 and v3.12.3 f6650f9a, github/docs pages at their commits. validate_convergence.py on the record exits 0. Mutated copies with a drifted workflow byte, an observation or adopt_within_scope exit 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dopted) docs/decisions/2026-10-03-macos-suite-parallelism-trial.md, lane foundation. The B1 fix, the oracle and the trial workflow belong to a draft pull request (PR-T) that is never merged. - What PR-A (validate-macos) and PR-A2 (validate) would change if their runner's verdict is adopt, including the B2 hardening-test edits and the macOS npm-pin environment difference PR-A must decide. - The preregistered checks, a handoff for PR-T: push once, no rebase without a TRIAL_BASE_SHA bump and a refreeze; conflicts with evidence.json or open #615 block pull_request runs. - Alternatives (pytest-xdist, matrix sharding, background steps, timeouts only), overturn conditions, evidence class and SOTA sources. Verified on 2026-10-03 with read-only GETs: pytest 9.1.1 docs list the load_tests protocol as unsupported; the workflow-syntax reference (github/docs c67a9362) documents background/wait/parallel steps, which corrects the plan's note that the page did not show them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix round before the first push (findings F5 and F8 of the reviews of b0b89d1d): - Every actions/upload-artifact step now sets overwrite: false (six steps: inventory, both control jobs, both arm jobs and compare; the brief counted five). action.yml at the pinned 043fb46d (:37-42): "If false, the action will fail if an artifact for the given name already exists." A job re-run therefore cannot replace an earlier attempt's artifact; its upload fails visibly. - Both control recorders now write runtime.json beside meta.json, with the same fields as a timed run, including run_attempt from GITHUB_RUN_ATTEMPT (github/docs contexts.md at 03d2e24b:209: 1 for a run's first attempt, one more per re-run). compare.py reads it from every run directory. - The clean-checkout step now treats only an empty git-status.txt as clean (it used strip()), matching compare.py's part-2 rule. - Comments: the discovery citation names CPython v3.13.16 (main.py :122-126 and :231-234, re-read at cbc944f4), not v3.13.15; the oracle's tested interpreters include 3.12.3; compare.py now applies part 2 itself and the step is a second check; the re-run policy. actionlint exit 0; zizmor --no-config --no-ignores --persona regular --strict-collection . exit 0 (offline mode, no findings); tests.test_workflow_hardening and tests.test_workflow_security_coverage: 84 tests OK (2 skips as at b0b89d1d; the PyYAML cross-check also passes under a scratch PyYAML 6.0.3 interpreter). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix round before the first push (findings F1-F6 and F8 of the reviews of b0b89d1d). Every new test fails against b0b89d1d's oracle (log: the new test_compare.py run in a git-archive export of b0b89d1d) and passes now: 44 tests OK on CPython 3.13.16, 3.12.3 with the lock and bare 3.12.3. F1 speed rule: compare.py compared round(ratio, 4) with 0.60 and 0.75. It now compares exact Fractions of the recorded seconds (the 1.10 unpooled preference too) and rounds only for display; result.json adds median_vs_s_median_exact and max_vs_s_fastest_exact. Probe: S 100000 s x3 with P3 (60004, 60004, 70000) gives reject (15001/25000, was adopt); P3 (50000, 50000, 75004) gives reject (18751/25000, was adopt); P3 (60000, 60000, 75000) gives adopt (3/5 and 3/4). F2 commands: each arm now carries its exact token list (ARMS[os][arm]["command"]); only S may append --durations N (N a positive integer). Probe: 'coverage run -m unittest -v', 'python3 -X dev -m unittest -v' and 'python3 -O -m unittest_parallel -j 3 --level module -v' are each flagged (all three returned [] before), as are timeout 300, python, reordered, -j3, --level=module, the console script and --durations on a parallel arm. F3 id mapping: the inventories may differ only by the prefix tests.test_native_maintenance. on exactly the 29 base ids of the six B1 classes (B1_CLASSES, B1_IDS = 29, counted from ids.py on 56473e4: 6+7+2+2+2+10), as a bijection; build_result gives every OS 'no verdict' when id_mapping.ok is false. Probe: 28 of 29 rewritten, another prefix for one module and another prefix for all give ok false (all ok before); a base with an extra id or 28 B1 ids turns adopt P3 / adopt L4 into no verdict on both OSes. On the integrator's real inventories the rule gives ok (29 ids, 6 classes, one prefix). F4 part 2: read_checkout_status reads every run directory's git-status.txt; missing, 'git status failed' or any content makes the run ineligible (control runs fail their arm). Probe: a dirty, missing, failed or blank-line file on P3 r2 turns adopt into reject; a dirty S run gives no verdict. F5 re-runs: read_run_attempt reads runtime.json run_attempt; above 1 or unrecorded (no file, no key, '', '0') is ineligible. Probe: attempt 2 on P3 r2 turns adopt into reject; on S r3 gives no verdict; on a control run fails the control. F6 make_fixtures.py: --work at or inside the checkout root (HERE.parents[2] and git rev-parse --show-toplevel) or non-empty is refused before anything runs; run directories are created, never deleted. Probe: --work . and --work tests from the checkout root are rejected with 'must be outside the checkout' (before: they passed the guard); a non-empty --work is refused and its content kept. Test harness: Layout now writes the workflow's full run layout (git-status.txt, runtime.json) and the production-shaped base/trial inventories, and adds 29 passing prefixed B1 records to every timed run, so the production id rule runs in every Layout test. Two existing IdMappingTests change visibly: the synthetic B1 set grows from 12 to the real 29-id shape, and test_identical_inventories_pass becomes test_identical_inventories_fail_without_the_b1_rewrite, because the rule now requires the rewrite. The CLI test passes the base inventory instead of the trial one as --inventory-base; its assertions are unchanged. New agreement tests tie ARMS to the workflow's set -- lines and the README table, and check overwrite: false and run_attempt in the workflow. README: exact commands in the table, git-status.txt and runtime.json in the input layout, both oracle parts, the strict id mapping, exact ratios, the re-run policy, the warnings-filter limit, where the control copies live, and that test_compare.py and ids.py run on ubuntu-24.04 only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix round before the first push (finding F7 of the reviews of b0b89d1d). The B1 fix loads the three blueprint modules at import time, so a blueprint that raised while loading aborted a named run such as python3 -m unittest tests.test_native_maintenance tests.test_x with an uncaught traceback and no Ran line. Now the wrapper catches Exception per module, drops the partial sys.modules entry, stops loading (as load_tests did at base) and raises the kept exception from load_tests, which loadTestsFromModule turns into one failing test (CPython loader.py, v3.12.3 :109-118, v3.13.16 :113-122). Probe (scratch copies of the wrapper, fake blueprints, the first or middle one raising RuntimeError at import): the fixed wrapper gives 'Ran 2 tests', FAILED (errors=1), exit 1, the failing test tests.test_native_maintenance (unittest.loader._FailedTest...) and the co-listed test_x ok, identical to the base wrapper, for the named form and for discovery; the b0b89d1d wrapper gave no Ran line on the named form. The failing child is not left in sys.modules. Healthy tree: python3 -m unittest tests.test_native_maintenance -v ran 29 tests OK with the same 29 ids and outcomes as at b0b89d1d; unittest-parallel 1.8.6 -j 2 --level module (1 suite) and --level class (6 suites across 2 workers, pooled and unpooled) give the same 29 records; ids.py on the head lists 9,800 ids, byte-identical to the integrator's head inventory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
experiment.json stays planned. Refrozen hashes: tests/ test_native_maintenance.py, the trial workflow, compare.py, README.md and test_compare.py (the locks, controls and ids.py are unchanged). The quality rule now states what compare.py does: exact commands (c), the run_attempt rule (j), part 2 applied by compare.py to every run directory with the workflow step as a second check, the strict B1 id mapping with no verdict on failure, exact ratios, and the re-run policy (the trial is repeated only by a new synchronize; a job re-run voids the affected arm-run, reported as a failed attempt; every attempt is reported). New limitations: the warnings-filter difference between S and the parallel arms (CPython 3.12.3 main.py:88-92, runner.py:231-234; unittest-parallel main.py:337-343 at bda5d77), the inventory built once on ubuntu-24.04's CPython 3.12.3 with only the lock, id_mapping.ok applied by compare.py, and the re-run residual. Primary sources add upload-artifact action.yml at 043fb46d and CPython 3.13.16 Lib/unittest/main.py. The decision record gains the fix round, the one-observed-record-per- adopting-OS rule (adopt_within_scope needs the qualifying run's scope to equal decision_and_scope.scope, scripts/validate_convergence.py :136-146), the no-verdict triage order (the S runs' git-status.txt first) and PR-A's log-format note on warnings. validate_convergence.py on the record: exit 0, valid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re freeze
compare.py's build_result listed a verdict only for an OS that had run
directories, so a runner whose jobs uploaded nothing vanished from
result.json and the summary. Every OS of ARMS is now listed with its
arms: one without any run directory (timed, control or crash) gets
"no verdict" with the reason "no run directory", and a failed id
mapping gives every OS "no verdict", with or without runs. The
exact-ratio, command, git-status, run_attempt and id-mapping rules are
unchanged.
test_compare.py: the CLI test now expects both OS lines
("ubuntu-24.04: no verdict"); three new tests cover an OS without run
directories, an empty results directory and a failed mapping with one
OS absent. They and the CLI test fail against aab78b12's oracle (3
failures, 1 error); every other test passes there.
A new FixtureFreezeTests test binds the 27 fixture logs to the per-file
sha256 values in fixtures/index.json and refuses unlisted or missing
files, so that freezing index.json freezes the logs.
python3 -m unittest discover -s <oracle dir> -p test_compare.py:
48 tests OK on CPython 3.13.16, bare 3.12.3 and 3.14.4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix round item 7 (a blueprint that raises while it loads must not abort a named run) was shown only by a scratch probe. test_b1_failure_mode.py covers it with a stdlib unittest beside the oracle, not in tests/, so the suite under trial does not change. It copies the wrapper and tests/__init__.py into temporary trees whose three blueprint directories are symbolic links to the checkout's own: - a healthy tree runs the 29 blueprint tests, OK; - a tree whose blueprint raises RuntimeError at import, in each of the three load positions, gives one failing _FailedTest, "Ran 2 tests", "FAILED (errors=1)", exit status 1 and a co-listed module that still runs; - importing the wrapper there leaves no partly loaded blueprint in sys.modules. Against b0b89d1d's wrapper (sha256 57cb42b0...) six broken-blueprint subtests fail: the named run aborts with a traceback and no Ran line, and the import raises. Against the base wrapper (56473e4, sha256 04b6a741...) all four tests pass. Child interpreters run with PYTHONDONTWRITEBYTECODE=1; the trial workflow does not run this file. 4 tests OK on CPython 3.13.16, bare 3.12.3 and 3.14.4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
experiment.json, README.md and the decision record:
- D1: name the three tests that failed on macOS only from 2026-09-25 to
2026-10-02 (test_k4_timing, the SIGQUIT subtest of
test_exit_code_and_signal_propagate, test_node_suite_passes; plan W2's
measurement: 30 macOS-only failures among 1,025 pull-request head
SHAs). All three ids are in the head inventory (ids.py, 9,800 ids).
A mismatch on them still counts under the zero-tolerance rule and is
triaged against that history; the verdict logic is unchanged.
- D2: the unpooled preference ("prefer P3F if within 10% of P3") is
stated as confirmed by the coordinator, for P3F and for L4F.
- D3: make_fixtures.py, fixtures/index.json and
fixtures/inventory-controls.txt join frozen_inputs.evaluation; the
logs are frozen through index.json's per-file sha256 values.
- D4: test_b1_failure_mode.py is frozen, and the record now says that
committed tests cover fix round items 1 to 7, each failing against
b0b89d1d's behaviour.
- D5: every OS is listed in result.json; one without run directories
gets "no verdict" (reason "no run directory").
- D6: the merge fallback adds the B1 check: after changing the base SHA,
run ids.py on the new base and confirm the six B1 classes still hold
exactly 29 ids, else update B1_CLASSES, B1_IDS, their comment and
B1_SHAPE in the same refreeze commit.
Refrozen in experiment.json (status planned): compare.py, README.md and
test_compare.py; added test_b1_failure_mode.py, make_fixtures.py,
fixtures/index.json and fixtures/inventory-controls.txt (15 hashed
entries). validate_convergence.py on the record: valid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The branch was rebased onto origin/main e88d59e (#622, then #639) before the first push, because it conflicted with main in manifests/evidence.json and GitHub runs no pull_request workflow on a conflicting pull request. This commit moves the trial base with it: - the workflow's TRIAL_BASE_SHA and experiment.json's base_revision and inventory command: 56473e4 -> e88d59e; - definitional mentions of the mapping base (README id-mapping paragraph, compare.py's B1 comment, test_compare.py's B1_SHAPE and IdMappingTests comments, the quality rule, the refreeze limitation and the record's handoff) now name e88d59e, keeping 56473e4 as the first base; - historical statements stay at 56473e4 (9,800 ids and the macOS cross-check with run 37073310931, the design agent's base, the units' worktrees), and the README and record now say that no macOS log exists at e88d59e; - a rebase entry in experiment.json, item 7 of the record's second round and a README bullet record the rebase; b0b89d1d and aab78b12 are annotated as local pre-rebase commits, never pushed. R2 checks, local integration evidence: ids.py on a git archive of e88d59e and of the rebased head lists 9,823 ids each, byte-identical under CPython 3.13.16 and the bare 3.12.3; the six B1 classes hold exactly 29 ids at e88d59e (6, 7, 2, 2, 2, 10), and compare.py's check_id_mapping on the two inventories is ok (29 mapped, 6 classes, one prefix, base_only 29, trial_only 29). The three macOS-only flaky ids are in the rebased head inventory. B1_CLASSES, B1_IDS and B1_SHAPE need no change. Refrozen (status planned): the workflow, compare.py, README.md, test_compare.py and test_b1_failure_mode.py. The oracle directory's tests: 52 OK on CPython 3.13.16, bare 3.12.3 and 3.14.4. validate_convergence.py on the record: valid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… first push The final evidence re-review found three low inaccuracies in the preregistration's prose. The known-failures limitation read as a complete list: it now says names were read for 17 of the 30 macOS-only head SHAs (7 in the three named tests, 10 in other tests, 13 not itemized). The rebase provenance claimed only the base SHA moved: it now lists the comment and prose edits of the re-basing commit and says no rule, command or oracle constant changed. The merge-fallback text differed between the record and experiment.json: both now give the same re-basing procedure and say a newer main breaks the id mapping only when it changes test ids. The README hash in experiment.json is refrozen in this commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
One registration built from the manifest of the trial base e88d59e: the 43 files the branch adds, the 3 existing files it changes, and the convergence record entry for the suite-parallelism preregistration. It is the branch's last commit, so a later rebase takes main's manifest and registers again (docs/lanes.md). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
seathatflowsinourveins
added
the
lane:foundation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
5 tasks done
Owner
Author
|
Closing without merging, as planned. The trial ran once (run 37109532421, head 1e4bb5a). Linux gave the official verdict: reject, because every unittest-parallel run crashed pickling the suite and none was eligible. macOS has no verdict: its 20 jobs never started and the run was cancelled. The decision record, the sanitized receipt and the root causes are in #652. This branch is kept for reference; please do not push to it, because every push re-runs the whole trial. |
seathatflowsinourveins
added a commit
that referenced
this pull request
Oct 3, 2026
…en inputs The observed record's frozen_inputs pointed only at the preregistration copy and listed result-trimmed.json, the oracle's output, as an evaluation input, so scripts/validate_convergence.py checked no byte of the preregistered oracle, controls, B1 fix or lock files (delta review of #652, D6). - frozen/ holds byte copies, stored as .txt, of the 15 files the preregistration froze, written with git show at the trial head 1e4bb5a (draft #646, closed unmerged): 15 of 15 byte-equal to their originals and to their preregistered sha256, 264,054 bytes, all passing scripts/validate.py --scan-file. No lock copy's name starts with requirements, so tests/test_osv_lockfile_coverage.py does not treat it as a lockfile. - experiment.json, regenerated from the raw run data: frozen_inputs cite the copies in the preregistration's role lists and order (3 sources, 3 inputs, 9 evaluation) plus the preregistration copy at the end of sources; result-trimmed.json stays an observation artifact only. A new limitation lists the other departures from the plan (command 11, one scope text per run, the job and step timestamps kept in the receipt's data.jobs), and each control observation's scope states its controls file's step seconds (0, 0, 1 and 0 s under S, L4, L4F and L4C; verifier note). rollback records #646 as closed. - README: a frozen/ section says what each copy is, that the originals live on the closed trial branch and that the hash equality with the preregistration is the point. - Receipt: data.retained_files lists all 28 files under the artifacts directory with their new hashes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins
added a commit
that referenced
this pull request
Oct 3, 2026
…ow each claim was read Delta review of #652 (D1 to D5) and two corrections found while verifying it. - D1: a "Sources missed before the trial" subsection names four sources that the preregistration and its decision record do not record reading (plan W1 and its Codex review, kept outside the repository, were not re-read): unittest-parallel's issue tracker (25 issues; issue 27, filed 2025-08-24, is titled with this trial's exact exception, and issue 22 reports a pickling failure of the family of root cause (b)), CPython's tracker (python/cpython#83438, whose only reply calls the pickling error expected: "contextvars are not compatible with multiprocessing"; no issue on pickling IsolatedAsyncioTestCase was found), this suite's own async and type()-built test modules, and GitHub's changelog. Matrix sharding, which plan W1 compared and the preregistration rejected, is listed with its blockers re-checked at the base (.github/main-ruleset.json:35 and :41, docs/github-automation.md:26-28, tests/test_workflow_hardening.py:1249-1257, GitHub's five-job macOS limit at github/docs 2bd66de8 limits.md:64-69 and the trial's measured macOS queue) and noted as outside the one-job task; a seed and a critic question follow. - D2: stestr's reader hang is VERIFIED at stestr/output.py:160-161 (4.2.1), and concurrencytest's repeated class fixtures under its default round-robin partitioning at concurrencytest.py:158-160 (0.1.11); only the lead's module-fixture wording stays LEAD-ONLY. - D3: the critique, the Evidence class and the SOTA sources say how each class of claim was read: repository files with GET requests at the release tag or commit; repository records, PyPI JSON, the changelog, the GNU announcement and the issue trackers with unpinned GET requests; the GNU xargs statuses from the installed manual page; the unittest.mock count and the matrix-sharding blockers locally. - D4: the second trial's local preflight is attributed to the coordinator and marked not verified here. - D5: the departures of the observed record's shape list all seven (command 11, the control observations, the skipped macOS observations, one scope text per run, the frozen inputs at other paths, the timestamps in the receipt, null token usage), with runner time in each executed observation's scope text. - Corrections: 106 of the 234 test modules import unittest.mock (100 at module level, 6 inside a test), not 100; and #646 was closed unmerged at 11:05:52Z on 2026-10-03, after this record's pull request opened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins
added a commit
that referenced
this pull request
Oct 4, 2026
…ed test jobs (#652) * Record the failed suite-parallelism trial: decision, receipt and sanitized artifacts Hosted run 37109532421 of draft PR #646 (head 1e4bb5a) failed its preregistered rule. On ubuntu-24.04 the serial arm passed 3 of 3 runs (1,688, 1,659 and 1,186 s); all nine unittest-parallel 1.8.6 runs (L4, L4F, L4C) ended with TypeError: cannot pickle '_contextvars.Context' object from pool.map, with 199 (module level) or 82 (class level) ids never run. The oracle gave ubuntu-24.04 reject and macos-15 no verdict; the coordinator cancelled the run after the Linux jobs finished and before any macOS job received a runner. Root causes, reproduced locally on CPython 3.12.3 and 3.13.16: the two IsolatedAsyncioTestCase classes hold a contextvars.Context, and four classes built by type() in tests/test_adoption_version_probes.py are not importable under their qualified names. Nothing is adopted. The receipt keeps the run and job records, per-run data, missing ids per class, the non-missing mismatches, the crash signature, the local reproductions and the upstream reads with pins; the artifacts keep sanitized excerpts, a trimmed oracle output and the hashes of the originals, which expire on 2026-11-02. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Log two anti-patterns from the suite-parallelism trial Two dated 2026-10-03 rows at the top of the anti-pattern log: a hosted whole-suite trial preregistered after a preflight of one 29-test module (the whole suite never ran locally under the candidate runner), and a trial workflow whose pull_request paths filter names its own file, so the record path has to be decided before the first push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct the trial outcome record, receipt and artifacts after evidence review Independent review of the records pull request (one medium, seven low findings) checked against the raw run artifacts; the decision is unchanged. - Serial outcome of the ids that never ran (F1), re-derived from the three S logs and equal in all three: module level 90 ok and 109 skipped, class level 8 ok and 74 skipped. All 66 IsolatedAsyncioTestCase tests and the 8 bash32 tests were skipped on ubuntu-24.04, so the async classes break unittest-parallel at pickling although none of their tests runs there (a class-level skip applies only in TestCase.run; CPython v3.12.3 case.py:159-161 and :612-617, loader.py:94). Per-class counts are in the receipt's data.missing_ids and the trimmed result; the serial seconds sentence now names the two missing ids in S-r3's durations table. - Ids with a result record versus records (F2): L4C-r3 has 9,743 records for 9,741 ids (three K4 subtest FAIL records). - n=3 limitation (F3), controls labelled as synthetic fixtures executed on the hosted runner and reproduction scope per interpreter and module (F4), usage statement with 11,524 ubuntu-24.04 job-seconds and unknown session model usage (F6), runner image recorded only by the 16 arm and control jobs (F7), and the two macOS phrasings (F8). - retained_files hashes updated for README.md and result-trimmed.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Narrow the suite-parallelism anti-pattern row to what was shown The whole suite was not run locally under the candidate runner, so the row now says a local whole-suite run would very likely have shown the crash, inferred from the per-module reproductions. It attributes the TypeError to the two IsolatedAsyncioTestCase modules and the masked PicklingError to the dynamic-class module (first-exception rule, Lib/multiprocessing/pool.py:822 at CPython v3.12.3). The unmeasured "in seconds" is dropped. Row format and position unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * State that the async tests were skipped because HAS_SDK was false in the trial environment, not because of the operating system Delta review finding on the outcome record: the skip is skipUnless(HAS_SDK), true only when both alpaca and requests import; the trial arms installed neither (requirements-ci.txt installs neither), so the 66 tests were skipped in every S run in that environment. Reworded the two sentences that tied the skip to ubuntu-24.04. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Add the validated observed convergence record and the completeness critique of the suite-parallelism trial The trial's convergence record is now evidence/artifacts/suite-parallelism-trial-20261003/experiment.json (status observed, decision reject), generated from the raw run directories, the jobs API and result.json. It has 36 observations: the 12 timed ubuntu-24.04 runs, the 4 ubuntu-24.04 control jobs and the 20 macos-15 jobs that never received a runner (skipped). It carries the preregistered metrics, commands, lane, roles and base revision unchanged; its frozen_inputs cite the retained preregistration copy, whose 15 frozen hashes were re-checked at 1e4bb5a. Token usage is null (unknown), not the preregistered observed zeros, and runner time is in each observation's scope text; both departures are in its limitations and the decision record. The decision record, the receipt and the artifacts README cite it as the convergence record; the receipt keeps the detailed data and the claim text, and no longer says it replaces the observed record. The decision record's new "Completeness critique" section names the missed modality, nine candidate classes with each relayed claim marked VERIFIED or LEAD-ONLY against its pinned upstream source, and seeds for the next ci-supply-chain landscape sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Log the two review findings of #652 as anti-patterns A free-form receipt in place of the preregistered observed convergence record, and a substantive trial closed without its completeness critique. Both rows sit above this pull request's two earlier 2026-10-03 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the anti-pattern row the decision record cites The table now has four rows dated 2026-10-03, so the two references to "the anti-pattern row of 2026-10-03" name the row by its title. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cite byte copies of the 15 frozen files as the observed record's frozen inputs The observed record's frozen_inputs pointed only at the preregistration copy and listed result-trimmed.json, the oracle's output, as an evaluation input, so scripts/validate_convergence.py checked no byte of the preregistered oracle, controls, B1 fix or lock files (delta review of #652, D6). - frozen/ holds byte copies, stored as .txt, of the 15 files the preregistration froze, written with git show at the trial head 1e4bb5a (draft #646, closed unmerged): 15 of 15 byte-equal to their originals and to their preregistered sha256, 264,054 bytes, all passing scripts/validate.py --scan-file. No lock copy's name starts with requirements, so tests/test_osv_lockfile_coverage.py does not treat it as a lockfile. - experiment.json, regenerated from the raw run data: frozen_inputs cite the copies in the preregistration's role lists and order (3 sources, 3 inputs, 9 evaluation) plus the preregistration copy at the end of sources; result-trimmed.json stays an observation artifact only. A new limitation lists the other departures from the plan (command 11, one scope text per run, the job and step timestamps kept in the receipt's data.jobs), and each control observation's scope states its controls file's step seconds (0, 0, 1 and 0 s under S, L4, L4F and L4C; verifier note). rollback records #646 as closed. - README: a frozen/ section says what each copy is, that the originals live on the closed trial branch and that the hash equality with the preregistration is the point. - Receipt: data.retained_files lists all 28 files under the artifacts directory with their new hashes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Complete the critique's missed sources and matrix sharding; qualify how each claim was read Delta review of #652 (D1 to D5) and two corrections found while verifying it. - D1: a "Sources missed before the trial" subsection names four sources that the preregistration and its decision record do not record reading (plan W1 and its Codex review, kept outside the repository, were not re-read): unittest-parallel's issue tracker (25 issues; issue 27, filed 2025-08-24, is titled with this trial's exact exception, and issue 22 reports a pickling failure of the family of root cause (b)), CPython's tracker (python/cpython#83438, whose only reply calls the pickling error expected: "contextvars are not compatible with multiprocessing"; no issue on pickling IsolatedAsyncioTestCase was found), this suite's own async and type()-built test modules, and GitHub's changelog. Matrix sharding, which plan W1 compared and the preregistration rejected, is listed with its blockers re-checked at the base (.github/main-ruleset.json:35 and :41, docs/github-automation.md:26-28, tests/test_workflow_hardening.py:1249-1257, GitHub's five-job macOS limit at github/docs 2bd66de8 limits.md:64-69 and the trial's measured macOS queue) and noted as outside the one-job task; a seed and a critic question follow. - D2: stestr's reader hang is VERIFIED at stestr/output.py:160-161 (4.2.1), and concurrencytest's repeated class fixtures under its default round-robin partitioning at concurrencytest.py:158-160 (0.1.11); only the lead's module-fixture wording stays LEAD-ONLY. - D3: the critique, the Evidence class and the SOTA sources say how each class of claim was read: repository files with GET requests at the release tag or commit; repository records, PyPI JSON, the changelog, the GNU announcement and the issue trackers with unpinned GET requests; the GNU xargs statuses from the installed manual page; the unittest.mock count and the matrix-sharding blockers locally. - D4: the second trial's local preflight is attributed to the coordinator and marked not verified here. - D5: the departures of the observed record's shape list all seven (command 11, the control observations, the skipped macOS observations, one scope text per run, the frozen inputs at other paths, the timestamps in the receipt, null token usage), with runner time in each executed observation's scope text. - Corrections: 106 of the 234 test modules import unittest.mock (100 at module level, 6 inside a test), not 100; and #646 was closed unmerged at 11:05:52Z on 2026-10-03, after this record's pull request opened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rename the two lock copies so no scanner pattern matches them, and apply the last delta-review wording fixes The two requirements-style lock copies (byte copies, same sha256) are renamed trial-ci-suite-<os>-pins.frozen: a .txt name containing requirements is what Python dependency scanners and the OSV inventory test match, so the .txt suffix alone did not keep them out. README and experiment.json now say what each name rule does and that the dependency-graph effect is expected, not verified. Also: the sources-missed list now includes the pytest and coverage reads the preregistration recorded, matrix sharding's attribution names the concurrent Adoption runs, the receipt separates pinned source reads from unpinned GETs, the experiment record hedges the absence claim, and the anti-pattern log gains the frozen-inputs row. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Register the trial outcome files and the observed convergence record (hot-file protocol; last commit) One registration on main's manifest at 4ced292: the decision record, the receipt, the artifacts including the 15 frozen byte copies (the two lock copies renamed so no scanner pattern matches them), docs/harness-defaults.md (five rows), and the observed record in convergence_records. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Receipt: refresh the README.md and experiment.json retained_files entries to the bytes after a986c4c (review 652 P1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hot-file protocol: reset manifests/evidence.json to the merge base before the final hot-file commit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reapply this branch's manifests/evidence.json edits on the merge base (hot-file protocol: every hot-file edit in the last commit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: seathatflowsinourveins <seathatflowsinourveins@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Scout <scout@local>
seathatflowsinourveins
deleted the
foundation/macos-suite-parallelism-trial-20261003
branch
October 8, 2026 17:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, never merged. This pull request exists to run a preregistered hosted trial. Its own required checks are informational; nothing in it reaches
mainthrough it. If the trial passes its preregistered rule, adoption lands as separate pull requests (PR-A forvalidate-macos, PR-A2 forvalidate).Scope
validate-macoson macos-15 andvalidateon ubuntu-24.04). The trial workflow runs only on this PR (itspull_requestpaths filter names its own file), and its oracle decides adoption by a rule fixed in advance. The PR also carries the one change the suite needs to run under spawned workers: the wrappertests/test_native_maintenance.pyloads its six blueprint TestCase classes when it is imported (as registered child modules) instead of insideload_tests, so a worker can unpickle them; the 29 tests and their outcomes are unchanged and only those six classes' ids gain the prefixtests.test_native_maintenance..e88d59e4b4670ca19211a9ab2adb45e1a28901c7(the trial base, the workflow'sTRIAL_BASE_SHA).mainhas moved on since (Layer consensus of 2026-10-02 in the manifest, plan and handbook (89 slots), and the WSL scope of the two-host decision #620, Keep the root directory as one PATH component; refuse a host HOME of /; six anti-pattern log rows #619). The branch is not rebased again because it merges cleanly and the workflow checks out the pull request's head commit, so the id inventory still maps the base onto the head; the decision record's handoff says what to do if a rebase becomes necessary.lane:foundation.github/workflows/macos-suite-parallel-trial.yml(new),.github/requirements-ci-suite-macos.txtand.github/requirements-ci-suite-linux.txt(new, hash-locked),.github/osv-scanner-lockfiles.json(two entries),tests/test_native_maintenance.py(the B1 fix),tests/test_workflow_security_coverage.py(expected-workflow set),blueprints/convergence-practice/macos-suite-parallelism-20261003/(oracle, controls, fixtures, README and theplannedpreregistrationexperiment.json),docs/decisions/2026-10-03-macos-suite-parallelism-trial.md, andmanifests/evidence.json(registration only, the branch's last commit, per the hot-file protocol indocs/lanes.md).SOTA sources
bda5d77dc1a2fa2df90f5f5a7de297ea375e345c(upstream publishes no tags; MIT):src/unittest_parallel/main.pyand the README, sections "Parallelism Level" and "Process and Thread Pools"; PyPI 1.8.6 and coverage 7.16.2 (its only hard dependency), both installed from hash-locked files with--only-binary=:all: --require-hashes.cbc944f4bc59639a444dd971c737788ba2283a91):Lib/unittest/loader.py,runner.py,result.py,main.pyandLib/pickle.py; CPython 3.12.3Lib/unittest.strategy.max-parallel, steptimeout-minutes) with its three-dot diff rule, contexts (github.run_attempt), events that trigger workflows (pull_requestGITHUB_SHA, merge conflicts) and the workflow jobs REST API.043fb46d1a93c77aae656e7c1c64a875d1fc6a0a:action.ymllines 37-42 (overwritedefaults tofalse, and then the action fails if an artifact of that name exists).cf470ec0bf7eb89cd97dd56df4859eae5db46447):doc/en/how-to/unittest.rst, the reason pytest-xdist is rejected here (pytest does not runload_tests, which would drop six classes and change the id set).docs/lanes.md(hot-file protocol),docs/acceptance-evidence-policy.md,docs/github-automation.md,docs/decisions/2026-10-02-github-automation-practice.md,tests/test_workflow_hardening.py(the rules the workflow must satisfy).Evidence-class table
main.py(worker count, header line, spawn pool, exit status)source_review: read-only primary-source reads, 2026-10-03experiment.jsonlocal_integrationpython3 -m unittest tests.test_native_maintenance -v(29 tests OK); the same 29 records underunittest-parallel -j 2, pooled and unpooled, module and class levelcompare.py) implements exactly the preregistered rule: exact fraction ratios, exact command match, strict id mapping, clean-checkout and run-attempt rules, every OS listedlocal_integrationandsynthetic(fixtures from real local runs, mutated logs)test_compare.py48,test_b1_failure_mode.py4) on CPython 3.13.16, 3.12.3 and 3.14.4; each fix round's behavioural tests fail against the previous head's behaviour (the two healthy-tree sanity tests oftest_b1_failure_mode.pypass there, as they should)result.jsonof thecomparejob; sanitized receipts are committed afterwardsLocal commands run
Exit status 0 unless stated. Run by the coordinator on this head,
1e4bb5ab:Run by the builder agent on the previous head and re-run by an independent verifier agent (39 of its 42 claims confirmed; the other three were the commit-trailer wording of the brief,
mainmoving during verification, and the absence of a system Python 3.12, for which the uv-managed 3.12.3 and the host's 3.14.4 were used):python3 -m unittest tests.test_workflow_hardening tests.test_workflow_security_coverage(84 tests, skipped=2),actionlint,zizmor --no-config --no-ignores --persona regular --strict-collection .(offline: its online audits did not run, so this PR's ownvalidatezizmor step is the first full run), and the 29-test wrapper run serially and under unittest-parallel. No whole-suite parallel run has happened yet; the hosted arms are the first.Decision record
docs/decisions/2026-10-03-macos-suite-parallelism-trial.md(alternatives, the preregistered decision rule, overturn conditions, handoff for this PR). Preregistration:blueprints/convergence-practice/macos-suite-parallelism-20261003/experiment.json, statusplanned, frozen by hash. The rule, fixed before any hosted run: an arm is eligible only with zero oracle mismatches; the fastest eligible arm is adopted only if its median is at most 0.60 times the serial median and its slowest run at most 0.75 times the serial fastest run; the unpooled arm is preferred when within 10%; otherwise the trial is recorded as a failed attempt.Host evidence
Not applicable: no files under
evidence/hosts/change.Checklist
tests/test_workflow_hardening.py.permissions: contents: read.Notes for the reviewer:
run_attempt. To repeat the trial, push a new head commit.max-parallel), so other pull requests'validate-macosjobs can queue for a while during the trial.comparejob writesresult.jsonand a step summary. A "no verdict" is triaged by reading the serial runs'checkout-status.jsonfirst. The preregistration is not changed after the first hosted run.manifests/evidence.jsonholds only this branch's registration, as the last commit; ifmainconflicts there, takemain's copy and register again (docs/lanes.md).🤖 Generated with Claude Code