Repository navigation
ci(OMN-19711): chain-replacement parity tool - #2949
jonahgabriel wants to merge 7 commits into
Conversation
scripts/ci/chain_replacement_parity.py turns the coverage-parity rule for deleting tests in favour of chain cases into a receipt with P1 to P5 (mutation kill set by mutant id, line and branch-arc set supersets per handler file, walker ERROR paths, 20-seed determinism, cost), each PASS, FAIL or MISSING. Any FAIL or MISSING, and any unreadable or invalid input, exits 1; zero mutants is MISSING, never a vacuous pass. The verdict layer (evaluate) is pure and unit-tested with planted inputs. The measure subcommand is lab-only: it runs coverage and mutmut over the handler modules in scratch copies, runs the chain cases under 20 seeds (PYTHONHASHSEED plus a seeded case order, since pytest-randomly is not a dependency here), and records junit times. Its default comparison suite is T plus C, which is stricter than a full-suite comparison; --baseline adds paths to both sides. mutmut 3 is added in a new parity dependency group, so the dev sync that CI runs is unchanged.
…classes Two defects found by the first h201 measure run over the four delegation handlers (verdict FAIL, exit 1, as it should on the planted pair): - naming the handler files in mutmut's paths_to_mutate copied only those files into mutants/, so mutants/src/omnimarket was a namespace portion that lost to the editable install; the tests imported the original code and mutmut stopped with no test case for any mutant (P1 MISSING). The scratch config now sets source_paths to src, only_mutate to H, and turns off git change detection in the .git-less scratch copy. - junit classname tests.x.test_y.TestZ was mapped to tests/x/test_y/TestZ.py, so all 21 class-based deleted cases had no time (P5 MISSING). The mapping now resolves the longest prefix naming a .py file and keeps the classes.
✅ Architectural Review — PASSEDErrors: 0 What this checks
No architectural violations found. Static architectural lint — no model inference (OMN-14176). |
measure passes --max-children to mutmut run, default 8. The first h201 run used every core and took the host's load from about 10 to 23.
…nimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240624462
✅ Hostile Reviewer — PASSEDBlocking findings (critical/error): 0 Gate semantics
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524) |
…nimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240724677
The pilot run with the whole delegation flow as baseline stopped at 'failed to collect stats': tests/test_delegation_env_scanner.py imports a module from scripts/ through a sys.path insert, and mutants/ held only src and tests. The scratch config now lists every other top-level entry in also_copy.
…nimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36242699101
mutmut rewrites every function of H into a trampoline, so a test that reads a handler's source text fails in mutants/ before any mutant is applied and mutmut stops at 'failed to collect stats' (the pilot run on h201 hit test_every_terminal_emission_inputs_names_both_fields). measure now deselects the tests a stats run reports FAILED, retries at most ten times, uses the same set on both sides, and records it as mutation_unmeasurable_tests. A deletion that removes one of those tests is P1 MISSING: its kills were never measured.
…nimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36243900762
mutmut's stats run stops at the first failing test, so setting tests aside one per retry took about five minutes each on the pilot (four retries, one parametrized source-reading test per retry). After a stats failure the tool now runs the same selection once inside mutants/ with no mutant active (MUTANT_UNDER_TEST empty, so every trampoline calls the original function) and sets aside every test that fails there.
…nimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36245125508
The stats file lives in a temporary scratch copy and was deleted with it. It records which tests cover each mutated function, which is the evidence needed to shrink a refused deletion set to an admissible one.
…nimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36246934084
…imarket#2949 (#11459) * evidence(OMN-19711): author OCC companion for OmniNode-ai/omnimarket#2949 OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 11283f4e5cc93127b9c33b0cb4d5d751c949aeec. * evidence(OMN-19711): self-bind OCC#11459 + rebind contract_sha256 * evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240624462 * evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240724677 * evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36242699101 * evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36243900762 * evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36245125508 * evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949 The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited. Ticket: OMN-16859 (AC3b) Onex-Workflow: OCC Receipt Runner Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36246934084 --------- Co-authored-by: omnimarket-bot <bot@omninode.ai> Co-authored-by: onexbot-occ-writer[bot] <307849072+onexbot-occ-writer[bot]@users.noreply.github.com>
|
Closing as superseded. omnimarket#2950 was stacked on this branch and squash-merged to dev as befa9cc, carrying every commit of this PR (head a350f3d). Verification from the canonical clone: |
Pull request was closed
Cites OMN-19711 (plan task B2 of knowledge-base-internal
beta/plans/2026-09-25-golden-chain-event-tests-for-pr-validation-plan.md, section 3.3, parent OMN-19680). Part B was approved by the operator ruling of 2026-09-26 (ledger RULING, lane validation-plans-83-ruling, ticket OMN-19680).What
scripts/ci/chain_replacement_parity.pyturns the coverage-parity rule for deleting test cases T in favour of chain cases C, which cover handler modules H, into a receipt. Each of P1 to P5 is PASS, FAIL or MISSING:Any FAIL or MISSING, and any unreadable or invalid input, exits 1.
evaluateis pure and unit-tested.measureis lab-only: it runs coverage and mutmut 3 over H in scratch copies and never edits the worktree's config. It runs C under 20 seeds, each settingPYTHONHASHSEEDand a seeded case order; pytest-randomly is not a dependency here, so this stands in for it. It also records junit times. The default comparison suite is T plus C, which is stricter than a full-suite comparison.--baselineadds paths to both sides.paritydependency group, so CI's dev sync is unchanged. The lock adds only mutmut, libcst, pyyaml-ft and setproctitle.Acceptance evidence
uv run pytest tests/unit/scripts/ci/test_chain_replacement_parity.py -qgives 25 passed. A planted lost branch fails P2. A planted surviving mutant fails P1. A no-op deletion passes.test_missing_input_fails_closedcovers each measurement set to None, plus an absent inputs file and a malformed one, and all exit 1.ruffandmypyare clean.Lab step (h201, this branch merged locally with OMN-19710's head
d6572dad)measureover the four pilot handlers (handler_delegation_workflow.py,handler_delegation_routing.py,handler_compliance_loop.py,handler_routing_intent.py). T was the 21 cases oftest_omn17802_v2_terminal_route_boundary.pyand C was OMN-19710's 2 real delegation chains. The run took 433 s of wall time and exited 1, as it should on this pair:The first lab run found two defects, which are fixed in the second commit. (1) mutmut copied only the named handler files, so the tests imported the original code and every mutant read "no tests". (2) junit ids dropped the test class, so the time of every class-based case read MISSING.
Not runtime-affecting: a lab-only CI tool plus a dependency group that CI does not sync.
Out of scope
The deletion gate (OMN-19712, stacked on this branch) and the pilot (OMN-19713). Nothing is armed or merged by this lane. It is handed to the landing lane by a ledger MSG.
Evidence-Ticket: OMN-19711
Evidence-Source: OCC#11459