Skip to content

ci(OMN-19711): chain-replacement parity tool - #2949

Closed
jonahgabriel wants to merge 7 commits into
devfrom
jonah/omn-19711-chain-replacement-parity-tool
Closed

jonahgabriel wants to merge 7 commits into
devfrom
jonah/omn-19711-chain-replacement-parity-tool

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Cites OMN-19711 (plan task B2 of knowledge-base-internal beta/plans/2026-09-25-golden-chain-event-tests-for-pr-validation-plan.md, section 3.3, parent OMN-19680). Part B was approved by the operator ruling of 2026-09-26 (ledger RULING, lane validation-plans-83-ruling, ticket OMN-19680).

What

scripts/ci/chain_replacement_parity.py turns the coverage-parity rule for deleting test cases T in favour of chain cases C, which cover handler modules H, into a receipt. Each of P1 to P5 is PASS, FAIL or MISSING:

check rule
P1 mutation every mutant of H killed before the deletion is killed after it, compared by mutant id with zero escapes. Zero mutants counts as MISSING
P2 coverage per file of H, covered lines and branch arcs after the deletion are supersets of those before it (set comparison, zero tolerance)
P3 walker every walker ERROR path T asserted is asserted by C
P4 determinism 20 runs, 20 distinct seeds, all pass, identical event lists per case
P5 cost C's summed case time is at most T's

Any FAIL or MISSING, and any unreadable or invalid input, exits 1.

  • evaluate is pure and unit-tested. measure is lab-only: it runs coverage and mutmut 3 over H in scratch copies and never edits the worktree's config. It runs C under 20 seeds, each setting PYTHONHASHSEED and a seeded case order; pytest-randomly is not a dependency here, so this stands in for it. It also records junit times. The default comparison suite is T plus C, which is stricter than a full-suite comparison. --baseline adds paths to both sides.
  • mutmut 3 goes in a new parity dependency group, so CI's dev sync is unchanged. The lock adds only mutmut, libcst, pyyaml-ft and setproctitle.

Acceptance evidence

  • AC1 and AC2: uv run pytest tests/unit/scripts/ci/test_chain_replacement_parity.py -q gives 25 passed. A planted lost branch fails P2. A planted surviving mutant fails P1. A no-op deletion passes. test_missing_input_fails_closed covers each measurement set to None, plus an absent inputs file and a malformed one, and all exit 1. ruff and mypy are clean.

Lab step (h201, this branch merged locally with OMN-19710's head d6572dad)

measure over the four pilot handlers (handler_delegation_workflow.py, handler_delegation_routing.py, handler_compliance_loop.py, handler_routing_intent.py). T was the 21 cases of test_omn17802_v2_terminal_route_boundary.py and C was OMN-19710's 2 real delegation chains. The run took 433 s of wall time and exited 1, as it should on this pair:

  • P1 FAIL: 4,685 mutants (the plan's 2,000 to 4,000 was an unmeasured estimate). Of the 1,182 killed by T plus C, only 868 are still killed by C alone, so 314 would escape.
  • P2 FAIL: 2 lines and 2 branch arcs lost.
  • P3 PASS, P4 PASS (20 of 20 seeds), P5 PASS (0.371 s against 1.605 s).

The first lab run found two defects, which are fixed in the second commit. (1) mutmut copied only the named handler files, so the tests imported the original code and every mutant read "no tests". (2) junit ids dropped the test class, so the time of every class-based case read MISSING.

Not runtime-affecting: a lab-only CI tool plus a dependency group that CI does not sync.

Out of scope

The deletion gate (OMN-19712, stacked on this branch) and the pilot (OMN-19713). Nothing is armed or merged by this lane. It is handed to the landing lane by a ledger MSG.

Evidence-Ticket: OMN-19711
Evidence-Source: OCC#11459

scripts/ci/chain_replacement_parity.py turns the coverage-parity rule for
deleting tests in favour of chain cases into a receipt with P1 to P5
(mutation kill set by mutant id, line and branch-arc set supersets per
handler file, walker ERROR paths, 20-seed determinism, cost), each PASS,
FAIL or MISSING. Any FAIL or MISSING, and any unreadable or invalid input,
exits 1; zero mutants is MISSING, never a vacuous pass.

The verdict layer (evaluate) is pure and unit-tested with planted inputs.
The measure subcommand is lab-only: it runs coverage and mutmut over the
handler modules in scratch copies, runs the chain cases under 20 seeds
(PYTHONHASHSEED plus a seeded case order, since pytest-randomly is not a
dependency here), and records junit times. Its default comparison suite is
T plus C, which is stricter than a full-suite comparison; --baseline adds
paths to both sides.

mutmut 3 is added in a new parity dependency group, so the dev sync that
CI runs is unchanged.
…classes

Two defects found by the first h201 measure run over the four delegation
handlers (verdict FAIL, exit 1, as it should on the planted pair):

- naming the handler files in mutmut's paths_to_mutate copied only those
  files into mutants/, so mutants/src/omnimarket was a namespace portion
  that lost to the editable install; the tests imported the original code
  and mutmut stopped with no test case for any mutant (P1 MISSING). The
  scratch config now sets source_paths to src, only_mutate to H, and turns
  off git change detection in the .git-less scratch copy.
- junit classname tests.x.test_y.TestZ was mapped to tests/x/test_y/TestZ.py,
  so all 21 class-based deleted cases had no time (P5 MISSING). The mapping
  now resolves the longest prefix naming a .py file and keeps the classes.
@github-actions

Copy link
Copy Markdown

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule Description
ARCH-TOPIC-001 No hardcoded Kafka topic strings in handler code
ARCH-DI-001 No event_bus=None bypass
ARCH-DI-002 No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003 No reinvented DI containers
ARCH-TOPIC-002 contract.yaml topics follow `onex.{cmd

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

measure passes --max-children to mutmut run, default 8. The first h201 run
used every core and took the host's load from about 10 to 23.
@jonahgabriel
jonahgabriel enabled auto-merge (squash) September 26, 2026 12:02
onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 26, 2026
…nimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240624462
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 5
Models succeeded: qwen3-review,gpt-oss-review


Gate semantics

Verdict Meaning Blocks merge?
passed >=2 models succeeded, no critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Fewer than 2 models succeeded (infra unavailable/timeout) Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 26, 2026
…nimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240724677
The pilot run with the whole delegation flow as baseline stopped at
'failed to collect stats': tests/test_delegation_env_scanner.py imports a
module from scripts/ through a sys.path insert, and mutants/ held only src
and tests. The scratch config now lists every other top-level entry in
also_copy.
onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 26, 2026
…nimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36242699101
mutmut rewrites every function of H into a trampoline, so a test that reads
a handler's source text fails in mutants/ before any mutant is applied and
mutmut stops at 'failed to collect stats' (the pilot run on h201 hit
test_every_terminal_emission_inputs_names_both_fields). measure now
deselects the tests a stats run reports FAILED, retries at most ten times,
uses the same set on both sides, and records it as
mutation_unmeasurable_tests. A deletion that removes one of those tests is
P1 MISSING: its kills were never measured.
onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 26, 2026
…nimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36243900762
mutmut's stats run stops at the first failing test, so setting tests aside
one per retry took about five minutes each on the pilot (four retries, one
parametrized source-reading test per retry). After a stats failure the tool
now runs the same selection once inside mutants/ with no mutant active
(MUTANT_UNDER_TEST empty, so every trampoline calls the original function)
and sets aside every test that fails there.
onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 26, 2026
…nimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36245125508
The stats file lives in a temporary scratch copy and was deleted with it.
It records which tests cover each mutated function, which is the evidence
needed to shrink a refused deletion set to an admissible one.
onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 26, 2026
…nimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36246934084
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 26, 2026
…imarket#2949 (#11459)

* evidence(OMN-19711): author OCC companion for OmniNode-ai/omnimarket#2949

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 11283f4e5cc93127b9c33b0cb4d5d751c949aeec.

* evidence(OMN-19711): self-bind OCC#11459 + rebind contract_sha256

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240624462

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36240724677

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36242699101

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36243900762

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36245125508

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2949

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/36246934084

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
Co-authored-by: onexbot-occ-writer[bot] <307849072+onexbot-occ-writer[bot]@users.noreply.github.com>
@jonahgabriel

Copy link
Copy Markdown
Contributor Author

Closing as superseded. omnimarket#2950 was stacked on this branch and squash-merged to dev as befa9cc, carrying every commit of this PR (head a350f3d).

Verification from the canonical clone: git diff origin/dev a350f3d4 -- scripts/ci/chain_replacement_parity.py tests/unit/scripts/ci/test_chain_replacement_parity.py is empty, and the only remaining differences in pyproject.toml and uv.lock are newer dev content (the 0.4.234 version bump and entries from other merged PRs); the parity dependency group this PR added is on dev. Nothing is left to land.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant