Skip to content

fix(OMN-16787): a worktree that could not be materialised is not "no contract" - #2169

Merged
jonahgabriel merged 1 commit into
devfrom
jonah/omn-16787-occ-worktree-fail-closed
Aug 27, 2026
Merged

jonahgabriel merged 1 commit into
devfrom
jonah/omn-16787-occ-worktree-fail-closed

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

The defect

EvidenceCollector._materialize_occ_dev_worktree has two independent failure classes. OMN-15454 closed one of them.

Failure class Return Behaviour before this PR
git fetch failed (dev_root, wt, FETCH_FAILED, sha) fail-closed — refuses, emits OCC_REF_REFRESH_FAILED, override via DOD_VERIFY_ALLOW_STALE_OCC_REF
git worktree add failed / timed out (None, None, refresh_outcome, None) fail-open — silently falls through to _DEFAULT_CONTRACT_ROOTS, the main-tracking working tree, while the run still stamps occ_governance_ref: origin/dev

OCC dev is ~4,198 commits and ~2,189 contracts ahead of main, so a dev-only contract is invisible to that fallback and the run returns a single skipped contract check, which handler_dod_verify reports as CONTRACT_MISSING.

"The clone could not be read at the ref I claim I read" was being laundered into "no contract exists" — a false-negative generator on the only sanctioned Done-flip tool.

Measured, not inferred (2026-08-27)

  • 3 of 22 CONTRACT_MISSING results in the beta DoD-verification sweep were this artifact. OMN-16690, OMN-16759 and OMN-16761 all have contracts on origin/dev and produced real verdicts when re-run against a detached worktree at a48d3c8b9. $OMNI_HOME/onex_change_control was [behind 33] at the time.
  • Why the add fails — timed on the live repo: git worktree add --detach --force <tmp> origin/dev checks out 32,382 files in 34.5 s single-threaded. _GIT_OP_TIMEOUT_S was 60 s, and the sweep runs dod_verify 5-way parallel against the same clone. The ceiling was tripping on ordinary load.
  • Corroborating residue: git worktree list on the canonical OCC clone shows 405 registrations against 42 surviving directories — 363 orphaned admin records, the exact signature of an add that timed out and had its temp dir rmtree'd without git worktree remove.

The fix

  1. Fail closed on worktree-materialisation failure, on OMN-15454's terms: a distinct OCC_WORKTREE_UNAVAILABLE FAILED result naming the ref and the override, honouring the same DOD_VERIFY_ALLOW_STALE_OCC_REF escape hatch — which proceeds but marks every result un-attributable and appends a standalone disclosure item.
  2. Stop the timeout being the normal case. The ceiling moves to 300 s and becomes operator-configurable via DOD_VERIFY_GIT_OP_TIMEOUT_S, so hitting the new refusal means something is genuinely wrong rather than that five verifiers shared a clone.

The refresh_outcome is not None guard is load-bearing: when no OCC root resolves at all no fetch is attempted, and that legitimate pre-existing shape (the CI shape) keeps its existing reporting rather than being relabelled a fault.

A pre-existing test that was green for the wrong reason

test_omn_15454_occ_ref_refresh_fail_closed.py::test_ref_lock_race_is_retried_once_ac3 built a repo with a local dev branch and no remote, then asked the collector to resolve origin/dev. The worktree add failed with invalid reference, and the assertion only passed because the collector fell back to the working tree — which carried the same contract, since main was branched off dev. It asserted "materialised at the post-fetch tip" while nothing was ever materialised.

The fixture now creates a real remote so origin/dev genuinely exists and the assertion means what it says. The fail-open that test was blind to is exactly what this PR closes.

Tests (TDD, red first)

New test_omn_16787_occ_worktree_unavailable_fail_closed.py, driving the real collect() path with real git subprocesses:

  • AC1 — worktree-add failure refuses; the decisive assertion is negative: the working-tree contract must not be picked up, and the refusal must not be disguised as CONTRACT_MISSING.
  • AC2 — the override proceeds, marks every result UNATTRIBUTABLE, and appends occ_worktree_unavailable_override.
  • AC3 — non-regression: no OCC root at all still reports CONTRACT_MISSING, never the new refusal.
  • AC4 — the default ceiling exceeds the measured 34.5 s cold checkout with headroom; and with DOD_VERIFY_GIT_OP_TIMEOUT_S=0 a real subprocess.TimeoutExpired from a real git worktree add converges on the same refusal, against a fixture whose genuine origin/dev carries the contract (so a silent fallback would have produced a fully VERIFIED table).
  • Non-regression: a healthy origin/dev worktree still resolves dev-first and records a real 40-char SHA.

Verified red before the implementation (ImportError on the new symbols), green after: 12 passed across the new file and the OMN-15454 suite. ruff format, ruff check, mypy --strict clean; the governed pre-push selector ran 7,974 tests.

Refs: OMN-16787

Evidence-Ticket: OMN-16787
Evidence-Source: OCC#7332

…contract"

`EvidenceCollector._materialize_occ_dev_worktree` has two failure classes.
OMN-15454 closed one: a failed `git fetch` refuses. The other — a failed or
timed-out `git worktree add` — returned `(None, None, ...)` and the caller
silently fell through to `_DEFAULT_CONTRACT_ROOTS`, the `main`-tracking
working tree, while the run still stamped `occ_governance_ref: origin/dev`.

OCC `dev` runs ~4,159 commits and ~2,184 contracts ahead of `main`, so a
dev-only contract is invisible to that fallback and the run reports
`CONTRACT_MISSING`. "The clone could not be read at the ref I claim I read"
was being laundered into "no contract exists" — a false-negative generator on
the only sanctioned Done-flip tool. Measured 2026-08-27: 3 of 22
`CONTRACT_MISSING` results in the beta sweep were this artifact, with the
canonical clone 33 commits behind origin/dev.

The trigger is ordinary load, not a rare fault. A real `git worktree add
--detach origin/dev` of the live OCC repo checks out 32,382 files in ~34.5 s
single-threaded, against a 60 s ceiling, under a 5-way-parallel sweep. The
residue is visible: 405 worktree registrations against 42 surviving
directories on the canonical clone — orphaned admin records from adds that
timed out and had their temp dir rmtree'd.

So both halves are fixed:

* the worktree-materialisation failure now refuses with a distinct
  `OCC_WORKTREE_UNAVAILABLE` result, honouring the same
  `DOD_VERIFY_ALLOW_STALE_OCC_REF` override, which (as with FETCH_FAILED)
  proceeds but marks every result un-attributable plus a standalone
  disclosure item. Never a silent working-tree read under a dev-ref banner.
* the git-op ceiling moves to 300 s and becomes operator-configurable via
  `DOD_VERIFY_GIT_OP_TIMEOUT_S`, so hitting it means something is genuinely
  wrong rather than that five verifiers shared a clone.

The `refresh_outcome is not None` guard is load-bearing: when no OCC root
resolves at all, no fetch is attempted, and that legitimate pre-existing
shape keeps its existing reporting rather than being relabelled a fault.

Also repairs OMN-15454's own AC3 fixture, which was green for the wrong
reason. It built a repo with a LOCAL `dev` branch and no remote, then asked
the collector to resolve `origin/dev`; the worktree add failed with `invalid
reference` and the assertion only passed because the collector fell back to
the working tree, which carried the same contract. It asserted "materialised
at the post-fetch tip" while nothing was ever materialised. The fixture now
creates a real remote, so the assertion means what it says — and the
fail-open it was blind to is exactly what this ticket closes.

Refs: OMN-16787
@coderabbitai

coderabbitai Bot commented Aug 27, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 55 minutes.

View limit details

Limit details: You’ve used the included review currently available. Your 131 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4311f938-ebc7-4e2c-a8e4-c44366736ddc

📥 Commits

Reviewing files that changed from the base of the PR and between cff0d76 and 029ff24.

📒 Files selected for processing (3)
  • src/omnimarket/nodes/node_dod_verify/services/evidence_collector.py
  • tests/unit/nodes/node_dod_verify/test_omn_15454_occ_ref_refresh_fail_closed.py
  • tests/unit/nodes/node_dod_verify/test_omn_16787_occ_worktree_unavailable_fail_closed.py

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

⚠️ Architectural Review — WARNINGS

Errors: 0
Warnings: 1

What this checks

Rule Description
ARCH-TOPIC-001 No hardcoded Kafka topic strings in handler code
ARCH-DI-001 No event_bus=None bypass
ARCH-DI-002 No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003 No reinvented DI containers
ARCH-TOPIC-002 contract.yaml topics follow `onex.{cmd

ARCH-DI-002: Direct Handler instantiation (warning)\n\n```\nsrc/omnimarket/nodes/node_dod_verify/services/evidence_collector.py:2635: output = HandlerDodEvidenceGithubEffect().handle(command)

src/omnimarket/nodes/node_dod_verify/services/evidence_collector.py:2671: output = HandlerDodEvidenceGithubEffect().handle(command)
src/omnimarket/nodes/node_dod_verify/services/evidence_collector.py:3245: output = HandlerDodEvidenceGithubEffect().handle(command)
src/omnimarket/nodes/node_dod_verify/services/evidence_collector.py:3276: output = HandlerDodEvidenceGithubEffect().handle(command)\n```\n\nHandler classes should be wired only in workflow_runner.py or adapter modules. Direct instantiation elsewhere bypasses the contract system.\n\n

Static architectural lint — no model inference (OMN-14176).

@github-actions

Copy link
Copy Markdown

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 0
Models succeeded: qwen3-review,qwen3-review-b


Gate semantics

Verdict Meaning Blocks merge?
passed >=2 models succeeded, no critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Fewer than 2 models succeeded (infra unavailable/timeout) Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

@jonahgabriel
jonahgabriel enabled auto-merge (squash) August 27, 2026 20:42
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Aug 27, 2026
…7332)

* evidence: OCC companion pass 1 for OmniNode-ai/omnimarket#2169

* evidence: OCC companion self-bind for #7332

---------

Co-authored-by: node-occ-companion-effect <occ-companion-effect@omninode.ai>
@jonahgabriel
jonahgabriel merged commit 6abd2c2 into dev Aug 27, 2026
143 of 151 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-16787-occ-worktree-fail-closed branch August 27, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant