Skip to content

fix(OMN-16989): lift h201's omnibase_infra denial — the 15 host-coupled failures were a gate-runner container artifact - #3041

Merged
github-actions[bot] merged 1 commit into
devfrom
jonah/omn-16989-h201-repos-denied
Aug 30, 2026
Merged

github-actions[bot] merged 1 commit into
devfrom
jonah/omn-16989-h201-repos-denied

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Aug 30, 2026 •

Copy link
Copy Markdown
Collaborator

OMN-16989 — the 15 "host-coupled" failures were a container artifact; h201 is green on the host

scripts/hooks/prepush_hosts.tsv carried repos_denied=omnibase_infra on h201
because OMN-16989 measured 15 host-coupled failures there. Every one of the 15
was measured inside the .201 gate-runner CONTAINER, driven by
scripts/ci/run_on_gate_runner.sh. That is not the environment this table routes
work to — the table routes to the .201 host over the OMN-16991 remote leg
(bundle transplant into a scratch workroot, uv sync in a fresh tree, the
wrapper's developer-shell PATH, and the committed table as the identity source
instead of an env override the pytest child then scrubs).

Re-measured over the REAL leg, not by hand

Driven through the shipped dispatcher's own functions (pick_capacity_host →
prepush_remote_run, extracted verbatim by line range), with placement forced to
h201 via the picker's own deterministic PREPUSH_SLOT_OVERRIDE_MAP surface — no
bypass knob was added, PREPUSH_FULL_SUITE was never set.

dispatch collected pytest_exit duration
tests/unit/ + tests/integration/chains/ (the heavy escalation argv) 24,939 0 695 s
tests/ci/ (the tree the 15 came from) 1,892 0 130 s

Both host_mode=authorizing, chosen_host=omninode-pc, chosen_label=h201.
Receipt on this PR's exact head:

{"ts":"2026-08-30T16:48:08Z","repo":"omnibase_infra","head_sha":"18c669be63c65941cf7cdd9ec2c27f1108a20c29",
 "chosen_host":"omninode-pc","chosen_label":"h201","host_mode":"authorizing",
 "selection_paths":"tests/unit/ tests/integration/chains/ ","pytest_exit":0,
 "collected":24939,"duration_s":695,
 "suite_log_sha256":"35e94ecfe3183e9b78838f5ceed313f644e22d0a433378ca72f2b12fb9c881de"}
{"ts":"2026-08-30T16:24:23Z","repo":"omnibase_infra","head_sha":"18c669be63c65941cf7cdd9ec2c27f1108a20c29",
 "chosen_host":"omninode-pc","chosen_label":"h201","host_mode":"authorizing",
 "selection_paths":"tests/ci/ ","pytest_exit":0,
 "collected":1892,"duration_s":130,
 "suite_log_sha256":"20e49c2acdac9741e48ce594eea2202c8b27a0799d42a50538b754ac17cc0faa"}

Both lines are verbatim from .onex_state/prepush_distribution/receipts.jsonl, filtered on this PR's head sha. An earlier revision of this body pasted the previous head's numbers (24,930 / 699 s / 2414e006…) under the new sha; corrected here against the file. host_load_ratio is empty because these two were driven by the verification driver, which pins h201 rather than ranking — host_load_ratio is populated by host_load_ratio(), which lives in the caller (prepush_smart_tests.sh), not in the dispatch library. The ratio is receipt metadata; the verdict is the marker-bound pytest_exit.

The 15, re-classified against the real leg

OMN-16989 class verdict on the host
(a) 4 × host-identity guard tests pass. OMN-16991 moved identity onto the committed table and omninode-pc is a first-class row, so the ".200 or refuse" encoding is already parametrized. One of the four still failed — on base ref 'origin/dev' could not be resolved, which is about the transplant, not identity (fix 2 below).
(b) 2 × env-parity, stale sibling clone skip, with the reason named — omninode_infra is not a sibling of a transplanted tree.
(c) 9 × host-environment coupling pass. TestEntrypointOrphanReap, the three TestImportEnvironmentIsolation, the four TestRedeployStartCLI and test_cache_effectiveness_metrics all pass on the 32-core host; the 4-CPU container cap that produced the timing/TimeoutExpired failures is not in play.

The denial was pinning a verdict from an environment nothing dispatches to.

Four real defects surfaced on the way — fixed, not denied around

  1. The remote wrapper's PATH prefix was macOS-only by construction.
    /opt/homebrew/bin has no meaning on a Linux row and h201 is the fleet's only
    one. Measured non-interactively 2026-08-30, .201's ssh PATH is
    /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
    — ~/.local/bin is absent and both uv and shellcheck live there, so the
    existing $(dirname "$UV") / ~/.local/bin entries already covered it (zero
    tool-missing reds in the full run). The Linux analogues are appended after
    every measured entry, so they can only add resolution.

  2. The transplanted tree has no remote-tracking refs. git bundle create <b> HEAD
    carries one ref, and this suite subprocesses this hook, which resolves
    ${PREPUSH_BASE_REF:-origin/dev} first. The dispatcher now hands the wrapper the
    base ref and its sha and the wrapper update-refs it after checkout. BASE_SHA
    is a merge-base, so it is always an ancestor of HEAD and already in the bundle.
    Unresolvable ⇒ skipped silently: it may only add resolution, never refuse a run.

  3. resolve_omni_home walks every parent of the repo root. On the transplant it
    reached /data/omninode and imported an omniclaude clone pinned at
    omniclaude#1600 — the host's current clone (fix(OMN-13062): migration-gate vacuity fix — sentinel discipline, wait-for-postgres, skip-manifest, sync-check nonzero (retro A-10) #1963) lives at
    /data/omninode/omni_home and is not an ancestor. Four tests failed on
    module '_deploy_gate_canonical_validator' has no attribute 'parse_evidence_metadata': a red about another repo's clone freshness on
    whichever host ran the suite
    , with nothing saying so — the same cross-repo
    hidden-input shape OMN-16989 named on the env-parity tests.
    load_canonical_validator now asserts the surface it calls and raises a named
    staleness error; the fixture skips with that reason attached. The hook itself
    still hard-errors, but with a remediation instead of an AttributeError five
    frames down.

  4. test_release_identity_passes_when_version_ahead asserted an ambient repo state.
    Between a release tag landing and the post-release bump landing (OMN-13912,
    chore(OMN-13912): post-release dev version bump to 0.38.15 #3033 — itself blocked on a stale runner-image identity lock) it is red on
    every developer clone and only there: actions/checkout does not fetch tags,
    so in CI git tag --list is empty, the gate is exempt and the test passes;
    locally the tags exist and it fails. Measured 2026-08-30 — dev at 0.38.14 with
    v0.38.14 cut at 13:24Z — it refused every local governed pre-push in the repo
    while being invisible in CI. The test now names the precondition and skips
    with it. The gate is untouched: check-release-identity (pre-commit) and the CI
    gate both still run the real script against the real diff.

Table + tests, per the runbook's promotion procedure

  • scripts/hooks/prepush_hosts.tsv: h201 repos_denied → -, note carries the
    measured numbers.
  • test_201_is_denied_for_omnibase_infra_until_omn16989_closes →
    test_201_denies_no_repo_since_omn16989_closed.
  • test_a_repo_denied_host_is_never_chosen now drives a synthetic table (the
    _SYNTHETIC_TABLE precedent set when h105 left shadow). Pinning the placement
    RULE to whichever repo the lab happens to deny today made a capacity-policy edit
    look like a mechanism regression — which is exactly the failure this run hit.
  • test_no_row_denies_a_repo_today_so_the_rule_needs_a_synthetic_fixture guards that
    fixture choice: the moment a real row denies a repo again it fails and says the
    live table may be pinned again.
  • docs/runbooks/lab-prepush-host-table.md records the lift, the container-vs-host
    distinction, and both transplant classes as host prerequisites.

Gate posture

Nothing here lets the gate accept less work. h201 moving from repo-denied to a
placement candidate can only add an execution target; every verdict still comes
from a real uv run pytest on an authorizing row of the committed table, read
back through a MARKER bound to {head_sha, argv_sha}. No new PREPUSH_* knob, no
--no-verify, no core.hooksPath override, no bypass token of the reject-skip-token family, no
PREPUSH_FULL_SUITE, no ENABLE_SMART_TESTS=off, no override grant consumed.

Local evidence

  • tests/unit/scripts/test_prepush_host_table.py: 65 → 69 tests, all green.
  • tests/ci/test_check_deploy_scope_dod.py: 6 → 8, all green.
  • Prepush surface locally: 117 green across test_prepush_host_table,
    test_check_deploy_scope_dod, test_prepush_hook_host_identity_guard,
    test_pytest_full_suite_host_guard; 109 green across the remaining
    test_prepush* files.
  • Governed pre-push hook ran to completion on this branch.

Closes OMN-16989.

Evidence-Ticket: OMN-16989
Evidence-Source: OCC#7733

Not fixed here — filed instead

tests/unit/scripts/test_heavy_lock.py::test_a_wedged_holder_is_distinguishable_from_a_working_one false-reds on a loaded .200: _group_is_gone gives async SIGKILL delivery a fixed 10 s budget, and the leader's returncode is already -9 when it trips. It is green on .201 (inside the 24,939 run above) and passes isolated on .200 in 10.08 s; it failed once in this PR's own 20-minute selection at ~1.8x load and passed on the re-run (4140 passed, 15 skipped). Same defect class as this ticket, different host, and it is OMN-16995's surface — filed as OMN-17239 rather than absorbed here.

Verification

  • Governed pre-push selector: 4140 passed, 15 skipped on the narrow impacted selection (scripts/ci/tests/ scripts/tests/ tests/ci/ tests/scripts/ tests/unit/scripts/). The selection is not whole-suite-equivalent, so it ran locally and the remote leg was not engaged.
  • pre-commit run --all-files: every hook Passed, tree clean.
  • Rebased onto origin/dev@5367b2882, which picks up OMN-17161's h101 promotion and OMN-13912's 0.38.15 bump. The prepush_hosts.tsv merge is clean and every capacity row now carries repos_denied=-.
  • No --no-verify, no PREPUSH_* override, no ENABLE_SMART_TESTS, no skip token, no override grant consumed. Runtime lanes (dev/stability/prod/judge compose projects) were never contacted — all work is filesystem plus a scratch workroot at /data/omninode/onex-prepush.

…e-red classes behind it

OMN-16989 recorded 15 host-coupled `omnibase_infra` failures on `.201` and set
`repos_denied=omnibase_infra` on `h201`. Every one of the 15 was measured inside
the `.201` GATE-RUNNER CONTAINER via `scripts/ci/run_on_gate_runner.sh` -- a
different execution environment from the one the table routes work to, which is
the `.201` HOST over the OMN-16991 remote leg.

Re-measured on the host over the real leg, receipts in
`.onex_state/prepush_distribution/receipts.jsonl`:

  tests/unit/ + tests/integration/chains/   24930 collected, pytest_exit=0, 695s
  tests/ci/ (the tree the 15 came from)      1892 collected, pytest_exit=0, 122s

both host_mode=authorizing, chosen_host=omninode-pc. Zero host-coupled failures.
The denial was pinning a verdict from an environment nothing dispatches to.

Four real defects surfaced on the way and are fixed, not denied around:

1. The remote wrapper's PATH prefix was macOS-only by construction.
   `/opt/homebrew/bin` has no meaning on a Linux row and h201 is the fleet's only
   one. Its measured non-interactive PATH omits `~/.local/bin`, where BOTH `uv`
   and `shellcheck` live -- already covered by the existing entries, so this
   caused no red on h201, but the list is now Linux-complete. The added entries
   go AFTER every measured one, so they can only add resolution.

2. The transplanted tree has no remote-tracking refs. `git bundle create <b> HEAD`
   carries one ref, and this suite subprocesses the hook, which resolves
   `${PREPUSH_BASE_REF:-origin/dev}` first. On h201 the whole
   `test_prepush_hook_host_identity_guard.py` behavioral proof reduced to
   `base ref 'origin/dev' could not be resolved`. The dispatcher now hands the
   wrapper the base ref and sha and the wrapper `update-ref`s it in. BASE_SHA is
   a merge-base, so it is always an ancestor of HEAD and already in the bundle.
   Unresolvable => skipped silently: it may only add resolution, never refuse.

3. `check_deploy_scope_dod.resolve_omni_home` walks EVERY parent of the repo
   root, so on the transplant it reached `/data/omninode` and imported an
   omniclaude clone pinned at omniclaude#1600 (the host's current clone, #1963,
   is not an ancestor). Four tests failed on `has no attribute
   'parse_evidence_metadata'` -- a red about another repo's clone freshness on
   whichever host ran the suite, with nothing saying so. Same
   cross-repo-hidden-input shape OMN-16989 named on the env-parity tests.
   `load_canonical_validator` now asserts the surface it calls and raises a NAMED
   staleness error; the fixture skips with that reason. The hook still
   hard-errors, but with a remediation instead of an AttributeError five frames
   down.

4. `test_release_identity_passes_when_version_ahead` asserted an AMBIENT
   repo-state precondition ("someone bumped the version in this PR") rather than
   stating it. Between a release tag landing and the post-release bump landing
   (OMN-13912, PR #3033, itself blocked on a stale runner-image identity lock) it
   is red on EVERY developer clone and ONLY there: `actions/checkout` does not
   fetch tags, so in CI `git tag --list` is empty, the gate is exempt and the
   test passes; locally the tags exist and it fails. Measured 2026-08-30 -- dev
   at 0.38.14 with `v0.38.14` cut at 13:24Z -- it refused every local governed
   pre-push in the repo while being invisible in CI. The test now names the
   precondition and skips with it. The GATE is untouched: `check-release-identity`
   (pre-commit) and the CI gate both still run the real script against the real
   diff.

Table + tests, per the runbook's promotion procedure:
- `scripts/hooks/prepush_hosts.tsv`: h201 `repos_denied` -> `-`, note carries the
  measured numbers.
- `test_201_is_denied_for_omnibase_infra_until_omn16989_closes` ->
  `test_201_denies_no_repo_since_omn16989_closed`.
- `test_a_repo_denied_host_is_never_chosen` now drives a SYNTHETIC table (the
  `_SYNTHETIC_TABLE` precedent set when h105 left shadow), so the placement RULE
  is no longer pinned to whichever repo the lab happens to deny today -- pinning
  it made a capacity-policy edit look like a mechanism regression, which is
  exactly the failure this run hit.
- `test_no_row_denies_a_repo_today_so_the_rule_needs_a_synthetic_fixture` guards
  that fixture choice: the moment a real row denies a repo again, it fails and
  says the live table may be pinned again.

Not fixed here: `pyproject.toml` version-vs-published is a release-cut condition
owned by OMN-13912; this diff touches neither `src/**` nor `pyproject.toml`, so
the release-identity gate itself does not apply to it.
@coderabbitai

coderabbitai Bot commented Aug 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 17 minutes.

View limit details

Limit details: You’ve used the included review currently available. Your 138 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 74782412-0541-43f0-a0ba-e0bfbcd6391e

📥 Commits

Reviewing files that changed from the base of the PR and between 7dcae42 and 18c669b.

⛔ Files ignored due to path filters (1)
  • scripts/hooks/prepush_hosts.tsv is excluded by !**/*.tsv
📒 Files selected for processing (6)
  • docs/runbooks/lab-prepush-host-table.md
  • scripts/ci/check_deploy_scope_dod.py
  • scripts/hooks/prepush_dispatch.sh
  • tests/ci/test_check_deploy_scope_dod.py
  • tests/scripts/test_fresh_deploy_fitness_gates.py
  • tests/unit/scripts/test_prepush_host_table.py

Comment @coderabbitai help to get the list of available commands.

@github-actions
github-actions Bot enabled auto-merge (squash) August 30, 2026 17:38
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Aug 30, 2026
…ibase_infra#3041 (#7733)

* evidence(OMN-16989): author OCC companion for OmniNode-ai/omnibase_infra#3041

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 18c669be63c65941cf7cdd9ec2c27f1108a20c29.

* evidence(OMN-16989): self-bind OCC#7733 + rebind contract_sha256

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8000 ] unreachable — preflight short-circuit (no models available)


Gate semantics (pilot phase)

Verdict Meaning Blocks merge?
passed No critical findings No
blocked CRITICAL findings found Yes
degraded All models unavailable (infra) No (pilot)

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review (OMN-8468/OMN-8524)

@github-actions
github-actions Bot merged commit 30c0158 into dev Aug 30, 2026
164 of 186 checks passed
@github-actions
github-actions Bot deleted the jonah/omn-16989-h201-repos-denied branch August 30, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant