Skip to content

fix(OMN-15137): provision omnibase_spi via a shared sibling-clone manifest - #2450

Merged
jonahgabriel merged 4 commits into
devfrom
jonah/omn-15137-ensure_runner_clonessh-never-provisions-omnibase_spi-sibling
Jul 25, 2026
Merged

jonahgabriel merged 4 commits into
devfrom
jonah/omn-15137-ensure_runner_clonessh-never-provisions-omnibase_spi-sibling

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator

OMN-15137

Fixes ensure_runner_clones.sh never provisioning omnibase_spi under the deploy runner's private OMNI_HOME, which blocked OMN-14900 stability deploy-hop re-fire iteration N+3 (run https://github.com/OmniNode-ai/omnibase_infra/actions/runs/30175447119, 2026-07-25). Rollback was clean, no prod/judge impact.

Root cause

RUNNER_CLONE_REPOS in ensure_runner_clones.sh hardcoded a 5-repo list. stage_workspace.sh's sibling-pin preflight (PREFLIGHT_REPO_ARGS → check_sibling_lock_pins.py's DEFAULT_PACKAGE_REPO_DIRS) has required a live OMNI_HOME/omnibase_spi clone since OMN-12977 — a 6-repo list. The two lists were independently hardcoded and drifted apart silently; the gap only surfaced 3 deploy hops deep, after OMN-15122 and OMN-15131 were fixed and a run finally reached this step.

Fix (root pattern, not just the one repo)

Added scripts/runtime_build/sibling_clone_manifest.sh as the single source of truth for the full 6-repo sibling-clone set the workspace build resolves clones for: omnibase_infra, omnibase_core, omnibase_spi, omnibase_compat, onex_change_control, omnimarket. Both ensure_runner_clones.sh (RUNNER_CLONE_REPOS) and stage_workspace.sh (PREFLIGHT_REPO_ARGS) now source this file instead of independently hardcoding their own copy — they can never diverge again. ensure_runner_clones.sh's existing existence/operability loop already fails fast and names any missing clone; feeding it the complete manifest makes that loop the "preflight names any missing clone before deploy starts" behavior the ticket asks for, with no new logic needed.

Audited full sibling set (Dockerfile.runtime + stage_workspace.sh):

  • omnibase_infra — Docker build context itself, never vendored as a "sibling," but still needs a clone for RUNNER_CLONE_REPOS/RT-1.
  • omnibase_core, omnibase_compat, onex_change_control, omnimarket — vendored as source via stage_workspace.sh's SIBLING_REPOS.
  • omnibase_spi — installed from the published PyPI wheel via uv sync (never staged from local source), but still needs a live OMNI_HOME clone so the pin-preflight can read its pyproject.toml version + git SHA against the consuming lock.

No 7th sibling exists in any lock file — check_sibling_lock_pins.py's DEFAULT_PACKAGE_REPO_DIRS has exactly these 6 entries, confirmed by the new parity test below.

Tests (new)

  • tests/scripts/test_ensure_runner_clones.py — end-to-end against real file:// git remotes: proves all 6 repos including omnibase_spi are cloned, idempotent re-run, fail-closed + named-repo error on missing upstream, fail-closed on an unwritable clone .git dir, fail-closed on missing OMNI_HOME. Confirmed RED against the pre-fix 5-repo manifest (7/8 assertions fail without omnibase_spi in the list).
  • tests/scripts/test_sibling_clone_manifest_parity.py — cross-language recurrence ratchet: asserts sibling_clone_manifest.sh's bash arrays exactly match check_sibling_lock_pins.py's DEFAULT_PACKAGE_REPO_DIRS, so a future 7th sibling added on one side and forgotten on the other fails CI immediately instead of failing 3 deploy hops deep on a real runner.

Verification (.200, env -u PYTHONPATH uv run pytest, homebrew bash 5.3 on PATH)

  • New files: 8/8 passed.
  • Full regression set (new + existing stage_workspace/check_sibling_lock_pins coverage): 67/67 passed.
  • ruff format --check + ruff check: clean.
  • shellcheck --severity=warning (all 3 shell files): clean.
  • bash -n: syntax OK.
  • pre-commit (scoped to changed files): all hooks passed, including SPDX.

Aside (not fixed here, out of scope): macOS system /bin/bash 3.2 has a pre-existing, unrelated nounset-on-empty-array bug that reproduces at stage_workspace.sh line ~204 (preflight_extra[@]: unbound variable) when tests invoke bash and it resolves to the system binary instead of a modern one. Confirmed identical on unmodified dev tip (not a regression from this PR); GNU bash 4.4+ (what the real Linux deploy runner uses) does not have this bug.

dod_evidence

  • Ticket: OMN-15137 (parent OMN-14900).
  • Root cause reproduced live, per ticket run https://github.com/OmniNode-ai/omnibase_infra/actions/runs/30175447119.
  • Acceptance per ticket: a fresh release-train-lab.yml stability-lane execute=true run reaching past check_sibling_lock_pins.py without an omnibase-spi clone-pin resolution error. This PR does not itself trigger that run (requires a fresh cut/deploy hop, out of scope for this PR); a RED-confirmed test now proves the fix, and the next stability deploy hop is the live acceptance readback.

Not merging this PR myself — Codex owns merge queue / merge per repo policy.

Evidence-Ticket: OMN-15137
Evidence-Source: OCC#4877

Evidence-Commit: d798061a08b31457c2213ba410d89bf9b12612c8

…ifest

ensure_runner_clones.sh never provisioned an omnibase_spi clone under the
deploy runner private OMNI_HOME, even though stage_workspace.sh sibling-pin
preflight (PREFLIGHT_REPO_ARGS -> check_sibling_lock_pins.py's
DEFAULT_PACKAGE_REPO_DIRS) has required a live OMNI_HOME/omnibase_spi clone
since OMN-12977. The two lists were independently hardcoded 5-repo and
6-repo bash arrays that silently drifted apart. This blocked OMN-14900
stability deploy-hop re-fire iteration N+3 3 steps downstream of clone
provisioning:

  ERROR: cannot resolve clone pin for omnibase-spi: missing pyproject.toml
  for omnibase-spi: /data/omninode/runner_omni_home/omnibase_spi/pyproject.toml

Fix targets the ROOT pattern, not just the missing repo: adds
scripts/runtime_build/sibling_clone_manifest.sh as the single source of
truth for the full 6-repo sibling-clone set (omnibase_infra, omnibase_core,
omnibase_spi, omnibase_compat, onex_change_control, omnimarket). Both
ensure_runner_clones.sh (RUNNER_CLONE_REPOS) and stage_workspace.sh
(PREFLIGHT_REPO_ARGS) now source this one file instead of hardcoding their
own copy, so the two can never diverge again. ensure_runner_clones.sh's
existing existence/operability loop is the fail-fast preflight the ticket
asks for -- once fed the complete manifest it already names any missing
clone before the deploy proceeds, with no new logic required.

Audited full sibling set consumed by the workspace build (Dockerfile.runtime
+ stage_workspace.sh): omnibase_infra is staged as the Docker build context
itself (never a vendored "sibling"); omnibase_spi is installed from the
published PyPI wheel via `uv sync` (never staged from local source) but
still needs a live OMNI_HOME clone so the pin-preflight can read its
pyproject.toml version + git SHA; omnibase_core/omnibase_compat/
onex_change_control/omnimarket are vendored as source via
stage_workspace.sh's SIBLING_REPOS. No 7th sibling exists in any lock file
(check_sibling_lock_pins.py's DEFAULT_PACKAGE_REPO_DIRS has exactly these 6
entries).

Tests (new):
- tests/scripts/test_ensure_runner_clones.py: end-to-end against real
  file:// git remotes -- proves all 6 repos including omnibase_spi are
  cloned, idempotent re-run, fail-closed + named-repo error on missing
  upstream, fail-closed on an unwritable clone .git dir, fail-closed on
  missing OMNI_HOME. Confirmed RED against the pre-fix 5-repo manifest
  (7/8 assertions fail without omnibase_spi in the list).
- tests/scripts/test_sibling_clone_manifest_parity.py: cross-language
  recurrence ratchet asserting sibling_clone_manifest.sh's bash arrays
  exactly match check_sibling_lock_pins.py's DEFAULT_PACKAGE_REPO_DIRS, so
  a future 7th sibling added on one side and forgotten on the other fails
  CI immediately instead of failing 3 deploy hops deep on a real runner.

Verification (.200, env -u PYTHONPATH uv run pytest, homebrew bash 5.3 on
PATH -- macOS system /bin/bash 3.2 has a pre-existing, unrelated nounset
empty-array bug in stage_workspace.sh line ~204 that reproduces identically
on unmodified dev tip; out of scope for this ticket):
- New files: 8/8 passed.
- Full regression set (new + existing stage_workspace/check_sibling_lock_pins
  coverage): 67/67 passed.
- ruff format --check + ruff check: clean.
- shellcheck --severity=warning (all 3 shell files): clean.
- bash -n: syntax OK.
- pre-commit (scoped to changed files): all hooks passed, including SPDX.

dod_evidence:
- Ticket: OMN-15137 (parent OMN-14900).
- Root cause reproduced live, per ticket run
  https://github.com/OmniNode-ai/omnibase_infra/actions/runs/30175447119.
- Acceptance per ticket: a fresh release-train-lab.yml stability-lane
  execute=true run reaching past check_sibling_lock_pins.py without an
  omnibase-spi clone-pin resolution error. This PR does not itself trigger
  that run (requires a fresh cut/deploy hop, out of scope for this PR); a
  RED-confirmed test now proves the fix and the next stability deploy hop
  is the live acceptance readback.

Not merging this PR myself -- Codex owns merge queue / merge per repo
policy.

Evidence-Ticket: OMN-15137
@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 17 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f7a071b6-10ea-400b-93c9-a0974708c859

📥 Commits

Reviewing files that changed from the base of the PR and between 3e7e487 and 5f2427f.

📒 Files selected for processing (6)
  • scripts/runtime_build/ensure_runner_clones.sh
  • scripts/runtime_build/sibling_clone_manifest.sh
  • scripts/runtime_build/stage_workspace.sh
  • tests/scripts/test_ensure_runner_clones.py
  • tests/scripts/test_sibling_clone_manifest_parity.py
  • tests/unit/scripts/test_runner_private_omni_home_safe_directory.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jonah/omn-15137-ensure_runner_clonessh-never-provisions-omnibase_spi-sibling

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8001 ] unreachable — preflight short-circuit (no models available)


Gate semantics (pilot phase)

Verdict Meaning Blocks merge?
passed No critical findings No
blocked CRITICAL findings found Yes
degraded All models unavailable (infra) No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)

docker-compose.runners.yml GIT_CONFIG_COUNT/GIT_CONFIG_KEY_*/VALUE_* (the
container-level defense-in-depth safe.directory list for the deploy
runner) was ALSO hardcoded to 5 entries, missing omnibase_spi -- the same
independently-hardcoded-list drift this ticket fixes elsewhere. Bumped to
6 and added the omnibase_spi entry.

tests/unit/scripts/test_runner_private_omni_home_safe_directory.py
(pre-existing OMN-14900 coverage) asserted the old 5-repo shape in three
places and its own file:// bare-fixture fixture only seeded 5 upstream
repos -- this is what CI Tests (Split 1/1) caught on PR #2450 after the
sibling_clone_manifest.sh fix made ensure_runner_clones.sh try to clone a
6th repo the test fixture never provided. Updated _ENSURE_REPOS to 6
(added omnibase_spi), the GIT_CONFIG_COUNT/KEY/VALUE assertions to 6, and
renamed the now-inaccurate test_..._all_five_... to
test_..._all_and_is_idempotent.

Verified (.200): tests/unit/scripts/test_runner_private_omni_home_safe_directory.py
20/20 passed; full regression set (with the two new OMN-15137 test files)
87/87 passed; pre-commit (scoped) all hooks passed.

Evidence-Ticket: OMN-15137
jonahgabriel added a commit to OmniNode-ai/onex_change_control that referenced this pull request Jul 25, 2026
…ibase_infra#2450 (#4873)

* evidence(OMN-15137): author OCC companion for OmniNode-ai/omnibase_infra#2450

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head cda61cf7dd6fc3741b24504066b4de43ab7c75d3.

* evidence(OMN-15137): self-bind OCC#4873 + rebind contract_sha256

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
The docker-compose.runners.yml GIT_CONFIG_* change from the previous
commit is reverted: it is a "runtime path" per deploy-gate, which
correctly blocked the PR (DEPLOY GATE FAILED: no cited ticket has a
falsifiable deploy probe) since this container-level env only takes
effect on a deploy-runner container recreate -- out of scope for a
code-only PR, same split as OMN-15134/#2448 (land now, deploy separately).

Kept: the test file correction (_ENSURE_REPOS now has 6 entries including
omnibase_spi, matching ensure_runner_clones.sh's real behavior -- this
part is NOT deploy-gated, it is test-only and was a genuine CI regression
introduced by the sibling_clone_manifest.sh fix). Reverted the
compose-specific GIT_CONFIG_COUNT/KEY/VALUE assertions in that same test
file back to 5, with a note that the container-level omnibase_spi
safe.directory gap is a tracked, non-blocking residual (the load-bearing
per-op `-c safe.directory=` scoping in the scripts already covers it).

Verified (.200): full regression set (unit/scripts + scripts/) 87/87
passed; pre-commit (scoped) all hooks passed.

Evidence-Ticket: OMN-15137
@jonahgabriel
jonahgabriel merged commit 49c886a into dev Jul 25, 2026
101 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-15137-ensure_runner_clonessh-never-provisions-omnibase_spi-sibling branch July 25, 2026 22:29
jonahgabriel added a commit that referenced this pull request Jul 26, 2026
…t catalog, T0/T1 readback, dev gotchas, prod pointer (#2454)

Extends docs/runbooks/release-train-lab.md (existing mechanism doc, not a
new file) with the operator-facing procedure proven live on run
30180376657 (2026-07-26, tag lab/stability/20260725T235956Z-87ec5b3165ce,
overall: PASS) — the terminal success of the six-iteration OMN-14900
hardening chain.

- Copy-paste tag-cut + watch procedure with expected output and a
  tag-content WARNING (never reuse a parked tag name).
- Preflight catalog: what each of the 6 landed fixes (#2450/#2446/#2444/
  #2452/#2434/#2448) catches, with pre-fix failure signatures.
- T0/T1 readback discipline: health 18085/18086, contract-count floor vs.
  regression, discovery_errors baseline, consumer groups, vcs_ref
  ancestry; FAILED_ROLLED_BACK equality-proof discipline.
- Dev-lane gotchas: stale workspace-build config YAML, OMN-14968 false
  FAILED on runtime-worker; pointer (not duplicate) to
  cold-lane-full-bringup.md for cold bring-up.
- Prod: pointer-only section citing CLAUDE.md rules 2a/12 and the
  onex_change_control#4892 prep-only grant-PR pattern; explicit
  raw-docker-mutation prohibition citing the no-raw-prod-bypass gate.
- Verification checklist + rollback section.

Co-authored-by: jonahgabriel <jonahgabriel@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant