Skip to content

feat(OMN-14466): RT-7 fail-loud release/deploy drift monitor - #2272

Merged
jonahgabriel merged 2 commits into
devfrom
jonah/omn-14466-rt-7-fail-loud-releasedeploy-drift-monitor
Jul 12, 2026
Merged

jonahgabriel merged 2 commits into
devfrom
jonah/omn-14466-rt-7-fail-loud-releasedeploy-drift-monitor

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Jul 12, 2026 •

Copy link
Copy Markdown
Collaborator

OMN-14466 — RT-7: fail-loud release/deploy drift monitor

Standalone scheduled monitor (Mechanical Release Trains, design docs/plans/2026-07-12-mechanical-release-trains.md §4 RT-7) that surfaces every silent release/deploy divergence same-day instead of after six weeks of green-and-unread silence. Epic OMN-13674.

What it checks (per repo)

  • Version drift: PyPI published version vs latest v* tag vs main vs dev pyproject version — fires on PyPI-behind-tag (stuck publish), main-ahead-of-tag (bump never tagged), main-behind-released (stale main), and a large dev→main backlog.
  • [tool.uv.sources] override drift: main carrying a git override that dev already dropped (the OCC omnibase-core rev=f325e759 case).
  • Release/deploy workflow health: last release.yml / runtime-rebuild-trigger.yml run failed, or stalled while main is ahead of the tag.

Output — feeds the existing substrate, no new store

On divergence it writes a friction record into the existing friction registry (.onex_state/friction/, consumed by node_friction_triage_orchestrator → Linear) and exits non-zero so the scheduled workflow goes RED (durable, observable, same-day). Friction-emit shape mirrors node_env_sync_alert_effect, the platform's existing scheduled drift→friction→Linear precedent. It builds no new store.

Seam

NEW standalone files only. Does not modify stage_workspace.sh, deploy-runtime.sh, release.yml, or the trigger scripts (other RT lanes own those).

  • scripts/release_drift_monitor_lib.py — pure evaluator (no I/O)
  • scripts/release_drift_monitor.py — I/O shell (gh api + PyPI JSON) + friction emit
  • .github/workflows/release-drift-monitor.yml — scheduled (every 6h) + workflow_dispatch
  • tests/scripts/test_release_drift_monitor.py — 12 unit tests

The per-lane running-signature check (.201 container signature vs intended ref) is deliberately out of scope here — it needs live SSH/introspection and is delivered by the RT-6 deploy-readback lane. No fake lane check is emitted.

dod_evidence — RED reproduces on TODAY's real drift (not a fixture)

Live run uv run python scripts/release_drift_monitor.py --json on 2026-07-12 → exit 1, diverged: true, blind: false, 0 probe errors, 13 findings across 6 repos, including:

  • PYPI_BEHIND_TAG P1 — omnibase-infra PyPI 0.36.1 behind latest tag v0.38.3 (the six-week-stuck case)
  • RELEASE_WORKFLOW_FAILED P1 — omnibase_infra release.yml last run = failure (2026-05-27)
  • MAIN_STALE_UV_OVERRIDE P1 — onex_change_control main still pins the omnibase-core [tool.uv.sources] git override that dev dropped
  • MAIN_AHEAD_OF_TAG / RELEASE_TRAIN_STALLED (infra 0.38.4 never tagged, release 46d stale); MAIN_BEHIND_RELEASED (core/compat/market/OCC); DEV_AHEAD_OF_MAIN (core +170, infra +294, market +547, OCC +1440)

The unit suite proves the check discriminates (per feedback_prove_red_against_exists_but_wrong): test_live_* fire on the exact live-shaped facts; test_aligned_repo_is_green passes on aligned facts, and its siblings flip one field to confirm the same check goes RED — a green means "checked and matched", not "couldn't see". A blind run (probe errors, no findings) exits 2, never a false green.

Local gates: 12/12 unit tests pass; ruff check + ruff format clean; mypy clean; full pre-commit green (SPDX, AI-slop, url-authority annotated for the PyPI public API, os.environ-block all pass).

Notes

  • pr_state substrate (OMN-14375/14374) is PR-scoped (repo, pr_number) — the wrong schema for release/deploy drift; the friction registry is the correct existing durable substrate reused here.
  • Requires CROSS_REPO_PAT in CI to read sibling repos; with only GITHUB_TOKEN the sibling probes fail and the monitor exits 2 (BLIND) rather than a false green.
  • DO NOT MERGE pending the RT-7 verifier + a non-self-authored OCC companion (Task Core Domain Models: Protocol-Based Health Architecture #11).

Evidence-Source: OCC#4048
Evidence-Commit: 15a088606656c9f62f97832b59e752695ac77bd0

Evidence-Ticket: OMN-14466

Standalone scheduled monitor that surfaces every silent release/deploy
divergence same-day instead of after six weeks of green-and-unread silence.

Per repo it compares PyPI-published version vs latest v* tag vs main/dev
pyproject version, flags [tool.uv.sources] git-override drift (main carrying
an override dev already dropped), and checks the last release.yml /
runtime-rebuild-trigger.yml run. On divergence it writes a friction record
into the existing friction registry (.onex_state/friction/, consumed by
node_friction_triage_orchestrator -> Linear) and exits non-zero so the
scheduled workflow goes RED. No new store; reuses the env_sync_alert
friction precedent.

- scripts/release_drift_monitor_lib.py: pure evaluator (no I/O), unit-tested
  against EXISTS-but-WRONG (fires) and aligned (green) facts.
- scripts/release_drift_monitor.py: I/O shell (gh api + PyPI JSON) + friction emit.
- .github/workflows/release-drift-monitor.yml: scheduled (every 6h) + dispatch.
- tests/scripts/test_release_drift_monitor.py: 12 tests, incl. live-drift RED.

dod_evidence: live run against GitHub/PyPI on 2026-07-12 exits 1 with 13
findings incl. omnibase_infra PyPI 0.36.1 behind tag v0.38.3 (P1),
omnibase_infra release.yml last run = failure (P1), onex_change_control main
stale omnibase-core [tool.uv.sources] override (P1). blind=false, 0 probe
errors. 12/12 unit tests pass; ruff + mypy clean; pre-commit green.

Epic OMN-13674. Design: docs/plans/2026-07-12-mechanical-release-trains.md section 4 (RT-7).
@coderabbitai

coderabbitai Bot commented Jul 12, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 23 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a0c5ef14-e349-4b4e-9ab5-a95281151884

📥 Commits

Reviewing files that changed from the base of the PR and between e811995 and 7e81e2d.

📒 Files selected for processing (4)
  • .github/workflows/release-drift-monitor.yml
  • scripts/release_drift_monitor.py
  • scripts/release_drift_monitor_lib.py
  • tests/scripts/test_release_drift_monitor.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jonah/omn-14466-rt-7-fail-loud-releasedeploy-drift-monitor

Comment @coderabbitai help to get the list of available commands.

@jonahgabriel
jonahgabriel merged commit 178680b into dev Jul 12, 2026
113 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-14466-rt-7-fail-loud-releasedeploy-drift-monitor branch July 12, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant