Repository navigation
feat(OMN-14466): RT-7 fail-loud release/deploy drift monitor - #2272
jonahgabriel merged 2 commits into
Conversation
Standalone scheduled monitor that surfaces every silent release/deploy divergence same-day instead of after six weeks of green-and-unread silence. Per repo it compares PyPI-published version vs latest v* tag vs main/dev pyproject version, flags [tool.uv.sources] git-override drift (main carrying an override dev already dropped), and checks the last release.yml / runtime-rebuild-trigger.yml run. On divergence it writes a friction record into the existing friction registry (.onex_state/friction/, consumed by node_friction_triage_orchestrator -> Linear) and exits non-zero so the scheduled workflow goes RED. No new store; reuses the env_sync_alert friction precedent. - scripts/release_drift_monitor_lib.py: pure evaluator (no I/O), unit-tested against EXISTS-but-WRONG (fires) and aligned (green) facts. - scripts/release_drift_monitor.py: I/O shell (gh api + PyPI JSON) + friction emit. - .github/workflows/release-drift-monitor.yml: scheduled (every 6h) + dispatch. - tests/scripts/test_release_drift_monitor.py: 12 tests, incl. live-drift RED. dod_evidence: live run against GitHub/PyPI on 2026-07-12 exits 1 with 13 findings incl. omnibase_infra PyPI 0.36.1 behind tag v0.38.3 (P1), omnibase_infra release.yml last run = failure (P1), onex_change_control main stale omnibase-core [tool.uv.sources] override (P1). blind=false, 0 probe errors. 12/12 unit tests pass; ruff + mypy clean; pre-commit green. Epic OMN-13674. Design: docs/plans/2026-07-12-mechanical-release-trains.md section 4 (RT-7).
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 23 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
OMN-14466 — RT-7: fail-loud release/deploy drift monitor
Standalone scheduled monitor (Mechanical Release Trains, design
docs/plans/2026-07-12-mechanical-release-trains.md§4 RT-7) that surfaces every silent release/deploy divergence same-day instead of after six weeks of green-and-unread silence. Epic OMN-13674.What it checks (per repo)
v*tag vsmainvsdevpyproject version — fires on PyPI-behind-tag (stuck publish), main-ahead-of-tag (bump never tagged), main-behind-released (stale main), and a large dev→main backlog.[tool.uv.sources]override drift:maincarrying a git override thatdevalready dropped (the OCComnibase-core rev=f325e759case).release.yml/runtime-rebuild-trigger.ymlrun failed, or stalled whilemainis ahead of the tag.Output — feeds the existing substrate, no new store
On divergence it writes a friction record into the existing friction registry (
.onex_state/friction/, consumed bynode_friction_triage_orchestrator→ Linear) and exits non-zero so the scheduled workflow goes RED (durable, observable, same-day). Friction-emit shape mirrorsnode_env_sync_alert_effect, the platform's existing scheduled drift→friction→Linear precedent. It builds no new store.Seam
NEW standalone files only. Does not modify
stage_workspace.sh,deploy-runtime.sh,release.yml, or the trigger scripts (other RT lanes own those).scripts/release_drift_monitor_lib.py— pure evaluator (no I/O)scripts/release_drift_monitor.py— I/O shell (gh api+ PyPI JSON) + friction emit.github/workflows/release-drift-monitor.yml— scheduled (every 6h) +workflow_dispatchtests/scripts/test_release_drift_monitor.py— 12 unit testsThe per-lane running-signature check (
.201container signature vs intended ref) is deliberately out of scope here — it needs live SSH/introspection and is delivered by the RT-6 deploy-readback lane. No fake lane check is emitted.dod_evidence — RED reproduces on TODAY's real drift (not a fixture)
Live run
uv run python scripts/release_drift_monitor.py --jsonon 2026-07-12 → exit 1,diverged: true,blind: false, 0 probe errors, 13 findings across 6 repos, including:PYPI_BEHIND_TAGP1 —omnibase-infraPyPI 0.36.1 behind latest tag v0.38.3 (the six-week-stuck case)RELEASE_WORKFLOW_FAILEDP1 —omnibase_infrarelease.ymllast run = failure (2026-05-27)MAIN_STALE_UV_OVERRIDEP1 —onex_change_controlmain still pins theomnibase-core[tool.uv.sources]git override that dev droppedMAIN_AHEAD_OF_TAG/RELEASE_TRAIN_STALLED(infra 0.38.4 never tagged, release 46d stale);MAIN_BEHIND_RELEASED(core/compat/market/OCC);DEV_AHEAD_OF_MAIN(core +170, infra +294, market +547, OCC +1440)The unit suite proves the check discriminates (per
feedback_prove_red_against_exists_but_wrong):test_live_*fire on the exact live-shaped facts;test_aligned_repo_is_greenpasses on aligned facts, and its siblings flip one field to confirm the same check goes RED — a green means "checked and matched", not "couldn't see". A blind run (probe errors, no findings) exits 2, never a false green.Local gates: 12/12 unit tests pass;
ruff check+ruff formatclean;mypyclean; fullpre-commitgreen (SPDX, AI-slop, url-authority annotated for the PyPI public API, os.environ-block all pass).Notes
pr_statesubstrate (OMN-14375/14374) is PR-scoped(repo, pr_number)— the wrong schema for release/deploy drift; the friction registry is the correct existing durable substrate reused here.CROSS_REPO_PATin CI to read sibling repos; with onlyGITHUB_TOKENthe sibling probes fail and the monitor exits 2 (BLIND) rather than a false green.Evidence-Source: OCC#4048
Evidence-Commit: 15a088606656c9f62f97832b59e752695ac77bd0
Evidence-Ticket: OMN-14466