feat(OMN-12559): auto-discover omnimarket node migrations into infra forward set

[jonahgabriel]

OMN-12559: auto-discover omnimarket node migrations into the infra forward set

Implements OMN-12559.

Addresses the integration gap surfaced by OMN-12489 (PR #1005): omnimarket projection nodes ship SQL under src/omnimarket/nodes/*/migrations/*.sql, but the forward-migration runner only read docker/migrations/forward/. Landing the delegation/savings views previously needed a manual host copy + a hand-renumber (savings 076 -> 084) to dodge a numeric collision with the flat infra 076. This eliminates that renumber-as-operational-pattern.

What changed

• Vendored node migrations under docker/migrations/forward/nodes/<node>/, read verbatim from omnimarket origin/dev (HEAD 26a343de): node_projection_delegation 0007–0010 (base tables + dashboard views) and node_projection_savings 076 (savings view).
• scripts/run-forward-migrations.sh: new discovery pass walks forward/nodes/<node>/*.sql and applies each under a namespaced node:<node>:<file> migration_id — a separate identity space from the flat docker/<file> sequence, so a node 076 never collides with infra flat 076. No renumber required.
• scripts/sync-node-migrations.sh: auto-discovers + vendors node migrations from an installed/cloned omnimarket (allowlist-scoped via nodes/.synced-nodes), with a --check drift guard wired as a pre-commit hook (enforcement, not detection).
• scripts/validation/validate_migration_sequence.py: excludes the namespaced nodes/ subtree from the flat sequence check.
• docker/Dockerfile.migrate: COPY is recursive, so the nodes/ subtree ships in the migrate image automatically.

dod_evidence (fresh-clone reproduction proof)

• Ran the real run-forward-migrations.sh against a throwaway local Postgres pointed at a temp MIGRATIONS_DIR (savings base 074 + the nodes/ subtree). The runner auto-discovered and applied all 5 node migrations under namespaced ids and created projection_delegation_summary, projection_delegation_model_routing, projection_delegation_quality_gate, projection_delegation_token_usage, projection_delegation_savings — with no manual copy. Re-run = all skipped (idempotent). Infra flat 076 and node 076 coexist as distinct schema_migrations rows (no collision).
• Tests: tests/unit/migrations/ + tests/integration/migrations/ + tests/ci/test_validate_migration_sequence.py green. Full unit suite: 19981 passed, 15 skipped. CI validator suite: 133 passed. The integration test that mutates a DB is opt-in (RUN_NODE_MIGRATION_DISCOVERY_TEST=1) and skips by default.
• pre-commit run --all-files: clean (also fixed a pre-existing SPDX year violation in test_handler_wiring_handle_async_dispatch.py).

Contract: contracts/OMN-12559.yaml (in-repo, deploy-gate) and the central OCC contract + receipts (Evidence-Source below).

Deploy-gate: no live deploy performed pre-merge; the next migrate image build carries the nodes/ subtree and the discovery runner.

Evidence-Ticket: OMN-12559
Evidence-Source: OCC#2080

Summary by CodeRabbit

• New Features

  • Added auto-discovery and application of node-owned database migrations with namespaced tracking
  • Created delegation events and generation events tables with comprehensive metrics
  • Added dashboard views for delegation summary, model routing, quality gates, token usage, and savings projections

• Infrastructure/Chores

  • Added pre-commit hook for migration sync validation
  • Enhanced migration runner to support node migration discovery with proper namespacing
  • Added deployment gate contract for node migration changes

• Tests

  • Added integration and unit tests for node migration discovery and validation

[coderabbitai]

Warning

Review limit reached

@jonahgabriel, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 39 minutes and 27 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fbcbaab1-6044-4b33-98d6-4619020d5020

📥 Commits

Reviewing files that changed from the base of the PR and between 7117d0f and 6e97b7f.

📒 Files selected for processing (24)

• .github/workflows/ci.yml
• .github/workflows/reusable-runtime-boot.yml
• .pre-commit-config.yaml
• contracts/OMN-12559.yaml
• docker/Dockerfile.migrate
• docker/migrations/forward/nodes/.synced-nodes
• docker/migrations/forward/nodes/node_projection_delegation/0007_delegation_events.sql
• docker/migrations/forward/nodes/node_projection_delegation/0008_generation_events.sql
• docker/migrations/forward/nodes/node_projection_delegation/0009_delegate_skill_projection_metrics.sql
• docker/migrations/forward/nodes/node_projection_delegation/0010_create_delegation_dashboard_projection_views.sql
• docker/migrations/forward/nodes/node_projection_savings/076_create_delegation_savings_projection_view.sql
• docker/runners/Dockerfile
• docker/runners/runner-image.lock.json
• pyproject.toml
• scripts/run-forward-migrations.sh
• scripts/sync-node-migrations.sh
• scripts/validation/validate_migration_sequence.py
• tests/ci/test_ci_workflow_resilience.py
• tests/ci/test_runner_image_node24_floor.py
• tests/ci/test_validate_migration_sequence.py
• tests/integration/migrations/__init__.py
• tests/integration/migrations/test_node_migration_discovery_applies.py
• tests/unit/migrations/__init__.py
• tests/unit/migrations/test_node_migration_discovery.py

📝 Walkthrough

Walkthrough

This PR implements node-owned migration auto-discovery and vendoring for omnimarket node projects. It introduces a sync script, updates the forward migration runner to apply node migrations with namespaced tracking, excludes node migrations from flat sequence validation, adds delegation and savings projection tables and views, and provides comprehensive unit and integration tests.

Changes

Node Migration Discovery & Vendoring

Layer / File(s)	Summary
Node migration allowlist configuration docker/migrations/forward/nodes/.synced-nodes	The .synced-nodes file defines which omnimarket node migration files are vendored into the forward migrations directory, with per-node glob filtering (e.g., node_projection_delegation for all files, node_projection_savings:076_* for basename pattern matching).
Forward migration runner with node discovery scripts/run-forward-migrations.sh	After applying infra migrations, the runner scans NODE_MIGRATIONS_DIR for node subdirectories, applies *.sql files in sorted order, and records each as node:<node>:<filename> in schema_migrations, keeping node and flat docker namespaces distinct.
Sync script for vendoring node migrations scripts/sync-node-migrations.sh	Auto-discovers node-owned migrations from omnimarket source (via OMNIMARKET_SRC, OMNI_HOME, or package lookup), filters by allowlist, vendors into forward migrations tree, and detects drift in --check mode used by pre-commit.
Exclude node migrations from sequence collision detection scripts/validation/validate_migration_sequence.py, tests/ci/test_validate_migration_sequence.py	Migration sequence validation treats docker/migrations/forward/nodes/ as a separate namespace, excluding node migrations from flat numeric sequence-uniqueness checks; two new tests verify that node migrations with the same prefix do not cause false collisions with flat migrations.
Delegation event tables with metrics docker/migrations/forward/nodes/node_projection_delegation/0007_delegation_events.sql, 0008_generation_events.sql, 0009_delegate_skill_projection_metrics.sql	Creates delegation_events and generation_events base tables with UUID primaries and unique correlation IDs; extends delegation_events with latency, token, and pricing metrics, including an expression index on total tokens.
Delegation dashboard projection views docker/migrations/forward/nodes/node_projection_delegation/0010_create_delegation_dashboard_projection_views.sql	Four aggregated read-optimized views over delegation_events: projection_delegation_summary (totals and rates), projection_delegation_model_routing (per-model distribution), projection_delegation_quality_gate (failure analysis), and projection_delegation_token_usage (cost and provenance), each with scalar and JSONB breakdowns.
Delegation savings projection view docker/migrations/forward/nodes/node_projection_savings/076_create_delegation_savings_projection_view.sql	Creates projection_delegation_savings view that normalizes and merges savings_estimates and delegation_events by session, deduplicates, limits to 500 recent rows, and returns cumulative metrics with per-session JSONB.
Node migration discovery unit tests tests/unit/migrations/test_node_migration_discovery.py, tests/unit/migrations/__init__.py	Unit tests verify vendored view/base-table migrations exist, runner uses namespaced discovery IDs, allowlist includes required nodes, and sync-script matches vendored tree to omnimarket source.
Node migration discovery integration tests tests/integration/migrations/test_node_migration_discovery_applies.py, tests/integration/migrations/__init__.py	Integration tests run real forward-migrations runner, verify views are created and queryable, confirm node:<node>:<file> migration IDs are recorded, and assert node migrations with same numeric prefix do not collide with flat migrations.
Pre-commit hook and deploy-gate contract .pre-commit-config.yaml, contracts/OMN-12559.yaml	Pre-commit hook runs sync-node-migrations.sh in --check mode; OMN-12559 deploy-gate contract enumerates validation commands for discovery, sequence behavior, drift checks, view creation, and migration runner in next image.
Documentation and miscellaneous updates docker/Dockerfile.migrate, pyproject.toml, tests/unit/runtime/auto_wiring/test_handler_wiring_handle_async_dispatch.py	Dockerfile.migrate adds comments on node-owned migration vendoring and namespace semantics; pyproject.toml adds S608 lint ignore for integration test; test headers updated with SPDX copyright; unrelated copyright year corrected.

🎯 4 (Complex) | ⏱️ ~60 minutes

🐰 Migrations dance in namespaced rows,
Views bloom where delegation flows,
Nodes discover, sync, and apply—
No flat collisions touch the sky!
A vendor's tale in SQL and Bash. 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 47.83% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the primary change: implementing auto-discovery of omnimarket node migrations (OMN-12559) and their integration into the infra forward-migration set.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jonahgabriel/omn-12559-auto-discover-node-migrations

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jonahgabriel]

Codex foreground tick: fixed the Receipt Gate metadata by replacing the placeholder Evidence-Source: OCC#__OCC_PR__ with Evidence-Source: OCC#2007, then reran the failed Receipt Gate run 26720541331. Also found OCC PR #2007 had a skip-receipt-gate token in its PR body; removed that token and reran the failed OCC receipt/reject-token gates there before relying on the evidence source.

[jonahgabriel]

Codex foreground tick: OCC#2007 now has a fresh pushed evidence commit 8a7fc1142 after local OCC pre-commit passed. Reran #1820 Receipt Gate run 26720541331 again so it re-evaluates Evidence-Source: OCC#2007 against the updated central evidence head.

[jonahgabriel]

Codex foreground tick: added missing Evidence-Ticket: OMN-12559 alongside Evidence-Source: OCC#2007. OCC#2007 Receipt Gate is now green on 8a7fc1142; rerunning #1820 Receipt Gate again after this PR-body schema fix.

[jonahgabriel]

Codex foreground tick: Receipt Gate run 26720541331 attempt 4 completed success after adding Evidence-Ticket: OMN-12559 and using updated OCC#2007 head 8a7fc1142. Current #1820 rollup has no failed Receipt Gate context.

[jonahgabriel]

Codex foreground tick: OCC#2007 is now merged with all fresh checks green, so OMN-12559 central evidence is durable in onex_change_control. Cleaned up the merged OCC worktree under $OMNI_HOME/omni_worktrees/OMN-12559/onex_change_control; no matching legacy external worktree existed under /Users/jonah/Code/omni_worktrees.

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

scripts/sync-node-migrations.sh (1)

168-205: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Detect/prune stale vendored migrations in drift checks.

Current logic only iterates allowlisted source files. Extra files already present under docker/migrations/forward/nodes/** are never flagged (check mode) or removed (sync mode), so obsolete migrations can still be applied.

Suggested fix

 DRIFT=0
 COPIED=0
+EXPECTED_REL="$(mktemp)"
+trap 'rm -f "${EXPECTED_REL}"' EXIT
 
 # Discover every node migration file and mirror it (allowlisted nodes only).
 while IFS= read -r src_file; do
@@
   filename="$(basename "${src_file}")"
   if ! file_is_allowed "${node_name}" "${filename}"; then
     continue
   fi
+  printf '%s/%s\n' "${node_name}" "${filename}" >> "${EXPECTED_REL}"
   dest_dir="${DEST_ROOT}/${node_name}"
   dest_file="${dest_dir}/${filename}"
@@
 done < <(find "${NODES_DIR}" -type f -path "*/migrations/*.sql" | sort)
+
+# Detect vendored files that are no longer expected (removed upstream or de-allowlisted).
+while IFS= read -r existing_dest; do
+  rel="${existing_dest#${DEST_ROOT}/}"
+  if ! grep -Fxq "${rel}" "${EXPECTED_REL}"; then
+    if [ "${CHECK_MODE}" -eq 1 ]; then
+      echo "[sync-node-migrations] DRIFT (stale vendored file): ${rel}" >&2
+      DRIFT=1
+    else
+      rm -f "${existing_dest}"
+      echo "[sync-node-migrations]   removed stale ${rel}"
+      COPIED=$((COPIED + 1))
+    fi
+  fi
+done < <(find "${DEST_ROOT}" -type f -name "*.sql" | sort)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/sync-node-migrations.sh` around lines 168 - 205, The script only
checks source migrations and never detects or removes extra files under
DEST_ROOT, so stale vendored migrations remain; add a second pass that discovers
every dest_file (find "${DEST_ROOT}" -type f -path "*/migrations/*.sql") and for
each compute node_name and filename (same basename/dirname logic), skip if
file_is_allowed(node_name,filename) is false, then if the corresponding src_file
("${NODES_DIR}/${node_name}/migrations/${filename}") does not exist treat it as
drift: in CHECK_MODE set DRIFT=1 and echo a DRIFT message, else in sync mode
delete the dest_file (and remove empty parent dir if desired), log removal and
increment a counter (or separate REMOVED counter) so output reflects removals;
ensure checks use the same file_is_allowed, CHECK_MODE, DRIFT, COPIED symbols
and produce consistent messages as existing logic.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@docker/migrations/forward/nodes/node_projection_savings/076_create_delegation_savings_projection_view.sql`:
- Around line 45-46: The COALESCE fallback for the latency column currently uses
delegation_latency_ms before latency_ms and should prefer the canonical
latency_ms first; update the projection so latency_ms is chosen when present
(i.e., use COALESCE(latency_ms, delegation_latency_ms)::int AS latency_ms) to
avoid returning stale delegation_latency_ms values while leaving the created_at
line unchanged.

In `@tests/ci/test_validate_migration_sequence.py`:
- Around line 225-264: Add the pytest unit marker to the two new test functions
in tests/ci/test_validate_migration_sequence.py: annotate
test_node_subtree_file_does_not_collide_with_flat_sequence and
test_node_subtree_excluded_even_when_flat_migration_staged with
`@pytest.mark.unit`; ensure pytest is imported at the top of the file if not
already (add import pytest) so the decorator resolves.

In `@tests/integration/migrations/test_node_migration_discovery_applies.py`:
- Around line 61-103: The tests call PostgresConfig.from_env() and run
subprocesses in _require_db, _query, and the test runner without first sourcing
~/.omnibase/.env and they replace the whole environment when calling
subprocess.run; update _require_db, any places that call
PostgresConfig.from_env(), _query, and the runner invocation to first load and
merge the dotenv (e.g. parse ~/.omnibase/.env into a dict) into os.environ (or
into a new env = {**os.environ, **dotenv_vars}) before using
PostgresConfig.from_env() and pass that merged env to subprocess.run instead of
the current env={...} overwrite so inherited vars (like DB DSN/KV settings) are
preserved for subprocesses and config resolution.

---

Outside diff comments:
In `@scripts/sync-node-migrations.sh`:
- Around line 168-205: The script only checks source migrations and never
detects or removes extra files under DEST_ROOT, so stale vendored migrations
remain; add a second pass that discovers every dest_file (find "${DEST_ROOT}"
-type f -path "*/migrations/*.sql") and for each compute node_name and filename
(same basename/dirname logic), skip if file_is_allowed(node_name,filename) is
false, then if the corresponding src_file
("${NODES_DIR}/${node_name}/migrations/${filename}") does not exist treat it as
drift: in CHECK_MODE set DRIFT=1 and echo a DRIFT message, else in sync mode
delete the dest_file (and remove empty parent dir if desired), log removal and
increment a counter (or separate REMOVED counter) so output reflects removals;
ensure checks use the same file_is_allowed, CHECK_MODE, DRIFT, COPIED symbols
and produce consistent messages as existing logic.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c24bbc68-4004-43da-8873-4b8df8c7d7dd

📥 Commits

Reviewing files that changed from the base of the PR and between 10a3b86 and 7117d0f.

📒 Files selected for processing (19)

• .pre-commit-config.yaml
• contracts/OMN-12559.yaml
• docker/Dockerfile.migrate
• docker/migrations/forward/nodes/.synced-nodes
• docker/migrations/forward/nodes/node_projection_delegation/0007_delegation_events.sql
• docker/migrations/forward/nodes/node_projection_delegation/0008_generation_events.sql
• docker/migrations/forward/nodes/node_projection_delegation/0009_delegate_skill_projection_metrics.sql
• docker/migrations/forward/nodes/node_projection_delegation/0010_create_delegation_dashboard_projection_views.sql
• docker/migrations/forward/nodes/node_projection_savings/076_create_delegation_savings_projection_view.sql
• pyproject.toml
• scripts/run-forward-migrations.sh
• scripts/sync-node-migrations.sh
• scripts/validation/validate_migration_sequence.py
• tests/ci/test_validate_migration_sequence.py
• tests/integration/migrations/__init__.py
• tests/integration/migrations/test_node_migration_discovery_applies.py
• tests/unit/migrations/__init__.py
• tests/unit/migrations/test_node_migration_discovery.py
• tests/unit/runtime/auto_wiring/test_handler_wiring_handle_async_dispatch.py