fix(OMN-15701): restore tenant_projection_writer TABLE grants reverted by #2632's stale-pin regeneration

[jonahgabriel]

What broke

omninode-runtime and omninode-runtime-effects CrashLoopBackOff on onex-dev (deployed digest sha256:5507667152e2…, infra dev SHA 2e5ef7da) with ValueError: Projection binding 'tenant_projection' principal 'tenant_projection_writer' lacks declared write priv for 8 contracts (canary_score_reducer, projection_context_roi, projection_cost_summary, projection_dep_health, projection_instruction_eval, projection_pattern_learning, projection_routing_decision, projection_skill_executions).

Root cause (systematic-debugging trace, full evidence on OMN-15701 / OMN-15655 comment 132a9d8b + correction comment e5b5cb46)

_require_projection_binding_privileges (handler_wiring.py:1855) is a pure static declaration check — no DB I/O. It compares each contract's required table privileges against principal.grants parsed from the shipped topology instance file (src/omnibase_infra/topology/instances/onex-dev.yaml, baked into the image, selected via ONEX_DATABASE_TOPOLOGY_PROFILE=onex-dev — live-confirmed correct in the onex-runtime-config ConfigMap).

• #2634 (merged 2026-08-03T05:05:30Z) correctly derived tenant_projection_writer TABLE grants for all nine house-tenant relations.
• #2632 (merged 2026-08-03T18:06:11Z, an ancestor of the deployed SHA) contains a refresh topology grants after dev rebase sub-commit that re-ran scripts/generate_application_database_table_grants.py --write against the CI workflow's hardcoded fallback omnimarket pin (ci.yml:2330 → 4637e625c99ef17c190aa471a5e51b7f646c6dfd, dated 2026-07-30) rather than omnimarket's actual dev HEAD. That pin still declares schema: omninode_internal for these relations. The regeneration silently reverted 8 of the 9 relations back to omninode_runtime/omninode_internal, and fully dropped delegation_judge_verdict_events and the omninode_runtime SELECT grant on nightly_loop_configs.
• #2632's own derivation-consistency gate reported PASS because it compared the regenerated branch against the same stale pin — a vacuous self-consistency check, not real cross-repo proof. This is the OMN-14208 seam-mismatch pattern.

Corrected timeline (round-1 remediation, 2026-08-04): the pivotal omnimarket commit is 8e27c27a (feat(OMN-15423): classify application relation ownership, merged 2026-07-31T11:50:07-04:00), which flipped node_canary_score_reducer/contract.yaml's schema omninode_internal → public — genuinely before both the stale pin's date and #2632's merge. A prior revision of this PR misidentified 485be549 as the pivotal commit and claimed it landed after #2632 merged; live timestamps show 485be549 (committer date 2026-08-03T15:56:54Z UTC) landed ~2h10m before #2632 merged (18:06:11Z), not after, and it is grant-equivalent to 8e27c27a anyway (public→tenant, both map to EnumDatabaseSchemaDomain.TENANT per application_database.py:113-118 and both derive to tenant_projection_writer) — it is not the commit the stale pin actually missed.

Independent, unrelated defect (do not conflate): the deploy's migrate job logged WARNING: no privileges were granted for "public" on each of the 9 RLS/tenant_id migrations — the GRANT USAGE ON SCHEMA public TO app_dashboard; statement in each migration. Mechanism correction (round-1 remediation): a prior revision called this a "benign, self-documented idempotent re-grant" — that mechanism is wrong. Reproduced on a scratch Postgres 14 cluster: a superuser/owner re-running the identical GRANT twice is silent (no warning at all — an idempotent re-grant cannot produce this message); a non-owner, non-superuser grantor running it produces the byte-identical WARNING: no privileges were granted for "public"; the same non-owner grantor attempting GRANT SELECT on a table it doesn't own raises a hard ERROR: permission denied, not a warning. So the warning proves the migration runner lacks USAGE WITH GRANT OPTION on schema public and the statement granted nothing — a silently-failing privilege statement, not a redundant no-op — while the sibling per-table GRANT SELECT succeeding without error proves the runner does own the tables. The "unrelated to the crash" conclusion is unchanged (the auto-wiring check never touches the DB); only the stated Postgres mechanism was wrong.

Fix

Regenerated local.yaml/onex-dev.yaml/onex-prod.yaml (+ the 7 rendered docker/catalog/database-topology/*.yaml catalogs) via the canonical authoring path:

uv run python scripts/generate_application_database_table_grants.py --write \
  --contracts-root <omnimarket>/src/omnimarket/nodes   # omnimarket dev HEAD 54356a83, includes 8e27c27a + 485be549

--check --prove confirms 43/43 PASS, 0 FAIL, 0 RESIDUAL on all 7 shipped profiles (including the omniintelligence residual from #2632, now clean too).

Omnimarket-Source-Ref: 54356a831e3d8876c69373cac884a3df2a5653f7

Correction (remediation round 2, 2026-08-04): the pin bump described below did NOT land in this PR. This PR was merged externally (mergedAt 2026-08-04T18:40:34Z, merge commit 57c3e7e) before the pin-bump commit could be pushed to this branch — dev still carries the stale pin (4637e625) as of this correction, confirmed via git show origin/dev:.github/workflows/ci.yml. The bump (.github/workflows/ci.yml:2330's hardcoded omnimarket fallback pin, 4637e625 (2026-07-30) → 54356a831e3d8876c69373cac884a3df2a5653f7, omnimarket dev HEAD as of 2026-08-03, the same SHA this PR's own trailer uses) instead ships as a separate PR, omnibase_infra#2657, as a point-in-time mitigation, not a structural fix. Without it, the gate's own --check step fails on the fixed topology files even with this PR's own regeneration — confirmed by running --check --prove against the pre-bump pin, which reports all three instance files drifted and instructs --write, which against the still-stale pin would silently reproduce this exact #2632 revert on the next PR that lacks a trailer. Structural fix tracked as a new follow-up: OMN-15703.

Trailer vacuity caveat: OMN-15701 AC-2 says the Omnimarket-Source-Ref: trailer makes the gate "non-vacuous." The trailer is validated (resolve_node_migration_source_ref.py::_parse_ref) only against a character-class regex — nothing checks the SHA is an ancestor of omnimarket dev. An author can point the gate at any omnimarket SHA that makes their own diff self-consistent — same vacuity class as the stale pin, just author-controlled. For this PR it is genuinely non-vacuous (trailer SHA independently confirmed = omnimarket dev HEAD, contains both pivotal commits, reproduces 43/43 PASS against the canonical clone) — the general claim doesn't hold for future PRs.

Regression test

Follows the existing OMN-15547 incident-replay pattern in tests/unit/topology/test_application_database_table_grants.py:

• tests/fixtures/omn15701/onex-dev-topology-reverted-tenant-grants.yaml.captured — byte-identical to the shipped instances/onex-dev.yaml at infra dev 2e5ef7da (the exact bytes the crash-looped pods loaded).
• test_omn15701_replay_captured_topology_reproduces_the_reverted_grant_failure / ..._fails_all_eight_reverted_relations — RED: reproduces the exact observed ValueError against the captured (broken) topology for all 8 relations.
• test_omn15701_replay_shipped_topology_now_accepts_all_reverted_relations / ..._restore_nightly_loop_configs_read — GREEN: all nine relations + the nightly_loop_configs read resolve cleanly on every shipped profile after the fix.

Verification

• uv run pytest tests/unit/topology/ -q → 105 passed (local Mac; no service dependency).
• ruff format / ruff check --fix clean.
• mypy clean on the touched test file (5 unrelated pre-existing errors in tests/helpers/{statistics_utils,replay_utils}.py, not touched by this PR).
• pre-commit run on the changed files: all applicable hooks passed, including Incident-replay coverage (OMN-15547).
• Pre-push governed selector (OMN-13973) escalated to the full local suite (multi-repo topology-file diff triggered the shared-module escalation path) and completed clean: 23161 passed, 0 failed, 40 skipped, 663.88s. Run on this Mac, not .200, because SSH to stickybeatz-studio failed with a publickey auth error in this session — stated exception per rule 11a.
• Live CI (2026-08-04, post round-1 remediation): Application Database Domain Enforcement (OMN-15361) grant-derivation step (generate_application_database_table_grants.py --check --prove) reports grant derivation check: 3 instance(s) in sync, 43/43 PASS, 0 FAIL, 0 RESIDUAL on all 7 profiles, confirming the fix + trailer independently of local runs. That job's overall conclusion is still red — see "Known open red" below; the grant check itself is green.

Known open red (2026-08-04, live, unresolved)

Application Database Domain Enforcement (OMN-15361) fails on its Rebuild and prove generated role and default-privilege gates step (docker/application-acl-proof compose), not the grant-check step above: AssertionError: psql ACL apply failed: ERROR: deadlock detected (proof/prove.py:1958, LOCK TABLE pg_catalog.pg_type IN SHARE MODE colliding with autovacuum ANALYZE on the same relation) — after all content phases (scaffold, 14 red controls, apply pass 1/2) already passed. First occurrence; looks like harness/container concurrency noise rather than a content defect, but it is a real, currently-blocking red — mergeStateStatus is BLOCKED, not clean. Not silently claimed green.

Also live as of this remediation round: occ-preflight/Hostile Review Gate/verify / verify — the three reds declared in the original report — cleared after bot PR onex_change_control#6072 merged (2026-08-04T18:13:49Z) and unblocked eligibility.

Correction (remediation round 2, 2026-08-04): the two claims above ("currently-blocking", "mergeStateStatus is BLOCKED", "CI Summary ... also now blocked") were true when written but became false ~1 minute before merge and are stale on this now-merged PR. Live check-runs on head b6c302ad0c: a second Application Database Domain Enforcement (OMN-15361) run (job 92093566188, 18:36:07Z–18:39:08Z) superseded the failed job 92089701605 (18:21:45Z) with conclusion=success; CI Summary reported success at 18:39:33Z; the PR merged clean at 2026-08-04T18:40:34Z (not 2026-08-03 as stated elsewhere in this thread — corrected here). The deadlock was a first-occurrence, non-reproducing red, not a merge-time blocker.

Not done in this PR (by design)

• No OCC companion authored — per repo convention a different, non-implementer agent authors the OCC evidence companion for this PR. (One already exists/merged: onex_change_control#6072.)
• No redeploy triggered. Proving this fix requires the onex-dev runtime image to be rebuilt + repinned at a SHA that includes this commit and then redeployed — that is the next lane's job, not this one's.
• Structural fix for the CI pin as a recurring staleness vector: tracked separately as OMN-15703. The one-time pin bump itself did NOT land in this PR either (see the "Correction (remediation round 2)" note above) — it ships in a separate PR, omnibase_infra#2657.
• Whether tenant_projection_writer holds live Postgres grants in the deployed DB: scripts/ci/prove_application_database_acl.py + docker/application-acl-proof/seed.sql both name the role and exercise a generated-ACL-against-live-Postgres path that may already answer this — not confirmed here, flagged as open on OMN-15701.

Refs: OMN-15655 (parent, umbrella ticket + root-cause comment + correction comment e5b5cb46), OMN-15701 (this defect), OMN-15703 (CI-pin staleness follow-up), OMN-14208 (seam-matching doctrine), OMN-15656 (grant derivation authoring path)
Evidence-Ticket: OMN-15701

Evidence-Source: OCC#6072

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 48 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 33cbb775-9db2-472f-b801-4d3b0985b606

📥 Commits

Reviewing files that changed from the base of the PR and between 1d1c2ad and b6c302a.

📒 Files selected for processing (12)

• docker/catalog/database-topology/judge.yaml
• docker/catalog/database-topology/local.yaml
• docker/catalog/database-topology/onex-dev.yaml
• docker/catalog/database-topology/onex-prod.yaml
• docker/catalog/database-topology/prod.yaml
• docker/catalog/database-topology/stability-test.yaml
• docker/catalog/database-topology/test.yaml
• src/omnibase_infra/topology/instances/local.yaml
• src/omnibase_infra/topology/instances/onex-dev.yaml
• src/omnibase_infra/topology/instances/onex-prod.yaml
• tests/fixtures/omn15701/onex-dev-topology-reverted-tenant-grants.yaml.captured
• tests/unit/topology/test_application_database_table_grants.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8001 ] unreachable — preflight short-circuit (no models available)

---

Gate semantics (pilot phase)

Verdict	Meaning	Blocks merge?
passed	No critical findings	No
blocked	CRITICAL findings found	Yes
degraded	All models unavailable (infra)	No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)