[jonahgabriel]

Summary

G3 of the gateway lift Phase 0 plan (docs/design/2026-08-08-gateway-node-architecture-lift.md, commit 9422d30b1, §3): a link-health projection materializing per-tenant-edge gateway liveness, driven by the contract's existing liveness thresholds, alerting on absence of progress rather than a missing row.

Adds two nodes mirroring the canonical node_pr_state_projection_compute / node_pr_state_write_effect ingress→projection pattern:

• node_gateway_link_health_projection_compute (COMPUTE) — subscribes to onex.evt.omnibase-infra.gateway-heartbeat.v1, emits a gateway_link_health.upsert intent.
• node_gateway_link_health_write_effect (EFFECT) — consumes the intent, UPSERTs one row per tenant_id into public.gateway_link_health (latest-known-state, ON CONFLICT (tenant_id) DO UPDATE — the row is never deleted).
• Migration docker/migrations/forward/100_create_gateway_link_health.sql — creates the table plus a read-time view, gateway_link_health_status. This is the load-bearing design choice: health is evaluated at query time, not stored at write time, so a tenant edge that goes silent shows up as a stale row, not a missing one — matching the ticket's "alert on absence of progress" requirement exactly. The view combines two independent signals: recency (NOW() vs last_seen_at against a 60s silence window) and the edge's own self-reported status. See the precedence table below.

---

Rebase onto current dev — 2026-08-18

The branch had been open 9 days and dev moved substantially underneath it. It has been rebased onto current dev and adapted to dev's interfaces — dev is authoritative throughout; this branch adapts to it, never the reverse.

The gateway-forwarder half of this PR is gone, because dev already has it

The original 43-file diff has become 35 files. Nothing was lost. Eight files — the forwarder runtime, its config and heartbeat models, the bus-forwarder contract, and their tests — landed on dev independently via #2736, including the _supervise_gateway_delivery restart/backoff supervisor this branch introduced. Those files are now byte-identical between this branch and dev, so they simply stopped being a diff. Verified per file rather than assumed.

What remains is exactly this ticket's own scope: the durable projection, its two nodes, the migration, and their tests.

Migration renumbered 098 → 100 (silent collision, caught by hand)

dev landed 098_create_omninode_internal_schema.sql and 099_create_omninode_internal_live_events.sql while this branch was open. This branch's migration was also numbered 098.

git rebase reported no conflict. Two differently-named files added at the same ordinal are not a textual conflict, so the duplicate 098 would have merged silently and been found later, on a deployment, by whichever ordering the applier happened to pick. dev owns the numbering; this branch moves to 100. The forward SQL, the rollback SQL, and the pinning test were renamed, their cross-references updated, and the schema fingerprint restamped (85 migration files, verify passes).

Scope disclosure corrected — it had become false

Three comments (the migration header, the compute contract, and the payload model docstring) asserted that G1 (#2690) and G2 (#2688) "have not merged as of this ticket." Both have since merged, so that was simply wrong and has been fixed.

The conclusion those comments drew is still correct, for a different reason than stated: lag_messages and lag_seconds are still always NULL, because G2 widened ModelGatewayHeartbeat with status / consecutive_failures / detail — not with lag telemetry. The inert-lag-arm reasoning stands unchanged.

The gap this rebase exposed — resolved in this branch

The rebase surfaced a real correctness problem, and it has been fixed here rather than deferred.

G2 added a status field ("active" / "degraded") and a consecutive_failures counter to the very heartbeat topic this projection consumes, and the projection was not reading either. Health came from recency alone, so an edge that was heartbeating perfectly on schedule while reporting itself degraded scored HEALTHY. That is the worst failure mode an observability surface can have: the wrong answer is indistinguishable from the right one, so nobody goes looking.

This branch now persists both fields and adds a DEGRADED_SELF_REPORTED verdict.

Precedence is fixed, documented above the view body, and tested — stale > self-reported > lag > healthy:

Situation	Verdict	Why it ranks there
Silent past the window	UNHEALTHY	A silent edge's last self-report is stale too, so it cannot be trusted to still describe reality.
Heartbeating, reports degraded	DEGRADED_SELF_REPORTED	A first-party statement outranks a metric inferred about the edge.
Heartbeating, active, lag breach	DEGRADED_LAG	Unchanged, and still inert until a producer populates lag.
Fresh, active, no breach	HEALTHY

Two design choices worth flagging for review, both deliberate:

• reported_status is free TEXT, and the view tests <> 'active' rather than matching a closed set. A status the producer adds later (a drain state, say) is therefore read as not-healthy rather than either crashing the projection or being scored healthy by omission. Validating against a Literal here would stall the whole projection the first time the producer grew a state.
• consecutive_failures drives no verdict of its own. The producer already folds it into status; re-deriving a second verdict from it here would let the two disagree about the same edge. It is persisted as the evidence behind a degraded status, so an operator can tell a blip from a sustained failure without consulting process memory.

The projection requires both fields rather than defaulting them. Defaulting a missing status to "active" would score a malformed or non-canonical producer HEALTHY — and that is the one error this surface must not make, since silence is already visible as staleness but a wrongly-healthy verdict is not.

How the verdicts are proven

The verdict is SQL, not Python, so a unit test that greps the migration text would prove the string is present, not that the arms evaluate correctly. tests/integration/migrations/test_100_gateway_link_health_verdicts.py applies the real migration to a throwaway Postgres cluster (reusing the existing ephemeral-cluster fixture) and reads health_status back out of the actual view — 13 cases, all passing:

• each verdict individually, including that NULL lag never decides anything;
• both simultaneous-signal precedence pairs (stale + degraded, degraded + lag);
• an unrecognised status scoring not-healthy;
• degraded → recovered, and silence → returned, each asserting the row is updated in place rather than duplicated or deleted;
• consecutive_failures readable but verdict-neutral;
• forward-migration idempotency and rollback.

Confirmed RED-before: deleting the new WHEN reported_status <> 'active' arm fails 4 of those tests, including the degraded-beats-recency case. The suite is not vacuous.

A cheap always-runs guard in tests/unit/db/test_migration_100.py pins the order of the CASE arms, since CASE stops at the first match and the order is the precedence — that one still fires when Postgres tooling is absent and the live suite skips.

What remains open on the ticket

Lag is the one declared threshold family still unobservable: G1 and G2 both landed without adding lag telemetry, so lag_messages / lag_seconds stay NULL and their arms stay inert. That is a materially smaller gap than before — silence is observable through last_seen_at, and an unwell-but-talking edge is now observable through status.

The ticket's remaining residual is the gated .201 live readback, which is blocked on OMN-15521 providing the sanctioned deploy path and is explicitly approval-gated. Not claimed here.

Landing order — cleared

The original landing-order block is obsolete: #2689, #2688, #2690, and #2687 have all merged. Nothing gates this PR.

---

Scope disclosure (read before trusting lag columns)

• lag_messages / lag_seconds are persisted as nullable columns and always NULL today — no producer publishes lag data on the heartbeat topic.
• The view's DEGRADED_LAG arms (lag_threshold_messages: 500, lag_threshold_seconds: 120) are inert (NULL comparisons are neither TRUE nor FALSE in SQL) until a future producer populates them — they never contribute a false verdict.
• The enforceable signals today are the silence window and the edge's self-reported status.

No omnidash widget — projection + contract only. Read surface for Bret: SELECT tenant_id, health_status, seconds_since_last_seen, reported_status, consecutive_failures FROM gateway_link_health_status WHERE health_status != 'HEALTHY'; — a normal projection-style read, no bespoke endpoint.

Heartbeat local-bus mirror

ServiceGatewayForwarder.publish_heartbeat now mirrors the same, untransformed envelope onto the local bus alongside the cloud wire topic. The projection subscribes on the local canonical topic (in-cluster consumers read local topics, never the tenant-prefixed cloud wire topic); before this, heartbeats only ever reached the cloud leg and the projection would never have seen a live event.

This reuses the exact dual-publish shape publish_status already established, and publishes the same envelope object to both legs so the envelope and correlation identifiers stay identical across them rather than minting two envelopes for one liveness tick. Confirmed against current dev: _publish_with_delivery_retry, _local_bus, and _build_status_envelope all exist there with the signatures this change uses.

Sanctioned allowlist additions (documented, not gate-weakening)

• scripts/ci/infra-node-allowlist.txt — 4 entries for the new handler files, same class and justification as the existing node_pr_state_* entries (platform/infra observability, not business-domain).
• src/omnibase_infra/validation/validation_exemptions.yaml — 3 entries for tenant_id/principal_id fields typed str rather than UUID, same seam and justification as the existing model_gateway_heartbeat.py exemptions immediately above them (these payloads carry the heartbeat's DNS-safe tenant slug and MSK principal verbatim; the principal's t- prefix makes UUID an invalid representation).
• scripts/check_subscribe_wiring_health.py baseline allowlist — 2 entries: the gateway-link-health-upsert.v1 command topic (intent-routed, not Kafka-published, same pattern as the existing pr-state-upsert.v1 entry) and the gateway-heartbeat.v1 event topic (published imperatively by ServiceGatewayForwarder rather than via a contract-declared publish list — a pre-existing gap in node_bus_forwarder_effect's own contract that this PR's new consumer surfaces for the first time, not one it introduces).
• pyproject.toml — both new nodes registered as entry points, required for contract auto-discovery.

Verification

Run on this session's host, confirmed Stickybeatz-Studio.local via hostname before any host-identity claim.

• Focused tests for the two new nodes, the bus-forwarder seam, the migration, and the live-apply verdict proofs: 132 passed (13 of them the live-Postgres verdict suite).
• mypy --strict on both new node packages: no issues in 16 source files.
• Dispatch-selection parity plus the full database test directory: green.
• Parity baseline delta checked field-by-field against dev's: exactly the 2 new dispatchers, the 2 new probes, and their derived counters (contracts 113→115, routes 124→126). No dev entry dropped, so no regeneration was needed.
• scripts/check_schema_fingerprint.py verify — passes at 85 migration files.
• Migration ordinals after renumber: 095–099 from dev, 100 from this branch. No duplicates.
• Full pre-commit hook chain green on every commit; the pre-push governed selector escalated to the full suite on its own (shared-module rule) and ran to completion before the push was accepted.

Not merged by this lane. No live .201 or AWS mutation — build and PR only.

Ticket: OMN-15570

Evidence-Ticket: OMN-15570
Evidence-Source: OCC#6215

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your current included review allowance is based on your included PR review attempts over the past 7 days.

Next review available in: 37 minutes

Limit details: You’ve used the included review currently available. Your 125 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bb9233a0-9c10-45eb-b62b-9856c2444012

📥 Commits

Reviewing files that changed from the base of the PR and between 4a72cbc and 29b1308.

⛔ Files ignored due to path filters (1)

• src/omnibase_infra/enums/generated/enum_omnibase_infra_topic.py is excluded by !**/generated/**

📒 Files selected for processing (40)

• .github/workflows/ci.yml
• docker/migrations/forward/100_create_gateway_link_health.sql
• docker/migrations/rollback/rollback_100_create_gateway_link_health.sql
• docker/migrations/schema_fingerprint.sha256
• docker/runners/runner-image.lock.json
• docs/evidence/OMN-12548/determinism-audit.json
• pyproject.toml
• scripts/check_subscribe_wiring_health.py
• scripts/ci/infra-node-allowlist.txt
• src/omnibase_infra/nodes/node_bus_forwarder_effect/services/service_gateway_forwarder.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/contract.yaml
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/handlers/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/handlers/handler_gateway_link_health_projection.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/models/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/models/model_payload_gateway_link_health_upsert.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/node.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/registry/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_projection_compute/registry/registry_infra_gateway_link_health_projection.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/contract.yaml
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/handlers/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/handlers/handler_gateway_link_health_upsert.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/models/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/models/model_gateway_link_health_upsert_result.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/node.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/registry/__init__.py
• src/omnibase_infra/nodes/node_gateway_link_health_write_effect/registry/registry_infra_gateway_link_health_write.py
• src/omnibase_infra/validation/application_database_domain_enforcement.py
• src/omnibase_infra/validation/validation_exemptions.yaml
• tests/fixtures/dispatch_parity/baseline-selection-v2.json
• tests/integration/migrations/test_100_gateway_link_health_verdicts.py
• tests/unit/db/test_migration_100.py
• tests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_service.py
• tests/unit/nodes/node_gateway_link_health_projection_compute/__init__.py
• tests/unit/nodes/node_gateway_link_health_projection_compute/test_handler_gateway_link_health_projection.py
• tests/unit/nodes/node_gateway_link_health_write_effect/__init__.py
• tests/unit/nodes/node_gateway_link_health_write_effect/handlers/__init__.py
• tests/unit/nodes/node_gateway_link_health_write_effect/handlers/test_handler_gateway_link_health_upsert.py
• tests/unit/validation/test_application_database_sql_enforcement_regressions.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

❓ Hostile Reviewer — UNKNOWN

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

---

Gate semantics (pilot phase)

Verdict	Meaning	Blocks merge?
passed	No critical findings	No
blocked	CRITICAL findings found	Yes
degraded	All models unavailable (infra)	No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)