feat(web): observable read-replica and slot replication health

[RSO]

Why

We had a US-west read replica silently stuck ~8 days behind (its walreceiver crash-looped after the primary recycled WAL it still needed). The existing `/api/internal/db/replication-lag` endpoint queried `pg_stat_replication`, which only lists currently-connected walsenders — so a broken replica has no row and is invisible, exactly the failure we most want to catch. Logical (Snowflake) slots disappear from that view the same way once their consumer disconnects.

What

New shared module `apps/web/src/lib/replication-health.ts` combining three signals:

• Replica-side lag (authoritative). Connects directly to each replica URL (`POSTGRES_REPLICA_US_URL`, `POSTGRES_REPLICA_EU_URL`, `POSTGRES_REPLICA_EU_URL_2`) and computes `replay_delay_seconds` from `pg_last_xact_replay_timestamp()` / `pg_is_in_recovery()`. Measured on the replica, so a stuck box reports days of delay (or `unreachable`) instead of vanishing. Short-lived `max: 1` pool with connect/statement/query timeouts; never throws.
• Slot health (primary). Reads `pg_replication_slots` for `active` / `wal_status` / retained WAL, flagging `at_risk` when `wal_status ∈ {unreserved, lost}` — catches lost Snowflake slots and slots approaching `max_slot_wal_keep_size`.
• `pg_stat_replication`, kept but reframed as "who is streaming right now".

`collectReplicationHealth()` isolates primary-query failures into `errors[]` and returns a single `healthy` boolean.

Endpoint A — `/api/internal/db/replication-lag`

Returns the full report (`healthy`, `replicas`, `walSenders`, `slots`, `errors`, `timestamp`). Same `X-Internal-Secret` auth. Response shape changed (no other consumers).

Cron B — `/api/cron/db-replication-health` (new, `*/5 * * * *`)

Reuses the module, emits per-replica/per-slot JSON to the Vercel→Axiom log drain, and `captureException`s to Sentry on lagging/unreachable replicas or at-risk slots.

Design note: I deliberately did not extend `db-pool-metrics`. That cron is a single-purpose Supabase Prometheus scraper, and the `physical_replication_lag_*` metric has a documented history of returning no data (and can share the same connected-only blind spot). A dedicated cron on the reliable replica-side SQL probe is cleaner and more trustworthy.

Testing

• `replication-health.test.ts`, both route tests — 13/13 pass (status classification, healthy/unreachable/lost-slot/primary-failure paths, auth).
• oxlint: 0 warnings/errors. `pnpm format` applied. `tsgo --noEmit` (apps/web) clean.

Follow-ups (non-blocking)

• `REPLICA_LAG_ALERT_SECONDS` (300s) and the `*/5` cadence are easy-to-tune defaults.
• The module opens a fresh pool per probe — fine at this frequency; would want cached pools if ever used on a hot path.

[kilo-code-bot]

Code Review Summary

Status: 1 Issue Found | Recommendation: Merge after optional cleanup

Executive Summary

Incremental review of the three follow-up commits: the previously flagged throw-out-of-probeReplica, missing pool error listener, and silent zero-replica blind spot are all fixed; the only remaining nit is that the new replica-inventory error is surfaced to Sentry under the misleading primary query failed prefix.

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	1

Issue Details (click to expand)

SUGGESTION

File	Line	Issue
apps/web/src/lib/replication-health.ts	258	Inventory error joins the same errors[] array the cron renders as primary query failed: ..., so a missing POSTGRES_REPLICA_* env var alerts as a primary-database failure

Previously reported, now resolved

• probeReplica throwing on a malformed connection string: fixed in e2e04c295 (createDrizzleClient moved inside try, client?.pool.end() guard) and per-probe catch isolation added in collectReplicationHealth, with regression tests for both.
• Missing pool.on('error') on the short-lived probe pool: fixed in 1113ef325.
• Zero configured replicas reporting healthy: true: fixed in 67135cd9a via EXPECTED_REPLICA_COUNT gated on VERCEL_ENV === 'production', with tests for the production, non-production, and full-inventory paths.

Files Reviewed (2 files)

• apps/web/src/lib/replication-health.ts - 1 issue
• apps/web/src/lib/replication-health.test.ts - 0 issues

Notes and assumptions

• Incremental scope: only aa04078f..67135cd9 changes were reviewed; unchanged files and unchanged lines (including the replay_delay_seconds === null discussion the author declined) were not re-raised.
• The production gate assumes Vercel system environment variables are exposed at runtime, which the repository already relies on elsewhere; if they were not, the check would simply never fire (no false alerts).
• getEnvVariable returns '' for unset variables, so getExpectedReplicaTargets() cannot throw on a missing replica URL.
• No tests were executed (read-only review).

Fix these issues in Kilo Cloud

Previous Review Summary (commit aa04078)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit aa04078)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

The new replication monitor has blind spots that can report healthy: true while replicas are unmonitored or provably stalled, plus two robustness gaps in probeReplica that can take down the whole report.

Overview

Severity	Count
CRITICAL	0
WARNING	4
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/lib/replication-health.ts	40	Zero configured replica URLs yields healthy: true with replicas: 0; a renamed/missing POSTGRES_REPLICA_* turns the monitor into a silent no-op
apps/web/src/lib/replication-health.ts	113	replay_delay_seconds === null (pg_last_xact_replay_timestamp() NULL until the first replayed commit) is classified ok, masking a replica applying no WAL
apps/web/src/lib/replication-health.ts	125	createDrizzleClient is outside the try, and probes run in a bare Promise.all; a malformed replica URL throws out of a "never throws" function and fails the entire report/endpoint
apps/web/src/lib/replication-health.ts	134	Short-lived probe pool has no pool.on('error') handler; an unhandled pg.Pool 'error' event terminates the process (long-lived pools in lib/drizzle.ts do attach handlers)

Files Reviewed (7 files)

• apps/web/src/lib/replication-health.ts - 4 issues
• apps/web/src/lib/replication-health.test.ts - 0 issues
• apps/web/src/app/api/cron/db-replication-health/route.ts - 0 issues
• apps/web/src/app/api/cron/db-replication-health/route.test.ts - 0 issues
• apps/web/src/app/api/internal/db/replication-lag/route.ts - 0 issues
• apps/web/src/app/api/internal/db/replication-lag/route.test.ts - 0 issues
• apps/web/vercel.json - 0 issues

Notes and assumptions

• The non-constant-time CRON_SECRET comparison in the new cron route matches the existing db-pool-metrics pattern, so it was not flagged.
• SQL correctness of the pg_replication_slots / pg_stat_replication queries and the response-shape break on /api/internal/db/replication-lag were reviewed against the PR description's claim of no other consumers; that claim was not independently verified.
• No tests were executed (read-only review).

Fix these issues in Kilo Cloud

---

_{Reviewed by claude-opus-5 · Input: 52 · Output: 14K · Cached: 1.5M}

_{Review guidance: REVIEW.md from base branch main}