[codex] Move provider stats fully to ClickHouse

deploy/README.md

@@ -144,7 +144,7 @@ CI (main) -> build dofek + dofek-ml (same tag)
    8. Validate required host bind-mount directories before deploying the stack. This must fail before `docker stack deploy` if paths such as `/mnt/dofek-data/redis` are missing, because Swarm rejects tasks with missing bind sources.
    9. `docker stack deploy -c deploy/stack.yml --with-registry-auth --prune --detach=false <stack>` — swarm performs a single stack-wide update, including `training-export-worker`, and CI waits for the rollout to converge before continuing. The deploy workflow bounds this wait at 20 minutes so a wedged Swarm rollback fails CI instead of running indefinitely.
       The workflow parses the Infisical dotenv file inside a child process for stack interpolation. Do not append the full dotenv file to `GITHUB_ENV`; GitHub Actions prints step environments and can expose Infisical-only secrets that GitHub does not automatically mask.
-   10. Wait for PeerDB and run the one-shot ClickHouse CDC setup command. The command loads `src/db/peerdb/metric-stream-cdc.sql`, substitutes deployment connection values, and creates the Postgres peer, ClickHouse peer, and `dofek_metric_stream_cdc` mirror if they do not already exist.
+   10. Wait for PeerDB and run the one-shot ClickHouse CDC setup command. The command loads `src/db/peerdb/metric-stream-cdc.sql`, substitutes deployment connection values, creates the Postgres and ClickHouse peers if missing, and applies the metric-stream, raw analytics, and provider inventory mirrors.
 
 When adding a new host bind mount under `/mnt/dofek-data`, update both
 `deploy/stack.yml` and the Terraform provisioner that creates the directory. If

docs/clickhouse-metric-stream.md

@@ -65,6 +65,13 @@ membership in Postgres. Stream, heart-rate-zone, power-zone, and activity
 summary reads then query stored ClickHouse `analytics.*` materialized views. The
 app does not issue raw `metric_stream` analytical reads for those endpoints.
 
+Provider record inventory uses the ClickHouse `analytics.provider_stats` read
+model for all provider-owned record counts displayed by sync/provider detail:
+activity, daily metric, sleep, body measurement, food entry, health event,
+metric stream, distinct nutrition day, lab panel, lab result, and journal entry
+counts. The provider detail UI still treats these as raw provider-owned record
+counts, not deduped analytical sample counts.
+
 ## Sync Model
 
 ClickHouse migrations run from the normal one-shot `migrate` container when
@@ -79,7 +86,9 @@ ClickHouse migrations create and update the databases and read models:
   refreshers.
 - `peerdb.metric_stream`: the PeerDB CDC validation target.
 - `postgres_fitness`: app-managed native ClickHouse raw mirrors with PeerDB CDC
-  metadata columns.
+  metadata columns. Besides the activity/sleep/body/daily/metric stream
+  analytics sources, this includes provider inventory mirrors for `food_entry`,
+  `health_event`, `lab_panel`, `lab_result`, and `journal_entry`.
 - `analytics.v_activity`, `analytics.v_activity_members`, `analytics.v_sleep`,
   `analytics.v_body_measurement`, and `analytics.v_daily_metrics`: ClickHouse
   read models over the raw mirrors.
@@ -106,8 +115,11 @@ Postgres runs with `wal_level=logical`, `max_replication_slots`, and
 `src/db/setup-clickhouse-cdc.ts` after `docker stack deploy`; that command
 loads `src/db/peerdb/metric-stream-cdc.sql`, substitutes deployment
 connection values, and applies the declarative PeerDB peer and mirror
-definition. The mirror uses a dedicated publication name, excludes `device_id`,
-`source_type`, and `vector`, and enables soft deletes so delete events are
+definition. Provider inventory tables are mirrored by
+`dofek_provider_inventory_raw_analytics` so existing raw analytics mirrors do
+not need to be rebuilt when inventory coverage expands. The mirrors use a
+dedicated publication name, exclude `device_id`, `source_type`, and `vector`
+from the metric stream mirrors, and enable soft deletes so delete events are
 represented in ClickHouse.
 ClickHouse's built-in `MaterializedPostgreSQL` engine is not the CDC path for
 `metric_stream`.

docs/production-incident-baseline.md

@@ -3205,3 +3205,79 @@ above the 75 break threshold.
 
 The targeted shard is now above threshold locally. Full CI still needs to rerun
 on GitHub Actions to verify the complete sharded matrix with runner timing.
+
+## 2026-05-07: Auth Session Queries Failed During Web DB Pool Starvation
+
+### Symptoms
+
+Production web logs showed repeated tRPC errors like:
+
+```text
+[trpc] settings.get: Failed query: SELECT user_id FROM fitness.session
+```
+
+The same failure appeared across many unrelated routes, including dashboard,
+sleep, stress, recovery, sync, provider guide, and mobile sensor push endpoints.
+OAuth callbacks and webhook lookups also failed around the same window.
+
+### User Impact
+
+Authenticated web and mobile requests intermittently failed because even the
+cheap session validation query could not get a Postgres client from the web
+process pool.
+
+### Evidence
+
+Axiom OpenTelemetry spans for `dofek-web` showed the underlying failure:
+
+```text
+name="pg-pool.connect"
+status.message="timeout exceeded when trying to connect"
+duration="9.999601248s"
+```
+
+The app log wrapper around the same time was:
+
+```text
+[trpc] settings.get: Failed query: SELECT user_id FROM fitness.session
+```
+
+Postgres itself was writable and not in recovery:
+
+```text
+pg_is_in_recovery = false
+to_regclass('fitness.session') = fitness.session
+session_count = 7
+connections = 20
+```
+
+`pg_stat_activity` showed many long-running app queries active for 28-35 minutes,
+including `fitness.v_daily_metrics`, `fitness.activity`, stress, recovery, and
+provider-stat read queries. The app DB pool is configured in `src/db/index.ts`
+with `max: 5` and `connectionTimeoutMillis: 10_000`.
+
+### Root Cause
+
+The immediate root cause was web-process Postgres pool starvation: expensive
+read queries held all available pg pool clients long enough that new requests
+timed out after 10 seconds while trying to acquire/connect a client. The
+`fitness.session` query was the first query most authenticated requests run, so
+it surfaced as an auth/session failure even though the session table existed and
+Postgres was writable.
+
+### Fix or Mitigation
+
+Moved `sync.providerStats` off the hot Postgres request path.
+`SyncRepository.getProviderStats()` now reads all provider record counts from
+the ClickHouse `analytics.provider_stats` read model. The ClickHouse raw mirrors
+were extended to include `food_entry`, `health_event`, `lab_panel`,
+`lab_result`, and `journal_entry`, so the endpoint no longer runs per-provider
+count subqueries against Postgres. The fix intentionally did not increase
+Postgres pool size or timeouts.
+
+### Remaining Risk
+
+Other expensive Postgres read paths were observed during the incident, especially
+daily-metrics/recovery/insights queries over `fitness.v_daily_metrics` and
+related views. Those paths still need follow-up migration or query tightening so
+dashboard/mobile bursts cannot exhaust each web process's small Postgres pool.

packages/server/src/repositories/sync-repository.test.ts

@@ -1,12 +1,19 @@
 import { describe, expect, it, vi } from "vitest";
+import type { z } from "zod";
 import { SyncRepository } from "./sync-repository.ts";
 
 // ---------------------------------------------------------------------------
 // Helpers
 // ---------------------------------------------------------------------------
 
-function makeRepository(rows: Record<string, unknown>[] = []) {
+function makeRepository(
+  rows: Record<string, unknown>[] = [],
+  clickHouseStatsRows: Record<string, unknown>[] = [],
+) {
   const execute = vi.fn().mockResolvedValue(rows);
+  const query = vi.fn(<TSchema extends z.ZodType>(schema: TSchema) =>
+    Promise.resolve(clickHouseStatsRows.map((row) => schema.parse(row))),
+  );
   const select = vi.fn().mockReturnValue({
     from: vi.fn().mockReturnValue({
       where: vi.fn().mockReturnValue({
@@ -17,8 +24,9 @@ function makeRepository(rows: Record<string, unknown>[] = []) {
     }),
   });
   const db: Pick<import("dofek/db").Database, "execute" | "select"> = { execute, select };
-  const repo = new SyncRepository(db, "user-1");
-  return { repo, execute, select };
+  const sensorStore = { query };
+  const repo = new SyncRepository(db, "user-1", sensorStore);
+  return { repo, execute, query, select };
 }
 
 // ---------------------------------------------------------------------------
@@ -116,28 +124,32 @@ describe("SyncRepository", () => {
 
   describe("getProviderStats", () => {
     it("returns empty array when no providers", async () => {
-      const { repo } = makeRepository([]);
+      const { repo, execute } = makeRepository([]);
       const result = await repo.getProviderStats();
       expect(result).toEqual([]);
+      expect(execute).not.toHaveBeenCalled();
     });
 
     it("maps rows to ProviderStatRow objects with numeric values", async () => {
-      const { repo } = makeRepository([
-        {
-          provider_id: "wahoo",
-          activities: "5",
-          daily_metrics: "30",
-          sleep_sessions: "0",
-          body_measurements: "2",
-          food_entries: "0",
-          health_events: "1",
-          metric_stream: "100",
-          nutrition_daily: "0",
-          lab_panels: "0",
-          lab_results: "0",
-          journal_entries: "3",
-        },
-      ]);
+      const { repo, execute } = makeRepository(
+        [],
+        [
+          {
+            provider_id: "wahoo",
+            activities: "5",
+            daily_metrics: "30",
+            sleep_sessions: "0",
+            body_measurements: "2",
+            food_entries: "8",
+            health_events: "1",
+            metric_stream: "100",
+            nutrition_daily: "6",
+            lab_panels: "4",
+            lab_results: "9",
+            journal_entries: "3",
+          },
+        ],
+      );
       const result = await repo.getProviderStats();
       expect(result).toHaveLength(1);
       expect(result[0]).toEqual({
@@ -146,53 +158,90 @@ describe("SyncRepository", () => {
         dailyMetrics: 30,
         sleepSessions: 0,
         bodyMeasurements: 2,
-        foodEntries: 0,
+        foodEntries: 8,
         healthEvents: 1,
         metricStream: 100,
-        nutritionDaily: 0,
-        labPanels: 0,
-        labResults: 0,
+        nutritionDaily: 6,
+        labPanels: 4,
+        labResults: 9,
         journalEntries: 3,
       });
+      expect(execute).not.toHaveBeenCalled();
     });
 
     it("handles multiple providers", async () => {
-      const { repo } = makeRepository([
-        {
-          provider_id: "wahoo",
-          activities: "5",
-          daily_metrics: "0",
-          sleep_sessions: "0",
-          body_measurements: "0",
-          food_entries: "0",
-          health_events: "0",
-          metric_stream: "0",
-          nutrition_daily: "0",
-          lab_panels: "0",
-          lab_results: "0",
-          journal_entries: "0",
-        },
-        {
-          provider_id: "strava",
-          activities: "10",
-          daily_metrics: "0",
-          sleep_sessions: "0",
-          body_measurements: "0",
-          food_entries: "0",
-          health_events: "0",
-          metric_stream: "0",
-          nutrition_daily: "0",
-          lab_panels: "0",
-          lab_results: "0",
-          journal_entries: "0",
-        },
-      ]);
+      const { repo } = makeRepository(
+        [],
+        [
+          {
+            provider_id: "wahoo",
+            activities: "5",
+            daily_metrics: "0",
+            sleep_sessions: "0",
+            body_measurements: "0",
+            food_entries: "0",
+            health_events: "0",
+            metric_stream: "0",
+            nutrition_daily: "0",
+            lab_panels: "0",
+            lab_results: "0",
+            journal_entries: "0",
+          },
+          {
+            provider_id: "strava",
+            activities: "10",
+            daily_metrics: "0",
+            sleep_sessions: "0",
+            body_measurements: "0",
+            food_entries: "0",
+            health_events: "0",
+            metric_stream: "42",
+            nutrition_daily: "0",
+            lab_panels: "0",
+            lab_results: "0",
+            journal_entries: "0",
+          },
+        ],
+      );
       const result = await repo.getProviderStats();
       expect(result).toHaveLength(2);
       expect(result[0]?.providerId).toBe("wahoo");
       expect(result[0]?.activities).toBe(5);
+      expect(result[0]?.metricStream).toBe(0);
       expect(result[1]?.providerId).toBe("strava");
       expect(result[1]?.activities).toBe(10);
+      expect(result[1]?.metricStream).toBe(42);
+    });
+
+    it("reads all provider counts from ClickHouse instead of the Postgres stats query", async () => {
+      const { repo, execute, query } = makeRepository(
+        [],
+        [
+          {
+            provider_id: "apple-health",
+            activities: "1",
+            daily_metrics: "2",
+            sleep_sessions: "3",
+            body_measurements: "4",
+            food_entries: "5",
+            health_events: "6",
+            metric_stream: "7",
+            nutrition_daily: "8",
+            lab_panels: "9",
+            lab_results: "10",
+            journal_entries: "11",
+          },
+        ],
+      );
+
+      await repo.getProviderStats();
+
+      expect(query).toHaveBeenCalledWith(
+        expect.anything(),
+        expect.stringContaining("analytics.provider_stats"),
+        { userId: "user-1" },
+      );
+      expect(execute).not.toHaveBeenCalled();
     });
   });
 });