Skip to content
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
c2e4143
Convert non-sensor ClickHouse read models to views
Asherlc May 23, 2026
cd33016
Make ClickHouse sensor dedupe incremental
Asherlc May 23, 2026
5a97193
Break ClickHouse deduped sensor import cycle
Asherlc May 23, 2026
c59c980
Address deduped sensor review feedback
Asherlc May 23, 2026
9b2e5fa
Parameterize deduped sensor dirty-key snapshot
Asherlc May 23, 2026
0810698
Fix ClickHouse migration CI failures
Asherlc May 23, 2026
fc97729
Split vertical ascent integration fixture
Asherlc May 23, 2026
faa1fb3
Address ClickHouse review comments
Asherlc May 23, 2026
9cee548
Cover deduped sensor mutation cases
Asherlc May 23, 2026
59353b0
Avoid empty aerobic efficiency diagnostic scan
Asherlc May 23, 2026
b9ae037
Short-circuit empty aerobic efficiency queries
Asherlc May 23, 2026
205c5f4
Short-circuit empty training analytics
Asherlc May 23, 2026
adc2fad
Avoid recursive activity view in analytics routes
Asherlc May 23, 2026
d7d76f0
Avoid recursive activity view in power analytics
Asherlc May 23, 2026
9f8f0f0
Bound PMC sample counts to query window
Asherlc May 23, 2026
6859383
Show chart empty states during background fetches
Asherlc May 23, 2026
14ffa06
Short-circuit empty cycling analytics reads
Asherlc May 23, 2026
d3552f8
Use Postgres activity preflights for empty analytics
Asherlc May 23, 2026
0d4098e
Remove obsolete chart skeleton assertions
Asherlc May 24, 2026
63f4533
Clarify conditional negative test guidance
Asherlc May 24, 2026
252a613
Kill training preflight Stryker mutants
Asherlc May 24, 2026
a8bdeca
Fix ClickHouse migration ordering for deduped sensor
Asherlc May 24, 2026
83cee5a
Fix migration test spell check
Asherlc May 24, 2026
6a4b278
Avoid long-lived SSH wait for deploy migrations
Asherlc May 24, 2026
6bd0eab
Chunk deduped sensor migration backfill
Asherlc May 24, 2026
b3fb1dc
Retry transient migration inspect failures
Asherlc May 24, 2026
66e7e92
Skip unnecessary deploy image prune
Asherlc May 24, 2026
dba182a
Validate deploy bind paths over SSH
Asherlc May 24, 2026
443cccd
Pull deploy dependency images over direct SSH
Asherlc May 24, 2026
ddec7bc
Retry ClickHouse startup timeouts
Asherlc May 24, 2026
ec8303b
Disable crashing CloudBeaver service
Asherlc May 24, 2026
5dd9fb3
Document CloudBeaver host recovery
Asherlc May 24, 2026
7f9c492
Bound ClickHouse production memory
Asherlc May 24, 2026
0769291
Detect stale ClickHouse deploy config
Asherlc May 24, 2026
08a5e02
Apply stack config before migrations
Asherlc May 24, 2026
996ba68
Detach pre-migration stack apply
Asherlc May 24, 2026
b83c0b1
Tune ClickHouse production memory cap
Asherlc May 24, 2026
c20d830
Rotate ClickHouse memory Swarm config
Asherlc May 24, 2026
210993f
Quiesce app services before migrations
Asherlc May 24, 2026
927579d
Avoid materializing ClickHouse views at startup
Asherlc May 24, 2026
e3807d7
Use metadata for ClickHouse startup smoke checks
Asherlc May 24, 2026
4f8b2bc
Document ClickHouse deploy recovery
Asherlc May 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 9 additions & 16 deletions docs/clickhouse-body-measurement-staleness-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ docker exec -i "$clickhouse" sh -lc 'clickhouse-client --password "$CLICKHOUSE_P
SELECT database, view, status, last_refresh_time, last_success_time, exception
FROM system.view_refreshes
WHERE database = 'analytics'
AND view IN ('deduped_sensor', 'provider_stats', 'v_body_measurement')
AND view = 'deduped_sensor'
ORDER BY view;

SELECT source, latest, rows FROM
Expand Down Expand Up @@ -84,19 +84,18 @@ SQL
REMOTE
```

If `analytics.deduped_sensor` or `analytics.provider_stats` is running a
full-history refresh over `postgres_fitness.metric_stream FINAL` and the host is
unhealthy, stop the active query and pause that refreshable view. Record the
query IDs first.
If a ClickHouse query is scanning all of `postgres_fitness.metric_stream FINAL`
and the host is unhealthy, stop the active query after recording the query ID.
The current sensor pipeline is incremental; do not pause or depend on a
refreshable sensor view.

```sql
KILL QUERY WHERE query_id IN ('<query-id-1>', '<query-id-2>') SYNC;
SYSTEM STOP VIEW analytics.deduped_sensor;
SYSTEM STOP VIEW analytics.provider_stats;
```

This is an operational mitigation: activity sensor analytics and provider stats
can remain stale while these views are disabled. Document it in
Then inspect `analytics.sensor_dirty_key` backlog and the latest
`analytics.sensor_scalar_sample._peerdb_synced_at` to confirm the incremental
pipeline is advancing. Document the incident in
`docs/production-incident-baseline.md`.

## 3. Decide Whether CDC Missed a Gap
Expand Down Expand Up @@ -170,19 +169,13 @@ The ClickHouse materialized view
`analytics.body_measurement_sample_ingest` should ingest these inserted raw rows
into `analytics.body_measurement_sample`.

## 4. Refresh and Verify the Body Read Model
## 4. Verify the Body Read Model

```bash
ssh dofek-server 'bash -s' <<'REMOTE'
set -euo pipefail
clickhouse=$(docker ps --format '{{.Names}}' | grep dofek_clickhouse | head -1)
docker exec -i "$clickhouse" sh -lc 'clickhouse-client --password "$CLICKHOUSE_PASSWORD"' <<'SQL'
SYSTEM REFRESH VIEW analytics.v_body_measurement;

SELECT database, view, status, last_refresh_time, last_success_time, exception
FROM system.view_refreshes
WHERE database = 'analytics' AND view = 'v_body_measurement';

SELECT source, latest, rows FROM
(
SELECT 'postgres_fitness.metric_stream' AS source, max(recorded_at) AS latest, count() AS rows
Expand Down
101 changes: 76 additions & 25 deletions docs/clickhouse-metric-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,26 +17,34 @@ Postgres/Timescale fitness.metric_stream
v v
ClickHouse postgres_fitness.metric_stream
|
| refreshable materialized view, 15-minute cadence
| native incremental projection
v
ClickHouse analytics.sensor_scalar_sample
|
| dirty-key bounded recompute
v
ClickHouse analytics.deduped_sensor
|
| refreshable materialized view, 15-minute cadence
| normal view
v
ClickHouse analytics.activity_summary
|
| refreshable materialized view, 15-minute cadence
| normal view
v
ClickHouse analytics.activity_trend_daily
|
v
Activity stream, zone, summary, and trend reads
```

Location rows are exposed through `analytics.deduped_location`, also as a normal
view. There are no ClickHouse full-refresh read models in this chain.

Runtime API queries must read `analytics.deduped_sensor`,
`analytics.activity_summary`, or `analytics.activity_trend_daily`, not the raw
metric stream. The raw ClickHouse table exists only as the source for
ClickHouse refresh jobs. Derived rows are never synced back to Postgres.
metric stream. The raw ClickHouse table exists only as the source for the
deduped sensor incremental projection and normal analytics views. Derived rows
are never synced back to Postgres.

## Local Development

Expand All @@ -62,8 +70,11 @@ Use these local URLs:
## Query Model

Activity routes resolve authorization, access windows, and canonical activity
membership in Postgres. Stream, heart-rate-zone, power-zone, and activity
summary reads then query stored ClickHouse `analytics.*` materialized views. The
membership in Postgres. Stream and heart-rate/power-zone reads query
`analytics.deduped_sensor` by user, channel, and activity time window; the
sensor table does not store `activity_id`. Activity summary, location, provider
stats, sleep/body/daily metrics, and trend reads query normal ClickHouse
`analytics.*` views over the raw mirrors and incremental projection tables. The
app does not issue raw `metric_stream` analytical reads for those endpoints.
Daily and weekly trend reads use `analytics.activity_trend_daily`; weekly rows
are rolled up from daily rows at query time.
Expand All @@ -75,6 +86,35 @@ metric stream, distinct nutrition day, lab panel, lab result, and journal entry
counts. The provider detail UI still treats these as raw provider-owned record
counts, not deduped analytical sample counts.

## Scalar And Location Projections

Scalar sensor samples and location samples use separate ClickHouse projections
because they have different deduplication semantics.

`analytics.deduped_sensor` is optimized for scalar channels such as heart rate,
power, cadence, speed, and altitude. For those channels, the useful atomic unit
is one `(user_id, channel, recorded_at)` key. The incremental pipeline can
recompute a single changed key and select the best live sample with
provider/device priority rules.

GPS location is different. The useful unit is a coherent route for an activity,
not an independently selected point at each timestamp. Providers can report the
same route with different sampling rates, timestamp rounding, pause trimming,
smoothing, altitude correction, and GPS filtering. If location were deduped
point-by-point with the scalar rules, a derived route could silently stitch
Apple, Garmin, Strava, or other provider points together. That stitched route
may not match any route a provider actually recorded, and route-sensitive
metrics such as distance, centroid, and map shape can be inflated, deflated, or
visibly jagged by small provider-to-provider differences.

For that reason, `analytics.deduped_location` remains a separate activity-route
projection. It selects a route source for an activity, currently by valid
location sample count with deterministic provider tie-breaking, and then uses
that source's ordered points for GPS-derived activity summary fields. This is
not mainly a table-shape constraint; the table shape could change. The core
constraint is preserving route coherence while keeping scalar sensor dedupe a
small per-key incremental pipeline.

## Sync Model

ClickHouse migrations run from the normal one-shot `migrate` container when
Expand Down Expand Up @@ -103,27 +143,36 @@ ClickHouse migrations create and update the databases and read models:
- `postgres_fitness`: app-managed native ClickHouse raw mirrors with PeerDB CDC
metadata columns. Besides the activity/sleep/body/daily/metric stream
analytics sources, this includes provider inventory mirrors for `food_entry`,
`health_event`, `lab_panel`, `lab_result`, and `journal_entry`.
`health_event`, `lab_panel`, `lab_result`, and `journal_entry`, plus
`sensor_provider_priority` and `sensor_device_priority` for sensor-channel
deduplication.
- `analytics.v_activity`, `analytics.v_activity_members`, `analytics.v_sleep`,
`analytics.v_body_measurement`, and `analytics.v_daily_metrics`: ClickHouse
read models over the raw mirrors.
`analytics.v_body_measurement`, `analytics.v_daily_metrics`, and
`analytics.provider_stats`: normal ClickHouse views over the raw mirrors and
body sample projection.
- `analytics.body_measurement_sample`: a narrow `ReplacingMergeTree`
projection of body-related `metric_stream` channels. It is backfilled once by
migration and kept current by `analytics.body_measurement_sample_ingest`, so
`analytics.v_body_measurement` can refresh without repeatedly scanning the
full metric stream mirror.
- `analytics.deduped_sensor`: a refreshable materialized view refreshed every
15 minutes from the copied raw rows and activity membership.
- `analytics.deduped_location`: a refreshable materialized view refreshed every
15 minutes from `postgres_fitness.metric_stream` location rows. While the
PeerDB mirror excludes `point`, new rows have `point = NULL` and this view
does not advance for new data.
- `analytics.activity_summary`: a refreshable materialized view refreshed every
15 minutes from `analytics.deduped_sensor`.
- `analytics.activity_trend_daily`: a refreshable materialized view with one
activity-linked sensor trend row per user and UTC day. It is derived from
`analytics.deduped_sensor`, refreshes every 15 minutes, and is safe to drop
and rebuild.
`analytics.v_body_measurement` does not scan the full metric stream mirror.
- `analytics.sensor_scalar_sample`: a narrow `ReplacingMergeTree` projection of
activity sensor scalar channels. It is backfilled once by migration and kept
current by `analytics.sensor_scalar_sample_ingest`.
- `analytics.sensor_dirty_key`: a bounded queue of changed
`(user_id, channel, recorded_at)` keys. The post-sync worker recomputes only
those keys into `analytics.deduped_sensor`.
- `analytics.deduped_sensor`: an activity-agnostic `ReplacingMergeTree` table
containing the best live scalar sample per `(user_id, channel, recorded_at)`
according to mirrored sensor provider/device priority tables. It has no
`activity_id`; activity reads join samples to activities by time window.
- `analytics.deduped_location`: a normal view over
`postgres_fitness.metric_stream` location rows. While the PeerDB mirror
excludes `point`, new rows have `point = NULL` and this view does not advance
for new data.
- `analytics.activity_summary`: a normal view over `analytics.deduped_sensor`,
`analytics.deduped_location`, and `analytics.v_activity`.
- `analytics.activity_trend_daily`: a normal view with one activity-linked
sensor trend row per user and UTC day. It is derived from
`analytics.deduped_sensor`.

Because `postgres_fitness.metric_stream` is an existing app-managed ClickHouse
table rather than a table created by PeerDB, it must include PeerDB's CDC
Expand All @@ -145,7 +194,9 @@ loads `src/db/peerdb/metric-stream-cdc.sql`, substitutes deployment
connection values, and applies the declarative PeerDB peer and mirror
definition. Provider inventory tables are mirrored by
`dofek_provider_inventory_raw_analytics` so existing raw analytics mirrors do
not need to be rebuilt when inventory coverage expands.
not need to be rebuilt when inventory coverage expands. Sensor priority tables
are mirrored by `dofek_sensor_priority_raw_analytics` so priority changes do
not require rebuilding the broader raw analytics mirrors.

**Mirror reconciliation**: The deploy CDC setup checks whether each mirror's
table list matches the expected mapping in `rawAnalyticsMirrorTableMappings`.
Expand Down
Loading
Loading