Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ If you are starting cold and do not want to hunt through agent notes, begin here
└─────────────┘ └───────────┘
```

Each data source is a **provider plugin** that implements a simple interface. The sync runner orchestrates all enabled providers. Raw data lands in a `fitness` Postgres schema. ClickHouse maintains stored analytics read models for heavy activity stream reads, with scalar `metric_stream` rows copied from Postgres through the tracked ClickHouse migration path. Incremental dbt models under `analytics/models/` define derived ClickHouse analytics tables such as `analytics.deduped_sensor` and `analytics.resting_heart_rate_sleep_window` outside the web/API request path. PeerDB is deployed as the internal CDC path from Postgres to ClickHouse, replicating `metric_stream` and raw fitness tables into `postgres_fitness.*` for analytics read models. Derived rows are not written back to Postgres. The web dashboard provides sync controls, provider health monitoring, insights, and data exploration. A companion iOS app (Expo + React Native) provides native HealthKit integration and on-the-go access. Nutrition logging on web and iOS supports natural-language AI meal input that can split one message into multiple food items. Long-running sync jobs are processed by BullMQ workers backed by Redis. In production, the `worker` container registers repeatable scheduled sync jobs in BullMQ, and the `analytics-worker` container runs the production-safe subset of dbt microbatch analytics builds every 15 minutes with a bounded retry delay; the `sync` mode remains available for manual one-shot runs. `analytics.sensor_scalar_sample` and `analytics.deduped_sensor` are enabled as bounded `recorded_at` microbatch models; `resting_heart_rate_sleep_window` and `activity_summary_rows` remain excluded until they are split into smaller chained incremental models.
Each data source is a **provider plugin** that implements a simple interface. The sync runner orchestrates all enabled providers. Raw data lands in a `fitness` Postgres schema. ClickHouse maintains stored analytics read models for heavy activity stream reads, with scalar `metric_stream` rows copied from Postgres through the tracked ClickHouse migration path. Incremental dbt models under `analytics/models/` define derived ClickHouse analytics tables such as `analytics.deduped_sensor` and `analytics.resting_heart_rate_sleep_window` outside the web/API request path. PeerDB is deployed as the internal CDC path from Postgres to ClickHouse, replicating `metric_stream` and raw fitness tables into `postgres_fitness.*` for analytics read models. Derived rows are not written back to Postgres. The web dashboard provides sync controls, provider health monitoring, insights, and data exploration. A companion iOS app (Expo + React Native) provides native HealthKit integration and on-the-go access. Nutrition logging on web and iOS supports natural-language AI meal input that can split one message into multiple food items. Long-running sync jobs are processed by BullMQ workers backed by Redis. In production, the `worker` container registers repeatable scheduled sync jobs in BullMQ, and the `analytics-worker` container runs the production-safe subset of dbt analytics builds every 15 minutes with a bounded retry delay; the `sync` mode remains available for manual one-shot runs. `analytics.sensor_scalar_sample`, `analytics.deduped_sensor`, `analytics.sleep_heart_rate_sample`, `analytics.activity_sensor_sample`, and `analytics.activity_location_sample` run as bounded `recorded_at` microbatch models; `analytics.resting_heart_rate_sleep_window`, the activity aggregate intermediates, and `analytics.activity_summary_rows` use dirty-key incremental models over those bounded inputs.

## Quick Start

Expand Down
40 changes: 24 additions & 16 deletions analytics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,31 @@ The call sites are:
Model dependencies are declared with dbt `ref()` calls. `sensor_scalar_sample` stages scalar
metric samples, `deduped_sensor` reads `sensor_scalar_sample`, and
`activity_vo2max_estimate` reads `deduped_sensor` to keep the expensive VO2
max activity/sample joins out of web/API requests. `resting_heart_rate_sleep_window`
also reads `deduped_sensor`.
max activity/sample joins out of web/API requests. `sleep_heart_rate_sample`,
`activity_sensor_sample`, and `activity_location_sample` are bounded
microbatch intermediates over sample time. `resting_heart_rate_sleep_window`
aggregates the sleep sample intermediary, while `activity_sensor_summary_rows`
and `activity_location_summary_rows` aggregate the activity sample
intermediaries before `activity_summary_rows` joins those compact per-activity
summaries.
The serving-facing `analytics.activity_summary` object is a thin ClickHouse view
over `analytics.activity_summary_rows FINAL`; the expensive activity/sample
joins belong in the incremental dbt model, not in web/API requests. Complex
offline ClickHouse models can set dbt `query_settings` locally;
`activity_summary_rows` raises `query_plan_max_optimizations_to_apply` for its
large insert plan and uses `max_threads=1` so the offline build does not compete
with request traffic.
joins belong in incremental dbt models, not in web/API requests. Complex
offline ClickHouse models can set dbt `query_settings` locally and use
`max_threads=1` so offline builds do not compete with request traffic.

Production `DBT_SAFE_MODELS` currently selects `sensor_scalar_sample`,
`deduped_sensor`, and `activity_vo2max_estimate`. The first two use dbt's
`microbatch` incremental strategy with `recorded_at` as the event time, daily
batches, and a short lookback so ClickHouse processes bounded sample-time
windows instead of one large dirty-key query. `activity_vo2max_estimate` uses
dirty activity/user keys and `max_threads=1`; it materializes reusable
per-activity VO2 max estimates, not final API responses. `activity_summary_rows`
is still excluded because the first single-query offline version OOM-killed
production ClickHouse. `resting_heart_rate_sleep_window` is also excluded until
it is converted to a bounded dbt-native strategy over sleep windows.
`deduped_sensor`, `sleep_heart_rate_sample`,
`resting_heart_rate_sleep_window`, `activity_sensor_sample`,
`activity_location_sample`, `activity_sensor_summary_rows`,
`activity_location_summary_rows`, `activity_summary_rows`, and
`activity_vo2max_estimate`. The sample-stage and sample-intermediate models use
dbt's `microbatch` incremental strategy with `recorded_at` as the event time,
daily batches, and short lookbacks so ClickHouse processes bounded sample-time
windows instead of one large activity/window query. The final resting heart
rate, activity aggregate, and activity summary models use dirty keys from those
intermediates and `max_threads=1` to keep the offline aggregate work out of
web/API requests.
`activity_vo2max_estimate` also uses dirty activity/user keys and
`max_threads=1`; it materializes reusable per-activity VO2 max estimates, not
final API responses.
87 changes: 87 additions & 0 deletions analytics/models/read_models/activity_location_sample.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
{{ config(
materialized='incremental',
incremental_strategy='microbatch',
unique_key='source_metric_stream_id',
event_time='recorded_at',
begin='2026-01-01',
batch_size='day',
lookback=7,
full_refresh=false,
concurrent_batches=false,
engine='ReplacingMergeTree(refresh_version)',
order_by='(user_id, activity_id, recorded_date, recorded_at, source_metric_stream_id)',
query_settings={
'max_threads': 1
}
) }}

WITH activity_members AS (
SELECT
activity_id,
user_id,
member_activity_id
FROM analytics.v_activity_members
),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

location_versions AS (
SELECT *
FROM {{ source('postgres_fitness', 'metric_stream') }}
WHERE channel = 'location'
AND (point IS NOT NULL OR _peerdb_is_deleted = 1)
),

location_rows AS (
SELECT
id,
argMax(activity_id, _peerdb_version) AS member_activity_id,
argMax(user_id, _peerdb_version) AS user_id,
argMax(recorded_at, _peerdb_version) AS recorded_at,
argMax(provider_id, _peerdb_version) AS provider_id,
argMax(point, _peerdb_version) AS point,
argMax(_peerdb_is_deleted, _peerdb_version) AS is_deleted
FROM location_versions
GROUP BY id
),

provider_counts AS (
SELECT
activity_members.activity_id AS activity_id,
location_rows.provider_id AS provider_id,
count() AS sample_count,
row_number() OVER (
PARTITION BY activity_members.activity_id
ORDER BY count() DESC, location_rows.provider_id ASC
) AS row_number
FROM location_rows
INNER JOIN activity_members
ON activity_members.member_activity_id = location_rows.member_activity_id
WHERE location_rows.is_deleted = 0
Comment thread
Asherlc marked this conversation as resolved.
Resolved
AND location_rows.point IS NOT NULL
GROUP BY activity_members.activity_id, location_rows.provider_id
Comment thread
Asherlc marked this conversation as resolved.
Resolved
),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

best_source AS (
SELECT
activity_id,
provider_id
FROM provider_counts
WHERE row_number = 1
)

SELECT
activity_members.activity_id AS activity_id,
activity_members.user_id AS user_id,
location_rows.recorded_at AS recorded_at,
toDate(location_rows.recorded_at) AS recorded_date,
location_rows.id AS source_metric_stream_id,
CAST(tupleElement(location_rows.point, 2), 'Nullable(Float32)') AS lat,
CAST(tupleElement(location_rows.point, 1), 'Nullable(Float32)') AS lng,
toUInt64(toUnixTimestamp64Nano(now64(9))) AS refresh_version,
location_rows.is_deleted AS is_deleted,
now64(9) AS refreshed_at
FROM location_rows
INNER JOIN activity_members
ON activity_members.member_activity_id = location_rows.member_activity_id
INNER JOIN best_source
ON best_source.activity_id = activity_members.activity_id
AND best_source.provider_id = location_rows.provider_id
179 changes: 179 additions & 0 deletions analytics/models/read_models/activity_location_summary_rows.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
{{ config(
materialized='incremental',
incremental_strategy='append',
engine='ReplacingMergeTree(refresh_version)',
order_by='(user_id, activity_id)',
query_settings={
'max_threads': 1
}
) }}

{% set initial_lookback_days = var('initial_lookback_days', 120) %}

WITH
{% if is_incremental() %}
target_state AS (
SELECT
coalesce(
max(refreshed_at),
toDateTime64('1970-01-01 00:00:00', 9, 'UTC')
) AS last_refreshed_at,
count() = 0 AS is_empty
FROM {{ this }}
),
{% endif %}

current_activity AS (
SELECT
id AS activity_id,
user_id,
started_at
FROM analytics.v_activity
),

existing_summary AS (
{% if is_incremental() %}
SELECT
activity_id,
user_id
FROM {{ this }} FINAL
Comment thread
Asherlc marked this conversation as resolved.
Resolved
{% else %}
Comment thread
Asherlc marked this conversation as resolved.
Resolved
SELECT
CAST(null, 'Nullable(UUID)') AS activity_id,
CAST(null, 'Nullable(UUID)') AS user_id
WHERE 1 = 0
{% endif %}
),

initial_dirty_keys AS (
SELECT
activity_id,
user_id
FROM current_activity
WHERE
{% if is_incremental() %}
(SELECT is_empty FROM target_state)
AND started_at >= now64(6, 'UTC') - INTERVAL {{ initial_lookback_days }} DAY
{% else %}
started_at >= now64(6, 'UTC') - INTERVAL {{ initial_lookback_days }} DAY
{% endif %}
),

location_dirty_keys AS (
SELECT DISTINCT
activity_id,
user_id
FROM {{ ref('activity_location_sample') }}
WHERE
{% if is_incremental() %}
NOT (SELECT is_empty FROM target_state)
AND refreshed_at > (SELECT last_refreshed_at FROM target_state)
{% else %}
1 = 0
{% endif %}
),

stale_dirty_keys AS (
SELECT
existing_summary.activity_id AS activity_id,
existing_summary.user_id AS user_id
FROM existing_summary
LEFT JOIN current_activity
ON current_activity.activity_id = existing_summary.activity_id
WHERE current_activity.activity_id IS null
),

dirty_keys AS (
SELECT DISTINCT
activity_id,
user_id
FROM (
SELECT
activity_id,
user_id
FROM initial_dirty_keys
UNION ALL
SELECT
activity_id,
user_id
FROM location_dirty_keys
UNION ALL
SELECT
activity_id,
user_id
FROM stale_dirty_keys
)
),

gps_points AS (
SELECT
location_samples.activity_id AS activity_id,
location_samples.user_id AS user_id,
location_samples.recorded_at AS recorded_at,
location_samples.lat AS lat,
location_samples.lng AS lng
FROM {{ ref('activity_location_sample') }} AS location_samples
INNER JOIN dirty_keys
ON dirty_keys.activity_id = location_samples.activity_id
WHERE location_samples.is_deleted = 0
AND location_samples.lat IS NOT null
AND location_samples.lng IS NOT null
),

gps_deltas AS (
SELECT
activity_id,
lat,
lng,
lagInFrame(lat) OVER (
PARTITION BY activity_id
ORDER BY recorded_at
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
) AS prev_lat,
lagInFrame(lng) OVER (
PARTITION BY activity_id
ORDER BY recorded_at
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
) AS prev_lng
FROM gps_points
),

distance_per_activity AS (
SELECT
activity_id,
CAST(sum(
2 * 6371000 * asin(sqrt(
pow(sin(radians(lat - prev_lat) / 2), 2)
+ cos(radians(prev_lat)) * cos(radians(lat))
* pow(sin(radians(lng - prev_lng) / 2), 2)
))
), 'Nullable(Float64)') AS total_distance
FROM gps_deltas
WHERE prev_lat IS NOT null
GROUP BY activity_id
),

location_centroids AS (
SELECT
activity_id,
user_id,
CAST(avg(lat), 'Nullable(Float64)') AS centroid_lat,
CAST(avg(lng), 'Nullable(Float64)') AS centroid_lng
FROM gps_points
GROUP BY activity_id, user_id
)

SELECT
assumeNotNull(dirty_keys.activity_id) AS activity_id,
assumeNotNull(dirty_keys.user_id) AS user_id,
coalesce(distance_per_activity.total_distance, CAST(0, 'Nullable(Float64)')) AS total_distance,
location_centroids.centroid_lat AS centroid_lat,
location_centroids.centroid_lng AS centroid_lng,
toUInt64(toUnixTimestamp64Nano(now64(9))) AS refresh_version,
if(location_centroids.activity_id IS null, 1, 0) AS is_deleted,
now64(9) AS refreshed_at
FROM dirty_keys
LEFT JOIN distance_per_activity
ON distance_per_activity.activity_id = dirty_keys.activity_id
LEFT JOIN location_centroids
ON location_centroids.activity_id = dirty_keys.activity_id
53 changes: 53 additions & 0 deletions analytics/models/read_models/activity_sensor_sample.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{{ config(
materialized='incremental',
incremental_strategy='microbatch',
unique_key=['activity_id', 'channel', 'recorded_at'],
event_time='recorded_at',
begin='2026-01-01',
batch_size='day',
lookback=7,
full_refresh=false,
concurrent_batches=false,
engine='ReplacingMergeTree(refresh_version)',
order_by='(user_id, activity_id, recorded_date, channel, recorded_at)',
query_settings={
'max_threads': 1
}
) }}

WITH current_activity AS (
SELECT
id AS activity_id,
user_id,
started_at,
coalesce(ended_at, started_at + INTERVAL 12 HOUR) AS ended_at
FROM analytics.v_activity
),
Comment thread
Asherlc marked this conversation as resolved.
Resolved

activity_samples AS (
SELECT
current_activity.activity_id AS activity_id,
samples.user_id AS user_id,
samples.recorded_at AS recorded_at,
samples.recorded_date AS recorded_date,
samples.channel AS channel,
samples.scalar AS scalar,
samples.is_deleted AS is_deleted
FROM {{ ref('deduped_sensor') }} AS samples
INNER JOIN current_activity
ON current_activity.user_id = samples.user_id
AND samples.recorded_at >= current_activity.started_at
AND samples.recorded_at <= current_activity.ended_at
)

SELECT
activity_id,
user_id,
recorded_at,
recorded_date,
channel,
scalar,
toUInt64(toUnixTimestamp64Nano(now64(9))) AS refresh_version,
is_deleted,
now64(9) AS refreshed_at
FROM activity_samples
Loading
Loading