Repository navigation
Convert analytics read models to bounded microbatches #1185
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| {{ config( | ||
| materialized='incremental', | ||
| incremental_strategy='microbatch', | ||
| unique_key='source_metric_stream_id', | ||
| event_time='recorded_at', | ||
| begin='2026-01-01', | ||
| batch_size='day', | ||
| lookback=7, | ||
| full_refresh=false, | ||
| concurrent_batches=false, | ||
| engine='ReplacingMergeTree(refresh_version)', | ||
| order_by='(user_id, activity_id, recorded_date, recorded_at, source_metric_stream_id)', | ||
| query_settings={ | ||
| 'max_threads': 1 | ||
| } | ||
| ) }} | ||
|
|
||
| WITH activity_members AS ( | ||
| SELECT | ||
| activity_id, | ||
| user_id, | ||
| member_activity_id | ||
| FROM analytics.v_activity_members | ||
| ), | ||
|
|
||
| location_versions AS ( | ||
| SELECT * | ||
| FROM {{ source('postgres_fitness', 'metric_stream') }} | ||
| WHERE channel = 'location' | ||
| AND (point IS NOT NULL OR _peerdb_is_deleted = 1) | ||
| ), | ||
|
|
||
| location_rows AS ( | ||
| SELECT | ||
| id, | ||
| argMax(activity_id, _peerdb_version) AS member_activity_id, | ||
| argMax(user_id, _peerdb_version) AS user_id, | ||
| argMax(recorded_at, _peerdb_version) AS recorded_at, | ||
| argMax(provider_id, _peerdb_version) AS provider_id, | ||
| argMax(point, _peerdb_version) AS point, | ||
| argMax(_peerdb_is_deleted, _peerdb_version) AS is_deleted | ||
| FROM location_versions | ||
| GROUP BY id | ||
| ), | ||
|
|
||
| provider_counts AS ( | ||
| SELECT | ||
| activity_members.activity_id AS activity_id, | ||
| location_rows.provider_id AS provider_id, | ||
| count() AS sample_count, | ||
| row_number() OVER ( | ||
| PARTITION BY activity_members.activity_id | ||
| ORDER BY count() DESC, location_rows.provider_id ASC | ||
| ) AS row_number | ||
| FROM location_rows | ||
| INNER JOIN activity_members | ||
| ON activity_members.member_activity_id = location_rows.member_activity_id | ||
| WHERE location_rows.is_deleted = 0 | ||
|
Asherlc marked this conversation as resolved.
Resolved
|
||
| AND location_rows.point IS NOT NULL | ||
| GROUP BY activity_members.activity_id, location_rows.provider_id | ||
|
Asherlc marked this conversation as resolved.
Resolved
|
||
| ), | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| best_source AS ( | ||
| SELECT | ||
| activity_id, | ||
| provider_id | ||
| FROM provider_counts | ||
| WHERE row_number = 1 | ||
| ) | ||
|
|
||
| SELECT | ||
| activity_members.activity_id AS activity_id, | ||
| activity_members.user_id AS user_id, | ||
| location_rows.recorded_at AS recorded_at, | ||
| toDate(location_rows.recorded_at) AS recorded_date, | ||
| location_rows.id AS source_metric_stream_id, | ||
| CAST(tupleElement(location_rows.point, 2), 'Nullable(Float32)') AS lat, | ||
| CAST(tupleElement(location_rows.point, 1), 'Nullable(Float32)') AS lng, | ||
| toUInt64(toUnixTimestamp64Nano(now64(9))) AS refresh_version, | ||
| location_rows.is_deleted AS is_deleted, | ||
| now64(9) AS refreshed_at | ||
| FROM location_rows | ||
| INNER JOIN activity_members | ||
| ON activity_members.member_activity_id = location_rows.member_activity_id | ||
| INNER JOIN best_source | ||
| ON best_source.activity_id = activity_members.activity_id | ||
| AND best_source.provider_id = location_rows.provider_id | ||
179 changes: 179 additions & 0 deletions
179
analytics/models/read_models/activity_location_summary_rows.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| {{ config( | ||
| materialized='incremental', | ||
| incremental_strategy='append', | ||
| engine='ReplacingMergeTree(refresh_version)', | ||
| order_by='(user_id, activity_id)', | ||
| query_settings={ | ||
| 'max_threads': 1 | ||
| } | ||
| ) }} | ||
|
|
||
| {% set initial_lookback_days = var('initial_lookback_days', 120) %} | ||
|
|
||
| WITH | ||
| {% if is_incremental() %} | ||
| target_state AS ( | ||
| SELECT | ||
| coalesce( | ||
| max(refreshed_at), | ||
| toDateTime64('1970-01-01 00:00:00', 9, 'UTC') | ||
| ) AS last_refreshed_at, | ||
| count() = 0 AS is_empty | ||
| FROM {{ this }} | ||
| ), | ||
| {% endif %} | ||
|
|
||
| current_activity AS ( | ||
| SELECT | ||
| id AS activity_id, | ||
| user_id, | ||
| started_at | ||
| FROM analytics.v_activity | ||
| ), | ||
|
|
||
| existing_summary AS ( | ||
| {% if is_incremental() %} | ||
| SELECT | ||
| activity_id, | ||
| user_id | ||
| FROM {{ this }} FINAL | ||
|
Asherlc marked this conversation as resolved.
Resolved
|
||
| {% else %} | ||
|
Asherlc marked this conversation as resolved.
Resolved
|
||
| SELECT | ||
| CAST(null, 'Nullable(UUID)') AS activity_id, | ||
| CAST(null, 'Nullable(UUID)') AS user_id | ||
| WHERE 1 = 0 | ||
| {% endif %} | ||
| ), | ||
|
|
||
| initial_dirty_keys AS ( | ||
| SELECT | ||
| activity_id, | ||
| user_id | ||
| FROM current_activity | ||
| WHERE | ||
| {% if is_incremental() %} | ||
| (SELECT is_empty FROM target_state) | ||
| AND started_at >= now64(6, 'UTC') - INTERVAL {{ initial_lookback_days }} DAY | ||
| {% else %} | ||
| started_at >= now64(6, 'UTC') - INTERVAL {{ initial_lookback_days }} DAY | ||
| {% endif %} | ||
| ), | ||
|
|
||
| location_dirty_keys AS ( | ||
| SELECT DISTINCT | ||
| activity_id, | ||
| user_id | ||
| FROM {{ ref('activity_location_sample') }} | ||
| WHERE | ||
| {% if is_incremental() %} | ||
| NOT (SELECT is_empty FROM target_state) | ||
| AND refreshed_at > (SELECT last_refreshed_at FROM target_state) | ||
| {% else %} | ||
| 1 = 0 | ||
| {% endif %} | ||
| ), | ||
|
|
||
| stale_dirty_keys AS ( | ||
| SELECT | ||
| existing_summary.activity_id AS activity_id, | ||
| existing_summary.user_id AS user_id | ||
| FROM existing_summary | ||
| LEFT JOIN current_activity | ||
| ON current_activity.activity_id = existing_summary.activity_id | ||
| WHERE current_activity.activity_id IS null | ||
| ), | ||
|
|
||
| dirty_keys AS ( | ||
| SELECT DISTINCT | ||
| activity_id, | ||
| user_id | ||
| FROM ( | ||
| SELECT | ||
| activity_id, | ||
| user_id | ||
| FROM initial_dirty_keys | ||
| UNION ALL | ||
| SELECT | ||
| activity_id, | ||
| user_id | ||
| FROM location_dirty_keys | ||
| UNION ALL | ||
| SELECT | ||
| activity_id, | ||
| user_id | ||
| FROM stale_dirty_keys | ||
| ) | ||
| ), | ||
|
|
||
| gps_points AS ( | ||
| SELECT | ||
| location_samples.activity_id AS activity_id, | ||
| location_samples.user_id AS user_id, | ||
| location_samples.recorded_at AS recorded_at, | ||
| location_samples.lat AS lat, | ||
| location_samples.lng AS lng | ||
| FROM {{ ref('activity_location_sample') }} AS location_samples | ||
| INNER JOIN dirty_keys | ||
| ON dirty_keys.activity_id = location_samples.activity_id | ||
| WHERE location_samples.is_deleted = 0 | ||
| AND location_samples.lat IS NOT null | ||
| AND location_samples.lng IS NOT null | ||
| ), | ||
|
|
||
| gps_deltas AS ( | ||
| SELECT | ||
| activity_id, | ||
| lat, | ||
| lng, | ||
| lagInFrame(lat) OVER ( | ||
| PARTITION BY activity_id | ||
| ORDER BY recorded_at | ||
| ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | ||
| ) AS prev_lat, | ||
| lagInFrame(lng) OVER ( | ||
| PARTITION BY activity_id | ||
| ORDER BY recorded_at | ||
| ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | ||
| ) AS prev_lng | ||
| FROM gps_points | ||
| ), | ||
|
|
||
| distance_per_activity AS ( | ||
| SELECT | ||
| activity_id, | ||
| CAST(sum( | ||
| 2 * 6371000 * asin(sqrt( | ||
| pow(sin(radians(lat - prev_lat) / 2), 2) | ||
| + cos(radians(prev_lat)) * cos(radians(lat)) | ||
| * pow(sin(radians(lng - prev_lng) / 2), 2) | ||
| )) | ||
| ), 'Nullable(Float64)') AS total_distance | ||
| FROM gps_deltas | ||
| WHERE prev_lat IS NOT null | ||
| GROUP BY activity_id | ||
| ), | ||
|
|
||
| location_centroids AS ( | ||
| SELECT | ||
| activity_id, | ||
| user_id, | ||
| CAST(avg(lat), 'Nullable(Float64)') AS centroid_lat, | ||
| CAST(avg(lng), 'Nullable(Float64)') AS centroid_lng | ||
| FROM gps_points | ||
| GROUP BY activity_id, user_id | ||
| ) | ||
|
|
||
| SELECT | ||
| assumeNotNull(dirty_keys.activity_id) AS activity_id, | ||
| assumeNotNull(dirty_keys.user_id) AS user_id, | ||
| coalesce(distance_per_activity.total_distance, CAST(0, 'Nullable(Float64)')) AS total_distance, | ||
| location_centroids.centroid_lat AS centroid_lat, | ||
| location_centroids.centroid_lng AS centroid_lng, | ||
| toUInt64(toUnixTimestamp64Nano(now64(9))) AS refresh_version, | ||
| if(location_centroids.activity_id IS null, 1, 0) AS is_deleted, | ||
| now64(9) AS refreshed_at | ||
| FROM dirty_keys | ||
| LEFT JOIN distance_per_activity | ||
| ON distance_per_activity.activity_id = dirty_keys.activity_id | ||
| LEFT JOIN location_centroids | ||
| ON location_centroids.activity_id = dirty_keys.activity_id | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| {{ config( | ||
| materialized='incremental', | ||
| incremental_strategy='microbatch', | ||
| unique_key=['activity_id', 'channel', 'recorded_at'], | ||
| event_time='recorded_at', | ||
| begin='2026-01-01', | ||
| batch_size='day', | ||
| lookback=7, | ||
| full_refresh=false, | ||
| concurrent_batches=false, | ||
| engine='ReplacingMergeTree(refresh_version)', | ||
| order_by='(user_id, activity_id, recorded_date, channel, recorded_at)', | ||
| query_settings={ | ||
| 'max_threads': 1 | ||
| } | ||
| ) }} | ||
|
|
||
| WITH current_activity AS ( | ||
| SELECT | ||
| id AS activity_id, | ||
| user_id, | ||
| started_at, | ||
| coalesce(ended_at, started_at + INTERVAL 12 HOUR) AS ended_at | ||
| FROM analytics.v_activity | ||
| ), | ||
|
Asherlc marked this conversation as resolved.
Resolved
|
||
|
|
||
| activity_samples AS ( | ||
| SELECT | ||
| current_activity.activity_id AS activity_id, | ||
| samples.user_id AS user_id, | ||
| samples.recorded_at AS recorded_at, | ||
| samples.recorded_date AS recorded_date, | ||
| samples.channel AS channel, | ||
| samples.scalar AS scalar, | ||
| samples.is_deleted AS is_deleted | ||
| FROM {{ ref('deduped_sensor') }} AS samples | ||
| INNER JOIN current_activity | ||
| ON current_activity.user_id = samples.user_id | ||
| AND samples.recorded_at >= current_activity.started_at | ||
| AND samples.recorded_at <= current_activity.ended_at | ||
| ) | ||
|
|
||
| SELECT | ||
| activity_id, | ||
| user_id, | ||
| recorded_at, | ||
| recorded_date, | ||
| channel, | ||
| scalar, | ||
| toUInt64(toUnixTimestamp64Nano(now64(9))) AS refresh_version, | ||
| is_deleted, | ||
| now64(9) AS refreshed_at | ||
| FROM activity_samples | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.