Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion analytics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,14 @@ active-row view over that dbt-owned canonical table. Insert-triggered
materialized views reduce provider changes to compact `(user_id, provider_id)`
arrival markers; `provider_change_watermark` reads only that compact state, and
`provider_stats` recounts at most one dirty provider per build so provider
inventory work cannot monopolize a cycle. A separate insert-triggered view
inventory work cannot monopolize a cycle. Its exact metric-stream count uses
the aggregate `by_provider_current_state` projection, which maintains the
latest deletion state per provider record ID as data arrives and lets the
recount merge precomputed `argMax` states instead of rebuilding a
high-cardinality latest-row hash table from the raw stream. ClickHouse
projections can precompute aggregates and are maintained automatically for new
inserts:
<https://clickhouse.com/docs/data-modeling/projections>. A separate insert-triggered view
reduces heart-rate arrivals to user/day markers. `sleep_heart_rate_window`
uses those markers to process at most 32 exact sleep windows, including
processed-empty and lifecycle rows, before `sleep_heart_rate_sample` reads
Expand Down
4 changes: 3 additions & 1 deletion analytics/models/read_models/provider_stats.sql
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@
query_settings={
'max_threads': 1,
'join_use_nulls': 1,
'enable_materialized_cte': 1
'enable_materialized_cte': 1,
'optimize_aggregation_in_order': 1,
'preferred_optimize_projection_name': 'by_provider_current_state'
}
) }}

Expand Down
4 changes: 4 additions & 0 deletions analytics/models/read_models/provider_stats.sql.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ describe("provider_stats model", () => {
expect(modelSql).toContain("metric_stream_current AS");
expect(modelSql).toContain("argMax(is_deleted, tuple(version, ingested_at)) AS is_deleted");
expect(modelSql).toContain("'enable_materialized_cte': 1");
expect(modelSql).toContain("'optimize_aggregation_in_order': 1");
expect(modelSql).toContain(
"'preferred_optimize_projection_name': 'by_provider_current_state'",
);
expect(modelSql).toContain("current_provider_state AS materialized");
expect(modelSql).toContain("providers AS materialized");
expect(modelSql).toContain("provider_dirty_key_batch_size");
Expand Down
61 changes: 58 additions & 3 deletions docs/clickhouse-metric-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,16 @@ Provider record inventory uses the ClickHouse `analytics.provider_stats` read
model for all provider-owned record counts displayed by sync/provider detail:
activity, daily metric, sleep, body measurement, food entry, health event,
metric stream, distinct nutrition day, lab panel, lab result, and journal entry
counts. The provider detail UI still treats these as raw provider-owned record
counts, not deduped analytical sample counts.
counts. Exact metric-stream counts prefer the
`ingest.metric_stream.by_provider_current_state` aggregate projection, which
stores mergeable latest-`is_deleted` state per `(user_id, provider_id, id)`.
This retains `argMax(version, ingested_at)` correctness for tombstones and
later live replacements without rebuilding the full per-ID state from raw
rows during every provider refresh. ClickHouse documents aggregate projections
as precomputed query data maintained on insert:
<https://clickhouse.com/docs/data-modeling/projections>. The provider detail UI
still treats these as raw provider-owned record counts, not deduped analytical
sample counts.

## Scalar And Location Projections

Expand Down Expand Up @@ -133,7 +141,10 @@ ClickHouse migrations create and update the databases and read models:
rows may still have been backfilled from Postgres, but new forward rows come
from Redpanda events. Its `version` and `is_deleted` columns encode replacement
order and logical deletion so current-state queries can select the latest live
row ([ReplacingMergeTree](https://clickhouse.com/docs/en/guides/replacing-merge-tree)).
row. Its `by_provider_current_state` aggregate projection computes the same
latest-row state in advance at provider-record grain for exact inventory counts
([ReplacingMergeTree](https://clickhouse.com/docs/en/guides/replacing-merge-tree),
[projections](https://clickhouse.com/docs/data-modeling/projections)).
- `ingest.metric_stream_delete_acknowledgement`: one receipt per version 2
deletion event, written only after the sink's tombstone insert completes.
- `ingest.metric_stream_processing_acknowledgement`: one receipt per stable
Expand Down Expand Up @@ -179,6 +190,50 @@ ClickHouse migrations create and update the databases and read models:
sensor trend row per user and UTC day. It is derived from
`analytics.deduped_sensor`.

### Provider inventory projection rollout

Migration `0061_provider_current_state_projection` adds the
`by_provider_current_state` definition. New metric-stream parts populate it
automatically, but existing parts do not receive a newly added projection until
an operator materializes it. ClickHouse documents `MATERIALIZE PROJECTION` as
the required existing-data step:
<https://clickhouse.com/docs/data-modeling/projections#filtering-on-columns-which-arent-in-the-primary-key>.

Materialization rewrites historical parts, so it is an explicit maintenance
operation rather than a deploy migration:

```sql
ALTER TABLE ingest.metric_stream
MATERIALIZE PROJECTION by_provider_current_state;
```

Monitor the mutation until `is_done = 1` and `latest_fail_reason` is empty:

```sql
SELECT
mutation_id,
command,
is_done,
latest_fail_reason
FROM system.mutations
WHERE database = 'ingest'
AND table = 'metric_stream'
ORDER BY create_time DESC;
```

Then verify every active base-table part contains the projection:

```sql
SELECT countIf(NOT has(
projections,
'by_provider_current_state'
)) AS missing_projection_parts
FROM system.parts
WHERE active
AND database = 'ingest'
AND table = 'metric_stream';
```

### Deletion protocol

New metric-stream deletions are version 2 Redpanda events with a unique event
Expand Down
52 changes: 52 additions & 0 deletions docs/production-incident-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -19333,3 +19333,55 @@ Drizzle schema and runtime Zod schemas. Findings and remediations:
- **Remaining risk / follow-up:** A second external download failure should be
treated as an unresolved upstream transport incident rather than papered over
with repository retry or timeout changes.

## 2026-07-27 — Provider Inventory Current-State Aggregation Timed Out (DOFEK-SERVER-5A)

- **Status:** Direct source fix reproduced and validated locally; deployment,
historical projection materialization, and production validation pending.
- **Symptoms:** The production analytics worker on release `f423b5b` continued
reporting
[DOFEK-SERVER-5A](https://east-bay-software.sentry.io/issues/DOFEK-SERVER-5A)
after compact dirty-key discovery had already reduced each build to one
provider.
- **User impact:** Every failed serial dbt cycle stopped before downstream
analytics and cache warming, so provider inventory and later dashboard,
sleep, recovery, training, cycling, and healthspan read models could remain
stale.
- **Evidence:** The exact failing command was the scheduled production
`provider_stats` dbt incremental insert. Its first fatal line was ClickHouse
code 159: `Timeout exceeded: elapsed 243415.214164 ms, maximum: 240000 ms`.
Query `177a3b92-6f01-4d04-806e-8d24525a4985` read 33,818,056 rows / 1.79
GiB, peaked at 5.59 GiB, and spent 166.9 CPU-seconds waiting. It did select
`ingest.metric_stream.by_provider_generation`; the selected dirty provider
was `apple_motion`, whose physical metric-stream range contained 33,730,657
rows. A direct physical count completed quickly, isolating the failure to
latest-version aggregation rather than provider-range pruning.
- **Root cause:** Dirty discovery was bounded, but exact record counting still
rebuilt one `argMax(is_deleted, (version, ingested_at))` hash entry for every
provider record ID. The covering generation projection pruned to the correct
provider but did not precompute that high-cardinality current-state
aggregation.
- **Fix / mitigation:** Add the aggregate
`by_provider_current_state` projection at
`(user_id, provider_id, id)` grain and have `provider_stats` prefer it with
in-order aggregation enabled. The projection stores mergeable exact
`argMax` states, so tombstones and later live replacements retain the
existing latest-version semantics. Migration `0061` adds only the projection
definition; historical materialization remains an explicit operator action
outside deploy migrations. ClickHouse documents projections as precomputed
query data maintained for new inserts and requires `MATERIALIZE PROJECTION`
for existing parts:
<https://clickhouse.com/docs/data-modeling/projections>.
- **Validation:** The regression first failed because the projection and
migration were absent. After the fix, focused unit and real-ClickHouse
integration suites pass, as do the full lint, TypeScript, and Docker-free test
gates (14,097 tests). The integration suite proves tombstone-then-live
replacement correctness across separate parts, automatic planner selection
without making the projection a correctness prerequisite, 50,000 exact IDs
within a 32 MiB query budget, and explicit materialization of parts created
before the projection.
- **Remaining risk / follow-up:** Merge and deploy through the normal release
path, materialize `by_provider_current_state` while monitoring
`system.mutations`, verify zero active parts lack the projection, observe a
complete production analytics-plus-cache cycle below the unchanged
four-minute ceiling, and only then resolve DOFEK-SERVER-5A.
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ const providerSchema = z.array(
provider_id: z.string(),
}),
);
const explainSchema = z.array(z.object({ explain: z.string() }));

function stripDbtModelWrapper(modelSql: string): string {
return modelSql
Expand Down Expand Up @@ -95,14 +96,23 @@ function renderDirtyProviderSelectionSql(
FROM providers`;
}

function renderMetricStreamCountSql(ingestDatabase: string): string {
function renderMetricStreamCountSql(
ingestDatabase: string,
projectionMode: "force" | "prefer" = "force",
): string {
const modelSql = readModelSql("provider_stats.sql");
const currentSql = extractCteSql(modelSql, "metric_stream_current").replace(
/\{\{\s*source\('ingest',\s*'metric_stream'\)\s*\}\}/g,
`${ingestDatabase}.metric_stream`,
);
const countsSql = extractCteSql(modelSql, "metric_stream_counts");

const projectionSettings =
projectionMode === "force"
? `force_optimize_projection = 1,
force_optimize_projection_name = 'by_provider_current_state'`
: "preferred_optimize_projection_name = 'by_provider_current_state'";

Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Resolved
return `WITH providers AS (
SELECT {userId:UUID} AS user_id, 'test_provider' AS provider_id
),
Expand All @@ -115,7 +125,11 @@ function renderMetricStreamCountSql(ingestDatabase: string): string {
SELECT metric_stream_counts.count AS metric_stream
FROM metric_stream_counts
WHERE user_id = {userId:UUID}
AND provider_id = 'test_provider'`;
AND provider_id = 'test_provider'
SETTINGS
${projectionSettings},
optimize_aggregation_in_order = 1,
max_memory_usage = 33554432`;
}

function renderProviderConnectionSql(database: string): string {
Expand Down Expand Up @@ -162,7 +176,7 @@ describe("provider stats read model", () => {
await client?.command({ query: `DROP DATABASE IF EXISTS ${ingestDatabase}` });
});

it("counts only the current live version of each metric-stream row", async () => {
it("counts the current live version through the provider current-state projection", async () => {
const userId = randomUUID();
const deletedId = randomUUID();
const liveId = randomUUID();
Expand All @@ -182,6 +196,10 @@ describe("provider stats read model", () => {
{deletedId:UUID}, NULL, {userId:UUID}, {recordedAt:DateTime64(6, 'UTC')}, 'heart_rate', 'test_provider',
NULL, NULL, 'api', 60, [], '', '', now64(9), 1, 2, 0
),
(
{deletedId:UUID}, NULL, {userId:UUID}, {recordedAt:DateTime64(6, 'UTC')}, 'heart_rate', 'test_provider',
NULL, NULL, 'api', 60, [], '', '', now64(9) + INTERVAL 1 SECOND, 0, 3, 0
),
(
{liveId:UUID}, NULL, {userId:UUID}, {recordedAt:DateTime64(6, 'UTC')}, 'heart_rate', 'test_provider',
NULL, NULL, 'api', 61, [], '', '', now64(9), 0, 1, 0
Expand All @@ -195,7 +213,83 @@ describe("provider stats read model", () => {
format: "JSONEachRow",
});

expect(providerCountSchema.parse(await result.json())).toEqual([{ metric_stream: 1 }]);
expect(providerCountSchema.parse(await result.json())).toEqual([{ metric_stream: 2 }]);
});

it("selects the provider current-state projection without making it mandatory", async () => {
const userId = randomUUID();
await client.command({
query: `INSERT INTO ${ingestDatabase}.metric_stream (
id,
user_id,
recorded_at,
channel,
provider_id,
source_type,
is_deleted,
version
) VALUES (
generateUUIDv4(),
{userId:UUID},
now64(6),
'heart_rate',
'test_provider',
'test',
0,
1
)`,
query_params: { userId },
});
const result = await client.query({
query: `EXPLAIN projections = 1
${renderMetricStreamCountSql(ingestDatabase, "prefer")}`,
query_params: { userId },
format: "JSONEachRow",
});

const explain = explainSchema
.parse(await result.json())
.map((row) => row.explain)
.join("\n");
expect(explain).toContain("ReadFromMergeTree (by_provider_current_state)");
});

it("keeps a high-cardinality exact recount within a bounded memory budget", async () => {
const userId = randomUUID();
await client.command({
query: `INSERT INTO ${ingestDatabase}.metric_stream (
id,
user_id,
recorded_at,
channel,
provider_id,
source_type,
is_deleted,
version
)
SELECT
toUUID(concat(
'00000000-0000-0000-0000-',
leftPad(toString(number), 12, '0')
)),
{userId:UUID},
now64(6),
'heart_rate',
'test_provider',
'test',
toInt8(0),
toInt64(1)
FROM numbers(50000)`,
query_params: { userId },
});

const result = await client.query({
query: renderMetricStreamCountSql(ingestDatabase),
query_params: { userId },
format: "JSONEachRow",
});

expect(providerCountSchema.parse(await result.json())).toEqual([{ metric_stream: 50000 }]);
});

it("tracks provider connection and disconnection versions by provider_id", async () => {
Expand Down
Loading
Loading