Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions analytics/models/read_models/activity_sensor_summary_rows.sql
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ initial_dirty_keys AS (
FROM current_activity
WHERE
{% if is_incremental() %}
(SELECT is_empty FROM target_state LIMIT 1)
(SELECT is_empty FROM target_state)
AND started_at >= now64(6, 'UTC') - INTERVAL {{ initial_lookback_days }} DAY
{% else %}
started_at >= now64(6, 'UTC') - INTERVAL {{ initial_lookback_days }} DAY
Expand All @@ -88,8 +88,8 @@ changed_sample_dirty_keys AS (
ON existing_summary.activity_id = sensor_sample.activity_id
AND existing_summary.user_id = sensor_sample.user_id
WHERE
NOT (SELECT is_empty FROM target_state LIMIT 1)
AND sensor_sample.refreshed_at > (SELECT last_refreshed_at FROM target_state LIMIT 1)
NOT (SELECT is_empty FROM target_state)
AND sensor_sample.refreshed_at > (SELECT last_refreshed_at FROM target_state)
AND (
existing_summary.activity_id IS null
OR sensor_sample.refreshed_at > existing_summary.refreshed_at
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -433,10 +433,10 @@ describe("production analytics read-model build", () => {
expect(normalizedSql).toContain("argMax(refreshed_at, refresh_version) AS refreshed_at");
expect(normalizedSql).toContain("argMax(is_deleted, refresh_version) AS is_deleted");
expect(normalizedSql).toContain("FROM existing_summary_state WHERE is_deleted = 0");
expect(normalizedSql).toContain("(SELECT is_empty FROM target_state LIMIT 1)");
expect(normalizedSql).toContain("NOT (SELECT is_empty FROM target_state LIMIT 1)");
expect(normalizedSql).toContain("(SELECT is_empty FROM target_state)");
expect(normalizedSql).toContain("NOT (SELECT is_empty FROM target_state)");
expect(normalizedSql).toContain(
"sensor_sample.refreshed_at > (SELECT last_refreshed_at FROM target_state LIMIT 1)",
"sensor_sample.refreshed_at > (SELECT last_refreshed_at FROM target_state)",
);
expect(normalizedSql).toContain(
"sensor_sample.refreshed_at > existing_summary.refreshed_at",
Expand Down
3 changes: 3 additions & 0 deletions cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,9 @@
"ingestion",
"ingestible",
"reingestion",
"Redrive",
"redrive",
"redrives",
"denormalized",
"denormalize",
"backfill",
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ Cross-provider reverse-engineering references:
| [clickhouse-read-model-deploy-runbook.md](clickhouse-read-model-deploy-runbook.md) | Deploy failures around ClickHouse CDC, analytics read models, and hot fitness views. |
| [clickhouse-cdc-health-runbook.md](clickhouse-cdc-health-runbook.md) | Preventing, diagnosing, and recovering lost PeerDB CDC slots. |
| [metric-stream-redpanda-r2-runbook.md](metric-stream-redpanda-r2-runbook.md) | Target Redpanda and R2 replay path for durable `metric_stream` rebuilds. |
| [provider-data-deletion-runbook.md](provider-data-deletion-runbook.md) | Generation-fenced provider deletion, transactional outbox delivery, BullMQ checkpoints, and ClickHouse tombstone verification. |
| [clickhouse-activity-dedup-runbook.md](clickhouse-activity-dedup-runbook.md) | Keeping ClickHouse activity read models on canonical deduped activity IDs. |
| [production-incident-baseline.md](production-incident-baseline.md) | Baseline knowledge from production incidents and recurring failure patterns. |
| [provider-sync-degradation-runbook.md](provider-sync-degradation-runbook.md) | Querying degraded provider sync rows and correlating pagination fingerprints with metrics and structured logs. |
Expand Down
78 changes: 78 additions & 0 deletions docs/production-incident-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -13770,3 +13770,81 @@ Drizzle schema and runtime Zod schemas. Findings and remediations:
deployment, verify its event ID appears in
`ingest.metric_stream_delete_acknowledgement`, confirm zero live logical IDs,
and verify the rebuilt analytics plus user cache.

## 2026-07-18 — Garmin Dump Tombstones Applied Without Acknowledgement

- **Symptoms:** Re-emitting the Garmin Dump provider deletion after deploying
image `sha-cbacad2` caused the analytics refresh job to time out waiting for
deletion event `42712966-7550-4eb4-8347-7ee99b6230af`. The ClickHouse sink
continued retrying the preceding provider-scoped event at Redpanda partition
0 offset `604114013`.
- **User impact:** The raw metric IDs were tombstoned, but the deletion workflow
did not reach its acknowledgement gate, so the requested analytics rebuild
and cache refresh did not complete.
- **Evidence:** The exact failing operation was the sink's provider-scoped
`INSERT INTO ingest.metric_stream SELECT ...` for event
`4e9d0632-3bce-4174-b685-5380687ce040`. The first fatal sink log line was
`Error: Timeout error` at `2026-07-18T15:27:03.430Z`; the next attempt failed
with ClickHouse code 241 (`MEMORY_LIMIT_EXCEEDED`) at
`2026-07-18T15:27:06.513Z`. Sentry issue
[DOFEK-SERVER-4T](https://east-bay-software.sentry.io/issues/DOFEK-SERVER-4T)
recorded four analytics acknowledgement timeouts. Production contained
12,522,091 inserted Garmin Dump versions and 3,130,320 tombstone versions.
Low-memory distinct estimates matched at 3,122,004 source IDs and 3,122,004
tombstoned IDs, with the latest tombstone committed at
`2026-07-18T15:35:26.697Z`, while the target acknowledgement count remained
zero. The linked
[deployment job](https://github.com/Asherlc/dofek/actions/runs/29632718641/job/88049486344)
deployed the sink image successfully even though the overall job failed when
`dofek_worker` rolled back.
- **Root cause:** The provider-wide tombstone query runs longer than the
ClickHouse client's 120-second request timeout. ClickHouse continues and
commits the server-side insert after the client disconnects, but the sink sees
a rejected command and therefore never writes the acknowledgement. Kafka then
retries the same expensive event, and the analytics job times out waiting for
a receipt that cannot be emitted by that attempt.
- **Fix / mitigation:** No runtime behavior was changed during verification. A
diagnostic latest-state query was canceled so it would not compete with the
production deletion. The committed tombstones were verified directly. The
repository follow-up replaces provider-wide tombstone queries with a
generation-fenced transactional outbox and a BullMQ worker that checkpoints
bounded batches of 10,000 exact-ID tombstones before acknowledgement.
- **Remaining risk / follow-up:** The bounded workflow is not active until its
PostgreSQL and ClickHouse migrations plus worker deployment complete. After
deployment, explicitly materialize the historical ClickHouse projection in
an approved maintenance window, resubmit the deletion request, and verify the
generation fence, acknowledgement, and zero active old-generation rows using
`docs/provider-data-deletion-runbook.md`.

## 2026-07-18 — Deletion Workflow PR Validation Failures

- **Symptoms:** The deletion workflow pull request failed its unit and
integration CI jobs. A subsequent local full-suite validation produced
cascading PostgreSQL `No space left on device` failures.
- **User impact:** No production impact; the failures blocked validation of the
replacement deletion workflow before deployment.
- **Evidence:** In [GitHub Actions run
29654216514](https://github.com/Asherlc/dofek/actions/runs/29654216514), the
unit job first failed on stale `LIMIT 1` SQL assertions and integration shard
3 first failed because ClickHouse rejects lightweight `DELETE` on a table
with projections. Locally, the Docker VM overlay reached 100% utilization and
PostgreSQL emitted `No space left on device`; the current workspace database
contained about 160 disposable `dofek_integration_template_*` and `test_*`
databases from integration runs.
- **Root cause:** Tests had not been updated for the corrected scalar subqueries
or projection-aware ClickHouse mutation semantics. Separately, completed
integration runs left disposable template databases in the shared Docker VM
until its filesystem was exhausted.
- **Fix / mitigation:** Updated the SQL expectations, changed ClickHouse test
cleanup to synchronous `ALTER TABLE ... DELETE` mutations, and dropped only
the current workspace's disposable test/template databases. The `health`
database, running containers, and all named volumes were preserved. Docker's
official guidance distinguishes pruning rebuildable caches and unused objects
from explicit volume removal
([Docker pruning guidance](https://docs.docker.com/engine/manage-resources/pruning/)).
- **Validation:** The affected 256 unit tests and 12 real PostgreSQL/ClickHouse
integration tests pass after the fixes.
- **Remaining risk / follow-up:** The shared Docker VM has limited free space,
so a non-sharded local full suite may still exceed capacity. Add deterministic
integration-template cleanup and a Docker disk preflight to the testing
runbook; use sharded CI as the authoritative full-suite validation until then.
123 changes: 123 additions & 0 deletions docs/provider-data-deletion-runbook.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Provider Data Deletion Runbook

Provider deletion is an asynchronous, generation-fenced workflow. It does not publish one broker event per stored metric. The API records one durable deletion request, and a BullMQ worker writes bounded batches of exact-ID tombstones directly to ClickHouse.

## Industry Terms

| Term | Meaning in Dofek |
|------|------------------|
| Transactional outbox | The provider record deletion, generation increment, and deletion request are committed in one PostgreSQL transaction. PostgreSQL transactions make those steps an all-or-nothing operation: <https://www.postgresql.org/docs/current/tutorial-transactions.html>. |
| Generation / fencing token | A monotonically increasing number for one `(user_id, provider_id)`. Events from an older generation are stale and cannot become active after deletion. |
| Idempotency key | The outbox `event_id`, also used as the BullMQ `jobId`. BullMQ custom job IDs prevent a second job with the same ID while the retained job exists: <https://docs.bullmq.io/guide/jobs/job-ids>. |
| Checkpoint | BullMQ job data containing the last processed metric-stream ID and cumulative batch/row counts. A retry resumes after the last durable checkpoint. |
| Tombstone | A new version of an existing metric-stream row with `is_deleted = 1`. Raw history remains available while current-state queries treat the ID as deleted. |
| Projection | ClickHouse's hidden, query-optimized ordering of the same table data. `by_provider_generation` orders IDs by user, provider, and generation so deletion scans do not require a full table scan. ClickHouse maintains projections for new inserts automatically: <https://clickhouse.com/docs/data-modeling/projections>. |
| Acknowledgement | A durable ClickHouse row proving the metric-stream deletion batches finished before downstream analytics refresh begins. |

## Control Flow

1. The API starts a PostgreSQL transaction.
2. It deletes the provider's relational records but preserves the provider connection token.
3. It increments `fitness.provider_data_generation.current_generation` and inserts one row into `fitness.provider_data_deletion_outbox` in the same transaction.
4. The outbox dispatcher adds a `provider-data-deletion` BullMQ job using `event_id` as the idempotency key, then marks the outbox row `dispatched`.
5. The worker advances the ClickHouse generation fence in `ingest.provider_data_generation`.
6. The worker scans active rows from older generations in batches of 10,000 IDs, writes exact-ID tombstones, and persists a checkpoint after every batch.
7. When no older active rows remain, the worker writes `ingest.metric_stream_delete_acknowledgement`, enqueues the provider analytics refresh, and marks the outbox row `completed`.

Metric-stream writers stamp every new event with the provider's current generation. After inserting a batch, the ClickHouse sink checks the generation fence and immediately tombstones exact IDs from older generations. The post-insert check closes the race between an in-flight sink batch and a newly advanced fence: either the deletion worker sees the row in its scan, or the sink sees the fence after insertion. The jobs are intentionally idempotent because BullMQ can deliver a job again after a worker failure; BullMQ documents this retry-safe design as idempotent jobs: <https://docs.bullmq.io/patterns/idempotent-jobs>.

## Deployment And Historical Projection

The ClickHouse migration adds `generation`, the generation-fence table, and the `by_provider_generation` projection definition. New parts populate the projection automatically. Existing parts remain correct but do not receive the reordered projection until an operator explicitly materializes it. ClickHouse requires `MATERIALIZE PROJECTION` for existing data: <https://clickhouse.com/docs/data-modeling/projections#filtering-on-columns-which-arent-in-the-primary-key>.

Materialization rewrites historical data, so do not put it in the deploy migration. Run it only in an approved maintenance window:

```sql
ALTER TABLE ingest.metric_stream
MATERIALIZE PROJECTION by_provider_generation;
```

Monitor progress and failures:

```sql
SELECT
mutation_id,
command,
is_done,
latest_fail_reason
FROM system.mutations
WHERE database = 'ingest'
AND table = 'metric_stream'
ORDER BY create_time DESC;
```

Verify that active projection parts exist:

```sql
SELECT
name,
sum(rows) AS rows,
formatReadableSize(sum(bytes_on_disk)) AS bytes_on_disk
FROM system.projection_parts
WHERE active
AND database = 'ingest'
AND table = 'metric_stream'
AND name = 'by_provider_generation'
GROUP BY name;
```

## Verify A Deletion

Find the durable request and its generation in PostgreSQL:

```sql
SELECT
event_id,
user_id,
provider_id,
generation,
status,
created_at,
dispatched_at,
completed_at
FROM fitness.provider_data_deletion_outbox
WHERE user_id = '<user UUID>'::uuid
AND provider_id = '<provider ID>'
ORDER BY created_at DESC;
```

For a completed request, confirm the ClickHouse fence and acknowledgement:

```sql
SELECT max(generation) AS active_generation
FROM ingest.provider_data_generation FINAL
WHERE user_id = toUUID('<user UUID>')
AND provider_id = '<provider ID>';

SELECT event_id, max(applied_at) AS applied_at
FROM ingest.metric_stream_delete_acknowledgement FINAL
WHERE event_id = toUUID('<event UUID>')
GROUP BY event_id;
```

Confirm no active rows remain from older generations:

```sql
SELECT count() AS active_old_generation_rows
FROM
(
SELECT id
FROM ingest.metric_stream
WHERE user_id = toUUID('<user UUID>')
AND provider_id = '<provider ID>'
AND generation < 3 -- Replace 3 with the active generation.
GROUP BY id
HAVING argMax(is_deleted, tuple(version, ingested_at)) = 0
);
```

The count must be zero before treating the metric-stream deletion as complete. Analytics can lag behind that acknowledgement while its refresh job runs.

## Deployment Transition

Deletion requests submitted before this outbox migration are not retroactively inserted into `fitness.provider_data_deletion_outbox`. After deploying this workflow, submit those deletion requests again so they receive a generation and durable outbox event.
27 changes: 27 additions & 0 deletions drizzle/0050_provider_data_deletion_outbox.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
CREATE TABLE fitness.provider_data_generation (
user_id uuid NOT NULL REFERENCES fitness.user_profile (id) ON DELETE CASCADE,
provider_id text NOT NULL,
current_generation bigint NOT NULL DEFAULT 0,
updated_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT provider_data_generation_pkey PRIMARY KEY (user_id, provider_id),
CONSTRAINT provider_data_generation_nonnegative CHECK (current_generation >= 0)
);
--> statement-breakpoint

CREATE TABLE fitness.provider_data_deletion_outbox (
event_id uuid PRIMARY KEY,
user_id uuid NOT NULL REFERENCES fitness.user_profile (id) ON DELETE CASCADE,
provider_id text NOT NULL,
generation bigint NOT NULL,
status text NOT NULL DEFAULT 'pending',
created_at timestamptz NOT NULL DEFAULT now(),
dispatched_at timestamptz,
completed_at timestamptz,
CONSTRAINT provider_data_deletion_outbox_generation_positive CHECK (generation > 0),
CONSTRAINT provider_data_deletion_outbox_status_valid
CHECK (status IN ('pending', 'dispatched', 'completed'))
);
--> statement-breakpoint

CREATE INDEX CONCURRENTLY provider_data_deletion_outbox_dispatch_idx
ON fitness.provider_data_deletion_outbox (status, created_at);
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@
"./providers/zos-app/provider": "./src/providers/zos-app/provider.ts",
"./db/migrate": "./src/db/migrate.ts",
"./db/metric-stream-writer": "./src/db/metric-stream-writer.ts",
"./db/provider-data-deletion": "./src/db/provider-data-deletion.ts",
"./db/token-user-context": "./src/db/token-user-context.ts",
"./db/tokens": "./src/db/tokens.ts",
"./security/credential-encryption": "./src/security/credential-encryption.ts",
Expand Down
Loading
Loading