[codex] Repair legacy ClickHouse metric stream engine by Asherlc · Pull Request #1099 · Asherlc/dofek

Asherlc · 2026-05-07T19:41:41Z

Summary

add a ClickHouse migration that repairs legacy postgres_fitness.metric_stream tables created as plain MergeTree
recreate the metric stream mirror as ReplacingMergeTree(_peerdb_version), backfill it, and refresh dependent analytics read models before provider stats runs
add regression coverage for the legacy engine path and document the deploy incident

Root Cause

Production still had a legacy postgres_fitness.metric_stream table created with the plain MergeTree engine. Newer ClickHouse read models query that raw mirror with FINAL, which requires ReplacingMergeTree(_peerdb_version). The existing bootstrap SQL used CREATE TABLE IF NOT EXISTS, so it did not repair the table engine.

Validation

pnpm install --frozen-lockfile
docker compose up -d db redis attempted; Redis started, DB bind was already occupied by the existing aloud-bike-db-1 container on 127.0.0.1:5435
pnpm test run src/db/clickhouse-migrations.test.ts
pnpm lint
pnpm tsc --noEmit
cd packages/server && pnpm tsc --noEmit
cd packages/web && pnpm tsc --noEmit
pnpm test:changed

Summary by CodeRabbit

Bug Fixes
- Resolved production deployment failure related to ClickHouse migration compatibility.
- Implemented automatic repair of legacy database table engines to support modern query operations.
Documentation
- Added incident baseline documentation detailing the production incident and resolution.

github-actions · 2026-05-07T19:43:08Z

Storybook previews for a448d77f are ready:

This comment updates automatically on each PR push.

coderabbitai · 2026-05-07T19:44:20Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b176e9f7-5247-41f5-b16f-429365867168

📥 Commits

Reviewing files that changed from the base of the PR and between edcd2a2 and 1092e1f.

📒 Files selected for processing (3)

docs/production-incident-baseline.md
src/db/clickhouse-migrations.test.ts
src/db/clickhouse-migrations.ts

📝 Walkthrough

Walkthrough

This PR adds a targeted ClickHouse migration to repair a production incident where legacy MergeTree tables block subsequent FINAL-dependent views. The fix conditionally detects and replaces the legacy postgres_fitness.metric_stream table with ReplacingMergeTree(_peerdb_version), then re-runs backfill and view refreshes. Tests validate the complete repair sequence, and incident documentation records the failure and mitigation.

Changes

Legacy Metric Stream Engine Repair

Layer / File(s)	Summary
Migration Logic & Engine Repair `src/db/clickhouse-migrations.ts`	New migration `0007_repair_legacy_metric_stream_engine` registers a conditional repair flow. Function `replaceLegacyMetricStreamIfNeeded` checks the metric_stream table engine and, when plain `MergeTree` is detected, drops dependent analytics views, drops the legacy table, re-applies bootstrap statements, backfills from Postgres, and refreshes/waits on dependent materialized views (`derived_resting_heart_rate`, `provider_stats`, `deduped_sensor`, `activity_summary`).
Test Coverage & Verification `src/db/clickhouse-migrations.test.ts`	Updated ClickHouse query mock to recognize and return `engine` field in `system.tables` queries. Adjusted expected migration count from 7 to 8 and `system.tables` query count from 17 to 18. Added comprehensive test case simulating legacy `MergeTree` engine, verifying the repair migration executes, and asserting that backfill tracking is dropped, `postgres_fitness.metric_stream` is recreated with `ReplacingMergeTree(_peerdb_version)`, backfill insertion occurs, and `analytics.provider_stats` is refreshed.
Production Incident Documentation `docs/production-incident-baseline.md`	New incident entry for 2026-05-07 describing the deploy failure triggered by `MergeTree` not supporting `FINAL` operator on the legacy `postgres_fitness.metric_stream` table. Documents the repair migration as the fix and notes constraints on local verification due to missing dependencies.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

Asherlc/dofek#1098: Both PRs modify ClickHouse migrations and tests to address metric_stream engine and backfill concerns.
Asherlc/dofek#1081: Both PRs update clickhouse-migrations.ts and clickhouse-migrations.test.ts to change metric_stream backfill and migration behavior.
Asherlc/dofek#1080: Both PRs modify ClickHouse metric_stream migration logic to handle legacy engine replacement and backfill/CDC setup.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: repairing a legacy ClickHouse metric stream engine via a new migration, which is the core objective of the PR.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reminiscent-dragon

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

This PR adds a targeted ClickHouse migration to detect and repair legacy postgres_fitness.metric_stream mirrors that were created with the plain MergeTree engine, which breaks newer analytics read models that query the mirror using FINAL.

Changes:

Add a new ClickHouse migration (0007_repair_legacy_metric_stream_engine) that detects the legacy engine, rebuilds the mirror as ReplacingMergeTree(_peerdb_version), backfills it, and refreshes dependent analytics read models.
Extend ClickHouse migration test coverage to exercise the legacy-engine replacement path.
Document the associated production deploy incident and mitigation in the incident baseline.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
src/db/clickhouse-migrations.ts	Adds a new migration that conditionally drops/recreates the metric stream mirror and refreshes dependent analytics models.
src/db/clickhouse-migrations.test.ts	Updates migration-count expectations and adds a regression test for the legacy MergeTree repair path.
docs/production-incident-baseline.md	Adds a dated incident entry documenting symptoms, root cause, and mitigation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    {
+      id: "0007_repair_legacy_metric_stream_engine",
+      run: replaceLegacyMetricStreamIfNeeded,
+    },


+    "DROP VIEW IF EXISTS analytics.provider_stats",
+    "DROP TABLE IF EXISTS analytics.provider_stats",
+    "DROP VIEW IF EXISTS analytics.derived_resting_heart_rate",
+    "DROP TABLE IF EXISTS analytics.derived_resting_heart_rate",
+    "DROP VIEW IF EXISTS analytics.activity_summary",
+    "DROP TABLE IF EXISTS analytics.activity_summary",
+    "DROP VIEW IF EXISTS analytics.deduped_sensor",
+    "DROP TABLE IF EXISTS analytics.deduped_sensor",


github-actions · 2026-05-07T19:54:05Z

Review app is ready:

This environment runs on a dedicated Hetzner server for PR #1099 and updates on each push.

[codex] Repair legacy ClickHouse metric stream engine

1092e1f

Copilot AI review requested due to automatic review settings May 7, 2026 19:41

Copilot started reviewing on behalf of Asherlc May 7, 2026 19:42 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

Asherlc merged commit 6443b44 into main May 7, 2026
67 checks passed

Asherlc deleted the reminiscent-dragon branch May 7, 2026 20:54

This was referenced May 21, 2026

Fix stale ClickHouse body measurements #1161

Merged

Remove ClickHouse full refresh read models #1174

Merged

Organize ClickHouse migrations #1175

Merged

coderabbitai Bot mentioned this pull request May 31, 2026

Fix ClickHouse metric stream bootstrap repair #1206

Merged

Navigation Menu

[codex] Repair legacy ClickHouse metric stream engine - #1099

[codex] Repair legacy ClickHouse metric stream engine#1099
Asherlc merged 1 commit into
mainfrom
reminiscent-dragon

Asherlc commented May 7, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

coderabbitai Bot commented May 7, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

❌ Failed checks (1 warning)

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Asherlc commented May 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root Cause

Validation

Summary by CodeRabbit

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

coderabbitai Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

❌ Failed checks (1 warning)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Asherlc commented May 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 7, 2026 •

edited

Loading