Truncate ClickHouse tables when CDC mirror reconciliation drops a mirror

[Asherlc]

Problem

Staging deploy fails at Configure ClickHouse CDC with:

table device_priority exists and is not empty

Root Cause

reconcileRawAnalyticsMirrors detects a table mapping mismatch → drops the mirror → SQL template tries CREATE MIRROR IF NOT EXISTS with do_initial_copy = true → PeerDB rejects because destination ClickHouse tables still have data from the previous mirror's initial copy.

Fix

After dropping a mirror during reconciliation, truncate all its ClickHouse destination tables so the subsequent CREATE MIRROR succeeds.

Changes

• src/db/clickhouse-cdc.ts: Add truncateClickHouseDestinationTables(), call it after DROP MIRROR in reconcileRawAnalyticsMirrors
• src/db/clickhouse-cdc.test.ts: Verify truncate commands fire for the dropped mirror's tables (and not for unaffected mirrors)

Summary by Sourcery

Ensure ClickHouse CDC mirror reconciliation can safely recreate mirrors when source table mappings change by clearing stale destination data.

Bug Fixes:

• Truncate ClickHouse destination tables after dropping an out-of-date raw analytics mirror so a subsequent mirror recreation with initial copy can succeed without data conflicts.

Tests:

• Extend ClickHouse CDC setup tests to assert that truncate commands are issued only for tables belonging to dropped mirrors and not for unaffected mirrors.

Summary by CodeRabbit

• Bug Fixes

  • Improved automatic cleanup of analytics destination tables when raw analytics mirrors are reconciled and recreated without expected source table mappings.

• Tests

  • Added test coverage for table truncation commands executed during mirror reconstruction and reconfiguration.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[sourcery-ai]

Reviewer's Guide

Adds truncation of ClickHouse destination tables when a raw analytics CDC mirror is dropped during reconciliation, wiring the ClickHouse client through setup and verifying the behavior in tests.

Sequence diagram for truncating ClickHouse tables when a CDC mirror is dropped

sequenceDiagram
  participant SetupClickHouseCdc as setupClickHouseCdc
  participant PeerDbClient as PeerDbClient
  participant ClickHouseClient as ClickHouseCommandClient
  participant TruncateTables as truncateClickHouseDestinationTables

  SetupClickHouseCdc->>PeerDbClient: reconcileRawAnalyticsMirrors(peerDbClient, clickHouseClient)
  activate PeerDbClient
  PeerDbClient-->>SetupClickHouseCdc: existingMirrorsWithConfigs
  loop eachRawAnalyticsMirror
    PeerDbClient->>PeerDbClient: query(DROP MIRROR mirrorName)
    PeerDbClient-->>SetupClickHouseCdc: mirrorDropped
    SetupClickHouseCdc->>TruncateTables: truncateClickHouseDestinationTables(clickHouseClient, tableNames)
    activate TruncateTables
    loop eachTableName
      TruncateTables->>ClickHouseClient: command(TRUNCATE TABLE IF EXISTS postgres_fitness.tableName)
    end
    deactivate TruncateTables
  end
  deactivate PeerDbClient

Loading

File-Level Changes

Change	Details	Files
Truncate ClickHouse destination tables whenever a raw analytics mirror is dropped during reconciliation to allow clean recreation.	Introduce truncateClickHouseDestinationTables helper that issues TRUNCATE TABLE IF EXISTS commands for each mapped destination table in the postgres_fitness schema using the ClickHouse client Extend reconcileRawAnalyticsMirrors to accept a ClickHouseCommandClient alongside the PeerDbClient After detecting a mismatch between configured tables and the mirror's mapping and issuing DROP MIRROR, call the truncation helper for that mirror's destination tables Update setupClickHouseCdc to pass the ClickHouse client into reconcileRawAnalyticsMirrors so truncation is available during setup	src/db/clickhouse-cdc.ts
Test that dropping a raw analytics mirror triggers truncation only for its affected ClickHouse tables.	Capture ClickHouse command invocations in the raw analytics mismatch test by recording options.query from clickHouseClient.command Filter recorded commands to TRUNCATE TABLE statements and assert that activity and user_profile tables are truncated while food_entry is not Keep existing expectations that the mirror is dropped and recreated via CREATE MIRROR IF NOT EXISTS	src/db/clickhouse-cdc.test.ts

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8defb031-84ce-4b6c-9df5-68ad9f0494c6

📥 Commits

Reviewing files that changed from the base of the PR and between 64021a9 and e64d70b.

📒 Files selected for processing (2)

• src/db/clickhouse-cdc.test.ts
• src/db/clickhouse-cdc.ts

---

📝 Walkthrough

Walkthrough

reconcileRawAnalyticsMirrors now accepts a ClickHouseCommandClient and truncates the corresponding destination tables when dropping raw analytics mirrors. A new helper truncateClickHouseDestinationTables issues TRUNCATE TABLE IF EXISTS statements, and the test verifies that truncation targets expected tables while skipping others.

Changes

Mirror Reconciliation Cleanup

Layer / File(s)	Summary
Table truncation helper and wiring src/db/clickhouse-cdc.ts	New truncateClickHouseDestinationTables helper executes truncate statements for specified table names. setupClickHouseCdc passes the ClickHouse client to reconcileRawAnalyticsMirrors to enable truncation during mirror reconciliation.
Test verification of truncation behavior src/db/clickhouse-cdc.test.ts	Test mock captures ClickHouse command queries and asserts that TRUNCATE TABLE IF EXISTS statements are issued for postgres_fitness.activity and postgres_fitness.user_profile while excluding postgres_fitness.food_entry.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

area/db, type/bug

🚥 Pre-merge checks | ✅ 1 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	Title exceeds the 70-character limit at 72 characters and uses imperative mood appropriately, but lacks an area prefix as required by guidelines.	Add an area prefix (e.g., [db] or [cdc]) and shorten to ≤70 chars, such as: [cdc] Truncate ClickHouse tables when mirror reconciliation drops

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Hey - I've left some high level feedback:

• The truncateClickHouseDestinationTables helper hardcodes the postgres_fitness schema, which could make this fragile if schemas differ across environments; consider deriving the schema from configuration or from the existing mirror/table mapping instead of embedding it in the query string.
• In truncateClickHouseDestinationTables, you currently truncate tables sequentially in a for loop; using Promise.all on a mapped array of clickHouseClient.command calls would reduce latency when truncating multiple tables.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The `truncateClickHouseDestinationTables` helper hardcodes the `postgres_fitness` schema, which could make this fragile if schemas differ across environments; consider deriving the schema from configuration or from the existing mirror/table mapping instead of embedding it in the query string.
- In `truncateClickHouseDestinationTables`, you currently truncate tables sequentially in a `for` loop; using `Promise.all` on a mapped array of `clickHouseClient.command` calls would reduce latency when truncating multiple tables.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[github-actions]

Storybook previews for 3ed0edca are ready:

• Web Storybook
• Mobile Storybook

This comment updates automatically on each PR push.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Asherlc]

Addressed sourcery-ai review feedback:

1. Promise.all for parallel truncation — Applied. Tables are independent, no reason to truncate sequentially.

   • Commit: e64d70b0

2. Hardcoded postgres_fitness schema — Not applying. postgres_fitness is hardcoded throughout this file (e.g., ensureAnalyticsPeerDbColumns). Changing it only here would be inconsistent, and there's no multi-schema scenario.