fix(chart): apply ClickHouse gold-view migrations on deploy (external CH)

[mitasovr]

Problem

The ClickHouse gold-view migrations (src/ingestion/scripts/migrations/*.sql — the insight.* VIEWs analytics-api reads: collab_bullet_rows, git_bullet_rows, people, wiki_bullet_rows, …) are applied only by scripts/init.sh via kubectl exec into a bundled statefulset/insight-clickhouse pod, invoked manually from run-init.sh. They are not applied on deploy, and the reconcile-loop (reconcile-connectors/main.sh) doesn't touch them.

When ClickHouse is external/layered — dev moved CH to a separate clickhouse Helm release in insight-infra — there is no in-namespace pod to kubectl exec into, and the toolbox image ships no clickhouse-client binary. So init.sh can't reach the live CH and no CH gold-view migration has landed since the move: every newly-released view change (e.g. the Zulip collab_bullet_rows branch from #1466) silently never applied, and the metric only worked after applying the SQL by hand.

This is broad — it affects all gold-view changes, not just Zulip.

Fix

A Helm post-install,post-upgrade Hook Job (clickhouse-migrate) that applies the migrations to clickhouse.host over the HTTP interface via a new host-driven scripts/apply_ch_migrations.py, using the toolbox image's bundled clickhouse-connect (a dbt-clickhouse dependency).

• No new binary, no Dockerfile change, no init.sh change — fully additive.
• Same transport + statement-split (-- strip, split on ;) the e2e rig already uses to apply these exact files (tests/e2e/e2e_lib/migration_applier.py).
• Mirrors the existing clickhouse-init-svcdbs Hook (which already dials clickhouse.host over HTTP for CREATE DATABASE).
• Gated by clickhouse.applyMigrations (default true).

Idempotency / no dupes

No version-tracking table — by design; migrations are re-applied in full each run and are written idempotently:

• VIEWs: DROP VIEW IF EXISTS + CREATE VIEW (e.g. gold-views.sql has 29 DROP VIEW IF EXISTS).
• placeholders/DBs: CREATE … IF NOT EXISTS; ALTER … DROP COLUMN IF EXISTS.
• the one data table, insight.jira_closed_tasks, is DROP TABLE IF EXISTS → CREATE → INSERT (rebuilt empty each run → no row duplication).

This is the same set the e2e applies and re-applies (204 statements × 27 files) every session.

Validation

• Parse logic unit-checked locally (51 placeholder stmts incl. zulip; collab migration → 2 stmts).
• End-to-end against the live dev external CH in the deployed toolbox image: clickhouse-connect connects over HTTP and applies the collab migration → collab_bullet_rows gains the zulip branch (has_zulip=1), idempotent on re-run.
• ⚠️ The Helm Hook wiring itself should be confirmed on the next dev deploy (I can't run a full helm install from here; the template mirrors the working clickhouse-init-svcdbs Job).

Note

Dev gitops already points clickhouse.host at clickhouse.insight-infra.svc.cluster.local, so no gitops change is needed — this PR makes the umbrella actually apply migrations there.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@mitasovr, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 4 minutes and 45 seconds. Learn how PR review limits work.

To continue reviewing without waiting, enable usage-based billing in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 587164c3-bc2b-422c-9b88-e82f78837c75

📥 Commits

Reviewing files that changed from the base of the PR and between bf8a620 and 5c8bd8c.

📒 Files selected for processing (3)

• charts/insight/templates/clickhouse-migrate-job.yaml
• charts/insight/values.yaml
• src/ingestion/scripts/apply_ch_migrations.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}