Skip to content

fix: align ClickHouse SQL URL type - #1241

Merged
Asherlc merged 11 commits into
mainfrom
Asherlc/slow-query-clickhouse-url-type-fix
Jun 4, 2026
Merged

Asherlc merged 11 commits into
mainfrom
Asherlc/slow-query-clickhouse-url-type-fix

Conversation

@Asherlc

@Asherlc Asherlc commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • Fix the Node URL import used by ClickHouse SQL path handling so package-level TypeScript checks pass in mobile and stacked PR worktrees.

Validation

  • pnpm lint
  • pnpm tsc --noEmit
  • cd packages/server && pnpm tsc --noEmit
  • cd packages/web && pnpm tsc --noEmit
  • cd packages/mobile && pnpm tsc --noEmit

Summary by cubic

Fixes the Node URL import and reduces slow queries by moving anomaly computation off the dashboard, serving provider/admin stats from a compact ClickHouse read model and Postgres catalog estimates, narrowing provider list queries, and caching body-weight fetches.

  • Refactors

    • Dashboard: anomalies are no longer computed in mobileDashboard.dashboard; mobile loads them via trpc.anomalyDetection.check, and pull-to-refresh refetches both.
    • Provider/admin stats: SyncRepository reads analytics.provider_stats (ReplacingMergeTree with is_deleted, refresh_version, refreshed_at); ClickHouse migrations and isolated test helpers updated to preserve the provider-stats SELECT.
    • Admin counts: use pg_class.reltuples estimates with pg_inherits-based chunk rollups for metric_stream.
    • Provider records: list queries select explicit columns instead of SELECT *.
    • Body analytics: per-instance cache dedupes identical weight-row fetches; failed fetches evict the cache and are reported to Sentry.
  • Bug Fixes

    • Import URL from node:url in ClickHouse SQL path handling to resolve TypeScript errors.
    • Allow pg_inherits catalog column names in cspell to pass linting.
    • Removed orphan review app agent links (deploy/review-apps/CLAUDE.md, deploy/review-apps/GEMINI.md).
    • Preserve provider-stats test SELECT in ClickHouse test helpers to avoid unintended rewriting.

Written for commit 4d73d17. Summary will update on new commits.

Review in cubic

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @Asherlc, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR implements the second phase of a staged slow-query optimization sequence by introducing a compact ClickHouse incremental read model for provider statistics, migrating server queries to consume it, replacing expensive COUNT(*) operations with Postgres catalog estimates, narrowing provider detail column selection, deduplicating body analytics row fetches in-memory, and decoupling mobile dashboard anomalies from the critical request path.

Changes

Slow Query Optimization - Provider Stats, Admin Counts, and Deduplication

Layer / File(s) Summary
Provider Stats dbt Read Model and Source Declaration
analytics/models/read_models/provider_stats.sql, analytics/models/sources.yml
New provider_stats incremental model using ReplacingMergeTree(refresh_version) aggregates per-(user_id, provider_id) activity counts from multiple postgres_fitness sources, deduplicates body measurements, computes refresh metadata (is_deleted, refresh_version, refreshed_at), and tracks provider deletion state.
ClickHouse Provider Stats Schema and Test Integration
src/db/clickhouse-read-models.ts, packages/server/src/routers/clickhouse-integration-test-helpers.ts, src/db/clickhouse-migrations.test.ts, src/db/clickhouse.test.ts
Extends ClickHouse read-model SQL generation to emit refresh_clock CTE and adds metadata columns. Updates test-helper schema definitions to treat provider_stats as a ReplacingMergeTree view with ORDER BY (user_id, provider_id) and view-specific SELECT rewriting for refresh fields.
Sync Repository - Provider Stats Read from ClickHouse Read Model
packages/server/src/repositories/sync-repository.ts, packages/server/src/repositories/sync-repository.test.ts
Simplifies SyncRepository.getProviderStats() to read directly from compact analytics.provider_stats FINAL table filtered by user_id and is_deleted = 0, replacing prior CTE-based aggregation across multiple postgres_fitness tables.
Admin Overview - Postgres Catalog-Based Row Count Estimates
packages/server/src/routers/admin.ts, packages/server/src/routers/admin.test.ts
Replaces exact COUNT(*) queries with estimated row counts from pg_class.reltuples, including special handling for metric_stream hypertable chunks via GREATEST(base, inherited) logic.
Provider Detail Repository - Explicit Column Selection
packages/server/src/repositories/provider-detail-repository.ts, packages/server/src/routers/provider-detail.test.ts
Adds listColumns(dataType) helper mapping each data type to an explicit column list and updates getRecords() to select only those columns instead of SELECT *. Validates exact column selection per data type.
Body Analytics Repository - Per-Instance Row Fetch Caching
packages/server/src/repositories/body-analytics-repository.ts, packages/server/src/repositories/body-analytics-repository.test.ts
Implements in-memory memoization of ClickHouse body-weight row fetches keyed by (days, endDate, requireBodyFat). Wraps fetchBodyWeightRows to reuse promises across analytics methods and evicts on rejection.
Mobile Dashboard - Standalone Anomaly Query
packages/mobile/app/(tabs)/index.tsx, packages/mobile/app/(tabs)/index.test.tsx
Adds separate trpc.anomalyDetection.check query with 10-minute staleTime, decoupling anomaly detection from critical dashboard query path. Pull-to-refresh refetches both queries concurrently.
Mobile Dashboard Router - Remove Anomaly from Critical Path
packages/server/src/routers/mobile-dashboard.ts, packages/server/src/routers/mobile-dashboard.test.ts
Removes anomaly computation from mobileDashboard.dashboard router by setting anomalies to null instead of calling AnomalyDetectionRepository.check(endDate). Verifies critical path remains unblocked.
Cleanup, Documentation, and Slow-Query Optimization Planning
cspell.json, docs/production-incident-baseline.md, docs/superpowers/plans/2026-06-04-slow-query-optimization-sequence.md, docs/superpowers/specs/2026-06-04-slow-query-optimization-split-design.md
Adds reltuples to cspell dictionary, records activity.stream verification findings, and documents the complete five-PR slow-query optimization sequence with task checklists, test expectations, and rollout order.
ClickHouse SQL Template Import Update
src/db/clickhouse-sql.ts
Expands node:url import to include URL for template file path construction.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • Asherlc/dofek#1098: Moves sync.providerStats to read from analytics.provider_stats via ClickHouse, directly dependent on the new provider_stats dbt model introduced here.
  • Asherlc/dofek#1174: Modifies provider_stats refresh behavior by switching to non-full-refresh "standard view" creation, affecting the incremental/refresh metadata handling added here.
  • Asherlc/dofek#1175: Changes provider_stats ClickHouse view creation strategy via standardViewHeader, impacting the view header and metadata column generation in src/db/clickhouse-read-models.ts.

Suggested labels

area/server, area/mobile, area/db, area/infra, type/refactor, type/performance

Suggested reviewers

  • cubic-dev-ai
🚥 Pre-merge checks | ✅ 1 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title uses 'fix:' prefix (semantic versioning) instead of the required imperative mood with area prefix format. Rewrite as '[server] align ClickHouse SQL URL type' or similar imperative form with area prefix, keeping under 70 characters.
✅ Passed checks (1 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Storybook previews for 9c4f206d are ready:

This comment updates automatically on each PR push.

@Asherlc
Asherlc marked this pull request as ready for review June 4, 2026 17:52
Copilot AI review requested due to automatic review settings June 4, 2026 17:52
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @Asherlc, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Auto-approved: This change adds the missing URL import from node:url to fix TypeScript type errors, and removes two orphaned symlink files—both are low-risk, trivial adjustments with no impact on runtime behavior or business logic.

Re-trigger cubic

@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Mobile Preview

Scan to open on device:

QR code for dofek://preview/pr-1241

Channel pr-1241
Deep Link dofek://preview/pr-1241
Commit 9c4f206

To test on device:

  1. Build and install the preview client: PREVIEW_CHANNEL=pr-1241 npx expo prebuild --clean -p ios
  2. Or tap deep link on an existing preview build: dofek://preview/pr-1241

Each PR gets its own channel. Build a preview client with PREVIEW_CHANNEL=pr-{N} to test.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 17 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/server/src/repositories/sync-repository.ts
Comment thread packages/server/src/routers/admin.ts Outdated
Asherlc added 2 commits June 4, 2026 12:35
* perf: cache body analytics rows

* fix: evict failed body cache fetches

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/superpowers/plans/2026-06-04-slow-query-optimization-sequence.md`:
- Line 3: The documentation line that mandates an agent-specific skill ("**For
agentic workers:** REQUIRED SUB-SKILL: Use
superpowers:subagent-driven-development (recommended) or
superpowers:executing-plans") makes the docs dependent on agent capabilities and
violates the rule that docs in docs/ must be human-actionable; update the
sentence in the plan to remove the "REQUIRED SUB-SKILL" phrasing and either
delete the agent-specific requirement entirely or convert it to an optional note
such as "Optional: developers familiar with
superpowers:subagent-driven-development or superpowers:executing-plans may use
those tools" so the document remains directly implementable by human developers
(locate and edit the offending line containing
"superpowers:subagent-driven-development" / "superpowers:executing-plans").

In `@packages/server/src/repositories/body-analytics-repository.ts`:
- Around line 324-327: The catch block that currently only evicts the cache and
rethrows the error should also call captureException(error) so failures are
reported to Sentry; update the catch in body-analytics-repository (the block
that contains this.#bodyWeightRowsCache.delete(cacheKey)) to call
captureException(error) before rethrowing, and add the appropriate import for
captureException at the top of the file if missing. Ensure the call uses the
caught error value (typed unknown) and does not suppress the existing cache
eviction or the subsequent throw.

In `@packages/server/src/routers/mobile-dashboard.test.ts`:
- Around line 119-129: Remove the negative test and its supporting mock setup:
delete the hoisted anomalyRepositoryMock declaration and the vi.mock replacement
for AnomalyDetectionRepository (the anomalyRepositoryMock and vi.mock(...)
block) and remove the test case that asserts deleted anomaly behavior (the test
around the `result.anomalies` null assertion and the expectation that
`anomalyRepositoryMock.check` is not called). If needed, replace the removed
test with a positive behavior test that exercises the current public API rather
than asserting the absence of deleted internals (refer to anomalyRepositoryMock,
AnomalyDetectionRepository.check, and the test that checks result.anomalies).

In `@packages/server/src/routers/provider-detail.test.ts`:
- Around line 621-652: The tests "does not select raw payload columns for
activity list records" and "does not select raw payload columns for metric
stream list records" should be changed from absence checks to positive contract
assertions: remove the expect(...not.toMatch(/SELECT \*/)) and
expect(...not.toContain("raw")) lines and instead assert the specific projected
columns you require when calling caller.records (use the same pattern with
mockExecute and extractSqlText(mockExecute.mock.calls[0][0]) to inspect SQL).
For the activity test (the one that calls caller.records({ providerId: "strava",
dataType: "activities" })) assert the exact column names your projection
contract requires (e.g., id, provider_record_id, recorded_at, etc.) and for the
metricStream test (caller.records({ providerId: "apple_health", dataType:
"metricStream" })) keep the positive assertions that check "recorded_at",
"channel", and "scalar" and add any other required projected columns; remove any
assertions that only test the absence of "raw" or "SELECT *".
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 53af8dae-ef79-4381-98dd-bd76eb58ca7e

📥 Commits

Reviewing files that changed from the base of the PR and between 339358a and 727e603.

📒 Files selected for processing (26)
  • analytics/models/read_models/provider_stats.sql
  • analytics/models/sources.yml
  • cspell.json
  • deploy/review-apps/CLAUDE.md
  • deploy/review-apps/GEMINI.md
  • docs/production-incident-baseline.md
  • docs/superpowers/plans/2026-06-04-slow-query-optimization-sequence.md
  • docs/superpowers/specs/2026-06-04-slow-query-optimization-split-design.md
  • packages/mobile/app/(tabs)/index.test.tsx
  • packages/mobile/app/(tabs)/index.tsx
  • packages/server/src/repositories/body-analytics-repository.test.ts
  • packages/server/src/repositories/body-analytics-repository.ts
  • packages/server/src/repositories/provider-detail-repository.ts
  • packages/server/src/repositories/sync-repository.test.ts
  • packages/server/src/repositories/sync-repository.ts
  • packages/server/src/routers/admin.test.ts
  • packages/server/src/routers/admin.ts
  • packages/server/src/routers/clickhouse-integration-test-helpers.test.ts
  • packages/server/src/routers/clickhouse-integration-test-helpers.ts
  • packages/server/src/routers/mobile-dashboard.test.ts
  • packages/server/src/routers/mobile-dashboard.ts
  • packages/server/src/routers/provider-detail.test.ts
  • src/db/clickhouse-migrations.test.ts
  • src/db/clickhouse-read-models.ts
  • src/db/clickhouse-sql.ts
  • src/db/clickhouse.test.ts
💤 Files with no reviewable changes (2)
  • deploy/review-apps/CLAUDE.md
  • deploy/review-apps/GEMINI.md

Comment thread docs/superpowers/plans/2026-06-04-slow-query-optimization-sequence.md Outdated
Comment thread packages/server/src/repositories/body-analytics-repository.ts
Comment thread packages/server/src/routers/mobile-dashboard.test.ts Outdated
Comment thread packages/server/src/routers/provider-detail.test.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 5 files (changes from recent commits).

Requires human review: This PR contains extensive refactoring across analytics models, server repositories, routers, and tests, including new caching logic, query rewrites, and data model changes that affect core performance paths, which carries notable risk of data inconsistency or behavioral regressions requiring...

Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Requires human review: The PR touches over 2000 lines across multiple critical subsystems including dashboard anomaly path, provider stats database model, admin table count queries, body analytics caching, and mobile UI, making the blast radius and risk of breakage too high for auto-approval.

Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Requires human review: This PR introduces substantial refactoring across multiple critical backend paths—including a new ClickHouse materialized view, changes to provider stats and admin count queries, removal of anomaly computation from the dashboard, and per-instance caching—which carry high risk of breaking data...

Re-trigger cubic

@Asherlc
Asherlc merged commit 1dba236 into main Jun 4, 2026
69 checks passed
@Asherlc
Asherlc deleted the Asherlc/slow-query-clickhouse-url-type-fix branch June 4, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants