Fix deploy CDC recovery and ClickHouse staleness - #1173
Merged
Merged
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
Contributor
Reviewer's GuideAdjusts ClickHouse CDC setup to avoid destructive resnapshots when raw analytics mirrors are absent but destination data exists, hardens the deploy workflow around migrations and post-stack data-service readiness, increases Netdata memory limits, and documents/operationalizes recent production incidents including a ClickHouse body-measurement staleness runbook. Sequence diagram for raw analytics CDC mirror reconciliation based on ClickHouse row countssequenceDiagram
participant Setup as setupClickHouseCdc
participant PeerDB as PeerDbClient
participant ClickHouse as ClickHouseCommandClient
Setup->>PeerDB: ensureAnalyticsPublication()
Setup->>PeerDB: ensureMetricStreamNoImuPublication()
Setup->>PeerDB: reconcileMetricStreamAnalyticsMirror()
Setup->>PeerDB: reconcileRawAnalyticsMirrors()
activate PeerDB
PeerDB-->>Setup: flows rows (raw_analytics_mirror_config)
deactivate PeerDB
loop each rawAnalyticsMirrorNames
alt mirror absent in flows
Setup->>ClickHouse: clickHouseDestinationTablesHaveRows(tableNames)
activate ClickHouse
ClickHouse->>ClickHouse: query({ query: SELECT coalesce(sum(rows)) AS row_count FROM system.parts ... })
ClickHouse-->>Setup: row_count
deactivate ClickHouse
alt row_count > 0
Setup->>Setup: rawAnalyticsInitialCopyValues[mirror] = false
else row_count == 0
Setup->>Setup: rawAnalyticsInitialCopyValues[mirror] = true
end
else mirror present in flows
Setup->>Setup: compare mirror config to rawAnalyticsMirrorTableMappings
alt config differs
Setup->>PeerDB: query(DROP MIRROR ...)
Setup->>ClickHouse: truncateClickHouseDestinationTables()
end
end
end
Setup->>Setup: renderPeerDbSqlTemplate(templateSql, templateValues, rawAnalyticsInitialCopyValues)
Setup-->>Setup: renderedSql
loop splitPeerDbSqlStatements(renderedSql)
Setup->>PeerDB: query(statement)
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Warning Review limit reached
Your plan currently allows 1 review/hour. Refill in 41 minutes and 34 seconds. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more review capacity refills, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughDeploy workflow now runs migrations in foreground with explicit cleanup and 4-hour timeout, adds post-deploy readiness checks for Postgres/ClickHouse, and increases Netdata memory. ClickHouse CDC conditionally skips initial bulk copies when destination tables contain existing rows, determined by querying ChangesProduction incident response and CDC improvements
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~30 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
Storybook previews for This comment updates automatically on each PR push. |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/deploy-web-stack.yml:
- Around line 477-516: The post-deploy readiness loops (the "Wait for Postgres
writable after stack deploy" and "Wait for ClickHouse after stack deploy" blocks
that iterate with for attempt in $(seq 1 36), use timeout 15s and sleep 5s and
set recovery_output / clickhouse_output) have a real worst-case runtime ≈720s
but their error messages claim 180s; either make the loop actually enforce a
180s deadline (replace the fixed seq loop with a deadline-based loop using a
start timestamp/SECONDS and break when elapsed > 180) or update the failure
messages ("::error::Postgres did not become writable within 180s after stack
deploy" and "::error::ClickHouse did not become reachable within 180s after
stack deploy") to the correct budget (e.g., 720s) so the reported timeout
matches the loop behavior. Ensure changes touch the attempt loop logic and the
exact error strings so behavior and logs are consistent.
In `@src/db/clickhouse-cdc.ts`:
- Around line 483-506: In clickHouseDestinationTablesHaveRows, the ClickHouse
JSON is parsed ad-hoc; define a Zod schema matching ClickHouseRowCount (e.g., {
row_count: z.union([z.string(), z.number(), z.null()]) }) and use it to parse
the result of await result.json() (validate the array shape) before calling
readInteger; update the code paths that currently destructure const [row] =
await result.json() to first z.parse the value and then pass parsedRow.row_count
into readInteger so the boundary is validated (keep references to
ClickHouseCommandClient/query, ClickHouseRowCount, readInteger,
peerDbStringLiteral).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: c8736024-a2c7-4a4b-a4d9-e5c673d6fec9
📒 Files selected for processing (8)
.github/workflows/deploy-web-stack.ymldeploy/README.mddeploy/stack.ymldocs/clickhouse-body-measurement-staleness-runbook.mddocs/production-incident-baseline.mdsrc/db/clickhouse-cdc.test.tssrc/db/clickhouse-cdc.tssrc/db/peerdb/metric-stream-cdc.sql
Contributor
|
Review app is ready: This environment runs on a dedicated Hetzner server for PR #1173 and updates on each push. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
Summary by Sourcery
Improve ClickHouse CDC raw analytics recovery behavior and harden the web deployment workflow against data-service and migration failures, while documenting recent production incidents and recovery steps.
Bug Fixes:
Enhancements:
Documentation:
Tests:
Summary by CodeRabbit
Release Notes
Chores
Documentation
Bug Fixes