Quiesce ClickHouse consumers during deploy CDC setup

[Asherlc]

Summary

Production deploy run 27889116821 failed at Configure ClickHouse CDC because ClickHouse was already at its memory cap while analytics-worker and the metric-stream sink were running. This adds a two-phase Swarm deploy: first apply deploy/stack.cdc-quiesce.yml to keep those services at zero replicas, run CDC setup, then redeploy the full stack to restore them. The full-stack redeploy still runs if CDC fails so analytics-worker is not left paused.

Test plan

• Merge and trigger a production Deploy Web run
• Confirm Configure ClickHouse CDC passes after the quiesced deploy
• Confirm analytics-worker and metric-stream-clickhouse-sink reach 1/1 after the final deploy step

Made with Cursor

---

Summary by cubic

Quiesces ClickHouse consumers during deploy so CDC setup runs safely and avoids memory-cap failures. First deploy uses deploy/stack.cdc-quiesce.yml to pause consumers, runs CDC, then redeploys the full stack and waits for consumers to converge.

• Bug Fixes
  • Add deploy/stack.cdc-quiesce.yml to set analytics-worker and metric-stream-clickhouse-sink replicas to 0; validate both the base stack and the overlay.
  • Deploy quiesced stack, run Configure ClickHouse CDC, then always redeploy the full stack if the quiesced step ran (even if it or CDC failed); wait for analytics-worker and metric-stream-clickhouse-sink to converge.
  • Fail the final convergence loop on paused Swarm updates; prune images and report disk usage only after the final deploy.

^{Written for commit 5e96618. Summary will update on new commits.}

Summary by CodeRabbit

• Chores
  • Enhanced deployment process with improved validation checks and optimized multi-phase service initialization for increased stability and reliability.
  • Added comprehensive pre-deployment readiness verification for all critical infrastructure services to ensure proper initialization before full deployment.
  • Refined service startup sequence to improve overall system initialization and convergence during deployment operations.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[sourcery-ai]

Sorry @Asherlc, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR introduces a two-phase Docker Swarm deployment around ClickHouse CDC setup. A new compose overlay (deploy/stack.cdc-quiesce.yml) sets analytics-worker and metric-stream-clickhouse-sink to zero replicas. The workflow first deploys with this overlay, runs CDC configuration, then performs a second full deploy without the overlay to restore those services.

Changes

Two-phase CDC-aware Docker Swarm deploy

Layer / File(s)	Summary
CDC quiesce overlay and stack validation deploy/stack.cdc-quiesce.yml, .github/workflows/deploy-web-stack.yml	New compose overlay sets analytics-worker and metric-stream-clickhouse-sink replicas to 0; docker stack config validation extended to include the quiesce override.
Quiesced deploy, CDC wiring, and full consumer restore .github/workflows/deploy-web-stack.yml	Single deploy step replaced by deploy_stack_quiesced (uses quiesce overlay); convergence wait adds metric-stream-clickhouse-sink; CDC step gains id: configure_clickhouse_cdc; new deploy_stack_full step, gated on quiesced deploy, runs the full deploy, waits for both consumer services, and owns image prune and disk-report cleanup.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• Asherlc/dofek#1087: Modifies the same CDC configuration sequencing in deploy-web-stack.yml, including Temporal search-attribute bootstrap steps adjacent to the CDC deploy flow touched here.
• Asherlc/dofek#1114: Also adds a compose overlay that scales specific services to replicas: 0 and extends docker stack config validation to include that overlay — the same pattern used in this PR.
• Asherlc/dofek#1173: Adds Postgres writability and ClickHouse reachability pre-checks before the CDC configuration step, which is the readiness gate that precedes the quiesced deploy introduced here.

Suggested labels

area/infra, type/bug

Suggested reviewers

• cubic-dev-ai

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title is in imperative mood, 52 characters (under 70), prefixed with area context, and has no trailing punctuation, meeting all requirements.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Storybook previews for 25b63742 are ready:

• Web Storybook
• Mobile Storybook

This comment updates automatically on each PR push.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/deploy-web-stack.yml:
- Around line 742-750: The paused service update case in the shell script is
using warn-and-continue behavior that masks failed rollouts. Instead of logging
a warning and continuing when the service update is paused but has running
tasks, treat the paused state as a failure condition and exit with error. Remove
the warning message and continue logic, and ensure that any paused service
update state causes the deployment to fail with an error message, consistent
with the guideline to not use warn-and-continue behavior that masks failures.
- Line 701: The restore deploy step is currently gated to only run when the
deploy_stack_quiesced step succeeds. This causes the restore phase to be skipped
if the quiesce step partially applies changes (like scaling down consumers) and
then fails during convergence checks, leaving services at zero replicas. Remove
or modify the condition on line 701 that checks for
deploy_stack_quiesced.outcome == 'success' to allow the restore deploy to always
run regardless of the quiesce step's outcome, ensuring the stack is always
returned to canonical service levels even when the quiesce step fails after
partial application.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d2e203f6-c03a-49a1-a1a3-008a2a394bfc

📥 Commits

Reviewing files that changed from the base of the PR and between 48a8489 and 609b7bf.

📒 Files selected for processing (2)

• .github/workflows/deploy-web-stack.yml
• deploy/stack.cdc-quiesce.yml