fix(ingestion): zoom participants substream deadlock — bump default_concurrency to 4 - #1488
Merged
mitasovr merged 3 commits intoJun 25, 2026
Conversation
…oncurrency to 4 The zoom `participants` stream is a SubstreamPartitionRouter over `meetings` (~30k partitions on a multi-month backfill). At `default_concurrency: 1` the concurrent CDK self-deadlocks once the substream generates >=10k partitions: the single worker throttles in `generate_partitions` waiting on partition-read futures that no other worker can execute. Symptom observed on dev-vhc: `users` (446) and `meetings` (29935) flush to bronze_zoom within minutes, then `participants` sits RUNNING emitting zero records for ~50 min while the source spins ~12% CPU forever. Same root cause and fix as the prior jira/confluence substream hang: bump `default_concurrency` 1 -> 4. Also bump `descriptor.version` 2026.05.04 -> 1.0.0 (strict semver per ADR-0015) so the reconciler detects the drift and republishes the manifest to the existing Airbyte source definition. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughUpdated the Zoom connector descriptor version to ChangesZoom connector settings
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
aleksdotbar
approved these changes
Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The zoom
participantsstream is aSubstreamPartitionRouterovermeetings(~30k partitions on a multi-month backfill). Atconcurrency_level.default_concurrency: 1the concurrent CDK self-deadlocks once the substream generates ≥10k partitions: the single worker throttles ingenerate_partitionswaiting on partition-read futures that no other worker can execute.Observed on dev-vhc during first connector sync:
users(446) andmeetings(29,935) flush tobronze_zoom.*within minutes ✅participantssitsRUNNINGemitting zero records / zero checkpoints for ~50 min while the source container spins ~12% CPU forever ❌This is the same root cause and fix as the earlier jira/confluence substream hang (concurrent-CDK self-deadlock at
default_concurrency: 1, confirmed there via py-spy).Fix
connector.yaml:default_concurrency1 → 4.descriptor.yaml:version2026.05.04 → 1.0.0(strict semver per ADR-0015) so the reconciler detects the drift and republishes the manifest to the existing Airbyte source definition.classify_bump("1.0.0", "2026.05.04")→migration(re-discover catalog, no full-refresh).Verification plan (post-deploy)
After the toolbox image ships and dev redeploys, the reconcile loop updates the zoom source manifest; a re-sync should show
participantscheckpoints/records flowing andbronze_zoom.participantsrow count climbing (mirrors the jira/confluence recovery: 0 → thousands/min once concurrency hit 4).🤖 Generated with Claude Code
Summary by CodeRabbit
1.0.0for clearer release tracking.