fix(ingestion): bump jira/confluence concurrency to 4 to break CDK partition deadlock - #1308
Merged
mitasovr merged 1 commit intoJun 12, 2026
Conversation
…rtition deadlock With concurrency_level.default_concurrency: 1 the concurrent declarative CDK self-deadlocks once a sync generates >= ~10k partitions (the CDK futures limit of 10,000): the single worker thread runs generate_partitions and throttles forever on prune_to_validate_has_reached_futures_limit() (airbyte_cdk .../concurrent/partition_enqueuer.py), while the partition-read futures it waits on have no other worker to execute them. The CDK's own comment assumes more workers than partition enqueuers — violated at concurrency 1. Symptom: CPU-busy spin with zero file/socket I/O and zero records, forever; the records counter frozen at exactly 10000 is the fingerprint. Hit in production by jira (full-window jira_issue_history fan-out, 3h+ spin diagnosed with py-spy inside the source container) and confluence (every sync froze at "Records read: 10000" on the wiki_page_versions fan-out). Reproduces on source-declarative-manifest 7.21.1 and :latest; global_substream_cursor and removing the child `step` do not help. Fix: default_concurrency 1 -> 4 for jira and confluence. Verified against the live instance: the same full-window read that deadlocked at concurrency 1 emitted 4208 records in the first 75 seconds at 4. The other nocode connectors keep 1 for now (no >=10k-partition fan-outs); the /connector skill now mandates >= 2 for new manifests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughDocumentation and configuration are updated across the connector package to enforce a minimum concurrency level of 2 and fix observed deadlock behavior in two production connectors by increasing their default concurrency from 1 to 4, with explanatory notes throughout. ChangesConcurrency Deadlock Resolution
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~4 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cypilot/.core/skills/connector/workflows/create.md`:
- Line 200: The concurrency example currently instructs setting
concurrency_level.default_concurrency: 1 which contradicts the new minimum
enforced in validate.md (>= 2) and can reintroduce the single-worker deadlock
seen in partition_enqueuer.py when generate_partitions hits the CDK futures
limit; update the example text to recommend default_concurrency >= 2 (and keep
default_concurrency: 4 only as the Jira/Confluence-sized example) so docs align
with validate.md and avoid the single-worker deadlock path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ece22aff-5a94-4956-ba2d-cd5c586fb6ce
📒 Files selected for processing (4)
cypilot/.core/skills/connector/workflows/create.mdcypilot/.core/skills/connector/workflows/validate.mdsrc/ingestion/connectors/task-tracking/jira/connector.yamlsrc/ingestion/connectors/wiki/confluence/connector.yaml
| - Route substreams through nullable parent fields (e.g. `parent_key: id_readable` / `record.get('idReadable')`). Use the parent's stable internal id (`record['id']`, surfaced as `youtrack_id` etc.). A `null` from the API silently routes to `.../None/<endpoint>` which 404s and drops the entire partition. | ||
| - Use a heavy stream (large per-record payloads, e.g. Jira `fields=*all` + `expand=names` ≈ 2 MB/response on ~1000-field instances) as a `SubstreamPartitionRouter` parent. The CDK **auto-caches every parent's HTTP responses in a SQLite requests-cache**; with multi-MB responses the cache balloons (observed: 226 MB after ~108 responses) and the read stalls silently — job stays "running", CPU busy, **0 records emitted, forever**. Instead, split roles: add a dedicated lightweight key-enumeration parent stream that requests a minimal field set (e.g. `fields: updated` — id/key arrive top-level for free) and point all children at it; keep the full-payload stream as a plain emitter (non-parent streams are not cached). This mirrors the official `source-jira` (`board_issues` parent uses `fields: ['key','created','updated']`; only the terminal `issues` emitter uses `*all`). Reference implementation: `jira_issue_keys` in `task-tracking/jira/connector.yaml`. | ||
|
|
||
| - Set `concurrency_level.default_concurrency: 1`. With a single worker the concurrent CDK **self-deadlocks** once one sync generates ≥ ~10k partitions (the CDK futures limit): the only worker thread runs `generate_partitions` and throttles on "futures limit reached" (`partition_enqueuer.py`), while the partition-read futures it waits on have no other worker to run them. Symptom: CPU-busy spin, zero I/O, zero records, forever — and the records counter freezing at exactly 10000 is the fingerprint. Hit in production by jira (`jira_issue_history` full-window fan-out) and confluence (`wiki_page_versions`). Use `default_concurrency: 4` (verified: the same full-window run that deadlocked at 1 emitted 4208 records in 75 s at 4). |
There was a problem hiding this comment.
Align the concurrency example with the new minimum.
This still tells authors to set default_concurrency: 1, which conflicts with validate.md's new >= 2 rule and reintroduces the single-worker deadlock path you’re documenting. Please change the generic guidance to >= 2 and keep 4 only as the Jira/Confluence-sized example.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@cypilot/.core/skills/connector/workflows/create.md` at line 200, The
concurrency example currently instructs setting
concurrency_level.default_concurrency: 1 which contradicts the new minimum
enforced in validate.md (>= 2) and can reintroduce the single-worker deadlock
seen in partition_enqueuer.py when generate_partitions hits the CDK futures
limit; update the example text to recommend default_concurrency >= 2 (and keep
default_concurrency: 4 only as the Jira/Confluence-sized example) so docs align
with validate.md and avoid the single-worker deadlock path.
mitasovr
added a commit
that referenced
this pull request
Jun 12, 2026
…#1308 (#1310) Reconcile republishes a nocode declarative manifest only on descriptor version drift (descriptor version vs the version stored on the Airbyte definition; equal -> noop per ADR-0015). #1308 changed connector.yaml (default_concurrency 1 -> 4) without bumping descriptor versions, so reconcile on the deployed 0.1.59 correctly no-op'd and the deadlock fix never reached Airbyte: the active jira manifest (v3) and confluence manifest (v2) still carry default_concurrency: 1. (#1283 only got published because an unrelated image-ref bump had already moved jira's descriptor to 1.2.0.) - jira: 1.2.0 -> 1.2.1 - confluence: 1.1.0 -> 1.1.1 Also encode the lesson in the /connector skill (deploy prerequisites + validate descriptor checklist): bump descriptor.yaml version in the same PR as any connector.yaml change, or the manifest silently never ships. Co-authored-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jun 12, 2026
mitasovr
added a commit
that referenced
this pull request
Jun 15, 2026
… staging (#1312) * feat(ingestion): add Outline wiki connector with Confluence-parity silver staging Second contributor to the wiki silver classes (class_wiki_pages, class_wiki_activity, class_wiki_engagement) alongside Confluence. Connector (wiki/outline, nocode, fully-inlined manifest — passes both validate and validate-strict): - wiki_spaces <- collections.list (full refresh) - wiki_pages <- documents.list, client-side incremental on updated_at, statusFilter [published, archived] - wiki_page_versions <- revisions.list (substream of wiki_pages) - wiki_comments <- comments.list workspace-wide (single threaded stream: reply = parentCommentId, inline = anchorText) - wiki_users <- users.list (email directory for identity resolution) The Outline API is RPC-style (POST {instance}/api/{method}, Bearer key, offset/limit pagination in the JSON body); 429 handled via Retry-After capped at 600s, same policy as Confluence. Live-tested against a self-hosted instance (47 collections / 11314 docs / 1758 comments / 484 users; all 5 streams green, incremental resume verified). Findings baked in: - statusFilter ["draft"] returns HTTP 500 on self-hosted Outline and drafts are owner-scoped anyway -> drafts excluded by design - embedded createdBy/updatedBy objects carry no email field on self-hosted instances -> staging models LEFT JOIN the connector's own wiki_users stream (coalesce embedded-first for cloud deployments) - 11k+ documents -> 11k+ substream partitions exceeds the 10k CDK futures limit -> default_concurrency: 4 (same deadlock class fixed for jira/confluence in #1308) dbt: outline__bronze_promoted (RMT promotion for 5 bronze tables) + 3 staging models tagged silver:class_wiki_*; column order kept positionally identical to the confluence__* models (union_by_tag emits positional UNION ALL). silver/wiki/schema.yml accepted_values extended with outline / insight_outline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech> * feat(ingestion): feed Identity Resolution from Outline user directory Add the standard identity-inputs macro chain for the outline connector — wiki_users exposes emails, so the connector must contribute observations to identity.identity_inputs like zoom/zulip-proxy/bamboohr/ms-entra do: - outline__users_snapshot: SCD2 snapshot of wiki_users (name/email/role/is_suspended) - outline__users_fields_history: field-level change log, entity_id = user_id (Outline user UUID) - outline__identity_inputs: email + display_name observations plus the canonical id binding row (emitted by the macro per ADR-0002); suspension (is_suspended=true) emits DELETE rows - silver/_shared/identity_inputs.sql: depends_on line for first-run build ordering Skill docs updated to close the gap that allowed this omission: - connector-create.md gains §3.6b — when the chain is REQUIRED, model templates for all three macros, and the rules (canonical id row is automatic, bool fields stringify to 'true'/'false', snapshot source must be RMT-promoted, shared-union depends_on) - connector-validate.md gains an "Identity Resolution inputs" checklist section Verified: dbt compile of the new chain plus the shared identity_inputs union is clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech> --------- Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After #1283 fixed the SQLite cache explosion, full-window jira syncs still stalled: 3+ hours of CPU-busy spin with zero file/socket I/O and zero records. Confluence has been failing the same way for weeks — every sync frozen at exactly
Records read: 10000.Root cause (diagnosed with py-spy inside the live source container)
concurrency_level.default_concurrency: 1+ a sync that generates ≥ ~10k partitions (the CDK futures limit of 10,000) = self-deadlock of the concurrent declarative CDK:generate_partitionsand throttles forever inwhile prune_to_validate_has_reached_futures_limit(): sleep(...)(airbyte_cdk/.../concurrent/partition_enqueuer.py),The frozen-at-10000 counter is the fingerprint (futures limit). Reproduces on
source-declarative-manifest:7.21.1and:latest;global_substream_cursor: trueand removing the childstepdo not help — all verified with isolatedreadruns. Narrow windows (≤ ~3.6k partitions) never hit the limit, which is why incremental nightly syncs of small connectors are unaffected.Fix
default_concurrency: 1 → 4for the two substream-heavy connectors:jira_issue_history/jira_comments/jira_worklogsfan-out (~10k issues)wiki_page_versionsfan-out overwiki_pages(froze at 10000 every run)Verified live: the same full-window read that deadlocked at concurrency 1 emitted 4208 records in the first 75 seconds at concurrency 4.
Also encodes the lesson into the
/connectorskill (create-rules + validate checklist:default_concurrencymust be ≥ 2). Other nocode connectors keep 1 for now — none have ≥10k-partition fan-outs.Follow-up (separate PR)
Confluence still carries a latent cache risk once syncs get past versions:
wiki_footer_comments/wiki_inline_commentsare substream parents (for replies) requestingbody-format: storage(full HTML bodies) — same cache-bomb class as jira'sfields=*allfrom #1283. Will be addressed with lightweight id-only parents after observing the first full confluence run.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Documentation