fix(jira): global_substream_cursor on issue streams — stop full re-sync every run - #1370
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📥 CommitsReviewing files that changed from the base of the PR and between 423aff56f233c53d9f5e6dd3540effff4a14a74c and f1e7e4e. 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughTwo streams in the Jira connector YAML — ChangesJira Connector: Global Substream Cursor Fix
Estimated code review effort🎯 2 (Simple) | ⏱️ ~5 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/ingestion/connectors/task-tracking/jira/connector.yaml (1)
906-908: ⚡ Quick winMake the sibling cursor-mode assumption explicit.
These comments rely on history/comments/worklogs crossing the CDK’s 10k partition auto-switch, but that is instance-size dependent. For Jira sources with fewer issue partitions, those streams may stay per-partition and hit the same non-persisted-state behavior; please verify the
<10kcase or setglobal_substream_cursor: trueexplicitly on those three issue substreams too.Also applies to: 8106-8108
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/ingestion/connectors/task-tracking/jira/connector.yaml` around lines 906 - 908, The comment assumes that history, comments, and worklogs substreams use global cursors because they exceed the 10k partition cap, but this behavior is instance-dependent. For Jira sources with fewer issue partitions, these three substreams may remain per-partition and encounter non-persisted-state issues. Either verify that the less-than-10k partition case works correctly for the history, comments, and worklogs substreams, or explicitly add global_substream_cursor: true to each of these three substreams (located at the line references mentioned in the comment). Apply this same resolution to the sibling configuration locations at lines 8106-8108.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/ingestion/connectors/task-tracking/jira/connector.yaml`:
- Around line 906-908: The comment assumes that history, comments, and worklogs
substreams use global cursors because they exceed the 10k partition cap, but
this behavior is instance-dependent. For Jira sources with fewer issue
partitions, these three substreams may remain per-partition and encounter
non-persisted-state issues. Either verify that the less-than-10k partition case
works correctly for the history, comments, and worklogs substreams, or
explicitly add global_substream_cursor: true to each of these three substreams
(located at the line references mentioned in the comment). Apply this same
resolution to the sibling configuration locations at lines 8106-8108.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: dc97f31e-0ff1-49b1-a8e8-aa59032ed799
📥 Commits
Reviewing files that changed from the base of the PR and between c32821d and 423aff56f233c53d9f5e6dd3540effff4a14a74c.
📒 Files selected for processing (2)
src/ingestion/connectors/task-tracking/jira/connector.yamlsrc/ingestion/connectors/task-tracking/jira/descriptor.yaml
…nc every run constructorfabric#1316 made jira_issue and jira_issue_keys substreams of jira_project_discovery. The CDK then kept their cursor state per-partition (keyed by project — 211 partitions, under the 10k cap that auto-switches to a global cursor), and that per-partition state did NOT persist: the connection state stored `{"states": []}` for both. Every sync therefore re-pulled all projects from jira_start_date — ~2.5h and 2M+ records per run, so the Argo poll timed out and the enrich/dbt steps never ran (jira silver stale; bronze jira_issue frozen at the last fully-committed run). Before constructorfabric#1316, jira_issue was a plain incremental stream and a nightly run was ~80k records in ~3 min. Fix: set `global_substream_cursor: true` on both streams so they use a single max(updated) cursor that persists (the same mechanism the history/comments/worklogs substreams already use automatically, since their per-issue partition count exceeds the 10k cap). Verified live (isolated read against virtuozzo Jira, full manifest + auto-discovery): - state now persists as {"use_global_cursor": true, "state": {"updated": "..."}, "parent_state": {"jira_project_discovery": {...}}} - first read (window from 2026-06-16): 5100 records - resume read with that state: 235 records (was: full re-sync every time) descriptor 2.0.0 -> 2.0.1 so reconcile republishes the manifest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
423aff5 to
f1e7e4e
Compare
3 participants
Problem
After #1316 (project auto-discovery) shipped, every jira sync became a ~2.5h / 2M+ record full re-sync — even nightly incremental runs. The Argo
pollstep times out before the Airbyte job finishes, so the enrich + dbt steps never run (jira silver stale,bronze_jira.jira_issuefrozen at the last fully-committed run). Before #1316 a nightly run was ~80k records in ~3 min.Root cause
#1316 made
jira_issueandjira_issue_keyssubstreams ofjira_project_discovery. The CDK kept their cursor state per-partition (keyed by project — 211 partitions, under the 10,000 cap that auto-switches to a global cursor), and that per-partition state did not persist:state/getreturned{"states": []}for both streams. With no cursor, every sync re-pulled all projects fromjira_start_date.The issue substreams
jira_issue_history/jira_comments/jira_worklogswere unaffected — their per-issue partition count (>130k) exceeds the 10k cap, so the CDK already auto-switched them to a global cursor, which persists fine.Fix
global_substream_cursor: trueonjira_issueandjira_issue_keys— onemax(updated)cursor across all projects, persisted reliably.Verified live (isolated
readagainst virtuozzo Jira, full auto-discovery manifest){"use_global_cursor": true, "state": {"updated": "2026-06-18 09:49"}, "parent_state": {"jira_project_discovery": {...}}}(was{"states": []})descriptor
2.0.0 → 2.0.1so reconcile republishes the manifest (manifest-only edits don't ship without a version bump).Note
The nightly
jira-virtuozzo-synccron is currently suspended on virtuozzo to stop the wasteful failing full-resyncs; it'll be re-enabled after this lands and a one-time full sync completes (after which runs are incremental again).🤖 Generated with Claude Code
Summary by CodeRabbit
jira_issueandjira_issue_keys, improving reliability and preventing unnecessary re-syncs.