feat(jira): auto-discover projects — drop the jira_project_keys allowlist - #1316
Merged
mitasovr merged 1 commit intoJun 15, 2026
Merged
Conversation
…list Replaces the manually-maintained jira_project_keys secret field with runtime project discovery. Supersedes constructorfabric#941 (idea and per-project JQL by mozhaev-dev, cyberfabric/cyber-insight#616), reworked against the current manifest: - New inline-only parent `jira_project_discovery` under `definitions` (never visible to discover, so reconcile cannot select it as a bronze table): GET /rest/api/3/project/search?expand=insight enumerates every project the API token can see. - BOTH jira_issue and jira_issue_keys (the lightweight parent of jira_issue_history/comments/worklogs from constructorfabric#1283) are partitioned per project with JQL `project = "<KEY>"`. constructorfabric#941 predates jira_issue_keys and would have silently broken all three substreams by removing the config their JQL still referenced. - Incremental discovery gate: client-side cursor on the hoisted insight.lastIssueUpdateTime — first sync emits all projects, later syncs only projects whose issues changed. Verified live: first read enumerated 211 projects (25 with fresh issues, 2478 records — the 4-project allowlist would have missed 9 of them), resume read touched only 3/211 partitions. Projects without issues default to epoch and stay filtered until their first issue. - Timezone tail fix done safely: lookback_window PT1H -> PT14H on the issue scans. constructorfabric#941 instead pushed end_datetime 14h into the future, which advances the cursor past the actual query time and permanently skips records updated in between. - jira_project_keys removed from spec, descriptor required_fields, secret example and README. - descriptor 1.2.1 -> 2.0.0 (major): per-project partitioning resets incremental state; reconcile dispatches a full refresh on major bumps, which is the intended migration. Known costs, documented in-manifest: the first sync after rollout is a full re-sync of every visible project since jira_start_date; the client-side gate compares strictly against the cursor (lookback does not widen it on resume), so a lagging insight aggregate delays that project until its next update. validate (CDK runtime): manifest valid. validate-strict: 14 pre-existing errors on main (jira is the known whole-object-$ref anti-template); unchanged by this diff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe Jira connector refactors project scoping from manual ChangesJira Project Auto-Discovery Refactor
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/ingestion/connectors/task-tracking/jira/README.md (1)
14-15:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winRemove outdated manual project-key prerequisite.
This still instructs users to manually choose project keys and says it is required, which conflicts with the new runtime auto-discovery behavior and can mislead setup.
Suggested doc fix
-3. Identify the project keys to sync (e.g. `TC`, `TNG`) — visible in any issue URL as the prefix before the hyphen. Jira Cloud rejects unbounded JQL queries, so this is **required**. +3. Ensure the API token can browse all projects you want ingested. Project scope is auto-discovered at runtime and queried per project with bounded JQL.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/ingestion/connectors/task-tracking/jira/README.md` around lines 14 - 15, Update the README.md to remove or replace the outdated instruction "Identify the project keys to sync (e.g. `TC`, `TNG`) — visible in any issue URL as the prefix before the hyphen. Jira Cloud rejects unbounded JQL queries, so this is **required**." with a note that project keys are auto-discovered at runtime (or optional) by the connector; ensure the wording around project key selection reflects the new runtime auto-discovery behavior and does not state it as a required manual step so users are not misled.src/ingestion/connectors/task-tracking/jira/connector.yaml (1)
8021-8025:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRegister
jira_issue_keysinautoImportSchema.This stream is now documented as a real bronze table, but
metadata.autoImportSchemastill omits it while the other persisted Jira streams are listed there. If schema import is driven by that map, reconcile can selectjira_issue_keyswhile the ingestion layer never auto-registers its schema.Suggested fix
metadata: autoImportSchema: jira_fields: true jira_projects: true jira_user: true jira_issue: true + jira_issue_keys: true jira_issue_history: true jira_comments: true jira_worklogs: trueAlso applies to: 8026-8181, 8237-8249
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/ingestion/connectors/task-tracking/jira/connector.yaml` around lines 8021 - 8025, The metadata.autoImportSchema map in connector.yaml omits the newly promoted bronze stream jira_issue_keys, causing schema auto-registration to skip it; update the metadata.autoImportSchema entry to include "jira_issue_keys" alongside the other persisted Jira streams so the ingestion layer will auto-register its schema (modify the autoImportSchema map where other jira bronze tables are listed to add the jira_issue_keys key and ensure its metadata aligns with the existing bronze entries).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/ingestion/connectors/task-tracking/jira/connector.yaml`:
- Around line 128-143: The incremental cursor logic uses a single global
last_issue_update_time (cursor_field) in jira_project_discovery which causes
projects that become newly visible but have older aggregate timestamps to be
permanently skipped; update the gating so it is project-aware: change
jira_project_discovery and/or the DatetimeBasedCursor usage to either persist
per-project cursor state keyed by project_key or always emit partitions for
unseen project_key values regardless of the global last_issue_update_time, and
update the issue streams that consume last_issue_update_time to read
project-scoped partitions (or consult the per-project cursor) so backfills for
permission/visibility changes are emitted.
---
Outside diff comments:
In `@src/ingestion/connectors/task-tracking/jira/connector.yaml`:
- Around line 8021-8025: The metadata.autoImportSchema map in connector.yaml
omits the newly promoted bronze stream jira_issue_keys, causing schema
auto-registration to skip it; update the metadata.autoImportSchema entry to
include "jira_issue_keys" alongside the other persisted Jira streams so the
ingestion layer will auto-register its schema (modify the autoImportSchema map
where other jira bronze tables are listed to add the jira_issue_keys key and
ensure its metadata aligns with the existing bronze entries).
In `@src/ingestion/connectors/task-tracking/jira/README.md`:
- Around line 14-15: Update the README.md to remove or replace the outdated
instruction "Identify the project keys to sync (e.g. `TC`, `TNG`) — visible in
any issue URL as the prefix before the hyphen. Jira Cloud rejects unbounded JQL
queries, so this is **required**." with a note that project keys are
auto-discovered at runtime (or optional) by the connector; ensure the wording
around project key selection reflects the new runtime auto-discovery behavior
and does not state it as a required manual step so users are not misled.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8d0c452d-c3a4-49a9-9e1b-a72b3bfa6817
📒 Files selected for processing (4)
src/ingestion/connectors/task-tracking/jira/README.mdsrc/ingestion/connectors/task-tracking/jira/connector.yamlsrc/ingestion/connectors/task-tracking/jira/descriptor.yamlsrc/ingestion/secrets/connectors/jira.yaml.example
💤 Files with no reviewable changes (1)
- src/ingestion/secrets/connectors/jira.yaml.example
Comment on lines
+128
to
+143
| incremental_sync: | ||
| type: DatetimeBasedCursor | ||
| cursor_field: last_issue_update_time | ||
| is_client_side_incremental: true | ||
| cursor_datetime_formats: | ||
| - "%Y-%m-%dT%H:%M:%S.%f%z" | ||
| datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z" | ||
| start_datetime: | ||
| type: MinMaxDatetime | ||
| datetime: "1970-01-01" | ||
| datetime_format: "%Y-%m-%d" | ||
| end_datetime: | ||
| type: MinMaxDatetime | ||
| datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%S.%f%z') }}" | ||
| datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z" | ||
| lookback_window: P3D |
There was a problem hiding this comment.
Parent incremental state can permanently skip newly visible projects.
jira_project_discovery stores a single max last_issue_update_time, and both issue streams only read partitions that this parent re-emits. If the token later gains access to a project whose aggregate timestamp is already older than that cursor, that project never gets partitioned at all until another issue update happens there. That silently drops backfill for permission changes, restored visibility, renamed projects, and other “newly seen but not freshly updated” cases. Consider making the gate project-aware (emit unseen project_key values regardless of timestamp, or persist per-project state) instead of relying on one global timestamp cursor.
Also applies to: 879-887, 8067-8075
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/ingestion/connectors/task-tracking/jira/connector.yaml` around lines 128
- 143, The incremental cursor logic uses a single global last_issue_update_time
(cursor_field) in jira_project_discovery which causes projects that become newly
visible but have older aggregate timestamps to be permanently skipped; update
the gating so it is project-aware: change jira_project_discovery and/or the
DatetimeBasedCursor usage to either persist per-project cursor state keyed by
project_key or always emit partitions for unseen project_key values regardless
of the global last_issue_update_time, and update the issue streams that consume
last_issue_update_time to read project-scoped partitions (or consult the
per-project cursor) so backfills for permission/visibility changes are emitted.
mozhaev-dev
approved these changes
Jun 15, 2026
mozhaev-dev
added a commit
that referenced
this pull request
Jun 15, 2026
Lesson applied from jira #1316: the incremental cursors on support_tickets and zendesk_satisfaction_ratings had no lookback. Zendesk's incremental export is eventually-consistent at the cursor boundary, so a record whose updated_at lands at the edge is permanently skipped on the next sync — and for a ticket that drops its entire audit activity (the only source of updates/comments/solved). Added lookback_window: P1D to both cursors so each sync re-queries a 1-day tail; append-only RMT bronze + read-time dedup absorb the re-delivery (verified: bronze distinct stable 32/5/102, silver exact 36/36 across 4 syncs, zero duplicates). NB the fix is lookback_window, NOT pushing end_datetime into the future (the #941 approach #1316 flagged: that advances the cursor past the real query time and permanently skips rows). descriptor 1.2.2 → 1.2.3 (patch: cursor state stays valid, lookback only widens the start each run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Sergei Mozhaev <mozhaev.dev@gmail.com>
mitasovr
pushed a commit
to mitasovr/insight
that referenced
this pull request
Jun 18, 2026
…nc every run constructorfabric#1316 made jira_issue and jira_issue_keys substreams of jira_project_discovery. The CDK then kept their cursor state per-partition (keyed by project — 211 partitions, under the 10k cap that auto-switches to a global cursor), and that per-partition state did NOT persist: the connection state stored `{"states": []}` for both. Every sync therefore re-pulled all projects from jira_start_date — ~2.5h and 2M+ records per run, so the Argo poll timed out and the enrich/dbt steps never ran (jira silver stale; bronze jira_issue frozen at the last fully-committed run). Before constructorfabric#1316, jira_issue was a plain incremental stream and a nightly run was ~80k records in ~3 min. Fix: set `global_substream_cursor: true` on both streams so they use a single max(updated) cursor that persists (the same mechanism the history/comments/worklogs substreams already use automatically, since their per-issue partition count exceeds the 10k cap). Verified live (isolated read against virtuozzo Jira, full manifest + auto-discovery): - state now persists as {"use_global_cursor": true, "state": {"updated": "..."}, "parent_state": {"jira_project_discovery": {...}}} - first read (window from 2026-06-16): 5100 records - resume read with that state: 235 records (was: full re-sync every time) descriptor 2.0.0 -> 2.0.1 so reconcile republishes the manifest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
mitasovr
added a commit
that referenced
this pull request
Jun 18, 2026
…nc every run (#1370) #1316 made jira_issue and jira_issue_keys substreams of jira_project_discovery. The CDK then kept their cursor state per-partition (keyed by project — 211 partitions, under the 10k cap that auto-switches to a global cursor), and that per-partition state did NOT persist: the connection state stored `{"states": []}` for both. Every sync therefore re-pulled all projects from jira_start_date — ~2.5h and 2M+ records per run, so the Argo poll timed out and the enrich/dbt steps never ran (jira silver stale; bronze jira_issue frozen at the last fully-committed run). Before #1316, jira_issue was a plain incremental stream and a nightly run was ~80k records in ~3 min. Fix: set `global_substream_cursor: true` on both streams so they use a single max(updated) cursor that persists (the same mechanism the history/comments/worklogs substreams already use automatically, since their per-issue partition count exceeds the 10k cap). Verified live (isolated read against virtuozzo Jira, full manifest + auto-discovery): - state now persists as {"use_global_cursor": true, "state": {"updated": "..."}, "parent_state": {"jira_project_discovery": {...}}} - first read (window from 2026-06-16): 5100 records - resume read with that state: 235 records (was: full re-sync every time) descriptor 2.0.0 -> 2.0.1 so reconcile republishes the manifest. Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
SharedQA
added a commit
to SharedQA/insight
that referenced
this pull request
Jun 22, 2026
…tion The jira descriptor lost the jira_project_keys required field when constructorfabric#1316 replaced the allowlist with auto-discovery; that change reached this branch via the main merge but the hardcoded expectation in test_fallback_parses_real_jira_descriptor still listed four fields, so the E2E suite failed on the stale assertion (the safe_load parity check passed). Align the expected list with the current three-field descriptor. Signed-off-by: Kenan Salim <kenan.salim@rolos.com>
4 tasks
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #941 (original idea and per-project JQL by mozhaev-dev, mirrored from cyberfabric/cyber-insight#616) — reworked from scratch against the current manifest, since #941 predates #1283/#1308/#1310 and can no longer apply cleanly.
What
jira_project_discovery— inline-only parent underdefinitions(invisible todiscover(), so reconcile/ADR-0015 cannot auto-select it as a bronze table):GET /rest/api/3/project/search?expand=insightenumerates every project visible to the API token.jira_issueandjira_issue_keysare partitioned per project (project = "<KEY>"JQL). This is the critical delta vs [PR #616] feat(jira): auto-discover projects — remove mandatory jira_project_keys #941: it predatesjira_issue_keys(fix(ingestion): stop silent jira sync hang via lightweight substream parent #1283) and, by deleting the config key its JQL still references, would have silently zeroed outjira_issue_history/jira_comments/jira_worklogs.insight.lastIssueUpdateTime— first sync emits all projects; subsequent syncs only projects whose issues changed since the previous run. Comments/worklogs bump the issue'supdated→ the project aggregate, so one gate covers all substreams.lookback_window PT1H → PT14Hon the issue scans (JQL bare datetimes are evaluated in the API user's local TZ). [PR #616] feat(jira): auto-discover projects — remove mandatory jira_project_keys #941 instead pushedend_datetime14h into the future — that advances the cursor past the actual query time and permanently skips records updated in between.jira_project_keysremoved everywhere (spec, descriptorrequired_fields, secret example, README).1.2.1 → 2.0.0(major → reconcile dispatches full refresh; the per-project partitioning resets incremental state anyway — intended migration, per the chore(connectors): bump jira/confluence descriptor versions to publish #1308 #1310 lesson the bump ships in the same PR as the manifest change).Verified live (virtuozzo Jira, isolated
readruns)jira_project_keysin configunique_key = virtuozzo-jira-main-<KEY>intactparent_state.jira_project_discovery.last_issue_update_timeadvancesvalidatevalidate-strict$refanti-template) — count and nature unchanged by this diffKnown costs (documented in-manifest)
jira_start_date— plan for a long first run.lookback_windowdoes not widen it on resume). If Atlassian'sinsightaggregate ever lags, that project's data is delayed until its next issue update pushes the aggregate past the cursor. Accepted: the aggregate was observed second-fresh on live Cloud; any full refresh re-covers everything.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Breaking Changes
jira_project_keysconfiguration parameter; projects are now auto-discovered based on available access.Documentation