[PR #616] feat(jira): auto-discover projects — remove mandatory jira_project_keys - #941
Closed
il10241024 wants to merge 6 commits into
Closed
[PR #616] feat(jira): auto-discover projects — remove mandatory jira_project_keys#941il10241024 wants to merge 6 commits into
il10241024 wants to merge 6 commits into
Conversation
Remove mandatory `jira_project_keys` config. The connector now queries `/rest/api/3/project/search` at the start of each sync to get all accessible project keys, then issues one JQL request per project: project = "<KEY>" AND updated >= "..." AND updated <= "..." This eliminates the need to manually update the secret when Jira projects are added or removed. Changes ------- - connector.yaml: add SubstreamPartitionRouter to jira_issue.retriever that uses an inline jira_project_keys parent stream - connector.yaml: update shared JQL to use stream_slice.project_key - connector.yaml: remove jira_project_keys from spec.required and spec.properties - descriptor.yaml: remove jira_project_keys from secret.required_fields - jira.yaml.example: remove jira_project_keys entry State note: existing connections lose per-source incremental state; full re-sync from jira_start_date will occur on first run after upgrade. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…N/PRD/README/YouTrack-ADR Review feedback from PR #616: 1. Rename inline partition stream jira_project_keys → jira_project_discovery to avoid confusion with the removed config field. 2. Add Jira ADR-001 (auto-project-discovery) documenting the rationale for removing the jira_project_keys allowlist in favour of SubstreamPartitionRouter + token-scoped permissions. 3. Update DESIGN.md: - Remove jira_project_keys row from connection spec table - Update JQL example to project = "<KEY>" form - Add ADR-001 reference and rationale note 4. Update PRD.md step 4: replace manual project scope selection with auto-discovery description. 5. Update Jira connector README: - Remove jira_project_keys from prerequisites, K8s Secret example, fields table, and operational constraints - Add project scope note pointing to ADR-001 6. Update YouTrack ADR-003: - Context now reflects that Jira previously had jira_project_keys but removed it (ADR-001); Jira and YouTrack converge on full-ingestion - Fix Option 2 description and Traceability section accordingly - Remove stale cross-reference to jira_project_keys Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Jira interprets bare datetime literals in the user's local timezone. Sending now_utc() as-is creates a blind spot for users east of UTC: e.g. Europe/Sofia (UTC+3) shifts the JQL upper bound 3 h back, silently excluding issues updated in the last 3 h of the scan window. Adding PT14H (the maximum positive UTC offset globally) ensures the JQL end bound is always in the future relative to the actual current moment for any Jira user timezone. Discovered during local end-to-end test against darthvolt.atlassian.net. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…DR-001
- PRD.md line 481: (see ADR-007) → (see ADR-001) — dangling ref caught
in review; would have failed Cypilot cross-reference validation.
- ADR-001 §Known Behaviors: document two pre-existing edge cases raised
in review as non-blockers:
- /project/search returns archived projects by default (follow-up:
add status=live filter if needed).
- jira_issue uses OffsetIncrement paginator; nextPageToken-based
pagination is a follow-up improvement.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Auto-discovery via SubstreamPartitionRouter changes the per-partition incremental cursor state shape. Existing connections will perform a full re-sync from jira_start_date on the first run after the upgrade, rebuilding the staging tables with the new state layout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve conflict in jira/descriptor.yaml: keep version 2.0.0 (major bump for the breaking removal of mandatory jira_project_keys) over main's 1.1.1 patch bump (#609, #614). 2.0.0 > 1.1.1, and main's patch fixes auto-merged cleanly into connector.yaml. Verified: merged tree vs main == the PR's original 8-file diff (+196/-31), unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThis PR implements a foundational shift in how the Jira connector discovers and syncs projects. Instead of requiring operators to manually specify allowed project keys via configuration, the connector now automatically discovers all projects accessible to the provided API token at sync start, using Jira's project search endpoint. Issues are then partitioned and queried per discovered project using token-scoped Browse Projects permissions as the access boundary. ChangesJira auto-discovery migration
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
docs/components/connectors/task-tracking/jira/specs/ADR/ADR-001-auto-project-discovery.md (1)
89-91: 💤 Low valueConsider rewording to reduce repetition.
Three consecutive lines begin with "
connector.yaml's", which affects readability slightly. While the static analysis tool flags this as a style concern, the current wording is clear and the repetition emphasizes the distinct surfaces being checked.✨ Optional refactor for variety
-- `connector.yaml`'s shared JQL uses `project = "{{ stream_slice.project_key }}"` (not `project IN (...)`). -- `connector.yaml`'s `spec.connection_specification.required` does **not** list `jira_project_keys`. -- `descriptor.yaml:secret.required_fields` does **not** list `jira_project_keys`. +- The shared JQL template uses `project = "{{ stream_slice.project_key }}"` (not `project IN (...)`). +- The `spec.connection_specification.required` array does **not** list `jira_project_keys`. +- `descriptor.yaml:secret.required_fields` does **not** list `jira_project_keys`.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/components/connectors/task-tracking/jira/specs/ADR/ADR-001-auto-project-discovery.md` around lines 89 - 91, Reword the three lines to avoid repeating "`connector.yaml`'s" while keeping the same checks: combine them into a single sentence or parallel list mentioning each surface by its unique symbol names (spec.connection_specification.required in connector.yaml, secret.required_fields in descriptor.yaml, and jira_project_keys in jira.yaml.example) and state that none include jira_project_keys; preserve the meaning but use varied phrasing (e.g., "connector.yaml's spec.connection_specification.required, descriptor.yaml's secret.required_fields, and jira.yaml.example do not include jira_project_keys") for improved readability.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@docs/components/connectors/task-tracking/jira/specs/ADR/ADR-001-auto-project-discovery.md`:
- Line 17: The link fragment `#option-1--static-allowlist-in-ks-secret` is
misspelled and doesn't match the actual heading
`#option-1--static-allowlist-in-k8s-secret`; update the link target in the text
(the string `#option-1--static-allowlist-in-ks-secret`) to
`#option-1--static-allowlist-in-k8s-secret` so the anchor correctly points to
the heading.
In `@docs/components/connectors/task-tracking/jira/specs/PRD.md`:
- Line 481: The PRD currently references the wrong ADR ("ADR-007") in the
sentence starting with "Connector auto-discovers all projects..." — update that
reference to the correct ADR identifier/name used in this PR (e.g.,
ADR-001-auto-project-discovery.md or the ADR id
"cpt-insightspec-adr-jira-auto-project-discovery") so the PRD points to the
actual ADR added; edit the text in PRD.md to replace "ADR-007" with the correct
ADR token and, if present elsewhere, update any related anchor/link text to
match the ADR filename/ID.
In `@src/ingestion/connectors/task-tracking/jira/connector.yaml`:
- Around line 779-823: Add an operator-side optional allowlist so upgrades don't
expand scope: introduce a config key (e.g. config['jira_project_keys'] as an
optional list) and apply it in the jira_project_discovery DeclarativeStream
before partitioning; specifically update the retriever/transformations for the
stream named jira_project_discovery (or insert a Filter transformation) to drop
any project records whose record['key'] is not in config['jira_project_keys']
when that config is present, and leave behavior unchanged when the config is
absent or empty; keep the existing parent_key (project_key) and partition_field
(project_key) unchanged.
---
Nitpick comments:
In
`@docs/components/connectors/task-tracking/jira/specs/ADR/ADR-001-auto-project-discovery.md`:
- Around line 89-91: Reword the three lines to avoid repeating
"`connector.yaml`'s" while keeping the same checks: combine them into a single
sentence or parallel list mentioning each surface by its unique symbol names
(spec.connection_specification.required in connector.yaml,
secret.required_fields in descriptor.yaml, and jira_project_keys in
jira.yaml.example) and state that none include jira_project_keys; preserve the
meaning but use varied phrasing (e.g., "connector.yaml's
spec.connection_specification.required, descriptor.yaml's
secret.required_fields, and jira.yaml.example do not include jira_project_keys")
for improved readability.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cb015067-218d-44d5-9cde-8186dc4a4330
📒 Files selected for processing (8)
docs/components/connectors/task-tracking/jira/specs/ADR/ADR-001-auto-project-discovery.mddocs/components/connectors/task-tracking/jira/specs/DESIGN.mddocs/components/connectors/task-tracking/jira/specs/PRD.mddocs/components/connectors/task-tracking/youtrack/specs/ADR/ADR-003-no-whitelist-full-ingestion.mdsrc/ingestion/connectors/task-tracking/jira/README.mdsrc/ingestion/connectors/task-tracking/jira/connector.yamlsrc/ingestion/connectors/task-tracking/jira/descriptor.yamlsrc/ingestion/secrets/connectors/jira.yaml.example
💤 Files with no reviewable changes (2)
- src/ingestion/connectors/task-tracking/jira/descriptor.yaml
- src/ingestion/secrets/connectors/jira.yaml.example
| - [Consequences](#consequences) | ||
| - [Confirmation](#confirmation) | ||
| - [Pros and Cons of the Options](#pros-and-cons-of-the-options) | ||
| - [Option 1 — Static allowlist in K8s Secret](#option-1--static-allowlist-in-ks-secret) |
There was a problem hiding this comment.
Fix broken link fragment.
The static analysis tool flags this link fragment as invalid. The target #option-1--static-allowlist-in-ks-secret does not match the actual heading #option-1--static-allowlist-in-k8s-secret (missing "8").
🔗 Proposed fix
- - [Option 1 — Static allowlist in K8s Secret](`#option-1--static-allowlist-in-ks-secret`)
+ - [Option 1 — Static allowlist in K8s Secret](`#option-1--static-allowlist-in-k8s-secret`)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - [Option 1 — Static allowlist in K8s Secret](#option-1--static-allowlist-in-ks-secret) | |
| - [Option 1 — Static allowlist in K8s Secret](`#option-1--static-allowlist-in-k8s-secret`) |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 17-17: Link fragments should be valid
(MD051, link-fragments)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@docs/components/connectors/task-tracking/jira/specs/ADR/ADR-001-auto-project-discovery.md`
at line 17, The link fragment `#option-1--static-allowlist-in-ks-secret` is
misspelled and doesn't match the actual heading
`#option-1--static-allowlist-in-k8s-secret`; update the link target in the text
(the string `#option-1--static-allowlist-in-ks-secret`) to
`#option-1--static-allowlist-in-k8s-secret` so the anchor correctly points to
the heading.
Comment on lines
+779
to
+823
| partition_router: | ||
| type: SubstreamPartitionRouter | ||
| parent_stream_configs: | ||
| - type: ParentStreamConfig | ||
| stream: | ||
| type: DeclarativeStream | ||
| name: jira_project_discovery | ||
| primary_key: [] | ||
| retriever: | ||
| type: SimpleRetriever | ||
| record_selector: | ||
| type: RecordSelector | ||
| extractor: | ||
| type: DpathExtractor | ||
| field_path: | ||
| - values | ||
| paginator: | ||
| $ref: "#/definitions/linked/SimpleRetriever/paginator" | ||
| requester: | ||
| type: HttpRequester | ||
| http_method: GET | ||
| authenticator: | ||
| $ref: "#/definitions/linked/HttpRequester/authenticator" | ||
| error_handler: | ||
| $ref: "#/definitions/linked/HttpRequester/error_handler" | ||
| url: "{{ config['jira_instance_url'] }}/rest/api/3/project/search" | ||
| transformations: | ||
| - type: AddFields | ||
| fields: | ||
| - type: AddedFieldDefinition | ||
| path: | ||
| - project_key | ||
| value: "{{ record['key'] }}" | ||
| schema_loader: | ||
| type: InlineSchemaLoader | ||
| schema: | ||
| type: object | ||
| properties: | ||
| project_key: | ||
| type: | ||
| - string | ||
| - "null" | ||
| additionalProperties: true | ||
| parent_key: project_key | ||
| partition_field: project_key |
There was a problem hiding this comment.
Keep an operator-side scope guard for project discovery.
Removing jira_project_keys means an upgrade can expand an existing connection from a curated subset to every project visible to the token, and the first post-upgrade sync will backfill that broader scope from jira_start_date. Please keep an optional allowlist/filter so teams can preserve data-minimization without having to redesign Jira permissions.
Also applies to: 7945-7999
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/ingestion/connectors/task-tracking/jira/connector.yaml` around lines 779
- 823, Add an operator-side optional allowlist so upgrades don't expand scope:
introduce a config key (e.g. config['jira_project_keys'] as an optional list)
and apply it in the jira_project_discovery DeclarativeStream before
partitioning; specifically update the retriever/transformations for the stream
named jira_project_discovery (or insert a Filter transformation) to drop any
project records whose record['key'] is not in config['jira_project_keys'] when
that config is present, and leave behavior unchanged when the config is absent
or empty; keep the existing parent_key (project_key) and partition_field
(project_key) unchanged.
il10241024
changed the title
Contributor
|
Superseded by #1316, which reworks this idea against the current manifest (crediting the original design from this PR / cyberfabric/cyber-insight#616). Why a rework rather than a rebase — the connector changed significantly since this was written:
Thanks for the groundwork — the inline discovery parent and per-project JQL shape carried over directly. |
mitasovr
added a commit
that referenced
this pull request
Jun 15, 2026
…list (#1316) Replaces the manually-maintained jira_project_keys secret field with runtime project discovery. Supersedes #941 (idea and per-project JQL by mozhaev-dev, cyberfabric/cyber-insight#616), reworked against the current manifest: - New inline-only parent `jira_project_discovery` under `definitions` (never visible to discover, so reconcile cannot select it as a bronze table): GET /rest/api/3/project/search?expand=insight enumerates every project the API token can see. - BOTH jira_issue and jira_issue_keys (the lightweight parent of jira_issue_history/comments/worklogs from #1283) are partitioned per project with JQL `project = "<KEY>"`. #941 predates jira_issue_keys and would have silently broken all three substreams by removing the config their JQL still referenced. - Incremental discovery gate: client-side cursor on the hoisted insight.lastIssueUpdateTime — first sync emits all projects, later syncs only projects whose issues changed. Verified live: first read enumerated 211 projects (25 with fresh issues, 2478 records — the 4-project allowlist would have missed 9 of them), resume read touched only 3/211 partitions. Projects without issues default to epoch and stay filtered until their first issue. - Timezone tail fix done safely: lookback_window PT1H -> PT14H on the issue scans. #941 instead pushed end_datetime 14h into the future, which advances the cursor past the actual query time and permanently skips records updated in between. - jira_project_keys removed from spec, descriptor required_fields, secret example and README. - descriptor 1.2.1 -> 2.0.0 (major): per-project partitioning resets incremental state; reconcile dispatches a full refresh on major bumps, which is the intended migration. Known costs, documented in-manifest: the first sync after rollout is a full re-sync of every visible project since jira_start_date; the client-side gate compares strictly against the cursor (lookback does not widen it on resume), so a lagging insight aggregate delays that project until its next update. validate (CDK runtime): manifest valid. validate-strict: 14 pre-existing errors on main (jira is the known whole-object-$ref anti-template); unchanged by this diff. Co-authored-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
mozhaev-dev
added a commit
that referenced
this pull request
Jun 15, 2026
Lesson applied from jira #1316: the incremental cursors on support_tickets and zendesk_satisfaction_ratings had no lookback. Zendesk's incremental export is eventually-consistent at the cursor boundary, so a record whose updated_at lands at the edge is permanently skipped on the next sync — and for a ticket that drops its entire audit activity (the only source of updates/comments/solved). Added lookback_window: P1D to both cursors so each sync re-queries a 1-day tail; append-only RMT bronze + read-time dedup absorb the re-delivery (verified: bronze distinct stable 32/5/102, silver exact 36/36 across 4 syncs, zero duplicates). NB the fix is lookback_window, NOT pushing end_datetime into the future (the #941 approach #1316 flagged: that advances the cursor past the real query time and permanently skips rows). descriptor 1.2.2 → 1.2.3 (patch: cursor state stays valid, lookback only widens the start each run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Sergei Mozhaev <mozhaev.dev@gmail.com>
mozhaev-dev
added a commit
that referenced
this pull request
Jun 18, 2026
…ts Link pagination The org/user daily-metrics streams advanced the incremental cursor per-slice unconditionally (the "Major #5" rule), starting the next run at cursor+1. But Copilot daily reports lag (often >24-48h; weekends later) and can be restated, so a recent day that returned HTTP 204 because its report wasn't ready yet still advanced the cursor past it — and was then NEVER re-fetched once the data landed. Silent data gaps for late-arriving / adjusted days (same class as the jira/#941 cursor-skip and the Zendesk lookback_window lesson). Fix: both report streams now re-query a trailing lookback window (config `metrics_lookback_days`, default 7) — start = max(start_date, cursor - lookback_days) instead of cursor+1. The recent tail self-heals every run; RMT dedup on unique_key ({tenant}-{source}-[user]-{day}) makes the overlap idempotent. Genuinely-empty historical days still don't get re-fetched forever (only the trailing window is revisited). lookback_days wired through source.py + spec.json. Also: seats pagination now follows GitHub's `Link: rel="next"` header (with the count heuristic as fallback) instead of stopping purely on "page < 100". descriptor 2.1.0 → 2.2.0 (minor: new optional config, no Bronze schema change). Verified live against a real Copilot Business org: with cursor=2026-06-13 + lookback=7 the read re-fetched 2026-06-06..06-15 (all 7 pre-cursor days, incl. active days the old cursor+1 would have skipped), 0 errors; seats paginated correctly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Sergei Mozhaev <mozhaev.dev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Jira connector required
jira_project_keys(comma-separated list) to be manually maintained in the K8s secret. As Jira projects are frequently added/removed, this became operationally painful.Solution
Switch
jira_issuestream to SubstreamPartitionRouter: on every sync the connector queries/rest/api/3/project/searchto get all accessible project keys, then runs one JQL request per project:New projects are picked up automatically on the next scheduled sync. Deleted/archived projects simply disappear from the parent stream and are no longer queried.
Changes
connector.yamlSubstreamPartitionRouterwith inlinejira_project_keysparent stream tojira_issue.retriever; updated JQL tostream_slice.project_key; removedjira_project_keysfrom specdescriptor.yamljira_project_keysfromsecret.required_fieldsjira.yaml.examplejira_project_keysentryState migration
Existing connections will lose their incremental cursor state and perform a full re-sync from
jira_start_dateon first run after upgrade. This is acceptable — current data has no production value.Architecture
🤖 Generated with Claude Code
Summary by CodeRabbit
Refactor
Documentation
Chores