constructorfabric · il10241024 · Jun 2, 2026 · Jun 2, 2026 · Jun 3, 2026 · Jun 3, 2026
@@ -0,0 +1,125 @@
+---
+status: accepted
+date: 2026-06-02
+---
+
+# Auto-discovery of Jira projects via SubstreamPartitionRouter
+
+<!-- toc -->
+
+- [Context and Problem Statement](#context-and-problem-statement)
+- [Decision Drivers](#decision-drivers)
+- [Considered Options](#considered-options)
+- [Decision Outcome](#decision-outcome)
+  - [Consequences](#consequences)
+  - [Confirmation](#confirmation)
+- [Pros and Cons of the Options](#pros-and-cons-of-the-options)
+  - [Option 1 — Static allowlist in K8s Secret](#option-1--static-allowlist-in-ks-secret)
-  - [Option 1 — Static allowlist in K8s Secret](#option-1--static-allowlist-in-ks-secret)
+  - [Option 1 — Static allowlist in K8s Secret](`#option-1--static-allowlist-in-k8s-secret`)
-  - [Option 1 — Static allowlist in K8s Secret](#option-1--static-allowlist-in-ks-secret)
+  - [Option 1 — Static allowlist in K8s Secret](`#option-1--static-allowlist-in-k8s-secret`)
+  - [Option 2 — Auto-discovery via SubstreamPartitionRouter](#option-2--auto-discovery-via-substreampartitionrouter)
+- [More Information](#more-information)
+- [Traceability](#traceability)
+
+<!-- /toc -->
+
+**ID**: `cpt-insightspec-adr-jira-auto-project-discovery`
+
+## Context and Problem Statement
+
+The original Jira connector required a `jira_project_keys` field in the K8s Secret — a comma-separated list of Jira project keys to sync (e.g., `TC,TNG`). The rationale at the time was that "Jira Cloud rejects unbounded JQL queries."
+
+In practice this created operational pain:
+
+- Jira projects are frequently created and archived. Every change required a manual Secret edit and connector re-trigger.
+- The "unbounded query" concern was a misconception: Jira Cloud's API restriction applies to queries with *no* bounds at all. The connector already queries in 30-day windows (`step: P30D`), which is a valid temporal bound. Queries bounded by project key *and* time window are equivalent in cost and reliability to queries bounded by time window alone.
+- The `jira_project_keys` allowlist diverges from the token's Browse Projects permission boundary. Rotating the token without updating the allowlist silently keeps ingesting a stale project set — or silently drops new projects.
+
+## Decision Drivers
+
+- Operational simplicity: no manual maintenance when projects are added or archived.
+- Permission-boundary alignment: the Jira API token already constrains what the connector can read.
+- Consistency with YouTrack: YouTrack ADR-003 already chose full-ingestion for the same reasons.
+- Correctness: per-project partitioning gives per-project incremental cursor state, which is strictly better than a single global cursor.
+
+## Considered Options
+
+1. **Static allowlist in K8s Secret** (`jira_project_keys`) — keep the current behaviour.
+2. **Auto-discovery via SubstreamPartitionRouter** — query `GET /rest/api/3/project/search` at the start of each sync to obtain all accessible project keys; partition `jira_issue` over those keys.
+
+## Decision Outcome
+
+Chosen option: **auto-discovery via SubstreamPartitionRouter** (Option 2).
+
+The connector now issues one JQL request per project partition:
+
+```sql
+project = "<KEY>" AND updated >= "<t_start>" AND updated <= "<t_end>"
+ORDER BY updated ASC
+```
+
+Project scope is delegated to Jira's Browse Projects permission on the API token. To limit ingestion to a specific subset of projects, operators scope the token in Jira — not in Insight config.
+
+The `jira_project_keys` field is removed from `spec.connection_specification.required`, `spec.connection_specification.properties`, `descriptor.yaml:secret.required_fields`, and `jira.yaml.example`.
+
+### Consequences
+
+**Positive**:
+
+- New or renamed projects are ingested automatically on the next scheduled sync without any config change.
+- Archived/deleted projects disappear from `/project/search` and are no longer queried — no stale state.
+- Per-project incremental cursor state: each project advances independently; a new project backfills from `jira_start_date` without affecting other projects.
+- Jira and YouTrack connectors converge on the same architecture (full-ingestion, token-scoped).
+- Eliminates the class of drift bugs where `jira_project_keys` is stale relative to the token scope.
+
+**Negative**:
+
+- Operators who want to limit ingestion to a subset of projects must manage token permissions in Jira rather than in Insight config. This is a one-time workflow change.
+- Syncing more projects costs more Jira API calls. For instances with hundreds of projects this adds latency to the directory-discovery phase. Jira rate-limits (429/503) are handled by the existing `Retry-After` backoff strategy.
+- The parent stream (`jira_project_discovery`) makes an additional `/project/search` call every sync run. This is negligible — project lists are small (typically < 200) and fully paginated in one or two requests.
+
+**State migration**:
+
+Existing connections lose their per-source incremental cursor state on upgrade. The connector will perform a full re-sync from `jira_start_date` on the first run after the change. Acceptable for current deployments (dev stage; data has no production value).
+
+### Confirmation
+
+Decision is confirmed when:
+
+- `connector.yaml`'s `jira_issue.retriever` contains a `SubstreamPartitionRouter` whose parent stream queries `/rest/api/3/project/search`.
+- `connector.yaml`'s shared JQL uses `project = "{{ stream_slice.project_key }}"` (not `project IN (...)`).
+- `connector.yaml`'s `spec.connection_specification.required` does **not** list `jira_project_keys`.
+- `descriptor.yaml:secret.required_fields` does **not** list `jira_project_keys`.
+- `jira.yaml.example` does **not** contain a `jira_project_keys` entry.
+
+## Pros and Cons of the Options
+
+### Option 1 — Static allowlist in K8s Secret
+
+- **Pros**: Explicit scope visible in config; operators control exactly which projects are ingested from Insight UI.
+- **Cons**: Manual maintenance on every project change; allowlist can drift from token scope; blocks automatic onboarding of new projects; diverges from YouTrack architecture.
+
+### Option 2 — Auto-discovery via SubstreamPartitionRouter
+
+- **Pros**: Zero maintenance; auto-pickup of new projects; per-project cursor state; consistent with YouTrack; permission boundary is the token (single source of truth).
+- **Cons**: Scope management moves to Jira token administration; proportional API cost on large instances.
+
+## More Information
+
+- Jira Cloud Project Search API: `GET /rest/api/3/project/search` — supports `startAt`/`maxResults` offset pagination; returns all projects accessible to the authenticated user.
+- YouTrack equivalent decision: `docs/components/connectors/task-tracking/youtrack/specs/ADR/ADR-003-no-whitelist-full-ingestion.md`.
+- Airbyte CDK `SubstreamPartitionRouter`: combines with `DatetimeBasedCursor` via cartesian product — each `(project_key, time_window)` pair becomes a request.
+
+## Traceability
+
+- Supersedes the `jira_project_keys` design documented in DESIGN §3.3 and PRD §3.1.
+- Mirrors YouTrack ADR-003 decision; Jira and YouTrack now converge on full-ingestion scope.
+- Implementation PR: `feat/jira-auto-project-discovery`.
+
+## Known Behaviors
+
+### Archived projects
+
+`GET /rest/api/3/project/search` returns **all project types** including archived projects by default (Jira Cloud does not filter by `status=live` unless explicitly requested). In practice archived projects return 0 issues via JQL and contribute negligible API cost. If this becomes a concern a follow-up can add `status=live` to the `jira_project_discovery` parent stream's `request_parameters` — that is a backwards-compatible, non-breaking change.
+
+### Issue pagination (`/rest/api/3/search/jql`)
+
+The connector currently uses an `OffsetIncrement` paginator (`startAt` / `maxResults`) against `/rest/api/3/search/jql`. Atlassian's enhanced-JQL endpoint supports cursor-based pagination via `nextPageToken` (already used by the connector's own `CursorPagination` paginator config). The two mechanisms coexist on this endpoint — offset pagination is functional but will be replaced with `nextPageToken`-based pagination in a follow-up to align with Atlassian's recommended approach for large result sets.
@@ -195,7 +195,7 @@ The connector writes only to Bronze tables. Cross-source unification, enum norma
 
 | Entity | Description | Maps To |
 |--------|-------------|---------|
-| `JiraInstance` | Connection configuration: URL, credentials, project scope | Connector config (spec section) |
+| `JiraInstance` | Connection configuration: URL, credentials | Connector config (spec section) |
 | `JiraIssue` | Issue with core fields: `id`, `key`, `project`, `issuetype`, `reporter`, `story_points`, `duedate`, `parent`, `created`, `updated` | `jira_issue` |
 | `JiraChangelog` | Per-issue changelog entries with `items[]` array; each item is a field change with `from`/`to` + display strings | `jira_issue_history` (one row per field change) |
 | `JiraWorklog` | Time entry: `id`, `issueKey`, `author`, `started`, `timeSpentSeconds`, `comment` | `jira_worklogs` |
@@ -400,7 +400,6 @@ In Phase 1, Atlassian Document Format (ADF) JSON from Jira Cloud REST API v3 com
 | `jira_api_token` | str (airbyte_secret) | Jira Cloud API token |
 | `insight_tenant_id` | str | Insight tenant identifier — injected into every record |
 | `insight_source_id` | str | Instance discriminator (e.g., `jira-team-alpha`) |
-| `jira_project_keys` | str | **Required.** Comma-separated project keys — Jira Cloud does not allow unbounded JQL queries |
 | `jira_start_date` | str | Earliest date to sync issues from, `YYYY-MM-DD` (default `2020-01-01`) |
 | `jira_page_size` | int | Page size for JQL search only (default 50, max 100). Passed as `request_parameters.maxResults` on the search stream. All other paginators use hardcoded `page_size` values optimized per endpoint API max: `paginator` 50 (projects, comments), `agile_paginator` 50 (boards, sprints), `child_paginator` 100 (changelog, worklogs), `user_paginator` 200 (user directory) |
 
@@ -592,9 +591,11 @@ All tables use `ReplacingMergeTree(_version)` with `_version = toUnixTimestamp64
 **JQL for Incremental Sync**:
 
 ```sql
-project IN ({project_keys}) AND updated >= "{last_cursor}" ORDER BY updated ASC
+project = "{project_key}" AND updated >= "{last_cursor}" ORDER BY updated ASC
 ```
 
+where `project_key` is supplied per-partition by the `SubstreamPartitionRouter` that discovers all accessible projects via `GET /rest/api/3/project/search` at the start of each sync (ADR-001). Scope is controlled by the API token's Browse Projects permission, not by connector config.
+
 The connector overlaps the cursor window by 1 hour (`lookback_window: PT1H`) to account for issues updated during the previous sync run. Deduplication at the storage level (`ReplacingMergeTree`) handles the overlap.
 
 **Key Endpoints by Stream**:

@@ -478,7 +478,7 @@ All timestamps persisted in the Bronze layer **MUST** be stored in UTC (ISO 8601
 1. Operator provides Jira instance URL and credentials
 2. System validates credentials against the Jira API
 3. System discovers available projects and their project styles (Classic/Next-gen)
-4. Operator selects project scope (all projects or specific project keys)
+4. Connector auto-discovers all projects accessible to the token via `GET /rest/api/3/project/search` — no operator input required (see ADR-001)
 5. System auto-detects the story points field: Next-gen projects use `customfield_10016`; Classic projects are scanned via the field metadata API
 6. If multiple candidate story points fields are found in Classic projects, system presents a selection list to the operator
 7. System initializes the connection with empty state

@@ -26,9 +26,9 @@ date: 2026-04-23
 **ID**: `cpt-insightspec-adr-youtrack-no-whitelist`
 ## Context and Problem Statement
 
-The Jira connector accepts a `jira_project_keys` K8s Secret field that restricts ingestion to a specific list of projects. This was driven by Jira-side requirements (Phase 1 customer instances often had hundreds of projects, of which only a handful were relevant for analytics).
+Originally the Jira connector accepted a `jira_project_keys` K8s Secret field that restricted ingestion to a specific list of projects. That field has since been removed (see Jira ADR-001 — auto-discovery via SubstreamPartitionRouter); Jira and YouTrack now converge on the same full-ingestion approach.
 
-For YouTrack, we have to decide whether to mirror this — adding a `youtrack_project_short_names` K8s Secret field that restricts the manifest's JQL-equivalent query and the project-fan-out streams — or to ingest everything the permanent token can reach.
+For YouTrack, we have to decide whether to add a `youtrack_project_short_names` K8s Secret field that restricts the manifest's JQL-equivalent query and the project-fan-out streams — or to ingest everything the permanent token can reach.
 
 The decision affects:
 
@@ -49,7 +49,7 @@ The decision affects:
 ## Considered Options
 
 1. **No whitelist** — ingest every project the token can reach.
-2. **Project-allowlist via K8s Secret** — mirror Jira's `jira_project_keys`.
+2. **Project-allowlist via K8s Secret** — add a `youtrack_project_short_names` field (Jira previously had this as `jira_project_keys`, but removed it in ADR-001).
 3. **Tenant-side scoping only** — operators create a service-account token restricted to specific projects in YouTrack; the connector trusts the token's scope.
 
 ## Decision Outcome
@@ -98,8 +98,8 @@ Decision is confirmed when:
 
 ### Option 2 — Project allowlist via K8s Secret
 
-- **Pros**: Symmetric with Jira; per-source operator UX consistent.
-- **Cons**: Duplicates token permission boundary; allowlist can drift from token scope without warning; adds maintenance burden as projects are added / archived.
+- **Pros**: Per-source operator UX: explicit scope list.
+- **Cons**: Duplicates token permission boundary; allowlist can drift from token scope without warning; adds maintenance burden as projects are added / archived. (Note: Jira previously had this option and removed it — see ADR-001.)
 
 ### Option 3 — Tenant-side scoping only (token permissions)
 
@@ -109,11 +109,11 @@ Decision is confirmed when:
 ## More Information
 
 - YouTrack permanent token permission model: <https://www.jetbrains.com/help/youtrack/devportal/Manage-Permanent-Token.html>.
-- Equivalent decision in Jira (allowlist retained): `docs/components/connectors/task-tracking/jira/specs/PRD.md` §3.1 `jira_project_keys`.
+- Jira converged on the same approach (ADR-001): `docs/components/connectors/task-tracking/jira/specs/ADR/ADR-001-auto-project-discovery.md`.
 - 6-month revisit horizon: Insight platform engineering review (Q3 2026 retrospective).
 
 ## Traceability
 
 - Implements DESIGN `cpt-insightspec-constraint-youtrack-no-whitelist`.
 - Pairs with Connector ADR-001 (project-scoped custom fields) — the per-project substream fans out across every project the token reaches.
-- Differs from the equivalent Jira decision (which kept a `jira_project_keys` allowlist) — documented divergence under PRD `cpt-insightspec-principle-youtrack-symmetry-with-jira`.
+- Jira ADR-001 documents the same decision for Jira (converged from the prior `jira_project_keys` allowlist approach).
@@ -10,8 +10,8 @@ Extracts projects, users, issues, issue history (changelog), comments, worklogs,
 ## Prerequisites
 
 1. Generate an Atlassian API token at [id.atlassian.com/manage-profile/security/api-tokens](https://id.atlassian.com/manage-profile/security/api-tokens).
-2. Use the email address of the Atlassian account that has **Browse Projects** on every target project.
-3. Identify the project keys to sync (e.g. `TC`, `TNG`) — visible in any issue URL as the prefix before the hyphen. Jira Cloud rejects unbounded JQL queries, so this is **required**.
+2. Use the email address of the Atlassian account that has **Browse Projects** on the target projects.
+3. **Scope the token via Jira permissions, not via connector config.** The connector automatically discovers all projects accessible to the token. If you want to limit ingestion to a subset of projects, restrict the token's Browse Projects permission in Jira to only those projects.
 
 ## K8s Secret
 
@@ -31,7 +31,6 @@ stringData:
   jira_instance_url: "https://myorg.atlassian.net"
   jira_email: "user@example.com"
   jira_api_token: "CHANGE_ME"
-  jira_project_keys: "PROJ1,PROJ2"
   # jira_start_date: "2024-01-01"   # optional, default = 2020-01-01
 ```
 
@@ -42,9 +41,10 @@ stringData:
 | `jira_instance_url` | Yes | Jira Cloud URL, no trailing slash (e.g. `https://myorg.atlassian.net`) |
 | `jira_email` | Yes | Atlassian account email for Basic Auth |
 | `jira_api_token` | Yes | Atlassian API token. Marked `airbyte_secret: true` — never logged |
-| `jira_project_keys` | Yes | Comma-separated project keys (e.g. `TC,TNG`). Jira Cloud rejects unbounded JQL queries |
 | `jira_start_date` | No | Earliest date to sync issues from, `YYYY-MM-DD`. Default `2020-01-01` |
 
+> **Project scope**: The connector discovers all projects accessible to the API token automatically (see ADR-001). To limit ingestion to specific projects, restrict the token's Browse Projects permission in Jira rather than configuring an allowlist here.
+
 ### Automatically injected
 
 These fields are added to every record by the connector — do **not** put them in the K8s Secret:
@@ -94,7 +94,7 @@ The `jira_boards` stream (`GET /rest/agile/1.0/board`) is the substream parent f
 
 - **Auth**: Basic Auth with email + API token. Missing/invalid token → HTTP 401; Jira project-level permission failures → 403. Both halt the run.
 - **Rate limits**: Atlassian caps per-user and per-IP API calls. The connector honours `Retry-After` on HTTP 429 and 503 (both used by Atlassian for throttling) with backoff.
-- **JQL scope**: `jira_project_keys` is required; Jira Cloud rejects unbounded queries (`project != EMPTY`) with an error.
+- **JQL scope**: Projects are discovered automatically via `SubstreamPartitionRouter` (ADR-001). Each project is queried as `project = "<KEY>" AND updated >= "..."`. Scope is controlled through the API token's Browse Projects permissions in Jira.
 - **Custom fields**: all custom fields are preserved in `jira_issue.custom_fields_json` for downstream dbt extraction.
 
 ## Related