fix(ingestion): stop silent jira sync hang via lightweight substream parent

[mitasovr]

Problem

jira syncs on large instances hang silently: the Airbyte job stays running with 0 records committed for hours, Argo poll times out (workflow Failed), and the orphaned replication pod keeps spinning. On the virtuozzo tenant this meant bronze_jira never received a single completed sync.

Root cause

The low-code CDK auto-caches a SubstreamPartitionRouter parent's HTTP responses in a SQLite requests-cache so child streams can reuse them. The parent of jira_issue_history / jira_comments / jira_worklogs was jira_issue, which requests fields=*all + expand=names. On an instance with ~1000 fields (44 system + 959 custom, mostly (migrated) duplicates), each /search/jql response is ~2 MB. The cache ballooned to ~226 MB after ~108 responses and the read stalled before emitting anything — diagnosed with an isolated read --debug run of the bare source (no destination/orchestrator), which reproduced the prod hang exactly.

Fix

Mirrors the official source-jira pattern (minimal-fields parent for partition enumeration, *all only on the terminal emitter):

• New lightweight stream jira_issue_keys: same /search/jql + JQL window, but fields=updated only (id/key arrive top-level for free, no expand).
• jira_issue_history, jira_comments, jira_worklogs repointed to it as their parent (#/streams/4 → #/streams/13).
• jira_issue stays the full-record emitter but is no longer a cached substream parent.

Since reconcile (ADR-0015) auto-selects every discovered stream, jira_issue_keys lands as a real (tiny: key + updated per issue) bronze table, so it:

• carries the standard identity stamp tenant_id / source_id / unique_key ({tenant}-{source}-{key}),
• hoists the updated cursor field to the top level (the search response nests it as fields.updated; jira_issue does the same hoist),
• is added to the promote_bronze_to_rmt list (jira__bronze_promoted.sql) so RMT dedup caps its growth. The macro skips missing tables, so ordering vs. first sync is safe.

Verification (isolated read runs against a live large instance)

	before	after
parent SQLite cache	226 MB, frozen at 108 responses	~4 KB
parent enumeration	stuck, 0 emitted	completes (9994 keys)
jira_issue_history	never starts	4210 records in a 2-day window, per-partition state advances
jira_issue_keys standalone	—	1579 records, unique_key=virtuozzo-jira-main-VSTOR-…, exit 0

No schema changes to existing bronze tables: children emit byte-identical records (id_readable comes from record['key'] in both old and new parent); dbt staging/silver untouched apart from the added promotion line.

Note: confluence has the same architecture (wiki_pages parents 3 substreams) and will get the same treatment in a follow-up.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added a lightweight Jira issue-key enumeration stream to improve cursor progress and incremental reads.
• Refactor
  • Substream partitioning for issue history, comments, and worklogs now uses the lightweight parent to reduce read stalls and cache growth.
• Maintenance
  • Bronze promotion updated to include the new issue-keys stream to help deduplicate and cap table growth.
• Documentation
  • Added guidance and troubleshooting on silent stalls, cursor hoisting, and validation rules for connector authors.

Also included

docs(skills): the /connector skill (cypilot/.core/skills/connector/) now encodes these lessons — forbidden heavy substream parents + the lightweight key-parent / inline-parent patterns, identity stamp & RMT promotion required on every top-level stream (reconcile auto-selects all), nested-cursor hoist rule, silent-stall diagnosis for read, and "never trust Airbyte job status — verify recordsCommitted / bronze freshness" ops guidance.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds a lightweight jira_issue_keys parent stream that emits only cursor/identity fields and hoists updated, re-points jira_issue_history, jira_comments, and jira_worklogs to use it for partitioning, promotes the new bronze table in dbt, and updates connector runbooks/workflows with cache and cursor-hoisting guidance.

Changes

Jira Issue Keys Parent Stream and Substream Re-parenting

Layer / File(s)	Summary
New jira_issue_keys parent stream definition src/ingestion/connectors/task-tracking/jira/connector.yaml	A new DeclarativeStream named jira_issue_keys is added with a DatetimeBasedCursor on updated, a JQL requester returning only fields.updated, transformations that hoist updated, stamp tenant/source/unique_key/id_readable/jira_id and collected_at, and an inline minimal schema.
Substream partition router re-parenting src/ingestion/connectors/task-tracking/jira/connector.yaml	jira_issue_history, jira_comments, and jira_worklogs have their SubstreamPartitionRouter parent stream reference changed from #/streams/4 to #/streams/13, altering the upstream source for per-issue partition keys.
dbt promotion for new jira_issue_keys stream src/ingestion/connectors/task-tracking/jira/dbt/jira__bronze_promoted.sql	bronze_jira.jira_issue_keys is added to promote_bronze_to_rmt(..., order_by='unique_key') in the promoted model.
Connector runbook and workflow docs cypilot/.core/skills/connector/*	Added guidance and validation rules warning against heavy SubstreamPartitionRouter parents, advising cursor hoisting for DatetimeBasedCursor, adding stall-diagnosis steps, and requiring identity stamping and bronze promotion lines in manifests.

Sequence Diagram

sequenceDiagram
  participant JiraAPI as Jira API
  participant IssueKeys as jira_issue_keys
  participant Substreams as jira_issue_history/comments/worklogs
  JiraAPI->>IssueKeys: JQL query (fields: updated)
  IssueKeys->>IssueKeys: Hoist `updated`, stamp identity, emit `unique_key`
  IssueKeys->>Substreams: Emit partitions (id_readable, unique_key)
  Substreams->>JiraAPI: Fetch per-issue payloads using partitions

Loading

🎯 4 (Complex) | ⏱️ ~45 minutes

🐰 I nibble keys light and neat,
hoist a timestamp, tidy the beat,
three old trails now follow my song,
small and steady—fast and strong. 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically identifies the main fix: introducing a lightweight substream parent to resolve silent Jira sync hangs caused by SQLite cache ballooning.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

cypilot/.core/skills/connector/SKILL.md (1)

57-61: ⚡ Quick win

Consider adding a brief example for checking aggregatedStats.recordsCommitted.

The guidance correctly identifies the real health signals (aggregatedStats.recordsCommitted and bronze freshness), but operators may not know how to retrieve aggregatedStats.recordsCommitted from the jobs API. Consider adding a one-line example or cross-referencing the Airbyte API endpoint (e.g., GET /api/v1/jobs/get → .attempts[].stats.recordsCommitted) to make the guidance immediately actionable.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cypilot/.core/skills/connector/SKILL.md` around lines 57 - 61, Add a one-line
example showing how to read aggregatedStats.recordsCommitted from the Airbyte
jobs API: reference the GET /api/v1/jobs/get endpoint and indicate the path
attempts[].stats.recordsCommitted (or jobs[].attempts[].stats.recordsCommitted /
.aggregatedStats.recordsCommitted as applicable) so operators can immediately
locate the metric; include this short example sentence near the paragraph that
recommends checking aggregatedStats.recordsCommitted and bronze freshness.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@cypilot/.core/skills/connector/SKILL.md`:
- Around line 57-61: Add a one-line example showing how to read
aggregatedStats.recordsCommitted from the Airbyte jobs API: reference the GET
/api/v1/jobs/get endpoint and indicate the path
attempts[].stats.recordsCommitted (or jobs[].attempts[].stats.recordsCommitted /
.aggregatedStats.recordsCommitted as applicable) so operators can immediately
locate the metric; include this short example sentence near the paragraph that
recommends checking aggregatedStats.recordsCommitted and bronze freshness.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 771fea94-3d49-4296-b465-561a0b1778a5

📥 Commits

Reviewing files that changed from the base of the PR and between d5388b0 and 1cd7878.

📒 Files selected for processing (4)

• cypilot/.core/skills/connector/SKILL.md
• cypilot/.core/skills/connector/workflows/create.md
• cypilot/.core/skills/connector/workflows/test.md
• cypilot/.core/skills/connector/workflows/validate.md

✅ Files skipped from review due to trivial changes (2)

• cypilot/.core/skills/connector/workflows/validate.md
• cypilot/.core/skills/connector/workflows/create.md