fix(jira): tag staging feeders 'staging' so silver builds on a fresh install (#1893)

[mitasovr]

Summary

Fixes #1893 — on a fresh install the jira silver dbt step fails, so no jira/task metrics build. The reported crash is Database Error in model jira__users_fields_history … code: 60 … Unknown table expression identifier 'staging.jira__users_snapshot'.

This is the same root cause as #1886 (merged in #1889), on more models.

Root cause

Unlike other connectors — which build via the legacy single pass tag:<connector>+ (pulls every connector-tagged model plus its descendants) — the jira pipeline builds in three narrow tag-scoped passes with no tag:jira catch-all:

pass	selector	source
staging	tag:staging,tag:jira	render_cronworkflow.py / render_sync_trigger.py (no +)
silver	tag:silver,tag:jira+	jira descriptor.yaml dbt_select
gold	tag:gold,tag:jira+	jira descriptor.yaml dbt_select

Seven jira models were tagged only ['jira'] (or ['jira','silver:class_*'] — note silver:class_* is a routing tag, not the bare silver tag), so they match none of these passes and are never materialized on a real sync:

jira__users_snapshot
jira__task_comments  jira__task_projects  jira__task_sprints
jira__task_statuses  jira__task_users     jira__task_worklogs

Their silver consumers then fail:

• jira__users_fields_history ref()s jira__users_snapshot → hard code: 60 crash (the reported bug).
• The shared class_task_* silver models union their jira feeders via union_by_tag, which on a fresh install raises the "no source tables … target not materialised" compiler error, and on a mixed tenant silently drops all jira rows → no jira task metrics.

Fix

Add the staging tag to all seven (matching the already-correct jira__issue_field_snapshot / jira__task_field_metadata) so the staging pass materializes them before enrich and silver. They already carry -- depends_on: jira__bronze_promoted, so ordering is preserved.

jira__task_field_history is intentionally left alone: it is ephemeral, has no DB relation, and is inlined into its consumers via ref(), so no pass needs to build it standalone.

Scope check (other connectors)

Audited every connector with dbt ls against its real pipeline selector:

• jira: 7 gaps (fixed here). Post-fix, the only uncovered jira model is the ephemeral jira__task_field_history — correct.
• youtrack is NOT affected despite mirroring these models: it has no enrich step, so its pipeline uses the legacy tag:youtrack+ pass, which already pulls in every youtrack-tagged model. Confirmed via dbt ls --select tag:youtrack+.
• All other connectors use tag:<connector>+ → no gaps.

Verification (ClickHouse 25.7.5)

• Pre-fix: seeded bronze_jira.jira_user as MergeTree, built jira__users_fields_history → reproduced the exact code: 60 … Unknown table 'staging.jira__users_snapshot'.
• Post-fix: the staging pass materializes jira__bronze_promoted → jira__users_snapshot (dedup via bronze FINAL), then jira__users_fields_history builds → Completed successfully.

Regression test

Added test_every_jira_model_is_built_by_some_prod_pass: parses the manifest, replays the three prod passes (tag intersection + + descendants), and asserts every non-ephemeral jira model is selected by at least one. Validated: fails pre-fix (lists the 7 gaps), passes post-fix. A metric e2e test can't cover this — the rig builds +<staging> which masks the gap by pulling ancestors.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved Jira data pipeline coverage by ensuring staging models are included in production processing.
  • Reduced the risk of missing Jira data during fresh installations or scheduled runs.

• Tests

  • Added regression coverage to verify every Jira model is built by at least one production processing pass.

[coderabbitai]

📝 Walkthrough

Walkthrough

Jira dbt models now include the staging tag, including the users snapshot. A new regression test parses the dbt manifest and verifies that non-ephemeral Jira models are covered by the configured production dbt passes.

Changes

Jira staging coverage

Layer / File(s)	Summary
Add staging tags src/ingestion/connectors/task-tracking/jira/dbt/*	Adds the staging tag to seven Jira dbt models without changing their SQL or materialization behavior.
Validate production pass coverage src/ingestion/tests/e2e/meta/test_dbt_runner.py	Adds a regression test that evaluates production tag selectors and reports uncovered non-ephemeral Jira models.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

• constructorfabric/insight#1893 — Covers the missing Jira staging tags and adds production-pass coverage validation.

Possibly related PRs

• constructorfabric/insight#1421 — Uses the Jira staging selector that depends on these model tags.
• constructorfabric/insight#1844 — Makes a related Jira dbt tagging update.
• constructorfabric/insight#1889 — Adds related Jira tagging and selector regression coverage.

Suggested reviewers: aleksdotbar

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding the staging tag to Jira feeder models so silver builds succeed on fresh installs.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/ingestion/tests/e2e/meta/test_dbt_runner.py`:
- Around line 141-144: Update the path substring checked by _is_jira_model to
remove the leading slash, matching dbt-project-relative paths under
connectors/task-tracking/jira/dbt/ while preserving the existing resource_type
model check.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4a68cd48-b052-417e-8bce-fd2f298d3774

📥 Commits

Reviewing files that changed from the base of the PR and between c9cb954 and bdd995e.

📒 Files selected for processing (8)

• src/ingestion/connectors/task-tracking/jira/dbt/jira__task_comments.sql
• src/ingestion/connectors/task-tracking/jira/dbt/jira__task_projects.sql
• src/ingestion/connectors/task-tracking/jira/dbt/jira__task_sprints.sql
• src/ingestion/connectors/task-tracking/jira/dbt/jira__task_statuses.sql
• src/ingestion/connectors/task-tracking/jira/dbt/jira__task_users.sql
• src/ingestion/connectors/task-tracking/jira/dbt/jira__task_worklogs.sql
• src/ingestion/connectors/task-tracking/jira/dbt/jira__users_snapshot.sql
• src/ingestion/tests/e2e/meta/test_dbt_runner.py

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
fd -a '^manifest\.json$' . -E target -E node_modules | while read -r manifest; do
  echo "== $manifest"
  jq -r '.nodes[]?.original_file_path // empty' "$manifest" |
    grep 'task-tracking/jira/dbt/' | head -5
done

Repository: constructorfabric/insight

Length of output: 163

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== candidate test file =="
fd -a 'test_dbt_runner.py$' . | sed 's#^\./##'

FILE="$(fd 'test_dbt_runner.py$' src/ingestion/tests/e2e/meta -1 || true)"
if [ -n "${FILE:-}" ]; then
  echo "== outline for $FILE =="
  ast-grep outline "$FILE" || true
  echo "== relevant lines =="
  sed -n '110,170p' "$FILE" | cat -n
fi

echo "== dbt-related files =="
fd -a 'dbt_project\.ya?ml|manifest\.json|model' . -E target -E node_modules -E .venv | sed 's#^\./##' | head -200

echo "== Jira connector references =="
rg -n 'task-tracking/jira/dbt|connectors/task-tracking/jira/dbt|/connectors/task-tracking/jira/dbt' . -E target -E node_modules -E .venv || true

Repository: constructorfabric/insight

Length of output: 4476

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== test full context =="
sed -n '1,120p' src/ingestion/tests/e2e/meta/test_dbt_runner.py | cat -n

echo "== dbt_project.yml =="
cat -n src/ingestion/dbt/dbt_project.yml

echo "== files under ingestion/dbt containing jira or task-tracking =="
find src/ingestion/dbt -maxdepth 6 \( -type f -o -type d \) | grep -E 'jira|task-tracking' | sort | sed -n '1,200p'

echo "== search repository (no binary) for jira paths =="
python3 - <<'PY'
import os, re
root='src/ingestion/dbt'
for dirpath, dirs, files in os.walk(root):
    dirs[:] = [d for d in dirs if not d.startswith('.')]
    for f in files:
        if f.startswith('.'): 
            continue
        p=os.path.join(dirpath,f)
        if not os.path.isfile(p):
            continue
        try:
            text=open(p,'r',encoding='utf-8',errors='ignore').read()
        except Exception as e:
            print(f"SKIP {p}: {e}")
            continue
        if 'jira' in text.lower() or 'task-tracking' in text:
            print(p)
PY

echo "== dbt package/source references =="
python3 - <<'PY'
import os, re
root='src/ingestion/dbt'
pattern=re.compile(r'connectors/task-tracking/jira/dbt|/connectors/task-tracking/jira/dbt|task-tracking/jira/dbt', re.I)
for dirpath, dirs, files in os.walk(root):
    dirs[:] = [d for d in dirs if not d.startswith('.')]
    for f in files:
        if f.startswith('.'): continue
        p=os.path.join(dirpath,f)
        try:
            s=open(p,'rb').readline(1000)
        except Exception:
            continue
        if b'connections' in s.lower() or b'jira' in s.lower() or b'task-tracking' in s.lower() or b'model' in s.lower():
            try:
                txt=open(p,'r',encoding='utf-8',errors='ignore').read()
            except Exception:
                continue
            if pattern.search(txt):
                for i,line in enumerate(txt.splitlines(),1):
                    if pattern.search(line):
                        print(f"{p}:{i}:{line}")
PY

Repository: constructorfabric/insight

Length of output: 11111

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== all files under src/ingestion/dbt that reference task-tracking/jira/dbt or jira models =="
python3 - <<'PY'
import os, re

def ok(path):
    root = 'src/ingestion/dbt'
    return path.startswith(root + os.sep) and not any(part.startswith(('.', 'node_modules', 'target', '.venv', '__pycache__')) for part in path.split(os.sep))

root = 'src/ingestion/dbt'
patterns = [
    re.compile(r'connectors/task-tracking/jira/dbt'),
    re.compile(r'/connectors/task-tracking/jira/dbt'),
    re.compile(r'jira__', re.I),
    re.compile(r'task-tracking/jira/dbt', re.I),
]
matches = {}
for dirpath, dirs, files in os.walk(root):
    dirs[:] = [d for d in dirs if not d.startswith('.') and d not in ('node_modules', 'target', '.venv', '__pycache__')]
    for f in files:
        if f.startswith('.'):
            continue
        path = os.path.join(dirpath, f)
        try:
            text = open(path, 'r', encoding='utf-8', errors='ignore').read()
        except Exception as e:
            print(f"SKIP {path}: {e}")
            continue
        for p in patterns:
            if not p.search(text):
                continue
            matches.setdefault(path, []).append(p.pattern)
            break

for path in sorted(matches):
    print(f"== {path} ==")
    lines = open(path, 'r', encoding='utf-8', errors='ignore').readlines()
    hits = [(i+1, line.rstrip('\n')) for i, line in enumerate(lines) if any(re.search(p, line) for p in matches[path])]
    for line_no, line in hits[:40]:
        print(f"{line_no}: {line[:300]}")
PY

echo "== files under entire repo that reference Jira path pattern =="
rg -n 'connectors/task-tracking/jira/dbt|/connectors/task-tracking/jira/dbt' -g '!target/**' -g '!node_modules/**' -g '!dist/**' -g '!build/**' . || true

echo "== test after current snippet from src/ingestion/tests/e2e/meta/test_dbt_runner.py =="
sed -n '120,150p' src/ingestion/tests/e2e/meta/test_dbt_runner.py | cat -n

Repository: constructorfabric/insight

Length of output: 3968

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== lock/package references to Jira dbt package =="
rg -n 'task-tracking/jira/dbt|jira/dbt|connectors/task-tracking/jira/dbt|src/ingestion/connectors/task-tracking/jira/dbt' . \
  --glob '!target/**' --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' --glob '!*.lock' --glob '!*.json' \
| sort -u || true

echo "== dbt config/package files =="
git ls-files | rg '(^|/)(dbt_project\.yml|packages\.yml|package-lock\.json|packages-config\.yml|dbt_project_exclude|packages\.yml)$' | sed -n '1,200p'

echo "== src/ingestion/connectors task-tracking entries =="
find src/ingestion/connectors -maxdepth 3 -type f -o -type d | sort | sed -n '1,200p'

Repository: constructorfabric/insight

Length of output: 10010

---

Avoid the leading slash in the Jira-model check.

original_file_path is dbt-project-relative for this package, so paths under the Jira connector are connectors/task-tracking/jira/dbt/.... Keeping / makes the filter miss Jira models entirely.

Proposed fix

-            "/connectors/task-tracking/jira/dbt/" in node.get("original_file_path", "")
+            "connectors/task-tracking/jira/dbt/" in node.get("original_file_path", "")

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	def _is_jira_model(node: dict) -> bool:
	return node.get("resource_type") == "model" and (
	"/connectors/task-tracking/jira/dbt/" in node.get("original_file_path", "")
	)
	def _is_jira_model(node: dict) -> bool:
	return node.get("resource_type") == "model" and (
	"connectors/task-tracking/jira/dbt/" in node.get("original_file_path", "")
	)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingestion/tests/e2e/meta/test_dbt_runner.py` around lines 141 - 144,
Update the path substring checked by _is_jira_model to remove the leading slash,
matching dbt-project-relative paths under connectors/task-tracking/jira/dbt/
while preserving the existing resource_type model check.