fix(jira): tag bronze_promoted 'staging' so prod promotes bronze before enrich (#1886)

[mitasovr]

Summary

Fixes #1886 — on a real Airbyte-synced install the jira pipeline crashes at the enrich step every run with Code: 181 … Storage MergeTree doesn't support FINAL (ILLEGAL_FINAL), so dbt silver/gold never runs and no jira/task metrics ever appear.

Root cause

The prod jira pipeline's staging dbt step uses the selector tag:staging,tag:jira (an AND-intersection, hardcoded in reconcile-connectors/python/render_cronworkflow.py and render_sync_trigger.py). The MergeTree → ReplacingMergeTree promotion lives in the jira__bronze_promoted model, which was tagged only ['jira'].

Because the selector requires both tags and has no + (so no upstream pull-in), it never selected the promote model. On a real sync bronze therefore stayed a plain MergeTree, and the enrich step's FROM bronze_jira.jira_issue AS ji FINAL is illegal on MergeTree in ClickHouse 25.7 → enrich exits 1 → pipeline aborts before silver/gold.

schema='staging' on the model sets the target database, not a dbt tag, so it never participated in tag selection.

This was masked everywhere we test:

• bootstrap-db (dev/CI) selects the promote model by name (--select jira__bronze_promoted).
• The e2e rig builds +<staging> (the + pulls the promote in as an upstream) and seeds silver directly, bypassing the Rust enrich binary.

So "works in dev" was never evidence the real Airbyte path worked.

Fix

Add the staging tag: tags=['jira', 'staging']. The existing prod staging step now selects the promote model and runs it (before enrich, via the depends_on edges the other staging models already declare). Minimal and aligns prod with the bootstrap-db contract.

jira is the only connector with an enrich step (verified: only connectors/task-tracking/jira has images.enrich / an enrich/ dir). Non-enrich connectors use the legacy tag:<connector>+ path, which already pulls in their <connector>__bronze_promoted, so no other connector needs this change. tag:staging is never used bare anywhere — only inside tag:staging,tag:jira — so this has no side effects elsewhere.

Verification (ClickHouse 25.7.5)

1. dbt ls --select tag:staging,tag:jira: before = 3 models (no promote); after = includes jira__bronze_promoted. Select-by-name (bootstrap-db) still works.
2. Live repro: seeded bronze_jira.jira_issue as plain MergeTree, ran the exact enrich query → Code: 181 ILLEGAL_FINAL. Ran dbt run --select jira__bronze_promoted → migrated to ReplacingMergeTree. Re-ran the FINAL query → succeeds and dedups (3 rows → 2 issues, latest version kept). Idempotent on re-run.

Regression test

Added a manifest-level meta test asserting the prod staging selector tag:staging,tag:jira selects jira__bronze_promoted. A metric e2e test can't cover this (the rig bypasses enrich and uses a masking selector), so a manifest-level guard is the correct layer.

Follow-up (out of scope)

render_cronworkflow.py / render_sync_trigger.py hardcode the staging selector as "tag:staging,tag:jira" if connector == "jira" else "". A second enrich connector would silently get "" (no staging step). Deriving the staging selector from the connector descriptor would prevent a future recurrence — worth a separate issue.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Fixed Jira staging pipeline selection so bronze promotion models are included correctly.
  • Prevented downstream failures caused by Jira bronze tables not being promoted to the expected table type.

• Tests

  • Added regression coverage to verify Jira staging models are selected as expected.
  • Improved validation of dbt selector behavior and worker-context variables.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4650ffe8-068a-4441-af3e-3ae3898f9ee6

📥 Commits

Reviewing files that changed from the base of the PR and between e11c2b9 and 399040c.

📒 Files selected for processing (2)

• src/ingestion/connectors/task-tracking/jira/dbt/jira__bronze_promoted.sql
• src/ingestion/tests/e2e/meta/test_dbt_runner.py

---

📝 Walkthrough

Walkthrough

The Jira bronze promotion model now carries the staging tag, and dbt runner coverage verifies that the combined staging/Jira selector includes it. Unused test imports and local assignments were removed.

Changes

Jira staging promotion

Layer / File(s)	Summary
Add staging tag to bronze promotion src/ingestion/connectors/task-tracking/jira/dbt/jira__bronze_promoted.sql	The model tags now include jira and staging, with documentation describing the selector requirement.
Verify staging selector coverage src/ingestion/tests/e2e/meta/test_dbt_runner.py	A regression test checks that tag:staging,tag:jira selects jira__bronze_promoted; unused imports and local assignments were removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• constructorfabric/insight#1421: Updates Jira staging tag selection and relates directly to this model’s selector behavior.
• constructorfabric/insight#1844: Adds the staging tag to another Jira dbt model required by the same selector.

Suggested reviewers: aleksdotbar

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly states the main change: adding the staging tag to jira bronze promotion for production ordering.
Linked Issues check	✅ Passed	The changes add the staging tag and a regression test so the production selector includes jira__bronze_promoted, addressing the failure in #1886.
Out of Scope Changes check	✅ Passed	The import cleanup and local variable removals are minor supporting changes and remain within the Jira promotion fix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}