fix(reconcile): jira staging dbt selector matched zero models (tag:jira-staging) - #1421
Conversation
…ra-staging)
The jira enrich pipeline runs three dbt phases: staging -> enrich -> silver.
Both renderers (render_sync_trigger.py, render_cronworkflow.py) hardcoded
the staging selector as `tag:jira-staging` for jira. No model carries that
tag — the staging models jira__changelog_items and jira__issue_field_snapshot
are tagged ['staging', 'jira']. So `dbt run --select tag:jira-staging`
selected zero nodes, exited 0 ("Nothing to do"), and the step went green
while building nothing.
Effect on virtuozzo: nightly jira syncs were green end to end (sync, enrich,
silver all Succeeded) but the staging tables were never rebuilt —
staging.jira_issue_field_snapshot stayed at 0 rows, jira_changelog_items
frozen, and the downstream silver (class_task_field_history, ephemeral off
changelog_items) frozen at the date of the last MANUAL run. Manual
run-sync.sh uses `tag:jira` for staging, which is why local/manual runs
populated the tables but reconcile-rendered cronworkflows did not.
Fix: select `tag:staging,tag:jira` (intersection) in both renderers —
exactly the two jira staging models, nothing broader. Verified the tags on
the models and that no model is tagged `jira-staging`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughTwo Python render scripts ( ChangesJira Staging Selector Fix
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… SELECT * (#1425) The model built one row per (issue, field) from bronze_jira.jira_issue. Its dedup subquery was: SELECT * FROM jira_issue ORDER BY _airbyte_extracted_at DESC LIMIT 1 BY source_id, jira_id `SELECT *` carries custom_fields_json (~2 MB/row, since the connector keeps all fields) through the ORDER BY buffer. On the virtuozzo instance (~140k issues) that blew past ClickHouse's query memory limit: Code: 241 ... (total) memory limit exceeded: would use 7.20 GiB ... MEMORY_LIMIT_EXCEEDED (model jira__issue_field_snapshot) So the model errored every run and staging.jira_issue_field_snapshot stayed at 0 rows — which fails the jira staging dbt step, so enrich + silver never run and the jira silver layer stays frozen. (Until #1421 this was masked: the staging selector matched no models, so it never even attempted to build.) Fix: - Project the small extracted columns in the same SELECT that does ORDER BY / LIMIT 1 BY, instead of `SELECT *` over a subquery. The 2 MB JSON is read per row to compute the projections but never enters the sort buffer. - Add max_bytes_before_external_sort / _group_by (2 GB) to the model settings as a spill safety net for the remaining heavy jira staging work. Verified against the live virtuozzo bronze with max_memory_usage capped at 4 GiB (well under the 7.2 GiB that previously failed): the full model (dedup + 10x UNION ALL) completes and returns 1,404,780 rows (140,478 issues x 10 fields), all unique_key distinct. Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Roman Mitasov <Roman.Mitasov@constructor.tech> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Problem
On virtuozzo, nightly jira syncs were green end to end (sync → enrich → silver all
Succeeded) yet the jira dbt data was frozen for ~2 weeks:staging.jira_issue_field_snapshot= 0 rowsstaging.jira_changelog_itemsfrozen (2.9M, last touched by a manual run)silver.class_task_field_history(ephemeral, reads changelog_items) frozen at the last manual run's datewhile
bronze_jira.*was fresh daily.Root cause
The jira enrich pipeline runs three dbt phases: staging → enrich → silver. Both renderers —
render_sync_trigger.pyandrender_cronworkflow.py— hardcode the staging selector astag:jira-stagingfor jira:No model carries the tag
jira-staging. The two jira staging models are tagged['staging', 'jira']:jira__changelog_items.sqljira__issue_field_snapshot.sqlSo
dbt run --select tag:jira-stagingselects zero nodes, exits 0 with "Nothing to do", and the step goes green while building nothing. enrich and silver then run on empty/stale staging.Manual
run-sync.shusesDBT_SELECT_STAGING="tag:jira"— which is why manual/local runs populated the tables but reconcile-rendered cronworkflows never did.Fix
Select
tag:staging,tag:jira(intersection) in both renderers — exactly the two jira staging models, nothing broader. Verified against the models' actual tags and that no model is taggedjira-staging.Rollout
App-repo change → toolbox build → reconcile re-renders the cronworkflows with the corrected selector. After that the nightly staging step rebuilds changelog_items + issue_field_snapshot, and the downstream silver unfreezes.
🤖 Generated with Claude Code
Summary by CodeRabbit