Skip to content

fix(reconcile): jira staging dbt selector matched zero models (tag:jira-staging) - #1421

Merged
mitasovr merged 2 commits into
constructorfabric:mainfrom
mitasovr:claude/fix-jira-staging-selector
Jun 22, 2026
Merged

fix(reconcile): jira staging dbt selector matched zero models (tag:jira-staging)#1421
mitasovr merged 2 commits into
constructorfabric:mainfrom
mitasovr:claude/fix-jira-staging-selector

Conversation

@mitasovr

@mitasovr mitasovr commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Problem

On virtuozzo, nightly jira syncs were green end to end (sync → enrich → silver all Succeeded) yet the jira dbt data was frozen for ~2 weeks:

  • staging.jira_issue_field_snapshot = 0 rows
  • staging.jira_changelog_items frozen (2.9M, last touched by a manual run)
  • silver.class_task_field_history (ephemeral, reads changelog_items) frozen at the last manual run's date
    while bronze_jira.* was fresh daily.

Root cause

The jira enrich pipeline runs three dbt phases: staging → enrich → silver. Both renderers — render_sync_trigger.py and render_cronworkflow.py — hardcode the staging selector as tag:jira-staging for jira:

dbt_select_staging = "tag:jira-staging" if args.connector == "jira" else ""

No model carries the tag jira-staging. The two jira staging models are tagged ['staging', 'jira']:

  • jira__changelog_items.sql
  • jira__issue_field_snapshot.sql

So dbt run --select tag:jira-staging selects zero nodes, exits 0 with "Nothing to do", and the step goes green while building nothing. enrich and silver then run on empty/stale staging.

Manual run-sync.sh uses DBT_SELECT_STAGING="tag:jira" — which is why manual/local runs populated the tables but reconcile-rendered cronworkflows never did.

Fix

Select tag:staging,tag:jira (intersection) in both renderers — exactly the two jira staging models, nothing broader. Verified against the models' actual tags and that no model is tagged jira-staging.

Rollout

App-repo change → toolbox build → reconcile re-renders the cronworkflows with the corrected selector. After that the nightly staging step rebuilds changelog_items + issue_field_snapshot, and the downstream silver unfreezes.

Context: this is the dbt-layer half of the jira recovery on virtuozzo. The ingestion-layer fixes (substream cache #1283, concurrency #1308, descriptor bumps #1310, global cursor #1370, cursor formats #1383, airbyte-server memory in gitops) are separate and already landed/deployed; bronze is fresh — only the transform layer was stuck, on this selector.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Corrected data model selection logic for the Jira connector to ensure staging models are properly targeted during nightly synchronization workflows.

…ra-staging)

The jira enrich pipeline runs three dbt phases: staging -> enrich -> silver.
Both renderers (render_sync_trigger.py, render_cronworkflow.py) hardcoded
the staging selector as `tag:jira-staging` for jira. No model carries that
tag — the staging models jira__changelog_items and jira__issue_field_snapshot
are tagged ['staging', 'jira']. So `dbt run --select tag:jira-staging`
selected zero nodes, exited 0 ("Nothing to do"), and the step went green
while building nothing.

Effect on virtuozzo: nightly jira syncs were green end to end (sync, enrich,
silver all Succeeded) but the staging tables were never rebuilt —
staging.jira_issue_field_snapshot stayed at 0 rows, jira_changelog_items
frozen, and the downstream silver (class_task_field_history, ephemeral off
changelog_items) frozen at the date of the last MANUAL run. Manual
run-sync.sh uses `tag:jira` for staging, which is why local/manual runs
populated the tables but reconcile-rendered cronworkflows did not.

Fix: select `tag:staging,tag:jira` (intersection) in both renderers —
exactly the two jira staging models, nothing broader. Verified the tags on
the models and that no model is tagged `jira-staging`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
@mitasovr
mitasovr requested a review from a team as a code owner June 22, 2026 08:13
@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ee204017-4862-4762-a22b-9eac645acf01

📥 Commits

Reviewing files that changed from the base of the PR and between d0775ac and 66a1156.

📒 Files selected for processing (2)
  • src/ingestion/reconcile-connectors/python/render_cronworkflow.py
  • src/ingestion/reconcile-connectors/python/render_sync_trigger.py

📝 Walkthrough

Walkthrough

Two Python render scripts (render_cronworkflow.py and render_sync_trigger.py) each had a jira-connector branch that set the dbt staging selector to the non-existent tag tag:jira-staging. Both are corrected to use the intersection selector tag:staging,tag:jira.

Changes

Jira Staging Selector Fix

Layer / File(s) Summary
Fix DBT_SELECT_STAGING for jira connector
src/ingestion/reconcile-connectors/python/render_cronworkflow.py, src/ingestion/reconcile-connectors/python/render_sync_trigger.py
The jira-specific branch in both scripts now sets the staging dbt selector to tag:staging,tag:jira instead of the previously used tag:jira-staging (which matched no models). Non-jira connectors continue to receive an empty selector.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐇 Hippity-hoppity, tags gone astray,
jira-staging led models astray!
Now comma-joined, intersection neat,
staging and jira finally meet.
The rabbit fixed it — hip hip hooray! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main bug fix: correcting the jira staging dbt selector from the incorrect non-existent tag 'tag:jira-staging' to the correct intersection 'tag:staging,tag:jira'.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@mitasovr
mitasovr merged commit 3d4af6d into constructorfabric:main Jun 22, 2026
18 checks passed
mitasovr added a commit that referenced this pull request Jun 22, 2026
… SELECT * (#1425)

The model built one row per (issue, field) from bronze_jira.jira_issue. Its
dedup subquery was:

    SELECT * FROM jira_issue ORDER BY _airbyte_extracted_at DESC
    LIMIT 1 BY source_id, jira_id

`SELECT *` carries custom_fields_json (~2 MB/row, since the connector keeps
all fields) through the ORDER BY buffer. On the virtuozzo instance (~140k
issues) that blew past ClickHouse's query memory limit:

    Code: 241 ... (total) memory limit exceeded: would use 7.20 GiB ...
    MEMORY_LIMIT_EXCEEDED  (model jira__issue_field_snapshot)

So the model errored every run and staging.jira_issue_field_snapshot stayed
at 0 rows — which fails the jira staging dbt step, so enrich + silver never
run and the jira silver layer stays frozen. (Until #1421 this was masked: the
staging selector matched no models, so it never even attempted to build.)

Fix:
- Project the small extracted columns in the same SELECT that does
  ORDER BY / LIMIT 1 BY, instead of `SELECT *` over a subquery. The 2 MB JSON
  is read per row to compute the projections but never enters the sort buffer.
- Add max_bytes_before_external_sort / _group_by (2 GB) to the model settings
  as a spill safety net for the remaining heavy jira staging work.

Verified against the live virtuozzo bronze with max_memory_usage capped at
4 GiB (well under the 7.2 GiB that previously failed): the full model
(dedup + 10x UNION ALL) completes and returns 1,404,780 rows
(140,478 issues x 10 fields), all unique_key distinct.

Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
Co-authored-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants