fix(jira): bronze promotion gaps, comment bodies nulled on ingest, staging dedup by the natural key

[mitasovr]

Closes part of #2609. Also removes the artifact tracked in #742.

Bronze promotion gap

Airbyte writes bronze tables as plain MergeTree with destinationSyncMode='append', and jira__bronze_promoted migrates each one to ReplacingMergeTree(_airbyte_extracted_at) ordered by unique_key. Five tables were never in that list:

• bronze_jira.jira_boards — promoted with a NULL-key filter, see the boards section below
• bronze_jira.jira_statuses
• bronze_jira.jira_issuetypes
• bronze_jira.jira_priorities
• bronze_jira.jira_resolutions

All five are full-refresh streams, so every sync re-appends the whole set. On a plain MergeTree nothing ever collapses those copies: background merges do not deduplicate, and FINAL raises Storage MergeTree doesn't support FINAL (#1886). The row count therefore grows without bound relative to the number of distinct entities, and any reader that does not deduplicate by hand multiplies its result by the number of syncs that have run.

The four catalogues already carry unique_key end to end and take the same order_by='unique_key'. The committed DDL snapshot captures post-dbt state, so their engines flip to ReplacingMergeTree there as well.

Boards: undeclared AddFields columns

CI caught a third instance of the AddFields-versus-schema mismatch class (the first two: body declared as an object, and benign number/string coercions). The boards stream computes tenant_id/source_id/unique_key/collected_at, but its inline schema never declared them — and the destination materialises columns from the catalog schema, so the values were silently dropped and the bronze table has no unique_key column at all.

Seeing the defect and the fix on an instance:

-- BEFORE: the AddFields columns never reached bronze — this returns only the
-- Airbyte columns and the raw API fields, with no unique_key / tenant_id /
-- source_id / collected_at among them.
SELECT name FROM system.columns
WHERE database = 'bronze_jira' AND table = 'jira_boards' ORDER BY position;

-- BEFORE: with no key, duplication is unbounded and unfixable — the same
-- board reappears once per sync and nothing can collapse the copies
-- (count() grows every sync while uniqExact(id) stays flat; FINAL is not
-- even applicable: Storage MergeTree doesn't support FINAL).
SELECT count() AS rows, uniqExact(id) AS boards, round(count() / uniqExact(id), 1) AS copies_per_board
FROM bronze_jira.jira_boards;

-- AFTER the deploy and the next scheduled sync: the columns exist, every
-- fresh row carries the key, and the engine dedups — one row per board.
SELECT count() AS rows_final, uniqExact(unique_key) AS boards
FROM bronze_jira.jira_boards FINAL
WHERE unique_key IS NOT NULL;
-- expected: rows_final = boards, and the engine line shows ReplacingMergeTree:
SELECT engine, sorting_key FROM system.tables
WHERE database = 'bronze_jira' AND name = 'jira_boards';

This PR declares the four fields in the schema, and promotes jira_boards with a guard: the promotion macro gains an optional CTAS filter, and boards passes where='unique_key IS NOT NULL'. Rows synced before the schema fix carry a NULL key, and ReplacingMergeTree treats NULL keys as equal — unfiltered, they would collapse into one phantom row. Filtering them away loses nothing: the stream is full refresh, so the first post-deploy sync rewrites the complete set with keys populated, and promotion runs in the workflow after the sync step, by which point the column exists. No state reset is needed for boards — unlike jira_comments below, every sync is already a full run.

Dropping the substream-parent artifact

jira_sprints is a substream and needs a parent stream to enumerate board ids. Reconcile auto-selects every stream that discover reports (ADR-0015), so a parent visible in the catalog materialises as a real bronze table — the leading underscore in the name is a convention and does not prevent it.

The current manifest declares that parent inline under parent_stream_configs[].stream, which keeps it out of the catalog entirely. bronze_jira._boards is what an earlier shape left behind: nothing writes to it, no model or script in the repository references it, and board data lives in bronze_jira.jira_boards. The migration drops it.

Comment bodies nulled by the destination

The comments stream declares body as an object, while the AddFields transformation writes the ADF document into that same path as a JSON string. The destination's typing step cannot serialize a string into a declared object, so it nulls the field and records this in _airbyte_meta:

{"changes":[{"field":"body","change":"NULLED","reason":"DESTINATION_SERIALIZATION_ERROR"}]}

Every row whose body is null carries that marker and no other row does. The bodies are still served by the API, so this is loss on the way in, and it recurs on every sync rather than being a one-off.

The three sibling projections that hand a tojson string to the same mechanism — jira_issue.custom_fields_json, jira_issue_history.items, jira_worklogs.comment — all declare string and carry no serialization errors. Declaring body the same way matches what the connector actually produces.

Dedup by the natural key, not the row id

jira__task_comments and jira__task_worklogs read append-only bronze through ORDER BY _airbyte_extracted_at DESC LIMIT 1 BY _airbyte_raw_id. _airbyte_raw_id is unique per physical row, so that LIMIT 1 BY removed nothing and every re-appended copy reached staging. Both now key on unique_key, which is what the ordering was written for.

This also gates recovery of the nulled bodies. A re-sync appends a fresh copy of each comment; with the old dedup both copies reach staging, _version is now64(3) and is evaluated once per query, so the ReplacingMergeTree tie-break between them is arbitrary and can keep the row whose body was nulled. With dedup on unique_key only the newest bronze row per comment reaches staging.

Rolling out on a warm instance

Deploying the fix stops the loss for new and edited comments, but does not bring back the bodies already nulled: the comments stream is incremental on updated and its parent carries incremental_dependency: true, so a comment nobody touches is never re-fetched. Recovery needs a state reset, in this order:

1. Deploy this change first.
2. Reset the connection state for jira_comments (its parent's cursor goes with it — incremental_dependency), then run a sync.
3. Rebuild the jira staging models and their silver targets.

The order matters because of the dedup fix above: resetting state before the new dedup lands would put both copies of every comment — the nulled one and the re-fetched one — into staging, where the now64(3) version gives them an identical tie-break and the engine may keep the nulled copy.

Bronze needs no cleanup: it is append-only ReplacingMergeTree keyed by unique_key with _airbyte_extracted_at as the version, so the re-fetched row wins any correct read and the stale one is swept by a background merge.

Verification after the re-sync — this must return zero:

SELECT count() AS still_lost
FROM (
    SELECT unique_key, argMax(body, _airbyte_extracted_at) AS latest_body
    FROM bronze_jira.jira_comments
    GROUP BY unique_key
)
WHERE latest_body IS NULL

Notes for review

• This does not make the data-quality register's count()-versus-unique_key row disappear, and it is not meant to. That check reads without FINAL, and a ReplacingMergeTree collapses duplicates only during background merges, so the row reflects merge lag once the engine is right. What the promotion fixes is the case underneath it: on plain MergeTree nothing collapses at all and FINAL errors outright. See Jira data-quality register: checks that cannot reach zero, and one that does not test what it reports #2609 for the register side.

• The promotion macro documents a race: rows inserted between its CREATE and EXCHANGE land in the copy it drops. The first promotion of these five tables should land when no Jira sync is active.

• The migration channel has no ledger and re-runs on every deploy, so it uses DROP TABLE IF EXISTS.

• dbt parse was not usable as a check here: the dbt available locally is dbt-fusion 2.0 preview, which fails on ~900 pre-existing test-argument deprecations across the project. No error referenced the changed model.

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a2b97945-998e-4e77-b860-babb5ea78435

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}