fix(dbt): align staging contributor types with their silver union targets by mitasovr · Pull Request #2101 · constructorfabric/insight

mitasovr · 2026-07-31T08:40:54Z

Why

The staging -> silver field-parity audit (#2080) fails on 13 type divergences. A silver class_* table is a positional UNION ALL of its staging contributors, so each divergence either silently changes the published silver type depending on which connectors are enabled, or coerces values at the insert boundary.

What

13 failures = 5 root causes; every fix is a one-line cast in the outlier staging branch, toward the type the data-carrying branch already publishes:

Root cause	Models	Fix
`commit_order` is a literal `0` (the APIs provide no ordering) which ClickHouse types as `UInt8`; bitbucket emits `Int64` from real data	github-v2, gitlab	`toInt64(0)`
`custom_fields` passed through as `Nullable(String)` while `heal_crm_table` ALTERs silver to `String DEFAULT '{}'` — NULLs were coerced to the default at insert time via `insert_null_as_default`	5 salesforce crm models (6 branches: activities has Task + Event)	`coalesce(custom_fields, '{}')`, matching hubspot's literal
`visited_page_count`: bronze carries the JSON `number` as `Nullable(Decimal(38, 9))`, passed through; onedrive branch emits `Nullable(Int64)`. A page count is integral	m365 sharepoint	`CAST(... AS Nullable(Int64))`
`close_date` narrowed to `Date` by `toDate()` while salesforce bronze is natively `Nullable(Date32)`	hubspot deals	`toDate32(...)`
`hire_date`/`termination_date`: constant-NULL placeholders typed `Nullable(Date)` vs the `Nullable(DateTime)` the data-carrying branches (bamboohr, workday) emit	ms-entra, active-directory	retype the NULL casts; zero rows affected by construction

The regenerated snapshot carries the visited_page_count type change and also normalizes silver.contract_version to dump-ddl.sh's statement format — that entry was appended by hand (semicolon on the statement line, a shape the dumper never produces), so the next convergence run would have flagged it as drift.

Rollout

No full-refresh is required for correctness: silver schemas on warm clusters do not change, and every new write is compatible with the old staging column types (Int64 value 0 into UInt8, String into Nullable(String), Int64 into Decimal, Date32 into Date, NULL into NULL). Gold is untouched in all modes — the one silver type that changes on fresh builds (visited_page_count) has no readers in migrations, gold, or silver (checked), and apply-ch-migrations.sh re-CREATEs gold views on every deploy anyway.

To converge the declared types of already-deployed staging tables, run in any convenient window:

dbt run --full-refresh --select github__pull_requests_commits gitlab__pull_requests_commits \
  salesforce__crm_accounts salesforce__crm_activities salesforce__crm_contacts salesforce__crm_deals salesforce__crm_users \
  m365__collab_document_activity_sharepoint m365__collab_meeting_activity \
  claude_enterprise__ai_dev_usage copilot__ai_dev_usage cursor__ai_dev_usage \
  github__commits github__file_changes github__pull_requests gitlab__commits gitlab__file_changes gitlab__pull_requests \
  slack__collab_chat_activity zulip_proxy__collab_chat_activity zoom__collab_meeting_activity \
  hubspot__crm_deals hubspot__crm_users jira__task_comments

(Scoped exactly to these models — no +. The github/gitlab rebuilds stamp fresh _versions, so the next silver increment after the refresh is one-time heavier.)

Second commit: nullable-widening warnings (61 -> 15)

The audit also reports 61 warnings — branches declaring T where the target publishes Nullable(T) because another branch is nullable. The published type is correct (NULL = "this source cannot measure this", distinct from a measured zero), but it was a supertype accident dependent on the connector set. The second commit pins it: 47 explicit toNullable() casts across 17 models. Values untouched, and the re-dumped snapshot is byte-identical — silver already published these types.

Two of those were masked in CI until now: heal_collab_chat_table ALTERs slack/zulip/m365 staging after dbt, so slack.direct_and_group_messages only surfaced when the table was rebuilt from the model. The model now owns its type instead of relying on the heal.

The 15 remaining warnings are deliberate: data_source, day, jira field-metadata keys, status_id/status_name, collected_at, timestamp — mandatory-by-meaning columns where the nullable branch is the defect and the contract should narrow to NOT NULL. That is the opposite direction (silver narrowing + not_null tests), tracked for the silver→gold contract work.

Verification

From-scratch bootstrap on this branch (all connectors green, dbt PASS=218 / ERROR=0), then the field-parity audit from #2080:

0 failure(s) (none), 15 warning(s), 0 unchecked, 39 union target(s), 273 relation(s) in the warehouse

— down from 13 failures and 61 warnings. The touched staging models were rebuilt with --full-refresh locally before re-auditing, and the snapshot was re-dumped after both commits.

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved consistency for numeric fields, including page visit counts, commit ordering, activity metrics, and code-change statistics.
- Preserved missing values across collaboration, CRM, Git, AI usage, and task-tracking data through consistent nullable field handling.
- Standardized date and datetime handling for deal and employee lifecycle dates.
- Ensured Salesforce custom fields return an empty JSON object instead of null when no values are available.
- Updated page visit count storage to use an integer type for more reliable reporting.

…gets The staging -> silver field-parity audit (#2080) fails on 13 type divergences across 5 root causes. A silver class table is a positional UNION ALL of its staging contributors, so each divergence either widens the published silver type depending on which connectors are enabled, or coerces values at the insert boundary. All fixes cast the outlier branch toward the type the data-carrying branch already publishes; silver schemas on warm clusters do not change (every write stays compatible), so no full-refresh is required for correctness — only for the deployed staging tables' declared types to converge, which can happen in any convenient window: dbt run --full-refresh --select github__pull_requests_commits \ gitlab__pull_requests_commits salesforce__crm_accounts \ salesforce__crm_activities salesforce__crm_contacts \ salesforce__crm_deals salesforce__crm_users \ m365__collab_document_activity_sharepoint * commit_order (github, gitlab): the models emit a literal `0` (the APIs provide no ordering), which ClickHouse types as UInt8; bitbucket emits Int64 from real data. toInt64(0) pins the branch to the contract type. * custom_fields (5 salesforce models, 6 branches): passed through from bronze as Nullable(String) while heal_crm_table ALTERs the silver tables to `String DEFAULT '{}'` — NULLs were being coerced to the default at insert time via insert_null_as_default. coalesce makes the '{}' fallback explicit in the model, matching hubspot's literal. * visited_page_count (m365 sharepoint): bronze carries the JSON `number` as Nullable(Decimal(38, 9)) and the model passed it through; the onedrive branch emits Nullable(Int64). A page count is integral — cast to Int64. Nothing downstream reads the column (checked migrations, gold, silver), so the silver type change on fresh clusters is safe. * close_date (hubspot deals): toDate() narrowed to Date while salesforce bronze is natively Nullable(Date32). toDate32 matches the wider type. * hire_date/termination_date (ms-entra, active-directory): constant-NULL placeholders typed Nullable(Date) against the Nullable(DateTime) the data-carrying branches (bamboohr, workday) emit. Retype the NULLs; zero rows are affected by construction. The regenerated snapshot carries the visited_page_count type change and also normalizes silver.contract_version to dump-ddl.sh's statement format — that entry was appended by hand with the semicolon on the statement line, a shape the dumper never produces, so the next convergence check would have flagged it as drift. Verified on a from-scratch bootstrap (all connectors green, dbt PASS=218/ERROR=0): the field-parity audit reports 0 failures, down from 13, across 39 union targets and 273 relations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>

coderabbitai · 2026-07-31T08:41:12Z

📝 Walkthrough

Walkthrough

Connector dbt models now use aligned nullable numeric, string, and datetime types. Salesforce models default null custom_fields values to '{}'. Git metrics and AI usage fields preserve nullable output types. The silver DDL reflects the updated document activity type.

Changes

Connector schema normalization

Layer / File(s)	Summary
Collaboration activity schema alignment `src/ingestion/connectors/collaboration/...`, `src/ingestion/scripts/connectors-ddl/silver.sql`	Collaboration models now emit nullable fields. `visited_page_count` uses nullable `Int64` in the model and silver DDL.
CRM output normalization `src/ingestion/connectors/crm/...`	HubSpot uses `Date32` and nullable status fields. Salesforce `custom_fields` values default to `'{}'` when null.
AI and activity metric nullability `src/ingestion/connectors/ai/...`, `src/ingestion/connectors/collaboration/...`	AI usage counts and activity metrics now use nullable numeric or string expressions.
Git metric and commit typing `src/ingestion/connectors/git/...`	Git change metrics use nullable values. `commit_order` defaults use explicit `Int64` values.
People and task field typing `src/ingestion/connectors/hr-directory/...`, `src/ingestion/connectors/task-tracking/jira/...`	HR hire and termination dates use nullable `DateTime` values. Jira `is_deleted` uses a nullable `UInt8` value.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

constructorfabric/insight#1861: Both PRs modify salesforce__crm_activities.sql for CRM field type normalization.
constructorfabric/insight#1892: Relates to connector model types and the generated silver DDL snapshot.

Suggested reviewers: ktursunov, aleksdotbar

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary change: aligning staging contributor types with their silver UNION ALL targets.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/field-parity-staging-types

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-31T08:44:22Z

⚠️ Regenerate the connectors-ddl snapshot

This PR changes src/ingestion/**. If your change affects any
bronze / silver / gold schema, regenerate the committed DDL snapshot
and include it in this PR.

Prerequisites (details: src/ingestion/scripts/bootstrap-db/README.md):

docker + a fresh throwaway ClickHouse 25.7.5 (README "Local ClickHouse for testing")
.env from .env.bootstrap.example pointing at it; use the host LAN IP,
reachable from both the host and connector containers
(host.docker.internal does not resolve on the macOS host itself)
python3.12 or python3.11 on PATH (pinned dbt venv)
HubSpot + Salesforce credentials in .env — their discover calls the
live APIs; without them, apply ../connectors-ddl/{hubspot,salesforce}.sql
(relative to bootstrap-db/) to seed their bronze, then run the dbt step

cd src/ingestion/scripts/bootstrap-db
set -a; source pins.env; source .env; set +a
./bootstrap-db.sh connectors-config.yaml   # fresh ClickHouse 25.7.5
./dump-ddl.sh                              # writes scripts/connectors-ddl/*.sql

Commit the resulting scripts/connectors-ddl/*.sql diff. If nothing
changed, no snapshot update is needed. (Regeneration is manual for now.)

The field-parity audit reports 61 nullable-widening warnings: a branch declares `T` where its silver union target publishes `Nullable(T)`, because some other branch of the same target is nullable. The published silver type is correct — NULL there means "this source cannot measure this" (bitbucket carries no diff stats, zoom cannot split meeting kinds) and must stay distinct from a measured zero — but it was a supertype accident, dependent on which connectors happen to be enabled. Pin it: every such branch now emits Nullable explicitly via toNullable(), so the silver type no longer depends on the connector set. 47 casts across 17 models (ai counters, m365 meeting counters, git files_changed/lines_added/lines_removed, chat/meeting user_name + email, crm and task boolean flags). Values are untouched — same numbers, same zeros; only the declared type widens. Two of these were previously masked in CI: heal_collab_chat_table ALTERs slack/zulip/m365 staging to the contract types after dbt runs, so slack.direct_and_group_messages only surfaced once the table was rebuilt from the model. The model now owns the type instead of relying on the heal. NOT aligned — left as warnings on purpose: data_source, day, insight_source_id/field_* (jira metadata), status_id/status_name, collected_at, timestamp (15 findings). Those are mandatory-by-meaning columns where the nullable branch is the defect and the contract should narrow to NOT NULL — the opposite direction, tracked for the silver→gold contract work rather than blanket-widened here. Verified against the same from-scratch warehouse: 0 failures, warnings 61 -> 15, and the re-dumped connectors-ddl snapshot is byte-identical — silver already published these Nullable types, so nothing downstream moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>

mitasovr requested a review from a team as a code owner July 31, 2026 08:40

cyberantonz approved these changes Jul 31, 2026

View reviewed changes

cyberantonz approved these changes Aug 3, 2026

View reviewed changes

mitasovr added this pull request to the merge queue Aug 3, 2026

Merged via the queue into main with commit 4c0a563 Aug 3, 2026
64 checks passed

mitasovr deleted the fix/field-parity-staging-types branch August 3, 2026 03:21

coderabbitai Bot mentioned this pull request Aug 5, 2026

feat(semantic): field catalog — types from ClickHouse + authored roles (#2208) #2211

Closed

Navigation Menu

Uh oh!

fix(dbt): align staging contributor types with their silver union targets - #2101

fix(dbt): align staging contributor types with their silver union targets#2101
mitasovr merged 2 commits into
mainfrom
fix/field-parity-staging-types

mitasovr commented Jul 31, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 31, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mitasovr commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Rollout

Second commit: nullable-widening warnings (61 -> 15)

Verification

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jul 31, 2026

⚠️ Regenerate the connectors-ddl snapshot

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mitasovr commented Jul 31, 2026 •

edited

Loading

coderabbitai Bot commented Jul 31, 2026 •

edited

Loading