fix(ingestion): heal positional schema drift in class contracts - #1784
Conversation
dbt-clickhouse incremental inserts map the model SELECT to table
columns positionally, and union_by_tag unions members with SELECT *,
so physical column order must equal model order. Three families
diverged:
- ai: label columns were patched onto pre-existing tables via
tail-appending ALTERs while the models declare them mid-SELECT;
syncs fail (CANNOT_PARSE_TEXT on staging inserts, NO_COMMON_TYPE on
the class union)
- collab chat: direct_and_group_messages was added mid-SELECT with
append_new_columns and no rebuild; same drift latent on pre-existing
member tables, and the class table never receives the column at all
- crm: salesforce members project custom_fields, hubspot members do
not; the class union fails whenever both connectors are configured
Fixes:
- derive ai tool/surface labels in gold from the discriminator codes
(macros/ai_labels.sql); drop the columns from the silver contract,
the label backfill migration, the staging repair functions, and the
per-deploy mutations they re-ran
- pin data columns (conversation_count, direct_and_group_messages,
custom_fields) to their model position with idempotent
ADD/MODIFY ... AFTER heals in the migrate hook, guarded for absent
tables and placeholders
- add custom_fields as a structural '{}' to the hubspot models at the
salesforce position; heal hubspot staging and crm class tables
- bound the table-materialized gold builds (query_settings: memory,
threads, disk spill) so deploy-time rebuilds degrade to disk instead
of failing the release
Every damaged instance converges on its next deploy + sync with no
manual intervention.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Aleksandr Barkhatov <pm@aleks.bar>
|
Warning Review limit reached
Next review available in: 26 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughAI usage models now emit canonical discriminator codes and derive labels through shared macros. AI staging schemas remove denormalized labels, HubSpot models add structural ChangesAI contract and label normalization
Gold build settings
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Aleksandr Barkhatov <pm@aleks.bar>
2 participants
Problem
dbt-clickhouse incremental inserts map the model SELECT to table columns positionally, and
union_by_tagunions members withSELECT *. Physical column order must equal model order. Three families diverged:CANNOT_PARSE_TEXTon staging inserts,NO_COMMON_TYPEon the class union.direct_and_group_messagesadded mid-SELECT withappend_new_columns, no rebuild. Same drift latent on pre-existing member tables; the class table (defaulton_schema_change='ignore') never receives the column at all.custom_fields, hubspot members do not. Class union fails withNUMBER_OF_COLUMNS_DOESNT_MATCHwhenever both connectors are configured.Fix
tool_label/surface_labelare constants derived fromtool/surface; the gold view now derives them viamacros/ai_labels.sql(same pattern asgit_source_label). Deletes the label backfill migration, both staging repair functions, and themutations_sync=2mutations they re-ran on every deploy.ADD/MODIFY COLUMN ... AFTERheals in the migrate hook +20260716000000_class_contract_heal.sql:conversation_count(ai),direct_and_group_messages(collab chat),custom_fields(crm). Guarded for absent tables and placeholders (the minimal gold-view placeholders lack the AFTER anchors).'{}'custom_fieldsat the salesforce position, in every positional branch.query_settings(memory, threads, disk spill) on both table-materialized observation models — deploy-time rebuilds degrade to disk spill instead of failing the release.Every damaged instance converges on its next deploy + sync. No manual intervention, no knowledge of per-instance damage state needed: DROP preserves relative order, ADD/MODIFY AFTER are metadata-only no-ops once converged.
Verification
dbt parseclean;bash -non both scripts.settingsvsquery_settings— table DDL settings would have rejected the memory bounds), fixed and re-verified against the adapter source. All heal anchors, types, and union branch parity verified against the model SELECTs.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
custom_fieldssupport to HubSpot CRM accounts, activities, contacts, deals, and users.Data Quality
Reliability