feat(ingestion): bootstrap-db — generate connectors-ddl snapshot [WIP] - #1876
Closed
mitasovr wants to merge 3 commits into
Closed
feat(ingestion): bootstrap-db — generate connectors-ddl snapshot [WIP]#1876mitasovr wants to merge 3 commits into
mitasovr wants to merge 3 commits into
Conversation
Replaces the hand-written scripts/create-bronze-placeholders.sh with a CI-generated DDL snapshot (scripts/connectors-ddl/*.sql) dumped from a real bootstrap-db run: connector discover -> destination-clickhouse write -> dbt -> SHOW CREATE. Unlike the hand-written placeholders the schemas come from the same source of truth as a real sync, so they cannot drift. Includes: - scripts/bootstrap-db/ tooling (generate config, discover->write, dump-ddl, pinned CH/destination/dbt versions in pins.env) - .github/workflows/connectors-ddl.yml — regenerates + auto-commits the snapshot on PRs (drift guard) - create-bronze-placeholders.sh rewritten as a snapshot applier - dev/test/CI ClickHouse bumped 24.8 -> 25.7.5 (prod parity; dev, virtuozzo and new instances all run 25.7.5) - ch_seeder string->numeric coercion; e2e migration_applier reads the snapshot - dbt contract fixes exposed by the real-schema snapshot (salesforce crm_deals Decimal->Float, m365 chat Decimal->Int, missing depends_on hints, etc.) WIP — snapshot is not yet complete on 25.7.5: see PR description for the two blocking dbt-model incompatibilities (bitbucket #1843 nested aggregates; identity seed correlated subqueries) and the pipeline ordering fix still needed. Closes #1831 once those land. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
added 2 commits
July 24, 2026 00:02
Bring in the bitbucket-cloud dbt illegal-aggregation fix (#1880) and other main changes. Resolved create-bronze-placeholders.sh in favour of the snapshot applicator (this PR's core change, #1831); upstream's warm-cluster legacy-bitbucket bronze reconcile is relocated to apply-ch-migrations heals in a follow-up change on this branch. Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
….7.5 Regenerate the full snapshot and fix the generation pipeline so a fresh cluster builds end-to-end (#1831, #1763). Generation / prod: - Order: create person/identity (init-identity) BEFORE dbt so the identity seed models' ANTI JOIN person.persons resolves; gold-view migrations after. - CH 25.7: set allow_experimental_correlated_subqueries at PROFILE level in all four dbt profile generators (config()-level does not reach the SELECT plan); cast the identity seed join key to UUID (String vs UUID NO_COMMON_TYPE). - dump-ddl also emits person/identity and the gold-referenced staging tables (only m365__collab_email_activity today) — NOT all of staging: pre-creating an incremental staging model empty poisons its first is_incremental build (max(date) FROM this over an empty MergeTree reads the Date type-max on 25.7). - SKIP_DBT_GOLD skips apply-ch-migrations' own gold build during generation (gold already built by the pinned dbt venv; avoids needing dbt on PATH). - Relocate the warm-cluster Bitbucket bronze reconcile (#1880) into the apply-ch-migrations heal section, now that create-bronze-placeholders is the snapshot applicator. e2e rig: - _ensure_table allows the real table to be a superset of the fixture schema; Decimal coercion + jira fixture schema/value alignment to the real bronze. - declarative `skip:`; git_metrics skipped pending a github-v2 fixture refresh. Snapshot: 53 gold views (incl git_metric_observations), all class_* incl git, person.persons. e2e green: metrics 34 passed/1 skipped, api 64 passed/12 xfail. The .github/workflows/connectors-ddl.yml that regenerates this snapshot in CI is committed separately (needs a token with `workflow` scope to push). Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Replaces the hand-written
create-bronze-placeholders.sh(ADR-0007) with agenerated DDL snapshot in
scripts/connectors-ddl/*.sql, dumped bybootstrap-dbfrom real connectors + dbt + migrations.create-bronze-placeholders.shis now a thin applicator of that snapshot, so a fresh cluster type-checks every
gold view without a hand-maintained placeholder list (#1831), and a fresh cluster
builds end-to-end (#1763). All environments are on ClickHouse 25.7.5 (test/compose
pins bumped to match).
Merges
upstream/main, including the bitbucket-cloud dbt fix (#1880).Generation pipeline / prod
person/identity(init-identity) are created before dbt so theidentity seed models'
ANTI JOIN person.personsresolves; gold-view migrations runafter dbt.
allow_experimental_correlated_subqueriesset at profile level in all four dbtprofile generators — a model
config()setting does not reach the SELECT plan.toUUID(...)(String vs UUIDNO_COMMON_TYPE).dump-ddlalso emitsperson/identityand only the gold-referenced stagingtables (today just
m365__collab_email_activity) — deliberately not all of staging:pre-creating an incremental staging model empty poisons its first
is_incrementalbuild (
max(date) FROM {{ this }}over an empty MergeTree reads the Date type-max on25.7 and excludes every row).
SKIP_DBT_GOLDlets generation skipapply-ch-migrations' own gold build (gold isalready built by the pinned dbt venv; avoids requiring
dbton PATH — absent /dbt-fusion outside the prod toolbox). Prod deploy leaves it unset.
apply-ch-migrationsheal section (create-bronze-placeholders is now the applicator).e2e rig (unified metrics)
_ensure_tableaccepts the real table being a superset of the fixture schema(connectors emit raw API columns the dbt models never read); seeder Decimal coercion +
jira fixture schema/value alignment to the real bronze.
skip:in*.test.yaml.Testing
PASS=204, ERROR=0, SKIP=0.git_metric_observations), allclass_*incl git,person.persons. Re-dump is byte-identical (converges); fresh-cluster apply validated.Follow-ups
git_metricse2e is skipped: its fixtures predate the github-v2 connector rewrite(
oid/commit_hash→sha, envelope columns). The git silver/gold pipeline itselfbuilds fine now that fix(git): de-shadow completed_at alias in Bitbucket staging models (ILLEGAL_AGGREGATION) #1880 is merged — only the fixtures need refreshing. Tracking
separately.
.github/workflows/connectors-ddl.ymlis committed on the branch butmust be pushed with a
workflow-scope token.