data: fix #1318 type mismatch + enforce contract on collab silver + dbt-coverage gate

[SharedQA]

First slice of the data-correctness work, three coherent parts.

1. Fix #1318 — the real type-mismatch bug

class_collab_document_activity failed ClickHouse NO_COMMON_TYPE: Float64 vs Int64: visited_page_count was Int64 from the OneDrive half and Float64 from SharePoint, so the silver tag-union couldn't pick a supertype. Pinned to Nullable(Float64) on both halves (OneDrive CAST(NULL …), SharePoint CAST(visitedPageCount …)).

2. Enforce a dbt contract on the collab silver model (first step to contract = 100%)

class_collab_document_activity now has contract: { enforced: true } with every column declared (name + data_type, taken from the live ClickHouse catalog). dbt verifies the built relation against the declared schema at build time, so a future Float64↔Int64 (or any) drift becomes a build failure, not a runtime ClickHouse error — the systematic prevention for the #1318 class.

3. scripts/ci/dbt_coverage.py — the data-contract gate

Walks the compiled dbt manifest and fails if any metric-path model (schema silver/gold, plus silver:* tag-union feeders) lacks an enforced contract or not_null+unique on its key. This is the mechanical enforcer for the "dbt/contract → 100%" targets. Wire-up: make contracts + a CI step after dbt parse (lands with the gate workflows once the enforcement PR merges).

Validation note

dbt can't be installed on the dev host, so I took the contract data_types from the deployed ClickHouse catalog (DESCRIBE TABLE) rather than guessing. CI's dbt build is the authority — if any declared type is off, the contract (correctly) fails the build.

Closes #1318 · Related #1319 (wired-empty connectors: seed-or-deregister decision).

Summary by CodeRabbit

• New Features
  • Added a “Data Contracts” CI workflow that generates dbt metadata, runs a coverage assessment, and publishes a coverage report plus the dbt manifest artifact.
• Tests
  • Added a coverage gate that inspects metric-path models to confirm enforced contracts and required not_null/unique tests for configured key columns (currently report-focused by default).
• Bug Fixes
  • Updated visited_page_count to use Float64 for M365 document-activity models to avoid type-union failures.
• Chores
  • Expanded and strengthened the collaboration document-activity silver schema with explicit column typing and stricter contract/on-schema-change settings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 39e88fd1-531a-454d-b8e6-6aa5f3220a4a

📥 Commits

Reviewing files that changed from the base of the PR and between 8e73749 and 59ab13e.

📒 Files selected for processing (2)

• .github/workflows/data-contracts.yml
• scripts/ci/dbt_coverage.py

🚧 Files skipped from review as they are similar to previous changes (2)

• .github/workflows/data-contracts.yml
• scripts/ci/dbt_coverage.py

---

📝 Walkthrough

Walkthrough

Adds an enforced, fully-typed silver schema for collaboration activity, aligns staging halves to use Nullable(Float64) for visited_page_count, and introduces a manifest-driven dbt_coverage CLI plus a GitHub Actions workflow to report (and optionally enforce) contract and key-test completeness.

Changes

Data Contract Enforcement and Type Alignment

Layer / File(s)	Summary
Silver schema contract and full column typing src/ingestion/silver/collaboration/schema.yml	class_collab_document_activity declares config.contract.enforced: true and on_schema_change: fail, adds explicit ClickHouse data_type for existing columns and new identity/metric/ingestion/version fields (e.g., unique_key: FixedString(16), collected_at, _version).
Type alignment in OneDrive and SharePoint staging models src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_onedrive.sql, src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_sharepoint.sql	Both staging halves cast visited_page_count to Nullable(Float64): OneDrive changes the NULL cast to Float64; SharePoint casts visitedPageCount to Nullable(Float64), resolving NO_COMMON_TYPE in the silver union.
dbt coverage validation script scripts/ci/dbt_coverage.py	New CLI loads a compiled dbt manifest.json, identifies metric-path models by schema/tag rules and naming patterns, checks contract.enforced and per-key not_null+unique tests, reports gaps and totals, exits non-zero with --check on contract/key-test violations.
GitHub Actions workflow for dbt coverage .github/workflows/data-contracts.yml	Workflow runs on PRs affecting ingestion paths, sets up Python 3.12 and dbt dependencies, executes dbt parse to generate manifest, runs the coverage script, writes report to step summary, and uploads report and manifest as artifact.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• constructorfabric/insight#1306: Introduces a new figma__bronze_promoted dbt model that matches the naming convention used by the dbt_coverage.py script to classify metric-path models for contract and key-test gating.

Suggested reviewers

• mozhaev-dev
• cyberantonz

Poem

"I hopped through manifests, nibbled at tags,
Fixed a float where an int once lagged.
Contracts now guard the silver gate,
CI watches metrics—nice and straight.
🐇📊"

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the three main changes: fixing the type mismatch (#1318), enforcing contract on collab silver, and adding the dbt-coverage gate script.
Linked Issues check	✅ Passed	The PR fully addresses #1318 requirements: fixes the visited_page_count type mismatch, pins both halves to Nullable(Float64), enforces dbt contract with data_type on the silver model, and adds a dbt-coverage gate script to systematically prevent recurrence.
Out of Scope Changes check	✅ Passed	All changes directly support the stated objectives: type fixes for the OneDrive and SharePoint models, schema contract enforcement for class_collab_document_activity, and the dbt-coverage gate infrastructure to validate contracts going forward.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/ci/dbt_coverage.py`:
- Around line 66-67: The code reads contract enforcement from n.get("contract")
but manifest stores it under n["config"]["contract"]["enforced"]; update the
enforcement check used to set the enforced variable (the expression that assigns
enforced in scripts/ci/dbt_coverage.py) to traverse config → contract → enforced
(e.g., use n.get("config", {}).get("contract", {}).get("enforced")) so enforced
is true for contract-enforced nodes; leave the tests_by_model lookup (kinds =
tests_by_model.get(uid, set())) unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f8b1c528-aaa2-4490-b09e-c1a0645776f0

📥 Commits

Reviewing files that changed from the base of the PR and between 43bfb9f and f96f59415c2d603d4593b5335650b9ac387bc329.

📒 Files selected for processing (4)

• scripts/ci/dbt_coverage.py
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_onedrive.sql
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_sharepoint.sql
• src/ingestion/silver/collaboration/schema.yml

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/data-contracts.yml:
- Line 63: The grep invocation "grep -E 'metric-path models|enforced
contract|key tests|bronze sources|freshness declared' /tmp/dbt-coverage.txt" can
return non-zero and fail the step on harmless output drift; change that command
to tolerate no matches by adding a fallback (for example append "|| true" or
otherwise ensure a non-zero grep exit does not make the job fail) so the step
remains report-only and won't break CI if grep finds nothing.
- Around line 53-54: Replace the failing-suppressed command "dbt deps || true"
with a plain "dbt deps" so dependency resolution failures surface and fail the
CI job; keep the subsequent "dbt parse --no-version-check" as-is so parsing
still runs only after deps succeed.

In `@scripts/ci/dbt_coverage.py`:
- Around line 59-67: The code computes freshness over all manifest "sources" but
prints "bronze sources"; narrow the computation to the intended bronze/Airbyte
subset by creating a filtered dict (e.g., bronze_sources = {uid: s for uid, s in
sources.items() if <bronze-predicate>(s)}) and then replace uses of sources in
the fresh calculation, the count/percentage print, and the for-loop with
bronze_sources (update the print message to still say "bronze sources" and keep
the per-source check using s.get("freshness") and s.get("loaded_at_field"));
alternatively, if you prefer not to filter, change the printed label and comment
from "bronze sources" to "all sources" and leave the existing computations
as-is.
- Around line 39-43: The is_metric_path predicate is too broad because it
currently includes every model in the bronze schema; update the logic in
is_metric_path to stop treating all bronze-schema models as in-scope — only keep
silver/gold schemas plus explicit bronze promotion names and silver-tagged
models. Concretely, change the return expression to remove schema in ("bronze")
(i.e. use schema in ("silver", "gold") or name.endswith("__bronze_promoted") or
any(t == "silver" or t.startswith("silver:") for t in tags)) so only promoted
bronze views, silver feeders, and silver/gold models are considered.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b58adc3c-867c-41d2-bbbe-ee80442864f4

📥 Commits

Reviewing files that changed from the base of the PR and between f96f59415c2d603d4593b5335650b9ac387bc329 and 507664312fdc944ca0166fa09835ea9cfc4b5041.

📒 Files selected for processing (3)

• .github/workflows/data-contracts.yml
• scripts/ci/dbt_coverage.py
• src/ingestion/silver/collaboration/schema.yml

🚧 Files skipped from review as they are similar to previous changes (1)

• src/ingestion/silver/collaboration/schema.yml

[SharedQA]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/ci/dbt_coverage.py`:
- Around line 75-89: The code only extracts single column_name values from tests
but unique_combination_of_columns tests store their columns in
kwargs.combination_of_columns as a list instead. After the existing column
extraction logic that looks for column_name, add fallback logic to extract the
list of columns from kwargs.combination_of_columns when column_name is not
present. Then modify the loop that processes dependencies and adds tests to
col_tests to handle both single columns and multiple columns from the
combination, ensuring the test kind is recorded for each column in the
combination so that composite unique tests are properly validated.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: cc1e1628-2051-414d-97e2-91cb8a51feb5

📥 Commits

Reviewing files that changed from the base of the PR and between 507664312fdc944ca0166fa09835ea9cfc4b5041 and 8e73749.

📒 Files selected for processing (4)

• .github/workflows/data-contracts.yml
• scripts/ci/dbt_coverage.py
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_onedrive.sql
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_sharepoint.sql

💤 Files with no reviewable changes (2)

• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_sharepoint.sql
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_onedrive.sql

🚧 Files skipped from review as they are similar to previous changes (1)

• .github/workflows/data-contracts.yml

[ktursunov]

reworked here #1695