test(data): source freshness on all bronze connectors (#1322)

[SharedQA]

What

Adds dbt source freshness to all 21 bronze_* connector sources — warn_after: 36h, error_after: 72h, loaded_at_field: _airbyte_extracted_at.

Why

A stopped/slow sync is currently invisible: the dashboard just shows stale numbers with no error. dbt source freshness turns that into a signal. Implements the bronze half of #1322 (EPIC #1321).

Notes

• Freshness is its own command (dbt source freshness), not part of dbt build/test — so sources not yet ingested (data: wired-but-empty connectors (Slack, Zoom, Zulip-Proxy) break silver build and yield no metrics — seed or deregister #1319) don't break the normal run.
• Verified _airbyte_extracted_at exists on the ingested bronze tables.
• Next in this branch: enforced contracts (data_type) on silver models — those need the warehouse-gated CI dbt run to validate the type declarations, so they land once data-checks.yml runs against a populated ClickHouse (can't be validated on a machine without dbt).

Draft until the freshness thresholds are confirmed with the team.

Summary by CodeRabbit

• New Features
  • Enabled DBT data freshness monitoring across 20+ connectors, so stale or missing upstream data now surfaces as freshness warnings/errors within configured time windows.
  • Added source-level defaults (based on ingestion/extraction time) and table-level overrides for better alignment to the most relevant business or event timestamps, with tuned thresholds (commonly ~36h/72h, and stricter settings like ~72–168h for specific datasets).

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 180a4d5f-8fdc-4ee5-bfec-c8450e4f5f8b

📥 Commits

Reviewing files that changed from the base of the PR and between f9bac1d and ab03586.

📒 Files selected for processing (3)

• src/ingestion/connectors/ai/openai/dbt/schema.yml
• src/ingestion/connectors/wiki/confluence/dbt/schema.yml
• src/ingestion/connectors/wiki/outline/dbt/schema.yml

🚧 Files skipped from review as they are similar to previous changes (2)

• src/ingestion/connectors/wiki/outline/dbt/schema.yml
• src/ingestion/connectors/wiki/confluence/dbt/schema.yml

---

📝 Walkthrough

Walkthrough

All 21 bronze Airbyte connector schema.yml files receive dbt source freshness configuration across AI, collaboration, CRM, git, HR-directory, task-tracking, UI-design, and wiki categories. Nine sources gain uniform loaded_at_field: _airbyte_extracted_at with warn_after: 36h and error_after: 72h. Twelve sources with windowed or parsed data add source-level defaults plus table-level overrides using parsed business dates and adjusted thresholds (72h/120h).

Changes

dbt Source Freshness Rollout

Layer / File(s)	Summary
Basic source freshness (36h/72h on _airbyte_extracted_at) src/ingestion/connectors/collaboration/zulip-proxy/dbt/schema.yml, src/ingestion/connectors/crm/hubspot/dbt/schema.yml, src/ingestion/connectors/crm/salesforce/dbt/schema.yml, src/ingestion/connectors/git/bitbucket-cloud/dbt/schema.yml, src/ingestion/connectors/git/github/dbt/schema.yml, src/ingestion/connectors/hr-directory/ms-entra/dbt/schema.yml, src/ingestion/connectors/task-tracking/jira/dbt/schema.yml, src/ingestion/connectors/task-tracking/youtrack/dbt/schema.yml, src/ingestion/connectors/ui-design/figma/dbt/schema.yml	Each bronze source definition receives uniform loaded_at_field: _airbyte_extracted_at and freshness thresholds (warn_after: 36h, error_after: 72h) to detect stalled Airbyte extracts.
Source + table-level freshness with parsed business dates src/ingestion/connectors/ai/chatgpt-team/dbt/schema.yml, src/ingestion/connectors/ai/claude-admin/dbt/schema.yml, src/ingestion/connectors/ai/claude-enterprise/dbt/schema.yml, src/ingestion/connectors/ai/claude-team/dbt/schema.yml, src/ingestion/connectors/ai/cursor/dbt/schema.yml, src/ingestion/connectors/ai/github-copilot/dbt/schema.yml, src/ingestion/connectors/ai/openai/dbt/schema.yml, src/ingestion/connectors/collaboration/m365/dbt/schema.yml, src/ingestion/connectors/collaboration/slack/dbt/schema.yml, src/ingestion/connectors/collaboration/zoom/dbt/schema.yml, src/ingestion/connectors/wiki/confluence/dbt/schema.yml, src/ingestion/connectors/wiki/outline/dbt/schema.yml	Each source sets source-level freshness defaults (36h/72h on _airbyte_extracted_at) and overrides with table-level loaded_at_field expressions (parseDateTimeBestEffortOrNull(date), parseDateTimeBestEffortOrNull(metric_date), etc.) and adjusted thresholds (72h/120h) for windowed, snapshot, or inferred datasets to reflect business date semantics.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

• data(bronze): source freshness on all sources + contracts on *__bronze_promoted views #1322: This PR directly implements the requirement to add loaded_at_field and freshness (warn_after/error_after) thresholds to bronze sources to detect stalled Airbyte connectors and surface data freshness through dbt freshness warnings/errors.

Possibly related PRs

• constructorfabric/insight#1306: Both PRs modify src/ingestion/connectors/ui-design/figma/dbt/schema.yml by adding freshness configuration to the bronze_figma source, extending schema enhancements from #1306.

• constructorfabric/insight#1312: Both PRs update src/ingestion/connectors/wiki/outline/dbt/schema.yml for the bronze_outline source and wiki_pages table, with this PR adding freshness thresholds alongside prior schema work.

Suggested reviewers

• mitasovr
• mozhaev-dev

Poem

🐇 Hopping through the YAML fields so bright,
Where bronze connectors track the night,
Thirty-six hours to warn, seventy-two to cry—
No stale Airbyte row shall slip on by!
Business dates parsed, thresholds aligned,
Fresh data blooms—no lag left behind! 🌿

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding dbt source freshness checks across all bronze connectors, which is the primary objective of this PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[dzarlax]

Coordinating — I have a ready "runtime/alerting" layer that builds directly on this PR.

This PR nails the declaration half of #1322 (per-source freshness blocks so dbt_coverage.py counts them toward 130/130). I've got the complementary runtime half already written against the old tree, and it sits cleanly on top of yours rather than competing with it:

• dbt-source-freshness-check CronWorkflow — runs dbt source freshness --select source:* daily after the sync window, parses target/sources.json, logs each breaching source with max_loaded_at + lag, and flips the workflow to Failed only on error/runtime error (warn stays green, still printed + emitted in payload).
• Driver-based notification (webhook / zulip / slack / teams) off a provider-agnostic JSON payload.
• Runtime trap-detector (freshness-trap-detect.py) for the "dbt couldn't even check" case (CH down, schema drift).
• PRD/DESIGN/FEATURE for a new ingestion-monitoring domain.

None of that exists in #1322's scope (which is declaration + coverage gate), so it's additive. Two things to align:

1. Thresholds — I had 30h/48h, you have 36h/72h. Happy to defer to yours; the cron reads whatever the sources declare, so no duplication.
2. Sequencing — my cron consumes the per-source declarations this PR adds, so it should land after this merges. I'd open it as a follow-up PR referencing EPIC: data-correctness contract coverage (bronze · silver · gold) → 100% #1321 as the runtime/alerting half of data(bronze): source freshness on all sources + contracts on *__bronze_promoted views #1322.

One design note for the record: my original took the inherited-default route (loaded_at_field + thresholds in dbt_project.yml, per-table opt-outs). That's functionally equivalent but wouldn't satisfy dbt_coverage.py, which counts per-source declarations — so your per-source approach is the right call and I'll build on it as-is.

Want me to open the follow-up PR once this merges, or fold the cron into this branch?

[SharedQA]

Thanks — agreed on all of it, and good that the runtime half is already written.

On the two alignment points:

1. Thresholds — keep 36h/72h here. The bronze syncs are daily, so 36h warn is roughly a 1.5-day grace (one missed sync isn't yet alarming) and 72h error covers a weekend gap before anyone gets paged. Since your cron reads whatever the sources declare, there's a single source of truth and no duplication — if we want to tighten later, it changes in one place.

2. Sequencing — open it as a separate follow-up after this merges, not folded in. This PR is deliberately declaration-only: static, no warehouse needed, mergeable as-is. Your runtime layer is a different concern with its own domain (ingestion-monitoring) and its own PRD/DESIGN, so keeping them apart keeps both reviewable and lets this gate land without waiting on the alerting design. The dependency direction is right too — your cron consumes these per-source blocks, so this lands first and yours sits on top.

For tracking: the runtime/alerting CronWorkflow is the "scheduled bronze freshness" item in the stabilization plan (the monitoring line, #949), so it slots in there as the operational counterpart to this declaration gate.

And confirmed your read on dbt_coverage.py — it counts per-source declarations, which is exactly why we went per-source instead of the dbt_project.yml inherited default. Please go ahead and open the follow-up once this merges.

[dzarlax]

Heads-up before this lands: flat _airbyte_extracted_at is false-green on the windowed/vendor-analytics connectors — verified on live data (Eternal installation).

The check correctly catches "sync fully stopped" for true incremental connectors. But for any connector whose API returns a fixed reporting window that the sync re-fetches every run (M365 Graph reports, ChatGPT Team, Claude Team), _airbyte_extracted_at is re-stamped on every successful sync regardless of whether the business data advanced — so it measures sync liveness, not data freshness.

Measured just now on the Eternal ClickHouse (now() - max(...)):

bronze source	max(_airbyte_extracted_at) age	max(<business_date>) age	distinct business dates re-emitted in the last extraction batch
m365.{email,teams,onedrive,sharepoint}_activity (reportRefreshDate)	10.6 h	108.7 h	24
chatgpt_team.chatgpt_team_chat_activity (date)	8.6 h	84.7 h	3
chatgpt_team.chatgpt_team_codex_user_daily (date)	8.6 h	60.7 h	4
claude_team.claude_team_code_metrics (metric_date)	8.6 h	60.7 h	6
youtrack.youtrack_issue (incremental, contrast)	9.4 h	13.2 h	—

Right now M365 business data is 4.5 days stale on this install, but warn_after: 36h / error_after: 72h on _airbyte_extracted_at reports green. The incremental connectors (youtrack/jira) are fine — ext_age ≈ biz_age — which is exactly the discriminator.

This matters because the false-green sources are the AI-usage connectors that Insight's value story leans on. The connector config is misleading here too — M365 declares lookback_window: P0D, but the Graph getReport endpoint returns the whole reportPeriod window, so the data re-fetches regardless.

Proposal: adopt the per-source model from the earlier #256 work as the declaration here — it's demonstrably the correct instrument on exactly these connectors. Concretely, additive to this PR's per-source structure:

1. Anchor loaded_at_field on the business-date column for the windowed connectors instead of _airbyte_extracted_at — reportRefreshDate (M365), date (ChatGPT / Slack), metric_date (Claude). Incremental connectors (git, jira, youtrack) keep _airbyte_extracted_at as-is.
2. Tier the thresholds so the windowed connectors get a higher warn_after/error_after that absorbs the legitimate 2–3 day vendor publish lag — otherwise anchoring on the business date would false-alarm. (This is why bug: frontend calls /api/identity-resolution/* removed by PR #398 (regression) #256 carried tiers; the dogfood numbers above show the lag is real, not a bug.)
3. Per-table opt-outs (freshness: null + a written reason) for the slow-moving lookup/catalog streams.

I have the full per-connector loaded_at_field + tier mapping from #256 ready to go. Happy to push it straight onto this branch so the declaration half ships correct in one piece, or land it as the immediate follow-up the runtime PR sits on — whichever you prefer for review flow.

[SharedQA]

This is the right catch, and it changes my call from earlier in the thread — thank you for dogfooding it on live data.

You're right that _airbyte_extracted_at measures sync liveness, not data freshness, and that it goes false-green on exactly the windowed vendor-analytics connectors (M365 Graph, ChatGPT Team, Claude Team) where the API re-emits a fixed reporting window every run. I corroborated the anchors you propose are real columns on those connectors (reportRefreshDate on M365, metric_date on Claude, date on the ChatGPT/Slack streams), so the per-source anchoring is feasible as a drop-in.

So I'm reversing my "keep it flat, declaration-only is fine as-is" position: the per-source + tiered model belongs in this PR, not a follow-up. A freshness gate that reads green while M365 business data is 4.5 days stale is worse than no gate — and it's worst on the AI-usage connectors that matter most. My earlier 36h/72h reasoning only held for the true-incremental connectors (ext_age ≈ biz_age); it's wrong for the windowed ones.

Please push the #256 per-connector mapping straight onto this branch (claude/dbt-test-coverage) — you have the live-verified loaded_at_field + tier values and I'd only be guessing the per-stream business-date columns and the lag-absorbing thresholds, which is exactly where a wrong guess turns into false-alarm. Concretely, to land the declaration correct in one piece:

1. Anchor loaded_at_field on the business-date column for the windowed connectors; keep _airbyte_extracted_at for the incrementals (git/jira/youtrack) — ext_age ≈ biz_age is the clean discriminator.
2. Tiered warn_after/error_after so the windowed connectors absorb the real 2–3 day vendor publish lag (your numbers show it's genuine).
3. Per-table freshness: null + a one-line reason for the slow-moving lookup/catalog streams.

The runtime/alerting CronWorkflow stays the separate follow-up we discussed — this keeps that PR sitting on a declaration that's actually correct. If pushing to the branch is awkward from your side, open it as a PR into claude/dbt-test-coverage and I'll merge it in.

[dzarlax]

Opened it as a PR into this branch: SharedQA#1 — anchors the windowed connectors (m365 / chatgpt_team / claude_team verified, the rest column-verified or inferred-and-marked) on their business-date column with tiered thresholds; incrementals untouched. The PR body breaks down the three confidence tiers and the one caveat (no end-to-end dbt source freshness run on the full set yet). Ready for you to merge in whenever it looks right.

[dzarlax]

One more before this lands — reviewing the merged anchors I caught a latent bug and opened SharedQA#2 against this branch:

• openai: bucket_start_time is Unix seconds (integer), not a date string — the connector's own models use fromUnixTimestamp(CAST(... AS UInt32)). The anchor I added in [PRE-01] Define Priority-1 connectors for VZ #1 used parseDateTimeBestEffortOrNull, which would runtime error on every openai usage/cost table once it's ingested. [PRE-02] Define Priority-1 charts and dashboards #2 fixes it to the epoch form.
• confluence/outline: aligned wiki_pages to parseDateTime64BestEffortOrNull(updated_at, 3) to match each connector's own model convention (functionally equivalent, just consistent).

openai is empty in the dogfood install so it wouldn't fail today, but it's a latent false-positive. Worth folding #2 in before #1346 merges so the gate ships correct. All other inferred anchors were re-checked against the connectors' silver models and parse fine.