data(bronze): anchor windowed connectors on business date + tiered freshness - #1
Merged
SharedQA merged 1 commit intoJun 16, 2026
Conversation
…eshness Flat _airbyte_extracted_at is false-green on windowed/vendor-analytics connectors: the API re-emits a fixed reporting window every sync, so the extracted-at timestamp tracks sync liveness, not data freshness. Verified live (Eternal): M365 business data 4.5d stale while extracted_at 10.6h fresh; same on ChatGPT Team / Claude Team. Incrementals (git/jira/youtrack) are fine (ext_age approx biz_age) and keep _airbyte_extracted_at. Per connector: - m365 / chatgpt_team / claude_team: VERIFIED on live data — anchor on reportRefreshDate / date,snapshot_date / metric_date, report tier (72/120h). - slack / cursor / zoom: column verified in CH, behavior inferred — anchor on date / day / event-time; slack report_extended (120/168h), zoom event (96/168h). - openai / claude-admin / claude-enterprise / github-copilot / confluence / outline: INFERRED, not live-verified (no rows in any reachable install; connector config is not a reliable signal, cf. M365 P0D). Anchored on the cursor business column with conservative tiers, marked inline. - rosters/lookups keep _airbyte_extracted_at (full-refresh → sync-liveness signal). Refs constructorfabric#1321, constructorfabric#1322. Builds the correct declaration the runtime gate (constructorfabric#949) sits on.
SharedQA
pushed a commit
that referenced
this pull request
Jun 16, 2026
Review of constructorfabric#1346 caught two anchor issues in the per-source freshness blocks: - openai: `bucket_start_time` is Unix seconds (integer), per the connector's own dbt models (toDate(fromUnixTimestamp(CAST(bucket_start_time AS UInt32)))). parseDateTimeBestEffortOrNull expects a string and would runtime-error on all usage/cost tables. Switch to fromUnixTimestamp(CAST(... AS UInt32)). - confluence/outline wiki_pages: align to parseDateTime64BestEffortOrNull(updated_at, 3) to match each connector's own model convention (functionally equivalent for freshness, but consistent). Follow-up to PR #1 on this branch; keeps constructorfabric#1346 correct before it merges to main. (cherry picked from commit cb13e9c) Signed-off-by: Kenan Salim <kenan.salim@rolos.com>
dzarlax
pushed a commit
to dzarlax/insight-back
that referenced
this pull request
Jun 17, 2026
The PRD/DESIGN/FEATURE/MONITORING docs described the original cohesive design (inherited +freshness in dbt_project.yml, env_var-driven thresholds, lint-bronze-freshness.py CI). That model was dropped in favor of the split agreed with QA: - declarations + literal per-source thresholds owned by the connector schemas (EPIC constructorfabric#1321/constructorfabric#1322 — PR constructorfabric#1346 + PR SharedQA#1), coverage counted by the QA dbt_coverage.py gate - this PR owns runtime/alerting only (CronWorkflow, parser, trap detector, driver notifications) Also folds in the empirically-verified false-green finding (windowed vendor-analytics connectors must anchor on a business-date column, not _airbyte_extracted_at) and corrects cyberfabric -> constructorfabric refs. Signed-off-by: Alexey Panfilov <alexey.panfilov@constructor.tech>
SharedQA
pushed a commit
that referenced
this pull request
Jun 23, 2026
Review note #1: sublabels still read "Confluence" though Outline now feeds the same 4 wiki_bullet_rows keys. Aligns sublabel with the already-neutral description; FE bullet-def updated in the paired PR. Cosmetic — no schema/data change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Sergei Mozhaev <mozhaev.dev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Anchors the windowed / vendor-analytics bronze sources on their business-date column with tiered freshness thresholds, on top of the per-source freshness blocks in this branch (constructorfabric#1346). Incremental connectors are left exactly as-is.
This is the per-source model from the earlier constructorfabric#256 work, landed in the per-source structure this branch uses (so
dbt_coverage.pystill counts every source). Opened againstclaude/dbt-test-coverageper the thread on constructorfabric#1346.Why
Flat
_airbyte_extracted_atis false-green on connectors whose API re-emits a fixed reporting window every sync: the extracted-at timestamp is re-stamped each run, so it measures sync liveness, not data freshness. Verified live (Eternal install):_airbyte_extracted_atagem365.*_activity(reportRefreshDate)chatgpt_team.chat_activity(date)claude_team.code_metrics(metric_date)youtrack.youtrack_issue(incremental, contrast)M365 business data was 4.5 days stale while a flat 72h-error gate read green — worst on the AI-usage connectors. Incrementals (
ext_age ≈ biz_age) are correct as-is.Confidence tiers (marked inline in each file)
m365,chatgpt_team,claude_team. Anchor + report tier (72/120h) backed by measuredext_agevsbiz_age.slack(date, report_extended 120/168h),cursor(day),zoom(event-time, event 96/168h). Columns confirmed in ClickHouse; windowed behavior inferred (no live rows).openai,claude-admin,claude-enterprise,github-copilot,confluence,outline. No rows in any reachable install, and connector config is not a reliable signal (M365 declaredlookback_window: P0Dyet re-fetches a 24-day window). Anchored on the cursor business column with conservative tiers; each markedINFERRED, NOT live-verified+confirm once ingested. Anchoring is harmless if a source turns out to be strictly incremental, and a wrong column surfaces as a visibleruntime error, never a silent false-green.Rosters / lookups (full-refresh) keep
_airbyte_extracted_atas a sync-liveness signal.Not done
dbt source freshnesshas not been run end-to-end against a populated warehouse on this set (YAML validated; theparseDateTimeBestEffortOrNull(...)expressions were checked individually against live ClickHouse). The tier-2/3 sources are the ones to watch on the first real run.Refs constructorfabric#1321, constructorfabric#1322. This is the declaration the runtime/alerting CronWorkflow (constructorfabric#949) sits on.