fix(ingestion): watermark collaboration staging on extract time, not business date

[mitasovr]

Problem

The incremental connector-staging models for collaboration gated incrementality on a forward-only business-date watermark:

{% if is_incremental() %}
  WHERE business_date > (SELECT max(business_date) - INTERVAL 3 DAY FROM {{ this }})
{% endif %}

This permanently strands backfilled / late-arriving bronze history: once the target's max business date advances, any source rows whose business date is older than that max never re-enter the model — even though they are present in bronze.

Observed impact (virtuozzo)

A user's Meeting Hours read ~19h/month ≈ ~17h/week — month ≈ week — for someone whose true month was ~67h.

Root cause traced layer by layer:

layer	Zoom meeting coverage
bronze_zoom.participants	128 days (2026-01 … 06-08)
staging.zoom__collab_meeting_activity	8 days (2026-06-01 … 06-08)

So the monthly metric was effectively M365 (full) + Zoom (last week only). Slack chat had the same gap (bronze from Jan, staging from late-Mar). M365/Cursor/Claude happened not to manifest only because their bronze never gained pre-existing history after the watermark advanced — the code is equally fragile.

Fix

Gate on the source extract time instead of the business date. Re-pulled / late rows always carry a fresh _airbyte_extracted_at, so we reprocess every business date that has any bronze row extracted within 3 days of the newest extract:

{% if is_incremental() %}
  AND (
    (SELECT count() FROM {{ this }}) = 0
    OR business_date IN (
      SELECT DISTINCT business_date
      FROM {{ source(...) }}
      WHERE _airbyte_extracted_at
            > (SELECT max(_airbyte_extracted_at) FROM {{ source(...) }}) - INTERVAL 3 DAY
    )
  )
{% endif %}

• The whole date is re-derived from full bronze, so the Zoom per-day aggregate (GROUP BY email, date) stays correct even when a day's participants arrive across several sync batches — a plain _airbyte_extracted_at > max row-watermark (as used by the 1:1 git/CRM staging models) would undercount aggregates here, which is why this uses a date-reprocess form.
• Pure WHERE change — no new output columns, so union_by_tag feeders stay schema-aligned and no --full-refresh redeploy is required.

Files (8 collaboration staging models)

• zoom__collab_meeting_activity
• m365__collab_meeting_activity, m365__collab_chat_activity, m365__collab_email_activity, m365__collab_document_activity_onedrive, m365__collab_document_activity_sharepoint
• slack__collab_chat_activity
• zulip_proxy__collab_chat_activity

Already remediated on virtuozzo

A one-off dbt run --full-refresh over the affected staging + silver chain has been applied: staging.zoom__collab_meeting_activity went 8 → 127 days, and the user's monthly Meeting Hours corrected 19.82h → 66.77h with no duplicates (silver union_by_tag QUALIFY-dedup + gold view).

Follow-up (same anti-pattern, not in this PR)

The AI usage models carry the identical business-date watermark and should get the same fix: cursor__ai_dev_usage, claude_admin__ai_{api,dev}_usage, claude_enterprise__ai_{assistant,dev}_usage, claude_team__ai_dev_usage, copilot__ai_{dev,org}_usage, the OpenAI to_ai_* models (max(report_date), no interval), and silver class_focus_metrics (max(day) - 3). They are not currently manifesting on virtuozzo.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Refactor
  • Updated incremental data refresh logic for collaboration data models (Microsoft 365, Slack, Zoom, Zulip). Data reprocessing now uses source extraction timestamps with a 3-day lookback window instead of previously stored business dates, improving consistency in data updates.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 98ac5b87-75b3-48dc-bc97-7a67d37e3c5d

📥 Commits

Reviewing files that changed from the base of the PR and between 735c3a1 and 9cacd55.

📒 Files selected for processing (8)

• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_chat_activity.sql
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_onedrive.sql
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_sharepoint.sql
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_email_activity.sql
• src/ingestion/connectors/collaboration/m365/dbt/m365__collab_meeting_activity.sql
• src/ingestion/connectors/collaboration/slack/dbt/slack__collab_chat_activity.sql
• src/ingestion/connectors/collaboration/zoom/dbt/zoom__collab_meeting_activity.sql
• src/ingestion/connectors/collaboration/zulip-proxy/dbt/zulip_proxy__collab_chat_activity.sql

---

📝 Walkthrough

Walkthrough

This PR updates eight dbt incremental collaboration models (M365 chat/email/meeting/document activity, Slack chat, Zoom meeting, Zulip chat) to use source extraction timestamps (_airbyte_extracted_at) for incremental watermarking instead of comparing against the target table's previously loaded dates. All models now detect empty targets and reprocess business dates from recent source extracts within the last 3 days.

Changes

Incremental Watermark Refactoring

Layer / File(s)

Summary

Incremental watermark replacement across all connectors src/ingestion/connectors/collaboration/m365/dbt/m365__collab_chat_activity.sql, src/ingestion/connectors/collaboration/m365/dbt/m365__collab_email_activity.sql, src/ingestion/connectors/collaboration/m365/dbt/m365__collab_meeting_activity.sql, src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_onedrive.sql, src/ingestion/connectors/collaboration/m365/dbt/m365__collab_document_activity_sharepoint.sql, src/ingestion/connectors/collaboration/slack/dbt/slack__collab_chat_activity.sql, src/ingestion/connectors/collaboration/zoom/dbt/zoom__collab_meeting_activity.sql, src/ingestion/connectors/collaboration/zulip-proxy/dbt/zulip_proxy__collab_chat_activity.sql

All incremental models switch from target max(date) watermarking to source _airbyte_extracted_at watermarking. Each model now includes an empty-target check and reprocesses business dates present in source rows extracted within the last 3 days relative to the source's maximum extraction timestamp.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

📅 Eight models dance in sync today,
From old dates swept to fresh data's way,
Extract timestamps now show the light,
Three days of change within our sight! 🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: shifting incremental watermarking from business date to extract time across collaboration staging models.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}