feat(gold): collab_person_counter_daily scaffold + Messaging counters (#1527)

[mozhaev-dev]

PR A of the #1527 split (see #1516 · release #1526) — the data-layer foundation. Establishes the shared gold view the other collab modalities (#1528–#1532) extend, and delivers the Messaging modality's two counters.

Split plan: A (this) → B backend query_ref + catalog seed → C e2e YAML → D FE (insight-front).

Changes

• New gold view insight.collab_person_counter_daily (migration 20260702000000), cloning feat: improve AI personal analytics queries #1514's ai_person_counter_daily:
  • honest-NULL UNION ALL over the per-vendor silver chat classes (M365 Teams · Slack · Zulip),
  • outer GROUP BY (person_id, metric_date) with if(countIf(x IS NOT NULL) > 0, sumIf(x, …), NULL) → a person with no source on a day is NULL, never a fake 0 (a real 0 row stays 0),
  • LEFT JOIN insight.people for org_unit_id (feeds the per-org_unit peer bands in PR B).
• Metrics

  metric	formula	note
  messages_sent	Σ total_chat_messages over M365 · Slack · Zulip	vendor semantics differ (Slack superset incl. replies; M365 excludes group chats & replies) — chat-engagement signal, not comparable absolute
  channel_posts	Σ (channel_posts + channel_replies) over M365 · Slack	Slack folds replies into channel_posts already (channel_replies NULL); M365 splits → add both for comparability; Zulip has no channel split → honest-NULL

• dbt singular test assert_collab_messaging_bounds.sql: counters ≥ 0.
• Register the view in silver/_shared/gold_sources.yml.

Design-for-extension

The view/test are the scaffold: later modalities add columns + UNION branches (not new vendor-named rows). Honest-NULL rendering ("No data" / source tooltip) is #1517; thresholds/bands land with PR B; this PR is data-layer only.

Testing / validation

• Static: YAML valid, migration timestamp-ordered last, no name collisions.
• ⚠️ Not executed against a live ClickHouse locally (no cluster available). View SQL is validated end-to-end by the e2e YAML in PR C and a dev-cluster no-loss spot check (ch.sh --target dev): messages_sent == m365_teams_chats + slack_messages_sent + zulip_messages_sent per QUALITY 5: data + UX: honest-NULL and showing missing data (collaboration counters) #1517 NULL rules.

Refs #1527 · #1516

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added a new daily collaboration metrics view with per-person, per-day totals for messages sent and channel posts across supported chat sources.
• Bug Fixes
  • Improved metric handling so days with no activity remain blank (instead of showing zero), and added warning-level checks when computed totals become negative.
• Chores
  • Updated the silver placeholder schema and added a Gold-layer catalog entry to support the new collaboration metrics, including an added channel replies field.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8bd3ad07-3908-41c4-900c-b255f08cc5d4

📥 Commits

Reviewing files that changed from the base of the PR and between f7d323a2ba6b7f421fd07cb9b1149e8dfbf0783b and 952394b.

📒 Files selected for processing (4)

• src/ingestion/dbt/tests/gold/assert_collab_messaging_bounds.sql
• src/ingestion/scripts/create-bronze-placeholders.sh
• src/ingestion/scripts/migrations/20260702000000_collab-person-counter-daily.sql
• src/ingestion/silver/_shared/gold_sources.yml

✅ Files skipped from review due to trivial changes (1)

• src/ingestion/silver/_shared/gold_sources.yml

🚧 Files skipped from review as they are similar to previous changes (3)

• src/ingestion/scripts/create-bronze-placeholders.sh
• src/ingestion/dbt/tests/gold/assert_collab_messaging_bounds.sql
• src/ingestion/scripts/migrations/20260702000000_collab-person-counter-daily.sql

---

📝 Walkthrough

Walkthrough

This PR adds a new ClickHouse view insight.collab_person_counter_daily for per-person daily messaging metrics, updates the underlying placeholder schema, registers the view as a gold source, and adds a dbt bounds test for negative values.

Changes

Collab Person Counter Daily Scaffold

Layer / File(s)	Summary
Bronze placeholder schema update src/ingestion/scripts/create-bronze-placeholders.sh	Adds channel_replies (Float64) to the silver.class_collab_chat_activity placeholder table definition.
Collab person counter daily view definition src/ingestion/scripts/migrations/20260702000000_collab-person-counter-daily.sql	Drops and recreates insight.collab_person_counter_daily, unioning M365/Slack/Zulip rows from silver.class_collab_chat_activity, aggregating messages_sent/channel_posts per person/day with honest-NULL semantics, and left-joining insight.people for org_unit_id.
Gold source registration and data quality test src/ingestion/silver/_shared/gold_sources.yml, src/ingestion/dbt/tests/gold/assert_collab_messaging_bounds.sql	Registers collab_person_counter_daily as a gold source and adds a dbt singular test that flags non-NULL negative messages_sent/channel_posts values with warn severity and store_failures.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

• catalog(messaging): messages_sent, channel_posts (+ shared scaffold) #1527: Both changes touch the same collab_person_counter_daily messaging scaffold and its dbt/gold-source wiring.
• catalog: modality-scoped collaboration & knowledge metrics — replace vendor-scoped rows #1516: Directly overlaps with the Messaging child for the same gold view and metrics.
• feat(catalog): collaboration & knowledge metric catalog — per-person counters #1515: Implements the per-person collaboration counters proposed in the retrieved catalog issue.
• feat(collab): implement messages_sent / emails_sent / meeting_hours / files_shared as sum + breakdown #1538: Centers on the messages_sent collaboration metric and its daily per-person aggregation.

Suggested reviewers: mitasovr, ktursunov, cyberantonz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the new gold scaffold and Messaging counters added in this PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/collab-person-counter-scaffold

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

src/ingestion/scripts/migrations/20260702000000_collab-person-counter-daily.sql (1)

46-92: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

Optional: collapse three FINAL scans of the same table into one pass.

The M365/Slack/Zulip branches each do a full FINAL scan of silver.class_collab_chat_activity filtered by data_source. A single scan with per-vendor CASE/multiIf projection over data_source IN (...) would avoid reading the table three times. Given this is a clone of ai_person_counter_daily, keeping the readable UNION shape is defensible — flagging only as a potential optimization if the source grows large.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@src/ingestion/scripts/migrations/20260702000000_collab-person-counter-daily.sql`
around lines 46 - 92, The current UNION ALL in the raw subquery scans
silver.class_collab_chat_activity with FINAL three times for M365, Slack, and
Zulip. Consider rewriting the extraction as a single pass over data_source IN
(...) and use CASE/multiIf to derive messages_sent and channel_posts per source,
while keeping the same person_id and metric_date logic in the raw subquery.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@src/ingestion/scripts/migrations/20260702000000_collab-person-counter-daily.sql`:
- Around line 46-92: The current UNION ALL in the raw subquery scans
silver.class_collab_chat_activity with FINAL three times for M365, Slack, and
Zulip. Consider rewriting the extraction as a single pass over data_source IN
(...) and use CASE/multiIf to derive messages_sent and channel_posts per source,
while keeping the same person_id and metric_date logic in the raw subquery.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 56a19a8f-be25-4461-ade5-4cfaefbf583e

📥 Commits

Reviewing files that changed from the base of the PR and between 84862c5 and f6e5a110373bfc34f87daeab98c61c51a772af09.

📒 Files selected for processing (4)

• src/ingestion/dbt/tests/gold/assert_collab_messaging_bounds.sql
• src/ingestion/scripts/create-bronze-placeholders.sh
• src/ingestion/scripts/migrations/20260702000000_collab-person-counter-daily.sql
• src/ingestion/silver/_shared/gold_sources.yml