-
Notifications
You must be signed in to change notification settings - Fork 9
feat(metrics): wiki on unified metric-results #1812
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
1c916f7
feat(metrics): wiki on unified metric-results
aleksdotbar bd6d39c
fix(ingestion): placeholder wiki_page_versions bronze tables
aleksdotbar 56c00fe
fix(gold): keep comments for pages without a creation timestamp
aleksdotbar 9c5eddf
Merge branch 'main' into feat/unified-metrics-wiki
aleksdotbar 6e76fae
Merge branch 'main' into feat/unified-metrics-wiki
aleksdotbar File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
14 changes: 14 additions & 0 deletions
14
src/ingestion/dbt/tests/gold/assert_wiki_observations_entity_id_shape.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| -- Build-integrity check (untagged → error severity under `dbt build`). | ||
| -- Unified entity ids for persons are lowercased emails; the runtime and the | ||
| -- cohort view join on exact string equality, so an empty, mixed-case, or | ||
| -- non-email id (an unresolved wiki account leaking past the staging email | ||
| -- gate) silently drops the person from every surface. | ||
| SELECT | ||
| entity_id, | ||
| measure_key, | ||
| count() AS row_count | ||
| FROM {{ ref('wiki_metric_observations') }} | ||
| WHERE entity_id = '' | ||
| OR entity_id != lower(entity_id) | ||
| OR entity_id NOT LIKE '%@%' | ||
| GROUP BY entity_id, measure_key |
10 changes: 10 additions & 0 deletions
10
src/ingestion/dbt/tests/gold/assert_wiki_observations_nonnegative.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| -- Build-integrity check (untagged → error severity under `dbt build`). | ||
| -- Every wiki measure is a count (pages, edit sessions, distinct pages, | ||
| -- comments) — non-negative by construction. A negative value is a | ||
| -- regression in the gold model, not a data condition. | ||
| SELECT | ||
| measure_key, | ||
| count() AS row_count | ||
| FROM {{ ref('wiki_metric_observations') }} | ||
| WHERE value < 0 | ||
| GROUP BY measure_key |
11 changes: 11 additions & 0 deletions
11
src/ingestion/dbt/tests/gold/assert_wiki_observations_subject_key_shape.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| -- Build-integrity check (untagged → error severity under `dbt build`). | ||
| -- The wiki family has no distinct-count measure, so the model is a single | ||
| -- value UNION branch that stamps subject_key = NULL on every row. A | ||
| -- non-NULL subject_key means a measure branch drifted into a distinct-count | ||
| -- shape — nothing downstream would count it, so it must never appear. | ||
| SELECT | ||
| measure_key, | ||
| countIf(subject_key IS NOT NULL) AS subject_rows | ||
| FROM {{ ref('wiki_metric_observations') }} | ||
| GROUP BY measure_key | ||
| HAVING subject_rows > 0 |
16 changes: 16 additions & 0 deletions
16
src/ingestion/dbt/tests/gold/assert_wiki_observations_unique_grain.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| -- Build-integrity check (untagged → error severity under `dbt build`). | ||
| -- All four wiki measures are day-grain sums: exactly one row per (tenant, | ||
| -- entity, date, measure, dimensions, subject). A duplicate means FINAL dedup | ||
| -- regressed on a class read or the engagement join fanned out, silently | ||
| -- inflating the sums. | ||
| SELECT | ||
| tenant_id, | ||
| entity_id, | ||
| metric_date, | ||
| measure_key, | ||
| dimensions, | ||
| subject_key, | ||
| count() AS row_count | ||
| FROM {{ ref('wiki_metric_observations') }} | ||
| GROUP BY tenant_id, entity_id, metric_date, measure_key, dimensions, subject_key | ||
| HAVING count() > 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| {{ config( | ||
| materialized='table', | ||
| engine='MergeTree', | ||
| order_by=['source_key', 'measure_key', 'entity_id', 'metric_date'], | ||
| schema='insight', | ||
| alias='wiki_metric_observations', | ||
| tags=['gold'] | ||
| ) }} | ||
|
|
||
| -- Source measure observations for the unified metrics runtime, wiki family. | ||
| -- Reads the wiki class contracts only (class_wiki_pages, class_wiki_activity, | ||
| -- class_wiki_engagement); every measure is emitted through the shape macros | ||
| -- in macros/metric_observation_measures.sql. No dimensions: wiki sources | ||
| -- (Confluence, Outline) feed one undifferentiated family. | ||
| -- | ||
| -- Materialized as a sorted table: the pipeline runs once per dbt build — | ||
| -- the only time the silver inputs can have changed — and the ordering key | ||
| -- mirrors the runtime's filter shape (source_key, measure_key, entity_id, | ||
| -- metric_date), so single-measure queries read index-pruned ranges. | ||
| -- | ||
| -- Grain per measure: | ||
| -- day-grain sums (creation date, page author): pages_created (1/page — | ||
| -- the page object's own author/created_at are the | ||
| -- canonical creation facts; a version-derived proxy | ||
| -- undercounts imported pages) | ||
| -- day-grain sums (edit date, version author): edits (logical edit | ||
| -- sessions — autosave bursts collapsed in silver, see | ||
| -- class_wiki_activity), pages_edited (distinct pages | ||
| -- touched that day) | ||
| -- day-grain sums (comment date, page author): comments (engagement | ||
| -- RECEIVED on the person's pages — footer + inline + | ||
| -- replies; the commenter is deliberately not the entity, | ||
| -- see class_wiki_engagement's page-centric design note) | ||
| -- | ||
| -- Attribution: entity_id = lower(author_email); only email-shaped keys pass. | ||
| -- Confluence resolves emails through the Jira directory join in staging and | ||
| -- yields NULL on tenants without Jira — those rows are excluded as | ||
| -- unmatchable rather than carried as dead entities (cohorts and API requests | ||
| -- address people by email). Outline resolves from its own user stream. | ||
| -- | ||
| -- Memory shape: every class read keeps FINAL (ReplacingMergeTree dedup) over | ||
| -- a pruned column set. class_wiki_activity is already (author, day) grain, | ||
| -- so its sums are near-free. The single join in the model attributes | ||
| -- page-day comment rollups to the page author: engagement ⋈ pages on | ||
| -- (tenant_id, source_id, page_id) — source_id is part of the key so a | ||
| -- page_id colliding across two wiki instances of one tenant cannot fan out. | ||
| -- | ||
| -- Peer measurability (who enters a metric's peer pool) is decided HERE, by | ||
| -- row emission — the runtime never fabricates zeros. All four measures are | ||
| -- engagement-gated: a row exists only where the source recorded authorship, | ||
| -- editing, or received comments. Rostered-but-inactive people take no | ||
| -- standing rather than dragging peer medians toward zero. | ||
|
|
||
| WITH | ||
| -- Page -> author attribution for the comment join. No created_at gate: | ||
| -- comment attribution needs only the page's author, so a page snapshot | ||
| -- without a creation timestamp still receives its comments. | ||
| pages AS ( | ||
| SELECT | ||
| tenant_id, | ||
| source_id, | ||
| page_id, | ||
| lower(author_email) AS entity_id | ||
| FROM {{ ref('class_wiki_pages') }} FINAL | ||
| WHERE author_email LIKE '%@%' | ||
| ), | ||
| -- Dated branch for pages_created; the creation date is the metric date, so | ||
| -- only here does a missing created_at exclude the page. | ||
| page_creations AS ( | ||
| SELECT | ||
| tenant_id, | ||
| lower(author_email) AS entity_id, | ||
| toDate(created_at) AS metric_date, | ||
| CAST([] AS Array(Tuple(key String, value String, label Nullable(String)))) AS no_dimensions | ||
| FROM {{ ref('class_wiki_pages') }} FINAL | ||
| WHERE author_email LIKE '%@%' | ||
| AND created_at IS NOT NULL | ||
| ), | ||
| activity AS ( | ||
| SELECT | ||
| tenant_id, | ||
| lower(author_email) AS entity_id, | ||
| day AS metric_date, | ||
| total_edits, | ||
| pages_edited, | ||
| CAST([] AS Array(Tuple(key String, value String, label Nullable(String)))) AS no_dimensions | ||
| FROM {{ ref('class_wiki_activity') }} FINAL | ||
| WHERE author_email LIKE '%@%' | ||
| AND day IS NOT NULL | ||
| ), | ||
| engagement AS ( | ||
| SELECT | ||
| e.tenant_id AS tenant_id, | ||
| p.entity_id AS entity_id, | ||
| e.day AS metric_date, | ||
| e.total_comments AS total_comments, | ||
| CAST([] AS Array(Tuple(key String, value String, label Nullable(String)))) AS no_dimensions | ||
| FROM ( | ||
| SELECT | ||
| tenant_id, | ||
| source_id, | ||
| page_id, | ||
| day, | ||
| total_comments | ||
| FROM {{ ref('class_wiki_engagement') }} FINAL | ||
| WHERE day IS NOT NULL | ||
| ) AS e | ||
| INNER JOIN pages AS p | ||
| ON e.tenant_id = p.tenant_id | ||
| AND e.source_id = p.source_id | ||
| AND e.page_id = p.page_id | ||
| ), | ||
| value_measures AS ( | ||
| {{ sum_measure('pages_created', 'page_creations', '1', 'no_dimensions') }} | ||
|
|
||
| UNION ALL | ||
|
|
||
| {{ sum_measure('edits', 'activity', 'total_edits', 'no_dimensions') }} | ||
|
|
||
| UNION ALL | ||
|
|
||
| {{ sum_measure('pages_edited', 'activity', 'pages_edited', 'no_dimensions') }} | ||
|
|
||
| UNION ALL | ||
|
|
||
| {{ sum_measure('comments', 'engagement', 'total_comments', 'no_dimensions') }} | ||
| ) | ||
| SELECT | ||
| assumeNotNull(tenant_id) AS tenant_id, | ||
| 'wiki' AS source_key, | ||
| 'person' AS entity_type, | ||
| assumeNotNull(entity_id) AS entity_id, | ||
| assumeNotNull(metric_date) AS metric_date, | ||
| CAST(NULL AS Nullable(DateTime64(3))) AS observed_at, | ||
| measure_key, | ||
| value, | ||
| CAST(NULL AS Nullable(String)) AS subject_key, | ||
| dimensions | ||
| FROM value_measures | ||
| WHERE tenant_id IS NOT NULL | ||
| AND entity_id IS NOT NULL | ||
| AND metric_date IS NOT NULL | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.