Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
83d7f5a
feat(analytics): unified metrics registry and /v1/metric-results runtime
aleksdotbar Jul 5, 2026
7c2c24b
feat(ingestion): managed metric observation gold models for AI usage
aleksdotbar Jul 5, 2026
20a48fc
docs(metrics): metrics domain spec with authoring guide
aleksdotbar Jul 5, 2026
f57ac21
refactor(ingestion): shape macros for metric observation branches
aleksdotbar Jul 5, 2026
7ce547c
docs(metrics): align references with the analytics service name
aleksdotbar Jul 6, 2026
69fc0e3
docs(metrics): concepts section — observations, definitions, results
aleksdotbar Jul 6, 2026
c0aa189
Merge branch 'main' into feat/unified-metrics
aleksdotbar Jul 6, 2026
265e483
feat(analytics): explanation field on metric definitions
aleksdotbar Jul 6, 2026
5d00919
Merge branch 'feat/unified-metrics' of https://github.com/constructor…
aleksdotbar Jul 6, 2026
8b81434
refactor(analytics): typed metric seed registry
aleksdotbar Jul 6, 2026
5354b45
refactor(analytics): typed schema statuses and source kinds in the ru…
aleksdotbar Jul 6, 2026
6a763b7
fix(analytics): keep corrupt input classification over custom-SQL rows
aleksdotbar Jul 6, 2026
3a8dbf6
refactor(analytics): close computation vocabulary to executable sum/r…
aleksdotbar Jul 6, 2026
eef2dbd
Merge remote-tracking branch 'origin/main' into feat/unified-metrics
aleksdotbar Jul 6, 2026
50a03b8
refactor(analytics): drop warehouse tenant filtering from metric results
aleksdotbar Jul 7, 2026
fbf0b25
feat(analytics): suppress peer percentiles below minimum pool size
aleksdotbar Jul 7, 2026
340e94a
feat(analytics): peer pools include only observed values
aleksdotbar Jul 7, 2026
119a6ca
fix(analytics): validate peer cohort keys, backfill class labels, fai…
aleksdotbar Jul 7, 2026
b5dc233
Merge remote-tracking branch 'origin/main' into feat/unified-metrics
aleksdotbar Jul 7, 2026
9c52454
Merge branch 'main' into feat/unified-metrics
aleksdotbar Jul 7, 2026
2866dc6
fix(analytics): duplicate-proof peer disclosure and pinned join seman…
aleksdotbar Jul 7, 2026
5361166
feat(analytics): periodic metric definition validation sweep
aleksdotbar Jul 7, 2026
deab55e
Merge branch 'feat/unified-metrics' of https://github.com/constructor…
aleksdotbar Jul 7, 2026
70b4ebd
feat(ingestion): build gold models at deploy in the ClickHouse migrat…
aleksdotbar Jul 7, 2026
ba479fd
feat(ingestion): backfill AI class-contract history via semver bumps …
aleksdotbar Jul 7, 2026
27ce88b
fix(analytics): restore canonical form of committed OpenAPI doc
aleksdotbar Jul 7, 2026
b05e5f9
Merge branch 'main' into feat/unified-metrics
aleksdotbar Jul 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,6 @@ cf-studio-path = ".cf-studio"
ALWAYS resolve and enforce prerequisites of skills/workflows/commands BEFORE applying user intent.
<!-- /@cf:root-agents -->

## Project Rules

ALWAYS open and follow `docs/domain/metrics/specs/DESIGN.md` WHEN adding or changing metrics, metric definitions, metric seeds, observation sources, or gold metric views
7 changes: 7 additions & 0 deletions charts/insight/templates/clickhouse-migrate-job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,13 @@ Idempotent: migrations are re-run on every upgrade and are CREATE OR
REPLACE / IF NOT EXISTS; placeholders are guarded by `ch_table_exists`.
No migration ledger — same contract as the legacy init.sh path.

The script's final step builds the dbt gold models (`dbt run --select
tag:gold`) so dbt-owned views exist right after deploy instead of after
the first connector sync — the analytics service reports metrics as
unavailable while an observation view is missing. The toolbox image
already bundles the dbt project; placeholders guarantee the view DDL
type-checks on a fresh cluster.

Gated on `clickhouse.runMigrations` (default true) AND the availability of
`ingestion.toolboxImage` (the image that ships the migration SQL).
*/}}
Expand Down
93 changes: 93 additions & 0 deletions docs/components/backend/analytics/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -1855,6 +1855,99 @@
"summary": "List queryable columns for a table"
}
},
"/v1/metric-results": {
"post": {
"operationId": "analytics_api.metric_results.create",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"type": "object"
}
}
},
"description": "Metric results"
},
"400": {
"content": {
"application/problem+json": {
"schema": {
"$ref": "#/components/schemas/Problem"
}
}
},
"description": "Bad Request"
},
"401": {
"content": {
"application/problem+json": {
"schema": {
"$ref": "#/components/schemas/Problem"
}
}
},
"description": "Unauthorized"
},
"403": {
"content": {
"application/problem+json": {
"schema": {
"$ref": "#/components/schemas/Problem"
}
}
},
"description": "Forbidden"
},
"404": {
"content": {
"application/problem+json": {
"schema": {
"$ref": "#/components/schemas/Problem"
}
}
},
"description": "Not Found"
},
"409": {
"content": {
"application/problem+json": {
"schema": {
"$ref": "#/components/schemas/Problem"
}
}
},
"description": "Conflict"
},
"429": {
"content": {
"application/problem+json": {
"schema": {
"$ref": "#/components/schemas/Problem"
}
}
},
"description": "Too Many Requests"
},
"500": {
"content": {
"application/problem+json": {
"schema": {
"$ref": "#/components/schemas/Problem"
}
}
},
"description": "Internal Server Error"
}
},
"security": [
{
"bearerAuth": []
}
],
"summary": "Compute metric results"
}
},
"/v1/metrics": {
"get": {
"operationId": "analytics_api.metrics.list",
Expand Down
1 change: 1 addition & 0 deletions docs/domain/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ Domain-level specifications for the Insight platform. Each domain represents a b
| [`ingestion/`](ingestion/) | Data pipeline from source APIs to Silver step 1 (Airbyte + Argo Workflows + dbt) | Accepted |
| [`connector/`](connector/) | Connector development: Insight Connector packages, nocode and CDK patterns, packaging, debugging | Accepted |
| [`identity-resolution/`](identity-resolution/) | Person identity matching and resolution across sources | Proposed |
| [`metrics/`](metrics/) | Unified metrics: typed registry + generic result runtime over source measure observations (`/v1/metric-results`) | Accepted |
78 changes: 78 additions & 0 deletions docs/domain/metrics/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Metrics Domain

The unified metrics system: metrics are defined once in a typed registry,
computed by one generic runtime over normalized source measure observations,
and served self-describing through `POST /v1/metric-results`. All new metrics
are authored through this system.

## Concepts

Three layers, one handshake in the middle.

An **observation** is a recorded fact: "alice, June 3rd, Claude Code,
`accepted_edit_actions` = 17". It says what happened, to whom, when, and how
much — and deliberately nothing about what it means. No name a user would
see, no formula, no chart. The data side (dbt gold models over silver
classes) produces millions of these in one fixed row shape, and its only job
is to record them honestly.

A **definition** is a catalog card holding meaning: "there is a metric
`ai.tool_acceptance_rate`; compute it as `accepted_edit_actions` divided by
`tool_use_offered`, times 100; show as percent; higher is better; may be
split by tool; compare within org unit". No data lives here — only meaning
and instructions, stored in the registry and authored as one Rust struct per
metric.

A **metric result** is the computed answer the user sees. It is not stored
anywhere: at request time the runtime applies a definition to the matching
observations — "alice, January: 312 ÷ 405 = 77%" — and returns it labeled and
ready to render.

The split exists because one fact serves many meanings and one meaning serves
many questions. The same `accepted_edit_actions` observation is the whole
value of `ai.accepted_edit_actions` and the numerator of
`ai.tool_acceptance_rate` — recorded once, interpreted twice. The same
definition answers any period, person, team, dimension split, or peer
comparison without new code. Each side changes without touching the other:
renaming a metric edits a card; a vendor API change fixes fact recording
while every card keeps working.

The handshake is the source measure observation contract (see
[`specs/DESIGN.md`](specs/DESIGN.md)): the data side promises to emit facts
in that shape, definitions reference facts only by measure key, and the
runtime can therefore connect any definition to any matching facts without
either side knowing the other exists.

| | Observation | Definition | Metric result |
|---|---|---|---|
| What | a fact | the meaning of facts | the computed answer |
| Lives | ClickHouse views over silver (computed on read, nothing stored) | registry (MariaDB, seeded from Rust) | nowhere — made per request |
| Knows | what happened | what it is called, how to compute, how to show | both, combined |
| Authored by | connector + gold model | one struct per metric | nobody — the runtime derives it |

## Documents

| Document | Description |
|---|---|
| [`specs/DESIGN.md`](specs/DESIGN.md) | System contract: observation contract, registry model, computations, result API, validation, authoring guide ("Adding a Metric") |

## Implementation

| Layer | Location |
|---|---|
| Metric registry (builtin seeds) | [`src/backend/services/analytics/src/domain/metric_definitions/builtin.rs`](../../../src/backend/services/analytics/src/domain/metric_definitions/builtin.rs) |
| Definition loading, reconciler, schema validator | [`src/backend/services/analytics/src/domain/metric_definitions/`](../../../src/backend/services/analytics/src/domain/metric_definitions/) |
| Result runtime (validation, query compiler, response builder) | [`src/backend/services/analytics/src/domain/metric_results/`](../../../src/backend/services/analytics/src/domain/metric_results/) |
| Result endpoint | [`src/backend/services/analytics/src/api/metric_results.rs`](../../../src/backend/services/analytics/src/api/metric_results.rs) |
| Registry schema migration | [`src/backend/services/analytics/src/migration/m20260625_000001_metric_definitions.rs`](../../../src/backend/services/analytics/src/migration/m20260625_000001_metric_definitions.rs) |
| Managed observation sources (dbt gold models) | [`src/ingestion/gold/`](../../../src/ingestion/gold/) |
| Class-contract data-quality tests | [`src/ingestion/dbt/tests/ai/`](../../../src/ingestion/dbt/tests/ai/) |

## Boundaries

- The AI class contracts feeding the observation models are documented in
[`src/ingestion/silver/ai/schema.yml`](../../../src/ingestion/silver/ai/schema.yml)
(activity invariant, label and conversation-count semantics).
- The legacy metric path ([`metric-catalog/`](../metric-catalog/) +
ad-hoc `insight.*` gold views) is frozen for new metrics and remains only
until its consumers migrate.
Loading
Loading