feat(metrics): add metric evidence drilldowns

docs/domain/metrics/README.md

@@ -28,6 +28,14 @@ anywhere: at request time the runtime applies a definition to the matching
 observations — "alice, January: 312 ÷ 405 = 77%" — and returns it labeled and
 ready to render.
 
+Metric **evidence** is the source-level population behind that answer. Each
+managed source exposes a normalized serving table.
+Definitions inherit drilldown support when all compatible inputs are backed by
+the same validated evidence relation, so adding a metric over an existing
+measure requires no drilldown-specific configuration.
+The backend applies the same entity, period, dimension, input-role, and
+computation semantics used by metric results.
+
 The split exists because one fact serves many meanings and one meaning serves
 many questions. The same `accepted_edit_actions` observation is the whole
 value of `ai.accepted_edit_actions` and the numerator of
@@ -43,12 +51,12 @@ in that shape, definitions reference facts only by measure key, and the
 runtime can therefore connect any definition to any matching facts without
 either side knowing the other exists.
 
-| | Observation | Definition | Metric result |
-|---|---|---|---|
-| What | a fact | the meaning of facts | the computed answer |
-| Lives | ClickHouse views over silver (computed on read, nothing stored) | registry (MariaDB, seeded from Rust) | nowhere — made per request |
-| Knows | what happened | what it is called, how to compute, how to show | both, combined |
-| Authored by | connector + gold model | one struct per metric | nobody — the runtime derives it |
+| | Observation | Evidence | Definition | Metric result |
+|---|---|---|---|---|
+| What | an aggregate-ready fact | the participating source population | the meaning of facts | the computed answer |
+| Lives | ClickHouse views or tables over silver | ClickHouse serving tables | registry | nowhere |
+| Knows | what happened numerically | which records participated | how to compute and display | all three, combined |
+| Authored by | connector + gold model | connector + gold model | one struct per metric | nobody |
 
 ## Documents
 
@@ -64,12 +72,19 @@ either side knowing the other exists.
 | Definition loading, reconciler, schema validator | [`src/backend/services/analytics/src/domain/metric_definitions/`](../../../src/backend/services/analytics/src/domain/metric_definitions/) |
 | Result runtime (validation, query compiler, response builder) | [`src/backend/services/analytics/src/domain/metric_results/`](../../../src/backend/services/analytics/src/domain/metric_results/) |
 | Result endpoint | [`src/backend/services/analytics/src/api/metric_results.rs`](../../../src/backend/services/analytics/src/api/metric_results.rs) |
+| Drilldown runtime | [`src/backend/services/analytics/src/domain/metric_drilldown/`](../../../src/backend/services/analytics/src/domain/metric_drilldown/) |
+| Drilldown endpoints | [`src/backend/services/analytics/src/api/metric_drilldown.rs`](../../../src/backend/services/analytics/src/api/metric_drilldown.rs) |
 | Registry schema migration | [`src/backend/services/analytics/src/migration/m20260625_000001_metric_definitions.rs`](../../../src/backend/services/analytics/src/migration/m20260625_000001_metric_definitions.rs) |
 | Managed observation sources (dbt gold models) | [`src/ingestion/gold/`](../../../src/ingestion/gold/) |
 | Class-contract data-quality tests | [`src/ingestion/dbt/tests/ai/`](../../../src/ingestion/dbt/tests/ai/) |
 
 ## Boundaries
 
+- Current deployments isolate one tenant per instance. Drilldown entity IDs
+  remain source-derived identifiers, commonly normalized email addresses.
+  Multi-tenant warehouse predicates, canonical person IDs, cross-source alias
+  resolution, and subordinate authorization belong to the identity-resolution
+  epic and are required before a multi-tenant instance enables drilldown.
 - The AI class contracts feeding the observation models are documented in
   [`src/ingestion/silver/ai/schema.yml`](../../../src/ingestion/silver/ai/schema.yml)
   (activity invariant, label and conversation-count semantics).

docs/domain/metrics/specs/DESIGN.md

@@ -59,7 +59,8 @@ Rules:
 ## Managed Source Ownership
 
 Managed observation sources and the cohort view are dbt gold models
-(`src/ingestion/gold/`), materialized as views in the `insight` database:
+(`src/ingestion/gold/`), materialized as views or MergeTree serving tables in
+the `insight` database:
 
 - `insight.ai_metric_observations`
 - `insight.metric_entity_cohorts_current`
@@ -81,7 +82,7 @@ contract; a source that needs different columns is a different source kind.
 
 Gold models are built at deploy time by the ClickHouse migrate hook
 (`dbt run --select tag:gold`, final step of
-`src/ingestion/scripts/apply-ch-migrations.sh`), so the views exist before
+`src/ingestion/scripts/apply-ch-migrations.sh`), so the relations exist before
 any connector sync — bronze/silver placeholders guarantee the DDL
 type-checks on a fresh cluster. Per-connector scoped dbt runs keep them
 current afterwards.
@@ -90,6 +91,104 @@ The cohort view is unique per `(tenant_id, entity_type, entity_id,
 cohort_key)`. The peer query relies on this; a dbt build-integrity test
 asserts it.
 
+## Metric Evidence Contract
+
+Each managed source may expose `<family>_metric_evidence` in the `insight`
+database:
+
+```sql
+tenant_id String,
+source_key String,
+entity_type String,
+entity_id String,
+metric_date Date,
+observed_at Nullable(DateTime64(3)),
+measure_key String,
+record_id String,
+record_kind String,
+granularity String,
+record_label String,
+contribution Nullable(Float64),
+subject_key Nullable(String),
+dimensions Array(Tuple(key String, value String, label Nullable(String))),
+details Map(String, String)
+```
+
+Evidence relations are MergeTree serving tables built from silver. Their
+ordering follows the drilldown predicate and cursor access pattern, avoiding
+repeated silver reconstruction for every page and export. Observation models
+derive their values from these evidence tables. The registry stores one
+evidence relation per source and one granularity per measure:
+
+- `event`: one source event, such as a commit.
+- `source_summary`: the finest summary preserved by silver.
+- `derived_population`: a source entity participating in a derived metric.
+
+Definitions do not declare a separate drilldown strategy. The runtime resolves
+the definition's existing input roles and source measures, requires every input
+to use the same evidence relation, and compiles the evidence selection from
+that metadata. A new metric over existing evidence-backed measures therefore
+inherits drilldown without metric-specific SQL, backend branches, or frontend
+configuration.
+
+The schema validator probes every standard column. Drilldown capability is
+absent until the probe is definitively healthy and every metric input has
+granularity metadata. Missing, unchecked, or invalid evidence fails closed.
+`POST /v1/metric-results` and `GET /v1/metric-definitions` expose that
+capability; consumers omit evidence actions when it is absent.
+
+The evidence runtime owns presentation. It projects the internal contract into
+typed human-facing columns rather than exposing `record_kind`, input role,
+dimensions, or other storage fields directly:
+
+- source-summary and derived-population measures default to date plus value.
+- event measures declare reusable detail keys by `(source_key, measure_key)`,
+  such as ref, title, repository, author, or issue type.
+- selected chart dimensions are added from the typed `dimensions` array.
+- ratio metrics return daily columns named after their numerator and
+  denominator measures instead of an ambiguous value column.
+- unknown detail keys are humanized and treated as strings; fields requiring
+  another label or type are added to the centralized presentation registry.
+
+Presentation is source-measure metadata in runtime code today. It is declared
+once per reusable measure shape, never once per metric and never in frontend
+configuration.
+
+`POST /v1/metric-drilldown` accepts one metric, one person entity, a period,
+declared dimension filters, and an encoded continuation cursor. It returns the
+canonical selection, typed server-owned columns, projected evidence rows, and
+a next cursor. Ordering is ascending over the complete evidence key. The
+cursor is versioned and bound to the normalized selection and request tenant.
+It is not an authorization token and modifying its ordering key cannot widen
+the server-owned relation or selection.
+
+`POST /v1/metric-drilldown/export` produces the complete selected population
+with the same projected columns as CSV or XLSX. Export is server-side and
+rejects results exceeding its row, byte, cell, execution-time, or concurrency
+limits. It never silently truncates.
+
+The evidence contract has these limitations:
+
+- Summary-grain silver cannot produce event-grain evidence. AI and
+  collaboration currently expose source summaries or derived populations.
+  Git exposes commit and pull-request events, task duration metrics expose
+  issue events, and wiki page creation exposes page events; their remaining
+  measures use the finest summary grain preserved by silver.
+- Metric results and evidence are not transactionally snapshot-isolated from
+  each other during a dbt rebuild. They reconcile after the complete gold
+  build because observations derive from evidence.
+- Pagination and each export are bound to the evidence table UUID. A rebuild
+  during the operation fails with `EVIDENCE_SNAPSHOT_EXPIRED`; the client must
+  restart the selection rather than mix rows from two builds. Previous table
+  snapshots are not retained.
+- Source links are omitted. Hosted services commonly use custom domains, and
+  the current silver contract does not preserve a canonical web base URL. A
+  future source registry can add a non-secret `web_base_url` keyed by source
+  instance and combine it with provider-specific record identifiers.
+- Drilldown preserves the existing metric entity and tenant behavior. This
+  change does not add identity-tree authorization or warehouse tenant
+  enforcement.
+
 ## Computations
 
 The computation vocabulary is closed and fully executable:
@@ -277,10 +376,20 @@ type MetricResult = {
   format: "integer" | "decimal" | "currency" | "percent"
   direction: "higher_is_better" | "lower_is_better" | "neutral"
   views: MetricResultView[]
+  selection: {
+    metric_key: string
+    entity: { type: string; ids: string[] }
+    period: { from: string; to: string }
+    filters: Array<{ dimension: string; values: string[] }>
+  }
+  drilldown?: {
+    granularity: Array<"event" | "source_summary" | "derived_population">
+  }
 } & (
   | { computation: "sum" }
   | { computation: "ratio"; scale: number }
   | { computation: "median" }
+  | { computation: "distinct_count" }
 )
 ```
 
@@ -424,8 +533,9 @@ one that applies.
 
 ### Case 1: metric over an existing measure
 
-The measure already appears in a managed observation source (check the
-`measures` list of the source in `builtin.rs` and the emitting gold model).
+The measure already appears in a managed source. Check the source's `measures`
+list in `builtin.rs`, the emitting evidence model, and the observation model
+derived from it.
 
 1. Add one `MetricSeed` to `BUILTIN_METRICS` in
    `src/backend/services/analytics/src/domain/metric_definitions/builtin.rs`:
@@ -435,54 +545,72 @@ The measure already appears in a managed observation source (check the
 2. Run `cargo test -p analytics` — the registry invariant tests validate
    key shapes, input/measure references, and computation field combinations.
 
-The reconciler seeds the definition on the next deploy. No SQL, no migration,
-no dbt change.
+The reconciler seeds the definition on the next deploy. If every input measure
+has healthy evidence metadata, the metric automatically receives drilldown,
+table, and CSV/XLSX export support. No drilldown-specific SQL, frontend
+configuration, migration, or dbt change is required.
 
 ### Case 2: new measure from an existing source
 
 The source exists but does not emit the measure yet.
 
-1. Add the measure branch to the source's gold model in `src/ingestion/gold/`:
-   one `UNION ALL` entry calling a shape macro from
-   `src/ingestion/dbt/macros/metric_observation_measures.sql` —
-   `sum_measure(measure_key, relation, value_expr, dimensions_col,
-   where=none)` for aggregated numerics, `presence_measure(measure_key,
-   relations)` for row-existence markers, `event_measure(measure_key,
-   relation, value_expr, dimensions_col, where=none)` for per-event values
-   feeding median metrics. Every branch is a shape-macro call; a new macro
-   is added only when a new computation kind becomes executable.
-   Read only class-contract columns; never vendor-specific ones — if the fact
-   you need is not in the class contract, extend the class contract first
-   (staging models declare semantics, see the class `schema.yml`).
-2. Add the `measure_key` to the gold model's `schema.yml` `accepted_values`
-   test.
-3. Add the measure key to the source's `measures` list in `builtin.rs`.
-4. Add the `MetricSeed` as in case 1.
-5. Validate: `dbt parse` + `cargo test -p analytics` (see Validation
+1. Add the measure to the source's `<family>_metric_evidence` model in
+   `src/ingestion/gold/`. Summary measures use the shared shape macros from
+   `src/ingestion/dbt/macros/metric_observation_measures.sql`;
+   event measures select stable source records into the evidence contract.
+   Read only class-contract columns; never vendor-specific ones. If the fact
+   is absent from the class contract, extend that contract first (staging
+   models declare semantics in their `schema.yml`).
+2. Choose the evidence granularity deliberately:
+   - emit one stable row per source record for `event`.
+   - emit the finest source-retained grouping for `source_summary`.
+   - emit the reconstructed participating population for
+     `derived_population`.
+   Event rows need deterministic `record_id` values and should place reusable
+   grouping fields in `dimensions` and human-facing fields in `details`.
+3. Derive the matching observation measure from the evidence relation. The
+   observation remains the aggregate-ready runtime input; the evidence row is
+   the population that explains it.
+4. Add the `measure_key` to the observation model's `schema.yml`
+   `accepted_values` test.
+5. Add the measure key to the source's `measures` list in `builtin.rs` and
+   classify its evidence granularity.
+6. Add or reuse a source-measure presentation rule when the default
+   date-plus-value table is insufficient. Declare detail keys there and add
+   explicit column metadata only for fields that are not humanized strings.
+7. Add the `MetricSeed` as in case 1.
+8. Validate: `dbt parse` + `cargo test -p analytics` (see Validation
    commands).
 
 ### Case 3: new observation source
 
 The metric family reads data no managed source covers.
 
-1. Create a dbt gold model in `src/ingestion/gold/` named
-   `<family>_metric_observations`, emitting the source measure observation
-   contract, `schema=insight`, `ref()`-ing silver models (medallion layering
-   rules: `docs/domain/ingestion-data-flow/specs/DESIGN.md`). Document columns
-   and measure keys in `src/ingestion/gold/schema.yml`.
+1. Create `<family>_metric_evidence` and
+   `<family>_metric_observations` dbt gold models in
+   `src/ingestion/gold/`, `schema=insight`, `ref()`-ing silver models
+   (medallion layering rules:
+   `docs/domain/ingestion-data-flow/specs/DESIGN.md`). The evidence model emits
+   the evidence contract; the observation model derives the aggregate-ready
+   observation contract from it. Document both in
+   `src/ingestion/gold/schema.yml`.
 2. Add a `BuiltinSource` (source + measures + dimensions) to `builtin.rs`,
-   with `source_ref` set to the relation name. No backend enum or table-name
-   code changes: the relation name is data, validated on load against the
-   `<family>_metric_observations` shape (`ObservationRelation`) and probed at
-   runtime by the schema validator.
-3. Add `MetricSeed`s as in case 1.
-4. Validate: `dbt parse` + `cargo test -p analytics` (see Validation
+   with `source_ref` and `evidence_ref` set to their relation names and every
+   measure assigned an evidence granularity. No backend enum or table-name
+   code changes are required: relation names are validated data and both
+   contracts are probed by the runtime schema validator.
+3. Add source-measure presentation rules for event shapes that need
+   human-facing detail columns.
+4. Add `MetricSeed`s as in case 1.
+5. Validate: `dbt parse` + `cargo test -p analytics` (see Validation
    commands). The runtime schema validator probes the new relation at
    startup.
 
 ### Rules that hold for every case
 
 - No metric-key-specific branches in runtime code.
+- Evidence presentation branches may depend on source and measure, never on
+  final metric key.
 - No vendor names, vendor columns, or label mappings in gold models — labels
   and taxonomy come from class-contract columns declared by staging.
 - Measure filter predicates (`where=` on shape macros) may reference only
@@ -540,6 +668,14 @@ Until one exists, custom definitions can be stored but cannot produce new source
 Frontend collection rendering:
 
 - requests metric keys and views.
+- treats the optional `drilldown` capability as the only evidence-action
+  switch; there is no frontend metric allowlist.
+- forwards the canonical metric selection returned by `/v1/metric-results`,
+  narrowing period and dimension filters for chart-point interactions.
+- renders server-owned typed evidence columns and rows without interpreting
+  the internal evidence contract.
+- uses the same canonical selection for table pagination and server-side
+  CSV/XLSX export.
 - treats configured required views as required.
 - normalizes response arrays only for local lookup.
 - renders using returned label, description, explanation, unit, format,