test(stand): cover metric drilldown for every metric

docs/TESTING.md

@@ -112,18 +112,18 @@ cd src/ingestion/tests/e2e
   **shallow acceptance validation** runs in **Beta**.
 - Every user-facing surface **should** have at least one smoke assertion.
 - A separate **compose-stand suite** (`tests/stand`, documented in `tests/stand/README.md`) drives a real Keycloak
-  login and four browser journeys against the SPA, plus an API-contract suite — all against a local
+  login and a set of browser journeys against the SPA, plus an API-contract suite — all against a local
   `docker-compose` stand seeded deterministically for tests (`deploy/seed`). Run it with
-  `./dev-compose.sh test-stand up|test|down`. It asserts no metric VALUE: the seed's `golden_metrics` is empty by
-  design, and a harness for it is being migrated separately.
+  `./dev-compose.sh test-stand up|test|down`. It asserts no metric VALUE against a declared expectation: the seed's
+  `golden_metrics` is empty by design, and a harness for it is being migrated separately. It does reconcile every
+  metric's drilldown evidence against that metric's own served value, which needs no declared expectation.
 
 **CI:** `functional-k3s.yml` — ephemeral k3d install. Today it only *installs*; a real smoke must build + import the
 PR's images and assert `/health` + a few golden metrics.
 
-**CI:** `e2e-stand.yml` — two **non-required** checks against the compose-stand suite: `api-smoke` (117 HTTP
-contract tests, no browser) and `ui-journeys` (10 tests: the four browser journeys, run inside the published
-`ui-tests` image). Neither blocks merge — both stand up a full stack against a live IdP and their flake rate is
-still unmeasured.
+**CI:** `e2e-stand.yml` — two **non-required** checks against the compose-stand suite: `api-smoke` (the HTTP
+contract tests, no browser) and `ui-journeys` (the browser journeys, run inside the published `ui-tests` image).
+Neither blocks merge — both stand up a full stack against a live IdP and their flake rate is still unmeasured.
 
 ---
 

src/backend/services/analytics/src/domain/metric_definitions/builtin.rs

@@ -203,6 +203,19 @@ mod tests {
         }
     }
 
+    #[test]
+    fn evidence_refs_parse_as_evidence_relations() {
+        use crate::domain::metric_definitions::definition::EvidenceRelation;
+        for builtin_source in builtin_sources() {
+            assert!(
+                EvidenceRelation::parse(&builtin_source.source.evidence_ref).is_some(),
+                "builtin source {} declares an invalid evidence relation {:?}",
+                builtin_source.source.key,
+                builtin_source.source.evidence_ref,
+            );
+        }
+    }
+
     #[test]
     fn every_source_declares_at_least_one_measure() {
         for builtin_source in builtin_sources() {

tests/stand/README.md

@@ -1,7 +1,7 @@
 # The compose-stand suite
 
-Deployed-stand tests for Insight: a real Keycloak login, four browser
-journeys against the SPA, and an API-contract suite — all run against a local
+Deployed-stand tests for Insight: a real Keycloak login, browser journeys
+against the SPA, and an API-contract suite — all run against a local
 `docker-compose` stand seeded deterministically for tests (`deploy/seed`).
 
 This suite assumes an **already-running, already-seeded** stand. It never
@@ -20,7 +20,7 @@ persona resolution) lives in `../lib`; both are one uv project
 | `api/analytics/` | The `/api/analytics` prefix, one module per path group. |
 | `api/identity/` | The `/api/identity` prefix, one module per concern. |
 | `api/test_gateway.py` | Neither service — the edge, sweeping 401 over every catalogued operation at once. |
-| `ui/` | The four browser journeys, plus `ui/pages/` (page objects). |
+| `ui/` | The browser journeys, plus `ui/pages/` (page objects). |
 
 Split by service because that is the axis along which a test's setup
 differs: identity's answers depend on **who is asking** (the org chart, the
@@ -76,6 +76,13 @@ disagree. Before adding a test, read it for:
   test here asserts a metric's exact value, and none should until the table
   has entries — reading a number off a running stand and asserting it back
   only proves that the code which produced it produced it.
+
+  What `api/analytics/test_drilldown.py` does is a different thing and is
+  allowed: it asks two independent serving relations the same question — the
+  evidence rows behind a metric, and the metric's own value — and requires them
+  to agree. Neither side is a number typed into the test, so the seed can change
+  underneath it, and a disagreement is a real defect rather than a stale
+  expectation. `drilldown_matrix.py` states what "agree" means per metric.
 - **capabilities** — e.g. `ingestion`, which this stand does not have
   (compose seeds silver/gold directly). A test that needs a capability the
   stand may lack should carry the matching marker (below), not assume it.
@@ -118,7 +125,7 @@ more API test than one more browser test whenever the two would prove the
 same thing.
 
 State the reason as a paragraph in the test module's docstring, in the
-shape the four shipped journeys already use — for example
+shape the shipped journeys already use — for example
 `ui/test_logged_out_access_refused.py`:
 
 > Why this is a browser test and not an API test, measured rather than

tests/stand/api/analytics/drilldown_matrix.py

@@ -0,0 +1,201 @@
+"""What each metric's evidence must add up to, one row per metric key.
+
+Every metric in the registry is drilldown-declared: a source carries an
+`evidence_ref` and a measure carries an `evidence_granularity`, both mandatory,
+so capability is decided at run time from the health of the evidence relation
+rather than per metric. Sweeping the catalogue is therefore the only way to
+notice that one source's evidence has drifted from the observations derived from
+it — the modal would show wrong numbers with every other test still green.
+
+The sweep needs an expectation per metric, and it cannot be one rule: an
+evidence row means something different at each granularity, and the period
+scalar is a different aggregate per computation. `Tier` is that expectation, and
+each variant is the STRONGEST statement provable from the serving path:
+
+* observations are derived from evidence by the gold models, under the same
+  scope predicate the drilldown compiler uses, so the two sides are the same
+  rows aggregated twice;
+* rows a person's identity did not resolve to reach neither side, so the two
+  cannot disagree about which rows exist;
+* `max`/`min` collapse across one person's several source accounts is the one
+  place they legitimately can, which is why those metrics get an inequality and
+  not an equality.
+
+Kept as a literal rather than derived from a live response so that collection
+stays offline, matching the rest of this suite, and so that adding a metric
+without deciding what its evidence means is a failure rather than a silent gap.
+`test_every_metric_definition_is_in_the_drilldown_matrix` is what enforces that.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from enum import StrEnum
+
+
+class Tier(StrEnum):
+    """The reconciliation a metric's evidence supports."""
+
+    #: Event grain whose contribution is a constant 1 and which projects no
+    #: value column: one row IS one unit of the metric.
+    EXACT_COUNT = "exact_count"
+    #: Additive measure summed into observations: the projected values sum to
+    #: the period scalar.
+    EXACT_SUM = "exact_sum"
+    #: Median over event rows passed through to observations one-for-one, so
+    #: the scalar is one of the projected values.
+    EXACT_MEDIAN = "exact_median"
+    #: Median over a measure whose value column is not projected, recoverable
+    #: from the detail columns it was computed from.
+    DERIVED_MEDIAN = "derived_median"
+    #: Ratio of two additive measures: the summed numerator over the summed
+    #: denominator, scaled, then transformed.
+    EXACT_RATIO = "exact_ratio"
+    #: Distinct count whose counted subject is the day itself.
+    EXACT_DISTINCT_DATES = "exact_distinct_dates"
+    #: Sum of a day flag that collapses by `max` or `min` across a person's
+    #: source accounts: evidence sums to at least the period scalar.
+    COLLAPSE_BOUNDED_SUM = "collapse_bounded_sum"
+    #: Ratio whose denominator is such a day flag: the evidence ratio is at
+    #: most the period scalar.
+    COLLAPSE_BOUNDED_RATIO = "collapse_bounded_ratio"
+    #: Distinct count whose counted subject is projected nowhere; only the
+    #: bound `1 <= period <= rows` survives.
+    STRUCTURAL_ONLY = "structural_only"
+
+
+@dataclass(frozen=True)
+class Transform:
+    """The affine-and-clamp a definition applies AFTER aggregation.
+
+    It reaches no response — `scale` is on the wire but this is not — so
+    reconciling a clamped metric means applying it here, in the same order the
+    SQL does, and a metric pinned at its bound compares exactly.
+    """
+
+    multiplier: float = 1.0
+    offset: float = 0.0
+    clamp_min: float | None = None
+    clamp_max: float | None = None
+
+    def apply(self, value: float) -> float:
+        transformed = self.multiplier * value + self.offset
+        if self.clamp_min is not None:
+            transformed = max(transformed, self.clamp_min)
+        if self.clamp_max is not None:
+            transformed = min(transformed, self.clamp_max)
+        return transformed
+
+
+@dataclass(frozen=True)
+class Expectation:
+    """One metric's drilldown expectation.
+
+    `source` is the evidence family, and it is what makes a failure readable:
+    capability and schema health are per source, so a whole family failing at
+    once is a different defect from one metric failing alone.
+    """
+
+    metric_key: str
+    source: str
+    tier: Tier
+    #: Ratio scale, from the definition's computation.
+    scale: float | None = None
+    transform: Transform | None = None
+    #: Detail columns a `DERIVED_MEDIAN` metric's value is the sum of.
+    derived_from: tuple[str, ...] = ()
+
+
+_PERCENT = Transform(clamp_max=100.0)
+_INVERTED_PERCENT = Transform(multiplier=-1.0, offset=100.0, clamp_min=0.0, clamp_max=100.0)
+
+MATRIX: Sequence[Expectation] = (
+    Expectation("ai.accepted_edit_actions", "ai", Tier.EXACT_SUM),
+    Expectation("ai.accepted_lines", "ai", Tier.EXACT_SUM),
+    Expectation("ai.active_days", "ai", Tier.COLLAPSE_BOUNDED_SUM),
+    Expectation("ai.assistant_actions", "ai", Tier.EXACT_SUM),
+    Expectation("ai.assistant_messages", "ai", Tier.EXACT_SUM),
+    Expectation("ai.chat_assistant_conversations", "ai", Tier.EXACT_SUM),
+    Expectation("ai.cost", "ai", Tier.EXACT_SUM),
+    Expectation("ai.dev_conversations", "ai", Tier.EXACT_SUM),
+    Expectation("ai.removed_lines", "ai", Tier.EXACT_SUM),
+    Expectation("ai.tool_acceptance_rate", "ai", Tier.EXACT_RATIO, scale=100.0),
+    Expectation("collab.active_days", "collab", Tier.EXACT_DISTINCT_DATES),
+    Expectation("collab.adhoc_meetings", "collab", Tier.EXACT_SUM),
+    Expectation("collab.breadth", "collab", Tier.STRUCTURAL_ONLY),
+    Expectation("collab.channel_posts", "collab", Tier.EXACT_SUM),
+    Expectation("collab.dm_ratio", "collab", Tier.EXACT_RATIO, scale=100.0),
+    Expectation("collab.emails_read", "collab", Tier.EXACT_SUM),
+    Expectation("collab.emails_received", "collab", Tier.EXACT_SUM),
+    Expectation("collab.emails_sent", "collab", Tier.EXACT_SUM),
+    Expectation("collab.files_engaged", "collab", Tier.EXACT_SUM),
+    Expectation("collab.files_shared", "collab", Tier.EXACT_SUM),
+    Expectation("collab.files_shared_external", "collab", Tier.EXACT_SUM),
+    Expectation("collab.files_shared_internal", "collab", Tier.EXACT_SUM),
+    Expectation("collab.focus_time_pct", "collab", Tier.EXACT_RATIO, scale=100.0),
+    Expectation("collab.meeting_free_days", "collab", Tier.COLLAPSE_BOUNDED_SUM),
+    Expectation("collab.meeting_hours", "collab", Tier.EXACT_SUM),
+    Expectation("collab.meetings_count", "collab", Tier.EXACT_SUM),
+    Expectation("collab.meetings_organized", "collab", Tier.EXACT_SUM),
+    Expectation("collab.messages_sent", "collab", Tier.EXACT_SUM),
+    Expectation("collab.msgs_per_active_day", "collab", Tier.COLLAPSE_BOUNDED_RATIO, scale=1.0),
+    Expectation("collab.scheduled_meetings", "collab", Tier.EXACT_SUM),
+    Expectation("git.code_lines", "git", Tier.EXACT_SUM),
+    Expectation(
+        "git.commit_size",
+        "git",
+        Tier.DERIVED_MEDIAN,
+        derived_from=("lines_added", "lines_removed"),
+    ),
+    Expectation("git.commits", "git", Tier.EXACT_COUNT),
+    Expectation("git.commits_per_active_day", "git", Tier.COLLAPSE_BOUNDED_RATIO, scale=1.0),
+    Expectation("git.lines_added", "git", Tier.EXACT_SUM),
+    Expectation("git.lines_removed", "git", Tier.EXACT_SUM),
+    Expectation("git.merge_rate", "git", Tier.EXACT_RATIO, scale=100.0),
+    Expectation("git.pr_cycle_time_h", "git", Tier.EXACT_MEDIAN),
+    Expectation("git.pr_size", "git", Tier.EXACT_MEDIAN),
+    Expectation("git.prs_created", "git", Tier.EXACT_COUNT),
+    Expectation("git.prs_merged", "git", Tier.EXACT_COUNT),
+    Expectation("tasks.avg_slip", "task", Tier.EXACT_RATIO, scale=1.0),
+    Expectation("tasks.bugs_fixed", "task", Tier.EXACT_COUNT),
+    Expectation("tasks.bugs_ratio", "task", Tier.EXACT_RATIO, scale=100.0),
+    Expectation("tasks.closed", "task", Tier.EXACT_COUNT),
+    Expectation("tasks.dev_time", "task", Tier.EXACT_MEDIAN),
+    Expectation("tasks.due_date_compliance", "task", Tier.EXACT_RATIO, scale=100.0),
+    Expectation(
+        "tasks.estimation_accuracy",
+        "task",
+        Tier.EXACT_RATIO,
+        scale=1.0,
+        transform=_INVERTED_PERCENT,
+    ),
+    Expectation("tasks.flow_efficiency", "task", Tier.EXACT_RATIO, scale=100.0, transform=_PERCENT),
+    Expectation("tasks.on_time_delivery", "task", Tier.EXACT_RATIO, scale=100.0),
+    Expectation("tasks.pickup_time", "task", Tier.EXACT_MEDIAN),
+    Expectation("tasks.reopen_rate", "task", Tier.EXACT_RATIO, scale=100.0),
+    Expectation("tasks.resolution_time", "task", Tier.EXACT_MEDIAN),
+    Expectation("tasks.stale_in_progress", "task", Tier.EXACT_SUM),
+    Expectation(
+        "tasks.worklog_accuracy", "task", Tier.EXACT_RATIO, scale=100.0, transform=_PERCENT
+    ),
+    Expectation("wiki.comments", "wiki", Tier.EXACT_SUM),
+    Expectation("wiki.edits", "wiki", Tier.EXACT_SUM),
+    Expectation("wiki.pages_created", "wiki", Tier.EXACT_COUNT),
+    Expectation("wiki.pages_edited", "wiki", Tier.EXACT_SUM),
+)
+
+#: One metric per distinct evidence presentation, plus the capable-but-empty
+#: case. A presentation is all an export can differ by — the column set and the
+#: header labels are everything it serializes — and every other metric in the
+#: catalogue reuses one of these, so exporting the whole catalogue would repeat
+#: these answers rather than add any.
+EXPORT_SHAPES: Sequence[str] = (
+    "git.prs_created",
+    "git.pr_cycle_time_h",
+    "tasks.closed",
+    "tasks.dev_time",
+    "git.merge_rate",
+    "collab.messages_sent",
+    "wiki.pages_created",
+)