diff --git a/.cf-studio/config/artifacts.toml b/.cf-studio/config/artifacts.toml index 8d231dfd5..ecd728f9c 100644 --- a/.cf-studio/config/artifacts.toml +++ b/.cf-studio/config/artifacts.toml @@ -5,7 +5,7 @@ name = "InsightSpec" slug = "insightspec" kit = "sdlc" codebase = ["docs"] -children = ["identity-svc", "ir", "person", "orgchart", "presentation", "metric-cat", "dataflow"] +children = ["identity-svc", "ir", "person", "orgchart", "presentation", "metric-cat", "dataflow", "semantic-layer"] [[systems.artifacts]] kind = "DESIGN" @@ -815,6 +815,24 @@ path = "docs/domain/presentation-layer/specs/PRD.md" name = "Presentation Layer PRD" traceability = "DOCS-ONLY" +[[systems]] +name = "Semantic Layer" +slug = "semantic-layer" +kit = "sdlc" +codebase = ["docs/domain/semantic-layer", "src/backend/services/analytics"] +children = [] + +[[systems.artifacts]] +kind = "DESIGN" +path = "docs/domain/semantic-layer/specs/DESIGN.md" +name = "Semantic Layer Design" + +[[systems.artifacts]] +kind = "PRD" +path = "docs/domain/semantic-layer/specs/PRD.md" +name = "Semantic Layer PRD" +traceability = "DOCS-ONLY" + [[systems]] name = "Ingestion Data Flow" slug = "dataflow" diff --git a/docs/domain/semantic-layer/specs/DESIGN.md b/docs/domain/semantic-layer/specs/DESIGN.md new file mode 100644 index 000000000..b0a114964 --- /dev/null +++ b/docs/domain/semantic-layer/specs/DESIGN.md @@ -0,0 +1,788 @@ +# Technical Design — Semantic Layer + + + +- [1. Architecture Overview](#1-architecture-overview) + - [1.1 Architectural Vision](#11-architectural-vision) + - [1.2 Architecture Drivers](#12-architecture-drivers) + - [1.3 Architecture Layers](#13-architecture-layers) +- [2. Principles & Constraints](#2-principles--constraints) + - [2.1 Design Principles](#21-design-principles) + - [2.2 Constraints](#22-constraints) +- [3. Technical Architecture](#3-technical-architecture) + - [3.1 Domain Model](#31-domain-model) + - [3.2 Component Model](#32-component-model) + - [3.3 API Contracts](#33-api-contracts) + - [3.4 Internal Dependencies](#34-internal-dependencies) + - [3.5 External Dependencies](#35-external-dependencies) + - [3.6 Interactions & Sequences](#36-interactions--sequences) + - [3.7 Database Schemas & Tables](#37-database-schemas--tables) +- [4. Additional context](#4-additional-context) +- [5. Traceability](#5-traceability) + + + +- [ ] `p3` - **ID**: `cpt-semantic-layer-design-semantic-layer` + +## 1. Architecture Overview + +### 1.1 Architectural Vision + +The semantic layer is a single system through which every analytical value is defined, validated, computed, and served. Definitions — datasets, measures, metrics, charts, dashboards — are data. A single compiler turns a definition plus a request into warehouse SQL; storage, caching, and materialization are private implementation details behind the definition contract. Semantics are owned and executed by the server, and the frontend is a renderer of definitions. + +The domain model is a strict ladder: a product-owned dataset publishes a typed field catalog; a measure is a declarative aggregation of one dataset; a metric composes measures; charts and dashboards are pure presentation. Datasets carry the entire correctness burden of ingestion (deduplication, mutability, late data) so nothing above them re-solves it. The one place free-form SQL exists is the gated custom-dataset layer, which reads only dedup-safe catalog views and is wrapped from outside with tenancy and resource guardrails — a dataset-sized blast radius, never the measure engine. + +Capability is a projection of code and definitions, never of stored derived rows, so an empty tenant has full authoring capability. Every query the compiler emits carries three server-injected scopes the client cannot widen: tenancy, org-scope entity visibility, and cohort isolation. Materialization is a compiler-managed, version-keyed cache of aggregated work — refreshed on a schedule, never on read, always falling back to live compute rather than serving a superseded value. The full rationale is in [REFERENCE.md](./REFERENCE.md); the migration sequence in [IMPLEMENTATION.md](./IMPLEMENTATION.md). + +### 1.2 Architecture Drivers + +Requirements that significantly influence architecture decisions. + +#### Functional Drivers + +| Requirement | Design Response | +|-------------|------------------| +| `cpt-semantic-layer-fr-definitions-as-data` | One canonical definition format defined as typed code in the owning service; authored YAML and stored rows are serializations of the same types; the definition store is the single runtime source of truth (`cpt-semantic-layer-component-definition-store`) | +| `cpt-semantic-layer-fr-one-compiler` | A single compiler (`cpt-semantic-layer-component-compiler`): definition + request → warehouse SQL; product and user definitions compile through the same path; a per-family flag selects a second executor only as a bounded cutover state | +| `cpt-semantic-layer-fr-gated-custom-dataset` | The custom-dataset component (`cpt-semantic-layer-component-custom-dataset`): role-gated, default off, references checked against the catalog, execution wrapped with tenancy from outside and resource guardrails, reads dedup-safe catalog views only | +| `cpt-semantic-layer-fr-capability-from-definitions` | Capability derived from the field catalog and stored definitions (`cpt-semantic-layer-component-field-catalog`); distinct dimension values served separately with an explicit unavailable state | +| `cpt-semantic-layer-fr-server-owned-semantics` | The compiler owns all rendering; responses are self-describing with provenance; the discovery API (`cpt-semantic-layer-component-discovery-api`) is the only vocabulary a client renders from | +| `cpt-semantic-layer-fr-query-time-grain` | Grain is a compiler request parameter; bucketing is in the tenant reporting time zone (`cpt-semantic-layer-constraint-reporting-timezone`); windows/cumulative are compiler features | +| `cpt-semantic-layer-fr-tenant-isolation` | The compiler's injected-scopes component (`cpt-semantic-layer-component-injected-scopes`) adds a tenancy predicate to every emitted query, sourced from request context | +| `cpt-semantic-layer-fr-org-scope-visibility` | The injected-scopes component adds an `entity ∈ visible_set` predicate resolved from the identity service, fail-closed if unavailable; the structural-safety-boundary principle governs it | +| `cpt-semantic-layer-fr-scope-isolation` | Cohort membership is an org-gated dataset; the injected-scopes component re-asserts the org boundary per cohort key; peer views are aggregates-only, memberless, floor-suppressed | +| `cpt-semantic-layer-fr-discovery-api` | The discovery API component serves the catalog, editor field catalogs, and paginated distinct values; requests validate against the same catalog | +| `cpt-semantic-layer-fr-materialization-cache` | The materialization cache component (`cpt-semantic-layer-component-materialization-cache`): version-keyed aggregated work, scheduled refresh, read-time fallback to live compute | +| `cpt-semantic-layer-fr-runtime-editing` | The editing API component (`cpt-semantic-layer-component-editing-api`): role-gated CRUD per layer sharing seed-reconciliation validators, version bumps, and appended revisions | + +#### NFR Allocation + +This table maps non-functional requirements from the PRD to specific design responses. + +| NFR ID | NFR Summary | Allocated To | Design Response | Verification Approach | +|--------|-------------|--------------|-----------------|----------------------| +| `cpt-semantic-layer-nfr-source-read-discipline` | Reads never bypass dataset dedup | Compiler + dataset metadata + custom-dataset views | Read discipline inherited from dataset metadata, applied by the compiler; custom SQL reads only dedup-safe catalog views | Adversarial tests: no dedup bypass; validator rejects raw-table references | +| `cpt-semantic-layer-nfr-tenant-isolation` | No cross-tenant rows | Injected-scopes component | Tenancy predicate on every emitted query, from request context, not widenable by definitions | Isolation e2e over every compiled read path returns zero cross-tenant rows | +| `cpt-semantic-layer-nfr-entity-visibility` | No out-of-scope persons | Injected-scopes component + identity service | `entity ∈ visible_set` predicate beside tenancy; fail-closed on unavailable source | Scope tests: zero out-of-scope persons returned/disclosed; unavailable source yields no rows | +| `cpt-semantic-layer-nfr-cohort-scope-isolation` | No cross-boundary cohorts | Cohort dataset + injected-scopes component | Org-gated membership at dataset build; boundary re-asserted per cohort key; memberless, floor-suppressed peer views | Cohort tests: zero cross-boundary members for any key; no peer view below the floor | +| `cpt-semantic-layer-nfr-executor-consistency` | Cutover parity | Compiler + e2e suite | Shadow-compare per family; divergence classes resolved before flip | 100% of existing e2e expectations green against the compiler | +| `cpt-semantic-layer-nfr-query-guardrails` | Every query bounded | Compiler | Row caps, timeout classes, memory bounds on every emitted query and cache rebuild | Every compiled query bounded; a pathological definition fails loudly | + +### 1.3 Architecture Layers + +```text +┌───────────────────────────────────────────────────────────────────┐ +│ SEMANTIC LAYER │ +│ │ +│ FE (renderer) ANALYTICS SERVICE (Rust) │ +│ ───────────── ──────────────────────── │ +│ ┌───────────────┐ discovery / query / definitions APIs │ +│ │ dashboards │──▶┌──────────────────────────────────────────┐ │ +│ │ editors │ │ Definition Store ──► Compiler │ │ +│ │ pickers │ │ Field Catalog (single executor) │ │ +│ └───────────────┘ │ Injected Scopes: tenant / org / cohort │ │ +│ │ Materialization Cache (version-keyed) │ │ +│ └──────────────────┬───────────────────────┘ │ +│ │ compiled SELECT │ +├─────────────────────────────────────── │ ─────────────────────────┤ +│ DATASETS (read-only, dedup-safe views) │ +│ ┌───────────────────────────────┐ ┌──────────────────────────┐ │ +│ │ product datasets (silver/gold)│ │ custom datasets (gated) │ │ +│ │ field catalog (generated) │ │ SQL over catalog views │ │ +│ └───────────────────────────────┘ └──────────────────────────┘ │ +└───────────────────────────────────────────────────────────────────┘ + ▲ org-chart visible set resolved from the identity service + ▲ base facts produced by bronze→silver ingestion (class contracts) +``` + +- [ ] `p3` - **ID**: `cpt-semantic-layer-tech-layers` + +| Layer | Responsibility | Technology | +|-------|---------------|------------| +| Presentation (FE) | Renderer of definitions: dashboards, editors, pickers, error states | FE app driven by discovery/dashboard payloads | +| Application | Discovery, query, and definition APIs; validation; injected scopes | Rust (analytics service) | +| Domain | Definition store, compiler, field catalog, materialization cache | Rust (`domain::definitions`, `domain::compiler`, `domain::catalog`) | +| Infrastructure | Datasets and version-keyed cache; org-chart visibility source | ClickHouse; identity service | + +## 2. Principles & Constraints + +### 2.1 Design Principles + +#### Definitions Are Data + +- [ ] `p2` - **ID**: `cpt-semantic-layer-principle-definitions-as-data` + +Anything a user may eventually edit is a row in the definition store, never code. Product-shipped and user-created definitions are the same kind of object with different ownership. Truth is base facts plus definitions; any derived value is a cached evaluation of a definition, and caches are invalidated by definition version — they are never the contract. + +#### One Executor + +- [ ] `p2` - **ID**: `cpt-semantic-layer-principle-one-executor` + +A single compiler turns definitions into queries. Two interpreters of one definition format diverge on null handling, time zones, and deduplication, so a second executor exists only as a bounded transitional state during cutover and retires on flip. + +#### No Invented Languages + +- [ ] `p2` - **ID**: `cpt-semantic-layer-principle-no-invented-languages` + +Every layer of the definition format reuses a proven external shape: the MetricFlow semantic-model envelope for the aggregation shape, MBQL / JSON-Logic structured predicate trees for filters, and allowlisted SQL fragments (parsed, AST-restricted) for scalar expressions. The system owns one small composition schema, not an expression language; each corner case (nulls, typing, precedence) has a reference answer in a published spec rather than a design meeting. + +#### Semantics Are Server-Owned + +- [ ] `p2` - **ID**: `cpt-semantic-layer-principle-server-owned-semantics` + +Clients request by key and render what they are given. Meaning — filters, formulas, windows, formatting identity — never lives in a client. Adding or changing a metric, chart, or dashboard is a data change that ships no frontend code; capability comes to the client only through discovery. + +#### The Safety Boundary Is Structural + +- [ ] `p2` - **ID**: `cpt-semantic-layer-principle-structural-safety-boundary` + +Users compose from typed catalogs through closed grammars; free-form SQL exists at exactly one gated layer with dataset-sized blast radius. The same structural discipline governs authorization: tenancy, org-scope entity visibility, and cohort isolation are injected server-side at the single compiler choke point, beside each other, and no definition or client input can widen them. Fail closed when the authorization source is unavailable. Cohort membership is org-gated at the dataset that produces it and re-asserted as an injected scope, so a shared tag cannot route a person across the org boundary. + +#### Capability from Code, Not Data + +- [ ] `p2` - **ID**: `cpt-semantic-layer-principle-capability-from-code` + +What can be asked is a function of shipped connectors and stored definitions; a tenant with zero ingested rows has full authoring capability. Dimension keys come from code and definitions; dimension values come from data through a dedicated distinct-values path with an explicit unavailable state. Capability is never inferred from stored derived rows. + +### 2.2 Constraints + +#### Editability Boundary + +- [ ] `p2` - **ID**: `cpt-semantic-layer-constraint-editability-boundary` + +Full composition freedom above the dataset line, none below it: datasets are product-owned (ingestion code); custom datasets are dataset-author-role, default off; the field catalog is nobody's to edit (derived from schemas); measures/metrics are admin-edited; charts/dashboards are admin and optionally end-user edited. Moving this line is an explicit design decision, never an incremental widening. + +#### SQL Only at the Dataset Layer + +- [ ] `p2` - **ID**: `cpt-semantic-layer-constraint-sql-only-at-dataset-layer` + +Free-form SQL produces rows (datasets); structure produces meanings (measures, metrics). The measure layer never accepts SQL: the final aggregation and column roles can never move into SQL because they are what the platform operates on (grain, scope, cache, composition, discovery). SQL at the measure layer would opt out of every platform feature, so it is confined to the gated custom-dataset layer, which only yields a schema. + +#### Code-Owned Schema, Defined Once + +- [ ] `p2` - **ID**: `cpt-semantic-layer-constraint-code-owned-schema` + +The definition format is typed code in the owning service; parser, validators, compiler, and editing APIs consume the same types, and machine-readable schema for external tooling is exported from them, never hand-maintained. Product definitions are validated at build time (an invalid definition fails the build) and structurally validated against the warehouse at startup; tenant definitions get the identical validators at write time. An invalid definition is never written. + +#### Reporting Timezone + +- [ ] `p2` - **ID**: `cpt-semantic-layer-constraint-reporting-timezone` + +Bucketing happens in a single declared per-tenant reporting time zone (a tenant profile setting, default UTC) with no per-request override. A per-request time zone would make the same request mean different things per client, violating server-owned semantics; exploration-style shifting, if ever wanted, is a separate explicit API capability, not a bucketing parameter. + +## 3. Technical Architecture + +### 3.1 Domain Model + +**Technology**: Rust (analytics service typed definitions), ClickHouse (datasets and cache) + +**Location**: [src/backend/services/analytics/](../../../../src/backend/services/analytics/) + +**Core Entities**: + +| Entity | Description | Location | +|--------|-------------|--------| +| Dataset | Product-owned relation with guaranteed dedup/tenant semantics; the correctness boundary | `domain::definitions` (dataset rows) | +| Custom dataset | Gated, role-authored SQL `SELECT` over dedup-safe catalog views, registered as a catalog entry | `domain::definitions` (custom-dataset rows) | +| Field catalog | Typed, role-annotated schema generated from dataset schemas; the authoring palette and validation universe | `domain::catalog` (build-time artifact) | +| Measure | Declarative aggregation of one dataset — filter tree, aggregation kind, event time, entity, dimensions | `domain::definitions` (measure rows) | +| Metric | Composition of measures — computation, transform (affine + clamp), display identity | `domain::definitions` (metric rows) | + +**Relationships**: +- Dataset → field catalog: each dataset publishes its typed fields (entity, dimension, measurable, event time). A field absent from the catalog does not exist at any layer above. +- Measure → dataset: a measure aggregates exactly one dataset; its `dimensions` list is its dimension capability, with no separate capability registry. +- Metric → measures: a metric composes one or more measures; its dimension capability is the intersection of its inputs' dimension sets. Cross-dataset metrics compose at the aggregated level (join on entity, time bucket, conformed dimensions), never row-level. +- Chart/dashboard → metric: pure presentation references; `ON DELETE RESTRICT` upward through the chain. + +### 3.2 Component Model + +```text +┌──────────────────────────────────────────────────────────────┐ +│ Analytics Service (Rust) │ +│ │ +│ ┌───────────────┐ ┌───────────────┐ ┌────────────────┐ │ +│ │ Editing API │──▶│ Definition │──▶│ Field Catalog │ │ +│ │ Discovery API │ │ Store │ │ (generated) │ │ +│ └───────────────┘ └───────┬───────┘ └────────┬───────┘ │ +│ │ │ │ +│ ▼ ▼ │ +│ ┌──────────────────────────────────────┐ │ +│ │ Compiler (single executor) │ │ +│ │ + Injected Scopes (tenant/org/cohort)│ │ +│ └───────┬───────────────────┬──────────┘ │ +│ │ │ │ +│ ▼ ▼ │ +│ Materialization Cache datasets / │ +│ (version-keyed) custom-dataset views│ +└──────────────────────────────────────────────────────────────┘ +``` + +#### Compiler + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-compiler` + +##### Why this component exists + +The single executor: definition plus request in, warehouse SQL out. One executor removes the drift class that two interpreters of one format would produce (nulls, time zones, deduplication). + +##### Responsibility scope + +- Catalog resolution and type checking; filter-tree and expression-AST rendering; per-dataset read discipline inherited from dataset metadata; dimension-capability validation. +- The time model: query-time grain bucketing in the tenant reporting time zone; windowed and cumulative modes parameterized at request time. +- Metric composition (all computation types) and the post-aggregation transform stage. +- Delegates every server-injected predicate to the injected-scopes component and bounds every emitted query with resource guardrails. + +##### Responsibility boundaries + +- Does NOT read scope values from client SQL or definitions — scopes come from request context via the injected-scopes component. +- Does NOT own storage or cache policy — it reads the store and the cache and falls back to live compute. +- Does NOT accept free-form SQL at the measure layer — that is the custom-dataset component. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-definition-store` — reads definitions from it +- `cpt-semantic-layer-component-injected-scopes` — delegates every server-side predicate to it +- `cpt-semantic-layer-component-materialization-cache` — reads cached work, falls back to live compute + +--- + +#### Definition Store + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-definition-store` + +##### Why this component exists + +Definitions live in application-owned storage so they can be authored, versioned, audited, and validated uniformly, and so the store is the single runtime source of truth for every consumer. + +##### Responsibility scope + +- Versioning: semantic changes increment `definition_version` (canonicalize semantic fields, compare, compare-and-set bump); presentation-only changes do not. Versions are strictly monotonic and never reused. Reads are version-keyed so superseded cache entries are logically dead the moment the version bumps. +- Referential integrity (metrics→measures, charts→metrics, dashboards→charts; deletion blocked while referenced or explicit cascade); ownership and tenancy (product definitions tenant-invariant and read-only; tenant definitions namespaced so they cannot shadow product keys); auditability (append-only revisions). +- Fail closed on the control plane only: a missing/unreadable store is a startup error; "no definitions found" never presents as "no capability"; warehouse state is never a startup gate. +- Warehouse-divergence is a state, not a crash: a continuous background probe reconciles declared contracts with the live warehouse and puts affected definitions into a self-healing `unavailable` state (alarmed if persistent). + +##### Responsibility boundaries + +- Does NOT execute queries — that is the compiler. +- Does NOT probe the warehouse inline on the serving path — serving and discovery read the stored availability state. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-editing-api` — the write path into the store +- `cpt-semantic-layer-component-compiler` — the primary reader +- `cpt-semantic-layer-component-field-catalog` — validates definitions at write time + +--- + +#### Field Catalog + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-field-catalog` + +##### Why this component exists + +Every dataset publishes a typed, role-annotated catalog of its fields — the editor's palette, the compiler's validation universe, and the discovery API's vocabulary at once. It is what makes capability a projection of definitions rather than data. + +##### Responsibility scope + +- Generated from dataset schemas (a Rust build-time generator sharing the backend's definition parsers, sourced from dbt `schema.yml` `meta:` blocks), never hand-maintained. Roles: entity, dimension, measurable, event time. +- Declares conformed dimensions — the same key means the same identity space across datasets — which is what makes cross-measure composition and shared filters meaningful. +- Serves as the validation universe: a field absent from the catalog does not exist at any layer above. + +##### Responsibility boundaries + +- Does NOT hold dimension values — those come from data via the discovery distinct-values path. +- Does NOT persist as a runtime store row until custom datasets exist — it is a build-time artifact through Phases 1–4, store-backed only once custom datasets are registered. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-dataset` — generated from dataset schemas +- `cpt-semantic-layer-component-discovery-api` — exposes the catalog vocabulary +- `cpt-semantic-layer-component-measure` — the palette measures compose from + +--- + +#### Dataset + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-dataset` + +##### Why this component exists + +A queryable relation with guaranteed semantics (deduplicated, tenant-scoped, stable names/types) that carries the entire correctness burden of ingestion, so nothing above it re-solves mutability, late data, or source quirks. Datasets are also the retention boundary — a measure evaluates only as far back as its dataset's history reaches, and that horizon is part of the served contract. + +##### Responsibility scope + +- Reuse existing bronze→silver ingestion, class contracts, and purpose-built gold relations (state intervals, cohorts) as named datasets; carry read discipline (dedup strategy, FINAL) as dataset metadata the compiler inherits. +- Cohort membership is a dataset with org-gated composition (a person's cohort is drawn from within their org scope; tags refine within the boundary, never across it). +- Expose row-level relations to custom SQL only through dedup-safe catalog views. + +##### Responsibility boundaries + +- Does NOT aggregate — aggregation is the measure layer. +- Does NOT decide scope at read time — the compiler injects tenancy and org/cohort scopes. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-field-catalog` — published from dataset schemas +- `cpt-semantic-layer-component-custom-dataset` — the gated authoring counterpart +- `cpt-semantic-layer-component-injected-scopes` — re-asserts cohort org-gating + +--- + +#### Custom Dataset + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-custom-dataset` + +##### Why this component exists + +The single gated home for everything the measure layer deliberately cannot express — cross-dataset joins, sequence/window logic, nested aggregation, correlated comparisons — with a dataset-sized blast radius: a defective custom dataset breaks itself and its dependents, visibly, never the measure engine or another dataset. + +##### Responsibility scope + +- Registration: parse the `SELECT`, check references against the catalog, capture the result schema, annotate result columns with catalog roles; the annotated schema becomes an ordinary catalog entry. +- Read surface: dedup-safe catalog views only — raw tables, source payloads, and pipeline intermediates are not referenceable, so read discipline is unbreakable by construction. +- Execution: always wrapped — tenancy applied from outside the statement, resource guardrails bounding it; lineage marks everything built on a custom dataset. +- Lifecycle: role-gated (default off), versioned with revisions, availability cascades to dependents; definitions form a DAG (cycles rejected at registration); deletion blocked while referenced; forking prefills an editor from a displayed body as a stopgap, with divergence surfaced. + +##### Responsibility boundaries + +- Does NOT guarantee semantic correctness of the SQL — that is the author's; the product guarantees safety and isolation. +- Does NOT bypass tenancy or guardrails — both are applied from outside the statement. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-dataset` — becomes a peer catalog entry +- `cpt-semantic-layer-component-field-catalog` — references validated against it +- `cpt-semantic-layer-component-measure` — composes on the registered schema + +--- + +#### Measure + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-measure` + +##### Why this component exists + +The lowest editable layer and the atom every metric is built from: a declarative aggregation of one dataset whose declarations are the SQL's type signature — five fields with no logic in them — keeping the platform feature set (grain, scope, cache, composition, discovery) mechanical. + +##### Responsibility scope + +- Envelope (MetricFlow shape): aggregation as a closed enum (`count`, `sum`, `avg`, `min`, `max`, `count_distinct`), an expression slot (`value_expr`), an explicit event-time field, an entity field, and a dimension list. +- Filter grammar: MBQL / JSON-Logic structured predicate trees (`all`/`any`/`not` over `{field, op, value}` leaves; closed operator enum; fields resolve in the catalog with a compatible type). +- Capability by declaration: an absent declaration switches a feature off, never rejects the definition; the two things that can never move into SQL are the final aggregation and column roles. + +##### Responsibility boundaries + +- Does NOT express row-level cross-dataset relations, sequence logic, or correlated comparisons — those go to a dataset below the line. +- Does NOT accept free-form SQL — only allowlisted scalar-expression fragments (parsed, AST-restricted). + +##### Related components (by ID) + +- `cpt-semantic-layer-component-field-catalog` — the palette it composes from +- `cpt-semantic-layer-component-metric` — composes measures into served values +- `cpt-semantic-layer-component-compiler` — renders the measure to SQL + +--- + +#### Metric + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-metric` + +##### Why this component exists + +Composes measures into a served value with a display identity, so the served contract is a named, formatted, direction-bearing thing rather than a raw aggregation. + +##### Responsibility scope + +- Computation over role-typed measure inputs (direct, ratio, derived expression over named inputs); an optional post-aggregation shaping stage (affine transform, clamping); display identity (direction, format, naming). +- Derived dimension capability: the intersection of inputs' dimension sets, with dimension-agnostic inputs (constants, global denominators) as identity elements. +- Cross-dataset composition at the aggregated level: each input aggregates within its dataset; the compiler joins aggregates on entity, time bucket, and conformed dimensions. + +##### Responsibility boundaries + +- Does NOT re-aggregate row-level across datasets — that need becomes a product or custom dataset. +- Does NOT hold presentation — view, layout, thresholds are chart/dashboard concerns. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-measure` — its inputs +- `cpt-semantic-layer-component-compiler` — composes and shapes it into SQL + +--- + +#### Materialization Cache + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-materialization-cache` + +##### Why this component exists + +A compiler-managed cache tier, invisible to the contract, that stores work not answers: one cached representation of each measure's aggregated rows at the finest served grain covers the entire request space by re-aggregation, unlike a request→response cache that pays off only when identical questions repeat. + +##### Responsibility scope + +- Cached row shape follows the aggregation math: additive → one row per entity × time bucket × dimension tuple; percentile-family → one row per source event; distinct-count → one row per counted subject. +- One shared cache relation for all measures, partitioned by `(measure key, definition version, time bucket)` so invalidation and refresh are atomic partition operations and tenant-created measures never trigger DDL. Custom-dataset materialization gets its own per-version relation (the one place DDL binds to a runtime object, bounded to promotion events). +- Caching is policy, not semantics: an enable/refresh/hot-window/watermark policy separate from the definition; toggling it never bumps the version. Refresh is scheduled (never read-triggered), rebuilds a hot window by partition replacement, and walks the definition DAG topologically. +- Read decision per input measure: policy enabled, cached version equals current definition version, requested range inside coverage watermarks, and availability == available → read cache; otherwise compile live. + +##### Responsibility boundaries + +- Does NOT serve a superseded definition — a version bump makes prior rows logically dead; after a semantic bump the fallback is live compute or `unavailable`, never the previous table. +- Does NOT gate the read path on a build — no request pays a build or stampedes one. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-compiler` — the only reader/writer of the cache +- `cpt-semantic-layer-component-definition-store` — supplies the version the cache is keyed by + +--- + +#### Discovery API + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-discovery-api` + +##### Why this component exists + +The server tells clients what exists so they never hardcode or probe; it is the single vocabulary that keeps requests and served capability from disagreeing. + +##### Responsibility scope + +- Serve the catalog of metrics and measures visible to the tenant (product ∪ tenant) with computation, format, direction, and allowed dimensions. +- For editors: dataset field catalogs with roles and types, including each dataset's read-only display body (custom datasets from their store row; product datasets as a build-generated artifact of the shipped model SQL, stamped with the build). +- On demand: distinct dimension values from data, paginated, with an explicit unavailable state for unscanned or oversized value sets. +- Validate requests against the same catalog it serves. + +##### Responsibility boundaries + +- Does NOT execute the shipped body or hold a second source of truth — the display copy is derived at build time, never authored, never executed. +- Does NOT compute values — that is the query path over the compiler. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-field-catalog` — the vocabulary it exposes +- `cpt-semantic-layer-component-definition-store` — the visible definitions it lists + +--- + +#### Editing API + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-editing-api` + +##### Why this component exists + +Role-gated runtime CRUD per definition layer so an authorized user changes analytics without a deploy, with the same assurance as seed reconciliation. + +##### Responsibility scope + +- CRUD per layer (charts/dashboards → metrics → measures → custom datasets), running the same validators as seed reconciliation with precise field-level errors; validate-only dry runs; fork; cache-policy changes. +- One write path: seed reconciliation and the editing API share validators. Semantic writes bump the version and append a revision; deletes are blocked with the referencing keys while referenced. +- This surface is also the agent/MCP tool surface — machine authors get the same endpoints and validation, never a separate path. + +##### Responsibility boundaries + +- Does NOT let a tenant definition shadow a product key — tenant keys are namespaced. +- Does NOT skip validation for any author, human or machine. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-definition-store` — the store it writes to +- `cpt-semantic-layer-component-field-catalog` — the validation universe it checks against + +--- + +#### Injected Scopes + +- [ ] `p2` - **ID**: `cpt-semantic-layer-component-injected-scopes` + +##### Why this component exists + +Authorization is structural, not per-endpoint: the compiler injects, on every query, three server-side scopes the client cannot widen, at the single choke point. A new read path cannot forget them because they live where every query is emitted. + +##### Responsibility scope + +- Tenancy: a tenancy predicate on every emitted query, sourced from request context, not from any definition or client input. +- Org-scope entity visibility: an `entity ∈ visible_set` predicate resolved from the identity service's org chart (self plus related subtree). People outside the set are never returned and their existence is not disclosed; fail closed if the source is unavailable. Definitions stay scope-agnostic; the scope is injected per request exactly like tenancy. +- Cohort isolation: re-assert the org boundary for every cohort key (not only `org_unit`) so a tag-based cohort cannot span org scope; peer/cross-cohort comparison is aggregates-only — distributions with no member ids, suppressed below a minimum distinct-member floor. + +##### Responsibility boundaries + +- Does NOT resolve the org chart itself — that is the identity service; this component consumes the visible set. +- Does NOT produce cohort membership — that is the cohort dataset; this component re-asserts its org-gating at read time. + +##### Related components (by ID) + +- `cpt-semantic-layer-component-compiler` — hosts the injection at the single choke point +- `cpt-semantic-layer-component-dataset` — cohort membership dataset it re-asserts over + +--- + +### 3.3 API Contracts + +- [ ] `p2` - **ID**: `cpt-semantic-layer-interface-query-endpoints` + +- **Implements**: `cpt-semantic-layer-interface-query-api` (PRD §7.1 Public API Surface) +- **Contracts**: `cpt-semantic-layer-contract-visibility-consumption`, `cpt-semantic-layer-contract-warehouse-read` +- **Technology**: REST / HTTP JSON +- **Base path**: `/v1/metric-results`, `/v1/metrics/queries` (stable across cutover) + +A query requests a served value by metric key with entity scope, date range, grain, dimensions, filters, and view. Responses are self-describing and carry provenance: definition version, cache-or-computed, data-available-from, and availability state. The request shape is validated against the same catalog discovery serves; the injected scopes (tenant, org, cohort) are applied server-side and are not request fields. + +**Endpoints Overview**: + +| Method | Path | Description | Stability | +|--------|------|-------------|-----------| +| `POST` | `/v1/metric-results` | Compile and serve one metric result (scope, range, grain, dimensions, filters, view); provenance in the response | stable | +| `POST` | `/v1/metrics/queries` | Batch metric queries in one request (the e2e parity surface) | stable | +| `GET` | `/v1/discovery/metrics` | Catalog of visible metrics/measures with capability | unstable | +| `GET` | `/v1/discovery/datasets` | Editor field catalogs with roles, retention, origin, and display body | unstable | +| `GET` | `/v1/discovery/dimension-values` | Paginated distinct dimension values with an explicit unavailable state | unstable | +| `POST` | `/v1/definitions/{layer}` | Role-gated create/edit per layer (shared validators, version bump, appended revision) | unstable | + +The query contract is held stable across the executor cutover; discovery and definition surfaces grow additively. Detailed request/response field specs are feature-level and out of scope for this design. + +### 3.4 Internal Dependencies + +| Dependency Module | Interface Used | Purpose | +|-------------------|----------------|----------| +| Definition store (`domain::definitions`) | In-process call | Read validated definitions and versions the compiler serves | +| Field catalog (`domain::catalog`) | In-process call | Validate definitions and requests against the typed vocabulary | +| Materialization cache | In-process call | Read cached aggregated work; fall back to live compute on miss/mismatch | + +**Dependency Rules**: +- The compiler is the only executor; product and tenant definitions compile through the same path. +- Injected scopes are always applied at the compiler; no read path bypasses them. +- `SecurityContext` is propagated across all in-process calls and is the sole source of scope values. + +### 3.5 External Dependencies + +#### ClickHouse + +| Aspect | Value | +|--------|-------| +| Datasets | Read-only through dedup-safe catalog views; read discipline (FINAL/dedup) inherited from dataset metadata | +| Cache | Compiler-managed version-keyed relation, partitioned by `(measure key, definition version, time bucket)`; custom-dataset materialization gets a per-version relation on promotion | +| Read semantics | Query-time grain bucketed in the tenant reporting time zone; guardrails (row caps, timeout classes, memory) on every emitted query | +| Convergence | A relation lagging its declared contract puts affected definitions into an `unavailable` state; the warehouse is never a startup gate | + +**Dependency Rules**: +- Only the compiler and the cache refresh jobs talk to ClickHouse. +- Custom SQL reads only catalog views, never raw tables or intermediates. + +#### Identity Service + +| Aspect | Value | +|--------|-------| +| Purpose | Authoritative source of org-chart visibility (the viewer's visible-person set) and cohort composition | +| Semantics | Token-forwarded visible-set resolution injected as the entity-visibility scope; fail closed if unavailable | +| Current state | Enforced today at the request boundary via `person_visibility` → `/v1/visible-persons`; moves into the compiler's shared `WHERE` under compiler-first | + +**Dependency Rules**: +- Analytics never invents scope; the identity service is authoritative. +- The visible set is injected per request and is never widened by definitions or client input. + +### 3.6 Interactions & Sequences + +#### Author and Serve a Definition + +**ID**: `cpt-semantic-layer-seq-author-and-serve` + +**Use cases**: `cpt-semantic-layer-usecase-author-measure-metric` + +**Actors**: `cpt-semantic-layer-actor-admin`, `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-clickhouse` + +```mermaid +sequenceDiagram + participant Ad as Admin (editor) + participant Ed as Editing API + participant St as Definition Store + participant Co as Compiler + participant CH as ClickHouse + + Ad ->> Ed: create measure/metric (validate) + Ed ->> St: write (bump version, append revision) + St -->> Ed: stored + Ad ->> Co: query metric key (scope, range, grain) + Co ->> St: read definition + version + Co ->> CH: compiled SELECT (injected scopes, FINAL, guardrails) + CH -->> Co: rows + Co -->> Ad: self-describing result (provenance) +``` + +**Description**: One write path validates and versions the definition; the compiler reads it and emits SQL at query-time grain with injected scopes. Capability appears in discovery the moment the definition is stored, independent of any ingested rows. + +#### Register a Custom Dataset + +**ID**: `cpt-semantic-layer-seq-register-custom-dataset` + +**Use cases**: `cpt-semantic-layer-usecase-register-custom-dataset` + +**Actors**: `cpt-semantic-layer-actor-dataset-author`, `cpt-semantic-layer-actor-compiler-svc` + +```mermaid +sequenceDiagram + participant DA as Dataset Author + participant Ed as Editing API + participant Cat as Field Catalog + participant St as Definition Store + + DA ->> Ed: register SELECT over catalog views + role annotations + Ed ->> Cat: check references (reject raw tables/intermediates) + Cat -->> Ed: result schema resolved + Ed ->> St: store custom dataset (versioned, DAG cycle-checked) + St -->> Ed: registered as catalog entry + Ed -->> DA: available; measures may compose on top +``` + +**Description**: The statement is parsed and validated against the catalog read surface; only dedup-safe views are referenceable. The annotated schema becomes an ordinary catalog entry with lifecycle, availability cascade, and lineage. + +#### Visibility-Scoped Read + +**ID**: `cpt-semantic-layer-seq-visibility-scoped-read` + +**Use cases**: `cpt-semantic-layer-usecase-promote-to-dashboard` + +**Actors**: `cpt-semantic-layer-actor-analyst`, `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-identity-svc` + +```mermaid +sequenceDiagram + participant An as Analyst (dashboard) + participant Co as Compiler + participant Id as Identity Service + participant CH as ClickHouse + + An ->> Co: query metric (entity/cohort scope) + Co ->> Id: resolve viewer visible-person set (token-forwarded) + Id -->> Co: visible_set (or unavailable) + alt source unavailable + Co -->> An: fail closed (no rows) + else + Co ->> CH: SELECT WHERE tenant + entity ∈ visible_set + cohort org-gated + CH -->> Co: rows (aggregates-only peer view, floor-suppressed) + Co -->> An: scoped result + end +``` + +**Description**: The compiler injects tenancy, an `entity ∈ visible_set` org-scope predicate, and the cohort org-boundary at the single choke point. An unavailable authorization source fails closed; peer/cross-cohort views are memberless aggregates suppressed below the distinct-member floor. + +### 3.7 Database Schemas & Tables + +- [ ] `p3` - **ID**: `cpt-semantic-layer-db-schemas` + +The store rewrite is additive tables plus seed repopulation, never in-place mutation of the old schema; migration burden is near zero because builtin rows are seed-reconciled from code and any `origin = 'custom'` rows get a one-shot mapping. Tables live in the analytics service database (MySQL via sea-orm, forward-only migrations). Column-level types are implementation detail; the tables and their roles are below. + +#### Table: `datasets` + +**ID**: `cpt-semantic-layer-dbtable-datasets` + +Product and custom dataset registry: the queryable relations measures build on. + +| Column | Type | Description | +|--------|------|-------------| +| `key` | String | Stable dataset identifier | +| `database_relation` | String | Warehouse database + relation (or the registered custom SELECT) | +| `read_discipline` | Enum | Dedup strategy the compiler inherits | +| `retention_horizon` | Interval | History depth part of the served contract | +| `origin` | Enum | `product` \| `custom` | + +**PK**: `key` + +**Additional info**: Custom-dataset rows carry the validated SELECT and its captured, role-annotated schema; availability state is stored on the row and read by serving/discovery. + +#### Table: `measures` + +**ID**: `cpt-semantic-layer-dbtable-measures` + +Declarative aggregation of one dataset — the lowest editable layer. + +| Column | Type | Description | +|--------|------|-------------| +| `key` | String | Stable measure identifier | +| `dataset_ref` | String | Dataset the measure aggregates | +| `filter` | JSON | Closed-enum MBQL-style predicate tree | +| `aggregation` | Enum | `count`, `sum`, `avg`, `min`, `max`, `count_distinct` | +| `value_expr` | String | Validated allowlisted SQL fragment (nullable) | +| `event_time` | String | Event-time catalog field | +| `entity` | String | Entity catalog field | +| `dimensions` | JSON | Dimension key → value/label catalog fields | +| `definition_version` | Integer | Monotonic; bumped on semantic change | + +**PK**: `key` + +**Additional info**: Tenant scoping and `origin` as in the current store; CHECK-style biconditionals between aggregation and expression follow existing migration conventions. + +#### Table: `metrics` + +**ID**: `cpt-semantic-layer-dbtable-metrics` + +Composition of measures into a served value with display identity. + +| Column | Type | Description | +|--------|------|-------------| +| `key` | String | Stable metric identifier | +| `computation` | JSON | Computation over role-typed measure inputs | +| `transform` | JSON | Affine transform + clamp | +| `format` | String | Display format | +| `direction` | Enum | Good-direction indicator | +| `entity_type` | String | Entity type | +| `cohort_key` | String | Cohort key (org-gated) | +| `definition_version` | Integer | Monotonic; bumped on semantic change | + +**PK**: `key` + +**Additional info**: Dimension capability is derived (intersection of inputs), not stored; same tenant scoping/`origin` as measures. + +#### Table: `definition_revisions` + +**ID**: `cpt-semantic-layer-dbtable-definition-revisions` + +Append-only audit written by every mutation path from the store's first day. + +| Column | Type | Description | +|--------|------|-------------| +| `id` | UUID | Revision identifier | +| `kind` | Enum | Definition layer (dataset/measure/metric/chart/dashboard) | +| `definition_key` | String | The definition changed | +| `version` | Integer | Version at this revision | +| `actor` | String | Who changed it | +| `body` | JSON | Snapshot of the definition body | +| `created_at` | DateTime | When | + +**PK**: `id` + +**Additional info**: Runtime editability without audit is not acceptable in a multi-tenant product; this table is written by seed reconciliation and the editing API alike. + +#### Table: `measure_cache` + +**ID**: `cpt-semantic-layer-dbtable-measure-cache` + +The one shared materialization relation (ClickHouse) for all measures — aggregated work, not answers. + +| Column | Type | Description | +|--------|------|-------------| +| `measure_key` | String | The cached measure | +| `definition_version` | Integer | Version the row was computed under | +| `time_bucket` | DateTime | Finest served grain bucket | +| `entity` | String | Entity (additive/distinct shapes) | +| `dimension_tuple` | JSON | Dimension tuple | +| `value` | Numeric | Aggregated partial | + +**PK**: `(measure_key, definition_version, time_bucket, entity, dimension_tuple)` + +**Additional info**: Partitioned by `(measure key, definition version, time bucket)` so invalidation/refresh are atomic partition operations and tenant-created measures never trigger DDL. Every row is stamped with its definition version; a mismatch means recompute or reject, never serve a superseded value. Custom-dataset materialization uses a separate per-version relation created only on promotion. + +## 4. Additional context + +Full design rationale — principles, expressiveness limits, capability model, time model, alternatives considered, and risks — is in [REFERENCE.md](./REFERENCE.md). The migration sequence and current-code inventory (keep/rewrite/delete, phase plan, decisions) are in [IMPLEMENTATION.md](./IMPLEMENTATION.md): Phase 1 definition core, Phase 2 compiler and cutover, Phase 3 deletion, Phase 4 catalog and discovery, Phase 5 runtime editing. The adoption review is in [FINDINGS.md](./FINDINGS.md). + +Open design decisions carried from the adoption review (see FINDINGS.md), pending an author/team ruling: + +1. **Percentile capability is under-specified.** The expressiveness class lists percentiles, but the measure aggregation enum omits it. This design adopts the metric-level reading (percentile as a metric computation over event-grain measures, computed at read over event rows — never a percentile-of-percentiles) unless the author decides otherwise. +2. **No serving a superseded custom-dataset table after a version bump.** The previous-table fallback is restricted to refresh failures under the same definition version; after a semantic bump the fallback is live compute or `unavailable`, never a table computed under a superseded definition. +3. **Gate cache reads on availability.** The cache read decision requires `availability == available` in addition to policy, version, and coverage; an `unavailable` definition serves neither cached nor live rows but the stored unavailable error. + +The declarative e2e metric suite is the cross-cutting safety rail through the executor swap and deletion; the query contract (`/v1/metric-results`, `/v1/metrics/queries`) is held stable while discovery grows additively. + +## 5. Traceability + +- **PRD**: [PRD.md](./PRD.md) +- **ADRs**: none yet. +- **Features**: to be created from the Phase 1–5 sub-issues of epic constructorfabric/insight#1803. #1974 shipped Phase 1's first step (metric definitions authored as data — the YAML registry replacing Rust constants), in the transitional observation-relation shape the target rewrites at cutover. diff --git a/docs/domain/semantic-layer/specs/FINDINGS.md b/docs/domain/semantic-layer/specs/FINDINGS.md new file mode 100644 index 000000000..4a9d5572a --- /dev/null +++ b/docs/domain/semantic-layer/specs/FINDINGS.md @@ -0,0 +1,143 @@ +# Adoption Findings — Semantic Layer + +Review of [REFERENCE.md](./REFERENCE.md) + [IMPLEMENTATION.md](./IMPLEMENTATION.md) +against the current codebase and the presentation-layer epic +(constructorfabric/insight#1803). The design is adopted as the Phase B target; +these are the reconciliation notes and the changes it implies to the existing +plan. + +## Alignment + +The design matches the principles the presentation split already commits to: + +- **No invented query language.** The definition format reuses proven shapes + (MetricFlow envelope, MBQL/JSON-Logic filter trees, allowlisted SQL + fragments) — a closed composition schema, not a DSL. +- **Raw SQL at exactly one gated layer** (custom datasets), with + dataset-sized blast radius; the measure/metric/chart layers stay structured. +- **Definitions as data, server-owned semantics, one compiler, capability from + code+config not from stored rows.** + +## Where the shipped registry (#1974) fits + +Phase 1 of the implementation plan is "product definitions as repo YAML, +embedded and parsed at compile time, replacing `builtin.rs` constants, +validated in CI." That is exactly what #1974 shipped: the metric registry moved +from Rust literals to `registry.yaml` (`include_str!`, validated by the +registry tests, `deny_unknown_fields`). So **#1974 is the first, shipped step of +this design's Phase 1** — the authoring-as-data move. + +Caveat: #1974's YAML still uses the *observation-relation* shape +(`source_ref: *_metric_observations`, per-measure `evidence_granularity`, +reconcile into `metric_source_measures`). The target **rewrites** that store +schema to the dataset/measure/metric domain model and **deletes** the dbt +observation gold models. So #1974's format direction is on-path; its schema +shape is transitional and gets rewritten at cutover. Migration cost stays near +zero because builtin rows are seed-reconciled. + +## Changes this implies to the epic sub-issues + +- **#1975 (metric passports + drift test)** and the generate-SQL framing of + **#1976** need re-scoping. The design is compiler-first and explicitly + rejects "registry-driven emission" and "drift gates for generated SQL" + because nothing is generated — one compiler over datasets removes the drift + class those tasks guard. Keep the *passport* idea (provenance surfaced per + value); drop the *generated-SQL drift gate*. +- **#1976 (semantic raw->derived compiler)** becomes the design's Phase 2 + compiler over datasets, not a raw->derived transpiler. +- **#1977/#1978 (FE rework, query->card promotion)** map onto Phases 4–5 + (discovery API + runtime editing). Promotion of a good ad-hoc query is the + custom-dataset -> measure -> metric ladder (runtime, role-gated), not a + bespoke per-card mechanism. +- **#1980 (subtree + row-policy backstop)** is where the caller-scope + predicates below become compiler-injected row filters. + +## Authorization: a scope the design must name + +The design specifies the **tenancy** predicate as injected on every compiled +query, and says entity scoping and peer modes are "injected uniformly," but it +does **not name org-chart visibility**. That is a real, required scope and must +be first-class in this document: + +- **Security — no people outside your org scope.** A viewer reads per-entity + values only for individuals within their org-chart scope (self + related + subtree/reports). People outside it are never returned and their existence is + not disclosed. The visible set is resolved from the org chart (owned by the + identity service); a caller-scoped predicate is injected server-side beside + the tenancy predicate, at the single compiler choke point, and the client can + never widen it. Fail-closed if the authorization source is unavailable. +- **Scope isolation — no other teams/cohorts.** No per-entity reads for + unrelated teams/cohorts. Cross-cohort comparison is aggregates-only: peer + views return distributions with no member ids, suppressed below a minimum + distinct-member floor so small groups cannot disclose individuals. +- **Cohorts are org-gated, not tag-gated.** A person's peer cohort is drawn + from within their org-chart scope; tags/attributes refine the cohort *inside* + that boundary and never pull a person across it. An R&D member and a Sales + member who share a tag are never placed in the same cohort, so a shared tag + cannot route around scope isolation — even the aggregate a viewer sees stays + within org scope, not merely anonymized. The org chart is authoritative over + cohort composition; tags are secondary and only subdivide within it. + + Current state / gap: only `org_unit` cohorts are *implicitly* org-scoped + today, because the peer pool is scoped by `cohort_key` value plus tenant + (`compile_peer_batch_query` / `push_cohort_scope`), never intersected with + the viewer's org-visible set. An arbitrary or tag-based cohort key would span + the org boundary. This rule must be enforced where cohort membership is + produced (the cohort dataset) and re-asserted as the compiler's injected + scope (#1980), so it holds for every cohort key, not just `org_unit`. + +Current state: analytics enforces this at the request boundary +(`domain/person_visibility.rs` -> identity `/v1/visible-persons`), forwarding +the caller's token and refusing out-of-scope ids (leaking only a count). Under +compiler-first this moves into the compiler's shared `WHERE` as an injected +`entity ∈ visible_set` filter (the #1980 work). Definitions stay +scope-agnostic; the scope is injected per request, exactly like tenancy. + +These two guarantees are recorded on the epic +(constructorfabric/insight#1803) and are carried into the governed +[DESIGN.md](./DESIGN.md) as named injected scopes (Principles & Constraints plus +the compiler component), alongside the tenancy predicate. + +## The real decision + +Adopting this design commits to **rewriting the observation-relation store +schema and deleting the dbt observation gold models**, replaced by one compiler +computing over datasets at query-time grain. That deletion — not the YAML +format — is the load-bearing decision. The e2e metric suite is the parity +invariant that makes the cutover safe (same seeds, same requests, same +expectations against the new executor). + +## Open items from review (in the reference design) + +Points raised against [REFERENCE.md](./REFERENCE.md) that need an author/team +decision; carried here rather than silently rewritten into the reference: + +- **Percentile capability is under-specified.** The expressiveness section + lists percentiles, but the measure aggregation enum + (`count | sum | avg | min | max | count_distinct`) omits it. Clarify whether + percentile is a **metric-level computation** (as median is today — + `ComputationSpec::Median` over event-grain measures) or a measure + aggregation, and state the read-time contract (percentiles compute over event + rows, never a percentile-of-percentiles). The governed DESIGN adopts the + metric-level reading unless the author decides otherwise. +- **Do not serve a superseded custom-dataset table after a semantic version + bump.** The materialization section both invalidates caches on a version bump + and allows a failed rebuild to keep serving the previous table. Restrict the + previous-table fallback to refresh failures **under the same definition + version**; after a semantic bump, fall back to live compute or return + `unavailable` — never serve values computed under a superseded definition + (the section's own "version mismatch means recompute or reject" rule). +- **Gate cache reads on definition availability.** The cache read decision + checks policy, version, and coverage but not availability; an `unavailable` + definition must not serve cached rows. Require `availability == available` + before cached or live execution, else return the stored unavailable error. + +## Governance status + +Adopted as governed `cfs` `sdlc` artifacts under +`docs/domain/semantic-layer/specs/`: [PRD.md](./PRD.md) and +[DESIGN.md](./DESIGN.md) are the template-conformant, registered artifacts; +[REFERENCE.md](./REFERENCE.md) is the detailed design narrative they distill and +cite for depth, and [IMPLEMENTATION.md](./IMPLEMENTATION.md) is the migration +plan. The reference and implementation docs are kept verbatim as adopted so the +governed specs never lose the original rationale. diff --git a/docs/domain/semantic-layer/specs/IMPLEMENTATION.md b/docs/domain/semantic-layer/specs/IMPLEMENTATION.md new file mode 100644 index 000000000..8acbdb771 --- /dev/null +++ b/docs/domain/semantic-layer/specs/IMPLEMENTATION.md @@ -0,0 +1,329 @@ +# Implementation Plan — Semantic Layer + +Status: proposal. Companion to `DESIGN.md` (target architecture). This +document decides what is kept, rewritten, or deleted, and sequences the +work. Unlike the design, it references current code on purpose: it is the +migration document. + +The strategy is **compiler-first**. The runtime compiler over datasets is +the target's center of gravity, so it is built early and everything the +target does not need is deleted early — instead of investing in machinery +that keeps the interim architecture consistent while it waits to be +demolished. Concretely rejected on that basis: generating dbt observation +models from measure definitions, drift gates for generated SQL, and +registry-driven emission. All of that would harden the layer the target +retires. + +## Current State (inventory) + +- **Definition store** — MySQL via sea-orm + (`src/backend/services/analytics/src/migration/m20260625_000001_metric_definitions.rs` + and follow-ups): `metric_sources` (observation relation registry with + schema-probe status machinery), `metric_source_measures` (measure keys + only — semantics live in dbt SQL), `metric_source_dimensions`, + `metric_definitions` (computation `sum|ratio|median|distinct_count`, + format, direction, tenant scoping, `origin builtin|custom`), + `metric_definition_inputs`, `metric_definition_dimensions`. Builtin + definitions are Rust constants + (`domain/metric_definitions/builtin.rs`, incl. affine + clamp shaping) + reconciled at startup (`seeds.rs`). +- **Emission** — dbt gold models + (`src/ingestion/gold/*_metric_observations.sql`) hand-author measure + semantics through shape macros + (`src/ingestion/dbt/macros/metric_observation_measures.sql`); day grain + baked into the observation contract (`metric_date Date`). +- **Validation** — `domain/metric_definitions/validator.rs` + + `domain/schema_validator/`: relation/column probes plus dimension + coverage **probed from observed rows** (empty tenant → `Unchecked`). +- **Serving** — `domain/metric_results/` compiles + period/timeseries/peer/breakdown/histogram queries over observation + relations; `/v1/metric-results`, `/v1/metrics/queries`. +- **Tests** — declarative e2e suite + (`src/ingestion/tests/e2e/metrics/*.test.yaml`) seeds bronze and asserts + through the batch endpoint; dbt tests on observation shapes. + +## Metric Lifecycle, Before and After + +The shortest way to see what changes. Adding one metric ("large PRs +merged"): + +Today: + +1. Hand-write a CTE + shape-macro call in the family's gold observation + model, dimension array included — the meaning, as dbt SQL. +2. Declare the measure key, metric, allowed dimensions, format, direction + in `builtin.rs` — the meaning again, as Rust. +3. Deploy both; dbt pre-computes observation rows; startup seeds the + store; the validator probes observed rows to confirm dimensions. +4. Serving reads the pre-computed rows at their baked day grain. +5. The chart is frontend code; changing it is a third deploy. + +Target: + +1. Write one YAML definition (dataset, filter, aggregation, event time, + dimensions) — the only place meaning exists. Later, an admin does the + same in an editor, writing a store row instead of a repo file. +2. The write path validates it against the field catalog and expression + allowlist; invalid definitions cannot exist. +3. Nothing is pre-computed. Capability is read off the definition; + an empty tenant exposes it identically. +4. The compiler turns definition + request into SQL over the dataset at + any grain; materialization appears later, per measure, as a + version-keyed cache — purely a latency decision. +5. The chart is a definition row referencing the metric; no deploy. + +Three deploys and two hand-synchronized restatements become one reviewed +definition; pre-computation moves from mandatory-and-authored to +optional-generated-versioned. + +## Keep / Rewrite / Delete + +| Component | Verdict | Reason | +| --- | --- | --- | +| bronze → silver ingestion, class contracts | **keep** | the dataset layer; carries dedup/mutability correctness the target builds on | +| purpose-built gold relations (state intervals, cohorts) | **keep** | become named datasets; derived-column logic belongs below the measure layer | +| declarative e2e suite + batch endpoint | **keep** | the cutover invariant: same seeds, same requests, same expectations against the new executor | +| public API contract (`/v1/metric-results`, `/v1/metrics/queries`) | **keep** | response shape is presentation contract, orthogonal to how values are computed | +| metric composition semantics (ratio/median/distinct-count, affine + clamp) | **keep semantics, rehome code** | proven definitions; they become data and compile in the new executor | +| store technology + conventions (MySQL, sea-orm, forward-only migrations, seed reconciliation) | **keep** | conventions are fine; the schema is not | +| store schema (`metric_sources` … `metric_definition_dimensions`) | **rewrite** | shaped around observation relations, not the domain model; measures are key-only; probe-status machinery solves a drift problem the target removes structurally | +| `metric_results` query builder | **rewrite** | its input model is observation rows at day grain; the compiler's input model is datasets at query-time grain — a second input path bolted on would preserve the wrong core | +| observation gold models + shape macros | **delete after cutover** | measure semantics move to definitions; materialization returns later as compiler-owned cache, not as authored models | +| row-probing dimension validation (`check_dimension_coverage`, `Unchecked` state, `dimension_not_covered` plumbing) | **delete** | capability must not depend on data; replaced by structural validation at startup and CI | +| `metric_sources.source_ref` relation-name-as-data + column probe machinery | **delete** | replaced by the dataset registry and catalog artifact | + +Migration burden of the store rewrite is near zero by construction: builtin +rows are startup-reconciled from code, so new tables repopulate from seeds. +Any `origin = 'custom'` rows get a one-shot mapping migration. + +## What Compiler-First Unlocks Immediately + +- Query-time grain (week/month/quarter without day-grain assumptions) and + tenant-timezone bucketing from day one — impossible over the baked + observation contract. +- New measure correct instantly with zero materialization — the editing + story's core mechanic, proven years before an editor exists. +- One executor from the start: no dual-execution window, no + Jinja-vs-compiler semantic drift to manage, no parity infrastructure + living forever. +- No generated-SQL machinery: nothing to codegen, gate, or keep from + drifting, because nothing is generated. + +## Phase 1 — Definition Core + +The new store, authored definitions, and validators. No serving change. + +1. **Schema, new tables** (one migration, domain-shaped): + - `datasets` — key, database + relation, read discipline (dedup + strategy), retention horizon, `origin` (`product` | `custom`). + - `measures` — key, dataset ref, filter (JSON, closed-enum MBQL-style + tree), aggregation ENUM(`count`,`sum`,`avg`,`min`,`max`,`count_distinct`), + `value_expr` / `subject_expr` (validated SQL fragments), `event_time` + field, entity field, dimension bindings (dimension key → + value/label catalog fields), `definition_version`, tenant scoping + + `origin` as today. + - `metrics` — key, computation over role-typed measure inputs, + transform (affine + clamp), format, direction, entity type, cohort + key; same scoping. + - `definition_revisions` — append-only audit (kind, id, version, actor, + body snapshot) written by every mutation path from the first day the + store exists, so no later retrofit. + - CHECK-constraint style follows the existing migrations (key shapes, + aggregation/expr biconditionals); the probe-status columns are not + carried over. +2. **Field catalog, minimal form.** The catalog generator (Rust, per + Decision 3) ships here, not in Phase 4: dataset field names, types, + and roles from dbt `schema.yml` `meta:` blocks, emitted as a + build-time artifact. Phase 1 needs it as the validation universe; + Phase 4 only extends it into the discovery surface. +3. **Validators.** `domain/definitions/expr.rs`: `sqlparser` (ClickHouse + dialect) + AST allowlist (catalog columns, literals, arithmetic, + allowlisted functions; everything else rejected). Filter tree as serde + types with closed operator enum. One write path — seed reconciliation + and future editing APIs share these validators. +4. **Authoring.** Product definitions as repo YAML + (`src/ingestion/measures/.yaml` and + `metrics/.yaml`), embedded and parsed at compile time, replacing + `builtin.rs` constants; a build-time test runs the full validator set + over every file (shape, enums, expressions, cross-references), so an + invalid product definition fails CI and cannot ship; startup seed + reconciliation as today. Repo YAML + is how product definitions are authored and reviewed; the store is the + single runtime source. +5. **Extraction.** Transcribe every measure from the gold observation + models into YAML against silver/gold datasets. Logic that exceeds the + measure schema (attribution tiers, file categorization) is not forced + into filter trees — it moves to (or stays in) named dataset relations. + The `presence` macro family is decomposed during extraction + (count-distinct over the event-time day) rather than becoming an + aggregation kind. +6. **Structural validation.** Two stages replacing the row-probing path + conceptually (the old path keeps serving until cutover). CI: definitions + validate against the field catalog derived from dbt `schema.yml` at the + same commit — contract mismatches cannot ship. Runtime: the probe + reconciles contracts with the live warehouse; absent or lagging + relations put affected definitions into an explicit self-healing + `unavailable` state, alarmed if persistent, never a startup failure and + never a silent capability reduction. The probe is a continuous + background loop over warehouse metadata (interval + backoff — the shape + of today's `schema_validator` loop, retargeted from row probing to + contract reconciliation), with an orchestrator poke after dbt runs for + fast healing; availability state is stored on the definition row and + read by serving and discovery, never probed inline. This also removes today's + deploy-time coupling: no dbt build runs at deploy, deploys ship code + only, and the warehouse converges asynchronously. + +Exit: every product measure and metric exists as validated YAML +reconciled into the new store; old system untouched and serving. + +## Phase 2 — Compiler and Cutover + +1. **Compiler.** `domain/compiler/`: definition + request (entity scope, + date range, grain, dimensions, filters) → ClickHouse SQL over + datasets. Owns filter/expression rendering from validated ASTs, + per-dataset read discipline (FINAL/dedup from dataset metadata), + tenancy predicates, grain bucketing in the tenant reporting time zone, + metric composition (all four computation types), the transform stage, + and guardrails (row caps, timeout classes, memory settings) so a + pathological definition degrades to an error, not an incident. +2. **Parity, then cutover, per source family.** "Family" (git, tasks, + wiki, collab, ai) is a migration-only unit — it is how the old + observation tables, seeds, and e2e fixtures are already grouped, so it + is the natural granule for flipping executors coherently. The concept + retires with the old system in Phase 3; steady-state decisions + (materialization, refresh) are per measure, never per family. The e2e + suite is the invariant: same bronze seeds, same requests, expectations green against + the compiler path. A temporary flag selects executor per family; + shadow-compare on live traffic per family until divergence classes + (dedup, timezone, null propagation) are resolved deliberately; flip; + next family. Shadow runs double as the latency measurement that + triggers Decision 1's cache pull-forward. The flag and comparison harness are explicitly temporary + — weeks per family, deleted at Phase 3, not a permanent dual-execution + capability. +3. **Materialization where measured, not assumed.** Cutover requires + parity and acceptable latency. Where query-time compute over datasets + misses the latency budget for a family, the compiler gains its cache + tier early: materialized results keyed by + `(definition key, definition_version)`, refreshed by + backend-scheduled jobs, reads falling back to compilation on version + mismatch or cache miss. This is the target's cache design pulled + forward on demand — never the old observation contract kept alive. + +Exit: every product metric serves through the compiler; observation +relations receive no reads. + +## Phase 3 — Deletion + +Immediately after the last family cuts over, while context is fresh: + +- gold observation models and shape macros, their dbt shape tests, and + the observation build step; +- the old serving query builder in `metric_results/` (request validation + and response DTOs remain with the API); +- row-probing validator machinery, `Unchecked` state, + `dimension_not_covered` plumbing, backoff/probe scaffolding that + existed to reconcile definitions with observation relations; +- old store tables, after the one-shot migration of any custom rows; +- the per-family executor flag and parity harness. + +Exit: one executor, one store schema, no dead branches. The system is +smaller than before the project started. + +## Phase 4 — Catalog and Discovery + +1. **Field catalog, full form.** The Phase 1 artifact (names, types, + roles) extends with everything discovery needs: value↔label pairing + surfaced per dimension, conformed dimension keys, display metadata. + Same generator, same `schema.yml` `meta:` source, same build-time + transport — until custom datasets exist, catalogs change only with + product deploys. +2. **Discovery API.** Extend `domain/catalog/`: metrics and measures with + derived allowed dimensions (intersection over inputs, + dimension-agnostic inputs as identity elements), dataset field + catalogs for editor scope, distinct-dimension-values endpoint + (paginated, explicit `unavailable` state, served from the cache tier + or bounded dataset scan). +3. **Frontend adoption.** Pickers consume discovery; no client-side + capability knowledge, per the server-owned-meaning boundary. + +Exit: an editor could be built against discovery alone; empty tenants +report full product capability. + +## Phase 5 — Runtime Editing + +Product order: dashboards/charts → metrics → measures. + +1. **Chart/dashboard definitions.** `chart_definitions`, + `dashboard_definitions` (metric ref, view type, dimension, layout, + thresholds), tenant-scoped, product seeds from today's hardcoded + dashboards, `ON DELETE RESTRICT` to metrics, revisions written like + every other definition. +2. **Editing APIs.** CRUD per layer, role-gated, tenant-namespaced keys, + sharing Phase 1 validators (one write path). Deletion blocked while + referenced. +3. **Editors.** Chart/dashboard first (configuration over validated + capability), then metric editor over measures, then measure editor + over dataset field catalogs. Each layer opens only when the layer + below is discovery-covered. +4. **Custom datasets (gated, last, optional).** Admin-authored SELECT + registered as a dataset: parsed, validated, schema captured into the + catalog; default off. Catalog transport moves from build-time artifact + to store-backed at this point, not before. + +Exit: an administrator creates a measure, composes a metric, places it on +a dashboard, and sees correct data — zero deploys, full audit history. + +## Cross-Cutting + +- **The e2e suite is the safety rail for the rewrite.** It pins behavior + through Phases 2–3 (executor swap, deletion) without pinning + implementation; new fixtures cover compiler-only behavior (grain + variants, timezone edges, adversarial expressions rejected by + validators). +- **API stability.** `/v1/metric-results` and `/v1/metrics/queries` + contracts do not change during the rewrite; discovery grows additively + in Phase 4. +- **Migrations** forward-only per convention; the store rewrite is + additive tables + seed repopulation + late drops, never in-place + mutation of the old schema. +- **Observability.** Compiler logs definition key + version + tier + (computed/cache) per query; shadow divergence and validator rejections + are first-class metrics during Phase 2. + +## Dependencies + +```text +Phase 1 ──► Phase 2 ──► Phase 3 (deletion) + │ + └──► Phase 4 ──► Phase 5 +``` + +Phase 4 needs only Phase 1 definitions plus compiler-derived capability, +so it can start during late Phase 2. Phase 5 chart/dashboard work can +start once Phase 4 discovery serves metrics. + +## Decisions + +1. **Materialization is exception, not default.** Query-time compute is + the default for every family. A family gets the cache tier pulled + forward only if its shadow-phase p95 on the batch endpoint exceeds the + current path's p95 by more than a fixed margin agreed before shadow + starts. The shadow harness already runs both executors on live + traffic, so the latency evidence arrives for free; no separate + measurement project. +2. **Reporting time zone is a tenant profile setting**, default UTC, with + no per-request override. A per-request time zone makes the same + request mean different things per client, which violates server-owned + semantics. Exploration-style TZ shifting, if ever wanted, is a new + explicit API capability — not a bucketing parameter. +3. **The catalog generator is Rust**, sharing the backend's definition + and catalog parsers. A generator in another language is a second + parser of the same source of truth — a small instance of exactly the + duplication this project exists to remove. The extra build wiring is + trivial against that. +4. **Peer comparison is a compiler query mode; cohort membership is a + dbt dataset.** Peer comparison evaluates the same metric over cohort + members and aggregates the results — request-shaped, not + definition-shaped — so it belongs to the compiler like grain does, + parameterized at request time. Cohort membership is derived data with + ingestion-grade correctness concerns, which is what datasets are for. diff --git a/docs/domain/semantic-layer/specs/PRD.md b/docs/domain/semantic-layer/specs/PRD.md new file mode 100644 index 000000000..c66e0c376 --- /dev/null +++ b/docs/domain/semantic-layer/specs/PRD.md @@ -0,0 +1,527 @@ +# PRD — Semantic Layer + + + +- [1. Overview](#1-overview) + - [1.1 Purpose](#11-purpose) + - [1.2 Background / Problem Statement](#12-background--problem-statement) + - [1.3 Goals (Business Outcomes)](#13-goals-business-outcomes) + - [1.4 Glossary](#14-glossary) +- [2. Actors](#2-actors) + - [2.1 Human Actors](#21-human-actors) + - [2.2 System Actors](#22-system-actors) +- [3. Operational Concept & Environment](#3-operational-concept--environment) + - [3.1 Module-Specific Environment Constraints](#31-module-specific-environment-constraints) +- [4. Scope](#4-scope) + - [4.1 In Scope](#41-in-scope) + - [4.2 Out of Scope](#42-out-of-scope) +- [5. Functional Requirements](#5-functional-requirements) + - [5.1 Definitions and Compiler (p1)](#51-definitions-and-compiler-p1) + - [5.2 Time and Scope (p1)](#52-time-and-scope-p1) + - [5.3 Serving and Authoring (p2)](#53-serving-and-authoring-p2) +- [6. Non-Functional Requirements](#6-non-functional-requirements) + - [6.1 NFR Inclusions](#61-nfr-inclusions) + - [6.2 NFR Exclusions](#62-nfr-exclusions) +- [7. Public Library Interfaces](#7-public-library-interfaces) + - [7.1 Public API Surface](#71-public-api-surface) + - [7.2 External Integration Contracts](#72-external-integration-contracts) +- [8. Use Cases](#8-use-cases) + - [Author a Measure and a Metric](#author-a-measure-and-a-metric) + - [Register a Custom Dataset](#register-a-custom-dataset) + - [Promote a Definition to a Dashboard](#promote-a-definition-to-a-dashboard) +- [9. Acceptance Criteria](#9-acceptance-criteria) +- [10. Dependencies](#10-dependencies) +- [11. Assumptions](#11-assumptions) +- [12. Risks](#12-risks) + + + +## 1. Overview + +### 1.1 Purpose + +The Semantic Layer is the system through which every analytical value in Insight is defined, validated, computed, and served. Definitions — datasets, measures, metrics, charts, dashboards — are data, not code: a row in a definition store that authorized users can eventually edit at runtime through structured editors. A single server-owned compiler turns each definition plus a request into warehouse SQL; storage, caching, and materialization are private implementation details behind the definition contract. + +This PRD covers Phase B of the presentation-layer split (epic constructorfabric/insight#1803), after Phase A drew the read-only, tenant-scoped boundary. It formalizes definitions-as-data, one compiler over datasets, a gated custom-dataset SQL layer, capability derived from definitions rather than stored rows, server-owned semantics with query-time grain and a tenant reporting timezone, a discovery API, runtime editing, and — critically — the server-injected scopes (tenant, org-scope entity visibility, cohort isolation) the client can never widen. The full design rationale lives in [REFERENCE.md](./REFERENCE.md); the migration sequence in [IMPLEMENTATION.md](./IMPLEMENTATION.md); the adoption review and open items in [FINDINGS.md](./FINDINGS.md). + +### 1.2 Background / Problem Statement + +Today metric meaning is authored twice and lives in two places at once: as dbt SQL in gold observation models and again as Rust constants for the definition store, then pre-computed to observation rows at a baked day grain. Capability is inferred by probing observed rows, so an empty tenant reports no capability and partial ingestion can silently mutate the served contract. Adding one metric takes three deploys and two hand-synchronized restatements, and the chart is frontend code. + +This blocks self-serve analytics and safe evolution. There is no single executor, so two interpreters of one definition can diverge on null handling, time zones, and deduplication. There is no runtime authoring, no query-time grain, and no place for expressive analytics (joins, windows, funnels) that is safe by construction. Authorization scopes beyond tenancy — who may see which people, which cohorts — are enforced at the request boundary rather than uniformly at the compiler, so a new read path can miss them. + +Phase B replaces this with one compiler over datasets and definitions-as-data. Meaning exists in exactly one place; capability is a projection of definitions; every query the compiler emits carries the same injected scopes. Expressiveness that the structured layers deliberately cannot reach lands in a single gated SQL layer (custom datasets) with dataset-sized blast radius, never in a loosened editor. + +**Target users**: admins authoring measures and metrics; dataset authors registering custom SQL datasets under a gated role; analysts and viewers reading served values within their scope; the compiler service that owns semantics; the identity service that owns org-chart visibility. + +### 1.3 Goals (Business Outcomes) + +| Goal | Success Criteria | +|------|------------| +| Meaning lives in one place | **Baseline**: metric semantics authored twice (dbt SQL plus Rust constants) and pre-computed. **Target**: each definition exists once as data; adding or changing a metric is a reviewed definition change, no generated-SQL machinery. **Timeframe**: end of Phase B. | +| Capability independent of data | **Baseline**: capability probed from observed rows; empty tenant reports none. **Target**: capability is a projection of definitions; a tenant with zero ingested rows has full authoring capability. **Timeframe**: end of Phase B. | +| One executor, no drift | **Baseline**: dbt-emitted rows and a separate query builder can diverge. **Target**: a single compiler serves every definition; cutover holds e2e parity (same seeds, same requests, same expectations). **Timeframe**: Phase B cutover. | +| Every read is scoped by construction | **Baseline**: tenancy is injected; org-scope and cohort scope are enforced only at the request boundary. **Target**: tenant, org-scope entity visibility, and cohort isolation are all injected server-side at the single compiler choke point; the client cannot widen them. **Timeframe**: end of Phase B. | +| Self-serve authoring without deploys | **Baseline**: adding a metric or chart is a code deploy. **Target**: an authorized user creates a measure, composes a metric, and places it on a dashboard with zero deploys and full audit history. **Timeframe**: end of Phase B. | + +### 1.4 Glossary + +| Term | Definition | +|------|------------| +| Dataset | A queryable relation with guaranteed semantics — deduplicated, tenant-scoped, stable column names/types. Carries ingestion correctness (mutability, late data) so nothing above it re-solves those problems. Product-owned, with the one gated custom-dataset exception. | +| Custom dataset | A dataset-author-role registration of a SQL `SELECT` over dedup-safe catalog views. The single gated SQL layer; default off per tenant. | +| Field catalog | The typed, role-annotated schema generated from dataset schemas: the editor palette, the compiler's validation universe, and the discovery vocabulary. | +| Measure | A declarative aggregation of one dataset — the lowest editable layer and the atom every metric is built from. | +| Metric | A composition of measures into a served value: computation, optional post-aggregation shaping, and display identity (direction, format, naming). | +| Chart / dashboard | Pure presentation over a metric — view, dimension, layout, thresholds, targets. No query semantics. | +| Compiler | The single executor: definition plus request in, warehouse SQL out. Owns catalog resolution, rendering, read discipline, the time model, and all injected scopes. | +| Definition store | Application-owned storage of definitions with versioning, referential integrity, ownership/tenancy, and audit. | +| Materialization cache | A compiler-managed, version-keyed cache of aggregated work (not answers), refreshed on a schedule, never on read. | +| Injected scope | A server-side predicate the compiler adds to every query from request context — tenancy, org-scope entity visibility, cohort isolation — that the client cannot widen. | +| Org scope | The set of individuals within a viewer's org-chart reach (self plus related subtree), resolved from the identity service; the boundary for entity visibility and cohort composition. | +| Reporting timezone | The single declared per-tenant time zone in which the compiler buckets time; not a per-request parameter. | + +## 2. Actors + +### 2.1 Human Actors + +#### Analyst / Viewer + +**ID**: `cpt-semantic-layer-actor-analyst` + +**Role**: Reads served metric values through charts and dashboards, requests by metric key with a scope, range, grain, dimensions, and view. Owns the correctness judgment on what a value means for their decision. +**Needs**: To read values that are correct, self-describing, and confined to their own scope, without ever seeing people or cohorts outside it. + +#### Admin (Definition Author) + +**ID**: `cpt-semantic-layer-actor-admin` + +**Role**: Authors and edits measures, metrics, charts, and dashboards through structured editors over the field catalog. Product definitions are reviewed code; tenant definitions are this actor's runtime creations. +**Needs**: To compose and change analytical definitions at runtime, validated before they exist, with full audit history and no deploy. + +#### Dataset Author + +**ID**: `cpt-semantic-layer-actor-dataset-author` + +**Role**: Holder of a distinct, per-tenant, default-off role who registers a SQL `SELECT` as a custom dataset — the sanctioned home for cross-dataset joins, sequence logic, and correlated comparisons the measure layer cannot express. +**Needs**: Full SQL expressiveness over the governed catalog read surface, with the platform guaranteeing tenancy, isolation, and resource safety around it. + +### 2.2 System Actors + +#### Compiler Service + +**ID**: `cpt-semantic-layer-actor-compiler-svc` + +**Role**: The analytics service (Rust) that owns the definition store, the single compiler, the field-catalog artifact, the materialization cache, and the discovery and definition APIs. It compiles every definition and injects every scope. + +#### ClickHouse + +**ID**: `cpt-semantic-layer-actor-clickhouse` + +**Role**: External warehouse holding datasets (silver/gold relations and custom-dataset views) and the compiler-managed materialization cache. Executes the compiled SQL; enforces read discipline through the dataset views. + +#### Identity Service + +**ID**: `cpt-semantic-layer-actor-identity-svc` + +**Role**: External authority for the org chart and entity visibility. Resolves the set of persons within a viewer's org scope, which the compiler injects as the entity-visibility scope. Authoritative over cohort composition. + +## 3. Operational Concept & Environment + +### 3.1 Module-Specific Environment Constraints + +- **Single reporting timezone per tenant**: time bucketing happens in one declared per-tenant zone; there is no per-request time-zone override, so a request never changes meaning across executors or cache tiers. +- **Warehouse convergence is asynchronous**: deploys ship code and definitions only; the warehouse converges on its own schedule, so definitions referencing not-yet-present relations enter an explicit `unavailable` state rather than failing a deploy. +- **Custom SQL reads dedup-safe catalog views only**: raw source payloads, physical tables, and pipeline intermediates are not referenceable; read discipline is unbreakable by construction. + +## 4. Scope + +### 4.1 In Scope + +- Definitions (datasets, measures, metrics, charts, dashboards) authored and stored as data in one canonical format. +- A single compiler that turns any definition plus request into warehouse SQL at query-time grain, with tenant-timezone bucketing. +- A gated custom-dataset layer: role-gated, default off, wrapped with tenancy and resource guardrails, reading dedup-safe catalog views only. +- Capability derived from definitions and configuration, never from stored derived rows. +- Server-injected scopes on every compiled query: tenant isolation, org-scope entity visibility, and cohort isolation. +- A materialization cache of aggregated work, version-keyed, refreshed on schedule; a discovery API; runtime editing in dependency order (charts/dashboards → metrics → measures → custom datasets). + +### 4.2 Out of Scope + +- Physically re-hosting or re-ingesting base facts — the dataset layer reuses existing bronze→silver ingestion and class contracts. +- A general-purpose BI platform beyond the editability boundary — moving that boundary is an explicit design decision, not incremental widening. +- A per-request time-zone parameter or client-composed queries — semantics are server-owned. +- **Safety-critical / life-critical behavior**: Not applicable — internal analytics platform, no physical actuation. +- **Accessibility, i18n, offline operation**: Not applicable — internal analytics platform with an authenticated, online operator audience for Phase B. +- **Payments, PCI, HIPAA**: Not applicable — internal analytics platform, no payment or regulated-health data handled. + +## 5. Functional Requirements + +> **Testing strategy**: All requirements verified via automated tests (unit, integration, e2e) targeting 90%+ code coverage unless otherwise specified. The declarative e2e metric suite is the cutover parity invariant. + +### 5.1 Definitions and Compiler (p1) + +#### Definitions as Data + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-definitions-as-data` + +The system **MUST** represent every analytical definition — dataset, measure, metric, chart, dashboard — as a validated row in a definition store in one canonical format, never as code. Product-shipped and user-created definitions **MUST** be the same kind of object, differing only in ownership. An invalid definition **MUST NOT** be writable. + +**Rationale**: Anything a user may eventually edit must be data so it can be authored, versioned, audited, and validated uniformly. + +**Actors**: `cpt-semantic-layer-actor-admin`, `cpt-semantic-layer-actor-compiler-svc` + +The authoring-as-data first step shipped: the metric registry moved from Rust constants to a validated YAML registry (#1974). That registry still uses the transitional observation-relation shape; the target rewrites it to the dataset/measure/metric domain model. This requirement stays open until the domain-model store is the single runtime source of truth for every consumer. + +#### One Compiler over Datasets + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-one-compiler` + +The system **MUST** serve every definition through a single compiler that takes a definition plus a request (entity scope, date range, grain, dimensions, filters) and produces warehouse SQL. Product and user definitions **MUST** compile through the same path. A second executor **MUST** exist only as a bounded transitional state during cutover. + +**Rationale**: Two interpreters of one definition format diverge on nulls, time zones, and deduplication; one executor removes that drift class structurally. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-clickhouse` + +#### Gated Custom-Dataset SQL Layer + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-gated-custom-dataset` + +The system **MUST** allow a holder of the dataset-author role (per-tenant, default off) to register a SQL `SELECT` as a custom dataset. The statement **MUST** read only dedup-safe catalog views (never raw tables, payloads, or pipeline intermediates); its references **MUST** be checked against the catalog and its result columns annotated with catalog roles. Execution **MUST** be wrapped so the tenancy predicate is applied from outside the statement and resource guardrails bound it. Deletion **MUST** be blocked while measures reference it. + +**Rationale**: Expressiveness the structured layers cannot reach (joins, windows, sequence logic) needs exactly one gated home with dataset-sized blast radius, safe by construction. + +**Actors**: `cpt-semantic-layer-actor-dataset-author`, `cpt-semantic-layer-actor-compiler-svc` + +#### Capability from Definitions + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-capability-from-definitions` + +The system **MUST** derive what can be asked (metric and measure catalog, dimension keys) from shipped code and stored definitions, never from stored derived rows. A tenant with zero ingested rows **MUST** report full authoring capability. Data values **MUST** determine only which concrete filter values exist, served through a dedicated distinct-values path with an explicit unavailable state. + +**Rationale**: Capability inferred from data makes empty tenants dead, lets partial ingestion mutate the contract, and lets ingestion defects silently remove features. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-admin` + +#### Server-Owned Semantics + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-server-owned-semantics` + +The system **MUST** own all query semantics — filters, formulas, windows, formatting identity — on the server. Clients **MUST** request by key and render what they are given, holding no metric vocabulary, dimension lists, or validation semantics. Query responses **MUST** be self-describing and carry provenance (definition version, cache-or-computed, data-available-from, availability state). + +**Rationale**: Meaning fragments and drifts if any of it lives in a client; server-owned meaning makes it structural rather than conventional. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-analyst` + +### 5.2 Time and Scope (p1) + +#### Query-Time Grain and Reporting Timezone + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-query-time-grain` + +The system **MUST** treat grain (day, week, month, quarter) as a query-time parameter, never baked into stored data, and **MUST** bucket time in a single declared per-tenant reporting time zone with no per-request override. Windowed and cumulative semantics **MUST** be compiler features parameterized at request time, not encoded ad hoc into definitions. + +**Rationale**: Grain baked into materializations and per-request time zones both break server-owned semantics and re-aggregation. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-analyst` + +#### Tenant Isolation + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-tenant-isolation` + +The system **MUST** inject a tenancy predicate on every query the compiler emits, sourced from request context and not from any definition or client input. No definition **MUST** be able to widen or remove it. + +**Rationale**: Multi-tenant isolation must be a property of the single choke point, not of each definition. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-clickhouse` + +#### Org-Scope Entity Visibility + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-org-scope-visibility` + +The system **MUST** inject an entity-visibility scope on every compiled query so a viewer reads per-entity values only for individuals within their org-chart scope (self plus related subtree). People outside the scope **MUST NOT** be returned and their existence **MUST NOT** be disclosed. The visible set **MUST** be resolved from the org chart owned by the identity service, injected server-side beside the tenancy predicate, and **MUST** fail closed if the authorization source is unavailable. + +**Rationale**: "No people outside your org scope" is a required security guarantee that must hold uniformly for every read path, not per endpoint. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-identity-svc` + +This scope is partially enforced today at the request boundary: analytics resolves the caller's org-visible set via `person_visibility` → identity `/v1/visible-persons` and refuses out-of-scope ids (leaking only a count). Under compiler-first it moves into the compiler's shared `WHERE` as an injected `entity ∈ visible_set` filter; definitions stay scope-agnostic. This requirement stays open until every compiled read path carries the injected scope. + +#### Cohort Scope Isolation + +- [ ] `p1` - **ID**: `cpt-semantic-layer-fr-scope-isolation` + +The system **MUST** prevent per-entity reads for unrelated teams or cohorts. A person's peer cohort **MUST** be drawn from within their org-chart scope; tags and attributes **MUST** only refine a cohort inside that boundary and **MUST NOT** pull a person across it. Cross-cohort comparison **MUST** be aggregates-only: peer views return distributions with no member ids and **MUST** be suppressed below a minimum distinct-member floor so small groups cannot disclose individuals. + +**Rationale**: "No other teams/cohorts" — a shared tag must never route around org-scope isolation, and small cohorts must not re-identify members. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-identity-svc` + +### 5.3 Serving and Authoring (p2) + +#### Discovery API + +- [ ] `p2` - **ID**: `cpt-semantic-layer-fr-discovery-api` + +The system **MUST** tell clients what exists rather than let them hardcode or probe: the catalog of metrics and measures visible to the tenant (product ∪ tenant) with computation, format, direction, and allowed dimensions; for editors, dataset field catalogs with roles and types and a read-only display body per dataset; and on demand, distinct dimension values with an explicit unavailable state. Requests **MUST** be validated against the same catalog discovery serves. + +**Rationale**: The frontend must be a renderer of server-owned definitions; discovery is the single vocabulary that keeps requests and capability from disagreeing. + +**Actors**: `cpt-semantic-layer-actor-analyst`, `cpt-semantic-layer-actor-admin` + +#### Materialization Cache + +- [ ] `p2` - **ID**: `cpt-semantic-layer-fr-materialization-cache` + +The system **MUST** treat materialized results as a compiler-managed cache of aggregated work (not request→response answers), keyed by definition version, invisible to the definition contract. Refresh **MUST** be scheduled, never triggered by a read, and every degraded state (stale version, uncovered range, disabled policy, unavailable definition) **MUST** fall back to live compute rather than serve a wrong or superseded value. + +**Rationale**: Caching is a latency policy, not semantics; the read path must stay read-only and never serve values computed under a superseded definition. + +**Actors**: `cpt-semantic-layer-actor-compiler-svc`, `cpt-semantic-layer-actor-clickhouse` + +#### Runtime Editing + +- [ ] `p2` - **ID**: `cpt-semantic-layer-fr-runtime-editing` + +The system **MUST** let authorized users create and edit definitions at runtime through role-gated, tenant-namespaced editors, opening layers in dependency order (charts/dashboards, then metrics, then measures, then gated custom datasets). Each write **MUST** run the same validators as seed reconciliation, bump the definition version on semantic change, append an audit revision, and be blocked from deleting a referenced definition. + +**Rationale**: Runtime authoring without audit or shared validators is not acceptable in a multi-tenant product; the order keeps each layer opening only over a validated one below it. + +**Actors**: `cpt-semantic-layer-actor-admin`, `cpt-semantic-layer-actor-dataset-author` + +## 6. Non-Functional Requirements + +### 6.1 NFR Inclusions + +#### Source Read Discipline + +- [ ] `p1` - **ID**: `cpt-semantic-layer-nfr-source-read-discipline` + +Every read the compiler emits **MUST** apply the dataset's declared read discipline (deduplication, mutable-read handling) inherited from dataset metadata, and custom SQL **MUST** be unable to query around it because it reads only dedup-safe catalog views. + +**Threshold**: 0 reads that bypass dataset deduplication in adversarial testing; 0 references to raw tables or pipeline intermediates admitted by the custom-dataset validator. + +**Rationale**: Correctness of every value above the dataset line depends on read discipline being unbreakable by construction. + +#### Tenant Isolation + +- [ ] `p1` - **ID**: `cpt-semantic-layer-nfr-tenant-isolation` + +Compiled reads for tenant A **MUST NOT** return rows for tenant B, regardless of definition content or client input. + +**Threshold**: 0 cross-tenant rows returned in isolation testing across every compiled read path. + +**Rationale**: Multi-tenant SaaS isolation guarantee. + +#### Entity Visibility + +- [ ] `p1` - **ID**: `cpt-semantic-layer-nfr-entity-visibility` + +A viewer's compiled reads **MUST NOT** return per-entity values for individuals outside their org scope, and out-of-scope existence **MUST NOT** be disclosed. The scope source being unavailable **MUST** fail closed. + +**Threshold**: 0 out-of-scope persons returned or disclosed in scope testing; unavailable authorization source yields no rows, never a widened read. + +**Rationale**: "No people outside your org scope" is a hard security boundary. + +#### Cohort Scope Isolation + +- [ ] `p1` - **ID**: `cpt-semantic-layer-nfr-cohort-scope-isolation` + +Peer and cohort reads **MUST NOT** cross an org boundary for any cohort key, and aggregate peer views **MUST NOT** disclose members below the distinct-member floor. + +**Threshold**: 0 cross-boundary cohort members for any cohort key (not only `org_unit`); 0 peer views served below the minimum distinct-member floor. + +**Rationale**: A shared tag must not route around scope isolation, and small cohorts must not re-identify individuals. + +#### Executor Consistency + +- [ ] `p1` - **ID**: `cpt-semantic-layer-nfr-executor-consistency` + +Cutover to the single compiler **MUST** preserve behavior: the declarative e2e suite **MUST** pass with the same bronze seeds, the same requests, and the same expectations against the new executor. + +**Threshold**: 100% of the existing e2e metric expectations green against the compiler path; any divergence class (dedup, timezone, null propagation) resolved deliberately before flip. + +**Rationale**: The e2e suite is the parity invariant that makes the store rewrite and executor swap safe. + +#### Query Guardrails + +- [ ] `p2` - **ID**: `cpt-semantic-layer-nfr-query-guardrails` + +Every query the compiler emits — product, tenant, or custom-dataset — **MUST** run under resource guardrails (row caps, timeout classes, memory bounds) so a pathological definition degrades to an error, not an incident. + +**Threshold**: 100% of compiled queries bounded by a guardrail; a definition exceeding a bound fails loudly rather than running unbounded. + +**Rationale**: Runtime-created definitions can be pathological; guardrails make that a performance event, not a correctness or availability event. + +### 6.2 NFR Exclusions + +- **Per-request query latency SLA**: Not specified for Phase B — materialization is pulled forward per measure only where shadow-phase latency evidence warrants it; no committed per-request latency target. +- **High availability / clustering**: Managed at ClickHouse infrastructure level, not by this module. +- **Encryption at rest**: Handled by ClickHouse infrastructure configuration. + +## 7. Public Library Interfaces + +### 7.1 Public API Surface + +#### Query API + +- [ ] `p1` - **ID**: `cpt-semantic-layer-interface-query-api` + +**Type**: REST API (HTTP/JSON) + +**Stability**: stable + +**Description**: Request a served value by metric key with entity scope, range, grain, dimensions, filters, and view. Responses are self-describing and carry provenance. This is the contract Phase B holds stable across the executor cutover; detailed endpoint contracts live in DESIGN. + +**Breaking Change Policy**: Stable — the response shape is a presentation contract, orthogonal to how values are computed; changes are additive. + +#### Discovery API + +- [ ] `p2` - **ID**: `cpt-semantic-layer-interface-discovery-api` + +**Type**: REST API (HTTP/JSON) + +**Stability**: unstable + +**Description**: The catalog of visible metrics and measures with capability, dataset field catalogs for editors, and paginated distinct dimension values with an explicit unavailable state. Everything a client renders derives from here. + +**Breaking Change Policy**: Unstable in Phase B; grows additively. + +#### Definitions API + +- [ ] `p2` - **ID**: `cpt-semantic-layer-interface-definitions-api` + +**Type**: REST API (HTTP/JSON) + +**Stability**: unstable + +**Description**: Role-gated CRUD per definition layer, running the same validators as seed reconciliation, with validate-only dry runs, fork, and cache-policy changes. This surface is also the agent/MCP authoring surface — machine authors get the same endpoints and validation. + +**Breaking Change Policy**: Unstable in Phase B; hardens as runtime editing opens per layer. + +### 7.2 External Integration Contracts + +#### Org-Chart Visibility Consumption + +- [ ] `p1` - **ID**: `cpt-semantic-layer-contract-visibility-consumption` + +**Direction**: required from external (the compiler resolves each viewer's org-visible person set from the identity service) + +**Protocol/Format**: Token-forwarded lookup of the caller's visible-persons set, resolved from the identity-owned org chart; the compiler injects the result as an entity-visibility scope. + +**Compatibility**: Fail-closed if unavailable; the visible-set contract is authoritative over cohort composition and never widened by definitions. + +#### Warehouse Read Contract + +- [ ] `p2` - **ID**: `cpt-semantic-layer-contract-warehouse-read` + +**Direction**: required from external (the compiler reads datasets and the cache from ClickHouse) + +**Protocol/Format**: Compiled `SELECT` over dataset relations and dedup-safe catalog views; the materialization cache is a compiler-managed relation. + +**Compatibility**: Dataset contracts evolve additively; a relation lagging its declared contract puts affected definitions into an `unavailable` state rather than failing. + +## 8. Use Cases + +### Author a Measure and a Metric + +- [ ] `p1` - **ID**: `cpt-semantic-layer-usecase-author-measure-metric` + +**Actor**: `cpt-semantic-layer-actor-admin` + +**Preconditions**: +- The field catalog for the target dataset is available. +- The admin holds the measure/metric editing role for the tenant. + +**Main Flow**: +1. Admin opens the measure editor over a dataset's field catalog and composes an aggregation (filter tree, aggregation kind, event time, entity, dimensions). +2. System validates the definition against the catalog and expression allowlist and writes it, bumping the version and appending a revision. +3. Admin composes a metric over one or more measures (computation, transform, display identity). +4. System validates and stores the metric; capability appears immediately in discovery. + +**Postconditions**: +- The measure and metric exist as validated store rows; the compiler can serve them at any grain with zero materialization. + +**Alternative Flows**: +- **Validation fails**: System rejects the write with a precise field-level error; nothing is stored. + +### Register a Custom Dataset + +- [ ] `p2` - **ID**: `cpt-semantic-layer-usecase-register-custom-dataset` + +**Actor**: `cpt-semantic-layer-actor-dataset-author` + +**Preconditions**: +- The dataset-author role is enabled for the tenant. +- The referenced catalog datasets exist. + +**Main Flow**: +1. Author submits a SQL `SELECT` over dedup-safe catalog views and annotates result columns with catalog roles. +2. System parses the statement, checks references against the catalog, captures the result schema, and rejects any reference to raw tables or intermediates. +3. System registers the annotated schema as an ordinary catalog entry; measures may compose on top. + +**Postconditions**: +- The custom dataset is a versioned catalog entry, query-time by default, promotable to cache; lineage marks everything built on it. + +**Alternative Flows**: +- **Reference or role violation**: System rejects registration; nothing enters the catalog. + +### Promote a Definition to a Dashboard + +- [ ] `p2` - **ID**: `cpt-semantic-layer-usecase-promote-to-dashboard` + +**Actor**: `cpt-semantic-layer-actor-admin` + +**Preconditions**: +- A validated metric exists and is discovery-visible. + +**Main Flow**: +1. Admin creates a chart definition referencing the metric (view, dimension, thresholds). +2. Admin places the chart on a dashboard definition (layout). +3. System validates references, stores the definitions, and blocks deletion of the referenced metric. + +**Postconditions**: +- The dashboard resolves with its charts and their metrics' current capability in one read; no deploy occurred. + +**Alternative Flows**: +- **Referenced metric missing/unavailable**: System surfaces the availability state in discovery; the chart renders its error state. + +## 9. Acceptance Criteria + +- [ ] Every product metric and measure exists as a validated definition row in the canonical domain-model format, authored once. +- [ ] A single compiler serves every definition; the e2e suite passes with the same seeds, requests, and expectations against the compiler path. +- [ ] A tenant with zero ingested rows reports full product authoring capability. +- [ ] Every compiled query carries an injected tenancy predicate the client cannot widen; cross-tenant reads return no rows. +- [ ] Every compiled per-entity read is confined to the viewer's org scope; out-of-scope persons are never returned or disclosed, and an unavailable authorization source fails closed. +- [ ] No cohort read crosses an org boundary for any cohort key, and peer views below the distinct-member floor are suppressed. +- [ ] Grain is a query-time parameter and time buckets in the tenant reporting time zone; no grain is baked into a contract-bearing materialization. +- [ ] An authorized user creates a measure, composes a metric, and places it on a dashboard with zero deploys and full audit history. + +## 10. Dependencies + +| Dependency | Description | Criticality | +|------------|-------------|-------------| +| ClickHouse | Stores datasets and the materialization cache; executes compiled SQL | `p1` | +| Analytics service (Rust) | Hosts the definition store, compiler, discovery and definition APIs | `p1` | +| Identity service | Resolves org-chart visibility and is authoritative over cohort composition | `p1` | +| dbt / ingestion (class contracts) | Produces the datasets the measure layer builds on; carries dedup/mutability correctness | `p1` | +| Declarative e2e metric suite | The cutover parity invariant (same seeds, requests, expectations) | `p1` | +| Presentation-layer Phase A boundary | The read-only, tenant-scoped boundary this phase builds semantics on | `p2` | + +## 11. Assumptions + +- Existing bronze→silver ingestion and class contracts are reused as the dataset layer; base facts are not re-ingested. +- Product definitions are authored and reviewed as repo files and reconciled into the store at startup; the store is the single runtime source of truth. +- The reporting time zone is a per-tenant profile setting (default UTC) with no per-request override. +- Materialization is the exception, not the default: query-time compute is the default, cache pulled forward only where latency evidence warrants. +- The identity service is the authority for org-chart visibility and cohort composition; analytics never invents scope. + +## 12. Risks + +| Risk | Impact | Mitigation | +|------|--------|------------| +| Executor drift during cutover | Compiler and legacy path disagree on nulls/timezone/dedup | Shadow-compare per family against the e2e invariant; resolve each divergence class before flip; delete the flag at deletion phase | +| Expression allowlist mis-calibration | Too narrow blocks valid measures; too wide leaks unsafe SQL | Start narrow; widen only by explicit reviewed additions the parser makes visible | +| Injected scope missing on a new read path | Cross-tenant, out-of-scope, or cross-cohort disclosure | Inject all scopes at the single compiler choke point; adversarial isolation tests over every compiled path; fail closed on unavailable authorization | +| Cohort key spanning the org boundary | A tag-based cohort discloses across scope | Enforce org-gating where cohort membership is produced and re-assert it as the compiler's injected scope for every cohort key | +| Serving a superseded definition from cache | Stale semantics presented as truth | Version-key every cached row; fall back to live compute on version mismatch, uncovered range, or unavailable definition | +| Runtime-created pathological definition | Unbounded query load | Bound every compiled query with row caps, timeout classes, and memory settings | diff --git a/docs/domain/semantic-layer/specs/README.md b/docs/domain/semantic-layer/specs/README.md new file mode 100644 index 000000000..0d84a5885 --- /dev/null +++ b/docs/domain/semantic-layer/specs/README.md @@ -0,0 +1,26 @@ +# Semantic Layer + +Adopted target architecture for Phase B of the presentation-layer split +(constructorfabric/insight#1803): every analytical value is defined, validated, +computed, and served through one compiler over datasets, with definitions as +data. + +Governed `cfs` artifacts: + +- [PRD.md](./PRD.md) — product requirements. +- [DESIGN.md](./DESIGN.md) — technical design (distills the reference narrative + into the governed template). + +Companion documents: + +- [REFERENCE.md](./REFERENCE.md) — the detailed design narrative (deep + rationale; the design of record for depth). +- [IMPLEMENTATION.md](./IMPLEMENTATION.md) — the migration plan (keep / rewrite + / delete, phased with a parity-checked cutover). +- [FINDINGS.md](./FINDINGS.md) — adoption review, sub-issue re-scope, the + org-scope authorization the design must name, and open review items. + +Read PRD.md + DESIGN.md before changing metric definitions, the compiler, or the +definition store; REFERENCE.md for the full rationale. The metrics-domain design +([docs/domain/metrics/specs/DESIGN.md](../../metrics/specs/DESIGN.md)) is the +current implementation contract this layer supersedes on cutover. diff --git a/docs/domain/semantic-layer/specs/REFERENCE.md b/docs/domain/semantic-layer/specs/REFERENCE.md new file mode 100644 index 000000000..cf3507f44 --- /dev/null +++ b/docs/domain/semantic-layer/specs/REFERENCE.md @@ -0,0 +1,651 @@ +# Technical Design — Semantic Layer + +Status: **adopted target architecture** for Phase B of the presentation-layer +split (epic constructorfabric/insight#1803) — the detailed design rationale. + +This is the in-depth reference narrative. The governed, template-conformant +artifacts are [PRD.md](./PRD.md) (product requirements) and +[DESIGN.md](./DESIGN.md) (technical design); they distill this narrative and +reference it for depth. The migration sequence is in +[IMPLEMENTATION.md](./IMPLEMENTATION.md); the adoption review and open items are +in [FINDINGS.md](./FINDINGS.md). Read this before changing metric definitions, +the compiler, or the definition store. + +The semantic layer is the system through which every analytical value is +defined, validated, computed, and served. Definitions — measures, metrics, +charts, dashboards — are data, editable at runtime by authorized users +through structured editors. Semantics are owned and executed by the server. +Storage, caching, and materialization are private implementation details +behind the definition contract. + +## Principles + +1. **Definitions are data.** Anything a user may eventually edit is a row in + a definition store, never code. Product-shipped definitions and + user-created definitions are the same kind of object with different + ownership. +2. **Truth is base facts plus definitions.** Any derived value is a cached + evaluation of a definition over base facts. Caches are invalidated by + definition version; they are never the contract. +3. **One executor.** A single compiler turns definitions into queries. + Two interpreters of one definition format diverge — null handling, time + zones, deduplication — so a second executor is only ever a bounded + transitional state. +4. **Capability comes from code and configuration, never from data.** What + can be asked is a function of shipped connectors and stored definitions. + A tenant with zero ingested rows has full authoring capability. Data + determines only which concrete values exist. +5. **No invented languages.** Every layer of the definition format reuses a + proven external shape. The system owns one small composition schema, not + an expression language. +6. **Semantics are server-owned.** Clients request by key and render what + they are given. Meaning — filters, formulas, windows, formatting + identity — never lives in a client. +7. **The safety boundary is structural.** Users compose from typed + catalogs through closed grammars. Free-form SQL exists at exactly one + gated layer with dataset-sized blast radius, or not at all. + +## Domain Model + +```text +dataset product-owned relation; the correctness boundary + └─ field catalog typed, role-annotated schema; the authoring palette + └─ measure aggregation of one dataset; lowest editable layer + └─ metric composition of measures; transform, format, direction + └─ chart / dashboard presentation only +``` + +### Dataset + +A dataset is a queryable relation with guaranteed semantics: deduplicated, +tenant-scoped, with stable column names and types. Datasets carry the entire +correctness burden of ingestion — mutability handling, late data, source +quirks — so that nothing above them ever re-solves those problems. + +Datasets are product-owned, with one gated exception — custom datasets — +specified below. + +### Custom datasets + +A holder of a distinct dataset-author role (per-tenant enablement, off by +default — authoring SQL is a sharper privilege than composing measures) +may register a SQL SELECT as a dataset. This is the sanctioned home for +everything the measure layer deliberately cannot express: cross-dataset +joins, sequence and window logic, nested aggregation, correlated +comparisons. + +The contract: + +- **Read surface.** Custom SQL reads the catalog'd datasets — which are + the row-level relations themselves (class contracts and purpose-built + relations), every row, every contract column — through dedup-safe + views rather than the underlying physical tables. Same rows, governed + door: the raw table objects, raw source payloads, and pipeline + intermediates are not referenceable. This makes read discipline + unbreakable by construction (an author cannot accidentally query + around deduplication) and preserves ingestion's freedom to rework + everything beneath the contract. Within that surface, full SQL — + joins, window functions, subqueries, nested grouping. +- **Registration.** The statement is parsed and its references checked + against the catalog; the result schema is captured, and the author + annotates result columns with catalog roles (entity, dimension, + measurable, event time). The annotated schema becomes an ordinary + catalog entry; measures compose on top structurally. +- **Execution.** Always wrapped: the tenancy predicate is applied from + outside the statement, and resource guardrails (timeout class, memory, + row caps) bound it. The product guarantees safety and isolation; + semantic correctness of the SQL is the author's, and lineage marks + everything built on a custom dataset so consumers see the provenance. +- **Lifecycle.** Versioned with revisions like every definition; + availability states apply (an upstream product dataset changing shape + sends the custom dataset and its dependents to `unavailable` through + the same reconcile loop); deletion is blocked while measures reference + it. Query-time by default, promotable to the version-keyed cache when + slow. +- **Chaining.** A registered custom dataset is a catalog entry, so + custom datasets may build on other custom datasets. Definitions form a + DAG: cycles are rejected at registration, and availability cascades — + an upstream dataset going `unavailable` takes its dependents with it, + each state carrying the root cause. Deep chains are the primary + candidates for cache promotion, since each layer is otherwise a + stacked view resolved at query time. +- **Blast radius.** A defective custom dataset breaks itself and its + dependents, visibly — never the measure engine, never another + dataset. +- **Forking.** Any dataset offers "derive custom dataset", prefilling + the editor with a starting point from its displayed body (product + bodies, shown as compiled SQL, may need mechanical adaptation to the + catalog read surface — the fork is a head start, not a guaranteed + verbatim copy). Combined with shadow definitions this is the sanctioned + runtime hotfix path when a product definition is believed wrong: fork + the dataset, fix it, build tenant measures and metrics on top, repoint + charts — corrected numbers with zero deploys, while product truth + stays untouched under its own keys. It is a stopgap by design: forks + do not track upstream changes, the divergence is surfaced on the fork + ("derived from X @ build N, upstream changed since"), and the expected + end state is a product fix followed by reverting the repoint and + deleting the fork. + +Datasets are the retention boundary. A measure can be evaluated only as far +back as its dataset's history reaches, and that horizon is part of the +served contract ("data available from …"), not an implementation surprise. +Retention is therefore a priced product commitment: measurable columns are +not pre-aggregated away on the assumption that no future definition will +need them. + +### Field catalog + +Every dataset publishes a typed catalog of its fields, each annotated with a +role: + +- **entity** — identifies who or what a row is about (person, team, + repository). Entity fields bind rows to the entity model used for + scoping and cohorting. +- **dimension** — groupable; a stable value key paired with a display + label. +- **measurable** — numeric, aggregatable. +- **event time** — timestamp candidates for time bucketing. + +The catalog is generated from dataset schemas, never hand-maintained. It is +simultaneously the editor's palette, the compiler's validation universe, and +the discovery API's vocabulary. A field absent from the catalog does not +exist, at any layer above. + +Dimensions with the same key across datasets are **conformed**: `repository` +means the same identity space wherever it appears. Conformance is declared +in the catalog, and it is what makes cross-measure composition and shared +filters meaningful. + +### Measure + +A measure is a declarative aggregation of one dataset — the lowest editable +layer, and the atom every metric is built from. + +```yaml +key: large_prs_merged +dataset: git_pull_requests +description: Merged pull requests changing at least 500 lines. +filter: + all: + - { field: state, op: eq, value: merged } + - { field: lines_changed, op: gte, value: 500 } +aggregation: count # count | sum | avg | min | max | count_distinct +value_expr: null # SQL fragment over catalog fields where the + # aggregation takes an operand +event_time: merged_at +entity: author +dimensions: [repository, source] +``` + +The format is a composition of proven shapes; none of it is invented: + +- **Envelope** — the dbt MetricFlow semantic-model shape: aggregation as a + closed enum, expression slots, an explicit aggregation time dimension. + Every corner case has a reference answer in a published spec rather + than a design meeting. +- **Filter grammar** — structured predicate trees in the MBQL / JSON Logic + style: `all` / `any` / `not` combinators over `{field, op, value}` + leaves, operators from a closed enum. Fields must resolve in the + dataset's catalog with a compatible type. +- **Scalar expressions** — SQL fragments, not a bespoke expression + language. Each fragment is parsed (warehouse dialect) and admitted only + if its AST contains exclusively: catalog column references, literals, + arithmetic, and allowlisted functions. Subqueries, table references, and + settings are rejected at the parser. SQL's fully specified semantics — + nulls, types, precedence — come for free; safety comes from the + allowlist, and every widening of the allowlist is an explicit reviewed + change. + +A measure's `dimensions` list is its dimension capability. There is no +separate capability registry: capability is a projection of definitions, +so it cannot drift from them. + +### Metric + +A metric composes measures into a served value: a computation over one or +more inputs (direct, ratio, derived expression over named inputs), an +optional post-aggregation shaping stage (affine transform, clamping), and +display identity (direction, format, naming). The MetricFlow metric types +are the reference vocabulary. + +A metric's dimension capability is derived, not declared: the intersection +of its inputs' dimension sets, where dimension-agnostic inputs (constants, +global denominators) are identity elements that do not shrink the set. + +Cross-dataset metrics compose **at the aggregated level**: each input +measure aggregates within its own dataset, and the compiler joins the +aggregates on entity, time bucket, and conformed dimensions. Row-level +joins across datasets are not a measure-layer capability; where a genuine +need exists, the join is built as a product-owned dataset and measured on +top. + +### Chart and dashboard + +Pure presentation: which metric, which view (timeseries, breakdown, single +value, table), which dimension, layout, thresholds, targets. No query +semantics of any kind. These are the first definitions to become +user-editable, because they are configuration over already-validated +capability. + +## Expressiveness and Its Limits + +The editable layer intentionally covers the aggregate-compositional class: +filtered counts, sums, averages, extrema, distinct counts, percentiles, +arbitrary arithmetic composition of those, over any catalog field, by any +declared dimension, at any grain, with windows, cohort comparison, and +shaping. This matches the expressiveness class the surviving semantic +layers converged on. + +What the editable layer deliberately cannot express, and where each case +goes instead — the rule being that anything row-relational or +order-dependent is dataset work below the line, while anything +aggregate-compositional stays above it. The first four are solvable +without product involvement wherever custom datasets are enabled: a +dataset author builds the dataset, and measures compose on top. + +- **Row-level cross-dataset relationships** → a joined dataset + (product-built, or custom), measured on top. +- **Sequence-dependent logic** (funnels, durations between events, + streaks) → a dataset that computes the ordering into columns + (intervals, stage timestamps); measures aggregate the result. +- **Second-order aggregation** (aggregate, regroup, aggregate again) → + a dataset holding the inner aggregation, or a compiler query mode if + the shape recurs broadly. +- **Correlated logic** (a row compared against its group's own + statistic) → a dataset with the baseline precomputed as a column. +- **Facts not ingested** → connector work; no layer conjures data. +- **As-of-history semantics** → snapshot-modeled datasets; measures see + only what the dataset preserves. +- **Non-affine shaping** → extension of the transform vocabulary, a + reviewed product change. + +Nothing useful is unreachable; the ladder decides who builds it and with +what safety guarantees. Every escape lands in the dataset layer or below, +where full SQL and ingestion-grade review are available — never in a +loosened editor. + +### Capability by declaration + +Structure is not all-or-nothing. All logic — joins, filters, computed +values, windows — may live in dataset SQL; what a definition *declares* +determines which platform features it receives, and every declaration is +optional. An absent declaration switches features off; it never rejects +the definition: + +| Declared | Unlocks | +| --- | --- | +| result schema only | rendering as a table or tile — legal, feature-dead | +| event-time field | timeseries, grains, windows, time zones | +| entity field | entity scoping, cohorts, peer comparison | +| dimensions (value/label bindings) | breakdowns, filters, discovery pickers | +| aggregation, platform-applied | re-aggregation across grain and scope, the rollup cache, composition into metrics | + +Two things can never move into the SQL, because they are what the +platform operates on: the final aggregation (embedding it bakes grain and +scope into a string and kills re-aggregation, caching, and composition) +and column roles (not reliably inferable from a statement). Everything +else is authoring preference. A fully declared measure is therefore not a +restriction on SQL — it is the SQL's type signature, five fields with no +logic in them, and it is the minimum structure that keeps the feature set +mechanical. + +Declarations are validated data, deliberately not row-shape conventions: +a declaration violation fails at write time with a precise error; a shape +convention fails at query time, in production. + +This grammar is also the intended surface for machine authorship. A +closed, catalog-validated definition language is a strictly better +target for AI assistants and connected agents than freeform SQL: +violations are machine-checkable and repairable, semantics are diffable +and auditable, and the editing API doubles as the agent tool surface. + +### Runtime completeness + +The runtime surface is complete over the class contracts: with custom +datasets enabled (including chaining), the entire product metric catalog +is expressible at runtime with no product code. Shipping product +definitions as reviewed code is therefore a governance choice — review, +data tests, CI, e2e coverage for semantics that deserve them — never a +capability boundary. The same metric is buildable either way; only its +ownership and assurance level differ. + +## Time Model + +Time is compiler-owned and uniform across every measure: + +- Each measure aggregates over its declared `event_time`. +- Grain (day, week, month, quarter) is a query-time parameter, not a + property of stored data. No grain is baked into materializations that + the contract depends on. +- Bucketing happens in a single declared reporting time zone per tenant; + the same request never changes meaning across executors or cache tiers. +- Windowed and cumulative semantics (trailing windows, to-date) are + compiler features parameterized at request time, not encoded into + definitions ad hoc. + +## Capability Model + +"What can be queried" has three independent layers. Conflating them produces +either dead editors on empty tenants or API contracts that drift with data. + +1. **Product capability** — what shipped datasets and definitions can + express. A function of code and product seeds. Identical for every + tenant, including one with zero ingested rows. +2. **Tenant configuration** — which definitions this tenant has created or + enabled. A function of the definition store. Runtime-editable. +3. **Data values** — which concrete repositories, categories, people + exist. A function of ingested rows. Serves filter value pickers only. + +Dimension *keys* (what can be grouped by) come from layers 1–2. Dimension +*values* (what can be filtered to) come from layer 3, through a dedicated +distinct-values endpoint with an explicit unavailable state for unscanned or +oversized value sets. Capability is never inferred from stored derived rows: +an empty installation would report no capability, partial ingestion would +mutate the contract, and an ingestion defect would silently remove features. + +## Compiler + +The compiler is the single executor: definition plus request (entity scope, +date range, grain, dimensions, filters) in, warehouse SQL out. It owns: + +- catalog resolution and type checking, +- filter-tree and expression-AST rendering, +- per-dataset read discipline (deduplication, mutable-read handling), + inherited from dataset metadata rather than re-decided per measure, +- dimension-capability validation, +- tenancy predicates on every query it emits, +- the time model, +- the post-aggregation transform stage. + +Product and user definitions compile through the same path; a product +measure is not a special case, only a differently owned row. + +## Materialization + +Materialized results are a compiler-managed cache tier, invisible to the +contract: the cache stores **work, not answers**. It holds each measure's +aggregated rows at the finest served grain, and any request shape is +served from them by re-aggregation (coarser grains, breakdowns, +composition, peer modes). One cached representation covers the entire +request space — the warehouse pre-aggregation pattern, as opposed to a +request→response cache, which pays off only when identical questions +repeat. A short-TTL response cache may sit on top for repeated identical +requests; it is an orthogonal add-on, never the mechanism for coverage or +correctness. + +What a cached row is depends on the aggregation kind, because the math +dictates the finest reusable form: + +- additive aggregations (`count`, `sum`, `avg` components, `min`, `max`) — + one row per entity × time bucket × dimension tuple; +- percentile-family — one row per source event (a median of medians is + wrong; percentiles compute at read over event rows); +- distinct-count — one row per counted subject, so cross-period distinct + counts stay exact. + +Physical layout is one shared cache relation for all measures — measure +results share one shape, so tenant-created measures never trigger DDL — +partitioned by `(measure key, definition version, time bucket)` so that +invalidation and refresh are atomic partition operations. Per-measure +relations are rejected: they bind schema operations to user actions and +grow unbounded with tenants × measures. + +**Caching is policy, not semantics.** Whether a measure is materialized +lives in a cache policy (enabled, refresh schedule, hot window, coverage +watermarks) separate from the definition; toggling it never changes the +definition version, because nothing about meaning changed. Product +definitions may ship a default policy; tenant definitions start uncached +and are promoted by policy change. Only measures are cached — metric +composition is cheap re-aggregation at read time, and caching terminal +results would multiply invalidation surface for no coverage gain. + +Refresh is scheduled, never triggered by reads: the read path stays +read-only, and no request ever pays a build or stampedes one. Because +datasets are mutable (late events, history rewrites), each refresh rebuilds +a recent hot window atomically by partition replacement; settled periods +stay put; a dataset reprocessing forces a full rebuild through the same +path. Coverage watermarks advance only after data lands, so the read path +never trusts a hole. + +**Dataset materialization.** Custom datasets follow the same principles +with row-level physics. Unmaterialized by default (a stored SELECT +resolved at query time), promoted by the same kind of cache policy when +slow. Because a dataset's schema is author-defined, each materialized +dataset gets its own relation per definition version — the one place DDL +is bound to a runtime object, bounded by happening only on promotion +(a rate-limited policy event, never on save) and only for promoted +datasets. Refresh is a full rebuild with an atomic swap: incremental +refresh cannot be inferred safely from arbitrary SQL over mutable +upstreams, so always-correct-and-expensive is the default and windowed +rebuild is a later author-declared opt-in. Version bumps invalidate +exactly as for measures (reads fall back to the live view). Refresh +walks the definition DAG topologically — upstream datasets before +dependent datasets before dependent measure caches — triggered by +ingestion completion like the availability probe. A failed rebuild keeps +serving the previous table and alarms: data staleness is a surfaced, +tolerable cache property ("as of" timestamps); semantic staleness never +is. Rebuilds run under the same resource guardrails as queries, so a +pathological definition fails its promotion loudly rather than becoming +a scheduled incident. + +The read decision, made per input measure independently: policy enabled, +cached version equals current definition version, and the requested range +inside the coverage watermarks → read cache; otherwise compile live from +the dataset. A single metric may mix cached and live inputs in one query. +Every degraded state — stale version, uncovered range, disabled policy — +falls back to live compute: slower, never wrong. + +Every cached row is stamped with the definition version it was computed +under. Version mismatch means recompute or reject — never silently serving +values computed under a superseded definition. History depth equals dataset +retention at evaluation time, in every tier. + +## Definition Store + +Definitions live in application-owned storage with these invariants: + +- **Versioning.** Semantic changes increment `definition_version`; + presentation-only changes do not. Versions drive cache invalidation and + audit. The version is never hand-maintained: every write path (seed + reconciliation, editing API) canonicalizes the definition's semantic + fields, compares against the stored body, and bumps on difference with a + compare-and-set — a forgotten manual bump would mean new semantics + served from an old cache, so the possibility is removed structurally. + Versions are strictly monotonic and never reused, even when a change + restores an earlier body: reusing a version would let a partially + invalidated cache present holes as truth. Correctness never depends on + cache cleanup: reads are version-keyed, so superseded cache entries are + logically dead the moment the version bumps, and physical cleanup is + unhurried background work. +- **Referential integrity.** Metrics reference measures; charts reference + metrics; dashboards reference charts. Deletion is blocked while + referenced or explicitly cascades. Keys are stable; display names change + freely. +- **Ownership and tenancy.** Product definitions are tenant-invariant and + read-only for tenants. Tenant definitions are tenant-scoped and + namespaced so they can never shadow product keys. Editing rights are a + role, per layer of the domain model. +- **Auditability.** Definition history is retained: who changed what, + when, from which version. Runtime editability without audit is not + acceptable in a multi-tenant product. +- **Fail closed on the control plane only.** A missing or unreadable + definition store is a startup error — the one dependency the service + cannot start without. "No definitions found" must never present as "no + capability". Warehouse state is never a startup gate: deploys ship + code and definitions only, and the warehouse converges on its own + schedule. +- **Warehouse divergence is a state, not a crash.** Product definitions + are validated in CI against the declared dataset contracts from the + same commit, so a definition referencing an undeclared field cannot + ship. At runtime the structural probe reconciles declared contracts + with the live warehouse: an absent relation (bootstrap, ingestion not + yet run) or a relation lagging the shipped contract puts the affected + definitions into an explicit `unavailable` state — excluded from + serving with a precise error, visible in discovery, self-healing when + the warehouse catches up; divergence persisting past a threshold is an + operations alarm. The probe is a continuous background reconcile loop + over warehouse metadata (with an ingestion-completion trigger as a + latency optimization, never the guarantee), distinct from + seed reconciliation, which runs at startup because its input changes + only on deploy. Definition availability state is stored with the + violation; serving and discovery read the stored state and never probe + inline. Tenant definitions invalidated by contract changes + get the same state, surfaced to the tenant's administrators. +- **One schema, defined once.** The definition format is defined as typed + code in the owning service; authored YAML and stored rows are + serializations of those types, and parser, validators, compiler, and + editing APIs consume the same types. Machine-readable schema for + external tooling is exported from them, never hand-maintained. +- **Validation before existence.** Product definitions are parsed and + semantically validated at build time (an invalid definition fails the + build), structurally validated against the warehouse at startup; tenant + definitions get the identical validators at API write time. An invalid + definition is never written. + +Product defaults ship as seeds and bootstrap into the store; from then on +the store is the single runtime source of truth for every consumer, +including whatever build-time machinery materializes caches. + +## Discovery API + +The server tells clients what exists; clients never hardcode or probe: + +- the catalog of metrics and measures visible to the tenant + (product ∪ tenant), with computation, format, direction, and allowed + dimensions per entry, +- for editors: dataset field catalogs with roles and types, including + each dataset's definition body for display — custom datasets from their + store row, product datasets as a build-generated artifact of the + shipped model SQL, read-only and stamped with the build it came from. + One source of truth per body is preserved: the display copy is derived + at build time, never authored, and never executed, +- on demand: distinct dimension values from data, paginated, with an + explicit unavailable state. + +Requests are validated against the same catalog the discovery API serves, +so the two can never disagree. The validation error path exists for stale +or handcrafted clients; the designed path is pickers that render only valid +choices. + +## API Surface + +Four groups; the query contract is stable, the rest additive. + +- **Query** — request by metric key with entity scope, range, grain, + dimensions, filters, view. Responses are self-describing and carry + provenance: definition version, cache-or-computed, data-available-from, + availability state. +- **Discovery** — the catalog of visible metrics and measures with their + capability; dataset field catalogs with roles, retention, origin, + display body, and lineage; paginated dimension values with an explicit + unavailable state. Everything a client renders comes from here. +- **Definitions** — role-gated CRUD per layer, running the same + validators as seed reconciliation with precise field-level errors; + validate-only dry runs; fork; cache-policy changes. Semantic writes + bump versions and append revisions; deletes are blocked with the + referencing keys while referenced. This surface is also the agent/MCP + tool surface — machine authors get the same endpoints and the same + validation, never a separate path. +- **Dashboards** — a dashboard definition resolved with its charts and + their metrics' current capability in one read. + +The frontend is a renderer of definitions: dashboards, pickers, editors, +and error states all derive from discovery and dashboard payloads. It +holds no metric vocabulary, no per-chart dimension lists, no validation +semantics, and synthesizes no metric copy — server-owned meaning becomes +structural rather than conventional. Adding or changing a metric, chart, +or dashboard is a data change that ships no frontend code. + +## Editability Boundary + +| Layer | Editable by | Mechanism | +| --- | --- | --- | +| datasets | product | ingestion code | +| custom datasets | dataset-author role, default off | registered SQL SELECT over catalog'd datasets | +| field catalog | nobody | derived from dataset schemas | +| measures | admin | structured editor over the field catalog | +| metrics | admin | structured editor over measures | +| charts / dashboards | admin, optionally end users | structured editor over metrics | + +The boundary is the product's answer to "how much BI freedom": full +composition freedom above the dataset line, none below it. Moving the line +is a design decision, never an incremental widening. + +## Adoption Order + +Independent of any current implementation, the layers come online in +dependency order, each shipping value alone: + +1. **Definitions as data.** All product measures and metrics exist as + store rows in the target format, whatever still executes them. + Versioning, integrity, and namespacing land here, when they are cheap. +2. **Single compiler on the read path.** The compiler serves product + definitions, shadow-verified for parity against whatever it replaces; + alternative executors retire on cutover. +3. **Runtime editing, presentation first.** Dashboards and charts, then + metrics, then measures — each layer opening only after the one below it + is served by the compiler and covered by discovery. + +## Alternatives Considered + +- **Deriving capability from stored derived rows.** Rejected: empty + tenants report no capability, partial ingestion mutates the contract, + defects silently remove features, discovery scans data. +- **Parallel declarations (emission-side and serving-side).** Rejected: + two sources of truth with no cross-check fail silently — unreachable + dimensions or valid-but-empty groups. +- **A complete invented expression DSL.** Rejected: invented languages + fail in their underspecified corners (nulls, typing, time); every + needed layer has a proven donor shape, so the system owns only a small + composition schema. +- **Embedding an external semantic-layer engine.** Rejected: an + additional service with its own caching, auth, and tenancy model to + operate and reconcile. Schema ideas are adopted; the executor is owned, + because it must enforce warehouse-specific read discipline and tenancy. +- **A key-value response cache (Redis-style) as the materialization + tier.** Rejected: it caches exact request→response pairs in a + combinatorial request space (low hit rate), cannot be joined or + re-aggregated by the warehouse, holds bulk rows at memory prices, and + invalidates by TTL and key patterns — the fuzzy invalidation that + serves stale semantics. Response caching remains available as an + orthogonal short-TTL layer on top. +- **Client-composed queries as the flexibility mechanism.** Rejected: + metric meaning fragments across clients, there is no governance seam + for access control, transforms, caching, or performance guardrails, and + storage schema becomes public API. +- **Users author SQL at the measure layer** ("why structure at all, if + the SQL hatch exists?"). Rejected because guardrails make SQL safe to + run, not understood — and every platform feature requires + understanding the definition: request-time grain and timezone need a + known event-time field and re-bucketable aggregation; dimension + capability, discovery, and validation derive from declared dimensions; + entity scoping and peer modes are injected uniformly; the rollup cache + depends on knowing the aggregation kind (a SQL result is opaque — + response-cacheable at best); contract changes are mechanically traced + to structured definitions but are string archaeology in SQL; and the + measure editor's audience is not SQL-literate. A metric authored as + SQL opts out of all of it — the documented tax of Metabase native + questions, and why Looker confines SQL to derived tables. The dividing + rule: SQL produces rows (datasets), structure produces meanings + (measures, metrics) — the layers the platform must operate on stay + structured; the layer that only yields a schema may be SQL. + +## Risks + +- **Allowlist calibration.** Too narrow blocks legitimate measures; too + wide leaks unsafe constructs. Start narrow; widen only by reviewed + additions, which the parser makes explicit. +- **Label instability.** Display labels ride data; a renamed entity + yields conflicting labels for one value key. The compiler owns one + resolution rule (latest observed label wins) stated in the read + contract. +- **Deploy and definition skew.** Additive capability changes are safe + mid-rollout; removals transiently reject valid requests and require a + deprecation window. +- **Editor-induced load.** Runtime-created measures compute at query time + until promoted; a pathological definition is a performance event, not a + correctness event. The compiler enforces guardrails (row limits, + timeout classes) as part of the contract. +- **Scope creep toward a general BI platform.** The editability boundary + table is the line. Changes to it are explicit design decisions with + this document updated first.