test(e2e): declarative YAML metric-test rig (replaces the CSV rig)

.claude/skills/metric-e2e-test/SKILL.md

@@ -0,0 +1,194 @@
+---
+name: metric-e2e-test
+description: "Author and validate declarative YAML e2e tests for analytics metrics (src/ingestion/tests/e2e/specs/*.test.yaml). Use when asked to write/scaffold/validate an e2e test for a metric, seed bronze data for a test, add a fixture for a dashboard metric, or check a *.test.yaml. Covers schemas/, templates/, $ref+sibling composition, bronze records with duplicates, the batch endpoint POST /v1/metrics/queries, and expect rules (in / mongo-style find / equal subset / CEL assert)."
+disable-model-invocation: false
+user-invocable: true
+allowed-tools: Bash, Read, Write, Edit, Glob, Grep
+---
+
+# Author a metric e2e test (declarative YAML)
+
+This skill writes and validates `*.test.yaml` fixtures that drive the full
+`bronze → dbt silver → gold view → analytics-api` path and assert the result.
+
+## Source of truth (reference — open only if you need the detail)
+
+This skill is self-contained for authoring. Consult these only when you need the
+precise algorithm/DoD, or when this file and the spec disagree (the spec wins) —
+no need to load them every time:
+
+- FEATURE: [docs/domain/bronze-to-api-e2e/specs/feature-yaml-rig/FEATURE.md](../../../docs/domain/bronze-to-api-e2e/specs/feature-yaml-rig/FEATURE.md) — flows, the `resolve` algorithm, the expect engine, DoD.
+- DESIGN: [docs/domain/bronze-to-api-e2e/specs/DESIGN.md](../../../docs/domain/bronze-to-api-e2e/specs/DESIGN.md) — principles `record-composition`, `schema-is-truth`; components `ref-resolver`, `schema-validator`, `expect-engine`.
+
+## Commands
+
+- `/metric-e2e-test create <name> --metric <uuid> --tables <t1,t2>` — scaffold a new `<name>.test.yaml` (+ any missing `schemas/` and `templates/`).
+- `/metric-e2e-test validate <path>` — resolve refs, schema-validate records, lint `cases`/`expect` without running ClickHouse.
+
+(Plain prose like "write an e2e test for the emails-sent metric" triggers the same flow.)
+
+## File layout
+
+```
+src/ingestion/tests/e2e/specs/
+  schemas/<db>.<table>.yaml      # one JSON schema per bronze table (all real columns)
+  templates/<group>.yaml         # reusable records (people, m365_email, …)
+  <name>.test.yaml               # the test (discovered by the *.test.yaml suffix)
+```
+
+Files under `schemas/` and `templates/` are NOT tests (no `cases`) and are skipped by discovery.
+
+## The format
+
+### Records, `$ref`, and overrides
+
+A record is a field map. It may carry `$ref: "<file>#/<json-pointer>"` to inherit
+from another record; **sibling keys override the base** (closest wins). Paths are
+relative to the file the `$ref` is written in; a `$ref` resolves in the context of
+its own file (a `#/...` ref inside `templates/people.yaml` stays local to it).
+
+```yaml
+# templates/m365_email.yaml
+templates:
+  m365_email:            # base — carries EVERY schema column (unused = null)
+    _airbyte_raw_id: "00000000-0000-0000-0000-000000000000"
+    _airbyte_extracted_at: "2026-01-05T00:00:00"
+    _airbyte_meta: "{}"
+    _airbyte_generation_id: 0
+    tenant_id: "00000000-0000-0000-0000-000000000000"
+    source_id: m365-test
+    sendCount: null
+    # … every other column …
+  alice_email:
+    $ref: "#/templates/m365_email"
+    userPrincipalName: alice@example.com
+```
+
+### `bronze` — what to seed
+
+Keyed by table name (the key IS the table + which schema validates it). Each row =
+`$ref` to a record + the fields under test. After resolution the row is **padded to
+the full schema** (missing columns → null) and validated (`additionalProperties:false`
+catches typos). Two identical rows = a real Airbyte re-sync duplicate (must dedup).
+
+```yaml
+bronze:
+  bronze_m365.email_activity:
+    - $ref: templates/m365_email.yaml#/templates/alice_email
+      reportRefreshDate: "2026-01-05"
+      unique_key: m365-alice-20260105
+      sendCount: 40
+    - $ref: templates/m365_email.yaml#/templates/alice_email   # duplicate → must NOT double
+      reportRefreshDate: "2026-01-05"
+      unique_key: m365-alice-20260105
+      sendCount: 40
+```
+
+### `cases` — batch request + expectations
+
+```yaml
+cases:
+  - name: <what this proves>
+    request:
+      url: /v1/metrics/queries
+      method: POST
+      body:
+        queries:
+          - id: collaboration            # echoed back as results[].id
+            metric_id: <uuid>
+            $top: 50
+            $filter: "person_id eq 'alice@example.com' and metric_date ge '2026-01-01' and metric_date le '2026-01-31'"
+            $orderby: metric_key
+    expect:
+      - assert: "status == 200"                       # HTTP code of the batch
+      - in: collaboration
+        assert: "result.status == 'ok'"                # this query's own status (batch HTTP stays 200 on per-query error)
+      - in: collaboration
+        find: { metric_key: m365_emails_sent }         # mongo-style selector → exactly one row (`it`)
+        equal: { value: 40, median: 20, range_min: 10, range_max: 40 }   # subset; unlisted fields ignored
+      - in: collaboration
+        assert: "size(items) == 20"
+      - in: collaboration
+        find: { metric_key: slack_dm_ratio }
+        equal: { value: null }
+```
+
+- `in` — select the batch result by request `id` (omit when there is one query).
+- `find` — exact field equality: `{field: value}` (selects one row). Anything richer (inequalities, counts, predicates) goes in a CEL `assert` — there is no second selector language.
+- `equal` — subset equality; use for exact ints / `null`.
+- `assert` — CEL boolean; use for inequalities / floats / counts.
+
+### `assert` (CEL) bindings
+
+Assembled in `e2e_lib/expect_engine.py::evaluate_case` (the `bindings` dict),
+converted to CEL in `_eval_cel`:
+
+| Binding | Value | Present when |
+|---|---|---|
+| `it` | the single row matched by `find` | only with `find` (else `null` → `it.x` errors) |
+| `items` | the selected result's `items` array | a result is selected (`in` or sole query) |
+| `result` | the selected result `{id, status, metric_id, items, page_info}` | a result is selected |
+| `results` | the full `results[]` of the batch | always |
+| `status` | the batch HTTP status code (int) | always |
+
+CEL is strictly typed and won't compare an `int` to a `double` — when a metric
+value may be integral (`40`) and you compare against a fractional literal, cast it:
+`double(it.value) > 39.5`. `status`/`size(...)` are ints (compare with int literals).
+For exact / `null`, use `equal` (Python `==`), not `assert`. CEL macros available:
+`size()`, `has()`, `.exists()`, `.all()`, `.map()`, `.filter()`.
+
+## Scaffolding a new test
+
+1. **Resolve the metric_id and its shape.** Find it in the seed catalog
+   (`grep -rn "<label>" src/backend/services/analytics-api/src/migration/*.rs`) and
+   the live `query_ref` rewrite for that metric. Note whether it returns a bullet
+   (`metric_key`/`value`/`median`/`range_*`) or per-person rows, and whether `median`
+   is company-wide (Team bullet) or team/org_unit (IC bullet).
+2. **Ensure a schema file per table.** If `schemas/<db>.<table>.yaml` is missing,
+   generate it from the REAL table (do not invent columns):
+   ```bash
+   export KUBECONFIG=<path to your dev cluster kubeconfig>
+   kubectl exec -n insight insight-clickhouse-0 -- clickhouse-client \
+     --query "SELECT name, type FROM system.columns WHERE database='<db>' AND table='<table>' ORDER BY position FORMAT TSV"
+   ```
+   Map CH types → JSON-schema: `Nullable(String)`→`[string,"null"]`, `Decimal/Float/Int`→`[number,"null"]` (`UInt*` non-null →`integer`), `Bool`→`[boolean,"null"]`, `DateTime*`→`{string, format: date-time}`, `JSON`→`[object,"null"]`. Set `additionalProperties: false` and list **every** column (incl. `_airbyte_*`).
+3. **Ensure base + variant templates.** The base record must contain every schema
+   column (incl. `_airbyte_*` — transforms depend on them); variants `$ref` the base
+   and override identity only.
+4. **Write `bronze`** with `$ref`+overrides; include a duplicate row when the metric
+   should dedup.
+5. **Write `cases`**: one batch `query` per metric under test; assert the few fields
+   that matter via `find`+`equal`, and counts/inequalities via `assert`.
+6. **Pick numbers that distinguish behaviors** — e.g. for a median test use values
+   where median ≠ mean (`[40,20,10]` → median 20, mean 23.33) so the test actually
+   pins the aggregation.
+
+## Validating a test (no ClickHouse needed)
+
+- Every `$ref` resolves (file + pointer exist); no cycles.
+- Each resolved+padded bronze record validates against `schemas/<table>.yaml`
+  (`additionalProperties:false`).
+- Base templates cover **all** schema columns (quick check):
+  ```bash
+  python3 - <<'PY'
+  import yaml
+  s=set(yaml.safe_load(open("schemas/<db>.<table>.yaml"))["schemas"]["<db>.<table>"]["properties"])
+  t=set(yaml.safe_load(open("templates/<group>.yaml"))["templates"]["<base>"]); t.discard("$ref")
+  print("missing", sorted(s-t), "extra", sorted(t-s))
+  PY
+  ```
+- Each `expect` rule has `find`+(`equal`|`assert`) or a bare `assert`; `in` matches a
+  declared query `id`; CEL expressions parse.
+
+## Running
+
+```bash
+cd src/ingestion/tests/e2e
+ls specs/*.test.yaml                       # list existing tests
+./e2e.sh test                              # run all tests (specs/ + meta/)
+./e2e.sh test -k <name>                    # run one test by name
+./e2e.sh test -k <name> -v                 # verbose (per-step log)
+./e2e.sh down                              # tear down the e2e compose stack + volumes (full reset)
+```
+
+`<name>` is the file stem (e.g. `collab_emails_sent` for `specs/collab_emails_sent.test.yaml`). Warm re-runs are fine — the session resets the multi-reader collab silver/staging tables at start (conftest). `./e2e.sh down` is only the e2e compose teardown (it is not a deploy), for when you want a fully clean ClickHouse.

cypilot/config/artifacts.toml

@@ -868,3 +868,9 @@ kind = "FEATURE"
 path = "docs/domain/bronze-to-api-e2e/specs/feature-csv-rig/FEATURE.md"
 name = "Bronze-to-API E2E Tests — CSV Rig Feature"
 traceability = "FULL"
+
+[[systems.artifacts]]
+kind = "FEATURE"
+path = "docs/domain/bronze-to-api-e2e/specs/feature-yaml-rig/FEATURE.md"
+name = "Bronze-to-API E2E Tests — Declarative YAML Rig Feature"
+traceability = "FULL"

docs/domain/bronze-to-api-e2e/specs/DESIGN.md

@@ -28,7 +28,8 @@
 
 ## Changelog
 
-- **v1.0** (current): Initial design. Establishes 7 components (fixture-loader, ch-seeder, dbt-runner, migration-applier, api-client, csv-asserter, session-rig), the data plane (docker compose with ClickHouse + MariaDB), the service plane (`analytics-api` binary on host with `cargo build --release`), and the assertion plane (pandas). Vertically slices through the bronze→silver→gold→API path defined in `cpt-dataflow-design-pipeline`.
+- **v1.1** (current): Authoring format moves from per-folder CSV (`bronze/*.csv` + `spec.yaml` + `expected/response.csv`) to a single declarative `<name>.test.yaml` (see `cpt-bronze-to-api-e2e-feature-yaml-rig`). Adds three components — `ref-resolver` (composition via `$ref` + sibling overrides), `schema-validator` (per-table JSON schema, padding, validation), `expect-engine` (exact-equality `find` + `equal` subset + CEL `assert` over the batch response) — and retires `csv-asserter`. `fixture-loader` is repurposed to load `*.test.yaml`. The API roundtrip targets the batch endpoint `POST /v1/metrics/queries`. The transformation path (bronze→silver→gold→API) is unchanged; only the authoring format and assertion engine change.
+- **v1.0**: Initial design. Establishes 7 components (fixture-loader, ch-seeder, dbt-runner, migration-applier, api-client, csv-asserter, session-rig), the data plane (docker compose with ClickHouse + MariaDB), the service plane (`analytics-api` binary on host with `cargo build --release`), and the assertion plane (pandas). Vertically slices through the bronze→silver→gold→API path defined in `cpt-dataflow-design-pipeline`.
 
 ## 1. Architecture Overview
 
@@ -129,7 +130,19 @@ Expensive setup (docker compose, ClickHouse migrations, dbt manifest parse, anal
 
 - [ ] `p2` - **ID**: `cpt-bronze-to-api-e2e-principle-fixtures-are-truth`
 
-The CSV under `expected/` is what the test asserts on. There is no "regenerate from current production" mode. The `--update-snapshots` flag exists only to bootstrap a new test or to acknowledge an intentional behavior change, and it MUST be invoked deliberately by a developer (never by CI).
+The test file is what the test asserts on. There is no "regenerate from current production" mode. Under the YAML rig (`feature-yaml-rig`) expectations are explicit `expect` rules; an author writes only the fields/conditions that matter, never a full response snapshot.
+
+#### Records compose by reference, not repetition
+
+- [ ] `p1` - **ID**: `cpt-bronze-to-api-e2e-principle-record-composition`
+
+A bronze row (or a template) is a field map that may carry a `$ref: "<file>#/<json-pointer>"` to inherit from another record; sibling keys override the base (closest wins). Reusable people/source records live in `specs/templates/*.yaml`. A test spells out only the fields it exercises; everything else is inherited. This keeps a test small while the seeded row stays complete.
+
+#### The table schema is the source of truth for a row's shape
+
+- [ ] `p1` - **ID**: `cpt-bronze-to-api-e2e-principle-schema-is-truth`
+
+Per-table JSON schemas live in `specs/schemas/<db>.<table>.yaml` (the file stem is the full bronze table name, e.g. `bronze_m365.email_activity.yaml`) and are resolved by that name (the `bronze` key IS the table). After `$ref` resolution a row is padded with every missing schema column as `null` and validated; `additionalProperties:false` catches a misspelled column. Base templates carry the full column set (including the non-nullable `_airbyte_*` CDK columns, which transforms such as `insight.people`'s `argMax(..., _airbyte_extracted_at)` depend on).
 
 ### 2.2 Constraints
 
@@ -278,10 +291,87 @@ Does **NOT**: assert response content (that's `csv-asserter`); insert MariaDB ro
 - `cpt-bronze-to-api-e2e-component-session-rig` — orchestrates spawn/teardown
 - `cpt-bronze-to-api-e2e-component-csv-asserter` — receives the `ApiResponse`
 
-#### CSV Asserter
+#### Ref Resolver
+
+- [ ] `p1` - **ID**: `cpt-bronze-to-api-e2e-component-ref-resolver`
+
+##### Why this component exists
+
+The readability of the YAML rig rests on `$ref` + sibling-override composition. Resolving that — across files, with cycle detection and base-in-its-own-file semantics — is a self-contained, pure transformation worth isolating and unit-testing in full (`cpt-bronze-to-api-e2e-dod-yaml-ref-resolution`, 12 invariants).
+
+##### Responsibility scope
+
+Implements `cpt-bronze-to-api-e2e-algo-yaml-resolve-refs`: walks a YAML node; on a `$ref` loads the target (cached), resolves the base in the target file's context, deep-merges sibling overrides on top; guards cycles. Pure — no I/O beyond reading referenced YAML files.
+
+##### Responsibility boundaries
+
+Does **NOT**: connect to ClickHouse; know about schemas (padding/validation is `schema-validator`); evaluate `cases`.
+
+##### Related components (by ID)
+
+- `cpt-bronze-to-api-e2e-component-fixture-loader` — calls the resolver on each record
+- `cpt-bronze-to-api-e2e-component-schema-validator` — consumes resolved records
+
+#### Schema Validator
+
+- [ ] `p1` - **ID**: `cpt-bronze-to-api-e2e-component-schema-validator`
+
+##### Why this component exists
+
+A resolved bronze row must be a complete, well-typed table row. Centralizing schema lookup-by-table-name, null-padding, and JSON-schema validation makes "the row matches the table" a single, fail-fast checkpoint.
+
+##### Responsibility scope
+
+Loads `specs/schemas/<db>.<table>.yaml`; pads a resolved record with missing schema properties as `null`; validates against the JSON schema (`additionalProperties:false`). Implements the post-step of `cpt-bronze-to-api-e2e-algo-yaml-resolve-refs`.
+
+##### Responsibility boundaries
+
+Does **NOT**: coerce values to ClickHouse types (that's `ch-seeder` at INSERT time); resolve `$ref` (that's `ref-resolver`).
+
+##### Related components (by ID)
+
+- `cpt-bronze-to-api-e2e-component-ref-resolver` — supplies resolved records
+- `cpt-bronze-to-api-e2e-component-ch-seeder` — consumes padded/validated records
+
+#### Expect Engine
+
+- [ ] `p1` - **ID**: `cpt-bronze-to-api-e2e-component-expect-engine`
+
+##### Why this component exists
+
+Replaces `csv-asserter`. Dashboard metrics return many rows over a batch of queries; an author asserts a few fields/conditions, not a whole CSV. This component owns selection (`in` + exact-equality `find`) and verdict (`equal` subset or CEL `assert`). Richer matching than equality is expressed in the CEL `assert`, so the rig carries no second selector language.
+
+##### Responsibility scope
+
+Implements `cpt-bronze-to-api-e2e-algo-yaml-eval-expect`: selects a batch result by `id`; filters `result.items` to exactly one row by exact field equality; compares a subset of fields (`equal`, explicit `null`) or evaluates a CEL boolean (`assert`). Renders a precise failing-rule report.
+
+The CEL `assert` bindings are assembled in `e2e_lib/expect_engine.py::evaluate_case` (the `bindings` dict) and converted to CEL in `_eval_cel`:
+
+| Binding | Value | Present when |
+|---|---|---|
+| `it` | the single row matched by `find` | only with `find` (else `null`) |
+| `items` | the selected result's `items` array | a result is selected |
+| `result` | the selected batch result `{id, status, metric_id, items, page_info}` | a result is selected |
+| `results` | the full `results[]` of the batch | always |
+| `status` | the batch HTTP status code (int) | always |
+
+CEL is strictly typed and won't compare `int` to `double`; bindings pass through unchanged, so an author casts a possibly-integral metric value (`double(it.value) > 39.5`). `status` and `size(...)` are `int`. Exact / `null` comparisons belong in `equal`.
+
+##### Responsibility boundaries
+
+Does **NOT**: call the API (consumes the deserialized `BatchResponse`); mutate state; support a snapshot-update mode (expectations are authored, per `cpt-bronze-to-api-e2e-principle-fixtures-are-truth`).
+
+##### Related components (by ID)
+
+- `cpt-bronze-to-api-e2e-component-api-client` — supplies the `BatchResponse`
+- `cpt-bronze-to-api-e2e-component-fixture-loader` — supplies the `cases`
+
+#### CSV Asserter (retired in v1.1)
 
 - [ ] `p1` - **ID**: `cpt-bronze-to-api-e2e-component-csv-asserter`
 
+> **Retired in v1.1** — superseded by `cpt-bronze-to-api-e2e-component-expect-engine`. Kept here for traceability from `feature-csv-rig`.
+
 ##### Why this component exists
 
 `pandas.testing.assert_frame_equal` is close to what we want but its diff output is awkward inside a pytest report and it doesn't honor `key_columns` for stable row ordering. This component is a thin, opinionated wrapper that produces the cell-precise diff format required by `cpt-bronze-to-api-e2e-nfr-diff-readability`.
@@ -332,7 +422,8 @@ The framework consumes the existing analytics-api HTTP surface. No new endpoints
 
 | Method | Path | Description | Stability |
 |--------|------|-------------|-----------|
-| `POST` | `/v1/metrics/{id}/query` | Execute a metric query (OData `$filter`, `$top`, `$orderby`, `$select`) | stable |
+| `POST` | `/v1/metrics/queries` | Batch metric query — `{queries:[{id, metric_id, $filter,...}]}` → `{results:[{id, status, items \| error}]}`. Primary roundtrip for the YAML rig. | stable |
+| `POST` | `/v1/metrics/{id}/query` | Execute a single metric query (OData `$filter`, `$top`, `$orderby`, `$select`) | stable |
 | `GET` | `/v1/metrics` | List metric definitions | stable |
 | `GET` | `/v1/metrics/{id}` | Get one metric definition | stable |
 | `GET` | `/v1/columns` | List column catalog | stable |