Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
706 changes: 706 additions & 0 deletions docs/components/backend/analytics/openapi.json

Large diffs are not rendered by default.

44 changes: 24 additions & 20 deletions docs/domain/presentation-layer/specs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Requirements that significantly influence architecture decisions.
| `cpt-presentation-fr-single-select-gate` | `validate_single_select` in `query_gate.rs`, wired through `parse_query_ref` — the one chokepoint metric SQL crosses on write and run |
| `cpt-presentation-fr-read-only-role` | Dedicated `presentation_ro` ClickHouse role: `SELECT` on the contract, `CREATE`/`INSERT` only in `presentation`, no `DROP`/`ALTER`/`TRUNCATE` |
| `cpt-presentation-fr-namespace` | New empty `presentation` database for new gold, saved-query results, and scratch; legacy gold left read-only in `insight` |
| `cpt-presentation-fr-saved-query-crud` | `presentation.queries` is a SeaORM entity in the service database (like metric definitions); CRUD mutates that metadata, not ClickHouse. Only `/run` reaches ClickHouse — it reuses the existing read path and executes the stored SQL as `presentation_ro`, so no write grant on the contract is ever needed |
| `cpt-presentation-fr-saved-query-crud` | The saved query (`presentation.queries` logically; the `saved_queries` table physically) is a SeaORM entity in the analytics **service database (MariaDB)**, like metric definitions; CRUD mutates that metadata, not ClickHouse. Only `/run` reaches ClickHouse — it reuses the existing read path and executes the stored SQL as `presentation_ro`, so no write grant on the contract is ever needed. Shipped (#1965) |
| `cpt-presentation-fr-query-params` | Named parameters, `tenant` always injected from context (not client SQL), `period` supported |
| `cpt-presentation-fr-tenant-filter` | Literal `insight_tenant_id = <ctx.tenant>` injected in one place — the compiler's shared `WHERE` — replacing the no-op |
| `cpt-presentation-fr-contract-surface-doc` | Contract surface documented as the read boundary (silver and identity objects) |
Expand Down Expand Up @@ -86,12 +86,15 @@ Requirements that significantly influence architecture decisions.
├──────────────────────────────────── │ ──────────────────────────┤
│ CONTRACT (read-only) ▼ presentation (write) │
│ ┌───────────────────────────┐ ┌───────────────────────────┐ │
│ │ silver class_*/fct_*/mtr_*│ │ presentation.queries │ │
│ │ identity person.*/identity│ │ new gold, results, scratch│ │
│ │ silver class_*/fct_*/mtr_*│ │ new gold / query results │ │
│ │ identity person.*/identity│ │ scratch (presentation DB) │ │
│ │ legacy gold in `insight` │ └───────────────────────────┘ │
│ └───────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
▲ produced by Engineering / ingestion layer (additive, versioned)
Saved-query metadata (`presentation.queries`) lives in the analytics
service DB (MariaDB), not the ClickHouse `presentation` namespace; only
`/run` reaches ClickHouse.
```

| Layer | Responsibility | Technology |
Expand Down Expand Up @@ -173,13 +176,13 @@ New gold does not add `insight_tenant_id` outside the coordinated engineering re

| Entity | Description | Location |
|--------|-------------|--------|
| `presentation.queries` | Saved query: a single `SELECT`/`WITH` over the contract, tenant-scoped | ClickHouse `presentation` DB |
| `presentation.queries` (`saved_queries` table) | Saved query: a single `SELECT`/`WITH` over the contract, tenant-scoped | Analytics service DB (MariaDB), like the metric CRUD entities |
| Query gate | `validate_single_select` — the syntactic read-only barrier | [query_gate.rs](../../../../src/backend/services/analytics/src/domain/query_gate.rs) |
| Metric compiler | Builds contract SQL; owns the shared `WHERE` where the tenant filter is injected | [metric_results/](../../../../src/backend/services/analytics/src/domain/metric_results/) |

**Relationships**:
- `presentation.queries.sql` → contract objects (silver, identity, legacy gold in `insight`): read-only `SELECT` validated by the gate.
- `presentation.queries.insight_tenant_id` → the tenant predicate injected on run.
- `presentation.queries.insight_tenant_id` → the tenant that owns the row; scopes all CRUD. It is also the source for the tenant predicate the run path will inject once #1967 replaces the compiler no-op.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

### 3.2 Component Model

Expand Down Expand Up @@ -257,23 +260,23 @@ The second, independent barrier behind the query gate: once analytics connects a

#### Saved-Query API

- [ ] `p2` - **ID**: `cpt-presentation-component-saved-query-api`
- [x] `p2` - **ID**: `cpt-presentation-component-saved-query-api`

##### Why this component exists

Plain CRUD over stored queries so a new analytics slice needs no engineering change and no re-ingest. The one new surface Phase A adds ("Data Analytics").
Plain CRUD over stored queries so a new analytics slice needs no engineering change and no re-ingest. The one new surface Phase A adds ("Data Analytics"). CRUD + run shipped (#1965); named parameters follow in #1966.

##### Responsibility scope

- CRUD over `presentation.queries`, tenant-scoped.
- Validate SQL via the query gate on create and update.
- Run: inject the tenant filter, execute read-only as `presentation_ro`, return untyped JSON rows (same shape as the existing metric query path).
- Named parameters: `tenant` always injected from context, `period` supported.
- CRUD over `presentation.queries` (the `saved_queries` service-DB table), tenant-scoped from the session `SecurityContext`. Handlers mirror the metric CRUD in `api::handlers`; delete is a hard delete.
- Validate SQL via the query gate (`validate_single_select`) on create, update, **and** run — the run-side re-validation keeps a stored SQL from reaching ClickHouse as anything but a single read.
- Run: execute the stored single-SELECT read-only as `presentation_ro` and return untyped JSON rows (`JSONEachRow`, same shape as the existing metric query path).

##### Responsibility boundaries

- Does NOT carry metric metadata, thresholds, or passports — those are Phase B.
- Does NOT bypass the gate or the tenant filter.
- Does NOT bypass the gate.
- Does NOT yet inject named parameters (`tenant`/`period`, #1966) or the tenant-row filter (#1967) — #1965 executes the stored SQL as authored; those cross-cutting concerns extend the run path in their own sub-issues.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

##### Related components (by ID)

Expand Down Expand Up @@ -335,8 +338,9 @@ Serves many experimental FE builds under one host against one shared read-only s

### 3.3 API Contracts

- [ ] `p2` - **ID**: `cpt-presentation-interface-saved-query-endpoints`
- [x] `p2` - **ID**: `cpt-presentation-interface-saved-query-endpoints`

- **Implements**: `cpt-presentation-interface-saved-query-api` (PRD §7.1 Public API Surface)
- **Contracts**: `cpt-presentation-contract-read-only-consumption`
- **Technology**: REST / HTTP JSON
- **Base path**: `/v1/queries`
Expand All @@ -352,7 +356,7 @@ Entity `presentation.queries`: `{ id, insight_tenant_id, name, description, sql,
| `GET` | `/v1/queries/{id}` | Fetch one | unstable |
| `PUT` | `/v1/queries/{id}` | Update (re-validates SQL) | unstable |
| `DELETE` | `/v1/queries/{id}` | Delete | unstable |
| `POST` | `/v1/queries/{id}/run` | Execute read-only, inject tenant filter, return rows | unstable |
| `POST` | `/v1/queries/{id}/run` | Execute read-only as `presentation_ro`, return rows (tenant filter injected in #1967) | unstable |

`run` executes as `presentation_ro` and returns untyped JSON rows, the same shape as the existing metric query path. No metric metadata, thresholds, or passports in Phase A.

Expand Down Expand Up @@ -455,15 +459,15 @@ sequenceDiagram

### 3.7 Database Schemas & Tables

- [ ] `p3` - **ID**: `cpt-presentation-db-schemas`
- [x] `p3` - **ID**: `cpt-presentation-db-schemas`

Only one new entity is added in Phase A. It lives in the `presentation` namespace; contract objects are unchanged (read-only).
Only one new entity is added in Phase A. It lives in the analytics **service database (MariaDB)** alongside the metric CRUD entities — not in the ClickHouse `presentation` namespace — because CRUD is metadata management, not a contract read. Contract objects and the `presentation` ClickHouse namespace are unchanged by CRUD; only `/run` reaches ClickHouse.

#### Table: `presentation.queries`
#### Table: `saved_queries` (`presentation.queries`)

**ID**: `cpt-presentation-dbtable-queries`

Saved query authored by an analyst: a single `SELECT`/`WITH` over the contract, tenant-scoped.
Saved query authored by an analyst: a single `SELECT`/`WITH` over the contract, tenant-scoped. Created by migration `m20260730_000001_saved_queries` (#1965).

| Column | Type | Description |
|--------|------|-------------|
Expand All @@ -477,7 +481,7 @@ Saved query authored by an analyst: a single `SELECT`/`WITH` over the contract,

**PK**: `id`

**Additional info**: Lives in the `presentation` namespace; written via the `presentation_ro` role's `CREATE`/`INSERT` grant. Reads and runs are tenant-scoped by `insight_tenant_id`.
**Additional info**: Lives in the analytics service DB (MariaDB), written by the service's own SeaORM connection — not the `presentation_ro` ClickHouse role, which is only the identity for `/run` reads. All CRUD reads/writes are tenant-scoped by `insight_tenant_id`.

## 4. Additional context

Expand All @@ -487,7 +491,7 @@ Ordered by quick win; each step ships value or safety on its own, and no step de

1. **Single-SELECT gate** (safety, done, #1962) — `validate_single_select`, applied on write and run via `parse_query_ref`. Shipped with no DB or infra change.
2. **`presentation_ro` role + empty `presentation` DB** (safety, done, #1963/#1964) — role + grant-less `presentation` user in CH bootstrap (`bootstrap-db/provision-presentation-access.sh`); empty DB via `clickhouse.initDatabases` + `apply-ch-migrations.sh`; analytics connects as the `presentation` user (existing `clickhouse_user`/`clickhouse_password` config → `gear.rs` `with_auth`), its password a required credential provisioned before analytics needs it.
3. **Saved-query CRUD** (value, #1965/#1966) — new entity plus migration for `presentation.queries`; handlers and routes per section 3.3; reuse the existing run path for `/run`.
3. **Saved-query CRUD** (value) — CRUD + run shipped (#1965): the `saved_queries` service-DB entity plus migration, handlers and routes per section 3.3, reusing the existing `JSONEachRow` read path for `/run`. Named parameters (#1966) extend the run path next.
4. **Tenant filter** (correctness, #1967) — replace the no-op with the injected predicate in the compiler's shared `WHERE`; `insight_tenant_id` first in `ORDER BY` for new gold; cover with an e2e metric test (`src/ingestion/tests/e2e`). Coordinated with engineering #1829.
5. **Query console** (value, FE, #1970) — thin stable app on the saved-query API: auth shell, author, list, run, render table / auto-chart. Tier-2 "promote to card" can follow.
6. **Preview environments** (infra, FE, #1971-#1973) — path-based `/exp/<name>` on a shared read-only synthetic backend; Redis-`state` return path; one route object per experiment; manual `kubectl`/`helm`.
Expand Down
8 changes: 4 additions & 4 deletions docs/domain/presentation-layer/specs/PRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,9 +185,9 @@ The system **MUST** provide an empty `presentation` ClickHouse database for new

#### Saved-Query CRUD and Run

- [ ] `p1` - **ID**: `cpt-presentation-fr-saved-query-crud`
- [x] `p1` - **ID**: `cpt-presentation-fr-saved-query-crud`

The system **MUST** allow an analyst to create, list, fetch, update, delete, and run saved queries scoped to their tenant. Create and update **MUST** validate the SQL through the single-SELECT gate; run **MUST** execute read-only and return rows. (#1965.)
The system **MUST** allow an analyst to create, list, fetch, update, delete, and run saved queries scoped to their tenant. Create and update **MUST** validate the SQL through the single-SELECT gate; run **MUST** execute read-only and return rows. (Shipped, #1965.)

**Rationale**: A new analytics slice needs no engineering change and no re-ingest.

Expand Down Expand Up @@ -305,13 +305,13 @@ Contract reads for tenant A **MUST NOT** return rows from tenant B, regardless o

#### Saved-Query API

- [ ] `p1` - **ID**: `cpt-presentation-interface-saved-query-api`
- [x] `p1` - **ID**: `cpt-presentation-interface-saved-query-api`

**Type**: REST API (HTTP/JSON)

**Stability**: unstable

**Description**: CRUD and read-only run over saved queries, tenant-scoped. The one new surface Phase A adds. Detailed endpoint contracts live in DESIGN.
**Description**: CRUD and read-only run over saved queries, tenant-scoped. The one new surface Phase A adds. Detailed endpoint contracts live in DESIGN. (CRUD + run shipped in #1965; named parameters follow in #1966.)

**Breaking Change Policy**: Unstable in Phase A; contract hardens in a later phase.

Expand Down
26 changes: 26 additions & 0 deletions src/backend/services/analytics/src/api/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ pub struct ThresholdError;
#[resource_error("gts.cf.insight.analytics_api.person.v1~")]
pub struct PersonError;

/// Resource namespace for `/v1/queries*` (saved-query CRUD + run, #1965).
#[resource_error("gts.cf.insight.analytics_api.saved_query.v1~")]
pub struct SavedQueryError;

/// Resource namespace for the metric-catalog domain (Refs #524, #525).
///
/// Distinct from [`MetricError`] (which scopes the legacy `/v1/metrics` CRUD
Expand Down Expand Up @@ -193,6 +197,28 @@ mod tests {
Ok(())
}

#[test]
fn saved_query_invalid_sql_envelope() -> Result<(), Box<dyn std::error::Error>> {
// POST/PUT /v1/queries with SQL the single-SELECT gate rejects: a 400
// pinning the `sql` field under the saved-query resource namespace.
let err = SavedQueryError::invalid_argument()
.with_resource("q-123")
.with_field_violation("sql", "query must be a single SELECT statement", "INVALID")
.create();
let p = problem(err)?;
assert_eq!(p["status"], 400);
assert_eq!(
p["context"]["resource_type"],
"gts.cf.insight.analytics_api.saved_query.v1~"
);
let violations = p["context"]["field_violations"]
.as_array()
.ok_or("field_violations must be an array")?;
assert_eq!(violations.len(), 1);
assert_eq!(violations[0]["field"], "sql");
Ok(())
}

#[test]
fn person_not_found_envelope() -> Result<(), Box<dyn std::error::Error>> {
let err = PersonError::not_found("person not found")
Expand Down
Loading
Loading