docs: clearer docs on data types, constraints and limitations. (#396)

Firstly this adds explicit pages in usage/data-modelling on:

- data type support
- constraints

Then it goes around and tidies up the various copy around limitations.

README.md

@@ -22,7 +22,7 @@
   <a href="https://github.com/electric-sql/electric/stargazers/"><img src="https://img.shields.io/github/stars/electric-sql/electric?style=social&label=Star&maxAge=2592000" /></a>
   <a href="https://github.com/electric-sql/electric/actions"><img src="https://github.com/electric-sql/electric/workflows/CI/badge.svg" alt="CI"></a>
   <a href="https://github.com/electric-sql/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache_2.0-green" alt="License - Apache 2.0"></a>
-  <a href="https://electric-sql.com/docs/reference/limitations"><img src="https://img.shields.io/badge/status-alpha-orange" alt="Status - Alpha"></a>
+  <a href="https://electric-sql.com/docs/reference/roadmap"><img src="https://img.shields.io/badge/status-alpha-orange" alt="Status - Alpha"></a>
   <a href="https://discord.electric-sql.com"><img src="https://img.shields.io/discord/933657521581858818?color=5969EA&label=discord" alt="Chat - Discord"></a>
   <a href="https://twitter.com/ElectricSQL" target="_blank"><img src="https://img.shields.io/twitter/follow/nestframework.svg?style=social&label=Follow @ElectricSQL"></a>
   <a href="https://fosstodon.org/@electric" target="_blank"><img src="https://img.shields.io/mastodon/follow/109599644322136925.svg?domain=https%3A%2F%2Ffosstodon.org"></a>

clients/typescript/src/migrators/bundle.ts

@@ -12,7 +12,7 @@ import { data as baseMigration } from './schema'
 import Log from 'loglevel'
 import { SatelliteError, SatelliteErrorCode } from '../util'
 
-export const SCHEMA_VSN_ERROR_MSG = `Local schema doesn't match server's. Clear local state through developer tools and retry connection manually. If error persists, re-generate the client. Check documentation (https://electric-sql.com/docs/reference/limitations) to learn more.`
+export const SCHEMA_VSN_ERROR_MSG = `Local schema doesn't match server's. Clear local state through developer tools and retry connection manually. If error persists, re-generate the client. Check documentation (https://electric-sql.com/docs/reference/roadmap) to learn more.`
 
 const DEFAULTS: MigratorOptions = {
   tableName: '_electric_migrations',

docs/api/ddlx.md

@@ -5,16 +5,16 @@ description: >-
 sidebar_position: 10
 ---
 
-:::caution Limitations
-The syntax and features described in this page are **not yet implemented**.
-
-Currently, DDLX rules are limited to [electrification by SQL procedure call syntax](../reference/limitations#ddlx-rules). See the [Limitations](../reference/limitations.md) page for more context.
-:::
-
 ElectricSQL extends the PostgreSQL language with the following DDLX statements.
 
 Use these in your [migrations](../usage/data-modelling/migrations.md) to [electrify tables](../usage/data-modelling/electrification.md) and expose data by granting [permissions](../usage/data-modelling/permissions.md) to roles and assigning roles to [authenticated](../usage/auth/index.md) users.
 
+:::caution Work in progress
+The syntax and features described in this page are not fully implemented.
+
+Currently, DDLX rules are limited to [electrification by SQL procedure call syntax](../reference/roadmap#ddlx-rules). See [Roadmap -> DDLX rules](../reference/roadmap.md#ddlx-rules) for more context.
+:::
+
 ## Electrification
 
 Tables must be [electrified](../usage/data-modelling/electrification.md) to include them in the replication machinery.

docs/examples/linear-lite.md

@@ -3,7 +3,7 @@ title: LinearLite
 description: Local-first project management app, based on a simplified Linear clone.
 ---
 
-:::caution Limitations
+:::caution Work in progress
 We haven't published this example yet. Check back soon for an online demo.
 :::
 

docs/integrations/backend/index.md

@@ -5,16 +5,10 @@ sidebar_position: 30
 
 import DocCardList from '@theme/DocCardList'
 
-:::caution Limitations
-See <DocPageLink path="usage/data-modelling" /> for current limitations on data modelling and migrations.
-:::
-
 ElectricSQL is designed to work with [any Postgres-backed system](../../usage/installation/postgres.md). You don't need a specific ([or, in fact, any](./other.md)) backend framework to use Electric.
 
 However, it's common to manage the Postgres data model using a migrations system. These are often provided by your web framework. Because ElectricSQL's [DDLX Rules](../../api/ddlx.md) are applied using DDL migrations, this section shows you how to do this using some popular frameworks.
 
-:::note
 Some frameworks also provide or work with Postgres change data capture tooling, such as logical replication consumers. This can be useful for [Event sourcing](../event-sourcing/index.md).
-:::
 
 <DocCardList />

docs/integrations/frontend/index.md

@@ -9,7 +9,7 @@ ElectricSQL is designed to integrate with existing reactivity and component fram
 
 This allows you to bind [live queries](../../usage/data-access/queries.md) to component state and hook into data and connectivity change events.
 
-:::caution Limitations
+:::caution Work in progress
 ElectricSQL currently only has official support for React (both ReactJS and React Native).
 
 If you're interested in helping to develop Solid, Svelte, Vue or any other frontend framework integrations [let us know on Discord](https://discord.electric-sql/com).

docs/integrations/frontend/solid.md

@@ -5,8 +5,8 @@ description: >-
 sidebar_position: 20
 ---
 
-:::caution Limitations
-SolidJS support is not yet implemented. See <DocPageLink path="reference/limitations" /> for context.
+:::caution Work in progress
+SolidJS support is not yet implemented.
 
 [Let us know on Discord](https://discord.electric-sql/com) if you're interested in helping to develop SolidJS support. 
 :::

docs/integrations/frontend/svelte.md

@@ -5,8 +5,8 @@ description: >-
 sidebar_position: 30
 ---
 
-:::caution Limitations
-Svelte support is not yet implemented. See <DocPageLink path="reference/limitations" /> for context.
+:::caution Work in progress
+Svelte support is not yet implemented.
 
 [Let us know on Discord](https://discord.electric-sql/com) if you're interested in helping to develop Svelte support. 
 :::

docs/integrations/frontend/vue.md

@@ -5,8 +5,8 @@ description: >-
 sidebar_position: 40
 ---
 
-:::caution Limitations
-Vue.js support is not yet implemented. See <DocPageLink path="reference/limitations" /> for context.
+:::caution Work in progress
+Vue.js support is not yet implemented.
 
 [Let us know on Discord](https://discord.electric-sql/com) if you're interested in helping to develop Vue.js support. 
 :::

docs/reference/limitations.md → docs/reference/roadmap.md

@@ -1,31 +1,40 @@
 ---
-title: Limitations
+title: Roadmap
 description: >-
   Summary of the maturity stage of the project and known limitations.
 ---
 
 ElectricSQL is in <strong className="warning-color">public alpha</strong> phase.
 
-:::caution
 APIs are not guaranteed to be stable. Backwards incompatible changes may (and will) be introduced in both minor and major version releases.
 
-Key aspects of the system *as currently documented* are not fully implemented yet, as outlined on this page.
-:::
-
 ## Practical limitations
 
-Key aspects of the system are not yet implemented, including:
+Key aspects of the system are not fully implemented yet:
 
-1. [DDLX rules](#ddlx-rules) &mdash; SQL syntax, permissions and validation are not supported
-2. [Shapes](#shapes) &mdash; currently limited to whole table sync
-3. [Data modelling](../usage/data-modelling/index.md) &mdash; limited data types and constraint support
-4. [Failure modes](#failure-modes) &mdash; you may need to workaround in development
+1. [Data modelling](#data-modelling) &mdash; limited data types and constraints
+2. [DDLX rules](#ddlx-rules) &mdash; limited to electrification with proceedure call syntax
+3. [Shapes](#shapes) &mdash; currently limited to whole table sync
 
-### DDLX rules
+Plus you may encounter [failure modes](#failure-modes) that you need to work around in development
+
+### Data modelling
+
+There are a number of [fundamental limitations](#fundamental-limitations) of the local-first model you should be aware of. These have a practical impact on data model support, for example:
 
-The SQL syntax you see documented on <DocPageLink path="api/ddlx" /> is not yet implemented. Neither are permissions, roles, validation or local SQLite commands.
+- **primary keys**: sequential IDs are not supported; you [must use binary UUIDs](../usage/data-modelling/types.md#primary-keys)
+
+In addition, there are a number of limitations of the current implementation that impact data model support:
+
+- **data types**: see the list of [supported data types](../usage/data-modelling/types.md#supported-data-types)
+- **constraints**: you must remove [unique and check constraints](../usage/data-modelling/constraints.md#unsupported)
+- **migrations**: you must use [additive, forward migrations](../usage/data-modelling/migrations.md#limitations)
+
+See <DocPageLink path="usage/data-modelling" /> for more information.
+
+### DDLX rules
 
-DDLX support is currently limited to electrifying tables using an SQL procedure call syntax:
+The SQL syntax you see documented on <DocPageLink path="api/ddlx" /> is not yet implemented. Neither are permissions, roles, validation or local SQLite commands. DDLX support is currently limited to electrifying tables using an SQL procedure call syntax:
 
 ```sql
 CALL electric.electrify('items')
@@ -35,42 +44,21 @@ You can track progress on DDLX support here [electric-sql/pg_proxy/pulls](https:
 
 ### Shapes
 
-[Shape-based sync](../usage/data-access/shapes.md) using the [`sync()` function API](../api/clients/typescript.md#sync) currently only supports whole table sync.
-
-There is no support for `where` clauses to filter the initial target rows or `select` clauses to filter the include tree. As a result, current calls to `db.tablename.sync({...})` will "over sync" additional data onto the device.
+[Shape-based sync](../usage/data-access/shapes.md) using the [`sync()` function](../api/clients/typescript.md#sync) currently supports whole table sync. There is no support for `where` clauses to filter the initial target rows or `select` clauses to filter the include tree. As a result, current calls to `db.tablename.sync({...})` will "over sync" additional data onto the device.
 
 :::note
 There is one temporary feature to filter data onto the local device: set an `electric_user_id` field on your table. If you do, then rows will only be synced if the value of that column matches the value of the authenticated user_id provided in your [auth token](../usage/auth/index.md).
 
 This is a very temporary workaround and will be removed soon!
 :::
 
-### Data modelling
-
-We don't currently support any constraints except primary keys. You must remove all unique and check constraints from your Postgres data model.
-
-We have limited data type support. Currently, we only support strings (`text` and non-length-limited `varchar`) and numbers (`smallint`, `integer`, `bigint`, `double precision`). The authoritative list of supported data types is maintained in the [`supported_pg_types/0` function][1].
-
-We also only support forward and additive migrations. Migrations that remove columns or make them more restrictive will fail when applied to electrified tables.
-
-[1]: https://github.com/search?q=repo%3Aelectric-sql%2Felectric+symbol%3Asupported_pg_types&type=code
-
 ### Failure modes
 
-Currently, you are likely to experience bugs or behaviour that leads to an inconsistent data state. This is **not** related to the core [consistency model](./consistency.md). It's a consequence of the maturity of the implementation.
-
-If you experience errors like:
-
-- `Migrations cannot be altered once applied`
-- `Cannot catch up to the server's current state` 
-- `FOREIGN KEY CONSTRAINT failed`
-
-You may need to reset your databases. This involves:
+Currently, you may experience bugs or behaviour that leads to an inconsistent data state. This is **not** related to the core [consistency model](./consistency.md). It's a consequence of the lack of validation and some recovery modes still pending implementation.
 
-- wiping your client databases by clearing IndexedDB in the browser
-- resetting your Potsgres database
+In development, you can usually recover from these bugs by resetting your database(s). In the browser, if you clear localStorage and IndexedDB (for example in Chrome, "Inspect" to open the developer tools -> Application -> Storage -> Clear site data) that will reset the client and your local app will re-sync from the server.
 
-To reset your Postgres database, you may need to drop the replication publication. This requires:
+If you need to re-set your Postgres database, if you're using Docker Compose (such as with the starter template or examples) you can usually use something like `yarn backend:down` or `docker compose -f backend/compose.yaml down --volumes`. Alternatively, if you can't just nuke your whole database folder, you may need to drop the replication publication manually:
 
 ```sql
 ALTER SUBSCRIPTION postgres_1 DISABLE;
@@ -82,7 +70,7 @@ You can then recreate your database, e.g.:
 
 ```shell
 dropdb -f intro
-createdb -T template0 -E UTF-8 intro
+createdb -T template0 -E UTF-8 electric
 ```
 
 Then:
@@ -93,7 +81,7 @@ Then:
 
 Keep an eye on [electric-sql/electric/pulls](https://github.com/electric-sql/electric) for the latest bugfixes.
 
-## Integrity limitations
+## Fundamental limitations
 
 ElectricSQL uses a [rich-CRDT data model](./consistency.md#rich-crdts) that allows building local-first applications without falling into common pitfalls of working with eventually consistent databases. However, you still need to follow certain constraints that can’t be verified or enforced automatically by ElectricSQL. The purpose of this section is to document these constraints and why they are required.
 

docs/top-level-listings/usage.md

@@ -12,7 +12,7 @@ This guide explains how to use ElectricSQL. It's broken down into four main sect
 ## See also
 
 See the [Integrations](./integrations.md) guide and the [API docs](./api.md).
-You might also be interested in some of the [Reference](./reference.md) docs, such as the [Limitations](../reference/limitations.md) and [Architecture](../reference/architecture.md) pages.
+You might also be interested in some of the [Reference](./reference.md) docs, such as the [Architecture](../reference/architecture.md) and [Roadmap](../reference/roadmap.md) pages.
 
 <div className="grid grid-cols-1 sm:grid-cols-2 gap-6">
   <div className="tile">

docs/usage/auth/index.md

@@ -40,12 +40,10 @@ Insecure mode is designed for development or testing. It supports unsigned JWTs
 
 If you have a backend for your app with a cookie-based login session mechanism already in place, we recommend creating an HTTP endpoint to generate one-off tokens for signed-in users and using that to obtain a fresh token before initializing the client.
 
-## Limitations
+<hr className="doc-divider" />
 
-We don't currently support unauthenticated use.
+:::caution Work in progress
+We don't currently support unauthenticated use, or changing the authentication state on an active replication connection.
 
-We don't currently support changing the authentication state on an active replication connection. To work around this, you can either defer instantiating your client until the user authenticates, or you can [stop][1] and [start][2] the [satellite process][3] manually yourself.
-
-[1]: https://github.com/electric-sql/electric/blob/2e8bfdf4992d355d0b1928a097fe406d283303bf/clients/typescript/src/satellite/process.ts#L293
-[2]: https://github.com/electric-sql/electric/blob/2e8bfdf4992d355d0b1928a097fe406d283303bf/clients/typescript/src/satellite/process.ts#L167-L170
-[3]: https://github.com/electric-sql/electric/blob/2e8bfdf4992d355d0b1928a097fe406d283303bf/clients/typescript/src/satellite/process.ts
+[Get in touch](https://discord.electric-sql.com) if this is a blocker or you need help with a workaround.
+:::

docs/usage/data-access/shapes.md

@@ -5,16 +5,16 @@ description: >-
 sidebar_position: 30
 ---
 
-:::caution Limitations
-Shapes are currently limited to whole-table sync. See the [limitations page](../../reference/limitations.md) for context.
-:::
-
 Shapes are the core primitive for controlling sync in the ElectricSQL system.
 
 Local apps establish shape subscriptions. This syncs data from the cloud onto the local device using the [Satellite replication protocol](../../api/satellite.md), into the local embedded SQLite database. Once the initial data has synced, [queries](./queries.md) can run against it.
 
 The [Electric sync service](../installation/service.md) maintains shape subscriptions and streams any new data and data changes onto the local device. In this way, local devices can sync a sub-set of a larger database for interactive offline use.
 
+:::caution Work in progress
+Shapes are currently limited to whole-table sync. See [Roadmap -> Shapes](../../reference/roadmap.md#shapes) for more information.
+:::
+
 ## What is a shape?
 
 A shape is a set of related data that's synced onto the local device. It is defined by:

docs/usage/data-modelling/constraints.md

@@ -0,0 +1,87 @@
+---
+title: Constraints
+description: >-
+  Constraints and invariants supported by the ElectricSQL system.
+sidebar_position: 50
+---
+
+Invariant support is currently limited to referential integrity and not-null constraints. Unique and check constraints are not yet supported.
+
+## Supported
+
+### Referential integrity
+
+ElectricSQL maintains referential integrity of foreign key references. So you can use foreign key relationships in your data model and rely on referential integrity:
+
+```sql
+CREATE TABLE posts (
+  id UUID PRIMARY KEY
+);
+
+CREATE TABLE comments (
+  id UUID PRIMARY KEY
+
+  post_id UUID REFERENCES(posts.id) ON DELETE CASCADE
+);
+```
+
+This works even when making writes locally in an offline database. See [Introduction -> Conflict-free offline -> Preserving data integrity](../../intro/offline.md#preserving-data-integrity) and the Rich-CRDT post on [Compensations](/blog/2022/05/03/introducing-rich-crdts#compensations) for more information.
+
+### Not-null constraints
+
+ElectricSQL supports [not-null constraints](https://www.postgresql.org/docs/current/ddl-constraints.html#id-1.5.4.6.6) as long as the constraint is defined when creating the column.
+
+I.e.: the not-null constraint must be defined in an [additive migration](./migrations.md#limitations). So the following is supported because creating the table with new columns is *additive*:
+
+```sql
+CREATE TABLE items (
+  -- implicit non null constraint
+  id UUID PRIMARY KEY
+
+  -- explicit non null constraint
+  foo TEXT NOT NULL
+
+  -- can be null
+  bar TEXT
+)
+```
+
+Adding a column with a not-null constraint is supported because it's *additive*:
+
+```sql
+ALTER TABLE items
+  ADD COLUMN baz TEXT NOT NULL;
+```
+
+Constraining an existing column by adding a not-null constraint to it is **not supported**:
+
+```sql
+-- Not supported
+ALTER TABLE items
+  ALTER COLUMN bar TEXT NOT NULL;
+```
+
+This is not supported because it may invalidate concurrent, in-flight operations. Specifically, writes that were accepted locally with null values would need to be rejected, which would violate the [finality of local writes](../../reference/architecture.md#local-writes).
+
+## Unsupported
+
+Unsupported constraints must be removed from a table before [electrifying](./electrification.md) it.
+
+### Check constraints
+
+ElectricSQL does not currently support [check constraints](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-CHECK-CONSTRAINTS).
+
+### Unique constraints
+
+ElectricSQL does not currently support [unique constraints](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-UNIQUE-CONSTRAINTS).
+
+<hr className="doc-divider" />
+
+:::caution Work in progress
+We're working to support:
+
+- unique constraints [using Reservations](/blog/2022/05/03/introducing-rich-crdts#reservations)
+- *single column* and then *multi-column* check constraints using validation
+
+See [Rich-CRDTs](/blog/2022/05/03/introducing-rich-crdts) for more information.
+:::