Skip to content

feat(sqlite): share a CRDT database between devices through a bucket, with no server - #666

Merged
pal-tamas merged 1 commit into
mainfrom
feat/crdt-bucket-sync
Aug 10, 2026
Merged

feat(sqlite): share a CRDT database between devices through a bucket, with no server#666
pal-tamas merged 1 commit into
mainfrom
feat/crdt-bucket-sync

Conversation

@pal-tamas

Copy link
Copy Markdown
Owner

Ships Rask.SQLite.Crdt's change feed over a bucket, so
several devices sharing one SQLite database converge with nothing between them. Item 6 of #642.

Stacked on #665. Retarget to main once that merges.

var engine = new CrdtSyncEngine(objectStore, new CrdtChangeFeed(context));
engine.Changed += status => Render(status);
await engine.SyncAsync();

One rule, everything else follows

Each device writes only under its own prefixcrdt/{site-id}/changes/ — and never touches
another's. No two devices ever write the same key, so there is nothing to lock, nothing to retry on
conflict, and no lease to renew or to leak if a device disappears mid-write. The site-id is
cr-sqlite's own, so a device cannot publish under a prefix that disagrees with the changes it is
publishing.

  • Forward-only reads. Keys carry the publisher's own db_version range in fixed-width hex, so they
    sort in the order changes were made and a remembered key resumes exactly where the last sync stopped.
  • Peers cost one listing. A grouped listing names the devices rather than every object they have
    written — which would undo the point of the watermark.
  • Only your own work is published. A replica's feed also carries every change it ever accepted, so
    publishing unfiltered would have each device re-uploading every other device's history. Uploads batch,
    because object storage charges per request.

The bug this design nearly had

I started to build peer watermarks on db_version. That would have been silent data loss.

A db_version is assigned by whichever database reads the change. Alice's single 13-column insert is
dbv=1 for all 13 in her database, and arrives in Bob's as dbv=1..13. So "everything peer X has after
N" is unanswerable from versions, and a layout built on them skips changes without ever throwing. The
watermark has to be the object key. Pinned by a test, and documented on GetDbVersionAsync where
someone will actually read it.

The watermark also advances only after changes commit locally — skipped changes never come back,
because the peer has no reason to publish them again.

Offline, and what is actually durable

The database is the queue. An edit is committed by SaveChanges before any of this runs, so an
unreachable bucket loses nothing, there is no "offline mode" to enter, and the next sync republishes the
same changes — safe precisely because applying twice does nothing. CrdtSyncPhase.Offline is
deliberately not a failure state.

That makes ICrdtSyncStore a cache, not a record: losing all of it costs re-uploading and
re-reading, never data. A materially better bargain than Rask.Sync.Client, where losing the queue
loses a user's offline edits — and why InMemoryCrdtSyncStore is a legitimate default rather than a test
double. A fresh state is answered from the bucket, so a reinstalled device does not republish its
history.

There is no conflict count in the status, on purpose. Merging is per column and automatic, so nothing
was silently discarded and there is nothing a user could be asked to resolve.

The wire format

Hand-written against Utf8JsonWriter: no reflection, so it survives trimming and AOT, and each value
is tagged with its SQLite storage class
. A change's value is dynamically typed, and one written back as
the wrong class is a different value, not a formatting difference — it lands on the receiving device as
a column that quietly changed type. The envelope carries a format version, so an object from a newer peer
is refused rather than half-applied; an object written today may be read years later by a device that has
been offline since.

Testing, both ways on purpose

The engine's tests run against a fake feed that models the two properties the layout depends on — a
change keeps its originating site_id forever, and applying one stamps it with the receiving replica's
version. A separate suite runs two real replicas through a bucket, so that model is checked against
the extension rather than against itself: concurrent per-column edits, every storage class, catch-up
after being offline.

dotnet test tests/Rask.SQLite.Crdt.Sync.Tests                          # 33 passed
RASK_CRSQLITE_PATH=… dotnet test tests/Rask.SQLite.Crdt.Sync.Tests     # 37 passed

ICrdtChangeFeed is extracted so a transport can be built and tested without a database or the native
extension behind it — otherwise the bucket layout would only ever be tested where the binary happened to
exist.

Local gates: dotnet format clean, dotnet build -warnaserror clean, 336 + 320 unit, 60 browser
journeys, 26 CLI build-gate.

… with no server

Rask.SQLite.Crdt.Sync ships Rask.SQLite.Crdt's change feed over Rask.ObjectStore, so several devices
sharing one SQLite database converge with nothing between them. Item 6 of #642. Stacked on #665.

The design rests on one rule: EACH DEVICE WRITES ONLY UNDER ITS OWN PREFIX
(crdt/{site-id}/changes/) and never touches another's. No two devices ever write the same key, so
there is nothing to lock, nothing to retry on conflict, and no lease to renew or to leak if a device
disappears mid-write. The site-id is cr-sqlite's own, so a device cannot publish under a prefix that
disagrees with the changes it is publishing. Everything else follows.

- Forward-only reads. Keys carry the publishing replica's own db_version range in fixed-width hex, so
  they sort in the order the changes were made and a remembered key resumes exactly where the last
  sync stopped: a sync costs what changed, not what exists. Peers are found with a grouped listing,
  so discovery costs one response naming the DEVICES rather than one listing every object they have
  ever written.
- Only a replica's own work is published. Its feed also carries every change it has ever ACCEPTED,
  so publishing unfiltered would have each device re-uploading every other device's history --
  growing with the number of peers rather than with what changed. Uploads batch, because object
  storage charges per request.
- A PEER WATERMARK IS A KEY, NOT A VERSION. A db_version is assigned by whichever database reads the
  change, so the same change carries a different version in every database holding it and
  "everything peer X has after N" is unanswerable from versions alone. Building the layout on peer
  versions would not have failed loudly -- it would have silently skipped changes. The watermark
  advances only after the changes commit locally, so an interrupted pull is retried rather than
  skipped; skipped changes never come back, because the peer has no reason to publish them again.
- Offline is the normal case, not an error, and more strongly here than for a queue-based sync: THE
  DATABASE IS THE QUEUE. An edit is committed by SaveChanges before any of this runs, so an
  unreachable bucket loses nothing, there is no "offline mode" to enter, and the next sync publishes
  the same changes -- safe precisely because applying a change twice does nothing.
- No conflict count in the status, deliberately. Merging is per column and automatic, so nothing was
  silently discarded and there is nothing a user could be asked to resolve; reporting a conflict
  would be reporting a decision that was never made.
- ICrdtSyncStore is a CACHE, NOT A RECORD. Losing all of it costs re-uploading and re-reading, never
  data, because SQLite already holds the truth -- which is why an in-memory implementation is a
  legitimate default rather than a test double. A fresh state is answered FROM THE BUCKET rather than
  assumed to mean "never published", so a reinstalled device does not re-upload its history.

The wire format is written by hand against Utf8JsonWriter: no reflection, so it survives trimming and
AOT, and each value is TAGGED WITH ITS SQLITE STORAGE CLASS. A change's value is dynamically typed,
and one written back as the wrong class is a different value rather than a formatting difference --
it would land in a peer's database as a column that quietly changed type. The envelope carries a
format version, so an object written by a newer peer is refused rather than half-applied; an object
written today may be read years from now by a device that has been offline since.

ICrdtChangeFeed is extracted so a transport can be built and tested without a database or the native
extension behind it. Otherwise the bucket layout would only ever be tested where the binary happened
to exist, which is not most machines.

Tested both ways on purpose. The engine's own tests run against a fake feed that models the two
properties the layout depends on -- a change keeps its originating site_id forever, and applying one
stamps it with the RECEIVING replica's version -- and a separate suite runs TWO REAL REPLICAS through
a bucket, so that model is checked against the extension rather than against itself. The real suite
skips without RASK_CRSQLITE_PATH; everything else always runs.
@pal-tamas
pal-tamas force-pushed the feat/crdt-bucket-sync branch from a42e929 to bff261c Compare August 10, 2026 09:09
@pal-tamas
pal-tamas merged commit fef2202 into main Aug 10, 2026
9 checks passed
@pal-tamas
pal-tamas deleted the feat/crdt-bucket-sync branch August 10, 2026 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant