feat(sqlite): merge several replicas of one database without conflicts, through plain EF Core - #665
Merged
Conversation
…s, through plain EF Core Rask.SQLite.Crdt wires the cr-sqlite extension into a DbContext, so several replicas of one SQLite database can be written independently and merged with no server, while application code stays ordinary EF Core: LINQ, change tracking, SaveChanges. Merging is PER COLUMN, not per row. Two devices editing different fields of the same record both keep their work; last-writer-wins applies only where two devices wrote the SAME field, which is the case where something genuinely has to be chosen. That is the whole reason to reach for a CRDT rather than a LastModified column, so it is what the merge tests assert. CrdtChangeFeed exposes the log with no transport attached -- ReadChangesAsync from a watermark, ApplyChangesAsync back. Where those bytes travel is the app's business, and keeping it that way is what lets the same log work over a bucket, a socket, or nothing at all. Reading from a watermark is what makes a sync cost what CHANGED rather than what EXISTS. Applying a change twice is a no-op, which is what makes re-sending safe after an upload whose outcome is unknown, and why a replica never has to track what its peers already hold. The package earns its place on three requirements that otherwise fail QUIETLY: - The extension is per connection, not per process, and Microsoft.Data.Sqlite pools connections -- so loading once at startup works until the pool recycles and then silently stops. It is now loaded on every open and finalized before every close. Pooling=False is required for the same reason: a handle returned to the pool mid-state and handed to somebody else corrupts quietly. - cr-sqlite refuses a NOT NULL column without a default, because a peer on an older schema has to be able to apply a change that never mentions it -- and EF emits exactly that shape for every required property. ApplyCrdtConventions() supplies them as EXPRESSIONS, since EF drops a default equal to the CLR default and a single bool column would otherwise come out bare, failing alone in a way that reads like a cr-sqlite bug rather than an EF one. - Loading the extension seeds bookkeeping tables, and EnsureCreated treats a database that already has tables as provisioned. Creating the schema through a context that loads the extension therefore creates NOTHING AT ALL, and the first symptom is the promotion complaining a table has no primary key. Create the schema without the extension, then promote. A non-SQLite connection is reported rather than skipped: silently not replicating produces an app that looks like it works and surfaces later as data loss. The native binary stays the app's to supply via ExtensionPath -- cr-sqlite ships one per platform, and which is right depends on where the app runs rather than on which package it referenced. Found by the tests rather than by review: the convention skipped every value type. Its guard used GetDefaultValue() is not null, which for a non-nullable value type hands back the BOXED CLR default whether or not one was configured -- so it reported "already has a default" for every int, bool, Guid and DateTime, leaving only string and byte[] working. TryGetDefaultValue answers the question that was actually being asked. Reverting it turns 9 of the 10 per-type assertions red. Merge behaviour is verified against the real extension (RASK_CRSQLITE_PATH) with two replicas exchanging change feeds; everything reachable without the native binary always runs. Item 5 of #642.
… batch atomically Two properties of cr-sqlite's feed that a transport has to build around. Both were found by probing the real extension while designing the bucket transport, not by reading the docs, and both are now pinned by tests rather than left as folklore. A replica's feed carries EVERY change it has ever accepted, not only the ones it made -- still stamped with the originating replica's site_id, which is the only thing that separates them. Publishing the unfiltered feed therefore has every device re-uploading every other device's history, growing with the number of peers rather than with what actually changed. ReadLocalChangesAsync() filters to this replica's own work; ReadChangesAsync() keeps the previous meaning for callers that want everything. A db_version belongs to the database it was READ FROM, not to the replica that originated the change: applying a peer's change stamps it with this replica's next version. So the same change carries a different version in every database holding it, and a version can order your own publishing but can NEVER express "everything peer X has after N". Remembering what has already been fetched from a peer is the transport's job, and building a bucket layout on peer db_versions would have silently skipped changes. Documented on GetDbVersionAsync where it will be read. ApplyChangesAsync now applies the whole batch in one transaction, joining an ambient one when the caller has already opened it. A peer's transaction lands atomically instead of a column at a time, and -- less obviously -- cr-sqlite assigns a fresh local db_version PER TRANSACTION, so applying row by row inflated the receiver's version by the size of every batch it ever took. A 13-column insert cost 13 versions; it now costs one, asserted directly. Parameters are bound once and rebound per change so the statement is prepared once rather than per column.
This was referenced Aug 10, 2026
pal-tamas
added a commit
that referenced
this pull request
Aug 10, 2026
… with no server Rask.SQLite.Crdt.Sync ships Rask.SQLite.Crdt's change feed over Rask.ObjectStore, so several devices sharing one SQLite database converge with nothing between them. Item 6 of #642. Stacked on #665. The design rests on one rule: EACH DEVICE WRITES ONLY UNDER ITS OWN PREFIX (crdt/{site-id}/changes/) and never touches another's. No two devices ever write the same key, so there is nothing to lock, nothing to retry on conflict, and no lease to renew or to leak if a device disappears mid-write. The site-id is cr-sqlite's own, so a device cannot publish under a prefix that disagrees with the changes it is publishing. Everything else follows. - Forward-only reads. Keys carry the publishing replica's own db_version range in fixed-width hex, so they sort in the order the changes were made and a remembered key resumes exactly where the last sync stopped: a sync costs what changed, not what exists. Peers are found with a grouped listing, so discovery costs one response naming the DEVICES rather than one listing every object they have ever written. - Only a replica's own work is published. Its feed also carries every change it has ever ACCEPTED, so publishing unfiltered would have each device re-uploading every other device's history -- growing with the number of peers rather than with what changed. Uploads batch, because object storage charges per request. - A PEER WATERMARK IS A KEY, NOT A VERSION. A db_version is assigned by whichever database reads the change, so the same change carries a different version in every database holding it and "everything peer X has after N" is unanswerable from versions alone. Building the layout on peer versions would not have failed loudly -- it would have silently skipped changes. The watermark advances only after the changes commit locally, so an interrupted pull is retried rather than skipped; skipped changes never come back, because the peer has no reason to publish them again. - Offline is the normal case, not an error, and more strongly here than for a queue-based sync: THE DATABASE IS THE QUEUE. An edit is committed by SaveChanges before any of this runs, so an unreachable bucket loses nothing, there is no "offline mode" to enter, and the next sync publishes the same changes -- safe precisely because applying a change twice does nothing. - No conflict count in the status, deliberately. Merging is per column and automatic, so nothing was silently discarded and there is nothing a user could be asked to resolve; reporting a conflict would be reporting a decision that was never made. - ICrdtSyncStore is a CACHE, NOT A RECORD. Losing all of it costs re-uploading and re-reading, never data, because SQLite already holds the truth -- which is why an in-memory implementation is a legitimate default rather than a test double. A fresh state is answered FROM THE BUCKET rather than assumed to mean "never published", so a reinstalled device does not re-upload its history. The wire format is written by hand against Utf8JsonWriter: no reflection, so it survives trimming and AOT, and each value is TAGGED WITH ITS SQLITE STORAGE CLASS. A change's value is dynamically typed, and one written back as the wrong class is a different value rather than a formatting difference -- it would land in a peer's database as a column that quietly changed type. The envelope carries a format version, so an object written by a newer peer is refused rather than half-applied; an object written today may be read years from now by a device that has been offline since. ICrdtChangeFeed is extracted so a transport can be built and tested without a database or the native extension behind it. Otherwise the bucket layout would only ever be tested where the binary happened to exist, which is not most machines. Tested both ways on purpose. The engine's own tests run against a fake feed that models the two properties the layout depends on -- a change keeps its originating site_id forever, and applying one stamps it with the RECEIVING replica's version -- and a separate suite runs TWO REAL REPLICAS through a bucket, so that model is checked against the extension rather than against itself. The real suite skips without RASK_CRSQLITE_PATH; everything else always runs.
pal-tamas
added a commit
that referenced
this pull request
Aug 10, 2026
…ween them samples/Rask.Example.Crdt runs three replicas -- Phone, Laptop, Tablet -- each with its own SQLite file and its own replica identity, sharing a bucket and nothing else. Normally these would be three phones; here they are three files in one process, which is the only difference that matters. Item 7 of #642. Stacked on #665 and feat/crdt-bucket-sync. The thing the sample exists to show: take two devices offline, edit DIFFERENT FIELDS of the same todo on each -- the priority on one, the done flag on the other -- then bring both back and sync. Both edits survive. Do the same with a LastModified column and one of them is gone. An E2E drives exactly that through a browser, so the claim is tested rather than asserted in prose. Each device's link to the bucket has a switch, and flipping it makes every call fail as a real client would see it. So the demo exercises the real offline path rather than a case the engine knows about, and the status line says "your edits are saved and will sync later" rather than reporting a failure -- the edit is already committed to that device's own database. FolderObjectStore is the other half: the same IObjectStore over a directory, which is what lets the sample run with no cloud credentials. It is not only a test double. It also covers a single-machine deployment with no reason to pay for object storage, and -- the interesting case -- A FOLDER SOMETHING ELSE ALREADY REPLICATES: pointed at a Syncthing share, devices converge with no central server at all; pointed at iCloud Drive or Dropbox, the replication is somebody else's problem. Objects are written beside their key and moved into place, so a reader listing concurrently sees either nothing or the whole object, which matters most when another process is replicating the folder while it is being written. Keys that would escape the root are refused rather than normalised, since a key can come back from a listing of a folder other people also write to. TryCreateAsync maps to the filesystem's own atomic create, so the conditional-create pattern works there too. cr-sqlite's native binary is per-platform and is not redistributed here, so without RASK_CRSQLITE_PATH the page explains what to download instead of failing at the first query -- a missing extension otherwise surfaces as "no such function: crsql_as_crr", which says nothing about what to do. That state is asserted too, so one of the two paths always runs in the E2E gate rather than the whole file quietly skipping on a machine without the binary. Verified by mutation: breaking the setup-card assertion turns it red on a run with no extension, so it is not passing vacuously. The sample joins the standalone-app shape of Rask.Example.Sqlite -- plain Bootstrap classes over the core elements, no Rask.Bootstrap or validation packages -- so it is added to the same in-repo implicit-usings exclusion list, whose generated usings would otherwise not resolve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Several replicas of one SQLite database, written independently and merged without conflicts — through
ordinary EF Core. Item 5 of #642.
Merging is per column, not per row
Two devices editing different fields of the same record both keep their work; last-writer-wins
applies only where two devices wrote the same field, which is the case where something genuinely has
to be chosen. That is the whole reason to reach for a CRDT rather than a
LastModifiedcolumn, so it iswhat the merge tests assert — not merely that "two writes survive".
CrdtChangeFeedexposes the log with no transport attached. Where the bytes travel is the app'sbusiness, and keeping it that way is what lets the same log work over a bucket, a socket, or nothing at
all. Reading from a watermark makes a sync cost what changed rather than what exists; applying a
change twice is a no-op, which is what makes re-sending safe after an upload whose outcome is unknown,
and why a replica never has to track what its peers already hold.
The package earns its place on three requirements that fail quietly
Microsoft.Data.Sqlitepools connections —so loading once at startup works until the pool recycles and then silently stops. It is now loaded on
every open and finalized before every close.
Pooling=Falseis required for the same reason: a handlereturned to the pool mid-state and handed to somebody else corrupts quietly rather than failing.
NOT NULLcolumn without a default — a peer on an older schema has to be ableto apply a change that never mentions it — and EF emits exactly that shape for every required
property.
ApplyCrdtConventions()supplies them as expressions, since EF drops a default equal tothe CLR default: a single
boolcolumn would otherwise come out bare and fail alone, reading like acr-sqlite bug rather than an EF one.
EnsureCreatedtreats a database that alreadyhas tables as provisioned. Creating the schema through a context that loads the extension therefore
creates nothing at all, and the first symptom is the promotion complaining a table has no primary
key. Create the schema without the extension, then promote.
A non-SQLite connection is reported rather than skipped: silently not replicating produces an app
that looks like it works and surfaces later as data loss.
The native binary stays the app's to supply via
ExtensionPath— cr-sqlite ships one per platform, andwhich is right depends on where the app runs rather than on which package it referenced.
Found by the tests, not by review
The convention skipped every value type. Its guard used
GetDefaultValue() is not null, which for anon-nullable value type hands back the boxed CLR default whether or not one was ever configured — so
it reported "already has a default" for every
int,bool,GuidandDateTime, leaving onlystringand
byte[]working.TryGetDefaultValueasks the question that was actually intended.Verified by mutation rather than trusted for being green: reverting it turns 9 of the 10 per-type
assertions red.
Also removed a dead
AddDefaultsForRequiredColumnsoption — nothing could ever read it, since theoptions object lives on the
DbContextOptionsBuilderwhile the convention only sees aModelBuilder.Calling or not calling
ApplyCrdtConventions()is the switch.Testing
Merge behaviour runs against the real extension — two replicas exchanging change feeds — gated on
RASK_CRSQLITE_PATHand skipped when the platform binary is absent, matching the CLI's build-gateshape. Everything reachable without it (conventions per CLR type, options, table resolution, the
connection lifecycle) always runs.
Local gates:
dotnet formatclean,dotnet build -warnaserrorclean, 320 unit tests, 60 browserjourneys, 26 CLI build-gate tests.
Not in this PR
The browser half. cr-sqlite now links and runs inside a .NET
browser-wasmpublish, but the builddepends on a hand-pinned Rust nightly plus a post-build rewrite of the archive, which is its own piece
of packaging work rather than something to hide in a csproj.