Add partitioned tables support #2197

magnetised · 2024-12-19T18:16:40Z

Allows for subscribing to partitions of a partitioned table (which you could already do) but also subscribing to the root, partitioned, table and receiving all updates across all partitions.

Fixes #2118

Major changes:

Add support for partitioned table types in the various inspector queries
Include partition hierarchy information in relation information from pg and in inspector relation info
Force reload of table information when receiving a new relation that is a partition
Add partition information to Filter so that it can correctly pass on a change on a partition table to a shape on the partition root.
For a shape subscribed to the partition root, changes coming in on partitions will be re-written so that their relation matches the partition root. The change keys are not being re-written - I don't think this is an issue as the keys are guaranteed to be unique and that's enough.
Relation messages notifying of a new partition table are forwarded onto root-partition consumers so the shape can be updated with knowledge of the new partition (this does not cause the shape to be deleted)
I've added a full-stack test to verify this stuff so that we're not making assumptions about pg's handling of partitions

Minor:

support querying the inspector based on the {schema, name} relation tuple, as well as the table name string because I needed to be able to look up table info based
Filter instances now have an inspector instance so they can do table info lookups to get partition information
The filter functions used by the dispatcher now return {filter, shape_ids} so that the filter's state can be updated
I've tweaked StubInspector so we can return richer info from the load_relation call
Added missing serialisation of a shape's replica setting

netlify · 2024-12-20T13:10:24Z

✅ Deploy Preview for electric-next ready!

Name	Link
🔨 Latest commit	`c59bdc2`
🔍 Latest deploy log	https://app.netlify.com/sites/electric-next/deploys/678634215b54b70008f94254
😎 Deploy Preview	https://deploy-preview-2197--electric-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

magnetised · 2025-01-13T11:03:06Z

Think i have a more up-to-date version of this on my laptop which I seem to have forgotten to push or have over-written with my old work-machine version. @robacourt please hold off review until I can recover the situation

magnetised · 2025-01-13T11:22:27Z

@robacourt situation recovered. review away. great start to the year!

balegas · 2025-01-13T12:45:07Z

Nice work! We should document the two possible ways to subscribe to partitioned tables.

magnetised · 2025-01-14T09:54:13Z

Nice work! We should document the two possible ways to subscribe to partitioned tables.

@balegas I've added basic docs highlighting the behaviour of shapes on partitioned tables - pls review

robacourt

Nice work! I've left a few comments.

I've also been thinking there might be an alternative implementation, whereby the replication steam is processed (by a PartitionTransformer for want of a better name) before it gets to the filter, and if it sees a change for a partition it inserts a change into the stream for the logical table. That way, all the partition logic can go in the PartitionTransformer. Shape, Filter, Consumer etc. would remain unchanged. Sure it will create some changes that might not be used, but that seems a good tradeoff for the simplicity. Is there a reason why this is not possible? Or undesirable?

robacourt · 2025-01-13T13:02:31Z

packages/sync-service/lib/electric/postgres/inspector.ex

              {:ok, relation_info()} | {:error, String.t()}

  @callback load_column_info(relation(), opts :: term()) ::
              {:ok, [column_info()]} | :table_not_found

+  # @callback introspect_relation()


I'm guessing this should be removed?

Suggested change

# @callback introspect_relation()

robacourt · 2025-01-13T15:08:15Z

packages/sync-service/test/electric/shapes/filter_test.exs

+      assert_affected(filter, insert, MapSet.new(["s1", "s2"]))
+    end
+
+    @tag :wip


Presumably this can be removed?

Suggested change

@tag :wip

robacourt · 2025-01-14T09:19:36Z

packages/sync-service/lib/electric/replication/shape_log_collector.ex

+      {:error, _} ->
+        # just ignore errors here, they're unlikely anyway
+        :ok


Surely the ShapeLogCollector should crash if there's an error? An error could be something as simple as loss of network connectivity to the database, which if we just swallow the error we won't clean the relation leading to stale cache and an inconsistent state, or have I missed something?

robacourt · 2025-01-14T09:21:58Z

packages/sync-service/lib/electric/replication/shape_log_collector.ex

+      {:ok, _} ->
+        # probably a malformed value from a test inspector
+        :ok


When would this happen?

balegas · 2025-01-14T14:29:41Z

website/docs/guides/shapes.md

@@ -56,6 +56,36 @@ This is the root table of the shape. All shapes must specify a table and it must

 The value can be just a tablename like `projects`, or can be a qualified tablename prefixed by the database schema using a `.` delimiter, such as `foo.projects`. If you don't provide a schema prefix, then the table is assumed to be in the `public.` schema.

+#### Partitioned Tables


magnetised force-pushed the 2118-support-shapes-on-partitioned-tables branch from 5e6a8f1 to 8aafed5 Compare December 19, 2024 18:17

magnetised requested review from icehaunter and robacourt December 19, 2024 18:18

magnetised force-pushed the 2118-support-shapes-on-partitioned-tables branch from 65582f7 to b9e1bd0 Compare January 13, 2025 10:39

Add partitioned tables support

b517c38

magnetised force-pushed the 2118-support-shapes-on-partitioned-tables branch from b9e1bd0 to b517c38 Compare January 13, 2025 11:09

increase stack ready timeout

0946d6d

add docs describing partitioned table support

c59bdc2

robacourt requested changes Jan 14, 2025

View reviewed changes

balegas reviewed Jan 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add partitioned tables support #2197

Add partitioned tables support #2197

magnetised commented Dec 19, 2024 •

edited

Loading

netlify bot commented Dec 20, 2024 •

edited

Loading

magnetised commented Jan 13, 2025 •

edited

Loading

magnetised commented Jan 13, 2025

balegas commented Jan 13, 2025

magnetised commented Jan 14, 2025

robacourt left a comment

robacourt Jan 13, 2025

robacourt Jan 13, 2025

robacourt Jan 14, 2025

robacourt Jan 14, 2025

balegas Jan 14, 2025

		@@ -56,6 +56,36 @@ This is the root table of the shape. All shapes must specify a table and it must

		The value can be just a tablename like `projects`, or can be a qualified tablename prefixed by the database schema using a `.` delimiter, such as `foo.projects`. If you don't provide a schema prefix, then the table is assumed to be in the `public.` schema.

		#### Partitioned Tables

Add partitioned tables support #2197

Are you sure you want to change the base?

Add partitioned tables support #2197

Conversation

magnetised commented Dec 19, 2024 • edited Loading

netlify bot commented Dec 20, 2024 • edited Loading

✅ Deploy Preview for electric-next ready!

magnetised commented Jan 13, 2025 • edited Loading

magnetised commented Jan 13, 2025

balegas commented Jan 13, 2025

magnetised commented Jan 14, 2025

robacourt left a comment

Choose a reason for hiding this comment

robacourt Jan 13, 2025

Choose a reason for hiding this comment

robacourt Jan 13, 2025

Choose a reason for hiding this comment

robacourt Jan 14, 2025

Choose a reason for hiding this comment

robacourt Jan 14, 2025

Choose a reason for hiding this comment

balegas Jan 14, 2025

Choose a reason for hiding this comment

magnetised commented Dec 19, 2024 •

edited

Loading

netlify bot commented Dec 20, 2024 •

edited

Loading

magnetised commented Jan 13, 2025 •

edited

Loading