Delete unreferenced state groups during purge #4006

erikjohnston · 2018-10-04T14:29:10Z

No description provided.

This is needed to efficiently check for unreferenced state groups during purge.

richvdh · 2018-10-09T16:52:09Z

synapse/storage/state.py

+    def _is_state_group_referenced(self, txn, state_group):
+        """Checks if a given state group is referenced, or is safe to delete.
+
+        A state groups is referenced if it or any of its descendants are


richvdh · 2018-10-09T16:58:29Z

synapse/storage/state.py

+
+        A state groups is referenced if it or any of its descendants are
+        pointed at by an event. (A descendant is a group which has the given
+        state_group as a prev group)


each state group only has one prev group, so this doesn't really make sense. "A descendant is a state_group whose chain of prev_groups includes the given state_group." ?

richvdh · 2018-10-09T17:02:55Z

synapse/storage/state.py

+        while state_groups_to_search:
+            state_group = state_groups_to_search.pop()  # Next state group to check
+
+            is_referenced = self._simple_select_one_onecol_txn(


the problem here is that we might be about to delete a bunch of the events :/. I think you need a WHERE event_id NOT IN events_to_delete.

richvdh · 2018-10-09T17:05:51Z

synapse/storage/state.py

+        state_groups_searched = set()
+
+        while state_groups_to_search:
+            state_group = state_groups_to_search.pop()  # Next state group to check


can we not do this for more than one state group at a time?

richvdh · 2018-10-09T17:07:11Z

synapse/storage/state.py

@@ -1082,6 +1133,12 @@ def __init__(self, db_conn, hs):
            columns=["state_key"],
            where_clause="type='m.room.member'",
        )
+        self.register_background_index_update(
+            self.EVENT_STATE_GROUP_INDEX_UPDATE_NAME,
+            index_name="event_to_state_groups_sg_index",


worth remembering that we already have this on matrix.org, so would ideally not spend ages building another copy of the index.

richvdh · 2018-10-12T15:32:11Z

synapse/storage/events.py

+                    # updating them
+                    continue
+
+                if not self._is_state_group_referenced(txn, sg):


if you're going to reference methods in StateGroupWorkerStore, can you make EventsStore inherit from it?

Oh, yes. I just assumed we already were (since we use functions from there elsewhere I think).

…rge_state_groups

erikjohnston · 2018-10-19T15:22:16Z

I've shuffled things about a bit to checking of unreferenced state groups in batches, rather than one at a time, as well as simplifying the initial query.

richvdh

looks generally sane to me. a few comments/questions

richvdh · 2018-10-23T15:28:41Z

synapse/storage/events.py

@@ -204,7 +205,8 @@ def f(self, *args, **kwargs):

 # inherits from EventFederationStore so that we can call _update_backward_extremities
 # and _handle_mult_prev_events (though arguably those could both be moved in here)
-class EventsStore(EventFederationStore, EventsWorkerStore, BackgroundUpdateStore):
+class EventsStore(StateGroupWorkerStore, EventFederationStore, EventsWorkerStore,


a comment on why we are inheriting from StateGroupWorkerStore mightn't hurt; otoh perhaps it's obvious enough.

I think its obvious enough, especially since we use it in a fair few places and not just in purge history

richvdh · 2018-10-23T15:59:44Z

synapse/storage/state.py

@@ -1041,6 +1041,86 @@ def _count_state_group_hops_txn(self, txn, state_group):

            return count

+    def _find_unreferenced_groups(self, txn, state_groups):


given this is now inherently tied to purging via the join on events_to_purge, I think it might make more sense to leave it alongside the other purge code.

richvdh · 2018-10-23T16:06:29Z

synapse/storage/state.py

+                next_to_search = set()
+            else:
+                lst = list(next_to_search)
+                current_search = set(lst[:100])


I'd be inclined to do:

current_search = set(islice(next_to_search, 100)) next_to_search -= current_search

richvdh · 2018-10-23T16:08:41Z

synapse/storage/state.py

+            """ % (",".join("?" for _ in current_search),)
+            txn.execute(sql, list(current_search))
+
+            referenced = set(sg for sg, cnt in txn if cnt > 0)


surely cnt will never be zero?

(so why not do SELECT DISTINCT state_group rather than the GROUP BY)

richvdh · 2018-10-23T16:12:44Z

synapse/storage/state.py

+
+            next_to_search.update(row["state_group"] for row in rows)
+            # We don't bother re-handling groups we've already seen
+            next_to_search -= state_groups_seen


should we not do this before adding to next_to_search?

prevs = set(row["state_group"] for row in rows) prevs -= state_groups_seen next_to_search |= prevs state_groups_seen |= prevs

…rge_state_groups

Addressed comments

richvdh

lgtm

rubo77 · 2018-10-30T18:16:25Z

Is this also deleting leftovers of the old Nuke Script?

Features -------- - Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled. ([\#4004](#4004), [\#4133](#4133), [\#4142](#4142), [\#4184](#4184)) - Support for replacing rooms with new ones ([\#4091](#4091), [\#4099](#4099), [\#4100](#4100), [\#4101](#4101)) Bugfixes -------- - Fix exceptions when using the email mailer on Python 3. ([\#4095](#4095)) - Fix e2e key backup with more than 9 backup versions ([\#4113](#4113)) - Searches that request profile info now no longer fail with a 500. ([\#4122](#4122)) - fix return code of empty key backups ([\#4123](#4123)) - If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events. ([\#4127](#4127)) - Fix table lock of device_lists_remote_cache which could freeze the application ([\#4132](#4132)) - Fix exception when using state res v2 algorithm ([\#4135](#4135)) - Generating the user consent URI no longer fails on Python 3. ([\#4140](#4140), [\#4163](#4163)) - Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if the remote server returns a Content-Type header with the chartype in quotes. ([\#4157](#4157)) - The hash_password script now works on Python 3. ([\#4161](#4161)) - Fix noop checks when updating device keys, reducing spurious device list update notifications. ([\#4164](#4164)) Deprecations and Removals ------------------------- - The disused and un-specced identicon generator has been removed. ([\#4106](#4106)) - The obsolete and non-functional /pull federation endpoint has been removed. ([\#4118](#4118)) - The deprecated v1 key exchange endpoints have been removed. ([\#4119](#4119)) - Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2. ([\#4120](#4120)) Internal Changes ---------------- - Fix build of Docker image with docker-compose ([\#3778](#3778)) - Delete unreferenced state groups during history purge ([\#4006](#4006)) - The "Received rdata" log messages on workers is now logged at DEBUG, not INFO. ([\#4108](#4108)) - Reduce replication traffic for device lists ([\#4109](#4109)) - Fix `synapse_replication_tcp_protocol_*_commands` metric label to be full command name, rather than just the first character ([\#4110](#4110)) - Log some bits about room creation ([\#4121](#4121)) - Fix `tox` failure on old systems ([\#4124](#4124)) - Add STATE_V2_TEST room version ([\#4128](#4128)) - Clean up event accesses and tests ([\#4137](#4137)) - The default logging config will now set an explicit log file encoding of UTF-8. ([\#4138](#4138)) - Add helpers functions for getting prev and auth events of an event ([\#4139](#4139)) - Add some tests for the HTTP pusher. ([\#4149](#4149)) - add purge_history.sh and purge_remote_media.sh scripts to contrib/ ([\#4155](#4155)) - HTTP tests have been refactored to contain less boilerplate. ([\#4156](#4156)) - Drop incoming events from federation for unknown rooms ([\#4165](#4165))

erikjohnston force-pushed the erikj/purge_state_groups branch 3 times, most recently from 56a33b0 to b07c0f6 Compare October 4, 2018 15:02

erikjohnston added 3 commits October 4, 2018 16:03

Delete unreferened state groups during purge

17d5857

Add state_group index to event_to_state_groups

4917ff5

This is needed to efficiently check for unreferenced state groups during purge.

Newsfile

d9f3db5

erikjohnston force-pushed the erikj/purge_state_groups branch from b07c0f6 to d9f3db5 Compare October 4, 2018 15:03

erikjohnston requested a review from a team October 4, 2018 15:11

richvdh previously requested changes Oct 9, 2018

View reviewed changes

richvdh assigned erikjohnston Oct 9, 2018

Fix up comments

67a1e31

richvdh reviewed Oct 12, 2018

View reviewed changes

erikjohnston added 3 commits October 19, 2018 15:48

Batch process handling state groups

47a9da2

Merge branch 'develop' of github.com:matrix-org/synapse into erikj/pu…

056f099

…rge_state_groups

pep8

67f7b9c

erikjohnston requested a review from a team October 19, 2018 15:21

erikjohnston removed their assignment Oct 22, 2018

richvdh reviewed Oct 23, 2018

View reviewed changes

richvdh assigned erikjohnston Oct 23, 2018

erikjohnston added 5 commits October 26, 2018 16:22

Merge branch 'develop' of github.com:matrix-org/synapse into erikj/pu…

03e634d

…rge_state_groups

Make SQL a bit cleaner

b2399f6

Don't make temporary list

f4f223a

Fix set operations thinko

664b192

Move _find_unreferenced_groups

ad88460

erikjohnston requested a review from richvdh October 29, 2018 14:26

richvdh approved these changes Oct 30, 2018

View reviewed changes

erikjohnston merged commit 0794504 into develop Oct 30, 2018

erikjohnston deleted the erikj/purge_state_groups branch December 12, 2018 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delete unreferenced state groups during purge #4006

Delete unreferenced state groups during purge #4006

erikjohnston commented Oct 4, 2018

richvdh Oct 9, 2018

richvdh Oct 9, 2018

richvdh Oct 9, 2018

richvdh Oct 9, 2018

richvdh Oct 9, 2018

richvdh Oct 12, 2018

erikjohnston Oct 12, 2018

erikjohnston commented Oct 19, 2018

richvdh left a comment

richvdh Oct 23, 2018

erikjohnston Oct 29, 2018

richvdh Oct 23, 2018

richvdh Oct 23, 2018

richvdh Oct 23, 2018

richvdh Oct 23, 2018

richvdh Oct 23, 2018

richvdh left a comment

rubo77 commented Oct 30, 2018

		@@ -1041,6 +1041,86 @@ def _count_state_group_hops_txn(self, txn, state_group):

		return count

		def _find_unreferenced_groups(self, txn, state_groups):

Delete unreferenced state groups during purge #4006

Delete unreferenced state groups during purge #4006

Conversation

erikjohnston commented Oct 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Oct 19, 2018

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

rubo77 commented Oct 30, 2018