Synapse migration support

[TheSunCat]

Hey, thanks for starting this fork! Exciting to see this project continue.
I'm opening this issue to hopefully track implementation of migration support from Synapse, as I'd love to try out tuwunel once I know I can move my users to it.

[erebion]

Hey,

just wanted to add that even if migrating is not complete, it might already be helpful. I certainly don't care about losing messages, I just want to migrate to something that uses less resources.

For my use-case what I'd need to migrate, roughly:

• accounts
• room memberships
• room addresses

Even though there's currently no path for a full migration, could I somehow already just migrate that?

[criadoperez]

Hi! I started to work on migrating Synapse to Tuwnunel.
The work so far is here.

However its important to mention that current code uses an API + federation approach. Meaning it first exports the data of Synapse using the API, then imports it in Tuwunel and lastly uses federation. This approach makes it fast and safe, however it will not migrate everything. This file explains the limitations of this approach.

A more complete (but more complex and risky) approach is a direct database migration, which I will start to look into. If anybody else is also looking into this, feel free to reach out.

[RGBCube]

Thanks for the excellent work! A question I had in mind was, would it be possible to use Syanpse's database schema inside Tuwunel without modification? Or would it be a bad idea as the Synapse schema isn't stable?

[jevolk]

@criadoperez This is exactly what's needed to jumpstart this endeavor. Thank you so much 🫶. I'm going to have a look at this as soon as I return to work next week. One of the things I'm going to be aiming for is how we can get e2ee migrated seamlessly. I think it might just be possible to perform the migration of devices and access tokens in-situ from the Synapse server to Tuwunel. It would be disruptive enough if users had to reset their identities and lose their message histories that it's worth investigating this further.

@RGBCube Unfortunately no, though it's always been an option to plug into Synapse's pgsql using a rust-pgsql library to exfiltrate the data. Using the pgsql in production is way too complicated and suffers from a multitude of issues where rocksdb and sql simply don't work well together under the same abstraction.

[criadoperez]

I'm glad to see others are also interested in this. :-)

As a full DB migration is a completely different approach I've created a separate new repository to test this approach.

The general approach (simplified explanation) I have in mind is:

1. Dump the Synapse DB, keys and media files to a backup folder
2. Create in an isolated network a new tuwunel server that uses the same domain name and keys (this is important).
3. Create a "migration script that will query data from the SQL DB, adapt it and write it in the RocksDB format.

That last point is the real hard and tricky part, so I plan to implement it by stages. So for example, first only attempt to migrate a minimum viable room with no E2EE, media, invites, etc.

Then I can test in the isolated environment if that worked, and if it does, gradually attempt to add more and more complex information. That way we can have a table of what gets migrated and what doesn't.

We will also need a configuration script that can read a Synapse homeserver.yaml file and creates a minimum and equivalent tuwunel.toml file (plus copy in RocksDB some data that is in Synapse stores in the homeserver.yaml file but tuwunel in the DB)

[criadoperez]

Small update on this. I have stopped developing the "new repository" approach as it turned up to be more time consuming that I can dedicate to it. The original repository that does a more limited migration is there in case it helps anybody.