Client very slow to restore session after Rust crypto v8 → v10 migration

[robintown]

Steps to reproduce

1. Be on develop.element.io, refresh to update your client
2. Notice that it's taking a really long time to load
3. Open the console and find that a Rust crypto migration is in progress

Outcome

What did you expect?

The migration should finish and I should be able to use my client again.

What happened instead?

After migrating all 80000-something sessions, it outputs a message to say it's now migrating the schema. At this point my CPU goes idle and no further interesting logs appear, so the client appears to be stuck on the loading screen. I refresh the page and the client now fails to start with an error in the console about failing to connect to IndexedDB.

Operating system

NixOS unstable

Browser information

Firefox 121.0.1

URL for webapp

develop.element.io

Application version

7c584df-react-a3a387a1dd8f-js-d178fbf9cdc3

Homeserver

Synapse 1.99.0

Will you send logs?

Yes

[andybalaam]

I get the same behaviour, and so does Valère. Investigating.

[andybalaam]

The scheme upgrade in question only deletes the old object store, nothing else.

A stackoverflow page suggests that you need to clear() an object store before deleting it, but we have not done this in the past and it worked fine.

[BillCarsonFr]

I waited instead of refreshing, and it finally completed:

https://rageshakes.element.io/api/listing/2024-02-01/102737-F7T44FAH/logs-0000.log.gz

2024-02-01T08:17:05.339Z I INFO matrix_sdk_indexeddb::crypto_store::migrations::v8_to_v10: IndexeddbCryptoStore upgrade schema -> v10 starting

and after 2 hours??

2024-02-01T10:23:27.105Z I INFO matrix_sdk_indexeddb::crypto_store::migrations::v8_to_v10: IndexeddbCryptoStore upgrade schema -> v10 complete

[BillCarsonFr]

Also it is now re-uploading all my keys to backup?
But maybe it's because my 4S is desynced

2024-02-01T10:23:27.763Z D Backup: Starting keys upload loop for backup version:83.
2024-02-01T10:23:27.953Z D FetchHttpApi: <-- GET https://matrix-client.matrix.org/_matrix/client/v3/room_keys/version [191ms 200]
2024-02-01T10:23:27.954Z D [PerSessionKeyBackupDownloader] Got current backup version from server: 83
2024-02-01T10:23:27.955Z D [PerSessionKeyBackupDownloader] Version for which we have a decryption key (82) doesn't match the version we are backing up to (83)

[andybalaam]

Since it eventually worked for @BillCarsonFr I changed the description of this PR to say it was very slow rather than failed.

I have a client that I refreshed part-way, and it is now being very slow to start, so we will see whether it leaves people's clients in a broken state.

@robintown were you able to use Element after refreshing?

[andybalaam]

I have a client that I refreshed part-way, and it is now being very slow to start, so we will see whether it leaves people's clients in a broken state.

Yes, it worked eventually but it took a long time (over an hour?)

[andybalaam]

I'm going to benchmark creating a lot of records and dropping the store, to see whether this is a general problem, or caused by a particular bug in our code.

[robintown]

@andybalaam were you able to use Element after refreshing?

No, that client is still unable to restore the session

[andybalaam]

No, that client is still unable to restore the session

@robintown please leave it running. After refresh, mine printed nothing for a long time, and then eventually it worked. (Took about 1-2 hours)

[andybalaam]

I now strongly suspect this is Firefox-only. I am able to reproduce slowness to delete a large store with an index in Firefox, but it's fine in Chrome.

[andybalaam]

This is a problem in Firefox, which I have documented here: https://www.artificialworlds.net/blog/2024/02/02/deleting-an-indexed-db-store-can-be-incredibly-slow-on-firefox/

I will log a bug against Firefox if one doesn't already exist.

[andybalaam]

This PR: matrix-org/matrix-rust-sdk#3090 should speed things up by about 3 times on Firefox, but I think that is all we can do.