Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce number of crypto database transactions when handling the sync response #7879

Merged
merged 15 commits into from
Jan 6, 2023

Conversation

bmarty
Copy link
Member

@bmarty bmarty commented Jan 3, 2023

Type of change

  • Feature
  • Bugfix
  • Technical
  • Other : Performance

Content

Batch crypto database writing to avoid multiple transactions which takes about 100ms each.
See commit per commit for the changes.

On my main Matrix account (~600 rooms) an initial sync (clear cache, so with non-empty crypto DB) was taking 8 minutes, with this change, it's only 1 minutes and 40 seconds. With a fake crypto store, it's only 30 seconds, but I do not want to dig more since this code will be replaced soon.

This PR also add a way to copy the access token to the clipboard.

Risk

Changing the way data is inserted in the Crypto database may have some side effect. Can you check that @BillCarsonFr ?

Motivation and context

Improve perf of the app and SDK.

Screenshots / GIFs

Tests

  • Perform initial sync.

Tested devices

  • Physical
  • Emulator
  • OS version(s):

Checklist

@ElementBot
Copy link

Warnings
⚠️

Please add a changelog. See instructions here

Generated by 🚫 dangerJS against 02e7157

Copy link
Member

@BillCarsonFr BillCarsonFr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvment, some comments

/**
* Container for the three cross signing keys: master, self signing and user signing.
*/
data class CryptoCrossSigningKeys(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd call that UserIdentity?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

/**
* Store a bunch of data related to the users. @See [UserDataToStore].
*/
fun storeUserDataToStore(userDataToStore: UserDataToStore)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name seems a bit redoudant. storeToStore

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to storeData.

if (cryptoStoreAggregator != null) {
Timber.e("cryptoStoreAggregator is not null...")
}
cryptoStoreAggregator = CryptoStoreAggregator()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I usually don't like when we add cache/buffer directly in the RealmStore.
We used to do it for inbound session and it ended up a bit hard to manage.
I'd prefer to an abstraction on top of the store (BatchedStoreHelper or something); that calls the store that is only doing storing (a bit like the session cache was moved to InboundGroupSessionStore).
But maybe too much?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, reworked.

@@ -176,7 +176,7 @@ internal class DefaultCreateLocalRoomTask @Inject constructor(
}

// Give info to crypto module
cryptoService.onStateEvent(roomId, event)
cryptoService.onStateEvent(roomId, event, null)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CallingcryptoService.onStateEvent here is probably useless, but this PR is not supposed to break anything.

@@ -431,7 +432,8 @@ internal class DefaultCryptoService @Inject constructor(
*
* @param syncResponse the syncResponse
*/
fun onSyncCompleted(syncResponse: SyncResponse) {
fun onSyncCompleted(syncResponse: SyncResponse, cryptoStoreAggregator: CryptoStoreAggregator) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • ⚠️ Documentation of onSyncCompleted is outdated

@sonarcloud
Copy link

sonarcloud bot commented Jan 5, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

Copy link
Member

@BillCarsonFr BillCarsonFr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bmarty bmarty merged commit b7076a1 into develop Jan 6, 2023
@bmarty bmarty deleted the feature/bma/still_investigating branch January 6, 2023 14:16
hughns added a commit to hughns/element-android that referenced this pull request Jan 6, 2023
commit a588989
Merge: 5606730 5ee3eef
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 18:45:18 2023 +0100

    Merge pull request element-hq#7875 from vector-im/feature/bma/releaseScript3

    Release script update

commit 5606730
Merge: baa4663 330a9be
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 18:44:55 2023 +0100

    Merge pull request element-hq#7905 from RiotTranslateBot/weblate-element-android-element-app

    Translations update from Weblate

commit baa4663
Merge: 93021a6 0d2fb8e
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 18:44:37 2023 +0100

    Merge pull request element-hq#7885 from vector-im/feature/bma/fixLint

    Fix lint false positive

commit 93021a6
Merge: f856142 e9d1de8
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 18:43:53 2023 +0100

    Merge pull request element-hq#7724 from vector-im/feature/bma/launchWhenResumed

    Observe ViewEvents only when resumed

commit e9d1de8
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 17:36:40 2023 +0100

    Fix compilation issue after rebase.

commit 330a9be
Merge: f856142 4f2550a
Author: Weblate <[email protected]>
Date:   Fri Jan 6 16:33:47 2023 +0000

    Merge branch 'origin/develop' into Weblate.

commit 4f2550a
Author: Linerly <[email protected]>
Date:   Wed Jan 4 22:47:24 2023 +0000

    Translated using Weblate (Indonesian)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/id/

commit 5734a27
Author: waclaw66 <[email protected]>
Date:   Wed Jan 4 16:04:26 2023 +0000

    Translated using Weblate (Czech)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/cs/

commit 0882e1b
Author: Jeff Huang <[email protected]>
Date:   Thu Jan 5 02:10:13 2023 +0000

    Translated using Weblate (Chinese (Traditional))

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/zh_Hant/

commit 53db988
Author: Christian Paul <[email protected]>
Date:   Thu Jan 5 16:06:48 2023 +0000

    Translated using Weblate (Esperanto)

    Currently translated at 2.2% (2 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/eo/

commit ff9cf8f
Author: Danial Behzadi <[email protected]>
Date:   Wed Jan 4 17:43:34 2023 +0000

    Translated using Weblate (Persian)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/fa/

commit 8a5aad1
Author: Priit Jõerüüt <[email protected]>
Date:   Thu Jan 5 07:22:25 2023 +0000

    Translated using Weblate (Estonian)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/et/

commit 2903a64
Author: Ihor Hordiichuk <[email protected]>
Date:   Wed Jan 4 19:24:53 2023 +0000

    Translated using Weblate (Ukrainian)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/uk/

commit 96363fb
Author: Jozef Gaal <[email protected]>
Date:   Wed Jan 4 16:18:18 2023 +0000

    Translated using Weblate (Slovak)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/sk/

commit 271b828
Author: Szimszon <[email protected]>
Date:   Wed Jan 4 18:55:35 2023 +0000

    Translated using Weblate (Hungarian)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/hu/

commit bd21f03
Author: Glandos <[email protected]>
Date:   Thu Jan 5 08:42:07 2023 +0000

    Translated using Weblate (French)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/fr/

commit 860df01
Author: Vri <[email protected]>
Date:   Wed Jan 4 16:20:15 2023 +0000

    Translated using Weblate (German)

    Currently translated at 100.0% (89 of 89 strings)

    Translation: Element Android/Element Android Store
    Translate-URL: https://translate.element.io/projects/element-android/element-store/de/

commit ed84212
Author: Besnik Bleta <[email protected]>
Date:   Wed Jan 4 17:01:03 2023 +0000

    Translated using Weblate (Albanian)

    Currently translated at 99.3% (2558 of 2576 strings)

    Translation: Element Android/Element Android App
    Translate-URL: https://translate.element.io/projects/element-android/element-app/sq/

commit f790921
Author: Mateus Rodrigues Costa <[email protected]>
Date:   Wed Jan 4 21:12:32 2023 +0000

    Translated using Weblate (Portuguese (Brazil))

    Currently translated at 100.0% (2576 of 2576 strings)

    Translation: Element Android/Element Android App
    Translate-URL: https://translate.element.io/projects/element-android/element-app/pt_BR/

commit 3098ec1
Author: overtinkering <[email protected]>
Date:   Thu Jan 5 18:02:33 2023 +0000

    Translated using Weblate (Spanish)

    Currently translated at 90.7% (2338 of 2576 strings)

    Translation: Element Android/Element Android App
    Translate-URL: https://translate.element.io/projects/element-android/element-app/es/

commit 725722d
Author: Christian Paul <[email protected]>
Date:   Thu Jan 5 16:14:10 2023 +0000

    Translated using Weblate (Esperanto)

    Currently translated at 76.0% (1960 of 2576 strings)

    Translation: Element Android/Element Android App
    Translate-URL: https://translate.element.io/projects/element-android/element-app/eo/

commit ea92464
Author: Christian Paul <[email protected]>
Date:   Thu Jan 5 16:51:43 2023 +0000

    Translated using Weblate (Danish)

    Currently translated at 10.2% (264 of 2576 strings)

    Translation: Element Android/Element Android App
    Translate-URL: https://translate.element.io/projects/element-android/element-app/da/

commit f856142
Merge: b7076a1 85cfa43
Author: Maxime NATUREL <[email protected]>
Date:   Fri Jan 6 16:07:44 2023 +0100

    Merge pull request element-hq#7886 from vector-im/feature/mna/past-polls-ui

    [Poll] Render past polls list of a room (PSG-1029)

commit b7076a1
Merge: 41bcdd7 dbf3b76
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 15:16:16 2023 +0100

    Merge pull request element-hq#7879 from vector-im/feature/bma/still_investigating

    Reduce number of crypto database transactions when handling the sync response

commit 7b1724f
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 15:13:01 2023 +0100

    changelog

commit 9768430
Author: Benoit Marty <[email protected]>
Date:   Mon Dec 19 18:32:07 2022 +0100

    Fix test compilation issue

commit 71bd4f4
Author: Benoit Marty <[email protected]>
Date:   Wed Dec 7 17:48:25 2022 +0100

    Ensure posted events from the ViewModel are consumed (once) by the UI
    Inspired from Kotlin/kotlinx.coroutines#3002

commit 9c79d23
Author: Benoit Marty <[email protected]>
Date:   Fri Dec 16 21:02:33 2022 +0100

    Ensure event are not sent if the lifecycle state is not RESUMED

commit 0dd1abb
Author: Benoit Marty <[email protected]>
Date:   Tue Dec 6 13:02:02 2022 +0100

    Rename method

commit 41bcdd7
Merge: b8da53b 7fc9705
Author: Maxime NATUREL <[email protected]>
Date:   Fri Jan 6 14:18:20 2023 +0100

    Merge pull request element-hq#7867 from vector-im/feature/mna/active-polls-ui

    [Poll] Render active polls list of a room (PSG-908)

commit 85cfa43
Author: Maxime NATUREL <[email protected]>
Date:   Fri Jan 6 14:13:58 2023 +0100

    Using ordinal of enum to render tabs

commit b8da53b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 6 11:56:11 2023 +0000

    Bump checker from 3.27.0 to 3.29.0 (element-hq#7903)

    Bumps [checker](https://github.com/typetools/checker-framework) from 3.27.0 to 3.29.0.
    - [Release notes](https://github.com/typetools/checker-framework/releases)
    - [Changelog](https://github.com/typetools/checker-framework/blob/master/docs/CHANGELOG.md)
    - [Commits](typetools/checker-framework@checker-framework-3.27.0...checker-framework-3.29.0)

    ---
    updated-dependencies:
    - dependency-name: org.checkerframework:checker
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit e88e874
Merge: 2e95d4f c3ad7fa
Author: Benoit Marty <[email protected]>
Date:   Fri Jan 6 10:35:14 2023 +0100

    Merge pull request element-hq#7865 from vector-im/dependabot/gradle/org.owasp-dependency-check-gradle-7.4.3

    Bump dependency-check-gradle from 7.4.1 to 7.4.3

commit 2e95d4f
Merge: f1bd9b2 87e661e
Author: Florian Renaud <[email protected]>
Date:   Fri Jan 6 09:10:00 2023 +0100

    Merge pull request element-hq#7899 from vector-im/bugfix/fre/buffering_on_last_chunk

    [Voice Broadcast] Stop listening if we reach the last received chunk and there is no last sequence number

commit 9b5fda2
Author: Maxime NATUREL <[email protected]>
Date:   Thu Jan 5 15:45:35 2023 +0100

    Fix after rebase

commit a5d076a
Author: Maxime NATUREL <[email protected]>
Date:   Wed Jan 4 10:49:07 2023 +0100

    Adding total votes status for ended poll items

commit 05363dc
Author: Maxime NATUREL <[email protected]>
Date:   Wed Jan 4 10:30:57 2023 +0100

    Adding winner option views for ended poll items

commit 1cc2644
Author: Maxime NATUREL <[email protected]>
Date:   Wed Jan 4 10:12:34 2023 +0100

    Renaming some ui fields

commit 3deae11
Author: Maxime NATUREL <[email protected]>
Date:   Tue Jan 3 17:32:41 2023 +0100

    Adding extra data for ended poll

commit cf82486
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 17:38:55 2022 +0100

    Adding mocked data for ended polls

commit 740591c
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 17:31:23 2022 +0100

    Updating unit tests

commit cb45056
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 17:28:57 2022 +0100

    Mutualizing list fragments and add ended polls tab

commit 0b53591
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 15:50:32 2022 +0100

    Adding changelog entry

commit 7fc9705
Author: Maxime NATUREL <[email protected]>
Date:   Thu Jan 5 16:37:06 2023 +0100

    Adding importantForAccessibility attribute to icon

commit 2dab6ed
Author: Maxime NATUREL <[email protected]>
Date:   Thu Jan 5 15:27:11 2023 +0100

    Fix horizontal margin of tabs

commit ff9e78b
Author: Maxime NATUREL <[email protected]>
Date:   Thu Jan 5 15:20:20 2023 +0100

    Use classical for loop instead of forEach

commit d604035
Author: Maxime NATUREL <[email protected]>
Date:   Thu Jan 5 15:09:41 2023 +0100

    Renaming of filter enum

commit 87e661e
Author: Florian Renaud <[email protected]>
Date:   Thu Jan 5 14:36:22 2023 +0100

    Add changelog file

commit 0d2fb8e
Author: Benoit Marty <[email protected]>
Date:   Wed Jan 4 10:17:35 2023 +0100

    Lint: fix KotlinNullnessAnnotation warning

commit dbf3b76
Author: Benoit Marty <[email protected]>
Date:   Thu Jan 5 11:54:19 2023 +0100

    Update doc.

commit 27d3218
Author: Benoit Marty <[email protected]>
Date:   Thu Jan 5 11:04:20 2023 +0100

    Aggregate data outside of the RealmCryptoStore.

commit 682bb8b
Author: Florian Renaud <[email protected]>
Date:   Wed Jan 4 14:06:58 2023 +0100

    VB - Stop listening if we reach the last received chunk and there is no last sequence number

commit 30940cb
Author: Benoit Marty <[email protected]>
Date:   Thu Jan 5 09:53:12 2023 +0100

    Rename `UserCrossSigningKeys` to `UserIdentity`

commit 7e26c4b
Author: Benoit Marty <[email protected]>
Date:   Thu Jan 5 09:48:25 2023 +0100

    Rename fun

commit 354554e
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 16:45:28 2022 +0100

    Ignore missing ContentDescription

commit e82c7af
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 15:48:14 2022 +0100

    Replace usage of colorAccent

commit 6c0c5e5
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 15:12:12 2022 +0100

    Rename poll item layout to be more generic

commit bd9c53a
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 14:57:37 2022 +0100

    Show message when list is empty

commit e0b7793
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 14:27:11 2022 +0100

    Changing the date format

commit bc985aa
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 14:19:50 2022 +0100

    Adding unit tests for ViewModel

commit 71b7edc
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 12:12:57 2022 +0100

    Adding debug log

commit bf67d25
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 12:08:55 2022 +0100

    Allow access of poll history only in debug variant

commit 8de86e7
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 11:59:48 2022 +0100

    Render mocked data get from use case

commit 77d3b7d
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 11:40:41 2022 +0100

    Fix missing id in Epoxy model

commit f20513e
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 10:56:44 2022 +0100

    Render the active polls list on fragment

commit 7b63f89
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 10:42:42 2022 +0100

    Epoxy controller to render active poll list

commit 9f97579
Author: Maxime NATUREL <[email protected]>
Date:   Fri Dec 30 10:07:50 2022 +0100

    Epoxy model for active poll

commit 10133bd
Author: Maxime NATUREL <[email protected]>
Date:   Thu Dec 29 17:46:07 2022 +0100

    Setup tab layout when landing on the room polls screen

commit 7436c2e
Author: Maxime NATUREL <[email protected]>
Date:   Thu Dec 29 16:41:42 2022 +0100

    Navigate to new empty screen

commit cba960f
Author: Maxime NATUREL <[email protected]>
Date:   Thu Dec 29 16:05:52 2022 +0100

    Adding new entry "Poll history" into room profile screen

commit e903dac
Author: Maxime NATUREL <[email protected]>
Date:   Thu Dec 29 15:40:59 2022 +0100

    Adding changelog entry

commit 437b93c
Author: Benoit Marty <[email protected]>
Date:   Wed Jan 4 11:35:04 2023 +0100

    Add some doc

commit 06f3c11
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 16:43:09 2023 +0100

    Changelog

commit 02e7157
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 16:16:17 2023 +0100

    Introduce CryptoCrossSigningKeys container

commit 4c4ef0d
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 15:57:39 2023 +0100

    Batch insertion of  user data after downloading keys.

commit f26178f
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 15:21:03 2023 +0100

    Avoid useless transaction

commit a386a47
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 15:18:32 2023 +0100

    Crypto store: Log realm transactions and the duration

commit c1a8bf8
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 15:15:15 2023 +0100

    Batch insertion of  `shouldEncryptForInvitedMembers`

commit 6f384c7
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 15:02:45 2023 +0100

    Batch insertion of  `shouldShareHistory`

commit 0e504e9
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 11:55:41 2023 +0100

    Format

commit 8375901
Author: Benoit Marty <[email protected]>
Date:   Tue Jan 3 11:55:32 2023 +0100

    Avoid launching coroutine for nothing.

commit 56986c3
Author: Benoit Marty <[email protected]>
Date:   Mon Jan 2 21:15:08 2023 +0100

    Add a way to get the access token from the advances settings.

commit 5ee3eef
Author: Benoit Marty <[email protected]>
Date:   Mon Jan 2 16:55:25 2023 +0100

    Pull branch sooner to ensure release version is correctly guessed

commit c3ad7fa
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Dec 29 23:02:43 2022 +0000

    Bump dependency-check-gradle from 7.4.1 to 7.4.3

    Bumps dependency-check-gradle from 7.4.1 to 7.4.3.

    ---
    updated-dependencies:
    - dependency-name: org.owasp:dependency-check-gradle
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

commit b3d578d
Author: Benoit Marty <[email protected]>
Date:   Thu Dec 15 12:44:40 2022 +0100

    Release script: Improve creation of the release on GitHub.

commit 5e1d3e6
Author: Benoit Marty <[email protected]>
Date:   Thu Dec 15 12:09:27 2022 +0100

    Escape %
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants