Remove dependency on libolm #8901

bmarty · 2024-09-11T10:34:26Z

Type of change

Feature
Bugfix
Technical
Other :

Content

Remove the dependency on the deprecated libolm library

Existing data is removed, which means that there is a crypto database migration to cleanup the tables
Encryption for content scanner now rely on the Rust implementation (need bindings: Expose the PkEncryption stuff in the crypto crate bindings matrix-org/matrix-rust-sdk#3971 and so a new crypto SDK release once the Rust PR is merged.)

Motivation and context

Remove the dependency on the deprecated libolm library. See more details here:

Screenshots / GIFs

Tests

Check migration from a previous installation to a new one

Tested devices

Physical
Emulator
OS version(s):

Checklist

Changes has been tested on an Android device or Android emulator with API 21
UI change has been tested on both light and dark themes
Accessibility has been taken into account. See https://github.com/element-hq/element-android/blob/develop/CONTRIBUTING.md#accessibility
Pull request is based on the develop branch
Pull request includes a new file under ./changelog.d. See https://github.com/element-hq/element-android/blob/develop/CONTRIBUTING.md#changelog
Pull request includes screenshots or videos if containing UI changes
Pull request includes a sign off
You've made a self review of your PR
If you have modified the screen flow, or added new screens to the application, you have updated the test UiAllScreensSanityTest.allScreensTest()

https://github.com/uhoreg/matrix-doc/blob/symmetric_ssss/proposals/2472-symmetric-ssss.md "The m.secret_storage.v1.curve25519-aes-sha2 method proposed in MSC1946 is removed."

… SDK.

…e it will not crash the application.

BillCarsonFr · 2024-09-13T08:24:41Z

...oid/src/main/java/org/matrix/android/sdk/internal/session/identity/IdentityBulkLookupTask.kt

+
+    @OptIn(ExperimentalStdlibApi::class)
+    private fun String.toSha256(): String {
+        return sha256.digest(toByteArray()).toHexString()


Found it surprising that we use toHexString here. The legacy sha was returning a base64 string no? https://gitlab.matrix.org/matrix-org/olm/-/blob/master/tests/test_olm_sha256.cpp#L16

Good catch thanks! f726d16

Thx, maybe just a nit, it could be named convertToSha256Base64Encoded

bmarty · 2024-09-17T08:09:03Z

I have tested again the database migration and this is working fine:

Checkout the current develop
Deploy the app
Login to an account
Verify the session, observe that existing messages can be decrypted
Send some messages in e2e rooms
Checkout the last commit of the branch
Use locally build crypto SDK (release still need to be done)
Deploy the application to the same device
Check in the log that the migration has been done
Observe that the application is not crashing and is still working as expected

Bumps [org.matrix.rustcomponents:crypto-android](https://github.com/matrix-org/matrix-rust-components-kotlin) from 0.4.1 to 0.4.3. - [Release notes](https://github.com/matrix-org/matrix-rust-components-kotlin/releases) - [Commits](matrix-org/matrix-rust-components-kotlin@crypto-v0.4.1...crypto-v0.4.3) --- updated-dependencies: - dependency-name: org.matrix.rustcomponents:crypto-android dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

ElementBot · 2024-09-20T08:31:18Z

	Warnings
⚠️	matrix-sdk-android/build.gradle#L164 - A newer version of net.java.dev.jna:jna than 5.13.0 is available: 5.14.0
⚠️	matrix-sdk-android/build.gradle#L164 - A newer version of net.java.dev.jna:jna than 5.13.0 is available: 5.14.0

Generated by 🚫 dangerJS against e89bec4

bmarty · 2024-09-20T11:07:57Z

...oid/src/main/java/org/matrix/android/sdk/internal/session/identity/IdentityBulkLookupTask.kt

+                    sha256Converter.convertToSha256(
+                            str = threePid.value.lowercase(Locale.ROOT) + " " + threePid.toMedium() + " " + pepper
+                    )
+            )


I confirm that this code works using vector.im identity server.

bmarty · 2024-09-23T15:03:29Z

I have tested the communication to the content scanner running a local synapse and a local instance of https://github.com/element-hq/matrix-content-scanner-python and the content scanner is able to decrypt the attachment.

Patch to apply (for information / future me):

contentScannerTest.patch

Result testing data from a file sent to a room (eicar.com.bin):

Content scanner logs:

2024-09-23 16:43:45,834 - matrix_content_scanner.scanner.file_downloader - 323 - INFO - POST-9 - Sending GET request to http://localhost:8080/_matrix/media/v3/download/localhost/KFBpUrxgQyLQrGbxJBYEVPXe
2024-09-23 16:43:45,847 - matrix_content_scanner.scanner.file_downloader - 181 - INFO - POST-9 - Remote server responded with 200
2024-09-23 16:43:45,848 - matrix_content_scanner.scanner.scanner - 417 - INFO - POST-9 - Decrypting encrypted file
2024-09-23 16:43:45,905 - matrix_content_scanner.scanner.scanner - 514 - ERROR - POST-9 - MIME type for file is forbidden: text/plain
2024-09-23 16:43:45,906 - matrix_content_scanner.scanner.scanner - 260 - INFO - POST-9 - Caching scan failure
2024-09-23 16:43:45,907 - aiohttp.access - 211 - INFO - POST-9 - 127.0.0.1 [23/Sep/2024:15:43:45 +0100] "POST /_matrix/media_proxy/unstable/scan_encrypted HTTP/1.1" 200 369 "-" "Element - dbg/1.6.22-dev (Google sdk_gphone64_arm64; Android 14; UE1A.230829.036.A4; Flavour GooglePlay; MatrixAndroidSdk2 1.6.22)"

BillCarsonFr

Approved

bmarty · 2024-09-24T06:46:42Z

Note: checking application upgrade from 1.5.32 to 1.6.22 and we got a migration error then the application crashes. Here are the log:

io.realm.exceptions.RealmMigrationNeededException: Migration is required due to the following errors:
- Property 'CryptoRoomEntity.rotationPeriodMs' has been added.
- Property 'CryptoRoomEntity.rotationPeriodMsgs' has been added.

Upgrading from 1.6.0 (which is 1-year old+) to 1.6.22 is working well.

bmarty added 6 commits September 11, 2024 11:23

Remove dependency on libolm - WIP

5908cd5

Better cleanup of the Crypto database.

15c86f3

Remove old unused algorithm m.secret_storage.v1.curve25519-aes-sha2

637a227

https://github.com/uhoreg/matrix-doc/blob/symmetric_ssss/proposals/2472-symmetric-ssss.md "The m.secret_storage.v1.curve25519-aes-sha2 method proposed in MSC1946 is removed."

Remove todo and cleanup

fc68f1c

Encrypt data for the content scanner using PkEncryption from the Rust…

355621b

… SDK.

New SDK: fix API break.

8056285

bmarty requested a review from BillCarsonFr September 11, 2024 10:34

bmarty added 3 commits September 12, 2024 21:05

ScanEncryptorUtils.getDownloadBodyAndEncryptIfNeeded can throw, ensur…

14a52d4

…e it will not crash the application.

Use local val and more compact code.

9dcb7c8

Fix quality issues.

51c20b4

BillCarsonFr reviewed Sep 13, 2024

View reviewed changes

Fix wrong sha256 conversion and add unit test.

f726d16

bmarty added the Z-NextRelease For issues and PRs which should be included in the NextRelease. label Sep 18, 2024

bmarty mentioned this pull request Sep 20, 2024

Bump org.matrix.rustcomponents:crypto-android from 0.4.1 to 0.4.3 #8912

Closed

Fix copyright issue.

f7195a8

bmarty commented Sep 20, 2024

View reviewed changes

use pkEncryption, to properly close it.

e89bec4

bmarty marked this pull request as ready for review September 23, 2024 15:03

BillCarsonFr approved these changes Sep 23, 2024

View reviewed changes

bmarty merged commit ac01523 into develop Sep 23, 2024
11 of 12 checks passed

bmarty deleted the feature/bma/removeLibolm branch September 23, 2024 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove dependency on libolm #8901

Remove dependency on libolm #8901

bmarty commented Sep 11, 2024 •

edited

Loading

BillCarsonFr Sep 13, 2024 •

edited

Loading

bmarty Sep 16, 2024

BillCarsonFr Sep 23, 2024

bmarty commented Sep 17, 2024

ElementBot commented Sep 20, 2024 •

edited

Loading

bmarty Sep 20, 2024

bmarty commented Sep 23, 2024

BillCarsonFr left a comment

bmarty commented Sep 24, 2024

Remove dependency on libolm #8901

Remove dependency on libolm #8901

Conversation

bmarty commented Sep 11, 2024 • edited Loading

Type of change

Content

Motivation and context

Screenshots / GIFs

Tests

Tested devices

Checklist

BillCarsonFr Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

bmarty Sep 16, 2024

Choose a reason for hiding this comment

BillCarsonFr Sep 23, 2024

Choose a reason for hiding this comment

bmarty commented Sep 17, 2024

ElementBot commented Sep 20, 2024 • edited Loading

bmarty Sep 20, 2024

Choose a reason for hiding this comment

bmarty commented Sep 23, 2024

BillCarsonFr left a comment

Choose a reason for hiding this comment

bmarty commented Sep 24, 2024

bmarty commented Sep 11, 2024 •

edited

Loading

BillCarsonFr Sep 13, 2024 •

edited

Loading

ElementBot commented Sep 20, 2024 •

edited

Loading