fix(crypto): Redecrypt non-UTD messages to remove no-longer-relevant warning shields #4644

andybalaam · 2025-02-07T14:16:21Z

Fixes element-hq/element-meta#2697
Fixes https://github.com/element-hq/crypto-internal/issues/398

I'm sorry it's a big change. I've tried to break it into decent commits, and I did a couple of preparatory PRs to make it less painful, but it's still a bit to get your head around.

The basic idea is that when a session is updated and we call retry_event_decryption, we don't only look at UTDs any more - now we also look at decrypted events, and re-request their EncryptionInfo, in case it has improved.

codecov · 2025-02-07T14:32:38Z

Codecov Report

Attention: Patch coverage is 85.71429% with 11 lines in your changes missing coverage. Please review.

Project coverage is 86.30%. Comparing base (6c9b1ef) to head (94b7776).

Files with missing lines	Patch %	Lines
...es/matrix-sdk-common/src/deserialized_responses.rs	0.00%	6 Missing ⚠️
...i/src/timeline/controller/decryption_retry_task.rs	94.82%	3 Missing ⚠️
...trix-sdk-ui/src/timeline/event_item/content/mod.rs	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4644      +/-   ##
==========================================
+ Coverage   86.26%   86.30%   +0.03%     
==========================================
  Files         290      290              
  Lines       34286    34346      +60     
==========================================
+ Hits        29578    29641      +63     
+ Misses       4708     4705       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

andybalaam · 2025-02-14T12:16:15Z

I swapped the Rust team reviewer to @bnjbvr since I know he has questions about including session ID in the timeline item.

richvdh

seems generally good to me. A few comments.

crates/matrix-sdk-ui/src/timeline/traits.rs

crates/matrix-sdk-ui/src/timeline/controller/state_transaction.rs

crates/matrix-sdk-ui/src/timeline/event_handler.rs

crates/matrix-sdk-ui/src/timeline/event_item/mod.rs

crates/matrix-sdk-ui/src/timeline/event_item/remote.rs

crates/matrix-sdk-ui/src/timeline/controller/mod.rs

richvdh · 2025-02-14T14:25:42Z

crates/matrix-sdk-ui/src/timeline/controller/state.rs

@@ -225,6 +228,45 @@ impl TimelineState {
        txn.commit();
    }

+    /// Try to fetch [`EncryptionInfo`] for the events with the supplied
+    /// indices, and update them where we succeed.
+    pub(super) async fn retry_event_encryption_info<P: RoomDataProvider>(


I have to say that having this here but retry_event_decryption_by_index in the other place feels rather asymmetric. Is there a reason they need to be different?

In my mind, TimelineState::retry_event_encryption_info is a partner to TimelineState::retry_event_decryption, not to TimelineController::retry_event_decryption_by_index. retry_event_decryption_by_index is needed because we need to construct a complex retry_one closure to pass to retry_event_decryption whereas retry_event_encryption_info can do its work directly.

bnjbvr · 2025-02-17T13:08:16Z

crates/matrix-sdk-ui/src/timeline/controller/mod.rs

+        if let Some(event) = item.as_event() {
+            if let Some(remote_event) = event.as_remote() {
+                if let Some(session_id) = &remote_event.session_id {
+                    if should_retry(session_id) {


uh, that's a lot of indent levels 😅

You could use .and_then on all the Option to chain them, as well as early continue when you're seeing something you're not expecting:

let Some(thing) = item.as_event() .and_then(|event| event.as_remote()) .map(/* or and_then, not sure what session_id is */|remote_event| *remote_event.session_id) else { continue; };

After the changes I've already made, this is a little flatter, and I don't think there is a nice way to make it better?

On similar lines, I had a look at TimelineState::retry_event_encryption_info which has a similar problem, but I couldn't see a way to make that better because it has an async call all the way over to the right. Suggestions welcome.

Restructured in the latest version.

andybalaam · 2025-02-17T15:40:14Z

This is ready for re-review. I think I would recommend looking at the full diff now, since the review changes reverted some of what I did in earlier commits. I will rebase into nice commits after review.

bnjbvr

A few stylistic change requests below. On the one hand, thanks for not storing any extra data in the TimelineItemContent itself, on the other… you know my opinion about adding more code to the timeline re-decryption. This all should be moved somewhere else, and tested in isolation there (could be in the crypto crate, the main SDK crate, inside or outside the event cache,… any place other than the timeline would make more sense).

crates/matrix-sdk-ui/src/timeline/controller/state_transaction.rs

bnjbvr · 2025-02-19T09:49:55Z

crates/matrix-sdk-ui/src/timeline/controller/mod.rs

+            if let Some(event) = item.as_event() {
+                if let TimelineItemContent::UnableToDecrypt(encrypted) = event.content() {
+                    if let Some(session_id) = encrypted.session_id() {
+                        if should_retry(session_id) {
+                            return Some(Either::Left(idx));
+                        }
+                    }
+                } else if let Some(remote_event) = event.as_remote() {
+                    if let Some(encryption_info) = &remote_event.encryption_info {
+                        if should_retry(&encryption_info.session_id) {
+                            return Some(Either::Right(idx));
+                        }
+                    }
+                }
+            }
+            None


style nit: this could be much less indented if you shortcut the Nones:

Suggested change

if let Some(event) = item.as_event() {

if let TimelineItemContent::UnableToDecrypt(encrypted) = event.content() {

if let Some(session_id) = encrypted.session_id() {

if should_retry(session_id) {

return Some(Either::Left(idx));

}

}

} else if let Some(remote_event) = event.as_remote() {

if let Some(encryption_info) = &remote_event.encryption_info {

if should_retry(&encryption_info.session_id) {

return Some(Either::Right(idx));

}

}

}

}

None

let event = item.as_event()?;

if let TimelineItemContent::UnableToDecrypt(encrypted) = event.content() {

let session_id = encrypted.session_id()?;

if should_retry(session_id) {

return Some(Either::Left(idx));

}

} else {

let remote_event = event.as_remote()?;

let encryption_info = remote_event.encryption_info.as_ref()?;

if should_retry(&encryption_info.session_id) {

return Some(Either::Right(idx));

}

}

None

Changed in the latest version, but thank you - I borrowed this style for f3618bc , which is lovely.

bnjbvr · 2025-02-19T09:50:38Z

crates/matrix-sdk-ui/src/timeline/controller/mod.rs

        if retry_indices.is_empty() {
            return;
        }

        debug!("Retrying decryption");

+        let mut state = state.clone().write_owned().await;


I think it's better to have the caller pass the owned lock here; that shows that the lock will be taken, at the call sites.

Done in restructured version

bnjbvr · 2025-02-19T09:51:57Z

crates/matrix-sdk-common/src/deserialized_responses.rs

+    #[serde(default)]
+    pub session_id: String,


Should it be an Option instead of a string that defaults to "" when the data was reloaded from the store, and we didn't have it? Seems dangerous to have this floating knowledge that "" means "didn't have this information in the store"

Done in restructured version

bnjbvr · 2025-02-19T09:52:35Z

crates/matrix-sdk-ui/src/timeline/controller/state.rs

+            if let Some(item) = item {
+                if let Some(event) = item.as_event() {
+                    let sender = event.sender.clone();
+                    if let Some(remote) = event.as_remote() {
+                        if let Some(encryption_info) = &remote.encryption_info {


too many indent levels. Please use either a closure that returns an option, or short-circuiting with let Some(thing) = ... else { continue; };

I spent quite a while on trying to make this nicer in the new version - hopefully it's better but if you have more ideas let me know.

crates/matrix-sdk-ui/src/timeline/controller/state.rs

bnjbvr · 2025-02-19T09:54:44Z

crates/matrix-sdk-ui/src/timeline/controller/state.rs

+
+                            let new_item = item.with_kind(TimelineItemKind::Event(new_event));
+
+                            new_info.push((idx, new_item));


Can you replace the item here, instead of storing it temporarily? Or can't because iterating on the items themselves?

Yeah, we need to separate the immutable reference to items from the mutable reference. Hopefully the new version is a little nicer.

andybalaam · 2025-02-20T13:48:14Z

crates/matrix-sdk-ui/src/timeline/controller/mod.rs

+
+        // Retry fetching encryption info for events that are already decrypted
+        let room_data_provider = self.room_data_provider.clone();
+        matrix_sdk::executor::spawn(async move {


From @bnjbvr

anyhow, one change that should happen now, and unnoted in this review because D4mir noticed it afterwards, was that this is spawning a task that's detached; it should be attached to the timeline, one way or another:

either as a one-off, making sure there's only one at any time
or a task that listens to a channel of commands, asking for retrying decryption of a set of tasks, and running those requests linearly

Right. This PR is going to need to be rebased on top of #4715 . Marking as draft for now.

removing my review request for now, then

OK, this is now rebased on top of #4715 . I will test it today and if it works, re-ask you both for review.

This will allow us to use it to find messages that need their encryption info updating because the session changed.

…-UTDs Fixes element-hq/element-meta#2697

andybalaam · 2025-03-03T10:33:11Z

I have confirmed that this fixes element-hq/element-meta#2697 but last time I tried it, it introduced a problem when testing https://github.com/element-hq/crypto-internal/issues/398 so trying that out before I ask for review.

andybalaam · 2025-03-03T11:34:30Z

I'm still getting duplicate messages when I try out the testcase from https://github.com/element-hq/crypto-internal/issues/398 with this PR, so I've asked for help from the EX team

andybalaam · 2025-03-05T13:24:28Z

I'm still getting duplicate messages when I try out the testcase from element-hq/crypto-internal#398 with this PR, so I've asked for help from the EX team

The duplicate messages problem was not caused by this change. I will log it separately.

andybalaam · 2025-03-05T13:33:08Z

Test failure should be fixed by #4755

andybalaam · 2025-03-05T14:11:13Z

The duplicate messages problem was not caused by this change. I will log it separately.

Logged here: #4758

richvdh

LGTM otherwise

richvdh · 2025-03-05T15:01:42Z

crates/matrix-sdk-ui/src/timeline/traits.rs

+    async fn get_encryption_info(
+        &self,
+        session_id: &str,
+        sender: &UserId,
+    ) -> Option<EncryptionInfo> {
+        self.get_encryption_info(session_id, sender).await
+    }


a naive reading of this is that it is a tight recursion. I suspect I'm missing something, but a comment to explain what would help.

richvdh · 2025-03-05T15:32:07Z

crates/matrix-sdk-ui/src/timeline/event_item/content/mod.rs

@@ -551,6 +551,14 @@ impl EncryptedMessage {
            _ => Self::Unknown,
        }
    }
+
+    pub(crate) fn session_id(&self) -> Option<&str> {


could maybe do with a doc-comment?

richvdh · 2025-03-05T15:40:21Z

crates/matrix-sdk-ui/src/timeline/controller/decryption_retry_task.rs

+        if let Some(session_id) = session_id {
+            // Should we retry this session ID?
+            should_retry(session_id)
+        } else {
+            // No session ID: don't retry this event
+            false
+        }


I'd probably write this as

Suggested change

if let Some(session_id) = session_id {

// Should we retry this session ID?

should_retry(session_id)

} else {

// No session ID: don't retry this event

false

}

session_id.is_some_and(should_retry)

but ymmv

richvdh · 2025-03-05T16:23:19Z

testing/matrix-sdk-integration-testing/src/tests/timeline.rs

-    send_message(&room_for_alice, "secret message").await;
-    bob.sync_once(SyncSettings::new()).await.expect("should not fail to sync");
+    let event_id = send_message(&room_for_alice, "secret message").await;
+    while timeline.item_by_event_id(&event_id).await.is_none() {


Suggested change

while timeline.item_by_event_id(&event_id).await.is_none() {

// Wait for the event to appear

while timeline.item_by_event_id(&event_id).await.is_none() {

andybalaam changed the title ~~refactor(crypto): Store the session ID of a UTD in RemoteEventTimelineItem~~ fix(crypto): Redecrypt non-UTD messages to remove no-longer-relevant warning shields Feb 7, 2025

andybalaam force-pushed the andybalaam/redecrypt-non-utds branch from 9a19fcf to 9cbfbfb Compare February 7, 2025 14:17

andybalaam force-pushed the andybalaam/redecrypt-non-utds branch 4 times, most recently from d83e187 to 414f03d Compare February 14, 2025 11:59

andybalaam marked this pull request as ready for review February 14, 2025 12:11

andybalaam requested review from a team as code owners February 14, 2025 12:11

andybalaam requested review from stefanceriu, richvdh and bnjbvr and removed request for a team and stefanceriu February 14, 2025 12:11

richvdh reviewed Feb 14, 2025

View reviewed changes

andybalaam marked this pull request as draft February 17, 2025 12:26

bnjbvr removed their request for review February 17, 2025 13:05

bnjbvr reviewed Feb 17, 2025

View reviewed changes

andybalaam requested review from bnjbvr and richvdh February 17, 2025 15:16

andybalaam marked this pull request as ready for review February 17, 2025 15:16

bnjbvr requested changes Feb 19, 2025

View reviewed changes

andybalaam commented Feb 20, 2025

View reviewed changes

andybalaam marked this pull request as draft February 25, 2025 15:15

andybalaam mentioned this pull request Feb 25, 2025

EX: Messages sent after an identity reset can sometimes be flagged as sent from insecure device element-hq/element-meta#2697

Open

richvdh removed their request for review February 26, 2025 08:45

andybalaam added 3 commits February 28, 2025 12:49

tool(ui): Allow getting encryption info via a Room instance

8f953b9

refactor(crypto): Store session ID in EncryptionInfo

362e232

This will allow us to use it to find messages that need their encryption info updating because the session changed.

tool(timeline): Allow fetching the session ID of an EncryptedMessage

cec1c3f

andybalaam force-pushed the andybalaam/redecrypt-non-utds branch from a9331cf to 7ea45d7 Compare February 28, 2025 15:01

fix(crypto): When a session updates, re-fetch encryption info for non…

d61ceca

…-UTDs Fixes element-hq/element-meta#2697

andybalaam force-pushed the andybalaam/redecrypt-non-utds branch from f3618bc to d61ceca Compare February 28, 2025 15:09

test(timeline): Test for updating the timeline on new session info

1071441

andybalaam requested review from bnjbvr and richvdh March 5, 2025 13:24

andybalaam marked this pull request as ready for review March 5, 2025 13:24

andybalaam added 2 commits March 5, 2025 13:39

Fix typos

77edb35

Clippy fixes

5825aa8

andybalaam added 4 commits March 5, 2025 14:13

Attempt to fix flake by waiting for the event to appear

9752a02

Merge branch 'main' into andybalaam/redecrypt-non-utds

a3f39b7

Merge branch 'main' into andybalaam/redecrypt-non-utds

3fc0b0f

Add a doc comment for get_encryption_info

94b7776

richvdh approved these changes Mar 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(crypto): Redecrypt non-UTD messages to remove no-longer-relevant warning shields #4644

fix(crypto): Redecrypt non-UTD messages to remove no-longer-relevant warning shields #4644

andybalaam commented Feb 7, 2025 •

edited

Loading

codecov bot commented Feb 7, 2025 •

edited

Loading

andybalaam commented Feb 14, 2025

richvdh left a comment

richvdh Feb 14, 2025

andybalaam Feb 17, 2025

bnjbvr Feb 17, 2025

andybalaam Feb 17, 2025

andybalaam Feb 28, 2025

andybalaam commented Feb 17, 2025

bnjbvr left a comment

bnjbvr Feb 19, 2025

andybalaam Feb 28, 2025

bnjbvr Feb 19, 2025

andybalaam Feb 28, 2025

bnjbvr Feb 19, 2025

andybalaam Feb 28, 2025

bnjbvr Feb 19, 2025

andybalaam Feb 28, 2025

bnjbvr Feb 19, 2025

andybalaam Feb 28, 2025

andybalaam Feb 20, 2025 •

edited

Loading

andybalaam Feb 25, 2025

richvdh Feb 26, 2025

andybalaam Feb 28, 2025

andybalaam commented Mar 3, 2025

andybalaam commented Mar 3, 2025

andybalaam commented Mar 5, 2025

andybalaam commented Mar 5, 2025

andybalaam commented Mar 5, 2025

richvdh left a comment

richvdh Mar 5, 2025

richvdh Mar 5, 2025

richvdh Mar 5, 2025

richvdh Mar 5, 2025


		let new_item = item.with_kind(TimelineItemKind::Event(new_event));

		new_info.push((idx, new_item));

	while timeline.item_by_event_id(&event_id).await.is_none() {
	// Wait for the event to appear
	while timeline.item_by_event_id(&event_id).await.is_none() {

fix(crypto): Redecrypt non-UTD messages to remove no-longer-relevant warning shields #4644

Are you sure you want to change the base?

fix(crypto): Redecrypt non-UTD messages to remove no-longer-relevant warning shields #4644

Conversation

andybalaam commented Feb 7, 2025 • edited Loading

codecov bot commented Feb 7, 2025 • edited Loading

Codecov Report

andybalaam commented Feb 14, 2025

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andybalaam commented Feb 17, 2025

bnjbvr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andybalaam Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andybalaam commented Mar 3, 2025

andybalaam commented Mar 3, 2025

andybalaam commented Mar 5, 2025

andybalaam commented Mar 5, 2025

andybalaam commented Mar 5, 2025

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andybalaam commented Feb 7, 2025 •

edited

Loading

codecov bot commented Feb 7, 2025 •

edited

Loading

andybalaam Feb 20, 2025 •

edited

Loading