Let devices request keys to decrypt past history for a room.

[ara4n]

If I'm in a room as device A, and join it later as device B, i need a way to request the old room key data from device A.

Similarly if I'm invited into a room by someone and they want to share past history with me, they need a way of transferring key data.

[apexo]

Could a possible solution to this be:

• create a special (maybe hidden per default) per-user channel on the user's home server
• each new device sends an (encrypted) poll message to all other devices (via this room)
• all other devices reply with the encrypted e2e state (sans identity keypair), or just all keys required for decrypting room history

[ara4n]

@apexo this is sort of the plan, except we now have a dedicated "to device" store-and-forward API for sharing data like this, which is encrypted in turn by olm. Also, we'd probably request the historical ratchet state from a specific device rather than all of them, and require the target user to approve/deny the request. e.g. "Matthew has added a new device 'iPad' and is requesting room history since Sept 18th. Do you want to share history?" or something.

[DrFlamingoo]

@ara4n So I am currently in a room listed with seven devices from three machines and as many browsers. well, I guess vector --> riot transition helped to increase the number. But with only one other person in the room, this is cumbersome.
I would prefer if a client-side certificate could be recognized as a user, and verification becomes an automated process that does not rely on devices or interaction of members in the room and I just see the history. Would work on private devices (i.e. not the library computer).
To understand the process, how are the IDs generated, and how long are they kept / stored? Do they change with version change of kernel, browser or if I purge cookies? I am pretty sure I am missing discussions on the same device I logged in at the time...

PS: I haven't seen another issue / discussion on this matter, so I hope this is the right place to put this :)

[madduck]

Maybe look at Bitcoin wallets, e.g. http://wallet.counterwallet.io, which ask a 12 dictionary word passphrase from the user. That passphrase is the private key.

Now when I log in to a device, I could be asked if I wanted to create a device-specific key, or if I wanted to instate the user's key from such a passphrase.

[csmart]

Can I check my understanding please, does the following represents current expected behaviour?

Say I'm in a room which has encryption turned on and we're all happily chatting away and everyone has verified keys. Then I sign out of riot and back in to see that all previous conversations are encrypted again and that there's no way to get the original key to you, so history remains unavailable?

Thanks!

[madduck]

It doesn't matter whether keys are verified. The point is that the encryption algorithm (megolm/ld double ratchet) encrypts to the current keys of all the people present.

Now if you log out of Riot, your key is discarded from local storage, and upon rejoining, a new key is made. Future messages will now be encrypted (also) to that key, but since you no longer have access to the previous key, the old messages aren't accessible any longer.

The solution is IMHO to allow one to export/safe/backup one's key and restore it upon rejoin.

[csmart]

Right, thanks @madduck

[ghost]

I've been having major usability problems with end to end encryption in riot, and I didn't realize that it was because logging out deletes your session key! riot-android/#869 would be nice, because I had no idea!

Now that I understand why I've been losing my keys -- and being locked out of my chat history -- I have to say I'm really annoyed. I'm sure everyone agrees you shouldn't have to stay logged in to your chat program continuously in order to be able to use it reliably. It's one thing to hide past messages from newcomers to a channel, but this same issue means that I can't share decryption keys with myself! In my opinion, E2E should not have been rolled out as a feature before this problem was solved. It's been a trainwreck for everyone I know who's tried turning it on.

[csmart]

@kebertx Aside from this issue (which I now understand) e2e has been awesome for me and those I've been chatting with.

IIRC, it was advertised as beta and at least in the Android app it says it's that e2e is an "experimental labs feature which may break in unexpected ways." So I think the warning has been fair enough.

Having said that, I do think things can be improved (a simple warning about losing keys when signing out, for example would be handy for new users).

IMO, it's been great to have the feature early and be able to test it.

[madduck]

Hey @kebertx, sorry you've had a bad experience. It is true that e2e at the moment is highly beta, and things certainly need to be improved, which is, in part, why this issue exists.

FWIW, I see no need to ever log out. I do have riot.im exempted from my cookie munger, and so I don't have this issue. Similarly, I don't log out the Android app, but the problem will come when I need to switch to a new phone, and by then I hope they've got a fix.

[ara4n]

(fwiw the warning was added)

[irunonayran]

I don't see a need to log out either, if the app supports multiple identities, which riot seems to not do, so logging in and out would be a workaround for me. Alternatively I could use a second android user, but that drains my battery more.

[pik]

@madduck You could also go the other-way an export an existing private key as a mnemonic - e.g. BIP39 uses a 2048 word list (https://github.com/bitcoin/bips/blob/master/bip-0039/english.txt) which is sufficient to store 128-bit keys (2048^12). Although one issue is that ed25519 and curve25519 keys are I believe independently derived in Olm? So you would need mnemonics for both, or support deriving one from the other.

@ara4n

Similarly if I'm invited into a room by someone and they want to share past history with me, they need a way of transferring key data.

I'm not sure I understand the original issue very well, can't an existing member just key-share all of the session_keys they have received from other members in that room via. the same sendToDevice api using an Olm Session as is normally done when a client shares their outbound session key?

[ara4n]

having read through prior comments - most of them don't relate to this bug, but are general UISI concerns (#2996) or key management concerns (#3611); please upvote and/or comment on the right bugs rather than using this one as a dumping ground O:-)

As per element-hq/element-web#2996 (comment) we're moving on and implementing this asap, both to provide an escape hatch for cleaning up remaining UISIs as well as to provide the feature of syncing history to new devices without having to manually export/re-import megolm session keys. @richvdh, @NegativeMjark and I just went through the discussion of how this should work, resulting in the plan:

• If a client receives a message which it doesn't have megolm session keys for (i.e. a UISI), it should send to_device messages to all devices (i.e. "*") belonging to themselves and the sender, requesting the missing key.
• By default senders will automatically re-share megolm keys to devices they have verified (if they sent the message to that device, or it's their own device).
• If the device isn't verified, they have the option to verify it before sharing, or to ignore the request (and/or blacklist the device?)
• We'll also provide a setting called "Do not automatically share E2E keys with new devices" in settings, so that paranoid users can explicitly authorize this.

Separately, we will obviously need to improve the verification UX, e.g. by cross-signing devices to automatically trust them, QR codes or mnemonics for verifying fingerprints, etc. This is all element-hq/element-web#2142 however and orthogonal to the problem of actually sharing keys.

We also discussed whether we should store megolm keys on the server, to try to both improve clientside key storage capacity problems (#3660) or to provide an easier way of backing up keys and/or sharing them - this is element-hq/element-web#3661. We concluded that this is basically an optimisation for key storage and wouldn't change the semantics of the crypto at all (i.e. if you're running out of clientside storage, you're welcome to use an encrypted cloud storage service to store data, but that's not going to change our E2E behaviour).

Finally, we considered the edge case that Alice and Bob are both pretty paranoid, and only ever use Matrix by firing up an incognito window, logging in, sending a message, and closing the tab (not implausible). As a result, at any given point they probably wouldn't have any devices live. Therefore when Alice sends a message to Bob, she won't know how to encrypt for him, and when Bob gets a UISI, there won't be any devices of Alice he can query for the megolm keys (unless Alice happens to be logged in and having re-imported her megolm keys at the moment that Bob requests history, or perhaps later). We concluded that there were two possible solutions to this:

1. Introduce the concept of 'virtual devices' so that Alice & Bob can maintain a virtual device which is stored encrypted on the server, and is passed between physical logins, letting them pick up conversations when desired - this would almost act as an 'identity key'.
2. Introduce the concept of deliberately 'dehydrating' and 'rehydrating' devices, so that when Bob stops using his Incognito window, he could explicitly export that device intact onto a USB stick or whatever, and subsequently rehydrate it next time he opens an incognito tab - effectively deliberately backing up & unbacking up the device.

In the end, both of these are tantamount to the same idea, whether the device is stored encrypted on the server or on a USB stick. Both suffer from element-hq/element-web#3822 - that if two identical devices ever exist concurrently Olm will completely wedge and E2E will break. So the conclusion there was to educate users that they need to keep at least one device active if they expect E2E to work, unless we ever get as far as dehydrating/rehydrating devices.

@pik: hopefully this resolves your confusion a bit :)

[uhoreg]

Will there also be a button somewhere to request all the keys from one of your own devices, in order to aid migration from one device to another? Without this, it sounds like when people want to replace one device with another, people would either need to manually export/import the keys (a bit less convenient than just clicking a button to request keys), or else go to each encrypted room and backfill to the beginning so that the device automatically requests keys (a lot less convenient).

[ara4n]

@uhoreg possibly. i'm a bit scared at how powerful such a button would be; at least an attacker needs to know the IDs of the megolm sessions they're trying to get keys for in the "send_missing_keys" to_device event outlined above. Whereas an "send_all_keys_ever" feels like a terrible thing to fall into the wrong hands...

[richvdh]

Didn't this get fixed?

[pmaier1]

Can we close this in favor of #749? To my knowledge element-hq/element-web#749 will be the way to go here and we want to schedule it soon(tm).

[richvdh]

element-hq/element-web#749 is an epic describing a desired end goal rather than a particular solution, whereas this was more an idea for a particular implementation.

Nevertheless I still think this has been done and should be closed; if anyone disagrees, we can reopen with more context.

[karolyi]

Pardon my intrusion but in which version did this get solved? I'm using 1.71 and it still doesn't work there.

[richvdh]

Pardon my intrusion but in which version did this get solved? I'm using 1.71 and it still doesn't work there.

Version 1.71 of what? Note that this is the element bug tracker.

The fix landed in Element Web in matrix-org/matrix-js-sdk#454. You're likely seeing a different problem; feel free to open a new issue and send bug reports from both the sender and recipient applications (see also #245)

[karolyi]

Oh sorry, the ecosystem around matrix is a bit convoluted. I'm using synapse 1.71 and element-web 1.1.17 and we're still seeing this problem.

[HarHarLinks]

Based on the problem statement in OP (without reading the whole thread) and the explanation in #647 (comment), I don't think this is "solved", merely worked around in one specific setting. To me the wording of both implies that element-hq/element-web#749 duplicates this issue (but is phrased more clearly).

[uhoreg]

No, this issue is for getting keys for messages sent while you were not in the room. For messages sent while you were in the room, and that other devices of yours can decrypt, it should be handled by key backup.

[jittygitty]

This issue started in 2016, any update? Is this possible now? I just had another friend want to QUIT using element chat!
He LOST his keys and everything, he does NOT want to install it again without way to regain access to chats!

I was basically in the chats with him, and my user/device should be able to send him keys to decrypt past messages.
I use Element Desktop. Is there a way he can recreate/reset his "user" and then allow me to send him decryption keys?

[mpeter50]

jittygitty, I'm really sorry it got to be this long.

This issue started in 2016, any update? Is this possible now?

As I know it was done in Element web (maybe desktop and android too), in a special way.

When you, from an Element client, invited a user, your client has sent all the keys for past messages (of course only those that it has known) to the invited user. Maybe they also needed to be using a device with an element client, possibly with a version that has implemented support for it. I think this was the MSC proposal for it.
For when a member of the room has lost the keys, the workaround to send them would be to kick them (or have them leave) and then invite them, and before you say this is silly, it is but there was a reason it worked this way, and it is related to security.

However, the clients since then have switched to the more efficient Rust-based cryptographic library, and this feature was not (yet?) reimplemented there. See the "Sharing room history" heading under "Notes" here.
There is a github issue in the Rust SDK repo for implementing the feature, and it is part of the "stabilize the crypto crate" milestone, so it will be done at some point, I guess? It also has a PR with an implementation, but it seems review has been stuck.
Element web also has an issue tracking this, it is currently waiting for the Rust SDK to implement it.
Last week it was also mentioned in the Element iOS repo, the comment might shed some light on what is happening.

I would appreciate if an Element employee could tell if this is accurate.

He LOST his keys and everything, he does NOT want to install it again without way to regain access to chats!

I totally agree that it is very uncomfortable. This is one of the strongest reasons why I don't yet bring people onto this platform, because I know this will bother them, and as it stands now, virtually everyone will experience it at least once.
However, let me tell you how do I look at it.

What is Matrix, when looking at its end to end encrypted rooms?
It is basically a Signal, but with the possibility of logging in on multiple devices (without requiring constant direct connection between them, e.g. at the home LAN like Signal and SimpleX "solves"' this), and even being able to read and send new encrypted messages.
Because of that it is (almost) a Telegram when looking at usability, but with security included. (except client performance, that is nowhere near it..)

So basically, I think Matrix is just more, because it allows more flexibility than Signal.
It is still more even when the encryption fails (safely), because you can still exchange new messages in that room, or participate in unencrypted rooms, or the other ones where encryption still works. In the absolute worst case, when you somehow lose all keys, it just falls back to a little bit above the level of Signal, where the keys are only stored on your single phone, and if you somehow lost them, you lost keys to all of your messages.

Why is Matrix miserable at implementing this reliably? I think it may be related to the reason why is it that they are the only one doing E2EE with multiple simultaneous logins at all: handling keys in this manner is very hard.
Most of the issues around encryption in Matrix are about keys not being there where they should, because they were lost somehow, and theres a million ways that could happen.

• It was lost on client storage,
• they were not persisted to client storage (because of a crash, killed process, platform crash, power loss or anything at the wrong moment),
• they were persisted but the client did not have the time needed to upload them to key backup before it was closed, and did/could not send to other devices either for some reason,
• they were lost in transit and not available anymore,
• many other reasons probably

Often this will fix itself because one of your devices has the key, but it is not online to respond to key transfer requests of your other device. At least they dont need to be online at the same time.

But this case is probably not even about a key handling bug that is that much complicated.
How did he lose his keys? Did he uninstall the app, or reset its data from the platform's settings, or what has happened?
Did he set up key backup? Probably not, because the clients do not push that hard enough, and I can imagine the prompt is just dismissed without a though (or is it even shown at all for new people?).
This is not about blaming the victim, but honest questions that need a response, for the problem to be solved or avoided in the future.

You know, when the same thing happened to me and got to learn the lesson, my account was fairly new and I didnt lose much.
But I too was very frustrated when I realized that new members of the encrypted room, not even just cannot read the former messages, they dont even see there are any because the client "helpfully" hides them. Even worse, it was with people I had barely known, I was just organizing a team! And then when they lost their keys because they reinstalled the app, switched phones and didnt use a desktop, or whatever, and werent able to see old messages..
But my point is, that once you understand that it is still a better outcome than what Signal or such could have allowed - because there the same user-stunt would have resulted in total loss of all messages in all chats - it becomes more clear that sometimes the problem is between keyboard and chair. Its a bit like not knowing your google password, wiping your phone, and being surprised that you cant log in to your gmail inbox or install your purchased apps.

Often this is the reason software orgs are afraid from implementing complicated features.
Not just because they lose reputation if it is buggy. But also because if the user doesnt see that doing specific things is dangerous, they will become angry when it does not work as they have expected. And communicating that it does not work that way is very hard, even more if we think about that most users just dismiss any dismissable dialog, and complain if it is not dismissable, but at the same time dont forget that most of the problems are not even coming from pressing buttons in a matrix client, but instead it being affected by its environment: getting suddenly closed and such.
See the situation of Matrix: Signal users mostly understand (I think?) that the messages are stored on their devices, and can conclude that deleting the app without a backup or losing the phone results in total loss of data. On Matrix, users understand that their data is stored on a server, but its not evident that access to part of it depends on a complicated thing called encryption, and that things are needed for it work that are never stored on the homeserver, for the encryption to not be nullified. And in turn, they expect that everything is stored on the server, like with an unencrypted messenger, and they expect that uninstalling the app can not have consequences. When they lose their keys, on next login they still see their old chat rooms and in case of unencrypted rooms even the contents, and they think that its a critical bug that they cant see their encrypted messages.

This got to be very much too long, but it is not a scolding or anything like that, I was just hoping I could give my perspective, in case it helps.
The most important thing about it is that (in my opinion) it must be kept in mind, that in the absolute worst case, you can lose at most as much as on Signal, but when things go wrong its often less. If you were using Signal, your friend would have lost all the messages anyway, probably (probably, because we dont know what caused this).
And also, seeing "unable to decrypt" does not always mean lost keys, but that your device did not have yet received the right keys. Maybe it is on the way among a thousand others and it takes time to process them, maybe the only device holding it has not been online since the request.

For the long term, I very much hope that we will get the possibility to send the keys for former messages to the other members.
I have read spec proposals that vision this happening even automatically, but they said a lot of changes need to be made until that can be done securely.

[jittygitty]

mpeter50 thanks for the very detailed write-up. I decided that until the complex crypto is stabilized that I will "only" be using encryption on rooms/chats with users on home servers I do not control. When I use on my own server, I think it makes more sense to simply encrypt the underlying storage via dm-crypt/Luks or other, and as long as everyone in chat is on the same home server, it doesn't seem much benefit to enable encryption at all in such cases, so I won't enable it.

Of course, for federation uses and connecting to home-servers you don't 'fully trust', encryption is "indispensable".

Also, t3chguy replied to me at #245
Anyone know what clients currently make use of matrix-org/matrix-js-sdk#454 ?

[mcg-matrix]

I was basically in the chats with him, and my user/device should be able to send him keys to decrypt past messages. I use Element Desktop. Is there a way he can recreate/reset his "user" and then allow me to send him decryption keys?

You may be able to transfer the keys to your friend in a file:

1. Create an export of all your keys (to a file on disk); in Element-Desktop this is called "Export E2E room keys".
2. Extract the keys you want someone else to have (by room id):
   • there's a tool called megolm_backup.py that can read the file with the exported keys, see https://github.com/cyphar/matrix-utils;
   • the filtering would be done by piping through something like jq 'map(select(.room_id == "!herestheroomid:example.org"))';
   • and again megolm_backup.py for writing the new file with only the extracted keys.
3. Pass a copy of the file with the extracted keys to your friend using some confidential way (though I believe the keys are protected by a password), e.g. in a Matrix room.
4. Let your friend import all keys from the file you gave him; in Element-Desktop this feature is called "Import E2E room keys".

Instructions for exporting and importing room keys in various versions of Element: https://ems-docs.element.io/books/element-cloud-documentation/page/export-and-import-e2e-room-keys

[jittygitty]

@mcg-matrix Awesome! Thanks for taking the time to share that! I'll try convincing my friend to give it another shot and reinstall element chat, and then will try that!

[mrlogick]

I really don't understand. If you are able to export the keys and manually give them to the other user, why can't you just make a simple button?
The user with the past keys should click the button to send the keys to the server (but encrypted). When the other user goes online he gets the past keys from the server (and decrypt them).
On Element there was already a button saying something like "Unable to decrypt message. Request keys...", but for some reason never works.

[mpeter50]

I really don't understand. If you are able to export the keys and manually give them to the other user, why can't you just make a simple button?

I think that would introduce a security vulnerability, backed by social engineering.
To begin with, the reason this is not done automatically is to protect the users from a malicious homeserver. A homeserver admin can add any user to any room (with database editing or modifying the server software), and if this would happen automatically then a new, "forged member" would receive the keys for past messages, nullifyng encryption. Its possible that only a single member would notice this because the server picked him and didnt tell the others, maybe even eradicate the events afterwards.

You suggest this to be done manually, but that has similar issues.
A hostile homeserver could take over your friend's account, or just not deliver his messages to you and instead forge custom messages in his name. By doing this, the admin could convince you to press this button to send him keys for past messages. Again, encryption nullified, because the user does not know better either.

If I can recall correctly, the rationale behind the past workaround (sending the keys to the user being invited by you) was generally that the inviting user (who still has the keys) can verify that he account he is inviting is actually the account of his friend, but I dont remember the specifics.

So to answer your question, this is actually safer because the homeserver operator cant fool you to spill your keys to him. This way you can transfer the keys to your friend "out of band", in a channel of your choosing that is not affected by the adversary, lets say on a pendrive handed over in person, or a shared encrypted cloud storage folder (but still making sure in person that its really them who requested your keys).
Also, exporting the keys is more cumbersome than pressing a handy "send all keys" button, so the user has time to realize that they should do this carefully.

Additionally, even if you believe that you have choosen a homeserver in which you trust, it could get compromised, or even confiscated by adversaries. Probably this does not affect most people, but its always good to be aware of the risks.

On the other hand, sharing your keys this way is risky because if you did not filter the backup correctly, your friend will also receive keys for other messages of yours.

On Element there was already a button saying something like "Unable to decrypt message. Request keys...", but for some reason never works.

I think that button just instructs your client to ask all your other logged in devices if they have the keys for that message.

[jittygitty]

@mrlogick Actually I was thinking the same. If this works just fine, then I don't see why users can't be given the option. I think the "user is too stupid" mantra is usually taken too far and ends up with oh let's lock out users out of their own $1500 device/phone that they bought, no Admin/root access to users, or they'll uninstall factory spyware? Encryption on phones with modems and baseband/master processors without true isolation, gives a "false" sense of security, agreed?

And so, while I sympathize with the points mpeter50 made in his reply to you, and I'd agree such measures shouldn't be done "automatically or even too easily" such that it could be accidental, a good trade-off 'balance' should be attainable.

I really wanted to try out Kopia soon, and now I want to try bupstash.io also!

But tell me, why force encryption like Kopia also? Why not let us skip it? Sometimes I just don't want encryption and I don't like being 'forced' to use it. It adds more complexity and harder recovery. And isn't it easier to do deduplication, or do it better even (not really sure how dedupe works with encryption), if there's no encryption enabled? Sorry I digressed.

[murlock1000]

The recent migration of element-web to the new rust crypto system without supporting MSC3061 has broken some of systems that relied on this feature. Specifically, the functionality where a bot would invite whitelisted users to a room and share room keys with them so they could see historic messages.

What is the current solution for an automated system that invites users to a room with existing messages and enables them to see historic messages now?

[mpeter50]

If this works just fine, then I don't see why users can't be given the option. I think the "user is too stupid" mantra is usually taken too far

Because its not necessarily a stupid user. Maybe one with a very short attention span or one who really is in hurry, and does not read the option but just proceed out of habit. The former is not a rarity, to say the least. I have had a hard time believing this until I have seen this happen regularly with my eyes. Not in the context of Matrix, but with tech, the simpler kind. A lot of people also just dont read, but rather agonize, literally. And in either case I'm not speaking about the elderly.

Having it behind 4 button presses, plus a 10 second "thinking timer" like some android phones do with security settings would probably work fine, though.

But tell me, why force encryption like Kopia also? Why not let us skip it?

You can skip it at room creation.

If your primary DM chat room's encryption breaks, you can create a new unencrypted one until you sort that out.
But a malicious homeserver may be able to get you into that state (e.g. by not forwarding to-device messages for you), so think twice before you do that.

What is the current solution for an automated system that invites users to a room with existing messages and enables them to see historic messages now?

This is nothing official and it would be much better if Element employees could give advice.
But as a workaround you could use an element client with the old crypto to join that room. The desktop app or a hosted web client, probably an older android app would work too.
This client would receive the keys, and hopefully it will be able to send them to your other clients, or upload it to key backup from where the others can download it.

---

I once again want to highlight that I am not an Element employee, I just wanted to give my 2 cents because for some issues I have already found a solution.

[jittygitty]

@mpeter50 Sorry for the confusion, you misunderstood my reply to @mrlogick because the entire section starting with "I really wanted to try out Kopia soon, and now I want to try bupstash.io also!" and ending with "Sorry I digressed" had nothing to do with Element chat at all, it was only about his "bupstash.io" project ( see https://github.com/andrewchambers/bupstash )

I knew that encryption in rooms is optional, I now no longer use it in some rooms which are hosted on my server (with encrypted drives) only and for which there's no federation and if only users on my server are allowed to join that room.

Anyway, I had already said that: "I'd agree such measures shouldn't be done "automatically or even too easily" such that it could be accidental, a good trade-off 'balance' should be attainable." So, of course I am perfectly fine to require a user to "TYPE" a long and exact phrase saying: "I Agree this is Dangerous and I insist on etc etc and know what I'm doing.."

Point was there's definitely ways like making people type something out which informs them that this is for advanced users and can be security risks involved etc, and such that nobody would be able to say they enabled it by "accident".

I'm just against policies that say, nope we're just not gonna give anyone such options, they'll have to go fork da code.

Anyway, I haven't had a chance yet to try this: #647 (comment)
Because I'm still trying to convince my friend to reinstall Element and give it another try, hope I'll succeed convince him.