Room events arrive faster than to-device messages during federation connectivity problems, causing decryption failures

[turt2live]

Suppose server A is struggling to send federation traffic to server B. (This might be because B is slow, is having connectivity problems, or there is just a huge queue of traffic.)

In this situation, B will often still receive encrypted messages from A: they may be pulled in from other servers in the room. However, the keys to these messages will be stuck in a queue on A. Users on B therefore see decryption errors.

ref element-hq/element-web#3754

[MadLittleMods]

Related to #966 (potential duplicate)

[richvdh]

Related to #966 (potential duplicate)

It's not a duplicate: #966 talks about the ordering of to-device messages relative to one another, while this is about to-device messages relative to room events.

[dkasak]

According to https://matrix-org.github.io/synapse/dev-docs/latest/modules/federation_sender.html#a-note-on-failures-and-back-offs, Synapse already mitigates this to some degree by restarting the to-device transmission loop once it receives an inbound request from the remote server.

[ara4n]

The core problem here is that todevice messages don't flow transitively, but timeline msgs do. So if server A can't route to server B, messages may all flow transitively A->C->B - but the keys will never get through, no matter what the retry schedule is.

Possible solutions off the top of my head:

• Forbid transitive backfill for encrypted rooms
• Let server B request missing todevice traffic from server A via server C. So:
  • server B realises it's receiving current events from server A via server C
  • server B assumes that A can't route to it directly
  • server B asks server C to ask server A for B's todevice msgs.

The latter is more fiddly, but preserves Matrix's somewhat desirable self-healing transitive delivery properties.

Alternatively, does MLS solve this by putting the key data in the timeline rather than todevice msgs? (cc @uhoreg)

[uhoreg]

Most of the MLS stuff happens in-room. The one thing that can happen in to-device messages is Welcome messages, but those could also be sent in-room. We haven't figured out if it would be better to use to-device messages or in-room messages for that.

[dkasak]

Presumably Matrix could've gotten away with putting it key data into the room too but for some reason chose not to. Are there any concerns with MLS key data being stored in the DAG forever? E.g. bloat or problems arising out of making permanent something that should be ephemeral?

[uhoreg]

Yeah, the main concerns about storing the MLS Welcome messages in the DAG are bloat-related. It would be irrelevant information to most of the people in the room.

[uhoreg]

We may be able to flag events that we got relayed from a different server, so that clients can indicate that there may be a decryption failure.

[kegsay]

I think there are two separate issues here:

• network partitions over federation i.e no amount of waiting will give you the to-device msgs.
• temporary lag where to-device msgs take ages to arrive.

The latter is quite frequent, and we see this (almost) daily between element.io <--> matrix.org. If we renegotiated room keys less frequently (or had more time to let the room key arrive by not sending it when typing..) this would concretely help I think.

[uhoreg]

(or had more time to let the room key arrive by not sending it when typing..)

I don't understand what you mean by this. By sending it when typing, that means that we send it earlier, which means it has more time to arrive before the room event is received?

[BillCarsonFr]

Alternatively, does MLS solve this by putting the key data in the timeline rather than todevice msgs?

A document on that subject was created https://docs.google.com/document/d/1f5XMf4qXEzoUBEFDu5IWJXvFAc76D9_dwqt0ReoCRpw/edit#heading=h.a4ncf0eur6o2

[turt2live]

MLS key data would most likely be sent over to-device in the linearized matrix models we've been discussing. The guarantees for delivery are a bit higher, though.