Backwardly-extensible history in bridged portal rooms (SPEC-440)

[matrixbot]

It would be lovely if bridged portal rooms that come from some bridged universe that supports history fetching (e.g. Gitter or Slack) could themselves be represented by lazy history backfill inside Matrix, matching the way that individual Matrix HSes can lazily backfill history from other HSes.

There's an annoying asymmetry in that native Matrix rooms can be lazily backfilled over server-to-server federation, but there isn't a similar ability for portal rooms bridged from a 3PN.

(Imported from https://matrix.org/jira/browse/SPEC-440)

(Reported by @leonerd)

[matrixbot]

Jira watchers: @leonerd

[ara4n]

Just had a big discussion with @leonerd and @erikjohnston on this. Quick notes were:

1. You could insert the ancient history as a big 'netsplit' in the DAG, forking from the m.room.create event, possibly putting some kind of metadata on m.room.create at room creation time in order to bypass anti-DoS heuristics for the HS sending the ancient history.

Erik doesn't like this though because it doesn't handle state nicely, and to display the netsplitted history in correct chronological order we'd need to be much smarter than the current implementation.

2. You could insert the ancient history as a 'u-turn' in the DAG at the point of the m.room.create event. I'm a bit unclear as to how the causality of the events work here and how it differs from Also send room members' presence in per-room initialSync matrix-spec-proposals#1 or Proposal for human ID rules. matrix-spec-proposals#3.

3. You could just create synthetic events which look like normal DAG events but exist before the m.room.create event. Their signatures and prev-events would all be missing, but they would be blindly trusted based on the HS who is allowed to serve them (based on metadata in the m.room.create event). Thus you'd have a perimeter in the DAG beyond which events are no longer decentralised or signed, but are blindly trusted to let HSes insert ancient history provided by ASes.

This seemed to be the preferred route, as it leaves the existing federation DAG code entirely as it is, and just has one whopping big special-case to support these "pre-history" events - in contrast with option matrix-org/matrix-spec-proposals#1 which meant improving and special-casing net-split semantics significantly to be aware of pre-history.

The convo came up based on whether ASes are better handling pre-history via a 'side-car' architecture where they just proxy Matrix straight through to the backing protocol - the conclusion is that Matrix itself needs to handle pre-history nicely, as per this bug.

[kegsay]

Uhhh, fairly sure we specced this (but not implemented) in AS API v1. See http://matrix.org/docs/spec/application_service/unstable.html#timestamp-massaging

[AndrewJDR]

I'm curious how this may fit in with message editing. Specifically @Half-Shot 's matrix-org/matrix-spec-proposals#1695 refers to m.relates_to ... could this same bit of metadata be used for inserting pre-history messages, but rather than redacting the message it relates to, it is chaining onto it?

Curious if there are opportunities for unification here on these problems.

[AndrewJDR]

Btw, I'm going to come at this from a puppet bridging perspective to make a few distinctions/points:

A facility that lazily pulls history from a 3rd party protocol on an as-needed basis is actually a bit different (and probably more) than what some of us on the puppet bridge side really want. That would entail matrix HS somehow brokering "I need some more history!" messages from the matrix client as they're scrolling back.

I'll break down the use cases I know of for back filling. a and b apply to puppet bridges, and c applies to (i think?) non puppet bridges.

There are two main use-cases I can think of:
a) On a 3rd party protocol, a message send failed, and the user clicks "Retry failed message attempt" within that protocol's app: We now need to insert that message back into its original context (having it appear at the end of the chat obviously makes little sense as many other messages may have been received in the intervening time). This needs some form of DAG back-insertion (i just made that up) or perhaps just some m.relates_to-like facility similar to what's been proposed for message editing, but slightly adapted for this usecase. This is not something that would need to happen a lot, just occasionally for a message or two here and there.

b) Backfilling the entirety of the old history upon initial creation of a bridge room: We can handle this on the bridge AS during an initial room population sequence when the bridge starts up. We just pull ALL the history and insert it in order. This way, DAG order matches chronological order. and we needn't have this ability to insert a whole slew of events "out-of-order". For many of us this is actually better than and preferred over c. (below), because part of the promise of matrix puppet bridging is to maintain a full, searchable history of "other protocol communications" hosted on a server that we have control over. If that proto shuts down (i.e. like hangouts is about to do next year), we still have our history. As far as I know, this is completely possible with current synapse, as it supports timestamp massaging (but not "DAG order massaging"; which is fine, because we're not using that here)

c) The dynamic-history-backfilling-as-user-scrolls-backward-in-their-client usecase. I think this would need a more exotic solution like some of those being proposed. I'll be honest -- I have no interest in this personally, because it ruins the aforementioned great promise of matrix puppet bridging (as mentioned in b.): maintaining searchable history on a server that I control. It also seems more complicated to implement than a/b, since it has to manage brokering of "I need more history!" events from matrix client, to matrix HS, to matrix bridge, to protocol, and then responses backward through that chain. I imagine this is more useful to those that are not doing puppet bridging, but are doing "regular bridging" (a la matrix-appservice-discord bridge), but somebody that does that kind of bridging can chime in and correct me if I'm off base here.