Eager federation /backfill mode

[erkinalp]

Description:

When enabled, Synapse should pull the events from the federation eagerly, regardless of whether clients requested something requiring events to be fetched. May be useful in certain high volume applications to even out the load throughout the day.

[anoadragon453]

Synapse already attempts to have the full state graph of a room that it is joined to, so there's nothing more to eagerly fetch.

Are you talking about non-state event fetching methods, such as /backfill?

[erkinalp]

@anoadragon453 Yes, fetching all events eagerly.

[MadLittleMods]

This is potentially interesting for the Matrix public archive so that our archive homeserver can have most dates ready to load and help with sort issues with MSC2716 historical messages if we don't pull directly from gitter.im,

Plan forward
[...]
2. For Gitter and the static archives, we can have a script manually scrollback across all of the rooms in the archive server before anyone else or Google spider crawls in some weird way. This way it will lock the sort in place for all of the historical messages. Or fetch directly from the gitter.im homeserver

-- matrix-org/matrix-spec-proposals#2716 (comment)

[MadLittleMods]

Copying some thoughts from the duplicate issue I created:

Currently, we only backfill when we receive a /messages?dir=b request which means any time you want a message and we don't already have it or there is a gap at the time you're paginating over, you have to wait for the /backfill request in the background before we can finally respond to your initial /messages request.

If we instead first attempt to backfill outstanding backward extremities before a /messages request comes in, we will either have the event in the best case, or recorded some event_failed_pull_attempts and we already be ready to backoff and skip the same event when we try to paginate around it later.

Ideally, we would be able to backfill more than real-time for all of the rooms and eventually catch up with everything possible to retrieve.

---

Also relates to Synapse trying to make a Matrix room eventually consistent on its own.

-- @MadLittleMods, #15652